---
title: 'Running AI Models Locally via Llama.cpp in fresh Ubuntu'
url: 'http://jayeshmahato.com/blog/technology/running-ai-models-locally-via-llama-cpp-in-fresh-ubuntu'
markdown: 'http://jayeshmahato.com/blog/technology/running-ai-models-locally-via-llama-cpp-in-fresh-ubuntu.md'
date: '2026-07-08'
description: 'Running AI Models via llama.cpp in Fresh Ubuntu (CUDA 13.1 + RTX 5070 Setup) Artificial Intelligence is becoming more accessible than ever, and with powerful GPUs like the RTX 5070, you can now run advanced AI models completely locally on your Ubuntu machine without relying on cloud services. In th…'
taxonomy:
  archives_month:
    - jul_2026
  archives_year:
    - '2026'
---

## Running AI Models Locally via Llama.cpp in fresh Ubuntu

# Running AI Models via llama.cpp in Fresh Ubuntu (CUDA 13.1 + RTX 5070 Setup)

Artificial Intelligence is becoming more accessible than ever, and with powerful GPUs like the RTX 5070, you can now run advanced AI models completely locally on your Ubuntu machine without relying on cloud services.

In this guide, I’ll show you how I transformed a fresh Ubuntu installation into a full local AI workstation using:

- NVIDIA CUDA 13.1
- llama.cpp
- RTX 5070 GPU acceleration
- Hugging Face GGUF models
- Gemma 4 Vision support
- Localhost AI chat server

By the end of this tutorial, you’ll be able to run multimodal AI models directly on your Linux machine with full GPU offloading.

---

# Why Run AI Models Locally?

Running models locally gives several advantages:

- Complete privacy
- No API costs
- Lower latency
- Offline usage
- Full control over models
- Open-source ecosystem

With modern quantized GGUF models, even consumer GPUs can deliver impressive AI performance.

---

# System Used in This Tutorial

## Hardware

- NVIDIA RTX 5070
- 12GB VRAM
- SSD storage

## Operating System

- Ubuntu 26.04 LTS

---

# Installing NVIDIA Drivers on Ubuntu

First, detect available NVIDIA drivers:

```bash
ubuntu-drivers devices
```

Ubuntu recommended:

```bash
nvidia-driver-595-open
```

Install it:

```bash
sudo apt install nvidia-driver-595-open
```

Reboot after installation.

---

# Verifying GPU Installation

Check that the GPU is detected properly:

```bash
nvidia-smi
```

If everything is working correctly, you should see your graphics card like RTX 5070 listed.

---

# Installing CUDA 13.1

Ubuntu repositories may not contain the latest CUDA packages, so install CUDA directly from NVIDIA.

Download CUDA from:

<https://developer.nvidia.com/cuda-downloads>

After installation, verify CUDA:

```bash
nvcc --version
```

---

# Fixing CUDA + GCC Compatibility Issues

While building llama.cpp, Ubuntu 26.04 introduced a compatibility issue involving:

```text
rsqrt
rsqrtf
```

inside CUDA math headers.

The fix involved:

- installing GCC 14
- explicitly setting CUDA host compiler
- patching CUDA headers with `noexcept(true)`

This resolved the build errors successfully.

---

# Building llama.cpp with CUDA Support

Clone the repository:

```bash
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
```

Configure build:

```bash
cmake -B build \
-DGGML_CUDA=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/gcc-14 \
-DCMAKE_CXX_COMPILER=/usr/bin/g++-14 \
-DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-14 \
-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
-DCMAKE_CUDA_FLAGS="--compiler-bindir=/usr/bin/g++-14"
```

Compile:

```bash
cmake --build build -j$(nproc)
```

After successful compilation, binaries like:

```text
llama-cli
llama-server
```

become available.

---

# Organizing AI Models

I created a separate folder for models:

```bash
mkdir -p ~/AI/models
```

Keeping models separate from the source code helps maintain a cleaner setup.

---

# Installing Hugging Face CLI

Using `pipx` is the cleanest method:

```bash
sudo apt install pipx
pipx ensurepath
source ~/.bashrc
```

Install Hugging Face CLI:

```bash
pipx install huggingface_hub
```

---

# Downloading Gemma 4 GGUF Model

I used Gemma 4 E4B GGUF from Hugging Face.

Download command:

```bash
hf download \
lmstudio-community/gemma-4-E4B-it-GGUF \
gemma-4-E4B-it-Q4_K_M.gguf \
--local-dir ~/AI/models
```

---

# Running Gemma 4 in Terminal Mode

Launch with full GPU offload:

```bash
~/llama.cpp/build/bin/llama-cli \
-m ~/AI/models/gemma-4-E4B-it-Q4_K_M.gguf \
-ngl 999 \
-c 16384
```

Explanation:

- `-ngl 999` → full GPU offload
- `-c 16384` → 16K context size

---

# Running AI Models on Localhost

One of the best features of llama.cpp is the built-in local server.

Start the server:

```bash
~/llama.cpp/build/bin/llama-server \
-m ~/AI/models/gemma-4-E4B-it-Q4_K_M.gguf \
-ngl 999 \
-c 16384 \
--host 0.0.0.0 \
--port 8080
```

Open browser:

```text
http://localhost:8080
```

Now you have a complete local AI chat interface running entirely on your machine.

---

# Adding Vision Support (Multimodal AI)

To enable image understanding, download the projection model:

```bash
hf download \
lmstudio-community/gemma-4-E4B-it-GGUF \
mmproj-gemma-4-E4B-it-BF16.gguf \
--local-dir ~/AI/models
```

Run server with vision support:

```bash
~/llama.cpp/build/bin/llama-server \
-m ~/AI/models/gemma-4-E4B-it-Q4_K_M.gguf \
--mmproj ~/AI/models/mmproj-gemma-4-E4B-it-BF16.gguf \
-ngl 999 \
-c 16384 \
--host 0.0.0.0 \
--port 8080
```

You can now upload images directly into the localhost chat UI.

---

# GPU Monitoring

Monitor GPU usage live:

```bash
watch -n 1 nvidia-smi
```

You should see:

- VRAM allocation
- CUDA utilization
- active GPU inference

---

# Performance on RTX 5070

The RTX 5070 handled Gemma 4 E4B extremely well.

Highlights:

- smooth inference
- full GPU offloading
- responsive localhost chat
- stable 16K context
- efficient VRAM usage

This setup is powerful enough for:

- coding assistants
- local AI chat
- vision analysis
- multimodal workflows
- offline AI tools

---

# Final Thoughts

Local AI on Linux has become incredibly powerful.

With:

- Ubuntu
- CUDA
- llama.cpp
- GGUF models
- RTX GPUs

you can now create your own private AI workstation completely independent of cloud providers.

The combination of:

- full GPU acceleration
- open-source models
- localhost APIs
- multimodal support

makes Linux one of the best platforms for AI enthusiasts and developers.

---

# Watch the Full Video Tutorial

If you want the complete step-by-step walkthrough, including:

- CUDA installation
- fixing compiler issues
- building llama.cpp
- enabling Gemma Vision
- localhost setup

watch the full video on my YouTube channel.

https://youtu.be/aaOLfjlvIPU

---

---

## Navigation

- Parent: [Technology](http://jayeshmahato.com/blog/technology.md)
- Previous: [Generate Unlimited AI Images Locally for Free with ComfyUI](http://jayeshmahato.com/blog/technology/generate-unlimited-ai-images-locally-for-free.md)
- Next: [How to Deploy Django with Nginx, Gunicorn and Postgres on Ubuntu](http://jayeshmahato.com/blog/technology/how-to-deploy-django-with-nginx-gunicorn-and-postgres-on-ubuntu.md)
