Run Google Gemma Locally with Ollama, Open WebUI, and Codex CLI

08/24/2026 08/13/2026

Tech Guru

The short answer: run a downloaded Gemma 4 tag in Ollama, keep Ollama on loopback, put Open WebUI behind a LAN or VPN boundary, and point Codex CLI's OSS mode at the local provider. That can keep model inference on your machine, but only after you disable optional Ollama cloud features and verify that web search, plugins, and connected providers are not sending data elsewhere.

This guide is for a single Linux homelab host or workstation. It covers model sizing, Open WebUI installation, Ollama, optional NVIDIA acceleration, and Codex CLI. It assumes you can use SSH, Docker, systemd, and a firewall; it does not assume that the machine is safe to expose to the public internet.

The target stack:

Google Gemma as the local open-weight model family
Ollama as the model runtime and local API server
Open WebUI as the ChatGPT-like browser interface
Codex CLI in OSS mode for local coding-agent workflows
Optional NVIDIA GPU acceleration
Optional coexistence with Tdarr or other GPU workloads

Important distinction: Gemma and Gemini are not the same thing. Gemini is Google's hosted API model family. Gemma is Google's open-weight model family that you can download and run locally.

Who This Is For

This guide is for homelab users, developers, and media-server owners who want to run Gemma locally instead of relying entirely on hosted AI services. It assumes basic comfort with Linux, Docker, SSH, and terminal commands.

What You'll Build

By the end, you will have:

Ollama serving a Gemma model locally on port 11434
Open WebUI available in a browser on port 3000
A working local chat interface for Gemma
A Codex CLI workflow that can target the local Ollama/Gemma provider
A safer mental model for sharing GPU resources with Tdarr or ffmpeg

Quick Start

For a smaller homelab system:

ollama pull gemma4:e2b
mkdir -p ~/open-webui
test -f ~/open-webui/open-webui.env || echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" > ~/open-webui/open-webui.env
chmod 600 ~/open-webui/open-webui.env

docker run -d \
  --name open-webui \
  --restart always \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --env-file ~/open-webui/open-webui.env \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

# After installing Codex CLI and signing in once:
codex --oss -c 'oss_provider="ollama"' -m gemma4:e2b

Open WebUI at:

http://SERVER-IP:3000

Security warning: Do not expose Open WebUI or Ollama directly to the public internet. Use a VPN, private LAN, or properly secured reverse proxy.

Interactive Flow Diagram: Local AI Architecture

Interactive Stack Flow: Browser to Local Model

Select each stage to see what it does and where it runs.

Select a stage

Click a stage in the flow to view operational details.

Hardware Planning

Local AI performance is mostly constrained by memory, especially GPU VRAM. CPU speed matters, but VRAM determines what model size can run comfortably.

Practical Gemma sizing:

Model class	Good for	Practical hardware
Gemma 4 E2B	Low-power chat, simple coding help, homelab testing	16GB system RAM recommended; 8GB system RAM can be tight. Current Ollama default is about 7.2GB, so 8GB VRAM can work only with limited headroom; 12GB+ VRAM is more comfortable.
Gemma 4 E4B	Better chat and reasoning, still lightweight	16-32GB RAM. Current Ollama default is about 9.6GB; 12GB VRAM is workable, 16GB+ is safer with longer context or other GPU workloads.
Gemma 4 12B	Stronger general model	32-64GB RAM. Current Ollama default is about 7.6GB; 12GB VRAM is workable for default quantized use, 16GB+ is safer. q8/bf16 variants need much more.
Gemma 4 26B A4B	Strong local reasoning tradeoff	64GB RAM. Current Ollama default is about 18GB; 24GB VRAM is the practical target. 16GB may require QAT/smaller quantization, reduced context, or CPU offload.
Gemma 4 31B	Best Gemma quality, heavier and slower	64-128GB RAM. Current Ollama default is about 20GB; 24GB VRAM is usable but cramped for short-context, no-competing-workload runs. 32GB+ is the practical local target; 48GB+ is better for q8/headroom.

Quick Sizing Graphic: Default Ollama Gemma Tags

E2B

~7.2GB

E4B

~9.6GB

12B

~7.6GB

26B

~18GB

31B

~20GB

These are current default Ollama tag sizes, not a guarantee of full runtime memory fit. KV cache, context length, quantization, images, and other GPU processes add overhead.

For Gemma 4 31B, Google lists approximate inference memory at about 17.5GB for 4-bit Q4_0, 34.9GB for 8-bit SFP8, and 69.9GB for BF16. Those figures are approximate GPU/TPU memory to load the model with overhead, not a guarantee that a full 256K-context Ollama session will fit; KV cache, runtime overhead, multimodal inputs, and other GPU processes add more.

Recommended 31B workstation:

GPU: NVIDIA RTX 5090 32GB or workstation RTX 6000 Ada 48GB preferred. RTX 4090 24GB or RTX 3090 24GB can run the default gemma4:31b/Q4-class model, but treat them as tight, short-context, single-workload options.
RAM: 64GB workable when the model is mostly GPU-resident; 128GB preferred for CPU offload, long contexts, and mixed workloads.
CPU: Ryzen 9, Core Ultra 9, or Threadripper depending on budget
Storage: 2TB NVMe minimum
PSU: 1000W to 1200W for high-end GPUs
OS: Linux preferred for server-style use

If you also run Tdarr, Plex, Jellyfin, Frigate, Stable Diffusion, or any other GPU workload, leave extra headroom. A model that barely fits while idle may fail when ffmpeg is already using VRAM.

Useful monitoring command:

watch -n 1 nvidia-smi

Step 1: Install System Packages

On Ubuntu or Debian:

sudo apt update
sudo apt install -y curl ca-certificates gnupg git htop nvtop

Install Docker:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker "$USER"

Privilege warning: Membership in the Docker group is effectively root-equivalent on most Linux systems. Only add trusted users.

Log out and back in so your user can run Docker without sudo.

Verify:

docker version
docker compose version

For NVIDIA GPU systems, make sure the driver is installed and visible:

nvidia-smi

Step 2: Install Ollama

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Install warning: Always review install scripts before piping them into a shell, especially on production systems.

Verify the service:

ollama --version
curl http://127.0.0.1:11434/api/tags

Pull a Gemma model:

ollama pull gemma4:e2b

For a stronger system, 12B is the next sensible step before 26B or 31B:

ollama pull gemma4:12b

For 24GB+ VRAM or high-RAM systems:

ollama pull gemma4:26b
ollama pull gemma4:31b

Test a prompt:

ollama run gemma4:e2b "Explain what Open WebUI does in one paragraph."

If you are on smaller hardware, start with E2B. Do not begin with 31B unless you know your system has the memory for it.

Step 3: Run Open WebUI

Open WebUI is the browser-based front end. It gives you a familiar chat interface, model selector, account support, prompt library, document upload, and local/private workflows.

Basic Docker run:

mkdir -p ~/open-webui
test -f ~/open-webui/open-webui.env || echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" > ~/open-webui/open-webui.env
chmod 600 ~/open-webui/open-webui.env

docker run -d \
  --name open-webui \
  --restart always \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --env-file ~/open-webui/open-webui.env \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

For production, consider pinning a known-good Open WebUI image tag instead of tracking :main.

By default, -p 3000:8080 publishes Open WebUI on all host interfaces. On a host with a public IP, that can expose the login page directly. For reverse-proxy-only access, bind to loopback with -p 127.0.0.1:3000:8080. For LAN-only access, bind to the server's LAN IP, for example -p 192.168.1.50:3000:8080.

Open:

http://SERVER-IP:3000

If Open WebUI runs in Docker and Ollama runs on the host, set the Open WebUI Ollama URL to:

http://host.docker.internal:11434

On native Linux Docker, host.docker.internal is not automatic; --add-host=host.docker.internal:host-gateway or Compose extra_hosts maps it to the Docker host gateway. If Open WebUI still cannot connect, remember that this address is not the host's 127.0.0.1; Ollama bound only to loopback will not be reachable from a bridge-network container. Either set OLLAMA_HOST=0.0.0.0:11434 in the Ollama systemd service and restrict port 11434 with a firewall, or use the host-network Open WebUI command below so 127.0.0.1:11434 means the host loopback.

Alternative host-network version:

docker run -d \
  --name open-webui \
  --restart always \
  --network=host \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --env-file ~/open-webui/open-webui.env \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

With host networking, Open WebUI listens on its container port directly, so this alternative is normally available at http://SERVER-IP:8080 unless you configure a different Open WebUI port.

Host networking shares the host network namespace. Docker -p mappings do not apply in this mode, so you cannot restrict exposure with -p 127.0.0.1:.... Use firewall rules or prefer bridge networking with an explicit host IP binding when you need interface-level restriction.

Then in Open WebUI:

Create the first admin account.
Go to Admin Settings.
Confirm Ollama connection.
Select gemma4:e2b, gemma4:e4b, or your preferred model.
Start a new chat.

Step 4: Docker Compose Version

For a cleaner homelab deployment, create:

mkdir -p ~/open-webui
cd ~/open-webui
nano docker-compose.yml

Use:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - open-webui:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
    extra_hosts:
      - "host.docker.internal:host-gateway"

volumes:
  open-webui:

For reverse-proxy-only access, use "127.0.0.1:3000:8080" instead of "3000:8080". For LAN-only access, bind to the server's LAN IP, for example "192.168.1.50:3000:8080".

Create a .env file beside docker-compose.yml:

echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" > .env

Keep this .env file and reuse it for upgrades. Do not regenerate WEBUI_SECRET_KEY for an existing Open WebUI volume unless you intentionally want to invalidate existing sessions and encrypted tokens.

Start:

docker compose up -d

Logs:

docker logs -f open-webui

Upgrade later:

cd ~/open-webui
docker compose pull
docker compose up -d

If you used the direct docker run path, update by recreating the container while reusing the same volume and ~/open-webui/open-webui.env file:

docker rm -f open-webui
docker pull ghcr.io/open-webui/open-webui:main
docker run -d \
  --name open-webui \
  --restart always \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --env-file ~/open-webui/open-webui.env \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Step 5: Tie In Codex CLI with Local Gemma

Codex CLI can run against local open-source providers. With Ollama installed, Codex can use Gemma through Ollama's local API.

Install or update Codex CLI:

curl -fsSL https://chatgpt.com/codex/install.sh | sh

The first run of Codex CLI prompts you to sign in with a ChatGPT account or an OpenAI API key. OSS mode only changes the model provider for that run; it does not make Codex CLI a fully unauthenticated standalone binary, and some Codex surfaces or features may still depend on the login type and plan.

Verify:

codex --version
codex --help

Make sure Ollama has a model:

ollama pull gemma4:e2b

Set Ollama as the default OSS provider by adding this as a root-level key near the top of ~/.codex/config.toml, before any [section] headers:

oss_provider = "ollama"

Put that in:

~/.codex/config.toml

Run Codex in OSS mode against Ollama:

codex --oss -c 'oss_provider="ollama"' -m gemma4:e2b

Run a one-shot prompt:

codex exec --oss -c 'oss_provider="ollama"' -m gemma4:e2b "Summarize this repository and identify the main entry points."

Inside a project directory:

cd /path/to/your/repo
codex --oss -c 'oss_provider="ollama"' -m gemma4:e2b

Some Codex CLI versions also expose an explicit local-provider flag:

codex --oss --local-provider ollama -m gemma4:e2b

Use codex --help to confirm which flags your installed version supports.

Important: Codex cloud/web does not use your local Gemma model. This setup is for the local Codex CLI. The CLI can inspect local files, run commands, and work in your local repo. The cloud version uses OpenAI-hosted infrastructure.

Command safety warning: A local coding agent can propose and run shell commands. Keep work in git, review diffs, and avoid granting broad filesystem access unless you understand the risk.

Interactive Flow Diagram: Codex + Gemma Coding Flow

Interactive Codex Flow: Local Gemma Provider

Click a step to view details.

Tdarr and GPU Sharing Notes

If the same system also runs Tdarr, your GPU is shared, but the contention is not identical: Tdarr/ffmpeg GPU transcodes usually use fixed-function NVENC/NVDEC engines, which are separate from the CUDA/Tensor compute Ollama uses. They can still conflict through VRAM, memory bandwidth, copy engines, power, thermals, and driver scheduling.

Check usage:

nvidia-smi

If you see ffmpeg processes using VRAM, Ollama has less room to load Gemma.

Practical scheduling:

E2B is the safest choice while Tdarr is active on 8-12GB GPUs.
On 16GB+ GPUs, E4B or 12B can often coexist with light transcodes, but watch VRAM and encoder/decoder utilization.
Pause Tdarr before loading 26B/31B on 24GB-class cards, and avoid 31B plus active transcodes unless you have substantial VRAM headroom and have tested it.
Use Tdarr schedules to keep heavy transcodes overnight.
Consider a second GPU if this machine is both a media server and an AI server.

For a dedicated AI box, keep it separate from Tdarr. For a combined homelab system, use smaller models and accept slower responses.

Security Recommendations

Do not expose Open WebUI directly to the internet without protection.

Better options:

Use Tailscale, WireGuard, or ZeroTier.
Put Open WebUI behind a reverse proxy with HTTPS.
Require strong accounts and passwords.
Keep the Ollama API port private.
Do not publish port 11434 to the public internet.
Back up the Open WebUI volume.

Example firewall stance:

Allow LAN/VPN access to 3000 for Open WebUI.
Keep 11434 local-only or LAN-only.
Do not expose Docker socket.
Do not run random model-generated shell commands without review.

Troubleshooting

Open WebUI cannot see Ollama:

curl -fsS http://127.0.0.1:11434/api/tags
docker run --rm --add-host=host.docker.internal:host-gateway curlimages/curl:latest -fsS http://host.docker.internal:11434/api/tags
docker logs open-webui

If the host curl works but the container curl fails on Linux, Ollama is probably listening only on loopback. Use OLLAMA_HOST=0.0.0.0:11434 with firewall restrictions, or run Open WebUI with host networking.

Try setting the Ollama URL in Open WebUI to:

http://host.docker.internal:11434

Model is too slow:

Use a smaller model.
Use a 4-bit quantized model.
Reduce context length.
Stop Tdarr/ffmpeg workloads.
Check whether the GPU is being used.

GPU memory error:

Run nvidia-smi.
Stop competing processes.
Use E2B or E4B instead of 12B/26B/31B.
Lower context.

Codex local model gives weak edits:

Try a coding-tuned model through Ollama for coding tasks.
Keep tasks small and specific.
Ask Codex to inspect before editing.
Use git and review diffs carefully.
For complex coding work, use a stronger hosted model or a larger local GPU.

Recommended Starting Commands

For a smaller homelab system:

ollama pull gemma4:e2b
mkdir -p ~/open-webui
test -f ~/open-webui/open-webui.env || echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" > ~/open-webui/open-webui.env
chmod 600 ~/open-webui/open-webui.env

docker run -d \
  --name open-webui \
  --restart always \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --env-file ~/open-webui/open-webui.env \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main
codex --oss -c 'oss_provider="ollama"' -m gemma4:e2b

For a 31B-capable workstation:

ollama pull gemma4:31b
codex --oss -c 'oss_provider="ollama"' -m gemma4:31b

For Open WebUI:

http://SERVER-IP:3000

Final Thoughts

The best first version of this stack is simple:

Ollama + Gemma E2B/E4B + Open WebUI

Once that works, add Codex CLI in OSS mode:

Codex CLI + Ollama + Gemma

That gives you two local workflows: a browser-based chat interface for everyday use and a terminal-based coding agent for repo work. Smaller Gemma models are reasonable for learning and tightly scoped tasks. A larger model may improve difficult repo work, but only a task-based evaluation can show whether the extra latency, memory, and power are justified.

For a mixed-use homelab server that also runs Tdarr, start small. For a dedicated local AI workstation, build around VRAM first.

How to Verify This Stack

Evidence status: this guide is documentation-backed and was fact-checked on July 15, 2026. The commands and architecture were reviewed against current Google, Ollama, Open WebUI, and Codex documentation. TechGeeks did not independently install this exact stack or measure its throughput, VRAM use, power draw, network egress, or Codex task success for this revision. Treat the checks below as your acceptance test, not as reported lab results.

Record versions and tags. Save the output of ollama --version, codex --version, docker image inspect, and ollama show gemma4:e2b. A mutable latest or main tag is not enough for a reproducible result.
Prove the execution path. While a prompt is running, check ollama ps and GPU telemetry such as nvidia-smi. Record processor placement, model size, context setting, and peak memory. A downloaded model file does not prove the runtime used the GPU.
Prove the service boundary. Use ss -ltnp to confirm port 11434 is bound only where intended. From an unauthorized VLAN or guest device, model enumeration and generation requests should fail.
Prove local-only behavior. Enable OLLAMA_NO_CLOUD=1 or disable_ollama_cloud, restart Ollama, and confirm the log says cloud is disabled. Then temporarily block WAN egress and repeat a local Open WebUI chat and a Codex task. Disable web search, remote providers, pipes, tools, and browser plugins for this test.
Test the real workload. Run at least five cold and five warm trials with a fixed prompt. Record time to first token, generation rate, peak RAM/VRAM, errors, and whether a simultaneous Tdarr job causes an out-of-memory event or unacceptable delay.
Test Codex in a disposable repository. Give it one bounded edit with a test, inspect every command and diff, and verify that sandbox and approval settings prevent writes outside the workspace. Do not place real credentials in the test repo.

Risk, Recovery, and Legal Boundaries

Before an Open WebUI update, stop the application and make a cold backup of its complete /app/backend/data volume plus the Compose file and protected .env. Restore that backup to a separate volume and sign in before calling it recoverable. Pin a known image tag or digest so rollback means recreating the old container against a compatible copy of the data, not guessing which build main used.

If the stack starts exposing data or consuming resources unexpectedly, the break-glass action is to stop Open WebUI and Ollama, remove the network route or firewall allowance, revoke connected provider keys, and preserve logs before changing configuration. Codex changes should live on a Git branch; rollback is a reviewed revert or branch deletion, not a blind cleanup command. A model rollback can restore behavior, but it cannot retract text already sent to a remote provider or tool.

Gemma is open-weight under Google's terms, not automatically Apache-licensed software. Review the exact model license and prohibited-use policy before redistribution or commercial use. Only index or submit code and documents you own or are authorized to process. Local execution does not remove copyright, privacy, employment, export, or client-confidentiality obligations.

What This Evidence Does Not Prove

Google's weight-memory table and Ollama's download sizes do not prove that a model fits at your chosen context length; key-value cache, vision input, runtime buffers, and competing GPU work add memory.
A successful chat does not prove coding-agent reliability, correct tool calls, safe command selection, or acceptable results across another repository.
OLLAMA_NO_CLOUD=1 does not control Open WebUI plugins, remote model connections, package downloads, Codex integrations, or third-party browser extensions.
A WAN-blocked test shows behavior during that observation window. It does not audit every future update, dependency, administrator change, or user-installed tool.
Loopback binding protects against direct remote access; it does not protect the host from a compromised local process or an overprivileged container.

References

Google Gemma docs: https://ai.google.dev/gemma/docs
Gemma 4 model and memory requirements: https://ai.google.dev/gemma/docs/core
Run Gemma with Ollama: https://ai.google.dev/gemma/docs/integrations/ollama
Open WebUI docs: https://docs.openwebui.com/
Open WebUI project: https://github.com/open-webui/open-webui
Codex CLI docs: https://developers.openai.com/codex/cli
Codex advanced configuration and local providers: https://developers.openai.com/codex/config-advanced
Ollama downloads: https://ollama.com/download
Ollama Gemma 4 library and current tag sizes: https://ollama.com/library/gemma4
Ollama cloud controls and network binding: https://docs.ollama.com/faq
Ollama Launch integration for Codex: https://ollama.com/blog/launch
Open WebUI hardening guidance: https://docs.openwebui.com/getting-started/advanced-topics/hardening/
GitHub Security Lab, Open WebUI tool-restriction bypass analysis: https://securitylab.github.com/advisories/GHSL-2026-002_Open_WebUI/

Need help applying this?

Bring TechGeeks into the real environment.

If you are working through this on a live network, WordPress site, Linux server, AI workflow, or PisoWiFi deployment, send the context and we can help turn it into a practical plan.

Request help Get field notes Recommended gear

Categories: AI, Development Tools, Homelab, Linux, and Local LLMs

Tags: Codex CLI Docker Gemma Gemma 4 Homelab Local AI Server NVIDIA Ollama Open WebUI Tdarr