For a long time, running a Large Language Model (LLM) on your phone meant compiling complex C++ code or using highly specialized, locked-down apps. But it is 2026, and you can now run a full-fledged local terminal-based AI experience on Android.
By combining Termux (the powerful Android terminal emulator) and Ollama (the standard for local LLM management), you can run models like Qwen 3.5, Llama 3.2, and Gemma directly in your pocket. This setup is 100% private, works offline in Airplane Mode, and costs nothing.
In this guide, I will take you step-by-step through installing and configuring Ollama inside Termux, running your first model, preventing background sleep kills, and connecting Web UIs.
Step 1: Prepare the Termux Environment
To ensure a smooth installation, you must use the version of Termux from F-Droid or GitHub. The Google Play Store version is severely outdated and will result in compile or package errors.
- Download and install Termux from F-Droid.
- Open Termux and update your packages to the latest versions:
pkg update && pkg upgrade -y - Install essential networking and process utilities:
pkg install curl git proot -y - Prevent Background Sleep Kills: Run the following command so Android does not kill the Ollama server when your screen turns off:
termux-wake-lock
Step 2: Install Ollama in Termux
Ollama has official ARM64 support, making the Termux installation direct and straightforward in 2026.
Run the following command to download and install the package:
pkg install ollama -y⚠️ Permission Warning: Always perform your Ollama commands and model storage in the home directory (
~) of Termux. If you attempt to save or run models from external shared storage (like/sdcard), Android’s file system restrictions will trigger standardPermission Deniederrors.
Step 3: Run the Ollama Server
Unlike a desktop machine where Ollama starts automatically as a system service, you must launch the Ollama server manually in Termux.
Start the Ollama background daemon:
ollama serve &The
&symbol pushes the process to the background, allowing you to continue using the same terminal screen.Verify that the server is active:
ollama listYou should see an empty table, indicating the server is running and ready to download models.
Step 4: Download and Chat with a Model
For mobile devices, you should stick to lightweight models designed for edge hardware. Models under 3 Billion parameters (3B) provide the perfect balance of fast token generation and minimal battery drain.
I highly recommend starting with Qwen 3.5 0.8B or Qwen 2.5 1.5B, which perform exceptionally well for their compact memory footprint.
To download and immediately start chatting with the model, run:
ollama run qwen3.5:0.8bOnce the download completes, you will be dropped into an interactive chat prompt. Type your question and hit Enter! To exit the chat at any time, type /exit.
Recommended Models for Mobile (2026)
| Model Name | Size | Best Use Case | Expected Speed | RAM Needed |
|---|---|---|---|---|
| Qwen 3.5 0.8B | ~1GB | General Q&A / Ultra-Fast Chat | 12–20 t/sec | ~1.2 GB |
| LFM 2.5 1.2B | ~731MB | Reasoning & On-Device Thinking | 8–14 t/sec | ~1.5 GB |
| Gemma 4 E2B | ~2.2GB | Multimodal & Complex Context | 6–10 t/sec | ~3.0 GB |
| Llama 3.2 1B | ~1.3GB | Coding & Conversational Logic | 8–15 t/sec | ~1.8 GB |
Connecting a Graphic Interface (Web UI)
If you don’t want to use the command line for everyday chatting, you can connect a graphic interface client.
Since your Ollama Termux server listens on http://127.0.0.1:11434, you can:
- Install mobile GUI clients like PocketPal AI and point it to localhost.
- Configure your local network or homelab Open-WebUI instance to use your phone as an inference worker node.
Troubleshooting Common Mobile Issues
- “Out of Memory / Killed” Error: If Termux kills the process during inference, your model is too large for free RAM. Switch to a smaller model quant like
qwen2.5:0.5bor close heavy background apps. - “Permission Denied” on Model Pull: Ensure you are working in
cd ~(Termux internal storage) and not an external SD card path. - Thermal Throttling / Battery Drain: If running 24/7, read my guide on Running 24/7 Local AI on an Old Android without Overheating.
💬 Frequently Asked Questions (FAQ)
Can I run Ollama in Termux without rooting my Android phone?
Yes. Ollama installs and executes completely in user-space inside Termux without requiring root permissions. All model downloads and binaries are stored safely inside Termux’s private home directory.
How do I prevent Android from killing the Ollama server in the background?
Run termux-wake-lock in your terminal before starting the Ollama daemon, and disable battery optimization for the Termux app in your Android system settings.
What is the best lightweight local LLM for Termux in 2026?
Qwen 3.5 0.8B and Qwen 2.5 1.5B are the best mobile models. They require under 1.5GB of RAM, start instantly, and generate 12 to 20 tokens per second on modern ARM64 chips.
Do I need internet access to use Ollama in Termux?
Internet access is only required once during the initial model download. Once the weights are stored locally, you can enable Airplane Mode and chat with the local AI 100% offline.
Related Mobile AI Tutorials
- Top 4 Ways to Run LLM Locally on Android & iOS (2026 Guide)
- How to Install llama.cpp on Windows, Mac, Linux, and Termux (2026)
- Running Gemma 4 Locally on iOS and Android: The Google AI Edge Gallery Guide
- Running 24/7 Local AI on an Old Android without Overheating
- How to Run LLM Models on Old Android Devices Locally
