Looking for simpler setups?
- For a fast one-line CLI setup with pre-built binaries, check out How to Install and Run Ollama in Termux on Android.
- For graphic apps on modern phones (PocketPal / Edge Gallery), check out Top 4 Ways to Run LLM Locally on Android & iOS.
- For cross-platform desktop & server setup, check out How to Install llama.cpp on Windows, Mac, Linux, & Termux.
When LLM models were first launched, we had to rely on cloud versions like ChatGPT or Gemini. However, the ecosystem has shifted toward local, private AI. We are now seeing lightweight edge models released every week that can run locally on mobile devices.
I have been running these models on my Windows machine using Ollama for a while, and even on my latest high-end Android phone with apps like Google AI Edge Gallery, PocketPal, and AnythingLLM.
BUT…
I wanted to try something different. I had an old Android phone (OnePlus 3T, 10 years old) lying around, and I wondered if I could repurpose it into a dedicated local AI server.
In cases where we have limited RAM (2GB to 6GB), the GGUF format via llama.cpp lets us run quantized models with minimal VRAM while maintaining solid accuracy.
🛠️ Installing Termux
To compile llama.cpp for Android, you need a full Linux user-space environment provided by Termux. Always download the APK from F-Droid or GitHub.
Update Termux and install the compilation toolchain:
pkg update && pkg upgrade -y
pkg install git clang cmake make -y👷🏼 Building llama.cpp from Source
To compile llama.cpp on Android:
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release -j$(nproc)/sdcard or /storage/emulated/0), Android will trigger a Permission Denied block when compiling. To fix this, always clone and compile inside the isolated home directory of Termux (~).🏃🏻 Running the Model & OpenAI-Compatible Server
Once built, the binaries are placed in build/bin/. You can launch the built-in HTTP server to serve OpenAI-compatible endpoints across your local network:
./build/bin/llama-server -m /path/to/model.gguf -c 2048 -n 4096 --host 0.0.0.0 --port 8080 -t 4Tested Models on Low-RAM Android:
| Model Name | Quantization | Size | RAM Usage | Speed (Old Phone) |
|---|---|---|---|---|
| Qwen 3 0.6B | Q4_K_M | ~450 MB | ~800 MB | 14–18 t/sec |
| LFM 2.5 1.2B Thinking | Q4_K_M | ~750 MB | ~1.3 GB | 6–10 t/sec |
| Gemma 3 1B | Q4_K_M | ~800 MB | ~1.4 GB | 5–8 t/sec |
| Qwen 2.5 1.5B | Q2_K / Q3_K_M | ~900 MB | ~1.6 GB | 4–7 t/sec |
Connecting a Web Interface (Open-WebUI)
Once the server is running on port 8080, you can connect to it from your computer or phone browser using Open-WebUI or any OpenAI-compatible client. Simply set the endpoint URL to:
http://<phone-local-ip>:8080/v1Preventing Overheating & Battery Degradation
Extended inference will cause older phones to throttle or swell batteries if kept plugged in at 100% charge. Follow these essential tips:
- Limit CPU Threads (
-t 2or-t 4): Don’t max out all cores; leave big cores headroom to avoid thermal throttling. - Use a Smart Plug / Charge Limit: Keep the battery hovering between 50% and 80%.
- Read my detailed guide on Running 24/7 Local AI on an Old Android without Overheating.
💬 Frequently Asked Questions (FAQ)
Can I run local AI models on old Android phones with 2GB to 4GB of RAM?
Yes. By building llama.cpp natively in Termux and using extreme GGUF quantizations (such as Q2_K, Q3_K_M, or Q4_K_M) on sub-1B parameter models like Qwen 2.5 0.5B or LFM 1.2B, you can achieve smooth offline inference.
How do I fix compile and permission denied errors when building llama.cpp on Android?
Ensure that you clone and compile llama.cpp inside Termux’s internal home directory (~ or /data/data/com.termux/files/home), rather than shared SD card storage where Android blocks binary execution.
How do I prevent an old Android device from overheating when running as a 24/7 AI server?
Limit llama-server to 2 or 4 CPU threads (using the -t flag), maintain the battery charge at 70-80% using smart plugs or battery bypass, and place the device on a ventilated surface.
Related Guides
- How to Install and Run Ollama in Termux on Android (2026 Guide)
- Top 4 Ways to Run LLM Locally on Android & iOS (2026 Guide)
- Running 24/7 Local AI on an Old Android without Overheating
- How to Install llama.cpp on Windows, Mac, Linux, and Termux (2026)
