AI on our own hardware
Medical data can't go to the cloud, and there's no money for new servers. Language models now run on GPUs that others had already written off.
- C# / .NET 10
- ASP.NET Core
- llama.cpp
- CUDA 12
- Python
- OpenRouter
Call analytics and medical documentation need language models. The cloud doesn't always fit: medical data has to stay inside the clinic. The clinic wasn't ready to buy new servers. What it did have was an old computer with three GTX 1080 Ti cards, 11 GB each.
A custom llama.cpp build
The GTX 1080 Ti is a Pascal card, and the new CUDA 13 no longer supports it. I built llama.cpp against CUDA 12 specifically for these cards. Gemma 4 with 26 billion parameters (21 GB) is split layer by layer across the three GPUs.
The LLA Control gateway
One OpenAI-compatible entry point for programs on the local network. The gateway launches the model, watches its health, queues requests and routes them by model name: to the local model, to another machine on the network or to a cloud model via OpenRouter.
Classes instead of names
Programs don't ask for a specific model but for a class: high, medium or low. The gateway sends the request to the live, least-loaded model of that class. Models can be swapped without touching a single program.
Cluster
Another machine with a model joins the gateway as a worker node, with no special protocol. The gateway checks that nodes are alive and protects against forwarding loops.
Honest metrics
The usual “tokens per second” mixes two different processes and understates generation speed. The gateway measures them separately — prompt processing and response generation — so the bottleneck is obvious. Models can also be compared live: a test chat asks two models the same question side by side.
A balancer for batch jobs
A separate Python balancer spreads thousands of calls across several servers in parallel.
The hard parts
Little memory
The machine has 16 GB of RAM, while the models weigh more than 20 GB. Everything has to live in GPU memory, and the launch settings are tuned for exactly that.
Failures must go unnoticed
If the model hangs or crashes, the gateway restarts it on its own — but not forever: there is protection against restart loops.
- C# / .NET 10, ASP.NET Corethe gateway and its control panel.
- llama.cpp on CUDA 12local Gemma 4 and Qwen models.
- Pythonbatch job balancer.
- OpenRoutercloud models, when the data allows it.
Stability test: a Qwen model ran for 30 days around the clock with no failures and without being unloaded from memory. I stopped the test manually once the work on medical records was finished.
Call analytics and the clinic's other AI tasks run on existing hardware — without buying new servers.