Experimenting with local LLMs
Can I run GenAI LLM models on local hardware, and if yes, at what cost? I had been asking myself this question for a long time, and thanks to a friend who kindly lent me an Asus Ascent GX10 machine, I was able to set up a more serious software stack compared to what I had on my personal computer.
The reasons one might want to run an LLM model locally are numerous: no subscription, models can be changed easily thanks to Ollama, data remains private, etc. Thanks to the open source community, you can pretty much do everything you do with online providers, but with a local model on your machine. I identified only two drawbacks. First, you need some skills to set up the network if you want to deploy your models on a different machine than the one you work on; and second, the size of models you can run is greatly limited by the amount of RAM on your machine.
On the latter point, the GX10 features a unified memory between CPU and GPU of 128 GB, which allows me to run models with around 120 billion parameters. However, I found that generation speed is only acceptable for models with less than 35 billion active parameters, regardless of whether they are dense or Mixture of Experts (MoE).
When I first started testing, I had a list of many models I wanted to try, but in the end, I kept only a few for experimentation: – Qwen3.5:27B dense model. – Qwen3.5:122B A10B MoE model. – GLM-4.7-Flash, which is a 30B A3B MoE model.
The objective was to deploy models as intelligent as Sonnet 4.6, and according to artificialanalysis.ai, no model that fits in the GX10 is as intelligent as Sonnet 4.6.
Note: It seems that the recently released Qwen3.8:27B model is similarly intelligent to Sonnet 4.6, although it's far from the latest Sonnet 5.
I ran a series of numeric tests, but I found that the numbers don't really yield meaningful insights. The most important metric is the number of tokens generated per second. This depends on the model and can go anywhere from 10 tokens/sec to 80 tokens/sec. Qwen3.5:27B averaged 12 tokens/sec and Qwen3.5:122B A10B averaged 24 tokens/sec. For a full message, expect somewhere between 30 seconds and a few minutes depending on the input.
I then conducted some conversational testing, although I can only evaluate based on my interpretation of the output messages.
From the three models tested, GLM-4.7-Flash was by far the worst, as it seemed to produce a great deal of hallucinated content. The only difference I saw between the two Qwen models was how each model asked questions to better understand what the user wanted. Qwen3.5:27B seemed closer to discussion, outputting only dry responses, whereas Qwen3.5:122B A10B asked many questions to keep the conversation going.
As for source code, I tried creating two pieces of software, although I must admit those would also be difficult for humans: a Zotero plugin, and a Joplin plugin. Paired with OpenCode, none of the tested models managed to produce a working first version of those plugins. Sonnet 4.6 managed to produce the Zotero one, though it too fell short on the Joplin one.
The main takeaway from these tests is that running LLMs locally takes a lot of time. I find it suitable for conversations where taking a few minutes to answer isn't an issue, but it borders on unusability for tasks such as coding (or vibe coding, as the youngsters call it).
On the price side, I made a simple pessimistic estimate. As for upfront costs, the machine costs at most 4,000 EUR. On the operating costs, the included power supply delivers up to 240 W. This means the machine consumes at most 5.76 kWh per day. At 0.30 EUR per kWh, a pessimistic value in France, this means a cost of at most 1.73 EUR per day, or 52 EUR per month.
With the obvious advantage that there's no usage limit, the investment may be worth it once you exceed the usage limits of the common 20 EUR subscriptions of popular online providers. The main drawback of this is that less and less open-weight models seem to be released nowadays, but given that Qwen3.8 has recently been released, the local-running LLM adventure might not be dead yet.
Thank you for reading. Reach me for further discussion and updates!