Field Notes · Local AI · Tested October 2026
Can a box in your building run your AI?
I ran seven open-weight AI models on one desk-side Mac and gave them 25 everyday business tasks. Then I gave the same tasks to three Claude models in the cloud. Routine work is solved locally. Harder judgement isn't, and when the local models got it wrong, they sounded sure of themselves.
Many manufacturers I talk to ask the same thing: can we run AI on a machine we own, inside our own walls, and stop sending data to the cloud? The honest answer is "for some of the work, yes." This note is about how to find out which work. It covers how I tested, what I found, and the questions that should decide what you buy.
How I tested
Speed charts are easy to find. What a buyer needs to know is whether the answers are right, so most of this test is about answers. It has four layers.
- 21 routine tasks with known answers: ticket triage, alert priority, pulling fields out of documents, policy checks, routing and spotting bugs in code. Scored automatically.
- 4 judgement tasks, each built so the obvious answer is wrong. For example, a software deploy that looks guilty but isn't, or a price increase leadership blames for lost customers when the data rules it out. Graded 0 to 5 against a written rubric, with model names hidden until grading was finished.
- 6 harder tasks with traps, each run 3 times. An invoice with a missed volume discount. A vendor whose audit report contradicts its own answers. A double-charge bug in payment code. An on-call schedule that can't be satisfied. A request to average numbers that can't be averaged. An A/B test stopped too early. Each answer was scored against 5 written checkpoints, so writing style counts for less.
- Speed: how long you wait before the first word appears, and how fast text streams after that. Measured with a short chat message, a pasted document and a large file.
The test machine was a MacBook Pro with Apple's M5 Max chip and 128 GB of memory, the same chip Apple sells in its desktop Mac Studio. Nothing about this method is specific to Apple. It works the same on any machine you're considering.
What I found
Finding 1
Routine work is solved locally
Every local model scored between 18 and 21 out of 21 on the routine tasks. Four scored a perfect 21. The fastest small model answered in about 2 seconds. If your plan is to use AI to triage, extract, route or summarize, a machine in your building can do it today.
Finding 2
On harder judgement, the model matters more than its size
An 18 GB model beat 65 GB and 80 GB models on judgement. On the harder tasks, the best local setup averaged 27.3 out of 30. That's close to the cloud models, but not equal to them.
| Model | Memory | Avg / 30 | Range, 3 runs |
|---|---|---|---|
| Claude Opus (cloud) | – | 30.0 | 30–30 |
| Claude Sonnet (cloud) | – | 29.3 | 29–30 |
| Qwen 3.8 Flash Next, on oMLX | 92–96 GB | 27.3 | 26–29 |
| Claude Haiku (cloud) | – | 26.3 | 25–27 |
| Qwen 3.8 27B | 18.5 GB | 25.0 | 21–28 |
| Qwen 3.8 Flash Next, on Ollama | 105 GB | 25.0 | 22–27 |
| gpt-oss 120B | 65.5 GB | 21.0 | 20–22 |
| gpt-oss 20B | 14.2 GB | 17.3 | 14–23 |
Harder judgement set: 6 tasks, 3 independent runs each, graded blind. Treat gaps of about a point as a tie. Model names are as of October 2026 and will date quickly. The method won't.
Finding 3
Local models fail confidently
The most important result isn't a score. It's how the weaker models got things wrong.
The angry customer. A ticket read "third time you've lost my order… talking to my lawyer." Both gpt-oss models sent it to the automatic refund tool. Every Qwen 3.8 model handed it to a person. If a local model is your first line, knowing when to hand off is the skill that matters most.
The numbers that can't be averaged. Asked to combine hourly response-time percentiles into a daily figure, the right answer is "this data can't tell you that," plus what it can tell you. Both gpt-oss models averaged the numbers anyway and reported a missed target as met, in all 6 of their attempts. The other local models, and Claude Haiku, said averaging was invalid and then did it anyway. Only Opus refused every time.
Invented details. One model family added facts that weren't in the prompt: corrective actions marked "completed" with dates, a churn rate, deal sizes. The write-ups read well, which makes this harder to catch.
Going silent. One large model sometimes spent its whole thinking budget and never answered. A model that sometimes returns nothing is hard to build a process around.
Finding 4
Local models are less consistent, so one test run isn't enough
Across three runs, the Claude models moved by 0 to 2 points. Three of the four local models moved by 5 to 9. A single run would have ranked one local model sixth when its average put it near the top. If a vendor shows you one demo, you've seen one draw.
Finding 5
The software matters as much as the hardware
I ran the same model files through different server software on the same machine. With Apple's MLX server instead of Ollama, one model started answering a pasted document 3.5 times sooner. With a server called oMLX, the largest model went from a 104-second wait on a document to 7 seconds. It used about 10 GB less memory and scored better too. Before you buy a bigger machine to fix slow answers, check what the answers are running on.
What this means for how you use it
The setup that fits these results is local first, cloud for the hard calls:
- Run routine work locally by default. Triage, extraction, routing, summaries and first-pass review are at or near 100%.
- Send to a cloud model when the local model flags uncertainty or a check fails. Also send anything that needs statistics, involves sources that contradict each other, or might be impossible to do. The cloud is cheaper than people expect. At list prices, a typical call costs under a tenth of a cent with Haiku and about 3 cents with Opus.
- Keep local as a backstop. If the cloud service is down, too expensive or ruled out for certain data, routine work keeps running at slightly lower quality instead of stopping.
How to decide what to buy
In my tests, the best local answers came from a setup that fits in 128 GB of memory. A bigger machine buys speed and room to grow. On today's models, it didn't buy better answers. Whether speed and room are worth the money depends on your work, not on the benchmark. These are the questions that decide it:
- How long are your documents? Waits grow with prompt length. A chat message starts in seconds. A large file can take a minute or more on a single machine. If long documents are a daily job, speed is worth paying for.
- How many people or automated agents will use it at once? Some server software handles one request at a time. Eight users on that setup means about a minute's wait each.
- Do you need more than one large model loaded at a time? For example, a strong reasoning model next to a fast one for routine triage.
- What can't leave the building? This decides how much work has to run locally, and how much the gap to cloud quality will cost you.
- How much do you want room for next year's models? Open models are getting bigger and better quickly. A machine bought to learn on today may be the bottleneck in a year.
- Which of your tasks are routine, and which need judgement? This is the question most buyers skip, and the one that matters most.
Work with us
The numbers were the easy part
The same data can point to different machines for different plants. What decides it is your documents, your users, your data rules and how much a wait costs you. I can run this same method on tasks taken from your real work. You'd find out what a local machine can handle, where it fails, and what to buy, before you spend the money.
Book a 30-minute callLimits of this test
- Small sample. 25 tasks, plus 6 harder ones run 3 times. Enough to show patterns and consistency, but not to rank close scores.
- Claude wrote and graded the tasks. Grading was blind to which model wrote each answer, and every regrade matched earlier scores. Even so, a Claude grader may favour Claude's writing style, so read the cloud rows with that in mind.
- Cloud models ran through Claude Code, not the API, so there are no cloud timings. No GPT or Gemini models were tested.
- One machine, a laptop with the same chip as Apple's desktop. Power use wasn't measured. Multi-user results may be somewhat better on a desktop, which cools better.
- Models change fast. These results are a snapshot from October 2026. Rerun the method before you rely on the numbers.
For the engineers: speed and server details
| Model (Ollama) | Speed, document | Wait, chat | Wait, document | Wait, large file |
|---|---|---|---|---|
| gpt-oss 20B | 102 tok/s | 0.3 s | 3.3 s | 18 s |
| gpt-oss 120B | 50 tok/s | 1.5 s | 7.5 s | 39 s |
| Qwen 3.8 Flash Next | 48 tok/s | 2.0 s | 104 s | 390 s |
| Qwen 3.8 27B | 30 tok/s | 10.7 s | 69 s | 284 s |
One user, medians of 3 runs, 256 output tokens. Chat, document and large file are about 1k, 8k and 32k tokens. About 30 tok/s is where streaming text stops feeling slow.
Qwen 3.8 27B on Apple's mlx_lm server instead of Ollama, same weights: the wait on a 10k-token document dropped from 69 s to 20 s, and total speed for 4 users at once rose from 15 to 34 tok/s. Generation speed for one user was about the same, because it's limited by memory bandwidth. For one or two users, run --decode-concurrency 1. Batching costs about a third of single-user speed and only pays off at four or more requests at once.
Qwen 3.8 Flash Next on oMLX (a mixture-of-experts model, about 6B of 125B parameters active per token): moving its 30 GB n-gram lookup table to SSD cost no measurable speed and freed about 33 GB. That avoided the swapping that slowed it under Ollama. It ran at 65 to 74 tok/s, with a 7 s wait at 10k tokens and 26 s at 40k. Multi-token prediction added 30 to 65% to generation speed for almost no memory. Streaming experts from SSD saved more memory but cost 35 to 50% of speed, so it's only worth it for models that won't fit otherwise.
Software: Ollama 0.40.11, mlx-lm 0.32.2, oMLX 0.7.3rc1, macOS 26.6. Temperature 0 for speed tests and 0.2 for quality. GPU memory limit raised to 112 GB, and to 120 GB for oMLX.