QLNI
How small can a language model get before it stops being useful?
That is the whole question. We build models from scratch to find out. Freeze the parts that don't need to learn. Cut the weights down to three values. Move memory onto the disk. Then see what is left.
Sometimes it breaks. Sometimes a 20 MB file answers like something ten times its size. Either way we publish all of it, the wins and the losses, with the code, data and checkpoints, so anyone can run it and check it.
What we have found so far
A frozen vocabulary is enough. In our paper, a small model whose vocabulary is a fixed 512-bit code per token, never trained, matched or beat a normal learned vocabulary on three of four benchmarks with 2.6 times fewer trainable parameters, and handled rare and brand-new words far better. With three-valued weights it became a 46 MB file running at 615 to 740 tokens a second on a laptop CPU. arXiv 2610.22428 · code, data, checkpoints · DOI 10.5281/zenodo.23382634
| model | what it is | size | on a laptop CPU |
|---|---|---|---|
| SHADOW 250M Instruct · GitHub | built from scratch, with a 100M-token memory on disk | 60 MB | about 400 tokens/s in 80 MB of RAM |
| SHADOW 50M Instruct · GitHub | three-valued weights, arithmetic circuits inside the model, memory on disk | 19.8 MB | about 2,000 tokens/s in 40 MB of RAM |
| Embeddings Are Overpriced | the three models from the paper | 46 MB (ternary) | 615 to 740 tokens/s |
| dataset | what it is |
|---|---|
| FLUX-Reason-6M as FLUX.2 latents | 5.6M images, ready to train as FLUX.2 latents |
Next: one small model, one country
Big models read the whole internet and try to know everything about everywhere. We want to see what happens when a small one is built around a single place instead.
SHADOW Assistant is that experiment. A small model that still knows the wider world, but where Australia is a quarter of everything it reads: its English, its places, its laws, its history, its everyday life. Does a small model become clearly better at one country's life? What does it give up for it? We don't know yet. That is why we are building it.
It is not a national AI and it is not a statement. It is a test, and Australia happens to be home.
It will see images, video and documents, keep a memory on disk, search the web when it is online, and run offline on a normal laptop. Its training data will be published here wherever the licences allow.
Thank you to CloudRift for sponsoring SHADOW Assistant with an AI Builder Grant. Their GPU credits are paying for the training.
Contact
Sai Kiran Bathula, New South Wales, Australia · qlni.io · saikiranbathula@qlni.io · GitHub
Releases are for non-commercial research and education unless a repository says otherwise.