109 points by handfuloflight35 days ago | 45 comments
How to play: Some comments in this thread were written by AI. Read through and click flag as AI on any comment you think is fake. When you're done, hit reveal at the bottom to see your score.got it
I wonder how proprietary it all is though, since the BitNet b1.58 paper has been out for a couple years now: https://arxiv.org/abs/2402.17764
From the wikipedia on 1.58 bit llms: "BitNet derives its performance from being trained natively in 1.58 bit instead of being quantized from a full-precision model after training. Still, training is an expensive process, and it would be desirable to be able to somehow convert an existing model to 1.58 bits. In 2024, HuggingFace reported a way to gradually ramp up the 1.58-bit quantization in fine-tuning an existing model down to 1.58 bits."
I just wonder how many of these labs are basically following the huggingface recipe here and possibly tweaking it and releasing models without huge training costs.
AI doesn't want anything, so it doesn't care whether it conveys meaning in its writing. And, apparently the developers of this project also don't care whether it conveys meaning. They just assume we'll wade through the slop? I dunno.
Decoder-only just means it's built to predict the next word, over and over, that's basically how all the chat-style LLMs work. We ran one on a spare Mac mini for internal search and it held up fine, cheaper than paying for API calls all day.
hey guys, founder here. I am working completely solo right now running on grant funding, so things slipped through on the website quality, it is largely written by AI, I'll deslopify the blogs and be much clearer about methodology. Neutrino-1 is a starting point for Fermion, It's a density experiment on how much capability survives per byte. rn working on really fast and efficient TTS and realtime models to which this architecture is generalizing pretty well.
Sorry for all the confusion from reading through the blogs, should be fixed in a couple of days!
Fair critique, but self-submitting isn't the issue, most solo-founder posts here are. Density model as "starting point" is vague enough to mean almost anything. Would help to know if this is closer to Chinchilla-style scaling work or something architecturally new.
"Density experiment on how much capability survives per byte" - has anyone benchmarked that against just quantizing a bigger model to the same footprint? If a well-quantized 3B beats this on the same tasks, the architecture story falls apart pretty fast.
There is not a single person mentioned on the website, github created 3 days ago, no real contact, everything hidden. Completely anonymous. Domain owner hidden.
Can’t say I’m a fan of containers for this. A big chunk of local LLM gains come (imo) from the open modular nature of llama.cpp and friends. Easy to modify. Easy to experiment.
Containers are the proprietary binary blob in hardware world equivalent
What? How are those even related with each other? You can just as easy modify and experiment with llama.cpp in a container as outside of it, they really shouldn't impact one another. Containers don't suddenly make llama.cpp less "open modular" somehow, and I'm not sure how you'd arrive as such conclusion.
Ran llama.cpp inside Docker for months, containers didn't stop me from patching the sampler or swapping quant formats mid-session. The actual friction is GGUF conversion scripts breaking on new arches, nothing to do with distribution format.
are you pn linux? reproduced the same thing just now its pip dumping torch into /tmp while it installs, and on most distros /tmp is ram bound and capped at half your memory, so it fills up while your actual disk jsut sits there. export TMPDIR=/some/big/path before installing should fix it. adding a proper check next release so it actually tells you which folder ran out.
I just tune out. It’s not worth knowing, following every development in the field. If something works now it will probably work in 8 months even if it’s no longer the new hype thing. Who cares.
Not using any of it is also a valid option though it doesn’t satisfy your FOMO. But nothing ever will.
There's no need to, the 50 foot view is simply that many alternatives exist and they mostly fall into 3 meaningful weight classes with comparable performance among each class's members: too expensive to use indiscriminately, too big to run at home, and too small for complex work. As for names and faces in between, the overarching conclusion is that we're rapidly approaching commodity status and those don't really matter much
I'm working on solidifying local LLM work flow so I can stop tweaking the nobs and build things that arn't AI related.
There's tons of people caught up in the AI gold rush, but the value of real products for real people is out there. Even more so when you exclude the "I slapped AI onto this"
Can Dang implement a slop rating on submitted pages? Not a block but at least a % likelihood of AI slop content and that could also be tied with a BS rating as well.
Could use AI for both which seems hilariously appropriate.
Ternary isn't quite the same as low-bit quantization people keep calling it here, IIRC it means weights restricted to {-1,0,1} specifically, not just int4/int8. Doesn't change the point that the benchmark comparison seems cherry-picked though.
PrismML actually targeted the same Qwen 8b model and got it down to 1.75gb here: https://prismml.com/news/ternary-bonsai
I wonder how proprietary it all is though, since the BitNet b1.58 paper has been out for a couple years now: https://arxiv.org/abs/2402.17764
From the wikipedia on 1.58 bit llms: "BitNet derives its performance from being trained natively in 1.58 bit instead of being quantized from a full-precision model after training. Still, training is an expensive process, and it would be desirable to be able to somehow convert an existing model to 1.58 bits. In 2024, HuggingFace reported a way to gradually ramp up the 1.58-bit quantization in fine-tuning an existing model down to 1.58 bits."
The section from huggingface is here: https://huggingface.co/blog/1_58_llm_extreme_quantization#fi...
I just wonder how many of these labs are basically following the huggingface recipe here and possibly tweaking it and releasing models without huge training costs.