How to play: Some comments in this thread were written by AI. Read through and click flag as AI on any comment you think is fake. When you're done, hit reveal at the bottom to see your score.got it
So there are a zillion of these, including ones that do what yours does. But that being said, I'm going to always upvote every single one of them because this is better and more important that what the big companies are doing across the board.
Thank you! Would love to hear which ones you've used. Working with local files without anything going to the cloud was the main thing I wanted. I tried Jan, OpenWork, Chatbox and Lobe, and couldn't get that working well in any of them. If there's one I missed it would be great to check it out!!
Fwiw, I'm liking the koboldcpp "ecosystem", including koboldlite; basically it's letting me try a little of everything, though there may be something of a "jack of all trades master of none" thing going on. But it has these nice little presets for, e.g. llms, voice, music, images, a "chatgpt"-esque interface, etc.
Same here, I upvote them all. The gotcha I keep hitting with local file tools is tool calling: anything under ~8B quantized fumbles multi-step file edits, so the model choice matters more than the app.
Haven't tried it yet, but I like that you shared your process in such an honest way, and kept it no frills, as in no LLM wall-of-text readme in the repo. Wish we could see that more often these days.
Looks great. One thing I am missing is the ability to change the installation directory for the models. For space reasons, I want them to be on a different partition.
Linux. I saw you support HOB_USER_DATA, which is great. But I want my config to remain in .config/Hob and only move the models (or whatever else needs lots of space) to another disk/directory.
Edit: Actually I think HOB_USER_DATA is enough for my use case.
It's still pretty tricky to do this as a solo dev to keep feature parity in my experience (fun fact, even Claude's own Desktop app is made in Electron).
That being said, majority of the heavy LLM lifting happens natively with llama.cpp :)
I did not enjoy that there was no way for it to clean up the 18GB of models it installed after I determined it was not really usable for any real work.
Which models does it ship with, and have you benchmarked them on actual multi-step tasks? I'd guess 18GB of local weights can't do reliable tool calling, so what's the fallback when they fail?
Here's roughly what it picks today from my testing:
- 8 GB Mac (M1/M2): Gemma 4 E2B
- 16 to 18 GB Mac: Gemma 4 E4B
- 24 to 64 GB Mac: Gemma 4 26B A4B
- 128 GB Mac Studio: gpt-oss 120B
- Windows/Linux with an RTX 3060 or 4070 (12 GB) and 32 GB RAM: Gemma 4 26B, split between the GPU and system memory
- RTX 4090: Gemma 4 26B fully on the GPU
- No GPU, 16 GB RAM: a compressed Gemma 4 E4B on the CPU
High level, it looks at your memory and memory bandwidth and then picks the smartest one that stays above about 15 tokens a second with room for a decent context.
Interesting, thanks for sharing this. I have some questions? I have a Macbook M5 with 16GB unified memory. What Model would you suggest I use with this. I want something that would replace dot or grokbot , do posting, some marketing work, and email handling. Not coding. What would you suggest?
I'm actually building something similar and thank you very much for sharing you gave me a few inspirations. I want mine to integrate seemlesly with an animated Ai companion (it's harder than it sounds haha). Anyways great stuff! Gave it a star.
We did something like this with VRM models in three.js. The hard part was lip sync. Driving the mouth from audio amplitude was good enough, and way simpler than proper visemes.
With the raise of AI, I believe: open-source and local software's will be on the rise, but with the open-source environment there is a real security issues that rise.
Never the less, really appreciate your contribution
Check the releases tab. If it's just binaries, "open source" in the title is doing a lot of work. Good luck debugging that at 3am when it falls over and you can't read the code.
“Garth is the most significant artist that I've worked with and I've worked with Lulu and four other people, so we’re talking crème de la crème.” — Dean Learner
As far as I know, tool-calling reliability is where small local models fall down, not raw chat quality. Curious which model Hob defaults to and whether you measured how often multi-step tasks actually complete.