Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

news.ycombinator.com

Post reply on HN
Print view View original

191–196 of 196 posts

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#191

Post by stuxnet79 »

disambiguation wrote: post #156
stuxnet79 wrote: post #80

Earlier quoted context omitted.

Any resources you can share for these experimental builds? This is something I was looking into setting up at some point. I'd love to take a look at examples in the wild to gauge if it's worth my time / money. An aside, if we ever reach a point where it's possible to run an OSS 20b model at reasonable inference on a Macbook Pro type of form factor, then the future is definitely here!

In reference to this post i saw a few weeks ago: https://lemmy.zip/post/50193734 (Lemmy is a reddit style forum) The author mainly demos their "custom tools" and doesn't elaborate further. But IMO is still an impressive showcase for an offline setup. I think the big hint is "open webui" which supports native function calls. Some more searching and i found this: https://pypi.org/project/llm-tools-kiwix/ It's possible…

It seems like the ecosystem around these tools has matured quite rapidly. I am somewhat familiar with Open WebUI, however, the last time I played around with it, I got the sense that it was merely a front-end to Ollama, the llm command line tool & it didn't have any capabilities outside of that.

I got spooked when the Ollama team started monetizing so I ended up doing more research into llama.cpp and realized it could do everything I wanted including serve up a web front end. Once I discovered this I sort of lost interest in Open WebUI.

I'll have to revisit all these tools again to see what's possible in the current moment.

> My sense is that you need ~1gb of RAM for every 1b paramters, so 32gb should in theory work here. I think macs also get a performance boost over other hardware due to unified memory.

This is a handy heuristic to work with, and the links you sent will keep me busy for the next little while. Thanks!

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#192

Post by embedding-shape »

gunalx wrote: post #189
embedding-shape wrote: post #167

Earlier quoted context omitted.

> had better results with the 20b model, over the 120b model The difference of quality and accuracy of the responses between the two is vastly different though, if tok/s isn't your biggest priority, especially when using reasoning_effort "high". 20B works great for small-ish text summarization and title generation, but for even moderately difficult programming tasks, 20B fails repeatedly while 120B gets it right on t…

But the 120b model has just as bad if not worse formatting issues, compared to the 20b one. For simple refactorings, or chatting about possible solutions i actually feel teh 20b halucinates less than the 120b, even if it is less competent. Migth also be because of 120b not liking being in q8, or not being properly deployed.

> But the 120b model has just as bad if not worse formatting issues, compared to the 20b one

What runtime/tools are you using? Haven't been my experience at all, but I've also mostly used it via llama.cpp and my own "coding agent". It was slightly tricky to get the Harmony parsing in place and working correct, but once that's in place, I haven't seen any formatting issues at all?

The 20B is definitely worse than 120B for me in every case and scenario, but it is a lot faster. Are you running the "native" MXFP4 weights or something else? That would have a drastic impact on the quality of responses you get.

Edit:

> Migth also be because of 120b not liking being in q8

Yeah, that's definitely the issue, I wouldn't use either without letting them be MXFP4.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#193

Post by ricardonunez »

gcr wrote: post #109

For new folks, you can get a local code agent running on your Mac like this: 1. $ npm install -g @openai/codex 2. $ brew install ollama; ollama serve 3. $ ollama pull gpt-oss:20b 4. $ codex --oss -m gpt-oss:20b This runs locally without Internet. Idk if there’s telemetry for codex, but you should be able to turn that off if so. You need an M1 Mac or better with at least 24GB of GPU memory. The model is pretty big, ab…

Similar on a MBpro 16 m4 max 128gb 1tb. I only use it for testing and did created a microservice to process some data for a client but no daily workflow.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#194

Post by johnwatson11218 »

I have a pipeline in Docker compose that starts up postgresql on one container and a python container. The python scripts will recursively read all the pdf files in a directory, use pdf plumber to parse the text to store in a postgres table. Then I use sentence_transformers to take 100 char, w/ 10 char overlap, chunks and embed each section as a 384D vector which is written back to the db. Then I average all the chunks to create a single embedding for the entire pdf file. I have used numpy as well as built in postgres functions to average and it fast either way.

Then I use HMAP + DBSCAN to create a 2D projection of my dataset. DBSCAN writes the clusters to a csv file. I read that back in to create topics, docs2topcs join table. Then I join each topic into a mega doc and consider the original corpus, I compute tf-idf, using only db functions. This gives me the top 5 or so terms per topic and serves as useful topic labels.

I can do 30 to 50 docs in an couple of hours. I imported 1100 pdf files and it took all weekend on an old gaming laptop w/ a ssd. I have a gpu, and I think the embedding steps would go faster but I'm still doing it all synchronously w/o any parallel processing.

Re: Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop

#195

Post by egberts1 »

knownjorbist wrote: post #188
egberts1 wrote: post #112

Ollama, 16-CPU Xenon E6320 (old), 1.9Ghz, 120GB DDRAM4, 240TB RAID5 SSDs, on Dell Precision T710 ("The Beast"). NO GPU. 20b (n oooooot f aah st at all). Pure CPU bound. Tweaked for 256KB chunking into RAG. Ingested election laws of 50 states, territories and Federal. Goal. Mapping out each feature of the election and deal with (in)consistent terminologies sprouted by different university-trained public administration…

So much intelligence devoted to what is obviously a huge con - the Big Lie?

Just yesterday, relevation of USPS stop scanning mail-in ballot envelope as ordered by both NYC and NJ government just before election day.

Might have reveal overloading in certain pickup areas.

Post reply on HN