61–70 of 139 posts
Earlier quoted context omitted.
Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.
Not sure if it is still updating https://hackyournews.com/
There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…
Yeah because dynamic digital price signs in shops based on what data vendors have about you and AI can extract from it are such fun! Total surveillance. More than what's already happening. Such fun!
Earlier quoted context omitted.
> (If you have a bunch of money and patience, you can also run something like GPT OSS 120B or GLM 4.5 Air locally.) Don't need patience for these, just money. A single RTX 6000 Pro runs those great and super fast.
Oh... 8 thousand of eurobucks for the thing.
We need custom inference chips at scale for this imho. Every computer (whatever formfactor/board) should have an inference unit on it so at least inference is efficient and fast and can be offloaded while the cpu is doing something else.
LLMs are, by definition, real time at any speed. 50,000 tokens per second? Real time. Only 0.0002 tokens per minute? Still real time. Eight tokens per second is "real time" in that sense, but that's also the kind of speeds that we used to mock old video games for, when they would show "computers" but the text would slowly get printed to a screen letter for letter or word for word.
For anyone interested in a comparative review of different models that can run on a Pi, here’s a great article [1] I came across while working on my project.
[0] https://github.com/syxanash/maxheadbox
[1] https://www.stratosphereips.org/blog/2025/6/5/how-well-do-ll...
Earlier quoted context omitted.
Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.
If you read a lot of comment sections, there are bot accounts showing up on LLM that try to do this constantly. Their output is not great so they get downvoted and spotted quickly.
Earlier quoted context omitted.
It should participate in all conversations, take initiative and experiment.
"Hey, hey, are you still asleep? Using spare cycles, I have designed an optimal recipe for mashed potatoes, as you mentioned ten days ago. I need you to go get some potatoes."
"Hey, are you still having trouble with[succinct summary of a problem it identified]?" "Yes" "I have a solution that meets your requirements as I understand them, and fits in your budget."
I've just tried replicating this on my Pi 5 16GB, running the latest llama.cpp... and it segfaults: ./build/bin/llama-cli -m "models/Qwen3-30B-A3B-Instruct-2507-Q3_K_S-2.70bpw.gguf" -e --no-mmap -t 4 ... Loading model... -ggml_aligned_malloc: insufficient memory (attempted to allocate 24576.00 MB) ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 25769803776 alloc_tensor_range: failed to al…