141–150 of 162 posts
Really there needs to be a product based off of LLMs similar to Alexa or Google home where instead of connecting to the cloud it’s a locally run LLM. I don’t know why one doesn’t exist yet or why no one is working on this
Earlier quoted context omitted.
That was an (as far as I can tell) unsubstantiated claim made by OpenAI. It doesn’t even make sense, as o1’s reasoning traces are not provided to the user.
One reason to believe OpenAI here is that R1 will occasionally claim to be made by OpenAI, which in e.g. LLaMA finetunes is indicative of using synthetic data generated by ChatGPT. Note that this isn't necessarily o1. While o1 is specifically trained to do CoT, you can also make 4o etc produce it with the appropriate prompts, and then train on that output.
Earlier quoted context omitted.
This is HN: my focus is technical (here specifically), maybe "technical" in world assessment and future prediction (in other pages). I.e.: I am just trying to understand the facts.
Sorry, it felt to me like you're trying to troll. Those behaviours are extremely likely intentionally added. I can't prove it, but the responses read like they are from a propaganda text book. Not the nuanced new fashioned kind of propaganda from social media, but classic blunt and authoritarian style. You really notice it from the answers. The output token come really fast, at least 3 times faster than in any other…
Sorry, I don't get this obsession.
Earlier quoted context omitted.
I honestly don't understand the meme with RPi clusters. For a little more money than 4 RPi 5's, you can find on eBay a 1U Dell server with a 32 core Epyc CPU and 64 GB memory. This gives you at least an order of magnitude more performance. If people want to talk about Beowulf clusters in their homelab, they should at least be running compute nodes with a shoestring budget FDR Infiniband network, running Slurm+Lustre…
You can buy it but you can't run it, unless you're fairly wealthy. In my country (italy) a basic colocation service is like 80 euros/month + vat, and that only includes 100Wh of power and a 100mbps connection. +100wh/month upgrades are like +100 euros. I looked up the kind of servers and cpus you're talking about and the cpu alone can pull something like 180W/h, without accounting for fans, disks and other stuff (stu…
Earlier quoted context omitted.
"ollama pull" is pretty close
Why tf isn't ollama in apt-get yet? F these curl|sh installs.
There are three broad groups of people in packaging:
A) People who package stuff you and others need.
B) People who don’t package stuff but use what’s available.
C) People who don’t package stuff and complain about what’s available without taking further action.
If you find yourself in group C while having no interest in contributing to group A or working within the limits of group B, then you have two realistic options: either open your wallet and pay someone to package it for you or accept that your complaints won’t change anything.
Most packaging work is done by volunteers. If something isn’t available, it’s not necessarily because no one sees value in it—it could also be due to policy restrictions, dependency complexity, or simply a lack of awareness. If you want it packaged, the best approach is to contribute, fund the work, or advocate for it constructively.
Does adding memory help? There's a Rpi 5 with 16GB RAM recently available.
Inference speed is heavily dependent on memory read/write speed versus size. As long as you can fit the model in memory, what’ll determine functionality is the mem bandwidth.
Earlier quoted context omitted.
Meanwhile on /r/localllama, people are running the full R1 on CPU with NVMe drives in lieu of VRAM.
Did they get the first token out? ;) Just curious, NVidia ported it, and they claim almost 4 tokens/sec on 8xH100 server. At this performance there are much cheaper option.
Suprisingly it's not *that* bad, with 3t/s for the quantized models: https://www.reddit.com/r/LocalLLaMA/comments/1in9qsg/boostin...
> NVidia ported it, and they claim almost 4 tokens/sec on 8xH100 server.
What? That sounds ridiculously low, someone just got 5.8t/s out of only one 3090 + CPU/RAM using the KTransformers inference library: https://www.reddit.com/r/LocalLLaMA/comments/1iq6ngx/ktransf...