81–90 of 153 posts
I missed the explanation for how the SSDs are connected. Maybe a dumb question.
SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
Earlier quoted context omitted.
SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
So if this all streamed from the internal NVME would you expect better performance?
(0.535 vs 1.038 tok/s at 128 tokens), since one fast drive still has to serve all 16 reads per layer while four drives split the load
https://raw.githubusercontent.com/argonautlabsai/deltafin/ma...
Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…
Earlier quoted context omitted.
My parents learned to program on punch cards. They told me it was a day of preparing the program, an hour of running it, just to get a syntax error.
Write the program, punch the cards, send the cards to another building to be loaded, program runs, printout comes out in another building, somehow this takes 2-3 days
I missed the explanation for how the SSDs are connected. Maybe a dumb question.
SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
Earlier quoted context omitted.
Maybe. Do you have an example of a prompt you've done recently? I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.
Stuff along the lines of implement controller service and tests for the following endpoints: - list of many endpoints with the JSON they receive and return and description of what they need to achieve Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy…
Earlier quoted context omitted.
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently). GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a…
This is incredible! Are there other big players in this space (freezing models to silicon)?
Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? Or something like that?
Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent.
Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discrete GPU it would need to fit in VRAM or be streamed too; and a desktop's PCIe lanes let you put NVMe drives on the bus directly rather than behind a ~7 GB/s Thunderbolt enclosure, which is our per-drive wall.
Whether that ends up faster is exactly the kind of thing that wants measuring rather than guessing.
Earlier quoted context omitted.
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
I would agree, but I want to add that I have real issues with combination of opencode plus slow inference (4-5tok/s). I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something. Its not a problem of the model of course, but it seems impractical atm. I wonder if anyone else had this kind of thing happening.
