Are there any well thought through stories about what this would look like? For example, I'm thinking about like nutrient flow, decision making, energy input, gravitational force, things like that seem to govern the value and speed of intelligence.
Hard to avoid spoilers here but Vernor Vinge hits on pretty much exactly this in A Fire Upon the Deep . Though he's interested less in the hard sci-fi aspects of how/why and more on the consequences of it (story-wise).
The OpenAI and Anthropic batch APIs give you a 50% discount. This is like 98%.
I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?
antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and memory bandwidth, and can barely fit the model weights on its maxed out internal storage (2TB). (This is why the linked OP resorts to streaming the sparse parameters from the network which is incredibly slow.)
I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?
antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and memory bandwidth, and can barely fit the model weights on its maxed out internal storage (2TB). (This is why the linked OP resorts to streaming the sparse parameters from the network which is incredibly slow.)
My answer doesn't change if it is M5s. Where is the math showing a 98% discount over K3 on openrouter. Heck, where is the math showing it is any % cheaper? How much electricity will your M5 sip to hit 1M input and 1M output tokens that would cost $3 + $15 there? I bet it is more expensive on the Mac.
0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
It is fun. Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone. Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200. SOTA LLMs are efficiently compression of all the knowkedge humanity has…
A post-apocalyptic desert landscape. The hero descends into the ruins, now a temple. Crude drawings of times before, people riding dinosaurs and mobile phones with arms and legs swinging a whip at the oppressed. He reaches the altar, a old MacBook, bottlecaps and fidgit spinners glued as a mosaic onto it. In the corner three starved men, riding on power generating bicycles providing some artificial light and supplying the machine with divine electricity. He proceeds and types: "we are starving. How can we get food?". Everybody waits in silence. Hours pass. Days. Slowly, letters emerge on the cracked screen: "you are absolutely right —" the screen flickers and fades. Panic. The link to God severed cynically. The hero has to seek out a new device, into distant lands. Maybe this time it will save them
It is fun. Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone. Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200. SOTA LLMs are efficiently compression of all the knowkedge humanity has…
A post-apocalyptic desert landscape. The hero descends into the ruins, now a temple. Crude drawings of times before, people riding dinosaurs and mobile phones with arms and legs swinging a whip at the oppressed. He reaches the altar, a old MacBook, bottlecaps and fidgit spinners glued as a mosaic onto it. In the corner three starved men, riding on power generating bicycles providing some artificial light and supplyin…
“How long?” he said.
“Seven and a half million years,” said Deep Thought.
SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 Soon decent speed across two Mac Studios with 512GB of RAM.
I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds? To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig. Or could you do the same with custom layout of cheap slower RAM? I wonder if any…
The problem is you'd have to traverse the PCI-E bus every transfer. Even with DMA it still has to physically get off the drive and onto the card. Even with the mythical PCI-E 6, if you had enough NVMEs to saturate, you have to do two transfers to get it to the inference hardware. And that tops out at 128 GB/s which is roughly the speed of DDR5 but with one extra hop.
Optane would actually be useful in this era. Intel was ahead of their time.
I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?
I just mean in terms of incremental inference cost. If you already committed to your hardware, huge models at single or sub-digit TPS are still useful. LLM-as-judge is a good use case. If your machine is gonna be idle overnight (and the power efficiency is excellent here), why pay openrouter if it’s not an interactive workload?
Because it's probably still more expensive in electricity costs compared to openrouter. Certainly it's not 98% cheaper.
0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
It is fun. Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone. Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200. SOTA LLMs are efficiently compression of all the knowkedge humanity has…
That's not a great argument for having an llm in an apocalyptic scenario.
People tend to think they are the thriving hero but most won't survive the first year.
I'd much rather have a USB stick of prepper disk. It works on any shitty machine I find, and it's not going to take days for an answer on how to make potash or fix a solar panel.