11–20 of 61 posts
I couldn't care less, tbh. This speed is ridiculously high, to the point where tool calls, not inference, become the bottleneck. Also, I'd rather run a large model at slower speeds than a smaller at insanely high speeds.
Also, the entire industry profits from the work that’s done at the bleeding edge. That’s the case in every industry.
I couldn't care less, tbh. This speed is ridiculously high, to the point where tool calls, not inference, become the bottleneck. Also, I'd rather run a large model at slower speeds than a smaller at insanely high speeds.
I couldn't care less, tbh. This speed is ridiculously high, to the point where tool calls, not inference, become the bottleneck. Also, I'd rather run a large model at slower speeds than a smaller at insanely high speeds.
I couldn't care less, tbh. This speed is ridiculously high, to the point where tool calls, not inference, become the bottleneck. Also, I'd rather run a large model at slower speeds than a smaller at insanely high speeds.
Have you considered parrallel processing? I always have 2-3 Cursor IDE open because I don't like wait either.
I couldn't care less, tbh. This speed is ridiculously high, to the point where tool calls, not inference, become the bottleneck. Also, I'd rather run a large model at slower speeds than a smaller at insanely high speeds.
Have you considered parrallel processing? I always have 2-3 Cursor IDE open because I don't like wait either.
Tool use that changes the mode of the environment is a good example where you cannot go parallel. I've built a recursive agent that can run a Unity editor and I can't just blindly run whatever it wants in parallel or combos like SwitchScene -> GetSceneOverview won't interleave correctly. You'll wind up with 15 calls that loop over every scene and then you grab the overview from the last scene you switched to 15 times.
There are ways to hack around it a bit, but at some level the underlying narrative does need to be serialized or you'll be wasting an incredible amount of resources.
Depth-first search doesn't guarantee the best solution, but on average it's guaranteed to find a solution faster than breadth-first search. It's worth waiting for those dependent calls and going super deep if you want some reasonable answer quickly.
The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages.
Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated with prompt processing included.
Model was a Deepseek 671B 32B MoE.
Looks to me that $20 for a month of coding is not very sustainable - let's enjoy the party while VCs are financing it! And keep an eye on your consumption...
Electricity costs seem negligable with ~$10,000 per year at 10cts per kWh but overall cost would be ~10% higher if electricity is more like 30cts like it is in Europe.
Edit: like it is pointed out by other commenters it is 2200t/s per single GPU thus the result needs to be divided by 16: $4/16 = $0.25. This actually somewhat matches the deepseek API pricing.
Impressive performance work. It's interesting that you still see these 40+% perf gains like this. Makes you think that you will continue to see the costs for a fixed level of "intelligence" dropping.
If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…
If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…
So if you work that through its $0.225 per 1M output tokens.
If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…
It's 2.2k tokens per second and GPU, so you have to multiply the token output by 16 and the price per million tokens works out to 22.5 cents.