21–30 of 61 posts
If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…
Now all we need is better support for AMD gpus, both CDNA and RDNA types
However from experience with an AMD Strix Halo, a couple of caveats: it's drastically slower than Ollama (tested over a few weeks, always using the official AMD vLLM nightly releases), and not all GPUs were supported for all models (but that has been fixed).
If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…
The VC money is there until they can solve the optimization problems
If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…
Nope - i live in one of the most expensive areas, and even the residential price has averaged 18c/kWh delivered including taxes. Businesses get a lower basic rate and also don't pay the VAT, so it works out around 13c/kWh for them.
https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...
If I followed the links correctly this benchmark was made on a 16xH200. At current prices I'd assume that is a system price of around $750,000. The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages. Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated wi…
> more like 30cts like it is in Europe Nope - i live in one of the most expensive areas, and even the residential price has averaged 18c/kWh delivered including taxes. Businesses get a lower basic rate and also don't pay the VAT, so it works out around 13c/kWh for them. https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...
Earlier quoted context omitted.
> more like 30cts like it is in Europe Nope - i live in one of the most expensive areas, and even the residential price has averaged 18c/kWh delivered including taxes. Businesses get a lower basic rate and also don't pay the VAT, so it works out around 13c/kWh for them. https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...
That's excluding tax, net prices around 0.20-0.30 EUR / Kwh we common.
Hey HN! I’m Seiji Eicher from Anyscale, one of the authors of this post :) Feel free to ask questions here.
Impressive performance work. It's interesting that you still see these 40+% perf gains like this. Makes you think that you will continue to see the costs for a fixed level of "intelligence" dropping.
vLLM needs to perform similar operations to an operating system. If you write an operating system in Python you will have scope for many 40% improvements all over the place and in the end it won’t be Python anymore, at least under the hood it won’t be.