Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

sfcompute.org

Post reply on HN
Print view View original

171–180 of 189 posts

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#171

Post by simonster »

haldujai wrote: post #150
simonster wrote: post #136

Earlier quoted context omitted.

A single TPUv2 host has 8 TPU cores with 64GB of total HBM (8GB per core), but like GPUs, TPUs can't directly access a network, so the host also needs CPUs and standard RAM to send data to them. They are fast, and the host has to be fast enough to keep them fed with data, so the host is pretty beefy. But FWIW, a TPUv2 host has somewhere around 330GB of RAM, not 1.4TB.

Thanks for clarifying, I misinterpreted the commenter as referring to the accelerator as the conversation was about TPU availability for purchase. I know just enough about the architecture to facilitate using TPUs for research training runs but I'm not sure what's so special about the host? Sure it's beefy but there are much beefier servers readily available.

There's nothing super-special about the host. The accelerators are the special part (and, as described elsewhere, they are orders of magnitude more powerful than the Edge TPU). However, if you're an academic/independent researcher, being able to access a system with that much system memory/CPU cores for free through TPU Research Cloud is potentially appealing even without the accelerators.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#172

Post by ec109685 »

version_five wrote: post #111
ec109685 wrote: post #101

Earlier quoted context omitted.

Power seems like a very small amount of cost of compute when it comes to GPU’s.

FWIW I tired to look up some numbers, i found California "industrial" electricity at $0.18/Kwh https://www.eia.gov/electricity/monthly/epm_table_grapher.ph... and H100s using 300-700w https://www.nvidia.com/en-us/data-center/h100/ which implies a worst case marginal cost of .18*.7 = $.126 / gpu / hour. Looks like Montana is cheapest at ~$.05 / kwh which would bring that down to $.035. So there may be about a $0.09 Ca…

Meanwhile, AWS is charging $8 an hour for their top of the line gpu server.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#173

Post by finnh »

xeromal wrote: post #170
stavros wrote: post #161

Earlier quoted context omitted.

I must say, this is the worst I've seen "beaucoup" spelled.

Hahahaha. I can honestly say no on has told me in my entire uneducated life that "bookoo" was a real word. I appreciate the lesson.

You just might enjoy Jumbo, a track off Underworld's "Beaucoup Fish" album

https://open.spotify.com/track/3VIMS1p3sNifH0RQnmDf7s

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#174

Post by PettingRabbits »

bnr4u wrote: post #71

Having hosted infrastructure in CA at multiple colos. I would advise you to host it elsewhere if you can, cost of power, other infrastructure is much higher in CA than AZ or NV.

Montreal would be the place to go for cheap power, and the CAD-USD advantage.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#176

Post by sillysaurusx »

zak wrote: post #142
sillysaurusx wrote: post #129

Earlier quoted context omitted.

It’s not euw4a. It’s everywhere. The allocation algorithm across the board kills off TPUs after no more than a couple hours. usc1f, usc1a, usc1c, euw4a; it makes no difference. It would be funny if someone set gpt-2-15b-poetry (our project) in some special way to prevent us from making TPUs that ever last more than a few hours, but from what I’ve heard from other people, this isn’t the case. That’s what I mean about…

It sounds like you're primarily using preemptible TPU quota, which doesn't come with any availability or uptime expectations at all. By default, the TRC program grants both on-demand quota and preemptible quota. If you are able to create a TPU VM with your on-demand quota, it should last quite a bit longer than a few hours. (There are situations in which on-demand TRC TPU VMs can be interrupted, but these ought to be…

I was using on-demand TPUs primarily, and preemptible TPUs secondarily. Neither would last more than an hour or two. And two was something of a minor miracle by the end.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#177

Post by wongarsu »

version_five wrote: post #111
ec109685 wrote: post #101

Earlier quoted context omitted.

Power seems like a very small amount of cost of compute when it comes to GPU’s.

FWIW I tired to look up some numbers, i found California "industrial" electricity at $0.18/Kwh https://www.eia.gov/electricity/monthly/epm_table_grapher.ph... and H100s using 300-700w https://www.nvidia.com/en-us/data-center/h100/ which implies a worst case marginal cost of .18*.7 = $.126 / gpu / hour. Looks like Montana is cheapest at ~$.05 / kwh which would bring that down to $.035. So there may be about a $0.09 Ca…

$0.09 for the GPU alone. Add power for mainboard, RAM, and fans, efficiency loss at the power supply, networking, etc. After that another flat 30% for HVAC, since all that "consumed" electricity got turned into heat and the heat has to go somewhere.

And when we are talking about low margins, a 5-10% difference in cost is very significant.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#178

Post by jiggawatts »

mikeravkine wrote: post #102
jiggawatts wrote: post #81

Earlier quoted context omitted.

It’s just corporate profits combined with market forces, not a some sort of malicious conspiracy. You can rent a 2-socket AMD server with 120 available cores and RDMA for something like 50c to $2 per hour. That’s just barely above the cost of the electricity and cooling! What do you want, free compute just handed to you out of the goodness of their hearts? There is incredible demand for high-end GPUs right now, and m…

Sorry where are these .50c many core servers you speak of exactly?

Azure's HB120rs_v3 size is about 36c per hour right now with Spot pricing in East US. These use 3rd generation AMD EPYC "Milan" processors.

The instances with the 4th generation "Genoa-X" processors (HB176rs_v4) cost about $2.88 per hour. The HX176rs_v4 model with 1.7 TB of memory is $3.46 per hour.

https://learn.microsoft.com/en-us/azure/virtual-machines/hbv...

https://learn.microsoft.com/en-us/azure/virtual-machines/hbv...

https://learn.microsoft.com/en-us/azure/virtual-machines/hx-...

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#179

Post by jiggawatts »

alex_lav wrote: post #119
jiggawatts wrote: post #81

Earlier quoted context omitted.

It’s just corporate profits combined with market forces, not a some sort of malicious conspiracy. You can rent a 2-socket AMD server with 120 available cores and RDMA for something like 50c to $2 per hour. That’s just barely above the cost of the electricity and cooling! What do you want, free compute just handed to you out of the goodness of their hearts? There is incredible demand for high-end GPUs right now, and m…

> You can rent a 2-socket AMD server with 120 available cores and RDMA for something like 50c to $2 per hour. Source required

https://news.ycombinator.com/item?id=36950422

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#180

Post by jiggawatts »

megakwood wrote: post #105
jiggawatts wrote: post #81

Earlier quoted context omitted.

It’s just corporate profits combined with market forces, not a some sort of malicious conspiracy. You can rent a 2-socket AMD server with 120 available cores and RDMA for something like 50c to $2 per hour. That’s just barely above the cost of the electricity and cooling! What do you want, free compute just handed to you out of the goodness of their hearts? There is incredible demand for high-end GPUs right now, and m…

Where can you get 120 cores for $2/hr?

https://news.ycombinator.com/item?id=36950422
Post reply on HN