Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

sfcompute.org

Post reply on HN
Print view View original

91–100 of 189 posts

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#91

Post by itissid »

itissid wrote: post #86

Noob Thought: So this would be a blue print on how a mid tier universities with older large compute cluster ops could do things in 2023 to support large LLM research? Perhaps its also a way for freshly applying grad students to look at a university looking to do research in LLMs that requires scale...

Like to clarify, a new grad students could look at the current group and ask "Hey I know you are working on LLMs, but how many $$ of your grant are dedicated to how many TPU hours per grad student?"

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#92

Post by beachy »

jiggawatts wrote: post #81
beachy wrote: post #74

Earlier quoted context omitted.

AWS and Azure would slit their own throats before they created a way for their customers to pool instances to save money. They want to do that themselves, and keep the customer relationship and the profits, instead of giving them to a middleman or the customer.

It’s just corporate profits combined with market forces, not a some sort of malicious conspiracy. You can rent a 2-socket AMD server with 120 available cores and RDMA for something like 50c to $2 per hour. That’s just barely above the cost of the electricity and cooling! What do you want, free compute just handed to you out of the goodness of their hearts? There is incredible demand for high-end GPUs right now, and m…

You mentioned malicious conspiracy, not me.

It's just business and I'd do the same if I was in charge of AWS.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#93

Post by haldujai »

sillysaurusx wrote: post #14
LoganDark wrote: post #12

Earlier quoted context omitted.

So the cloud TPUs are more powerful...? Or what are you saying?

Yeah, it’s a silly branding thing. One TPU (not even a pod, just a regular old TPUv2) has 96 CPU cores with 1.4TB of RAM, and that’s not even counting their hardware acceleration. I’d love to buy one.

Huh, this doesn't seem right. Based on #s you seem to be referring to pods but even then I'm not familiar with such a configuration existing.

A single TPUv2 chip has 1 core and 8gb of memory. A single device comes in the v2-8 configuration with 8 cores and 64gb of memory.

Pod variants come in v2-32 to v2-512 configurations.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#94

Post by haldujai »

sillysaurusx wrote: post #3

I hope you succeed. TPU research cloud (TRC) tried this in 2019. It was how I got my start. In 2023 you can barely get a single TPU for more than an hour. Back then you could get literally hundreds, with an s. I believed in TRC. I thought they’d solve it by scaling, and building a whole continent of TPUs. But in the end, TPU time was cut short in favor of internal researchers — some researchers being more equal than…

My experience has been different. Considering how easy the application is I think they're still being fairly generous as I've been offered multiple v3-8s and v3-32s x 30days as well as pre-emptible v3-64s x 28 days for a few different projects within the last 6 months.

Are you affiliated with an academic institution? Otherwise I'm not sure why they're been more generous with me, my projects have been mildly interesting at best.

They're certainly a lot stingier with larger pods than they used to be though.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#95

Post by hgsgm »

Frost1x wrote: post #67
camhart wrote: post #30

Earlier quoted context omitted.

Optimism is (almost) always required in order to accomplish anything of significance. Those who lose it, aren't living up to their potential. I'm not encouraging the false belief that everything you do will work out. Instead I'm encouraging the realization that the greatest accomplishments almost always feel like long shots, and require significant amounts of optimism. Fear and pessimism, while helpful in appropriate…

>Optimism is (almost) always required in order to accomplish anything of significance. Those who lose it, aren't living up to their potential. I argue that realism trumps optimism. It's perfectly normal in a realist farming to see something difficult, acknowledge the high risk and failure potential, and still pursue something with intent to succeed. I've personally grown tired of over optimism everywhere because it c…

Realism doesn't work in business. Business success requires 10 people to try for 1 person to succeed. If those 10 people were realists, they wouldn't try.

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#96

Post by aabhay »

whack wrote: post #70

> Rather than each of K startups individually buying clusters of N gpus, together we buy a cluster with NK gpus... Then we set up a job scheduler to allocate compute In theory, this sounds almost identical to the business model behind AWS, Azure, and other cloud providers. "Instead of everyone buying a fixed amount of hardware for individual use, we'll buy a massive pool of hardware that people can time-share." Outsi…

They are working on this. All the major clouds have initiatives to do short term requests/reservations. It’s just not a feature that has ever been of much use pre-GenAI. How often do you need to request 1000 CPU nodes for 48 hours in a single zone?

Secondly, there is a fundamental question of resource sharing here. Even with this project by Evan and AI Grant (the second such cluster created by AI Grant btw), the question will arise — if one team has enough money to provision the entire cluster forever, why not do it? What are the exact parameters of fair use? In networking, we have algorithms around bandwidth sharing (TCP Fairness, etc.) that encode sharing mechanisms but they don’t work for these kinds of chunky workloads either.

But over the next few months, AWS and others are working to release queueing services that let you temporarily provision a chunk of compute, probably with upfront payment, and at a high expense (perhaps above the on demand rate).

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#98

Post by turbobooster »

xwdv wrote: post #79
flaque wrote: post #40

Earlier quoted context omitted.

Oh no, definitely not. We just got a loan. Neither Alex or I are currently VCs, and this has no affiliation with any venture fund. We want to be a customer of the sf compute group too!

I’m curious, how are those loans guaranteed?

The only guarantee is them not paying it back

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#99

Post by zak »

sillysaurusx wrote: post #3

I hope you succeed. TPU research cloud (TRC) tried this in 2019. It was how I got my start. In 2023 you can barely get a single TPU for more than an hour. Back then you could get literally hundreds, with an s. I believed in TRC. I thought they’d solve it by scaling, and building a whole continent of TPUs. But in the end, TPU time was cut short in favor of internal researchers — some researchers being more equal than…

Actually, the TPU Research Cloud program is still going strong! We've expanded the compute pool significantly to include Cloud TPU v4 Pod slices, and larger projects still use hundreds of chips at a time. (TRC capacity has not been reclaimed for internal use.)

Check out this list of recent TRC-supported publications: https://sites.research.google/trc/publications/

Demand for Cloud TPUs is definitely intense, so if you're using preemptible capacity, you're probably seeing more frequent interruptions, but reserved capacity is also available. Hope you email the TRC support team to say hello!

Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups

#100

Post by flaque »

agajews wrote: post #76
xeromal wrote: post #72

Earlier quoted context omitted.

I assume it's up to them to say no. They did say they're not in it to make bookoo bucks

Yeah we aren’t going to let anyone book the whole thing for years. If we ever have to make the choice, we’ll choose the startups over the big companies.

Yeah, if someone doesn't care about the cost and wants to buy whole cluster, they might be better off using an existing provider.
Post reply on HN