Having hosted infrastructure in CA at multiple colos. I would advise you to host it elsewhere if you can, cost of power, other infrastructure is much higher in CA than AZ or NV.
Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
71–80 of 189 posts
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#72Earlier quoted context omitted.
Ah, we're running a medium amount of compute at zero-margin. The point is not to go sell the Fortune 500, but to make sure a grad student can spend a $50k grant. Right now, it's pretty easy to get a few A/H100s (Lambda is great for this), but very hard to get more than 24 at a reasonable price ($~2 an hour). One often needs to put up a 6+ month commitment, even when they may only want to run their H100s for an 8 hour…
So what happens when some big bucks VC backed closed source LLM company buys all your compute inventory for the next 5 years? This is not that unlikely. Lambda Labs a little while back was completely sold out of all compute inventory.
I assume it's up to them to say no. They did say they're not in it to make bookoo bucks
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#73Correct me if I’m wrong but doesn’t Lambda Labs already provide them at 1.89$? What’s the point if you’re starting out not the cheapest
Ah that’s only if you pay for 3 years of compute upfront. Most startups, especially the small ones, really can’t afford that
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#74> Rather than each of K startups individually buying clusters of N gpus, together we buy a cluster with NK gpus... Then we set up a job scheduler to allocate compute In theory, this sounds almost identical to the business model behind AWS, Azure, and other cloud providers. "Instead of everyone buying a fixed amount of hardware for individual use, we'll buy a massive pool of hardware that people can time-share." Outsi…
AWS and Azure would slit their own throats before they created a way for their customers to pool instances to save money.
They want to do that themselves, and keep the customer relationship and the profits, instead of giving them to a middleman or the customer.
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#75Earlier quoted context omitted.
For consumer-grade cards, that's already here. Make money off your GPU with vast.AI https://cloud.vast.ai/host/setup
> Requirements > Ubuntu 18.04 or newer (required) > Dedicated machines only - the machine shouldn't be doing other stuff while rented well that's certainly not what I expected. ctrl-f "virtual" gives nothing, so it seems they really mean "take over your machine" > Note: you may need to install python2.7 to run the install script. what kind of nonsense is this? Did they write the script in 2001 and just abandon it?
I just skimmed their FAQ at https://vast.ai/faq, and it seems like it could use an update. E.g., it says "Initially we are supporting Ubuntu Linux, more specifically Ubuntu 16.04 LTS.". That version of Ubuntu has been end-of-life'd for several years, and when I just tried vast.ai out, it seemed to be using Ubuntu 20.04. There were also a couple of words with letters missing (probably trivial typos) that could be found with a spell checker. The questions in their FAQ are really interesting though, in terms of highlighting what users care about (e.g., there's a lot devoted to "how do I use vast.ai + google colab together"?). I also wonder when vast.ai started? Sometimes you can get insight from a company blog page, but the vast.ai blog seems to start in Feb 2023: https://vast.ai/blog . There's a bunch of "personal experiences" with vast.ai from 3 years ago in this discussion though: https://www.reddit.com/r/MachineLearning/comments/hv49pd/d_c...
A comment in that discussion mentions yet another competitor in this space that I've never heard of: https://www.qblocks.cloud/ -- I just tried Q blocks out and the new user experience wasn't as good for me as with vast.ai: you have to put in $10 money to try it, instead of getting to try it initially for free; there is a manual approval process before you can try data center class GPUs; you only see that your instance is in Norway (say) after you try to start it, not before; it seems like there's no ssh access, and they only provide Jupyter to connect; neither pytorch nor tensorflow seemed to be installed. They could probably update their pages too, e.g., https://www.qblocks.cloud/vision is all about crypto mining and smartphones, which feels a bit dated... :-)
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#76Earlier quoted context omitted.
So what happens when some big bucks VC backed closed source LLM company buys all your compute inventory for the next 5 years? This is not that unlikely. Lambda Labs a little while back was completely sold out of all compute inventory.
I assume it's up to them to say no. They did say they're not in it to make bookoo bucks
Yeah we aren’t going to let anyone book the whole thing for years. If we ever have to make the choice, we’ll choose the startups over the big companies.
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#77Earlier quoted context omitted.
> Requirements > Ubuntu 18.04 or newer (required) > Dedicated machines only - the machine shouldn't be doing other stuff while rented well that's certainly not what I expected. ctrl-f "virtual" gives nothing, so it seems they really mean "take over your machine" > Note: you may need to install python2.7 to run the install script. what kind of nonsense is this? Did they write the script in 2001 and just abandon it?
> what kind of nonsense is this? Did they write the script in 2001 and just abandon it? Anything AI/ML is a hot mess of cobbled-together bits and pieces of Python barely holding together. I recently read somewhere that there should be a new specialization of "ML DevOps Engineer"... and hell I'm supporting that.
there should be a new specialization of "ML DevOps Engineer"
Do you mean MLOps? Nothing new about it. We have two full-time MLOps engineers at our startup.
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#78Earlier quoted context omitted.
> Your project has a youthful optimism that I hope you won’t lose as you go. And in fact it might be the way to win in the long run. This is the nicest thing anyone has said to us about this. We're gonna frame this and hang it out on our wall. > So whenever someone comes knocking, begging for a tiny slice of your H100s for their harebrained idea, I hope you’ll humor them. Absolutely! :D
Optimism is (almost) always required in order to accomplish anything of significance. Those who lose it, aren't living up to their potential. I'm not encouraging the false belief that everything you do will work out. Instead I'm encouraging the realization that the greatest accomplishments almost always feel like long shots, and require significant amounts of optimism. Fear and pessimism, while helpful in appropriate…
YC startup founder here,
Mostly agree, except the market is not an optimistic place — it’s the market.
There are a multitude of reasons you lose your optimism, mostly because people take it away — your optimism is their money
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#79Earlier quoted context omitted.
From sentence one of the post, it clearly states that they are VC funders who are doing this for a round of startups they just funded, and they're looking for others to be a part of it.
Oh no, definitely not. We just got a loan. Neither Alex or I are currently VCs, and this has no affiliation with any venture fund. We want to be a customer of the sf compute group too!
I’m curious, how are those loans guaranteed?
Re: Show HN: San Francisco Compute – 512 H100s at <$2/hr for research and startups
#80I love the idea of community assets. could it be the start of a GPU co-op?
Serious Q, as I dont know Twitters internal infra at all... but with a shrinking in revenue from ads, or maybe less engagement by users, and the influx of Threads - maybe twitter can use from slices of its infra (even if its rack space, VMs, Containers, connectivity, who knows what, to support startups such as this? Basically twitter devolves into the Colos of the late 90s :-) - For those who didnt notice, it was ton…
Generally when you just stop paying your bills the datacenter holds your hardware and eventually auctions it off to cover some of your debt. I seriously doubt Twitter has any access to the two of three datacenters Elon decided to not pay for.