Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I regret building this $3000 Pi AI cluster

jeffgeerling.com

Post reply on HN
Print view View original

301–310 of 377 posts

Re: I regret building this $3000 Pi AI cluster

#301

Post by geerlingguy »

milesvp wrote: post #298
geerlingguy wrote: post #98

Earlier quoted context omitted.

That's the biggest regret; but I've backed 6 Kickstarter projects over the years. Median time to deliver is 1 year. Somehow I've actually gotten every item I backed shipped at some point (which is unexpected). Hardware startups are _hard_, and after interacting with a number of them (usually one or two people with a neat idea in an underserved market), it seems like more than half fail before delivering their first r…

Seriously. Imagine the fractal nature of software problems (you do need firmware right?), and add that on top of a layer that has poorly documented errata and non standard peripheral variants. I'm currently making a board to solder onto an existing production run because we finally found the 1 sentence in a 1700page data sheet that explained why many of our MCUs are overheating. Had the datasheet simply labeled VREFP…

The stories hardware devs have about individual chip datasheets causing hours or weeks of pain...

It's most fun when you can prove the vendor's datasheet is lying about some pin or some function, but they still don't update it after a decade or more. So everyone integrating the chip who hasn't before hits the exact same speed bump!

Re: I regret building this $3000 Pi AI cluster

#302

Post by subscribed »

qmr wrote: post #282
densh wrote: post #238

For anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 virtual machines on it. 8 virtual machines, with 4 hyper threads pinned per machine, and 1/8 of total RAM per machine. Create a network between them virtually within your virtualization software of choice (such as Proxmox). And suddenly you can start playing with…

No need for so much CPU power, any old quad core would work.

Old quad core won't have all the virtualisation extensions.

Re: I regret building this $3000 Pi AI cluster

#303

Post by geerlingguy »

jonathanlydall wrote: post #228
wccrawford wrote: post #116

Earlier quoted context omitted.

Geerling's titles have been increasingly click-bait for a while now. It's pretty sad, because I like his content, but hate the click-bait BS.

If it makes an appreciable difference to how much money he makes on YouTube then I can’t begrudge him for doing it. Don’t hate the player, hate the game.

Just to add context — I've been experimenting on my 2nd channel (Level 2 Jeff) with titles that are straight/barebones exactly describing the content of the video, vs a slight bit of clickbait (never untrue, but certainly more intriguing and not describing the exact topic of the video).

The ones that are dead straight with no clickbait are 10/10 (the worst performers), and usually by a massive margin. Even with the same thumbnail.

The sad fact is, if you want your work seen on YouTube, you can't just say "I built a 10 node Raspberry Pi blade cluster and ran HPL and LLMs on it".

Some people are fine with a limited audience. And that's fine too! I don't have to write on my blog at all—I earn negative income from that, since I pay for hosting and a domain, but I hope some people enjoy the content in text form like I do.

Re: I regret building this $3000 Pi AI cluster

#304

Post by geerlingguy »

djhworld wrote: post #271

I watched the video and enjoyed it, I think the most interesting part to me was running the distributed Llama.cpp, Jeff mentioned it seems to work in a linear fashion where processing would hop between nodes. Which got me thinking about how do these frontier AI models work when you (as a user) run a query. Does your query just go to one big box with lots of GPUs attached and it runs in a similar way, but much faster?…

ServeTheHome has a few videos covering AI servers and interconnects.

And yes, they basically have 1 Tbps+ interconnects and throw tens or hundreds of GPUs at queries. Nvidia was wise to invest so much in their networking side—they have massive bandwidth between machines and shared memory, so they can run massive models with tons of cards, with minimal latency.

It's still not as good as tons of GPU attached to tons of memory on _one_ machine, but it's better than 10, 25, or 40 Gbps networking that most small homelabs would run.

Re: I regret building this $3000 Pi AI cluster

#305

Post by ssl-3 »

kllrnohj wrote: post #257
Kubuxu wrote: post #234

Earlier quoted context omitted.

For 8+ bays you just need a SAS HBA card and one free PCI-E slot. Not to mention that many motherboards will have 6+ SATA ports already. If anything, 2nd hand AMD gaming rigs make more sense than old servers. I say that as someone with always off r720xd at home due to noise and heat. It was fun when I bought it during winter years ago, until summer came.

> For 8+ bays you just need a SAS HBA card and one free PCI-E slot. Not to mention that many motherboards will have 6+ SATA ports already. And what case are you putting them into? What if you want it rack mounted? What about >1gig networking? What if I want a GPU in there to do whisper for home assistant? Used gaming rigs are great. But used servers also still have loads of value, too. Compute just isn't one of them.

> And what case are you putting them into?

Maybe one of the Fractal Designs cases with a bunch of drive bays?

> What if you want it rack mounted?

Companies like Rosewill sell ATX cases that can scratch that itch.

> What about >1gig networking?

What about PCI Express card? Regular ATX computers are expandable.

> What if I want a GPU in there to do whisper for home assistant?

I mean... We started with a gaming rig, right? Isn't a GPU already implicit?

Re: I regret building this $3000 Pi AI cluster

#306

Post by Computer0 »

privatelypublic wrote: post #287
michaelt wrote: post #265

Earlier quoted context omitted.

Then you should take a good hard look at older, much cheaper Raspberry Pis. Then look at Apple’s ARM offerings, and AWS Graviton if you need ARM with raw power. If you need embedded/GPIO you should consider an Arduino, or clone. If you need GPIOs and Internet connectivity, look at an ESP32. GPIOs, ARM and wired ethernet? Consdier the the STM32H. Robotics/machine vision applications, needing IO and lots of compute pow…

I've heard nothing but horror stories on the Jetson & Tegra in general. I'd Avoid it unless the project MUST use a SoM w/ CUDA. which will basically only be Professional stuff. I've never heard of anything hobby level where a PCIe slot was a deal breaker- even with high vibration. (PCIe 4.0 isn't terrible difficult to get good flex cables for)

I hate my Jetson, and the hardware is fine for my purpose.

Re: I regret building this $3000 Pi AI cluster

#307

Post by kortex »

michaelt wrote: post #265
Computer0 wrote: post #253

Earlier quoted context omitted.

What if you need ARM? What is the best sub $100 sbc that I am missing? Orange Pi hardware always looks good but I hear a lot of negativity about the software that I don't really experience with Raspbian.

Then you should take a good hard look at older, much cheaper Raspberry Pis. Then look at Apple’s ARM offerings, and AWS Graviton if you need ARM with raw power. If you need embedded/GPIO you should consider an Arduino, or clone. If you need GPIOs and Internet connectivity, look at an ESP32. GPIOs, ARM and wired ethernet? Consdier the the STM32H. Robotics/machine vision applications, needing IO and lots of compute pow…

There are a lot of reasons you probably would want a Pi over an ESP32 (or in addition to one), e.g. you want GPIO, plus, internet connectivity, and want to run certain linux programs (e.g. full python, not micropython), or need timesharing, or any number of reasons you might want a linux box over an embedded.

But single board computers with something external to do your GPIO is often way more compelling.

Re: I regret building this $3000 Pi AI cluster

#308

Post by malux85 »

densh wrote: post #238

For anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 virtual machines on it. 8 virtual machines, with 4 hyper threads pinned per machine, and 1/8 of total RAM per machine. Create a network between them virtually within your virtualization software of choice (such as Proxmox). And suddenly you can start playing with…

Yeah this is how I practiced Postgres hot standby and read replicas,

It was also how I learned to setup a Hadoop cluster, and a Cassandra cluster (this was 10 years ago when these technologies were hot)

Having knowledge of these systems and being able to talk about how I set them up and simulated recovery directly got me jobs that 2x and then 3x my salary, I would highly recommend all medium skilled developers setup systems like this and get practicing if you want to get up into the next level

Re: I regret building this $3000 Pi AI cluster

#310

Post by oasisaimlessly »

privatelypublic wrote: post #288
ssl-3 wrote: post #268

Earlier quoted context omitted.

One cannot (or at least, one is not supposed to be able to) patent someone else's invention.

You theoretically have a year before you even have to apply- but patents are expressly "first to file."

Public use of an idea still prevents someone else from patenting it.
Post reply on HN