Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building A16Z's Personal AI Workstation

a16z.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Building A16Z's Personal AI Workstation

#61

Post by refulgentis »

CamperBob2 wrote: post #56
refulgentis wrote: post #16

Earlier quoted context omitted.

It's been such a mind-boggling decline in intellect, combined with really odd and intense conspiratorial behavior around crypto, that I went into a bit a few months ago. My weak, uncited, understanding from then they're poorly positioned, i.e in our set they're still the guys who write you a big check for software, but in the VC set they're a joke: i.e. they misunderstood carpet bombing investment as something that s…

If you're inferencing, you're not getting much more out of this than you would a ~$2K Framework desktop. Well, you're getting the ability to maintain a context bigger than 8K or so, for one thing.

Well, no, at least, we're off by a factor of about 64x at the very least: 64 GB GPU M2 Max/M4 max top out at about 512K context for 20B params, and the Framework desktop I am referencing has 128 GB unified memory.

Re: Building A16Z's Personal AI Workstation

#62

Post by karmakaze »

marcusb wrote: post #45

A rough price out using new egg/microcenter pricing, substituting a few items where they didn't specify a specific item/brand. I didn't bother trying to figure out what case they used. Grand total: ~ $41,000 Motherboard https://www.newegg.com/gigabyte-mh53-g40-amd-ryzen-threadrip... $895 CPU https://www.microcenter.com/product/674313/amd-ryzen-threadr... - $3500 Cooler https://www.newegg.com/p/3C6-013W-002G6 $585 RAM…

This could have been much more expensive. They built it from off-the-shelf parts anyone could get (except for the cost) and listed them which I appreciate. Microcenter/Newegg could put together a bundle as a joke and they'd likely get a few orders.

Of course I don't personally have any use for this but it's good to have an idea what it takes to run the best openweight models in a secure/controlled environment. To get started a single 96GB GPU system is only $16,115. For perspective I spent about $10k (today dollars) for a Toshiba Portege 320CT laptop with as much memory and accessories as I could get in 1998.

Re: Building A16Z's Personal AI Workstation

#63

Post by CamperBob2 »

refulgentis wrote: post #61
CamperBob2 wrote: post #56

Earlier quoted context omitted.

If you're inferencing, you're not getting much more out of this than you would a ~$2K Framework desktop. Well, you're getting the ability to maintain a context bigger than 8K or so, for one thing.

Well, no, at least, we're off by a factor of about 64x at the very least: 64 GB GPU M2 Max/M4 max top out at about 512K context for 20B params, and the Framework desktop I am referencing has 128 GB unified memory.

What's the TTFT like on a GPU-poor rig, though, once you actually take advantage of large contexts?

Re: Building A16Z's Personal AI Workstation

#64

Post by refulgentis »

CamperBob2 wrote: post #63
refulgentis wrote: post #61

Earlier quoted context omitted.

Well, no, at least, we're off by a factor of about 64x at the very least: 64 GB GPU M2 Max/M4 max top out at about 512K context for 20B params, and the Framework desktop I am referencing has 128 GB unified memory.

What's the TTFT like on a GPU-poor rig, though, once you actually take advantage of large contexts?

I guess I'd say, why is the framework perceived as GPU poor? I don't have one but I also don't know why TTFT would be significantly lower than M-series (it's a good GPU!)

Re: Building A16Z's Personal AI Workstation

#65

Post by CamperBob2 »

refulgentis wrote: post #64
CamperBob2 wrote: post #63

Earlier quoted context omitted.

What's the TTFT like on a GPU-poor rig, though, once you actually take advantage of large contexts?

I guess I'd say, why is the framework perceived as GPU poor? I don't have one but I also don't know why TTFT would be significantly lower than M-series (it's a good GPU!)

Compared to 4x RTX 6000 Blackwell boards, it's GPU poor. There has to be a reason they want to load up a tower chassis with $35K worth of GPUs, right? I'd have to assume it has strong advantages for inference as well as training, given that the GPU has more influence on TTFT with longer contexts than the CPU does.

Re: Building A16Z's Personal AI Workstation

#66

Post by jryle70 »

teaearlgraycold wrote: post #29

If some rich guys want to role play as engineers - sure go ahead. But making this article should be a humiliation.

HN at its worse. Whining about anything and everything. Role playing as engineers? Don't you think there are many engineers working for AZ16?

> But making this article should be a humiliation.

Article? It's their f*king website. Did you say the same when someone post their complaint about LLM on their blog?

Re: Building A16Z's Personal AI Workstation

#67

Post by jryle70 »

chis wrote: post #7

A16Z is consistently the most embarrassing VC firm at any given point in time. I guess optimistically they might be doing “outrage marketing” but it feels more like one of those places where the CEO is just an idiot and tells his employees to jump on every trend. The funny part is that they still make money. It seems like once you’ve got the connections, being a VC is a very easy job these days.

> consistently the most embarrassing VC firm at any given point in time

Based on what? your feelings?

> being a VC is a very easy job these days.

There you go. Why hasn't everyone who have connections became VC.

Re: Building A16Z's Personal AI Workstation

#68

Post by refulgentis »

CamperBob2 wrote: post #65
refulgentis wrote: post #64

Earlier quoted context omitted.

I guess I'd say, why is the framework perceived as GPU poor? I don't have one but I also don't know why TTFT would be significantly lower than M-series (it's a good GPU!)

Compared to 4x RTX 6000 Blackwell boards, it's GPU poor. There has to be a reason they want to load up a tower chassis with $35K worth of GPUs, right? I'd have to assume it has strong advantages for inference as well as training, given that the GPU has more influence on TTFT with longer contexts than the CPU does.

Right - I'd suggest the idea that 128 GB of GPU RAM gives you an 8K context shows us it may be worth revising priors such as "it has strong advantages for inference as well as training"

As Mr. Hildebrand used to say, when you assume, you make...

(also note the article specifically frames this speccing out as about training :) not just me suggesting it)

Re: Building A16Z's Personal AI Workstation

#69

Post by voidskie »

marcusb wrote: post #45

A rough price out using new egg/microcenter pricing, substituting a few items where they didn't specify a specific item/brand. I didn't bother trying to figure out what case they used. Grand total: ~ $41,000 Motherboard https://www.newegg.com/gigabyte-mh53-g40-amd-ryzen-threadrip... $895 CPU https://www.microcenter.com/product/674313/amd-ryzen-threadr... - $3500 Cooler https://www.newegg.com/p/3C6-013W-002G6 $585 RAM…

The case is an InWin Dubili with a modified, and color mismatched, front panel: https://estore.in-win.com/products/dubili?variant=4711876034...

Re: Building A16Z's Personal AI Workstation

#70

Post by ksec »

danjl wrote: post #39

If you actually want a good multi-GPU system, just get an NVIDIA-designed system. There's no good reason to try to design and build one yourself when you will be relying almost entirely on NVIDIA cards and their multi-GPU communication.

I think the DGS Station only has a Single Blackwell GPU. Are there any other variants that has Multi-GPU that is not a Cluster or a Rack?
Post reply on HN