Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.1

llama.meta.com

Post reply on HN
Print view View original

261–270 of 279 posts

Re: Llama 3.1

#261

Post by karolist »

api wrote: post #146
paxys wrote: post #137

Earlier quoted context omitted.

You don't need a model of this scale for personal use. Llama 3.1 8B can easily run on your laptop right now. The 70B model can run on a pair of 4090s.

I have the 70b model running quantized just fine on an M1 Max laptop with 64GiB unified RAM. Performance is fine and so far some Q&A tests are impressive. This is good enough for a lot of use cases... on a laptop. An expensive laptop, but hardware only gets better and cheaper over time.

I have the same machine, may I ask which model file and program are you using? Is it partial GPU offload?

Re: Llama 3.1

#262

Post by rovr138 »

nomel wrote: post #166
fngjdflmdflg wrote: post #160

Earlier quoted context omitted.

As far as I know it's not just the weights. it's everything but the dataset. So the code used to generate the weights is also open source.

Is there any other case where "open source" is used for something that can't be reproduced? Seems like a new term is required, in the concept of "open source, non-reproducible artifacts". I suppose language changes. I just prefer it changes towards being more precise, not less.

Academia - nowadays source is needed is a lot of conferences, but the datasets, depending on where/how it might have be obtained, just can't be used or not available and the exact results can't be reproduced.

Not sure if the code is required under an open source license, but it's the same issue.

---

IMO, source is source and can be used for other datasets. Dataset isn't available, bring your own.

In this case, the source is there. The output is there, and not technically required. What isn't available is the ability to confirm the output comes from that source. That's not required under open source though.

What's disingenuous is the output being called 'open source'.

Re: Llama 3.1

#263

Post by evilduck »

diffeomorphism wrote: post #236
api wrote: post #146

Earlier quoted context omitted.

I have the 70b model running quantized just fine on an M1 Max laptop with 64GiB unified RAM. Performance is fine and so far some Q&A tests are impressive. This is good enough for a lot of use cases... on a laptop. An expensive laptop, but hardware only gets better and cheaper over time.

Just for reference, the current version of that laptop costs 4800€ (14 inch macbook pro, m3 max, 64gb of ram, 1TB of storage). So price-wise that is more like four laptops.

I think they were referring to the form factor not the price. But even then the price of four laptops is not out of line for enthusiast hobby spending.

Ever priced out a four wheeler, a jet-ski, a filled gun safe, what a "car guy" loses in trade in values every two years, what a hobbyist day-trader is losing before they cut their losses or turn it around, or what a parent who lives vicariously through their child and drags them all over their nearby states for overnight trips so they can do football/soccer/ballet/whatever at 6am on Saturdays against all the other kids who also won't become pro athletes? What about the cost of a wingsuit or getting your pilots license? "Cruisers" or annual-Disney vacationers? If you bought a used CNC machine from a machine shop? But spend five grand on a laptop to play with LLMs and everyone gets real judgmental.

Re: Llama 3.1

#264

Post by bick_nyers »

angoragoats wrote: post #145
bick_nyers wrote: post #131

Earlier quoted context omitted.

Energy costs are an important factor here too. While Quadro cards are much more expensive upfront (higher $/VRAM), they are cheaper over time (lower Watts/Token). Offsetting the energy expense of a 3090/4090/5090 build via solar complicates this calculation but generally speaking can be a "reasonable" way of justifying this much hardware running in a homelab. I would be curious to see relative failure rates over time…

Agree 100% that energy costs are important. The example system in my other post would consume somewhere around 300W at idle, 24/7, which is 219 kWh per month, and that's assuming you aren't using the machine at all. I don't have any actual figures to back this up, but my gut tells me that the fact that enterprise GPUs are an order of magnitude (at least) more expensive than, say a, 3090, means that the payback period…

Agreed, but there are other costs associated with supporting 10-16x GPUs that may not necessarily happen with say 6 GPUs. Having to go from single socket (or Threadripper) to dual socket, PCIE bifurcation, PLX risers, etc.

Not necessarily saying that Quadros are cheaper, just that there's more to the calculation when trying to run 405B size models at home

Re: Llama 3.1

#265

Post by bick_nyers »

lostmsu wrote: post #185
bick_nyers wrote: post #131

Earlier quoted context omitted.

Energy costs are an important factor here too. While Quadro cards are much more expensive upfront (higher $/VRAM), they are cheaper over time (lower Watts/Token). Offsetting the energy expense of a 3090/4090/5090 build via solar complicates this calculation but generally speaking can be a "reasonable" way of justifying this much hardware running in a homelab. I would be curious to see relative failure rates over time…

I don't think this is correct. 5 years power usage of 4090 is $2600 giving TCO of ~$4300. RTX 6000 Ada starts at $6k for the card itself. https://gpuprices.us

To be fair, you need 2x 4090 to match the VRAM capacity of an RTX 6000 Ada. There is also the rest of the system you need to factor into the cost. When running 10-16x 4090s, you may also need to upgrade your electrical wiring to support that load, you may need to spend more on air conditioning, etc.

I'm not necessarily saying that it's obviously better in terms of total cost, just that there are more factors to consider in a system of this size.

If inference is the only thing that is important to someone building this system, then used 3090s in x8 or even x4 bifurcation is probably the way to go. Things become more complicated if you want to add the ability to train/do other ML stuff, as you will really want to try to hit PCIE 4.0 x16 on every single card.

Re: Llama 3.1

#266

Post by lostmsu »

bick_nyers wrote: post #265
lostmsu wrote: post #185

Earlier quoted context omitted.

I don't think this is correct. 5 years power usage of 4090 is $2600 giving TCO of ~$4300. RTX 6000 Ada starts at $6k for the card itself. https://gpuprices.us

To be fair, you need 2x 4090 to match the VRAM capacity of an RTX 6000 Ada. There is also the rest of the system you need to factor into the cost. When running 10-16x 4090s, you may also need to upgrade your electrical wiring to support that load, you may need to spend more on air conditioning, etc. I'm not necessarily saying that it's obviously better in terms of total cost, just that there are more factors to consi…

With 2x 4090 you will have 2x speed of RTX 6000 A. So same energy per token.

Will need more space, true.

Re: Llama 3.1

#267

Post by 93po »

Teknomancer wrote: post #81
93po wrote: post #65

Earlier quoted context omitted.

according to another comment, ~10x 4090 video cards.

That was the punchline of a joke.

lol thanks, i know nothing about the hardware side of things for this stuff

Re: Llama 3.1

#268

Post by angoragoats »

bick_nyers wrote: post #264
angoragoats wrote: post #145

Earlier quoted context omitted.

Agree 100% that energy costs are important. The example system in my other post would consume somewhere around 300W at idle, 24/7, which is 219 kWh per month, and that's assuming you aren't using the machine at all. I don't have any actual figures to back this up, but my gut tells me that the fact that enterprise GPUs are an order of magnitude (at least) more expensive than, say a, 3090, means that the payback period…

Agreed, but there are other costs associated with supporting 10-16x GPUs that may not necessarily happen with say 6 GPUs. Having to go from single socket (or Threadripper) to dual socket, PCIE bifurcation, PLX risers, etc. Not necessarily saying that Quadros are cheaper, just that there's more to the calculation when trying to run 405B size models at home

The system I outlined in my other post [0] has ten GPUs and does not require dual socket CPUs as far as I'm aware. It could likely scale easily to 14 GPUs as well (assuming you have sufficient power), with an x8/x8 bifurcation adapter installed in each PCIe slot. This is pushing the limits of the PCIe subsystem I'm sure, but you could also likely scale up to 28 GPUs, again assuming sufficient power, by simply bifurcating at x4/x4/x4/x4 vs x8/x8.

I think it should work as-is with the components listed, but if you disagree please let me know!

[0] https://news.ycombinator.com/item?id=41047689

Re: Llama 3.1

#269

Post by angoragoats »

lostmsu wrote: post #266
bick_nyers wrote: post #265

Earlier quoted context omitted.

To be fair, you need 2x 4090 to match the VRAM capacity of an RTX 6000 Ada. There is also the rest of the system you need to factor into the cost. When running 10-16x 4090s, you may also need to upgrade your electrical wiring to support that load, you may need to spend more on air conditioning, etc. I'm not necessarily saying that it's obviously better in terms of total cost, just that there are more factors to consi…

With 2x 4090 you will have 2x speed of RTX 6000 A. So same energy per token. Will need more space, true.

Yeah, after digging more into RTX 6000 Ada cards, I don't see any way they'd be more economical even over many years, no matter how you slice it.

Re: Llama 3.1

#270

Post by jiriro »

Can this Llama process ~1GB of custom XML data?

And answer queries like:

Give all which refer to which refer to an Indo-European .

Post reply on HN