161–170 of 213 posts
How will Meta leverage LLMs at scale to drive revenue? It's not clear.
2. virtually free moderation for their existing (facebook, instagram, threads) and future social media services. likewise, their volume is too insane to even consider paying someone else to process it.
the models they do release are probably toys in comparison to their internal models.
OK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line GPUs falling off: In this case, GPUs are not detected by the host on PCIe.
I was imagining that some sys admin has to walk to the server, take out the GPU, blow against the PCI-E pins like a game cartridge, and put it back to try again.
Earlier quoted context omitted.
> GPU falling off the bus I'm wondering if we could prompt llama3 with the above statement. What kind of response would it give?
With temperature set to 1, it recognizes the joke, but proceeds to explain what the "bus" is in computer terms, picks a problem this prompt could mean, and explains how to solve it. In ~20 tries it always gave me something along the lines of: The infamous "GPU falling off the bus" issue! This problem typically occurs when a graphics processing unit (GPU) is not properly seated or connected to its expansion slot, such…
Earlier quoted context omitted.
With temperature set to 1, it recognizes the joke, but proceeds to explain what the "bus" is in computer terms, picks a problem this prompt could mean, and explains how to solve it. In ~20 tries it always gave me something along the lines of: The infamous "GPU falling off the bus" issue! This problem typically occurs when a graphics processing unit (GPU) is not properly seated or connected to its expansion slot, such…
+1, interesting findings! I like how it was able to infer the meaning from such a short phrase in a limited context.
You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.
And then the query was:
GPU falling off the bus
And yes, I imagine it read that query as ending with an implied "pls help!".
These seem classic challenges with running distributed systems loads that are not specific to training LLMs. Anyone of the super computers listed here https://en.wikipedia.org/wiki/TOP500 suffers from the same issues. Think about it. While the national labs use these systems to model serious stuff -such as climate or nuclear weapons- Meta uses them to train LLMs. What a joke, honestly!
Would be nice to read how do they collect/prepare data for training. Which data sources? How much of Meta users data (fb, instagram… etc). How do they sanitize PII?
By literally opting everyone into their training data set and making it very cumbersome to opt out: https://threadreaderapp.com/thread/1794863603964891567.html
Yikes, the little Infiniband+A100 cluster I installed for my previous company seemed useful at the time (12 GPUs) and that was at a cost of around $300k. With LLMs it feels like game over for non-cloud applications if you are not a mega-corp.
Well, yes, but Not all models need to be "super large." Smaller models, specialized in specific tasks, working together - and then reporting to a slightly larger model is the way to go. Think of everything being connected to a "Home Computer" in those "Future House of 2020" videos that were out there in 70s or what not. Another example (very rough) would be something like "Weather data gets to a small model via an AP…
I want to believe, but I'm still yet to see this kind of set up being anywhere near GPT-4 level.
The weather example seems quite contrived. Why not just display the alerts for your area? Why is a complex system of smaller models reporting up to a slightly larger model necessary?
Earlier quoted context omitted.
They are building racks of 4090s. Nobody can get H100s in any reasonable volume. Hell, Microsoft is renting GPUs from Oracle Cloud to get enough capacity to run Bing.
Who is "they"? RTX 4090s are terrible for this task. Off the top of my head: - VRAM (obviously). Isn't that where the racks come in? Not really. Nvidia famously removed something as basic as NVLink between two cards from the 3090 to the 4090. When it comes to bandwidth between cards (crucial) even 16 lanes of PCIe 4 isn't fast enough. When you start talking about "racks" unless you're running on server grade CPUs (co…
A few years ago, if you wanted a lot of GPU power you would buy something like [1] - a 4/5U server with space for ten dual-slot PCIe x16 cards and quadruple power supplies for 2000W of fully redundant power. And not a PCIe riser in sight.
I share your scepticism about whether it's common to run >2 4090s because nvidia have indeed sought to make it difficult.
But if there was some sort of supply chain issue that meant you had to, and you had plenty of cash to make it happen? It could probably be done.
Some of the more value-oriented GPU cloud suppliers like RunPod offer servers with multiple 4090s and I assume those do something along these lines. With 21 slots in the backplane, you could probably fit 6 air-cooled three-slot GPUs, even if you weren't resorting to water cooling.
[1] https://www.supermicro.com/en/products/system/4U/4028/SYS-40...