131–140 of 331 posts
> The human brain has about 20 PFLOPS of compute. Where is this number coming from? The number of spikes per second? Edit: doing a quick search, it doesn’t seem like there’s a consensus on the order of magnitude of this. Here’s a summary of various estimates: https://aiimpacts.org/brain-performance-in-flops/
Used 20 PFLOPS of compute to simulate it.
> I started tinygrad in Oct 2020. It started as a toy project to teach me about neural networks Shows you what is possible in 2.5 years. Keeps me motivated to learn.
The math isn't super difficult. Some books will try to throw a mess of differential equations at you, but some simple calculus is all you need for backpropagation.
For background, "geohot", is George Hotz, who's a known hacker / tech personality[0] This project fits the pattern of his previous projects: he gets excited about the currently hot thing in tech, makes his own knockoff version, generates a ton of buzz in the tech press for it, and then it fizzles out because he doesn't have the resources or attention span to actually make something at that scale. In 2016, Tesla and s…
My opinion of geohot definitely dropped after he started tweeting how easy it would be to fix Twitter, and then he started soliciting free work. He obviously underestimated the difficulty of shipping a feature across web and mobile. Hacking a prototype is trivial. Making it work well for all platforms, fully accessible, and across all supported languages is a bigger hurdle. It just gave me the impression that he thou…
Why wouldn't AMD throw a few million at this? Worst case they lose a small amount of money, but best case they finally get good software for their hardware. The past decade or so, they haven't been able to create any good software for their hardware. They made small improvements but the competition, Nvidia, has also made improvements to their already good software. It too the point where their software is the reason…
They tried at some point. Like another commenter pointed out elsewhere HW people just don't care about SW. They think HW is superior and SW is the joke. I don't think much of the culture has changed.
A lot of their understanding = workaround the HW, make it work, etc.
Earlier quoted context omitted.
AFAIK the "AMDs drivers are bad" meme is outdated. Sure, their AI/ML software is garbage, but the graphics drivers are fine.
I just had my AMD-based machine crash - repeatedly - every time I tried to use Google Maps inside Firefox for longer than a few minutes. I haven't confirmed, but I strongly assume that either their graphics drivers or something Ubuntu does with Wayland are not fine.
Earlier quoted context omitted.
They're slowing burning through their VC money trying to make a business out of the hobbyist market while Cruise and Waymo have fully autonomous cars deployed in SF and are scaling up.
As a very happy user of Comma, I think it is reasonable to say the company is going to fail, but that ignores that the product they created is still awesome. Comma is light years better than any built-in driving assist in any non-Tesla car. And it's comparable to Tesla for far less money. The reason the company might fail is because their main thesis, being that car manufacturers would just license the self driving t…
Can anyone comment on the TinyBox they are taking preorders for? The tinybox 738 FP16 TFLOPS 144 GB GPU RAM 5.76 TB/s RAM bandwidth 30 GB/s model load bandwidth (big llama loads in around 4 seconds) AMD EPYC CPU 1600W (one 120V outlet) Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks) $15,000
I like George's style and wish him well. But I'm not optimistic about their chances of selling $15k servers that are $10k in parts (or whatever the exact numbers are). It's just too easy for anyone to throw together a Supermicro machine with 6x GPUs in it, which is what it sounds like they'll be doing. My guess is they'll end up creating some premium extensions to the software and selling that to make money. Or maybe…
Have you seen what a DGXA100 costs? It starts at $199k for 8 40GB A100's, which have a list price of $10k each. So the GPU costs are $80k. What do you get for the extra $120k? 1TB ram, 2 2TB NVMe OS drives, 4 4TB NVME general storage, and 8x200Gbit infiniband. I would guess no more than 20k all of the remaining hardware. So that's a ~$100k computer selling for $200k. And that's with NVDA likely making massive margins already on the A100 and the Infiniband hardware.
The reality is that companies want to buy complete solutions, not to build and manage their own hardware. A $15k a computer that's $10k in parts is not a large markup at all for something like this.
> I think the only way to start an AI chip company is to start with the software. The computing in ML is not general purpose computing. 95% of models in use today (including LLMs and image generation) have all their compute and memory accesses statically computable. Agree with this insight. One thing Nvidia got right was a focus on software. They introduced CUDA [1] back in 2007 when the full set of use cases for it…
I know the founders of an AI chip company that taped out and got working chips on their first go. They got their chip done, it’s pretty solid. Chip has great perf and is super power efficient, a solid delivery. I knew they'd nail it and they did. The SW story is a train wreck, though. The problem basically was that they couldn’t hire any good SW people. As I said I know the founders. They are both genuinely decent gu…
> In 2022, FuriosaAI remained the only startup to submit results in MLPerf Inference... This time, through purely enhancements in the compiler, our team was able to double the performance on the exact same silicon.
Maybe it helped they are based in South Korea. Other places to work in South Korea doing system programming is not very attractive.
> There’s a [Radeon RX 7900 XTX 24GB] already on the market. For $999, you get a 123 TFLOP card with 24 GB of 960 GB/s RAM. This is the best FLOPS per dollar today, and yet…nobody in ML uses it. > I promise it’s better than the chip you taped out! It has 58B transistors on TSMC N5, and it’s like the 20th generation chip made by the company, 3rd in this series. Why are you so arrogant that you think you can make a bet…
I know a fair amount about this problem, my last startup built a working prototype of a performance-portable deep learning framework that got good performance out of AMD cards. The compiler stack is way harder than most people appreciate because scheduling operations for GPUs is very specific to the workload, hardware, and app constraints. The two strongest companies I'm aware of that are working in this area now are…
When we're presented with problems where the two potential answers are "it's a lot harder than it looks" and "the people working on it are idiots" I tend to lean towards the former. But hey, when it is the latter there's usually a good market opportunity. Just I've found that domain expertise is seeing the nuance that you miss when looking at 10k ft.
Earlier quoted context omitted.
I know a fair amount about this problem, my last startup built a working prototype of a performance-portable deep learning framework that got good performance out of AMD cards. The compiler stack is way harder than most people appreciate because scheduling operations for GPUs is very specific to the workload, hardware, and app constraints. The two strongest companies I'm aware of that are working in this area now are…
This. Modular and OctoML are building on top of MLIR and TVM respectively. > It's pretty hard to really beat NVIDIA for developer support though, they've invested a lot of work into the CUDA ecosystem over the years and it shows. Yup, strong CUDA community and dev support. That said, more ergonomic domain specific languages like Mojo might finally give CUDA some competition though - it's still a very high bar for sur…