81–87 of 87 posts
Wow, software is hard! Imagine an entire company working to build an insanely huge and expensive wafer scale chip and your super smart and highly motivated machine learning engineers get 1/3 of peak performance on their first attempt. When people say NVIDIA has no moat I'm going to remember this - partly because it does show that they do, and partly because it shows that with time the moat can probably be crossed...
Earlier quoted context omitted.
> There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time. Well now the compiler is the bottleneck isn't it? And you would still need human check for bugs that aren't caught by the compiler. Still nice to have inference spe…
Something will always be the bottleneck, and it probably won’t be the speed of electrons for a while ;) Some compilers (go) are faster than others (javac) and some languages are interpreted and can only be checked through tests. Moving the bottleneck from AI code gen step to the same bottleneck as a person seems like a win.
I wonder if there is a token/watt metric. Afaiu cerebras uses plenty of power/cooling.
I found this on their product page, though just for peak power: > At 16 RU, and peak sustained system power of 23kW, the CS-3 packs the performance of a room full of servers into a single unit the size of a dorm room mini-fridge. It's pretty impressive looking hardware. https://cerebras.ai/product-system/
Earlier quoted context omitted.
Whisper runs so well locally on any hardware I’ve thrown at it, why run it in the cloud?
That's pretty much exactly how I started. Ran whisper.cpp locally for a while on a 3070Ti. It worked quite well when n=1. For our use case, we may get 1 audio file at a time, we may get 10. Of course queuing them is possible but we decided to prioritize speed & reliability over self hosting.
Earlier quoted context omitted.
Whisper runs so well locally on any hardware I’ve thrown at it, why run it in the cloud?
Does it run well on CPU? I've used it locally but only with my high end (consumer/gaming) GPU, and haven't got round to finding out how it does on weaker machines.
Wow, software is hard! Imagine an entire company working to build an insanely huge and expensive wafer scale chip and your super smart and highly motivated machine learning engineers get 1/3 of peak performance on their first attempt. When people say NVIDIA has no moat I'm going to remember this - partly because it does show that they do, and partly because it shows that with time the moat can probably be crossed...
make it work, make it work right(ish), now make it fast.
Earlier quoted context omitted.
Something will always be the bottleneck, and it probably won’t be the speed of electrons for a while ;) Some compilers (go) are faster than others (javac) and some languages are interpreted and can only be checked through tests. Moving the bottleneck from AI code gen step to the same bottleneck as a person seems like a win.
Spelling out the code in editor is not really the bottleneck.
With LLM-pair-programing, you can basically say "add a button to this widget that calls this callback" or "call this API with the result of this operation", and the LLM will spit out code that does that thing. If your change is entirely within 1-2 files, and It's human-driven, and the LLM just handles the writing. The LLM isn't doing large refactors, nor is it designing scalable systems on its own. A human is doing that still. But it does speed up the process noticeably.