261–270 of 314 posts
Earlier quoted context omitted.
Well since I'm not a ML engineer of any kind - that's good!
at the end of the day, you are still moving, storing and manipulating 1's and 0's, whether you are a front end engineer or a backend engineer or systems engieer or an ML engineer or an infra engineer
Earlier quoted context omitted.
The real question is: why aren't they? They had the infrastructure needed to seed a cloud offering 10 years ago. Heck, if Oracle managed to be in 5th (6th? 7th?) place, Facebook for sure could have been a top 5 contender, at least.
Because they make more money using their servers for their own products than they would renting them to other people. Meta has an operating margin of 41% AFTER they burn a ton on Reality Labs, while AWS has a 21% margin with more disciplined spending. Social media is a more profitable business than infrastructure.
Having lived through the dot-com era, I find the AI-era slightly dispiriting because of the sheer capital cost of training models. At the start of the dot-com era, anyone could spin up an e-commerce site with relatively little infrastructure costs. Now, it seems, only the hyper-scale companies can build these AI models. Meta, Google, Microsoft, Open-AI, etc.
Earlier quoted context omitted.
> And if you think any Jack or Jill can just come in and text prompt a whole movie, you're crazy. It's still hard work and a metric ton of good taste. If you want anything good , yes. If you just want something … I reckon it'd take a week to assemble an incomprehensible-nonsense-film pipeline, after which it's just a matter of feeding the computer electricity. Short-term, this is going to funnel resources away from t…
> If you want anything good, yes. If you just want something ... You don't even need AI for that. https://en.wikipedia.org/wiki/YouTube_poop https://en.wikipedia.org/wiki/Skibidi_Toilet The idea that AI isn't going to be used as a creative tool too and that it won't lead to more and better art is a defeatist, Luddite attitude. Similarly shaped people thought that digital cameras would ruin cinema and photography. > S…
Earlier quoted context omitted.
Is there float8 support in any common CPU intrinsics? It sounds interesting but curious what will be the impact if any on CPU inference.
I’m curious if there’s a meaningful quality difference between float8 and some uint8 alternative (fixed precision or a look up table).
Plain uint8 wouldn’t allow for the same accuracy range as float8 and it’s the accuracy not the precision (which uint would win for the largest values it can represent) that counts most.
Earlier quoted context omitted.
I’m curious if there’s a meaningful quality difference between float8 and some uint8 alternative (fixed precision or a look up table).
A LUT could be a significant performance penalty would it not? Instead of a float8 (potentially multiple in simd case) in a register, you’re now having to head out to at least L1 cache to dereference the value in the LUT. Plain uint8 wouldn’t allow for the same accuracy range as float8 and it’s the accuracy not the precision (which uint would win for the largest values it can represent) that counts most.
The issue with LUTs is don't you have to update the LUT itself? You can select which memory address to load up, but the LUT itself has to be differentiable maybe? TBH I'm not an expert on LUTs.
On fixed point - similarly ye you have to fix the precision ranges as well, so again I'm unsure on how one changes the fixed point numbers over time. I'll have to read more on fixed point.
Maybe 1.58bit using (-1, 0, 1) which gets rid of multiplications and just additions might be more useful, although you'll only get a 2x FLOP boost since you still need fp8 or fp16 addition.
Earlier quoted context omitted.
FB does not have the flywheel of running data centres - all three of those mentioned run hyper scale datacentres that they can then juice by “investing” billions in AI companies who then turn around and put those billions as revenue in the investors OpenAI takes money from MSFT and buys Azure services Anthropic takes Amazon money and buys AWS services (as do many robotics etc) I am fairly sure it’s not illegal but it…
NVidia also invests in their AI customers.
Earlier quoted context omitted.
A LUT could be a significant performance penalty would it not? Instead of a float8 (potentially multiple in simd case) in a register, you’re now having to head out to at least L1 cache to dereference the value in the LUT. Plain uint8 wouldn’t allow for the same accuracy range as float8 and it’s the accuracy not the precision (which uint would win for the largest values it can represent) that counts most.
Oh oh was just gonna comment as well, but saw this! I think x86 has like pshufb for LUTs (used them like ages ago, but forgot now :() I think also some game (was it Spiderman) used loads of lookup tables. The issue with LUTs is don't you have to update the LUT itself? You can select which memory address to load up, but the LUT itself has to be differentiable maybe? TBH I'm not an expert on LUTs. On fixed point - simi…
There is also VPERMI2B [0] which operates on a 128 byte LUT.
Earlier quoted context omitted.
Oh oh was just gonna comment as well, but saw this! I think x86 has like pshufb for LUTs (used them like ages ago, but forgot now :() I think also some game (was it Spiderman) used loads of lookup tables. The issue with LUTs is don't you have to update the LUT itself? You can select which memory address to load up, but the LUT itself has to be differentiable maybe? TBH I'm not an expert on LUTs. On fixed point - simi…
>I think x86 has like pshufb for LUTs There is also VPERMI2B [0] which operates on a 128 byte LUT. [0] https://en.wikichip.org/wiki/x86/avx512_vbmi
Earlier quoted context omitted.
Another way to compete with the big tech incumbents is instead of hardware, try maths and software hacks to level the playing field! Training models is still black magic, so making it faster on the software side can solve the capital cost issue somewhat!
This kind of research is also incredibly capital intensive. You have to pay some of the smartest people around to work in it.