It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."
> the law prohibits trading on material nonpublic information. Isn’t it public information the moment it’s said audibly in a public space?
No. It's not. However, as pointed out elsewhere, you can trade on many types of non-public information. Indeed, hedge funds engage in all sorts of surveillance in order to get non-public material information to trade on which gives them a proprietary edge.
This was sort of the dream of KNL but today I noticed Xeon Phi CPUs support (a.k.a. Knight Landing and Knight Mill) are marked as deprecated. GCC will emit a warning when using the -mavx5124fmaps, -mavx5124vnniw, -mavx512er, -mavx512pf, -mprefetchwt1, -march=knl, -march=knm, -mtune=knl or -mtune=knm compiler switches. Support will be removed in GCC 15. the issue was that coordinating across this kind of hierarchy was…
I actually wasn't so much talking from that perspective, as simply from the perspective of the design of individual pieces. There were rather clever things done in e.g. older multipliers or adders or similar which, I think , could apply to most modern parallel architectures, be that GPGPU, SP, MPE, FPGA, or whatever, in order to significantly increase density at a cost of slightly reduced serial performance. For mach…
Truth is that there is another startup called graph core that is doing exactly that, and also a really big chip
Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…
Scaling up compute can improve throughput, but can't easily improve latency between tokens. Generation is usually bottlenecked by the time it takes to go through the network for each token. To speed that up, you need to perform these computations faster, which is a hard problem after you've exhausted all the obvious options (use the fastest accelerator you can find, cache what you can etc).
At batch size 1 LLMs are memory bandwidth bound, not compute bound…as in you spend most time waiting for model weights to load from vram. At higher batch sizes this flips.
But this is why Groq is built around large numbers of chips with small amount of very fast sram.
The Groq demo was indeed impressive. I work with LLM alot in work, and a generation speed of 500+ tokens/s would definitely change how we use these products. (Especially considering it's an early-stage product) But the "completely novel silicon architecture" and the "self-developed LPU" (claiming not to use GPUs)... makes me bit skeptical. After all, pure speed might be achievable through stacking computational power…
I work at Groq. We arent using GPUs at all. This is a novel hardware architecture of ours that allows this high throughput and latency. Nothing sketchy about it.
I don't think it should be quadratic in input length. Why do you think it is?
You can ask your website: "What is the computational complexity of self-attention with respect to input sequence length?" It'll answer something along the lines of self-attention being O(n^2) (where n is the sequence length) because you have to compute an attention matrix of size n^2. There are other attention mechanisms with better computational complexity, but they usually result in worse large language models. To…
Changing the way transformer models works is orthogonal to gaining good performance on Mistral. Groq did great work reducing the latency considerably of generating tokens during inference. But I wouldn't be surprised if they etched the A matrix weights in some kind of fast ROM, used expensive SRAM for the the skinny B matrix, and sent everything else that didn't fit to good old fashioned hardware. That's great for generating text, but prompt processing is where the power is in AI. In order to process prompts fast, you need to multiply weights against 2-dimensional matrices. There is significant inequality in software implementations alone in terms of how quickly they're able to do this, irrespective of hardware. That's why things like BLAS libraries exist. So it'd be super interesting to hear about how a company like Groq that leverages both software and hardware specifically for inference is focusing on tackling its most important aspect.
(Groq Employee) You're right - we are comparing to independently-clocked logic. I wonder whether async logic would be feasible for reconfigurable "Spatial Processor" type architectures [1]. As far as LPU architectures go, they fall in the "Matrix of Processing Engines"[1] family of architectures, which I would naively guess is not the best suited to leverage async logic. 1: I'm using the "Spatial Processor" (7:14) an…
Curiously, almost all of this video is mostly covered by computer architectures lit in the late 90's early 00's. At the time, I recall Tom Knight had done most of the analysis in this video, but I don't know if he ever published it. It was extrapolating into the distant future. To answer your questions: - Spatial processors are an insanely good fit for async logic - Matrix of processing engines are a moderately good…
Sweet, thanks! It seems like this research ecosystem was incredibly rich, but Moore's law was in full swing, and statically known workloads weren't useful at the compute scale of back then.
So these specialized approach never stood a chance next to CPUS. Nowadays the ground is.. more fertile.