41–50 of 52 posts
It's good to see more effort for making things not device specific but I only see benchmarks for NVIDIA B200 and AMD MI350X. Also what's the experience of using one of these Python DSLs like? Are the tools good enough to make code completion, jump to definition, setting breakpoints, watching variables, copying as expression etc. nice?
Generally you are unlikely to get Python-level debugging for code that is going to run on GPUs.
I switched from pytorch to jax just before triton appeared. Does anyone know how jax compares to this autotuning machinery in pytorch ? I know jax does jit, but i don't have a good intuition if jit is better than this type of autotuning.
Earlier quoted context omitted.
> how much of ML development these days touches these “lower level” parts of the stack? I’d expect that by now most of the work would be high level Every time the high level architectures of models change, there are new lower level optimizations to be done. Even recent releases like GPT-OSS adds new areas for improvements, like MXFP4, that requires the lower level parts to created and optimized.
How often do hardware optimizations get created for lower level optimization of LLMs and Tensor physics? How reconfigurable are TPUs? Are there any standardized feature flags for TPUs yet? Is TOPS/Whr a good efficiency metric for TPUs and for LLM model hosting operations? From https://news.ycombinator.com/item?id=45775181 re: current TPUs in 2025; "AI accelerators" : > How does Cerebras WSE-3 with 44GB of 'L2' on-chi…
> How often do hardware optimizations get created for lower level optimization of LLMs and Tensor physics?
LLMs? all the time? "tensor physics" (whatever that is) never
> How reconfigurable are TPUs?
very? as reconfigurable as any other programmable device?
> Are there any standardized feature flags for TPUs yet?
have no idea what a feature flag is in this context nor why they would be standardized (there's only one manufacturer/vendor/supplier of TPUs).
> Is TOPS/Whr a good efficiency metric for TPUs and for LLM model hosting operations?
i don't see why it wouldn't be? you're just asking is (stuff done)/(energy consumed) a good measure of efficiency to which the answer is yes?
How does this compare against other DSLs?
I dont get the point of helion as compared to its alternatives like gluon. For best performance I would presume one needs low-level access to hardware knobs. And, these kernel primitives are written one-time and reused. So, what is the point of a DSL that dumbs things down as a wrapper around triton.
One of the main values of Triton is that it significantly expanded the scope of folks who can write kernels - I think Helion could expand the scope even more.
Interesting choice from PyTorch to release yet another DSL, on positive side it's one more point in the design space on the other hand it's even more difficult to choose the right technology among Triton, Gluon, CuTe, ThunderKittens and a few others.
IMO, this is something that makes sense for PyTorch to release, as "neutral ground" in the industry.
Earlier quoted context omitted.
How often do hardware optimizations get created for lower level optimization of LLMs and Tensor physics? How reconfigurable are TPUs? Are there any standardized feature flags for TPUs yet? Is TOPS/Whr a good efficiency metric for TPUs and for LLM model hosting operations? From https://news.ycombinator.com/item?id=45775181 re: current TPUs in 2025; "AI accelerators" : > How does Cerebras WSE-3 with 44GB of 'L2' on-chi…
this is like 5 different questions all across the landscape - what exactly do you think answers will do for you? > How often do hardware optimizations get created for lower level optimization of LLMs and Tensor physics? LLMs? all the time? "tensor physics" (whatever that is) never > How reconfigurable are TPUs? very? as reconfigurable as any other programmable device? > Are there any standardized feature flags for TP…
X86, ARM, and RISC have all standardized on feature flags which can be reviewed on Linux with /proc/cpuinfo or with dmidecode.
cat /proc/cpuinfo | grep -E '^processor|Features|^BogoMIPS|^CPU'
> How reconfigurable are TPUs?
TIL Google's TPUs are reconfigurable with OCS Optical Circuit Switches that can be switched between for example 3D torus or twisted torus configurations.
(FWIW also, quantum libraries mostly have Line qubits and Lattice qubits. There is a recent "Layer Coding" paper; to surpass Surface Coding.)
But classical TPUs;
I had already started preparing a response to myself to improve that criteria; And then paraphrasing from 2.5pro:
> Don't rank by TOPS/wHr alone; rank by TOPS/wHr @ [Specific Precision]. Don't rank by Memory Bandwidth alone; rank by Effective Bandwidth @ [Specific Precision].
Hardware Rank criteria for LLM hosting costs:
Criterion 1: EGB (Effective Generative Bandwidth) Memory Bandwidth (GB/s) / Precision (Bytes)
Criterion 2: GE (Generative Efficiency) EGB / Total Board Power (Watts)
Criterion 3: TTFT Potential Raw TOPS @ Prompt Precision
LLM hosting metrics: Tokens Per Second (TPS) for throughput, Time to First Token (TTFT) for latency, and Tokens Per Joule for efficiency.
Earlier quoted context omitted.
this is like 5 different questions all across the landscape - what exactly do you think answers will do for you? > How often do hardware optimizations get created for lower level optimization of LLMs and Tensor physics? LLMs? all the time? "tensor physics" (whatever that is) never > How reconfigurable are TPUs? very? as reconfigurable as any other programmable device? > Are there any standardized feature flags for TP…
> have no idea what a feature flag is in this context nor why they would be standardized (there's only one manufacturer/vendor/supplier of TPUs). X86, ARM, and RISC have all standardized on feature flags which can be reviewed on Linux with /proc/cpuinfo or with dmidecode. cat /proc/cpuinfo | grep -E '^processor|Features|^BogoMIPS|^CPU' There are multiple TPU vendors. I listed multiple AI accelerator TPU products in t…
There are not - TPU is literally a Google trademark:
> Tensor Processing Unit (TPU) is an AI accelerator application-specific integrated circuit (ASIC) developed by Google.
https://en.wikipedia.org/wiki/Tensor_Processing_Unit
The rest of what you're talking about is irrelevant