at some point the cost of transferring will dwarf the cost you pay to NVIDA. I bet that is their bet
Amazon launches Trainium3
61–70 of 75 posts
Re: Amazon launches Trainium3
#62I've had to repeatedly tell our AWS account reps that we're not even a little interested in the Trainium or Inferentia instances unless they have a provably reliable track record of working with the standard libraries we have to use like Transformers and PyTorch. I know they claim they work, but that's only on their happy path with their very specific AMI's and the nightmare that is the neuron SDK. You try to do any…
IMO AWS once you get off the core services is full of beta services. S3, Dynamo, Lambda, ECS, etc are all solid. But there are a lot of services they have that have some big rough patches.
Lightsail fortunately behave like core services.
Re: Amazon launches Trainium3
#63Earlier quoted context omitted.
IMO AWS once you get off the core services is full of beta services. S3, Dynamo, Lambda, ECS, etc are all solid. But there are a lot of services they have that have some big rough patches.
RDS, Route53, and Elasticache are decent, too. But yes, I've also been bitten badly in the distant past by attempting to rely on their higher-level services. I guess some things don't change. I wonder if the difference is stuff they dogfood versus stuff they don't?
My understanding is that AWS productizes lots of one-offs for customers (like Snowball), so that makes sense
Re: Amazon launches Trainium3
#64Earlier quoted context omitted.
RDS, Route53, and Elasticache are decent, too. But yes, I've also been bitten badly in the distant past by attempting to rely on their higher-level services. I guess some things don't change. I wonder if the difference is stuff they dogfood versus stuff they don't?
I once used one of their services (I forget which, but I think it was there serverless product) that “supported” Java. … but the official command line tools had show-stopper bugs if you were deploying Java to this service, that’d been known for months, and some features couldn’t be used in Java, and the docs were only like 20% complete. But this work-in-progress alpha (not even beta quality because it couldn’t plausi…
I'm assuming you're talking about Lambda. I don't mess with their default images. Write a Dockerfile and use containerized Lambdas. Saves so many headaches. Still have to deal with RIE though, which is annoying.
Re: Amazon launches Trainium3
#65Earlier quoted context omitted.
IMO AWS once you get off the core services is full of beta services. S3, Dynamo, Lambda, ECS, etc are all solid. But there are a lot of services they have that have some big rough patches.
True with Cloudflare too. Just stick with Workers, R2, Durable Objects, etc...
Not even sure about R2 with it's unpredictable latencies.
Re: Amazon launches Trainium3
#66Earlier quoted context omitted.
True with Cloudflare too. Just stick with Workers, R2, Durable Objects, etc...
Not even sure about R2 with it's unpredictable latencies.
Hmm is it actually that bad? Keep in mind r2 is only stored in one region which is chosen when the bucket is first created so that might be what you're seeing
But I've never really looked too closely because I just use it for non-latency critical blob storage
Re: Amazon launches Trainium3
#67Not a single mention of any benchmarks or performance.
yea, they "officially" dont release benchmarks even if we asked the AWS reps
Re: Amazon launches Trainium3
#68Earlier quoted context omitted.
Yes, but Google benefit from people using their TPUs, while Anthropic gains nothing unless AWS throws money at them for saying it.
This. Anthropic is not going to interrupt their competitors if their competitors don't want to use trainium. Neither would you, I, nor anyone else. The only potential is downside. There's no upside potential for them at all in doing so. From Anthropic's perspective, if the rest of us can't figure out how to make trainium work? Good. Amazon will fix the difficulty problem with time, but that's time Anthropic can use t…
I am not sure, I would imagine enthusiastic quotes will lead to huge discounts and in that scale that matters
Re: Amazon launches Trainium3
#69Re: Amazon launches Trainium3
#70Single-chip specs according to:
https://awsdocs-neuron.readthedocs-hosted.com/en/latest/abou...
https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/...
Eight NeuronCore-v4 cores that collectively deliver:
2,517 MXFP8/MXFP4 TFLOPS
671 BF16/FP16/TF32 TFLOPS
2,517 FP16/BF16/TF32 sparse TFLOPS
183 FP32 TFLOPS
SRAM: 32 MiB * 8 = 256 MiB (ignoring 2 MiB * 8 = 16 MiB of PSUM which is not really general-purpose nor DMA-able)
Interconnect: 2560 GB/s (I think bidirectional, i.e. Jensen Math™)
----
At 3nm process node the FLOP/s is _way_ lower than competition. Compare to B200 which does 2250 BF16, x2 FP8, x4 FP4. TPU7x does 2307 BF16, x2 FP8 (no native FP4). HBM also lags behind (vs ~192 GiB in 6 stacks for both TPU7x and B200).
The main redeeming qualities seem to be: software-managed SRAM size (double of TPU7x; GPUs have L2 so not directly comparable) and on-paper raw interconnect BW (double of TPU7x and more than B200).