11–20 of 21 posts
• Devs forever want choice.
• Open-source LLMs are getting better
• Anthropic ships fantastic models
• Doesn't expose your app’s data to multiple companies
• Consolidated security, billing, config in AWS
• Power of AWS ecosystem
Increasingly bullish on AWS Bedrock. • Devs forever want choice. • Open-source LLMs are getting better • Anthropic ships fantastic models • Doesn't expose your app’s data to multiple companies • Consolidated security, billing, config in AWS • Power of AWS ecosystem
I have not read too deeply into this but, do any of these serverless environments offer GPUs? I'm sure there are ... reasons but the lack of GPU support in Lambda and Fargate remains a major paint point for AWS users. It's been keeping me wrangling EC2 instances for ML teams but I do wonder how much longer that will last.
A more general issue is that the workloads that tend to run on GPU are much bigger than a standard Lambda-sized workload (think a 20Gi image with a smorgasbord of ML libraries). I've spent time working around this problem and wrote a bit about it here: https://www.beam.cloud/blog/serverless-platform-guide
I have not read too deeply into this but, do any of these serverless environments offer GPUs? I'm sure there are ... reasons but the lack of GPU support in Lambda and Fargate remains a major paint point for AWS users. It's been keeping me wrangling EC2 instances for ML teams but I do wonder how much longer that will last.
Increasingly bullish on AWS Bedrock. • Devs forever want choice. • Open-source LLMs are getting better • Anthropic ships fantastic models • Doesn't expose your app’s data to multiple companies • Consolidated security, billing, config in AWS • Power of AWS ecosystem
I am worried about AWS imposing their own political rules on the models. For example, they may impose censorship, or safety requirements. It is hard for me to trust them as a central platform in this ecosystem
Last time I checked Bedrock was quite expensive to operate in a small scale.
Do you mean specifically the Bedrock Knowledgebase/RAG -- that uses serverless OpenSearch which costs at minimum $200ish/month bc it doesn't scale to zero?
I have not read too deeply into this but, do any of these serverless environments offer GPUs? I'm sure there are ... reasons but the lack of GPU support in Lambda and Fargate remains a major paint point for AWS users. It's been keeping me wrangling EC2 instances for ML teams but I do wonder how much longer that will last.
The major clouds don't support serverless GPU because the architecture is fundamentally different from running CPU workloads. For Lambda specifically, there's no way of running multiple customer workloads on a single GPU with Firecracker. A more general issue is that the workloads that tend to run on GPU are much bigger than a standard Lambda-sized workload (think a 20Gi image with a smorgasbord of ML libraries). I'v…
You can do this with SR-IOV enabled hardware.
https://docs.nvidia.com/networking/display/mlnxofedv581011/s...
P.S: I am from Inferless