Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's AI Platform: an inference layer designed for agents

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Cloudflare's AI Platform: an inference layer designed for agents

#55

Post by vladgur »

mips_avatar wrote: post #49

So it's basically just openrouter with cloudflare argo networking? I feel like they could do some much more interesting stuff with their replicate acquisition. Application specific RL is getting so good but there's no good way to deploy these models in a scalable way. Even the providers like fireworks which claim to let you deploy LORAs in a scalable way can't do it. For now I literally have to host base load on my a…

Curious which models are you able to run and how many 3090s do they require at scale?

Re: Cloudflare's AI Platform: an inference layer designed for agents

#56

Post by 6thbit »

nhecker wrote: post #16
6thbit wrote: post #10

Earlier quoted context omitted.

i got shivers thinking about a future ai dynamic pricing and automatic gateway choosing the cheapest provider available

Openrouter already does this, unless I've misunderstood the premise.

They can route between models but you pay the standard rate for whichever model is selected (plus 5% fee). Afaik all current model providers have fixed prices per tokens which don't vary depending on, say, demand or hardware availability.

Re: Cloudflare's AI Platform: an inference layer designed for agents

#57

Post by mips_avatar »

vladgur wrote: post #55
mips_avatar wrote: post #49

So it's basically just openrouter with cloudflare argo networking? I feel like they could do some much more interesting stuff with their replicate acquisition. Application specific RL is getting so good but there's no good way to deploy these models in a scalable way. Even the providers like fireworks which claim to let you deploy LORAs in a scalable way can't do it. For now I literally have to host base load on my a…

Curious which models are you able to run and how many 3090s do they require at scale?

4 3090s with nvlinks on each pair. Super fast inference on Moe models around 20-36b

Re: Cloudflare's AI Platform: an inference layer designed for agents

#58

Post by ignoramous »

eis wrote: post #39
whereistejas wrote: post #13

This actually looks very useful. Cloudflare seems to be brining together a great set of tools. Not to mention, D2 is literally the only sqlite-as-a-service solution out there whose reliability is great and free tier limits are generous.

D1 reliability has been bad in our experience. We've had queries hanging on their internal network layer for several seconds, sometimes double digits over extended periods (on the order of weeks). Recently I've seen a few times plain network exceptions - again, these are internal between their worker and the D1 hosts. And many of the hung queries wouldn't even show up under traces in their observability dashboard so…

> And many of the hung queries wouldn't even show up under traces in their observability dashboard

How did you work around this problem? As in, how do you monitor for hung queries and cancel them?

> D1 reliability has been bad in our experience.

What about reads? We use D1 in prod & our traffic pattern may not be similar to yours (our workload is async queue-driven & so retries last in order of weeks), nor have we really observed D1 erroring out for extended periods or frequently.

Post reply on HN