It's great to see more such platform popping up. It's good for the ecosystem. We need more hosting options that are clear, secure and have the ability to help people run as many models as possible.
Cloudflare's AI Platform: an inference layer designed for agents
101–106 of 106 posts
Re: Cloudflare's AI Platform: an inference layer designed for agents
#102So it's basically just openrouter with cloudflare argo networking? I feel like they could do some much more interesting stuff with their replicate acquisition. Application specific RL is getting so good but there's no good way to deploy these models in a scalable way. Even the providers like fireworks which claim to let you deploy LORAs in a scalable way can't do it. For now I literally have to host base load on my a…
The interesting part is that you can use the same API with Workers AI models (hosted at the edge) and proxied models (OpenRouter-style). Disclaimer: I work at Cloudflare, but not on this.
It's the same problem as fireworks, the only models supporting LORA are like year old dense models that perform horribly on most tasks. If you want to do anything close to relevant you still need to rent/own dedicated GPUs, which seems insane to me when vLLM fully support dynamic LORA loading.
Re: Cloudflare's AI Platform: an inference layer designed for agents
#103Earlier quoted context omitted.
There is always one thing that bites you because Cloudflare is different. I just built an AI game (sleuththetruth.com) and the primary reason it's so slow to prompt a new board is actually not because of AI latency. It's because CF workers have a limit of 6 connections (including spawned workers). There is no way to gulp down all the wiki images I want all at once. If I had put the backend on Railway I don't think I'…
You can farm out the requests to a bunch of Durable Objects. Each DO will have a separate six-concurrent limit. And you can send unlimited concurrent requests to Durable Objects. (This is not an exploit, this is working as intended. The concurrency limit exists to prevent creating excessive connections from a single machine; farming to DOs means the requests are spread out.) Also note that as of recently, the concurr…
Great tip. I knew about #2 which still doesn't help me but #1 is nowhere in their docs!
Re: Cloudflare's AI Platform: an inference layer designed for agents
#104Sexy, but I wouldn't trust it. Why ? because Cloudflare AI Gateway is reporting inaccurate/wrong price for flagship models such as Nano Banana 2 and Nano Banana pro (I run production app using those). Been reporting it on discord and twitter, and they don't care. Entreprise client here :)
hi, I am the PM for AI Gateway. We want to make sure our pricing is correct. I found your tweets about this and will dig in!
thank you
Re: Cloudflare's AI Platform: an inference layer designed for agents
#105Re: Cloudflare's AI Platform: an inference layer designed for agents
#106Earlier quoted context omitted.
* D1, but agreed. I wish Cloudflare would offer a built-in D1-R2 backups system though! (Can be done with custom code in a worker, but wish it was first-party)
yeah this really sucks. No downtime snapshots would be the best but I'd be quite happy with a blocking backup on a set schedule that can be set from the GUI / from the cli / from a config file. Its a huge PITA having to play 'trust me bro' to clients and their admins with custom workers and backups. I currently stream it D1 dump -> worker(encrypt w/ key wrapping) -> R2 on a schedule, then have a container spin up onc…
would https://litestream.io/ be a good solution here?