Show HN: I built an OSS alternative to Azure OpenAI services • Hacker News Board

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#21

Post by HellsMaddy » Mon, Dec 11, 2023, 9:27 PM UTC

This reminds me of an idea I had for an OpenAI proxy that transparently handles batching of requests. The use case is that OpenAI has rate limits not only on tokens but also requests per minute. By batching multiple requests together you can avoid hitting the requests limit.

This isn’t really feasible to implement if your app runs on lambda or edge functions, you’d need a persistent server.

Here’s a diagram I drew of a simple approach that came to mind: https://gist.github.com/b0o/a73af0c1b63fccf3669fa4b00ac4be52

It would be awesome to see this functionality built into BricksLLM.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#22

Post by ajcp » Mon, Dec 11, 2023, 9:30 PM UTC

gigel82 wrote: post #15

ajcp wrote: post #12

Earlier quoted context omitted.

Many here think that I'm afraid of my companies data being used to train models, but that is only half-correct. The value proposition of Azure OpenAI to the enterprise is that it's bound by the Enterprise Agreement for not just data-use, but data access writ-large. I don't want my data to be accessible for *any reason* by a vendor that isn't covered explicitly in the EA. While OpenAI says they delete your data after…

I suspect a lot of folks never worked with Enterprise / Gov customers and don't understand the restrictions and compliance requirements (like data residence, access control, FedRAMP, TISAX, reliability SLAs, etc. which you get with Azure but not with some "move fast and break things" startup like OpenAI). My comment is not dissing on the author, I'm just pointing out what most folks get from Azure is compliance (and…

Yup, and I agree with you.

Even beyond enterprise thought people seem to think that the only thing their data is good for as far as OpenAI is concerned is for training-datasets, but that's just not the case.

The authors of this have built something great, but it doesn't protect you from any of those non-training use-cases.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#23

Post by toomuchtodo » Mon, Dec 11, 2023, 9:30 PM UTC

robertlagrant wrote: post #20

gigel82 wrote: post #15

Earlier quoted context omitted.

I suspect a lot of folks never worked with Enterprise / Gov customers and don't understand the restrictions and compliance requirements (like data residence, access control, FedRAMP, TISAX, reliability SLAs, etc. which you get with Azure but not with some "move fast and break things" startup like OpenAI). My comment is not dissing on the author, I'm just pointing out what most folks get from Azure is compliance (and…

> most folks get from Azure is compliance To be fair, this doesn't prevent problems. Paperwork doesn't plug any security gaps in any cloud provider.

It potentially plugs the contractual and liability risks, which might be more important (talk to your legal and compliance folks). None of your data is going to launch nuclear missiles, if it leaks it would be unfortunate, but not as much as the litigation and regulatory costs you could potentially incur.

Everyone gets popped eventually. It's your job to show you operated from a commercially reasonable security posture (and potentially your third party dependency graph, depending on regulatory and cyber insurance requirements).

(i report to a CISO, and we report to a board, thoughts and opinions are my own)

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#24

Post by ralph84 » Mon, Dec 11, 2023, 9:39 PM UTC

robertlagrant wrote: post #20

gigel82 wrote: post #15

Earlier quoted context omitted.

I suspect a lot of folks never worked with Enterprise / Gov customers and don't understand the restrictions and compliance requirements (like data residence, access control, FedRAMP, TISAX, reliability SLAs, etc. which you get with Azure but not with some "move fast and break things" startup like OpenAI). My comment is not dissing on the author, I'm just pointing out what most folks get from Azure is compliance (and…

> most folks get from Azure is compliance To be fair, this doesn't prevent problems. Paperwork doesn't plug any security gaps in any cloud provider.

Compliance isn’t about preventing problems. It’s about identifying risks and determining who is responsible for mitigating those risks and who is on the hook for damages if the risks aren’t sufficiently mitigated.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#25

Post by computerex » Mon, Dec 11, 2023, 9:43 PM UTC

HellsMaddy wrote: post #21

This reminds me of an idea I had for an OpenAI proxy that transparently handles batching of requests. The use case is that OpenAI has rate limits not only on tokens but also requests per minute. By batching multiple requests together you can avoid hitting the requests limit. This isn’t really feasible to implement if your app runs on lambda or edge functions, you’d need a persistent server. Here’s a diagram I drew of…

OpenAI API doesn't support batching afaik.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#26

Post by swyx » Mon, Dec 11, 2023, 9:48 PM UTC

HellsMaddy wrote: post #21

This reminds me of an idea I had for an OpenAI proxy that transparently handles batching of requests. The use case is that OpenAI has rate limits not only on tokens but also requests per minute. By batching multiple requests together you can avoid hitting the requests limit. This isn’t really feasible to implement if your app runs on lambda or edge functions, you’d need a persistent server. Here’s a diagram I drew of…

how exactly are you intending to batch different prompts together in the openai api? its not like they accept an array of parallel inputs

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#27

Post by heyn05tradamu5 » Mon, Dec 11, 2023, 9:52 PM UTC

HellsMaddy wrote: post #21

This reminds me of an idea I had for an OpenAI proxy that transparently handles batching of requests. The use case is that OpenAI has rate limits not only on tokens but also requests per minute. By batching multiple requests together you can avoid hitting the requests limit. This isn’t really feasible to implement if your app runs on lambda or edge functions, you’d need a persistent server. Here’s a diagram I drew of…

They’ve recently added this functionality to AWS Bedrock thankfully. Doesn’t support OpenAI models, but does support Anthropic.

https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-be...

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#28

Post by HellsMaddy » Mon, Dec 11, 2023, 9:53 PM UTC

computerex wrote: post #25

HellsMaddy wrote: post #21

This reminds me of an idea I had for an OpenAI proxy that transparently handles batching of requests. The use case is that OpenAI has rate limits not only on tokens but also requests per minute. By batching multiple requests together you can avoid hitting the requests limit. This isn’t really feasible to implement if your app runs on lambda or edge functions, you’d need a persistent server. Here’s a diagram I drew of…

OpenAI API doesn't support batching afaik.

They do: https://platform.openai.com/docs/guides/rate-limits/batching...

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#29

Post by treprinum » Mon, Dec 11, 2023, 9:55 PM UTC

computerex wrote: post #25

HellsMaddy wrote: post #21

This reminds me of an idea I had for an OpenAI proxy that transparently handles batching of requests. The use case is that OpenAI has rate limits not only on tokens but also requests per minute. By batching multiple requests together you can avoid hitting the requests limit. This isn’t really feasible to implement if your app runs on lambda or edge functions, you’d need a persistent server. Here’s a diagram I drew of…

OpenAI API doesn't support batching afaik.

Embeddings can be batched.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#30

Post by luyuanxin1995 » Mon, Dec 11, 2023, 9:56 PM UTC

mdeeks wrote: post #19

Looks interesting! We're in the middle of building something similar right now for ourselves. We may look at this as an alternative for ourselves. By the way, I saw this after a quick glance poking through the code. This isn't encryption, it is hashing. Not sure where or how it is used but it is worth a rename at least: https://github.com/bricks-cloud/BricksLLM/blob/main/internal...

you are right. will update the language