Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an OSS alternative to Azure OpenAI services

github.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#41

Post by luyuanxin1995 »

aaronharnly wrote: post #39

Congratulations on shipping! We are currently evaluating replacing our homegrown version of an LLM proxy with this project: https://github.com/BerriAI/litellm Any comparison or contrast you would point out?

Litellm proxy is a pretty good project on its own. I am obviously biased because we are competitors. Here are my thoughts.

* Litellm is declarative and it let you define everything in yaml * Bricks is not declarative and you control everything via API

* Litellm does not have an UI * Bricks has a non open source UI

* Litellm is written in python * Bricks is written in Golang

* Litellm does not persist rate limits. Therefore can't accurately rate limit across distributed instances * Bricksllm let you create API keys with accurate rate limits and spend limits that work across distributed instances

* Litellm provides high level spend metrics on API keys * Bricks provides granular spend, request and latency metrics breakdown by model and custom id

* Litellm is not compatible with OpenAI SDK. You have to adopt Litellm python client * Bricks is designed to be compatible with OpenAI SDK

* Litellm only supports OpenAI completion and embedding * Bricks supports almost all OpenAI endpoints except image and audio

* Litellm has exact request caching * Bricks does not have caching as for now

* Litellm has OpenTelemetry integration * Bricks has statsd integration

* Litellm supports orchestration of API calls. When this API call fails, use this model or call this API endpoint instead * Bricks does not support orchestration of API calls since I believe that it is something that the client should handle

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#42

Post by [deleted] »

kristjansson wrote: post #13
happytiger wrote: post #11

Earlier quoted context omitted.

Whoa there. That’s only half the story. https://techcrunch.com/2023/03/01/addressing-criticism-opena... They said only if they opt in not that they wouldn’t do it. And they used to and could change those terms again after the market matures and developers have deeply integrated application investments. Plenty of companies have followed this strategy in the past. Let’s not oversimplify the situation as it was a recent…

That's only for ChatGPT. The APIs are governed by [0] > We will only use Customer Content as necessary to provide you with the Services, comply with applicable law, and enforce OpenAI Policies. We will not use Customer Content to develop or improve the Services. [0]: https://openai.com/policies/business-terms

[deleted]

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#43

Post by voiceblue »

kristjansson wrote: post #13
happytiger wrote: post #11

Earlier quoted context omitted.

Whoa there. That’s only half the story. https://techcrunch.com/2023/03/01/addressing-criticism-opena... They said only if they opt in not that they wouldn’t do it. And they used to and could change those terms again after the market matures and developers have deeply integrated application investments. Plenty of companies have followed this strategy in the past. Let’s not oversimplify the situation as it was a recent…

That's only for ChatGPT. The APIs are governed by [0] > We will only use Customer Content as necessary to provide you with the Services, comply with applicable law, and enforce OpenAI Policies. We will not use Customer Content to develop or improve the Services. [0]: https://openai.com/policies/business-terms

That language seems to be doing a lot of heavy lifting. A more straightforward phrasing would've been "we will never process or otherwise consume customer content once it has been delivered to you". As written, they could use it to train GPT-5 or what-have-you and refuse to share it with you, making that exempt as it is apart from "services". Or all manner of other shenanigans, if they have competent lawyers.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#44

Post by aaronharnly »

luyuanxin1995 wrote: post #41
aaronharnly wrote: post #39

Congratulations on shipping! We are currently evaluating replacing our homegrown version of an LLM proxy with this project: https://github.com/BerriAI/litellm Any comparison or contrast you would point out?

Litellm proxy is a pretty good project on its own. I am obviously biased because we are competitors. Here are my thoughts. * Litellm is declarative and it let you define everything in yaml * Bricks is not declarative and you control everything via API * Litellm does not have an UI * Bricks has a non open source UI * Litellm is written in python * Bricks is written in Golang * Litellm does not persist rate limits. The…

Thank you, I appreciate the earnest and thoughtful reply!

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#45

Post by toomuchtodo »

cebert wrote: post #40
toomuchtodo wrote: post #23

Earlier quoted context omitted.

It potentially plugs the contractual and liability risks, which might be more important (talk to your legal and compliance folks). None of your data is going to launch nuclear missiles, if it leaks it would be unfortunate, but not as much as the litigation and regulatory costs you could potentially incur. Everyone gets popped eventually. It's your job to show you operated from a commercially reasonable security postu…

> (i report to a CISO, and we report to a board, thoughts and opinions are my own) That sounds like an interesting role. How did you get there? Did you start as a security analyst and work your way up?

Word of mouth referral into the org, last ~5 years as a security architect/cybersecurity subject matter expert, before that DevOps/infra engineer. 20+ years in tech. I rely solely on network and reputation.

Be interesting to people who can provide you opportunity, and ask whenever an opportunity presents itself. If you don’t ask, the answer is default no. Being genuinely curious and desiring to help doesn't hurt either.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#47

Post by kristjansson »

voiceblue wrote: post #43
kristjansson wrote: post #13

Earlier quoted context omitted.

That's only for ChatGPT. The APIs are governed by [0] > We will only use Customer Content as necessary to provide you with the Services, comply with applicable law, and enforce OpenAI Policies. We will not use Customer Content to develop or improve the Services. [0]: https://openai.com/policies/business-terms

That language seems to be doing a lot of heavy lifting. A more straightforward phrasing would've been "we will never process or otherwise consume customer content once it has been delivered to you". As written, they could use it to train GPT-5 or what-have-you and refuse to share it with you, making that exempt as it is apart from "services". Or all manner of other shenanigans, if they have competent lawyers.

You can't really read the language at face value. 'Services' is defined term in the contract:

> “Services” means any services for businesses and developers we make available for purchase or use, along with any of our associated software, tools, developer services, documentation, and websites, but excluding any Third Party Offering.

Of course it's still language, and one can quibble with any language, but it's reasonably restrictive.

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#48

Post by donovan-so »

weekay wrote: post #46

If I were to self host an open source model like mistral or llama , are there options similar to this as an api gateway to proxy and authenticate , create api keys , monitor spends by api etc .,? How are people running open source LLM”s in production ? Thanks

We do support self-hosted models, as long as they're exposed as an API. Would this endpoint work for you? https://github.com/bricks-cloud/BricksLLM?tab=readme-ov-file...

Re: Show HN: I built an OSS alternative to Azure OpenAI services

#50

Post by sharathr »

aaronharnly wrote: post #39

Congratulations on shipping! We are currently evaluating replacing our homegrown version of an LLM proxy with this project: https://github.com/BerriAI/litellm Any comparison or contrast you would point out?

Have you taken a look at: https://www.getjavelin.io
Post reply on HN