AI Proxy to swap in any LLM while using OpenAI's SDK

#1

Post by davidatbt » Mon, Nov 20, 2023, 9:41 PM UTC

AI Proxy to swap in any LLM while using OpenAI's SDK
twitter.com

Re: AI Proxy to swap in any LLM while using OpenAI's SDK

#2

Post by westurner » Mon, Nov 20, 2023, 9:49 PM UTC

How does this compare to LocalAI? https://github.com/mudler/LocalAI

Re: AI Proxy to swap in any LLM while using OpenAI's SDK

#3

Post by Tiberium » Mon, Nov 20, 2023, 10:10 PM UTC

westurner wrote: post #2

How does this compare to LocalAI? https://github.com/mudler/LocalAI

The AI Proxy from the post is for using multiple local and remote LLMs over the OpenAI API (along with API key management apparently), while LocalAI is only for using local LLMs.

Re: AI Proxy to swap in any LLM while using OpenAI's SDK

#4

Post by Havoc » Tue, Nov 21, 2023, 12:36 AM UTC

Been meaning to build similar. Specifically with a view towards mixing local LLMs and openAI. Locals are cheaper and more interesting but less powerful. eg some tasks will need gbnf grammar some won’t. Some can get away with 3.5 others need 4

Suspect I’ll end up duplicating OPs efforts though because hosted doesn’t really work for that.

Re: AI Proxy to swap in any LLM while using OpenAI's SDK

#5

Post by davidatbt » Tue, Nov 21, 2023, 2:32 AM UTC

Havoc wrote: post #4

Been meaning to build similar. Specifically with a view towards mixing local LLMs and openAI. Locals are cheaper and more interesting but less powerful. eg some tasks will need gbnf grammar some won’t. Some can get away with 3.5 others need 4 Suspect I’ll end up duplicating OPs efforts though because hosted doesn’t really work for that.

Sounds interesting - feel free to reach out or discuss on the Braintrust discord :)!

Re: AI Proxy to swap in any LLM while using OpenAI's SDK

#6

Post by davidatbt » Tue, Nov 21, 2023, 2:33 AM UTC

westurner wrote: post #2

How does this compare to LocalAI? https://github.com/mudler/LocalAI

This looks awesome - thanks for sharing! Yeah I think it's just local vs. hosted models.

Re: AI Proxy to swap in any LLM while using OpenAI's SDK

#7

Post by westurner » Tue, Nov 21, 2023, 4:01 AM UTC

Tiberium wrote: post #3

westurner wrote: post #2

How does this compare to LocalAI? https://github.com/mudler/LocalAI

The AI Proxy from the post is for using multiple local and remote LLMs over the OpenAI API (along with API key management apparently), while LocalAI is only for using local LLMs.

promptfoo and ChainForge do multi-LLM comparisons and benchmarking: https://news.ycombinator.com/item?id=37447885