Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

github.com

Post reply on HN
Print view View original

111–120 of 349 posts

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#111

Post by Q6T46nT668w6i3m »

tikkun wrote: post #41
TuringNYC wrote: post #16

Curious if anyone has done a side-by-side analysis of this offering vs just running LLaMA? I'm currently running a side-by-side comparison/evaluation of MSFT GPT via Cognitive Services vs LLaMA[7B/13B/70B] and intrigued by the possibility of a truly air-gapped offering not limited by external computer power (nor by metered fees racking up.) Any reads on comparisons would be nice to see. (yes, I realize we'll eventual…

I did one. I took a few dozen prompts from my ChatGPT history and ran them through a few LLMs. GPT-4, Bard and Claude 2 came out on top. Llama 2 70b chat scored similarly to GPT-3.5, though GPT-3.5 still seemed to perform a bit better overall. My personal takeaway is I’m going to continue using GPT-4 for everything where the cost and response time are workable. Related: A belief I have is that LLM benchmarks are all…

I don’t know what you mean by “too research oriented.” A common complaint in LLM research is the poor quality of evaluation metrics. There’s no consensus. Everyone wants new benchmarks but designing useful metrics is very much an open problem.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#112

Post by gdiamos »

albert_e wrote: post #2

is there away to run this on AWS instead. we were looking to explore Llama2 for internal use

We can run llama 2 on an AWS vm if you have enough GPUs: https://lamini.ai/

Install in 10 minutes.

Make sure you have enough GPU memory to fit your llama model if you want good perf

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#113

Post by paxys »

Xenoamorphous wrote: post #75
paxys wrote: post #30

Would it be too much to mention somewhere in the README what this repo actually contains? Just docs? Deployment files? Some application (which does..something)? The model itself?

The repo contains the UI code, not the model or anything else around ChatGPT, it just uses Azure’s ChatGPT API which doesn’t share data with OpenAI.

So basically – what you really need to do to run Azure ChatGPT is go and click some buttons in the Azure portal. This repo is a sample UI that you could possibly use to talk to that instance, but really you will probably always build your own or embed it directly into your products.

So calling the repo "azurechatgpt" is misleading. It should really be "sample-chatgpt-api-frontend" or something of that sort.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#114

Post by register »

tikkun wrote: post #41
TuringNYC wrote: post #16

Curious if anyone has done a side-by-side analysis of this offering vs just running LLaMA? I'm currently running a side-by-side comparison/evaluation of MSFT GPT via Cognitive Services vs LLaMA[7B/13B/70B] and intrigued by the possibility of a truly air-gapped offering not limited by external computer power (nor by metered fees racking up.) Any reads on comparisons would be nice to see. (yes, I realize we'll eventual…

I did one. I took a few dozen prompts from my ChatGPT history and ran them through a few LLMs. GPT-4, Bard and Claude 2 came out on top. Llama 2 70b chat scored similarly to GPT-3.5, though GPT-3.5 still seemed to perform a bit better overall. My personal takeaway is I’m going to continue using GPT-4 for everything where the cost and response time are workable. Related: A belief I have is that LLM benchmarks are all…

How did you measure the performance?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#115

Post by ajhai »

bhanu423 wrote: post #103
ajhai wrote: post #77

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at le…

Interesting project - was trying it out, found an issue in building the image - have opened an issue on github - please take a look. Also do you have plan to support llama over openai models.

Thanks for the issue. Will take a look. In the meantime, you can try the registry image with `cp .env.prod .env && docker compose up`

> Also do you have plan to support llama over openai models.

Yes, we plan to support llama etc. We currently have support for models from OpenAI, Azure, Google's Vertex AI, Stability and a few others.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#117

Post by BoorishBears »

shubb wrote: post #85
extr wrote: post #79

One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they never work quite as well, it's obvious that you run out of context after a few conversation turns. Is ChatGPT doing some embedding lookup inside the conversation thread to make the context feel infinite? I've noticed anecdotally it definitely isn't infinite, but i…

This is one of the things that make me uncomfortable about proprietary llm. They get task performance by doing a lot more than just feeding a prompt straight to an llm, and then we performance compare them to raw local options. The problem is, as this secret sauce changes, your use case performance is also going to vary in ways that are impossible for you to fix. What if it can do math this month and next month the h…

I'm not sure you realize how proprietary LLMs are being built on.

No one is doing secret math in the backend people are building on. The OpenAI API allows you to call functions now, but even that is just a formalized way of passing tokens into the "raw LLM".

All the features in the comment you replied to only apply to the web interface, and here you're being given an open interface you can introspect.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#118

Post by scrum-treats »

gdiamos wrote: post #89
ajhai wrote: post #77

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at le…

I find it interesting to see how competitive this space got so quickly. How do these stacks differentiate?

Quality and depth of particular types of training data is one difference. Another difference is inference tracking mechanisms within and between single-turn interactions (e.g., what does the human user "mean" with their prompt, what is the "correct" response, and how best can I return the "correct" response for this context; how much information do I cache from the previous turns, and how much if any of it is relevant to this current turn interaction).

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#119

Post by BoorishBears »

extr wrote: post #101
SOLAR_FIELDS wrote: post #95

Earlier quoted context omitted.

They definitely do some proprietary running summarization to rebuild the context with each chat. Probably a RAG like approach that has had a lot of attention and work

This is effectively my question. I assume there is some magic going on. But how many engineering hours worth of magic, approximately? There is a lot of speculation around GPT-4 being MoE and whatnot. But very little speculation about the magic of the ChatGPT front end specifically that makes it feel so fluid.

That's mostly because there's very little value in deep speculation there.

It's not particularly more fluid than anything you couldn't whip up yourself (and the repo linked proves that) but there's also not much value in trying to compete with ChatGPT's frontend.

For most products ChatGPT's frontend is the minimal level of acceptable performance that you need to beat, not an maximal one really worth exploring.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#120

Post by kuchenbecker »

lsaferite wrote: post #108
extr wrote: post #100

Earlier quoted context omitted.

I don't believe that's the whole story. Other conversational implementations use sliding context windows and it's very noticable as context drops off. Whereas ChatGPT seems to retain the "gist" of the conversation much longer.

I mean, I explicitly have the LLM summarize content that's about to fall out of the window as a form of pre-emptive token compression. I'd expect maybe they do something similar.

I feel like we're describing short vs long term memory.
Post reply on HN