Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

github.com

Post reply on HN
Print view View original

161–170 of 349 posts

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#161

Post by qwertox »

extr wrote: post #100
MaxLeiter wrote: post #83

Earlier quoted context omitted.

It uses a sliding context windows. Older tokens are dropped as new ones stream in

I don't believe that's the whole story. Other conversational implementations use sliding context windows and it's very noticable as context drops off. Whereas ChatGPT seems to retain the "gist" of the conversation much longer.

At least in 3.5 it's very noticeable when the context drops. They could use summarization, akin to what they are doing when detecting the topic of the chat, but applied to question-answer-pairs in order to "compress" the information. But that would require additional calls into a summarization LLM so I'm really not sure if it is worth it. Maybe they dump some tokens they have on a blacklist or text snippets like "I want to" or replace "could it be that" with "chance of".

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#162

Post by ttul »

jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for economic reasons.

Over the long run, open source will eventually overtake. Chances are this will happen once the researchers who are making magic happen get their liquidity and can start working for free again out in the open.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#163

Post by robertnishihara »

ajhai wrote: post #77

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at le…

> we believe most companies prefer locally installed solutions to cloud based ones

We've also seen a strong desire from businesses to manage models and compute on their own machines or in their own cloud accounts. This is often part of a hybrid strategy of using API products like OpenAI for rapid prototyping.

The majority of (though not all) businesses we've seen tend to be quite comfortable using hosted API products for rapid prototyping and for proving out an initial version of their AI functionality. But in many cases, they want to complement that with the ability to manage models and compute themselves. The motivation here is often to reduce costs by using smaller / faster / cheaper fine-tuned open models.

When we started Anyscale, customer demand led us to run training & inference workloads in our customers' cloud accounts. That way your data and code stays inside of your own cloud account.

Now with all the progress in open models and the desire to rapidly prototype, we're complementing that with a fully-managed inference API where you can do inference with the Llama-2 models [1] (like the OpenAI API but for open models).

[1] https://app.endpoints.anyscale.com/

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#164

Post by gerdesj »

Well, your AChatGPT connection might be private but your Windows machine will leak like a sieve. It is embarrassing how needy the blasted things are about signing in via Azure/Microsoft, instead of a local or AD account. Even worse is the naff "choose how insidious you would like us to be" series of questions. How would you like your ads? Targeted or non targeted? How about not at all? Nope.

In this day and age, exactly how private does anyone expect their comms/thoughts/files/data to be? I recall reading a recent MS EULA and it seems I have to say three Hail Marys every third Tuesday for using Arch Linux on my PCs. I could install Edge, and did but I don't like the nasty homepage - a bit right wing ... - why on earth is a browser pushing "news"? Its a browser. To be fair I had to dump all the homepage crap that Firefox pushed when I finally dumped anything to do with Chrome.

Please don't use the words private and secure when you have your fingers crossed behind your back.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#165

Post by robertnishihara »

stavros wrote: post #136
mark_l_watson wrote: post #90

This seems like such an obvious thing to do. I see the use of general purpose LLMs like ChatGPT, but smaller fine tuned models will probably end up being more useful for deployed applications in most companies. Off topic, but I was experimenting with LLongMA-2-7b-16K today, running it very inexpensively in the cloud, and given about 12K of context text it really performed well. This is an easy model to deploy. 7B par…

Is there an easy way to play with these models, as someone who hasn't deployed them? I can download/compile llama.cpp, but I don't know which models to get/where to put them/how to run them, so if someone knows about some automated downloader along with some list of "best models", that would be very helpful.

If you want to try out the Llama-2 models (7B, 13B, 70B), you can get started very easily with Anyscale Endpoints (~2 min). https://app.endpoints.anyscale.com/

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#166

Post by xist »

gerdesj wrote: post #164

Well, your AChatGPT connection might be private but your Windows machine will leak like a sieve. It is embarrassing how needy the blasted things are about signing in via Azure/Microsoft, instead of a local or AD account. Even worse is the naff "choose how insidious you would like us to be" series of questions. How would you like your ads? Targeted or non targeted? How about not at all? Nope. In this day and age, exac…

The article is referring to enterprise usage - and you're quoting all the consumer level attributes (aka cheap/sometimes subsidized version).

At the enterprise level where this is intended to be ran, things are much diffrerent.

If you're not aware of the differences or use cases, perhaps you're not the target audience who should be using or configuring it.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#167

Post by robertnishihara »

ttul wrote: post #162
jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for e…

> Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4

I think you're right about this, and benchmarks we've run at Anyscale support this conclusion [1].

The caveat there (which I think will be a big boon for open models) is that techniques like fine-tuning makes a HUGE difference and can bridge the quality gap between Llama-2 and GPT-4 for many (but not all) problems.

[1] https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#169

Post by [deleted] »

dbish wrote: post #129
padolsey wrote: post #127

I'm confused. If this is just a front-end for the OpenAI API then how does it remove the data privacy concern? Your data still ends up with Azure/OpenAI, right? It doesn't stay localized to your instance; it's not your GPU running the transformations. You have no way of knowing whether your data is being used to train models. If customer data is sensitive, I'm pretty sure running a 70B llama (or similar) on a bunch o…

Azure is hosting and operating the service themselves rather then for OpenAI, with all the security requirements that come with that. I assume this comes with different data and access restrictions as well and ability to run in secured instances (and nothing sent to OpenAI the company). Most companies use cloud already for their data, processing, etc. and aren’t running anything major locally, let alone ML models, th…

[deleted]

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#170

Post by FrenchDevRemote »

baobabKoodaa wrote: post #102
FrenchDevRemote wrote: post #51

Earlier quoted context omitted.

it's the SOTA for chat(prove me wrong), and you can always use the API directly even for simple tasks they're less reliable and needs more prompt engineering

> it's the SOTA for chat(prove me wrong) GPT-4 beats ChatGPT on all benchmarks. You can easily google these.

that is why i said FOR CHAT.

even through the API you can't easily use the regular models for chat, the parsing would be atrocious and there are hundreds of edge cases to handle.

ChatGPT4 through the API is the SOTA

Post reply on HN