Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

Post by [deleted] » Sun, Aug 13, 2023, 11:31 PM UTC

dbish wrote: post #129

padolsey wrote: post #127

I'm confused. If this is just a front-end for the OpenAI API then how does it remove the data privacy concern? Your data still ends up with Azure/OpenAI, right? It doesn't stay localized to your instance; it's not your GPU running the transformations. You have no way of knowing whether your data is being used to train models. If customer data is sensitive, I'm pretty sure running a 70B llama (or similar) on a bunch o…

Azure is hosting and operating the service themselves rather then for OpenAI, with all the security requirements that come with that. I assume this comes with different data and access restrictions as well and ability to run in secured instances (and nothing sent to OpenAI the company). Most companies use cloud already for their data, processing, etc. and aren’t running anything major locally, let alone ML models, th…

Post by stavros » Sun, Aug 13, 2023, 11:36 PM UTC

tayo42 wrote: post #173

stavros wrote: post #136

Earlier quoted context omitted.

Is there an easy way to play with these models, as someone who hasn't deployed them? I can download/compile llama.cpp, but I don't know which models to get/where to put them/how to run them, so if someone knows about some automated downloader along with some list of "best models", that would be very helpful.

For llama, the 4bit quantized ones, small models like the 7b one. The ggml format. That will run on your local cpu. Google those terms too. you can look on hugging face for the actual model to download then load it and send prompts to it

Post by jgalt212 » Sun, Aug 13, 2023, 11:40 PM UTC

ttul wrote: post #162

jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for e…

Post by Havoc » Sun, Aug 13, 2023, 11:46 PM UTC

asabla wrote: post #97

Havoc wrote: post #94

How does this work in terms of utilization? The isolation presumably means buying gpu capacity and only using a %?

Basically you get N tokens/second (or if it was minute, can check tomorrow if you're really interested) per deployment. So if you would outgrow on deployment, just deploy another one (with the associated costs of course). One deployment = a deployed model which you can query On top of that, depending on the model you're using, you also see a cost increment for each 1000 request you make.

Post by gerdesj » Mon, Aug 14, 2023, 12:01 AM UTC

xist wrote: post #166

gerdesj wrote: post #164

Well, your AChatGPT connection might be private but your Windows machine will leak like a sieve. It is embarrassing how needy the blasted things are about signing in via Azure/Microsoft, instead of a local or AD account. Even worse is the naff "choose how insidious you would like us to be" series of questions. How would you like your ads? Targeted or non targeted? How about not at all? Nope. In this day and age, exac…

The article is referring to enterprise usage - and you're quoting all the consumer level attributes (aka cheap/sometimes subsidized version). At the enterprise level where this is intended to be ran, things are much diffrerent. If you're not aware of the differences or use cases, perhaps you're not the target audience who should be using or configuring it.