Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

github.com

Post reply on HN
Print view View original

221–230 of 349 posts

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#221

Post by jmorgan »

pyrophane wrote: post #181
jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

> While I haven't tested it extensively, 70B model is supposed to rival Chat GPT 3.5 in most areas, and there are now some new fine-tuned versions that excel at specific tasks That has been my experience. Having experimented with both (informally), Llama 2 is similar to GPT-3.5 for a lot of general comprehension questions. GPT-4 is still the best amongst the closed-source, cutting edge models in terms of general conv…

RE 2 - neat! What are some tasks you've been using smaller models (with perhaps larger context sizes) for?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#222

Post by sailfast »

c0nsumer wrote: post #209
weird-eye-issue wrote: post #191

Earlier quoted context omitted.

Microsoft says it is more secure. And that it is enterprise . That's about it

There are legal agreements backing the separation of company data from other parties. This is what's important to big corps.

I have to imagine Big Corps are also concerned about liability / risk when generating things with OpenAI products - at least until there is some sort of settled law around using models trained on this kind of data.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#223

Post by tayo42 »

stavros wrote: post #174
tayo42 wrote: post #173

Earlier quoted context omitted.

For llama, the 4bit quantized ones, small models like the 7b one. The ggml format. That will run on your local cpu. Google those terms too. you can look on hugging face for the actual model to download then load it and send prompts to it

Thanks, maybe it's as easy as downloading the ggml and running it with Llama.cpp. I'll try that, thanks!

there is also a python wrapper that has a web ui built in for llama.cpp, if it wasnt easy enough already

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#224

Post by scrum-treats »

cosbgn wrote: post #123
toomuchtodo wrote: post #84

Earlier quoted context omitted.

Can you plug this together with tools like api2ai to create natural language defined workflow automations that interact with external APIs?

You can use unfetch.com to make API calls via LLMs and build automations. (I'm building it)

Is it possible to not use Google with unfetch.com?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#225

Post by ramraj07 »

pyrophane wrote: post #181
jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

> While I haven't tested it extensively, 70B model is supposed to rival Chat GPT 3.5 in most areas, and there are now some new fine-tuned versions that excel at specific tasks That has been my experience. Having experimented with both (informally), Llama 2 is similar to GPT-3.5 for a lot of general comprehension questions. GPT-4 is still the best amongst the closed-source, cutting edge models in terms of general conv…

Could you give examples of smaller models trained on specific datasets?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#226

Post by BoorishBears »

DebtDeflation wrote: post #189
BoorishBears wrote: post #160

Earlier quoted context omitted.

It sounds like a cop-out but: it's one made for your use-case. If you're letting people do fun long-form roleplay adventures using summarization alongside some sort of named entity K-V store driven by the LLM would be a good strategy. If you're building a tool that's mostly for internal data, something that leans heavily into detailed answers with direct verbatim citations and having your frontend create new threads…

>alongside some sort of named entity K-V store driven by the LLM I'd be curious to hear more about how exactly this works. You do NER on the prompt (and maybe on the completion too) and store the entities in a database and then what? How does the LLM interact with it?

LLMs thrive at completely ambiguous classifications: you can have them extract entities and something like "a list of notable context".

Let's say we want to let our chat remember the character slammed the door last time they were in Village X with the mayor in their presence and have the mayor comment next time they see the player.

Every X tokens we can fire a prompt with a chunk of conversation and a list of semantically similar entities that already exist, letting the LLM return an edited list along the lines of:

   entity: mayor

   location: village X

   priority: HIGH

   keywords: town hall, interact, talk

   "memory, likelyEffect"[]: door slammed in face, anger at player
Now we have:

- multiple fields for similarity search

- an easy way to manage evictions (sweep up lowest priority)

- most importantly: we're providing guidance for the LLM to help it ignore irrelevant context

When the user goes back to village X we can fetch entities in village X and whittle that list down based on priority and similarly to the user prompt.

None of this has any determinism: instead you're optimizing for the illusion of continuity and trading off predictability.

You're aiming for players being shocked that next time they talk to the mayor he's already upset with them, and if they ask why he can reply intelligently.

And to my original point while this works for a game-like experience, you wouldn't want to play around with this kind of fuzzy setup for your companies internal CRM bot or something. You're optimizing for the exact value proposition of your use-case rather than just trying to throw a raw RAG setup at it

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#227

Post by wodenokoto »

Xenoamorphous wrote: post #75
paxys wrote: post #30

Would it be too much to mention somewhere in the README what this repo actually contains? Just docs? Deployment files? Some application (which does..something)? The model itself?

The repo contains the UI code, not the model or anything else around ChatGPT, it just uses Azure’s ChatGPT API which doesn’t share data with OpenAI.

Isn’t there also some sort of backend stuff in there? How else would it keep track of history and accept documents.

I don’t know enough typescript to understand where the front end stops and the backend begins I this code

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#228

Post by xist »

gerdesj wrote: post #180
xist wrote: post #166

Earlier quoted context omitted.

The article is referring to enterprise usage - and you're quoting all the consumer level attributes (aka cheap/sometimes subsidized version). At the enterprise level where this is intended to be ran, things are much diffrerent. If you're not aware of the differences or use cases, perhaps you're not the target audience who should be using or configuring it.

Why don't we give the willy waving a miss? Win 10 and 11 are steering you to cloud first, out of the box. That's fine if you like it, but I don't and quite a lot of my customers don't. The real problem is about data sovereignty. I'm a Brit and ... MS isn't.

The article is about use in an enterprise. An enterprise runs professional/enterprise/ltsc versions which do NOT steer you to the cloud - what data sovereignty concerns have you seen in those editions of windows/server? They've gone through a lot of pains to ensure those concerns are taken care of for enterprises/governments so i'm curious the ones you think they missed.

You can make the argument for their consumer editions sure, but that's a different product with different features, different price point for different users.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#229

Post by hug »

sailfast wrote: post #222
c0nsumer wrote: post #209

Earlier quoted context omitted.

There are legal agreements backing the separation of company data from other parties. This is what's important to big corps.

I have to imagine Big Corps are also concerned about liability / risk when generating things with OpenAI products - at least until there is some sort of settled law around using models trained on this kind of data.

Yes, those concerns exist, but they're also practically impossible to enforce.

At my enterprise, it's a three step solution, two of which don't work.

1. Written policy concerning LLM output and its risks, disallow it for being used for any kind of official documentation or decision making. (This doesn't work, because no one wants to use their own brain to do tedious paperwork.)

2. Block access to public LLM tools via technical means from company owned end-user devices. (This doesn't work because people will just open ChatGPT on their home PC or mobile.)

3. Write and provide our own gpt-3.5 frontend, so that when people ignore rules #1 and #2 we have logs, and we know we're not feeding our proprietary info to to OpenAI.

Post reply on HN