Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

github.com

Post reply on HN
Print view View original

181–190 of 349 posts

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#181

Post by pyrophane »

jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

> While I haven't tested it extensively, 70B model is supposed to rival Chat GPT 3.5 in most areas, and there are now some new fine-tuned versions that excel at specific tasks

That has been my experience. Having experimented with both (informally), Llama 2 is similar to GPT-3.5 for a lot of general comprehension questions.

GPT-4 is still the best amongst the closed-source, cutting edge models in terms of general conversation/reasoning, although 2 things:

1. The guardrails that OpenAI has placed on ChatGPT are too aggressive! They clamped down on it quite hard to the extent that it gets in the way of a reasonable query far too often.

2. I've gotten pretty good results with smaller models trained on specific datasets. GPT-4 is still on top in terms of general purpose conversation, but for specific tasks, you don't necessarily need it. I'd also add that for a lot of use cases, context size matters more.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#183

Post by pwarner »

PoignardAzur wrote: post #138

> However, ChatGPT risks exposing confidential intellectual property. One option is to block corporate access to ChatGPT, but people always find workarounds Pretty bold thing to say to your potential clients. "You can always tell your employees not to use our product, but they won't listen to you."

It's almost like employees might have their own computers?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#184

Post by shubb »

BoorishBears wrote: post #117
shubb wrote: post #85

Earlier quoted context omitted.

This is one of the things that make me uncomfortable about proprietary llm. They get task performance by doing a lot more than just feeding a prompt straight to an llm, and then we performance compare them to raw local options. The problem is, as this secret sauce changes, your use case performance is also going to vary in ways that are impossible for you to fix. What if it can do math this month and next month the h…

I'm not sure you realize how proprietary LLMs are being built on. No one is doing secret math in the backend people are building on. The OpenAI API allows you to call functions now, but even that is just a formalized way of passing tokens into the "raw LLM". All the features in the comment you replied to only apply to the web interface , and here you're being given an open interface you can introspect.

Thank you for pointing that out - I had assumed that things were not how they are.

Although performance has varied over time https://arxiv.org/pdf/2307.09009.pdf I also notice that the API allows you to use a frozen version of the model which avoids the worries I mentioned.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#185

Post by smoldesu »

MuffinFlavored wrote: post #171
robertnishihara wrote: post #167

Earlier quoted context omitted.

> Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4 I think you're right about this, and benchmarks we've run at Anyscale support this conclusion [1]. The caveat there (which I think will be a big boon for open models) is that techniques like fine-tuning makes a HUGE difference and can bridge the quality gap between Llama-2 and GPT-4 for many (but not all) problems. [1] https://www.anysc…

can I fine tune it on like 2,000 repos at a corporation (code based) and have it understand the architecture?

I don't think you can do that with any AI models. It almost feels like a fundamental misrepresentation of how they work.

You could fine-tune a conversational AI on your codebase, but without loading said codebase into it's context it is "flying blind" so-to-speak. It doesn't understand the data structure of your code, the relation between files and probably doesn't confidently understand the architecture of your system. Without portions of your codebase loaded into the 'memory' of your model, all that your finetuning can do is replicate characteristics of your code.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#186

Post by 37112741 »

Nothing in the repo details how this addresses privacy concerns of running inference on someone else's LLM. To be isolated from other users of the service is not the same thing as having a private inference engine.

> Private: Built-in guarantees around the privacy of your data and fully isolated from those operated by OpenAI.

Do tell.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#189

Post by DebtDeflation »

BoorishBears wrote: post #160
extr wrote: post #121

Earlier quoted context omitted.

What front end is better than ChatGPT? Is the OP implementation doing running summarization or in-convo embedding lookup?

It sounds like a cop-out but: it's one made for your use-case. If you're letting people do fun long-form roleplay adventures using summarization alongside some sort of named entity K-V store driven by the LLM would be a good strategy. If you're building a tool that's mostly for internal data, something that leans heavily into detailed answers with direct verbatim citations and having your frontend create new threads…

>alongside some sort of named entity K-V store driven by the LLM

I'd be curious to hear more about how exactly this works. You do NER on the prompt (and maybe on the completion too) and store the entities in a database and then what? How does the LLM interact with it?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#190

Post by jmorgan »

jgalt212 wrote: post #176
ttul wrote: post #162

Earlier quoted context omitted.

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for e…

OK, fair enough. Please give me an example of a customer facing chatbot that Llama 2 (and unbearable to use) and GPT 4 customer facing chatbot that is a joy to use. I think at the end of the day, you still have customers dreading such interactions.

It's early, and this definitely isn't customer facing in the traditional sense, but a team member of mine set up a Discord bot running Llama 2 70B on a Mac studio and we've been quite impressed by its responses to folks who test it.

IIRC chat bots are central the vision Facebook has with LLMs (e.g. every instagram account has a personal chat bot), so I would expect the Llama models to get increasingly better at this task.

That said the 7B and 13B models definitely don't quite seem ready yet for production customer interaction :-)

Post reply on HN