Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

github.com

Post reply on HN
Print view View original

211–220 of 349 posts

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#211

Post by RockyMcNuts »

37112741 wrote: post #186

Nothing in the repo details how this addresses privacy concerns of running inference on someone else's LLM. To be isolated from other users of the service is not the same thing as having a private inference engine. > Private: Built-in guarantees around the privacy of your data and fully isolated from those operated by OpenAI. Do tell.

So where do you draw the line? No cloud instances, no cloud SQL like Snowflake, no Teams or Office 365, no S3/blob storage? Run everything on-prem like 10 years ago?

It's only going to get more impossible. All that VC money going in at 100x revenue needs a return and they aren't going leave money on the table with full-featured open-source or CentOS type alternatives.

All those data engineering startups, database providers with 'open-source' + cloud hosting, the 'open-source' is going to be just 'open' enough to claim there is some fallback for someone else to pick up the mantle using the community version, if the cloud version gets enshittified beyond reason.

You're not going to even be able to run the full-featured software version on-prem because the economics of cloud are so much better.

Unless you are writing and compiling your own code you are going to be out of luck if your privacy standard is that high. That war has been lost. And Web3 sure ain't gonna save you either.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#212

Post by RockyMcNuts »

klysm wrote: post #50
albert_e wrote: post #2

is there away to run this on AWS instead. we were looking to explore Llama2 for internal use

Msft spent a lot of money to ensure that was not an option w chatgpt

https://about.fb.com/news/2023/07/llama-2/ https://huggingface.co/blog/hugging-face-endpoints-on-azure

You can of course run Llama2 in Azure, but you can't host OpenAI models in AWS

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#214

Post by villgax »

speedgoose wrote: post #43
villgax wrote: post #18

Earlier quoted context omitted.

Have your engineers set this up internally https://huggingface.co/spaces/huggingface-projects/llama-2-7...

You can’t really replace ChatGPT 4 with llama2 7B.

Yeah right try getting the same answer after two months

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#215

Post by oceanplexian »

xcdzvyn wrote: post #197
ttul wrote: post #162

Earlier quoted context omitted.

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for e…

> The closed source players have the best researchers Is that definitely why? GPT 3.5 and GPT 4 are far larger than 70B, right? So if a 70B, local model like LLaMA can even remotely rival them, would that not suggest that LLaMA is fundamentally a better model? For example, would a LLaMA model with even half of GPT 4's parameters be projected to outperform it? Is that how it works? [I'm not super familiar with LLM tec…

LLamA 2 at 70B is, let’s say pessimistically 70% as good as GPT3.5. This makes me think that OpenAI is lying about their parameter count, are vastly less efficient than LLaMA, or, the lager model sizes have diminishing returns. Either way, your point is a good one. Something doesn’t add up.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#216

Post by llm_thr »

ttul wrote: post #162
jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for e…

You seriously underestimate just how much _not_ having to tune your llm for SF sensibilities benefits performance.

As an example from the last six months: people on tor are producing better than state of the art stable diffusion because they want porn without limitations. I haven't had the time to look at llm's but the degenerates who enjoy that sort of thing have said they can get the Llama2 model to role play their dirty fantasies and then have stable diffusion illustrate said fantasies. It's a brave new world and it's not on the WWW.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#217

Post by nl »

ttul wrote: post #162
jmorgan wrote: post #152

This appears to be a web frontend with authentication for Azure's OpenAI API, which is a great choice if you can't use Chat GPT or its API at work. If you're looking to try the "open" models like Llama 2 (or it's uncensored version Llama 2 Uncensored), check out https://github.com/jmorganca/ollama or some of the lower level runners like llama.cpp (which powers the aforementioned project I'm working on) or Candle, the…

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for e…

> The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that.

Llama2 came out of Meta's AI group. Meta pays researcher salaries competitive with any other group, and their NLP team is one of the top groups in the world.

For researchers it is increasingly the most attractive industrial lab because they release the research openly.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#218

Post by nl »

xcdzvyn wrote: post #197
ttul wrote: post #162

Earlier quoted context omitted.

Llama 2 might by some measures be close to GPT 3.5, but it’s nowhere near GPT 4, nor Anthropic Claude 2 or Cohere’s model. The closed source players have the best researchers - they are being paid millions a year with tons of upside - and it’s hard to keep pace with that. My sense is that the foundation model companies have an edge for now and will probably stay a few steps ahead of the open source realm simply for e…

> The closed source players have the best researchers Is that definitely why? GPT 3.5 and GPT 4 are far larger than 70B, right? So if a 70B, local model like LLaMA can even remotely rival them, would that not suggest that LLaMA is fundamentally a better model? For example, would a LLaMA model with even half of GPT 4's parameters be projected to outperform it? Is that how it works? [I'm not super familiar with LLM tec…

If you read the Llama2 paper it is very clear that small amounts of data (thousands of records) make vast difference at the instruction turning stage. From the Llama2 paper:

> Quality Is All You Need.

> Third-party SFT data is available from many different sources, but we found that many of these have insufficient diversity and quality — in particular for aligning LLMs towards dialogue-style instructions. As a result, we focused first on collecting several thousand examples of high-quality SFT data, as illustrated in Table 5. By setting aside millions of examples from third-party datasets and using fewer but higher-quality examples from our own vendor-based annotation efforts, our results notably improved. These findings are similar in spirit to Zhou et al. (2023), which also finds that a limited set of clean instruction-tuning data can be sufficient to reach a high level of quality. We found that SFT annotations in the order of tens of thousands was enough to achieve a high-quality result. We stopped annotating SFT after collecting a total of 27,540 annotations. Note that we do not include any Meta user data.

It's likely OpenAI has invested in this and has good coverage in a larger range of domains. That alone probably explains a large amount of the gap.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#220

Post by Buttons840 »

samrolken wrote: post #92
Buttons840 wrote: post #32

Earlier quoted context omitted.

As an example of this. I found that GPT4 wouldn't agree with me that C(A) = C(AA^T) until I explained the proof. A few weeks later it would agree in new chats and would explain using the same proof I did presented the same way.

I’ve found that the behavior of ChatGPT can vary widely from session to session. The recent information about GPT4 being a “mixture of experts” might also be relevant. Do we know that it wouldn’t have varied in its answer by just as much, if you had tried in a new session at the same time?

I tested it several times, new chats never got this right at first. I tried at least 6 times. I was experimenting and found that GPT4 couldn't be fooled by faulty proofs. Only a valid proof could change its mind.

Now it seems to know this mathematical property from first prompt though.

Post reply on HN