Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

github.com

Post reply on HN
Print view View original

101–110 of 349 posts

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#101

Post by extr »

SOLAR_FIELDS wrote: post #95
extr wrote: post #79

One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they never work quite as well, it's obvious that you run out of context after a few conversation turns. Is ChatGPT doing some embedding lookup inside the conversation thread to make the context feel infinite? I've noticed anecdotally it definitely isn't infinite, but i…

They definitely do some proprietary running summarization to rebuild the context with each chat. Probably a RAG like approach that has had a lot of attention and work

This is effectively my question. I assume there is some magic going on. But how many engineering hours worth of magic, approximately? There is a lot of speculation around GPT-4 being MoE and whatnot. But very little speculation about the magic of the ChatGPT front end specifically that makes it feel so fluid.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#102

Post by baobabKoodaa »

FrenchDevRemote wrote: post #51
baobabKoodaa wrote: post #38

Earlier quoted context omitted.

> and since LLMs aren't even that good to begin with, it's obvious you want the SOTA to do anything useful unless maybe you're finetuning This is overkill. First of all, ChatGPT isn't even the SOTA, so if you "want SOTA to do anything useful", then this ChatGPT offering would be as useless as LLaMA according to you. Second, there are many individual tasks where even those subpar LLaMA models are useful - even without…

it's the SOTA for chat(prove me wrong), and you can always use the API directly even for simple tasks they're less reliable and needs more prompt engineering

> it's the SOTA for chat(prove me wrong)

GPT-4 beats ChatGPT on all benchmarks. You can easily google these.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#103

Post by bhanu423 »

ajhai wrote: post #77

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at le…

Interesting project - was trying it out, found an issue in building the image - have opened an issue on github - please take a look. Also do you have plan to support llama over openai models.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#104

Post by mritchie712 »

ilyt wrote: post #61
mritchie712 wrote: post #54

Earlier quoted context omitted.

ahhhh, but they're pasting the 3 page email into ChatGPT ("summarize this"). The future is here.

Wouldn't be surprised if that was next Outlook feature. Cue someone making some horrible error because some crucial information didn't survive ChatGPT->ChatGPT round-trip

it's already here... https://blogs.microsoft.com/blog/2023/03/16/introducing-micr...

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#106

Post by TuringNYC »

TillE wrote: post #91
tikkun wrote: post #41

Earlier quoted context omitted.

I did one. I took a few dozen prompts from my ChatGPT history and ran them through a few LLMs. GPT-4, Bard and Claude 2 came out on top. Llama 2 70b chat scored similarly to GPT-3.5, though GPT-3.5 still seemed to perform a bit better overall. My personal takeaway is I’m going to continue using GPT-4 for everything where the cost and response time are workable. Related: A belief I have is that LLM benchmarks are all…

I think tests like "can this LLM pass an English literature exam it's never seen before" are probably useful, but yeah there's a lot of silly stuff like math tests. I suppose the question is where are they most commercially viable. I've found them fantastic for creative brainstorming, but that's sort of hard to test and maybe not a huge market.

>> I suppose the question is where are they most commercially viable.

Fair point, though I'm not aiming to start a competing LLM SaaS service, rather i'm evaluating swapping out the TCO of Azure Cognitive Service OpenAI for the TCO of dedicated cloud compute running my own LLM -- to serve my own LLM calls currently being sent to a metered service (Azure Cognitive Service OpenAI)

Evaluation points would be: output quality; meter vs fixed breakeven points; latency; cost of human labor to maintain/upgrade

in most cases, i'd outsource and not think about it. BUT we're currently in some strange economics where the costs are off the charts for some services

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#107

Post by justinlloyd »

Interesting release, though still lacking a few features I've had to resort building myself such as code summary, code base architecture summary, and conversation history summary. ChatGPT (the web UI) now has the ability to execute code, and make function callbacks, but I prefer running that code locally, especially if I am debugging. This latter part, conversation history summary, is something that ChatGPT web UI does reasonably well, giving it a long history, but a sentiment extraction and salient detail extraction before summarizing is immensely useful for remembering details in the distant past. I've been building on top of the GPT4 model and tinkering with multi-model (gpt4 + davinci) usage too, though I am finding with the MoE that Davinci isn't as important. Fine tuning has been helpful for specific code bases too.

If I had the time I'd like to play with an MoE of Llama2, as a compare and contrast, but that ain't gonna happen anytime soon.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#108

Post by lsaferite »

extr wrote: post #100
MaxLeiter wrote: post #83

Earlier quoted context omitted.

It uses a sliding context windows. Older tokens are dropped as new ones stream in

I don't believe that's the whole story. Other conversational implementations use sliding context windows and it's very noticable as context drops off. Whereas ChatGPT seems to retain the "gist" of the conversation much longer.

I mean, I explicitly have the LLM summarize content that's about to fall out of the window as a form of pre-emptive token compression. I'd expect maybe they do something similar.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#109

Post by stavros »

baobabKoodaa wrote: post #102
FrenchDevRemote wrote: post #51

Earlier quoted context omitted.

it's the SOTA for chat(prove me wrong), and you can always use the API directly even for simple tasks they're less reliable and needs more prompt engineering

> it's the SOTA for chat(prove me wrong) GPT-4 beats ChatGPT on all benchmarks. You can easily google these.

The distinction between GPT-4 and ChatGPT is blurry, as ChatGPT is a chat frontend for a GPT model, and you can use GPT-4 with ChatGPT. The parent probably means ChatGPT with GPT-4.
Post reply on HN