Live data from Hacker News

Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

github.com

81–90 of 349 posts

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#81

Our company is pushing everyone to use a similar offering. Most of the company is doing low value work … still using excels even though we have a custom ERP. Now seeing people who couldn’t write a coherent email before write 3 page emails. The illusion of being productive by doing more work even though it has zero impact on the bottom line. It’s insane how inefficient organisations are. No doubt we’ll have some KPI s…

If anything it's less productive because people have to parse all that nonsense. I was gobsmacked to hear a friend say that their work guidance is to use ChatGPT to write letters to external clients for example. I know for sure I'd be insulted if someone sent me paragraphs of text to read created from a sentence long prompt. I'd rather have the prompt, my time is valuable as well.

The amount of othewise very smart people who completely lose the ability to think critically when it comes to "AI" is really interesting to me.

I'm not anti-AI; I've recommended that we use it at work a few times where it made sense and was backed by evidence/bencharmks. But for essentially any problem that comes up someone will try to solve it with ChatGPT, even if it demonstrably can't do the job. And these are not business folks, these are engineering leaders who absolutely have the capability to understand this technology.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#82
This is awesome to see, feels heavily inspired (in a good way) by the version we made at Vercel[1]. Same tech stack: Next.js, NextAuth, Tailwind, Shadcn UI, Vercel AI SDK, etc.

I'd expect this trend of managed ChatGPT clones to continue. You can own the stack end to end, and even swap out OpenAI for a different LLM (or your own model trained on internal company data) fairly easily.

[1]: https://vercel.com/templates/next.js/nextjs-ai-chatbot

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#83
post #79

One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they never work quite as well, it's obvious that you run out of context after a few conversation turns. Is ChatGPT doing some embedding lookup inside the conversation thread to make the context feel infinite? I've noticed anecdotally it definitely isn't infinite, but i…

It uses a sliding context windows. Older tokens are dropped as new ones stream in

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#84
post #77

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at le…

Can you plug this together with tools like api2ai to create natural language defined workflow automations that interact with external APIs?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#85
post #79

One thing I still don't understand is what _is_ the ChatGPT front end exactly? I've used other "conversational" implementations built with the API and they never work quite as well, it's obvious that you run out of context after a few conversation turns. Is ChatGPT doing some embedding lookup inside the conversation thread to make the context feel infinite? I've noticed anecdotally it definitely isn't infinite, but i…

This is one of the things that make me uncomfortable about proprietary llm.

They get task performance by doing a lot more than just feeding a prompt straight to an llm, and then we performance compare them to raw local options.

The problem is, as this secret sauce changes, your use case performance is also going to vary in ways that are impossible for you to fix. What if it can do math this month and next month the hidden component that recognizes math problems and feeds them to a real calculator is removed? Now your use case is broken.

Feels like building on sand.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#86

This is potentially a huge deal. Companies are concerned using ChatGPT might violate data privacy policies if someone puts in user data or invalidate trade secrets protections if someone uploads sections of code. I suspect many companies have been waiting for an enterprise version.

This is a web UI that talks to a (separate) Azure OpenAI resource that you can deploy into your subscription as a SaaS instance.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#88
post #77

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at le…

Can you plug this together with tools like api2ai to create natural language defined workflow automations that interact with external APIs?

There is a generic HTTP API processor that can be used to call APIs as part of the app flow which should help invoke tools. Currently working on improving documentation so it is easy to get started with the project. We also have some features planned around function calling that should make it easy to natively integrate tools into the app flows.

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#89
post #77

A lot of companies are already using projects like chatbot-ui with Azure's OpenAI for similar local deployments. Given this is as close to local ChatGPT as any other project can get, this is a huge deal for all those enterprises looking to maintain control over their data. Shameless plug: Given the sensitivity of the data involved, we believe most companies prefer locally installed solutions to cloud based ones at le…

I find it interesting to see how competitive this space got so quickly.

How do these stacks differentiate?

Re: Azure ChatGPT: Private and secure ChatGPT for internal enterprise use

#90
This seems like such an obvious thing to do.

I see the use of general purpose LLMs like ChatGPT, but smaller fine tuned models will probably end up being more useful for deployed applications in most companies. Off topic, but I was experimenting with LLongMA-2-7b-16K today, running it very inexpensively in the cloud, and given about 12K of context text it really performed well. This is an easy model to deploy. 7B parameter models can be useful.

Post reply on HN