Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

github.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#21

Post by vertis »

jasonwcfan wrote: post #18
vertis wrote: post #14

Trying to run this locally and once I get past a few gotchas (local.env, needing to be renamed to .env) and needing to `pip3 install poetry`. I start getting back responses like "D I'm sure I'm doing something wrong :)

Thanks for the callout! We'll add the local.env instructions to the readme. Are you using it with input docs or without? Locally it uses GPT4all which isn't nearly as good as Llama or Falcon. I saw a project that is docker for Llama 2 so we might use that instead!

I tried both. I'll certainly try it remotely as well. Was just pottering through HN before bed.

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#23

Post by jasonwcfan »

e12e wrote: post #13

This looks like a great project. Given the costs, I imagine many might want to run on dedicated hardware with GPU - yet: > GPT4All: When you run locally, RAGstack will download and deploy Nomic AI's gpt4all model, which runs on consumer CPUs. > Falcon-7b: On the cloud, RAGstack deploys Technology Innovation Institute's falcon-7b model onto a GPU-enabled GKE cluster. > LLama 2: On the cloud, RAGstack can also deploy t…

Yep the docker containers should run fine on local hardware, but the terraform config only supports GCP right now.

In terms of cost - just ran our deployed cluster through GCP's pricing calculator and it's about $300 USD per month. Definitely not cheap for individual use, but pretty affordable for enterprise use. Running the 40B parameter version will be significantly more.

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#24

Post by jasonwcfan »

rozap wrote: post #22

So this dumps the documents returned from the vector store into a prompt to the LLM. How does it work when there are many documents returned? What's the upper limit there?

Yep. We use LangChain's basic text splitter to chunk the documents and the QA chain to stuff it into the prompt. But AFAIK it doesn't check for context length so that's a piece that's still missing.

Upper limit depends on the model, Llama 2 is 4k including the prompt.

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#25

Post by jasonwcfan »

WillPostForFood wrote: post #6

Approximately, what would the hourly cost of running this be on Google Cloud? >In the default-pool > Nodes tab, set: >Machine Configuration from General Purpose to GPU >GPU type: Nvidia TF >Number of GPUs: 1 >Enable GPU time sharing >Max shared clients per GPU: 8 >Machine type: n1-standard-4 >Boot disk size: 50 GB >Enable nodes on spot VMs Not familiar with GCP, but I see n1-standard-4's are in an instance type that…

Just ran our deployed cluster through GCP's pricing calculator and it's about $300 USD per month with Llama 2

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#26

Post by e12e »

jasonwcfan wrote: post #23
e12e wrote: post #13

This looks like a great project. Given the costs, I imagine many might want to run on dedicated hardware with GPU - yet: > GPT4All: When you run locally, RAGstack will download and deploy Nomic AI's gpt4all model, which runs on consumer CPUs. > Falcon-7b: On the cloud, RAGstack deploys Technology Innovation Institute's falcon-7b model onto a GPU-enabled GKE cluster. > LLama 2: On the cloud, RAGstack can also deploy t…

Yep the docker containers should run fine on local hardware, but the terraform config only supports GCP right now. In terms of cost - just ran our deployed cluster through GCP's pricing calculator and it's about $300 USD per month. Definitely not cheap for individual use, but pretty affordable for enterprise use. Running the 40B parameter version will be significantly more.

Out of curiosity how does that gcp instance compare to my modest gaming rig (Nvidia 3080 24(?)gb ram/Ryzen 7/64gb ram)? (Since I'm paying 0/month for it ...).

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#27

Post by kwerk »

jasonwcfan wrote: post #23
e12e wrote: post #13

This looks like a great project. Given the costs, I imagine many might want to run on dedicated hardware with GPU - yet: > GPT4All: When you run locally, RAGstack will download and deploy Nomic AI's gpt4all model, which runs on consumer CPUs. > Falcon-7b: On the cloud, RAGstack deploys Technology Innovation Institute's falcon-7b model onto a GPU-enabled GKE cluster. > LLama 2: On the cloud, RAGstack can also deploy t…

Yep the docker containers should run fine on local hardware, but the terraform config only supports GCP right now. In terms of cost - just ran our deployed cluster through GCP's pricing calculator and it's about $300 USD per month. Definitely not cheap for individual use, but pretty affordable for enterprise use. Running the 40B parameter version will be significantly more.

What is the capacity for that price?

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#30

Post by unstatusthequo »

I see a few nice connectors, but doesn't seem to support network shares to unstructured data. Not all enterprises host on cloud. Not all use Google Cloud, or will be willing to. Some want it local to their network. This also becomes a lot more interesting with M365 integration (OneDrive, SharePoint, Teams, Loop, etc) and boring old network share paths or local filesystem sources? I'd love to try it at least now on the two latter things without connectors to SaaS things. I have dozens of terabytes of data to test with this. Any plans for these more common sources?
Post reply on HN