Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

github.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#11

Post by neilv »

jasonwcfan wrote: post #8
neilv wrote: post #5

Earlier quoted context omitted.

I disagree. Open source involves the "source" being available, not just the "compiled".

The concept of “source” is nebulous for ML models. If you have the weights you can recreate a model without access to the source code originally used to train it, and similarly just having the source code without the training data won’t allow you to recreate the model. While it would be nice to have the data set Meta used I think open sourcing the weights is good enough.

I think some marketers are trying to use this term "open source" to try to ride on the goodwill and perceived benefits of open source, without actually doing it.

Also, people who just want to be able to run something on their computer without paying money for it shouldn't call it "open source", unless it actually is.

These distinctions have been going on for decades, for very good reasons. No need to throw away that progress now.

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#12

Post by e12e »

wmf wrote: post #10
mmastrac wrote: post #4

Earlier quoted context omitted.

I don't think we've collectively figured out how to describe what "weights openly available" means, so open-source is probably a reasonable descriptor.

Maybe we could call it... Open Weights™.

More like Weights Available in the case of llama2 (and Bring Your Own Pirate Treasure in case of llama1?).

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#13

Post by e12e »

This looks like a great project. Given the costs, I imagine many might want to run on dedicated hardware with GPU - yet:

> GPT4All: When you run locally, RAGstack will download and deploy Nomic AI's gpt4all model, which runs on consumer CPUs.

> Falcon-7b: On the cloud, RAGstack deploys Technology Innovation Institute's falcon-7b model onto a GPU-enabled GKE cluster.

> LLama 2: On the cloud, RAGstack can also deploy the 7B paramter version of Meta's Llama 2 model onto a GPU-enabled GKE cluster.

Why not llama2 on dedicated/local hardware? Memory and download size requirements?

Ed: After reading the linked tutorial - it looks like the built docker container will run fine on local/dedicated hardware?

https://www.psychic.dev/post/how-to-deploy-llama-2-to-google...

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#15

Post by jasonwcfan »

Q6T46nT668w6i3m wrote: post #9
jasonwcfan wrote: post #8

Earlier quoted context omitted.

The concept of “source” is nebulous for ML models. If you have the weights you can recreate a model without access to the source code originally used to train it, and similarly just having the source code without the training data won’t allow you to recreate the model. While it would be nice to have the data set Meta used I think open sourcing the weights is good enough.

No. The weights encode recorded parameters they don’t encode essential components like hyperparameters or modules without recorded parameters.

You're right. Either way it's impossible to recreate Llama 2 without the data set so perhaps "free to use model" is a better description than "open source model"

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#18

Post by jasonwcfan »

vertis wrote: post #14

Trying to run this locally and once I get past a few gotchas (local.env, needing to be renamed to .env) and needing to `pip3 install poetry`. I start getting back responses like "D I'm sure I'm doing something wrong :)

Thanks for the callout! We'll add the local.env instructions to the readme.

Are you using it with input docs or without? Locally it uses GPT4all which isn't nearly as good as Llama or Falcon. I saw a project that is docker for Llama 2 so we might use that instead!

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#19

Post by jasonwcfan »

Jayakumark wrote: post #16

Does it use openAI embeddings or other free ones ?

It uses all-MiniLM-L6-v2 from huggingface by default

https://huggingface.co/sentence-transformers/all-MiniLM-L6-v...

You can also specify a specific embeddings model from SentenceTransformers to use in /server/.env

Re: Show HN: RAGstack – private ChatGPT for enterprise VPCs, built with Llama 2

#20

Post by jasonwcfan »

generalizations wrote: post #17

Is there a version of this set up to be cpu-only, as in something that can use ggml tech? I'd love to deploy this on some servers with lots of ram and cpu horsepower, but no gpus.

Not yet, but we can definitely add it. Created an issue: https://github.com/psychic-api/rag-stack/issues/2

In the meantime it uses GPT4all when running locally so you can technically deploy it as well, but it's not very good.

Post reply on HN