ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#1

Post by jarrell_mark » Fri, Mar 10, 2023, 9:51 PM UTC

ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI
colab.research.google.com

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#2

Post by johntash » Sat, Mar 11, 2023, 7:36 AM UTC

Can this be run locally without beefy GPUs by any chance?

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#3

Post by ops » Sat, Mar 11, 2023, 8:10 AM UTC

johntash wrote: post #2

Can this be run locally without beefy GPUs by any chance?

I haven't used this yet, but I am currently running GPT-J on my Mac Studio, so I suspect so.

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#4

Post by jerpint » Sat, Mar 11, 2023, 1:09 PM UTC

johntash wrote: post #2

Can this be run locally without beefy GPUs by any chance?

There have been CPU implementations of LLAMA (7b parameters, comparable in size) with very impressive performance

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#5

Post by serendipty01 » Sat, Mar 11, 2023, 2:20 PM UTC

Github repo : https://github.com/jarrellmark/chatgpt-j

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#6

Post by quesomaster9000 » Sat, Mar 11, 2023, 4:11 PM UTC

johntash wrote: post #2

Can this be run locally without beefy GPUs by any chance?

ggml (https://github.com/ggerganov/ggml) has a GPT-J example, the 6B parameter model runs happily on the CPU 16gb of ram and 8 cores at a couple of words per second, no GPUs necessary.

    gptj_model_load: ggml ctx size = 13334.86 MB
    gptj_model_load: memory_size =  1792.00 MB, n_mem = 57344
    gptj_model_load: model size = 11542.79 MB / num tensors = 285
    main: number of tokens in prompt = 12

    An example of GPT-J running on the CPU is shown in Fig. [4](#Fig4

    main: mem per token = 16179460 bytes
    main:     load time =  7463.20 ms
    main:   sample time =     3.24 ms
    main:  predict time =  4887.26 ms / 232.73 ms per token
    main:    total time = 13203.91 ms

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#7

Post by two_handfuls » Sat, Mar 11, 2023, 4:20 PM UTC

"Privacy-First", but also working in a colab notebook - meaning running on someone else's machine? That doesn't seem very private.

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#8

Post by was_a_dev » Sat, Mar 11, 2023, 7:25 PM UTC

two_handfuls wrote: post #7

"Privacy-First", but also working in a colab notebook - meaning running on someone else's machine? That doesn't seem very private.

Download the notebook and run locally?

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#9

Post by jarrell_mark » Mon, Mar 13, 2023, 5:11 AM UTC

johntash wrote: post #2

Can this be run locally without beefy GPUs by any chance?

It should work with about 12gb GPU RAM.

I got it to load on a GTX 1070 with 8GB GPU RAM, but then it crashed before it could generate a response.

It needs less RAM than regular GPT-J because the weights are converted to 8-bit

Re: ChatGPT-J: The Privacy-First, Self-Hosted Chatbot Built on GPT-J's Powerful AI

#10

Post by jarrell_mark » Mon, Mar 13, 2023, 7:54 AM UTC

was_a_dev wrote: post #8

two_handfuls wrote: post #7

"Privacy-First", but also working in a colab notebook - meaning running on someone else's machine? That doesn't seem very private.

Download the notebook and run locally?

Yes, the GitHub has the Jupyter .ipynb notebook that can be run locally: https://github.com/jarrellmark/chatgpt-j

And even in Colab, it's privacy first in the sense that user input or model output isn't being sent anywhere. The data is local to your Colab session.