Were transformers used in other contexts ? biochemistry ? geometry .. whatever.
The Coming of Local LLMs
111–120 of 214 posts
Re: The Coming of Local LLMs
#112Earlier quoted context omitted.
Because OpenAI or Google being the gate keepers is a show stopper for most serious people in the space. They will be able to shutdown your startup on a whim. Even if they didn't politicians and regulators would be a huge risk. Without democratization we get blade runner . Not that democratization has no problems, just that it is the way a lot of us are wanting it to go.
And basing your startup on a model pirated from Meta isn't risky? I get it, I understand why people like decentralization - but the open source community doesn't even have close to the capability to train an actually open-source LLaMA equivalent.
Mozilla or Wikimedia foundation have budgets with the right number of zeroes to support an effort like this.
While those organizations might not be the right fit, I think they serve as an "existence proof" that a large scale open / nonprofit project is not completely inconceivable.
Could possibly see an industry consortium (of players too small to compete on their own) funding an open effort.
Last idea sounds crazy but hear me out: how much would Nvidia spending $100M on "open" models boost spending on graphics cards? I hope someone's running the numbers on that...
Re: The Coming of Local LLMs
#113I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…
Super curious how you did this! Doesn’t 30B model require a hefty computer to run locally (assuming you’re tuning a non-quantized version)
Not at all. Even a Raspberry Pi would do - you only need ~6GiB RAM for a 4-bit quantized LLaMA model (though it's gonna be quite slow). A decent modern desktop machine would do just fine, no need for anything extra fancy.
What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes.
Re: The Coming of Local LLMs
#114Out of doubt, which seems to be spreading around the internet. The LLaMa model weights weren't "leaked" AFAIK but rather explicitly given access to to researchers, isn't it right?
I know the article goes on to speak about something else, but I'm not sure why this claim that the LLaMa model weights were leaked, as in unintendenly made available is being done.
Re: The Coming of Local LLMs
#115Re: The Coming of Local LLMs
#116Earlier quoted context omitted.
And basing your startup on a model pirated from Meta isn't risky? I get it, I understand why people like decentralization - but the open source community doesn't even have close to the capability to train an actually open-source LLaMA equivalent.
Mozilla or Wikimedia foundation have budgets with the right number of zeroes to support an effort like this. While those organizations might not be the right fit, I think they serve as an "existence proof" that a large scale open / nonprofit project is not completely inconceivable. Could possibly see an industry consortium (of players too small to compete on their own) funding an open effort. Last idea sounds crazy b…
> how much would Nvidia spending $100M on "open" models boost spending on graphics cards?
Probably not as much as in a zero-sum game where everyone is trying to train their own model. Every leap in CPU inference makes this an increasingly less appealing option for them.
But I agree, some industry consortium might try to do it. I think they would first have to be relying heavily on LLMs before they were willing to do that, and its possible by then that the lead will have gotten too large to easily surmount, especially now that everyone has stopped publishing.
Re: The Coming of Local LLMs
#117Earlier quoted context omitted.
is there a difference in the quality of llm one would be able to train or run on a gpu with 8, 12, 16, all the way up to 24gb? I'm trying to decide whether it's worth while to splurge on a more expensive 4090 vs a 4070 or whatever.
It makes 0 economical sense to buy a GPU to train a model. If you want to train a model, train it on the cloud.
Can you calculate that for me on a napkin? Every calculation I make for training, but certainly for inference, makes me break even after well under a year and then it’s vastly less if I buy the hardware myself.
Re: The Coming of Local LLMs
#118I expect we will see the biggest jump in performance if (when) consumer-grade coprocessors like mobile GPUs start incorporating attention layers as a primitive building block at the hardware level, e.g., with instructions and memory layouts engineered specifically to make ultra-low-precision (say, 4-bit) transformer layers as compute- and memory-efficient as possible on consumer devices. That seems almost inevitable…
There are those Google accelerators that plug into an M.2 slot. You could plausibly do this today, although I am not sure what sort of memory constraints those accelerators have.
Re: The Coming of Local LLMs
#119Earlier quoted context omitted.
It makes 0 economical sense to buy a GPU to train a model. If you want to train a model, train it on the cloud.
Can you calculate that for me on a napkin? Every calculation I make for training, but certainly for inference, makes me break even after well under a year and then it’s vastly less if I buy the hardware myself.
You can infer on your local machine.
I doubt it will take you a year to train a model.
Vicuna-13B cost $300 to train/fine-tune [0]. They trained on an A100 which costs $10k [1].
[0]: https://vicuna.lmsys.org/ [1]: https://github.com/lm-sys/FastChat/blob/main/scripts/train-v...