Unfortunate, an opportunity to further enlighten others, but the author took a dismissive and antagonistic perspective.
Some Remarks on Large Language Models
21–30 of 87 posts
Re: Some Remarks on Large Language Models
#22The dismissal of biases and stereotypes is exactly why AI research needs more people who are part of the minority. Yoav can dismiss this because it just doesn't affect him much. It's easy to say "Oh well, humans are biased too" when the biases of these machines don't: misgender you, mistranslate text that relates to you, have negative affect toward you, are more likely to write violent stories related to you, have lo…
[flagged]
Re: Some Remarks on Large Language Models
#23Unfortunate, an opportunity to further enlighten others, but the author took a dismissive and antagonistic perspective.
[deleted]
Re: Some Remarks on Large Language Models
#24Earlier quoted context omitted.
I mean he is presumably Jewish and lives in Israel, so I would guess he knows quite a bit about being a minority and experiencing bias.
Jews are a minority in Israel?
We're the "majority" by virtue of internationally gerrymandered borders. In the region as a whole? Yes, we're an indigenous minority.
Re: Some Remarks on Large Language Models
#25While not unsolvable, I think the author is understating this problem a lot: > Also, let's put things in perspective: yes, it is enviromentally costly, but we aren't training that many of them, and the total cost is miniscule compared to all the other energy consumptions we humans do. Part of the reason LLMs aren't that big in the grand scheme of things is because they haven't been good enough and businesses haven't…
>Part of the reason LLMs aren't that big in the grand scheme of things is because they haven't been good enough and businesses haven't started to really adopt them. That will change, but the costs will be high because they're also extremely expensive to run. I think the author is focusing on the training costs for now, but that will likely get dwarfed by operational costs. What then?
Now maybe I'm naive somehow because I'm a machine-learning person who doesn't work on LLMs/big-ass-transformers, but uh... why do they actually have to be this large to get this level of performance?
Re: Some Remarks on Large Language Models
#26Sometimes I read text like this and really enjoy the deep insights and arguments once I filter out the emotion, attitude, or tone. And I wonder if the core of what they're trying to communicate would be better or more efficiently received if the text was more neutral or positive. E.g. you can be 'bearish' on something and point out 'limitations', or you can say 'this is where I think we are' and 'this is how I think…
Sometimes I read scientific and technical texts that are candy-coated to appeal to people who can't stand criticism, and wish the author was allowed to say what they really think.
Re: Some Remarks on Large Language Models
#27Re: Some Remarks on Large Language Models
#28While not unsolvable, I think the author is understating this problem a lot: > Also, let's put things in perspective: yes, it is enviromentally costly, but we aren't training that many of them, and the total cost is miniscule compared to all the other energy consumptions we humans do. Part of the reason LLMs aren't that big in the grand scheme of things is because they haven't been good enough and businesses haven't…
>Part of the reason LLMs aren't that big in the grand scheme of things is because they haven't been good enough and businesses haven't started to really adopt them. That will change, but the costs will be high because they're also extremely expensive to run. I think the author is focusing on the training costs for now, but that will likely get dwarfed by operational costs. What then? Now maybe I'm naive somehow becau…
Dunno! It could be the case that there just needs to be a trillion parameters to be useful enough outside of highly-constrained scenarios. But I would certainly challenge those who work on LLMs to figure out how to require far less compute for the same outcome.
Re: Some Remarks on Large Language Models
#29While not unsolvable, I think the author is understating this problem a lot: > Also, let's put things in perspective: yes, it is enviromentally costly, but we aren't training that many of them, and the total cost is miniscule compared to all the other energy consumptions we humans do. Part of the reason LLMs aren't that big in the grand scheme of things is because they haven't been good enough and businesses haven't…
I think quantization (e.g. 4-bit, https://arxiv.org/abs/2212.09720) and sparsity (e.g. SparseGPT, https://arxiv.org/abs/2301.00774) will bring down inference cost.
Edit: This isn’t handwaving btw, this is to say some fairly decent solutions are available now.
Re: Some Remarks on Large Language Models
#30I would like to see the section on "Common-yet-boring" arguments cleaned up a bit. There is a whole category of "researchers" who just spend their time criticizing LLMs with common-yet-boring arguments (Emily Bender is the best example) such as "they cost a lot to train" (uhhh have you seen how much enterprise spends on cloud for non-LLM stuff? Or seen the power consumption of an aluminum smelting plant? Or calcuated the costs of all the airplanes flying around taking tourists to vacation?)
By improving this section I think we can have a standard go-to doc to refute the common-but-boring arguments. By pre-anticipating what they say (and yes, Bender is very predictable... yuo could almost make a chatbot that predicts her) it greatly weakens their argument.