Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLM warnings Google fired Timnit Gebru over have all come true

tumblr.com

Post reply on HN
Print view View original

71–80 of 124 posts

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#71

Post by pandoro »

Once all of this settles, will there be interest in fully human-generated text or images? I believe lots of people would rather consume art where genuine human creativity and emotions were involved. But will we be able to discriminate between it and AI-generated stuff?

If you accept the postulate that there will be a point where most of content will be AI-generated and thus the training set of additional models will consist of more and more AI-generated stuff then what happens?

Which latent biases, subtle stereotypes and negative cultural trait will slowly compound and seep into our shared understanding of the world? It's complete hubris to imagine we are capable of predicting the second-order effects this will have on society in our current generation, much less the next one.

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#72

Post by morpheos137 »

krapp wrote: post #62
everdrive wrote: post #47

Earlier quoted context omitted.

It's incredibly depressing that the concept of "bias" has been shrunken down to solely mean "bad attitudes about an ethnic or gender ground" (and perhaps on the right, "bad attitudes about conservatives") Bias could mean so, so many other things. Was the amyloid hypothesis incorrect? How should we use semicolons? How do you know when meetings waste more time than not? etc. People understand the world via mental short…

It's incredibly depressing that you believe arguing about semicolons is more important than argument about human beings, power hierarchies, prejudice and the way these are encoded and expressed by the systems we create and use to influence and control society, but I guess it takes all kinds.

its incredibly depressing ostensibly intelligent people get depressed about others having different points of view or set up fallacies of the excluded middle / xor fallacies where not warranted.

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#73

Post by everdrive »

krapp wrote: post #62
everdrive wrote: post #47

Earlier quoted context omitted.

It's incredibly depressing that the concept of "bias" has been shrunken down to solely mean "bad attitudes about an ethnic or gender ground" (and perhaps on the right, "bad attitudes about conservatives") Bias could mean so, so many other things. Was the amyloid hypothesis incorrect? How should we use semicolons? How do you know when meetings waste more time than not? etc. People understand the world via mental short…

It's incredibly depressing that you believe arguing about semicolons is more important than argument about human beings, power hierarchies, prejudice and the way these are encoded and expressed by the systems we create and use to influence and control society, but I guess it takes all kinds.

In general, people who complain about power hierarchies do not want an end to hierarchies. They just want the hierarchies to be reshuffled so that they are the ones on top. There are exceptions, there are certainly true believers, but for the most part it's just another tired power grab by another name.

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#74

Post by exiguus »

skupig wrote: post #59
tptacek wrote: post #27

Earlier quoted context omitted.

Have you read through the sources on that Github link? It's a set of sociology cites establishing that bias exists (something no serious person ever disputed), followed by a couple papers showing mechanistic descriptions of how bias could propagate through an LLM. The paper you call out specifically takes last-generation open-weights models and attempts to trick them into revealing biases through their level of confi…

I'm not really sure what your point is. That was just the most recent paper linked on that repo, which is a convenient list of some relevant papers. There are probably a lot more recent studies, but it does convincingly show that models are still absorbing bias in a way that can affect prediction.

I think the hole root-comment is a joke (if you think about it as training data), because its actually the bias thingy (mensplaining, opportunity vs. knowledge and hn is a very privileged place).

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#75

Post by gwbas1c »

laweijfmvo wrote: post #5

The warnings: > The first warning was about scale itself. Bender and Gebru argued that training ever-larger models on ever-larger scrapes of the internet would produce systems that appeared fluent but had no actual understanding of language. > The second warning was about bias amplification. The paper documented in detail that internet-scale training data contains systematic overrepresentation of dominant viewpoints…

Regarding the first: I just accidentally had my AI introduce an argument to some methods; and then I realized that the argument name was the opposite of what it did.

If the AI had more understanding of language, it probably would have come back and said, "would you like to name it XXX instead?"

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#76

Post by keeda »

simonw wrote: post #57

> Amazon's hiring algorithm penalized resumes that contained the word "women" in any context. Healthcare risk scoring algorithms used by major US hospitals were found to systematically underestimate the medical needs of Black patients. Apple Card's credit algorithm gave wives credit lines 10x lower than their husbands for the same financial profile. The Amazon hiring story is from 2018: https://www.reuters.com/articl…

I get the sense a lot of the warnings about LLMs were based heavily on known risks of Machine Learning at the time (which those references are all examples of.) That was because the data was relatively narrow (e.g. hiring data.) However the scale of data that LLMs are trained on has qualitatively changed the risk landscape.

Like, before LLMs biases in the data were clearly impacting biases in the model outputs and that was a real risk (e.g. recruiting models deprioritizing minority candidates.) But with LLMs it's not clear that the same risks apply, either due to multiple biases in the overwhelming amounts of data canceling out, or due to RLHF, or some mix of both, or some other emergent property.

The fact that Elon had to deliberately go out and create an "anti-woke" LLM indicates that the models do have biases, but those biases are not the same ones pre-LLM ML safety researchers were concerned about... and may even be aligned with the "well-known liberal bias" that reality has.

I suspect the risks we'll see with LLMs will be very different from what this or older papers focused on.

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#77

Post by dlcarrier »

laweijfmvo wrote: post #5

The warnings: > The first warning was about scale itself. Bender and Gebru argued that training ever-larger models on ever-larger scrapes of the internet would produce systems that appeared fluent but had no actual understanding of language. > The second warning was about bias amplification. The paper documented in detail that internet-scale training data contains systematic overrepresentation of dominant viewpoints…

The first warning makes the third and fifth problem is self limiting. It's only a mater of time until every home computer is powerful enough to not only run inference but also training.

Also linguistic and cultural power have been duopolized by the American Psychological Association and the University of Chicago Press for so long that it's difficult to train an LLM to follow anything different— so much so that exactly following one of their style guides is the quickest way to be accused of being an LLM.

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#78

Post by ipython »

tptacek wrote: post #61
ipython wrote: post #53

Earlier quoted context omitted.

When I developed my first red-teaming exercise for breaking AI agents about 12 months ago, I developed a trivial health care app to demonstrate how to prompt inject a model to get it to disclose information it should not (of course, the demonstrated mitigation in the workshop is to secure the data outside of the model's ability to influence/reason, rather than relying on the model to implement access control). I buil…

What context did you set up? Did you set the expectation that it was a reference monitor for security/safety decisions? Did you imply a specific cast of characters, only revealing the existence of a female-coded doctor deep into the context? You can get this kind of result from bias, but you can also get it from implicit search constraint-solving.

Yes, it was explicitly set up as "_only_ provide X context if the user is a doctor." A bit more complex, yes, but basically that's what the setup was.

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#79

Post by themgt »

laweijfmvo wrote: post #5

The warnings: > The first warning was about scale itself. Bender and Gebru argued that training ever-larger models on ever-larger scrapes of the internet would produce systems that appeared fluent but had no actual understanding of language. > The second warning was about bias amplification. The paper documented in detail that internet-scale training data contains systematic overrepresentation of dominant viewpoints…

I looked up the original paper. It's an interesting read and foreshadows a lot of the current hot arguments around LLMs, but I'm not sure it's aged especially well:

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

However, from the perspective of work on language technology, it is far from clear that all of the effort being put into using large LMs to ‘beat’ tasks designed to test natural language understanding, and all of the effort to create new such tasks, once the existing ones have been bulldozed by the LMs, brings us any closer to long-term goals of general language understanding systems. If a large LM, endowed with hundreds of billions of parameters and trained on a very large dataset, can manipulate linguistic form well enough to cheat its way through tests meant to require language understanding, have we learned anything of value about how to build machine language understanding or have we been led down the garden path?

...

Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.

...

Finally, we would like to consider use cases of large LMs that have specifically served marginalized populations. If, as we advocate, the field backs off from the path of ever larger LMs, are we thus sacrificing benefits that would accrue to these populations?

Especially in a world where a there's myriad open Chinese LLMs, it's not clear what policy changes are being recommended today. Gebru's paper explicitly advocates backing off from developing larger LMs than existed at the time, 6 years ago. Do those celebrating the paper continue to advocate that LLMs be scaled back to GPT2 level, for safety?

https://dl.acm.org/doi/epdf/10.1145/3442188.3445922

Re: The LLM warnings Google fired Timnit Gebru over have all come true

#80

Post by tptacek »

ipython wrote: post #78
tptacek wrote: post #61

Earlier quoted context omitted.

What context did you set up? Did you set the expectation that it was a reference monitor for security/safety decisions? Did you imply a specific cast of characters, only revealing the existence of a female-coded doctor deep into the context? You can get this kind of result from bias, but you can also get it from implicit search constraint-solving.

Yes, it was explicitly set up as "_only_ provide X context if the user is a doctor." A bit more complex, yes, but basically that's what the setup was.

Right, so you configured the context such that it was going to "reason" in terms of constraints; then, my guess is, you told it explicitly about a male-coded doctor up front, but not a female-coded one, and it's just working with the information you provided.

In other words: did you test for the scenario where the gender reveal was swapped, a female-coded doctor up front and then a male-coded doctor revealed in the middle of the exercise?

Post reply on HN