[deleted]
We read the paper that forced Timnit Gebru out of Google
11–20 of 683 posts
Re: We read the paper that forced Timnit Gebru out of Google
#12This doesn't strike me as anything that needs to be buried. The energy argument is tenuous and at this point models use relatively little compute power. The language bias is a better tack but I don't think this is earth shattering to anyone. Most of the internet is from Western sources and some fraction of that is racist/prejudiced. I think this is obvious to anyone who has ever used the internet.
Re: We read the paper that forced Timnit Gebru out of Google
#13This doesn't strike me as anything that needs to be buried. The energy argument is tenuous and at this point models use relatively little compute power. The language bias is a better tack but I don't think this is earth shattering to anyone. Most of the internet is from Western sources and some fraction of that is racist/prejudiced. I think this is obvious to anyone who has ever used the internet.
It sounds like it should be buried because it's clickbait headline generating fluff that only would get attention because of where the authors come from. And I say that after reading a summary that sounds like the authors had a positive opinion of the paper.
Google (probably) wanted to bury it because many of those clickbait headlines would have been negative to google.
Re: We read the paper that forced Timnit Gebru out of Google
#14While this is a lot of electricity and CO2 to consider, is it fair to compare them to cars? I don't think this so much means that one should get rid of training these big models but rather do their best to ensure that the data centers that these models are trained on are getting their electricity from mostly 0 emission sources. This actually seems like a doable task since you don't need to be physically near your compute cluster. I think Google and Facebook could also turn this into a PR move by putting some research money into green tech (ambiguous).
Re: We read the paper that forced Timnit Gebru out of Google
#15> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
I find these CO2 arguments from model training extremely misleading. If this is what you care about, why don't you actually try to make electricity cleaner. Realistic, that's a much more effective way to address this tiny issue, together with the grand total of 50% of humanity's CO2 emissions on top. To me it always seems like virtue signalling, showing that you "care", while not having to actually make any realistic progress. Go build nuclear power plants, or at least prevent them from shutting down, if you want you're electricity to be at 12 g CO2 / kWh rather than 500 from natural gas or 800 from oil (you can also move your servers to France or Ontario). Do not focus on tiny, 10% efficiency gains in NLP. That's a marginal gain within a miniature cause to start with.
Re: We read the paper that forced Timnit Gebru out of Google
#16I was a little disappointed in the quality of this research. I can see where Google is coming from. Some of these topics seem regurgitated (e.g. bias in online text) and some are just irrelevant (CO2 emissions). Training language models does not contribute in any meaningful way to global CO2 emissions. Overall not a very strong paper in contrast to the other reactions I've been seeing online.
Re: We read the paper that forced Timnit Gebru out of Google
#17> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
1438 pounds of CO2 is 250 liters of gasoline, or thereabouts, that's pumping gas five times. How long does a tank of gas last the average Google employee? Less than a week, considering what East Bay traffic looks like. People get paid to publish that stuff?
Re: We read the paper that forced Timnit Gebru out of Google
#18Re: We read the paper that forced Timnit Gebru out of Google
#19Is this paper on arxiv? This overview doesn't answer any critical questions. For example, it's easy to fill up 128 references and a reader shouldn't blindly trust a claim that, "The version of the paper we saw does also nod to several research efforts on reducing the size and computational costs of large language models, and on measuring the embedded bias of models." If a key part of Google's claim is that the paper…
No, the paper is not on arxiv
> [...] Though Bender asked us not to publish the paper itself because the authors didn’t want such an early draft circulating online, it gives some insight into the questions Gebru and her colleagues were raising about AI that might be causing Google concern.
Re: We read the paper that forced Timnit Gebru out of Google
#20> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
(Having not read any of this). Isn't Google committed to be net carbon neutral/negative? If so, does the claimed extra electricity usage matter?
Does OpenAI? Does the rest of the industry? Everybody is training larger and larger models, with research groups competing on who will be the first to scale up to some number of parameters.
It seems obvious that this would be something an ethicist would be interested in researching to figure out what the impact will be and start discussions about how to offset it.