We read the paper that forced Timnit Gebru out of Google
technologyreview.com
We read the paper that forced Timnit Gebru out of Google
1–10 of 683 posts
Re: We read the paper that forced Timnit Gebru out of Google
#2It's nice to finally get to see some of the content of the paper and it's awesome that Bender was willing to step up and give some context to world about what the hell it was about.
Re: We read the paper that forced Timnit Gebru out of Google
#3> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate
So... they're saying it used about $100 worth of electricity.
[ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ]
[ https://www.statista.com/statistics/190680/us-industrial-con... ]
Re: We read the paper that forced Timnit Gebru out of Google
#4> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
(Having not read any of this). Isn't Google committed to be net carbon neutral/negative? If so, does the claimed extra electricity usage matter?
Re: We read the paper that forced Timnit Gebru out of Google
#5Is this paper on arxiv? This overview doesn't answer any critical questions. For example, it's easy to fill up 128 references and a reader shouldn't blindly trust a claim that, "The version of the paper we saw does also nod to several research efforts on reducing the size and computational costs of large language models, and on measuring the embedded bias of models."
If a key part of Google's claim is that the paper omits relevant research, an author should have simply posted their 128 references and openly asked what work was missing. This whole saga could be easily solved instead of being dragged out for clicks.
Re: We read the paper that forced Timnit Gebru out of Google
#6> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
(Having not read any of this). Isn't Google committed to be net carbon neutral/negative? If so, does the claimed extra electricity usage matter?
Re: We read the paper that forced Timnit Gebru out of Google
#7> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
(Having not read any of this). Isn't Google committed to be net carbon neutral/negative? If so, does the claimed extra electricity usage matter?
The paper should have been probably focused on the title only, because the CO2 usage looks like a publicity stunt. She also didn't subtract the amount of CO2 saved by improved productivity by getting better search results.
Re: We read the paper that forced Timnit Gebru out of Google
#8> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
There are models much larger than BERT with a much larger footprint, GPT-3 being the most well kniwn example.
Models like BERT aren't just trained once either when they are developed, but trained again with different domains, different parameters, different tasks in some cases. There is also fine-tuning (more frequent, less carbon intendive), so these are real environmental problems, and others have pointed them out.
Re: We read the paper that forced Timnit Gebru out of Google
#9> A version of Google’s language model, BERT, which underpins the company’s search engine, produced 1,438 pounds of CO2 equivalent in Strubell’s estimate So... they're saying it used about $100 worth of electricity. [ https://www.eia.gov/tools/faqs/faq.php?id=74&t=11 ] [ https://www.statista.com/statistics/190680/us-industrial-con... ]
There are models much larger than BERT with a much larger footprint, GPT-3 being the most well kniwn example. Models like BERT aren't just trained once either when they are developed, but trained again with different domains, different parameters, different tasks in some cases. There is also fine-tuning (more frequent, less carbon intendive), so these are real environmental problems, and others have pointed them out.
Google claims they neutralized their legacy emissions over the entire history of the company.
Now about the billion cars on the road and the 40% of the world’s electricity being generated from coal.
Re: We read the paper that forced Timnit Gebru out of Google
#10Are these numbers the energy to train a model? The whole point of these new NLP models is transfer learning, meaning you train the big model once and fine-tune it for each use case with a lot less training data.
5 cars worth of carbon emissions is not a lot given that it is a fixed cost. Very few are retraining BERT from scratch.
EDIT:
The other two points are also disingenuous.
* "[AI models] will also fail to capture the language and the norms of countries and peoples that have less access to the internet and thus a smaller linguistic footprint online. "
NLP in "low resource" languages is a major area of research, especially because that's where the "next billion users" are for Big Tech. Facebook especially is financially motivated to solve machine translation to/from such languages. https://ai.facebook.com/blog/recent-advances-in-low-resource...
* "Not as much effort goes into working on AI models that might achieve understanding, or that achieve good results with smaller, more carefully curated datasets (and thus also use less energy)."
This is also a major area of research. Achieving understanding falls under the purview of AGI, which itself carries ethical and safety concerns. There are certainly research groups working toward this. And reducing parameter sizes of big networks like GPT-3 is the next big race. See https://news.ycombinator.com/item?id=24704952