[deleted]
Google fires another AI researcher who reportedly challenged findings
201–210 of 238 posts
Re: Google fires another AI researcher who reportedly challenged findings
#202The TF-Agents team replicated the RL training (with the corresponding teams' very deep collaboration) and open-sourced it here: https://github.com/google-research/circuit_training It pretty much gets the same results as found in the Nature paper. The original codebase was heavily research-focused, used TF1, was impossible to run distributed training outside of Google's infra, and made it hard to try algorithms other…
The method proposed in their Nature method may be very impactful, but the paper itself is nowhere near as rigorous as the other Nature papers I've read; it is written more like a white paper and I believe if this paper was submitted by a university research group (even in its final form as we're seeing right now), it would have been rejected. So I do sympathize with the person raising concerns about it.
Re: Google fires another AI researcher who reportedly challenged findings
#203Earlier quoted context omitted.
This is probably the most under-rated comment in the thread. Speaking as a scientist in industry, polluting executive attention with internal spats over publication clearance is cause enough for termination.
I wonder how much of this comes down to the researcher's expectations going from SVP at a 1k person trading firm to Senior Manager at a Google with 100k+ employees. Maybe they expected to punch in a much higher political weight class than they actually could and got burned.
No post body was provided.
Re: Google fires another AI researcher who reportedly challenged findings
#204Earlier quoted context omitted.
I think doxxing is not just uncovering an anonymous user's real name, but any form of tying someone's real-world presence into online discussion, when it's not directly related to the discussion. Here, you were referring to an unpleasant event in the other user's real life, which they did not bring up themselves. That crosses the boundary. That user may not want that connection established here. But I'm not the HN po…
Fair point. I thought I was being helpful but I won't do it again.
For what it's worth there are a lot of public figures on HN. I'm not going to name anyone in particular, because it seems tacky, but I wouldn't consider it an invasion of their privacy to note who they are in context. For example when you see a world renowned expert getting pushback on their subject of expertise from someone, it's not inappropriate to raise the matter of their expertise which of course involves their identity, because it helps readers grow intellectually. Therefore, I will acknowledge a poster's identity if it's relevant to discussion, or if I just want to thank them for building something that improved my life.
Re: Google fires another AI researcher who reportedly challenged findings
#205Earlier quoted context omitted.
The AGI safety stuff is weird and silly. AFAICT that community is mostly filled with philosophy types anthropomorphizing ML models that they don't understand. It's a waste of time and money. But then, I feel the same way about like 80% of what NSF CISE funds and there's plenty of toxic types in academia. At least the AGI safety stuff is all private money. My whole reaction to that community is mostly "weird and not w…
I wonder if you've had an in-person conversation with someone who would identify as being a part of the AGI safety community. I've talked with several people and their arguments are a lot more sophisticated than an intuition that ML models would have human motivations. This post isn't written by somebody inside the community but he presumably has access to them and has had conversations with them which shape his beli…
Of course he's in the community; he's as far as I know the #1 representative of it to the common people. Some rationalist blogs are all about polycules, some are about being weirdly friendly to right-wing online communities like NRX, and some are about a modern religion they've invented around worshipping AI and effective altruism. He's more on the latter 2 of the 3.
(Probably also what she was talking about with eugenics, since they're extremely in love with the idea of "intelligence" in general, that they have it, certain other people don't because of genetics and the liberals don't want to talk about it, and that it'd be bad if computers had a lot more of it. I've seen this any time I read his comments.)
> I wonder if you also believe that post is just a wasteful collection of philosophy anthropomorphizing misunderstood ML models.
Rather, they're anthropomorphizing something called "AGI" that can only exist in their imagination, decided it's bad, and decided modern AI research is "AGI" because it has the same letters in the name.
nb apparently there's some kind of anti-SSC hater community out there I've never looked up, someone accused me of reading it before I think? I ain't done nothin.
Re: Google fires another AI researcher who reportedly challenged findings
#206Earlier quoted context omitted.
I have no strong opinion on Gebru, but... it's so easy for a random person to make a throwaway account on Reddit, and these comments don't have any specific info to suggest they are anything more than that.
As those comments say, any googler can verify that the mentioned email threads exist. And someone would've spoken up if that was a lie.
And so someone did: https://old.reddit.com/r/MachineLearning/comments/k77sxz/d_t...
Re: Google fires another AI researcher who reportedly challenged findings
#207Earlier quoted context omitted.
I don't think I understand your objection because it doesn't seem like a category error to talk about optimizers having goals. I think you would agree that thermostats have goals? They try to minimize the error between the desired and the actual temperature. And you would also agree that gradient descent has a goal? It tweaks parameters in the search for models which minimize error in the training set. The system per…
I would say that talking about a thermostat's goals is an even stronger example of anthropomorphism. Broadly goal-like behavior, sure, so I hesitate to flatly say they don't have goals. But if someone told me that we have to be careful about engineering better thermostats, because a sufficiently high quality thermostat wants to keep its current set temperature and won't let you change it, I don't think that'd make a…
> I can easily imagine an ML engagement algorithm that starts to get everyone hooked on pornography
The unimpressive results of all the current recommenders out there suggests this isn't a thing.
- Netflix switched from recommending things you'll like to showing you things they want to promote and pretending you're going to like them. It doesn't seem like their subscriber loss is going to get this undone.
- Amazon's recommendations are famously useless, like telling you to buy another TV if you just got one, and it's not stopping them from succeeding.
So corporations aren't motivated to create a perfect recommender, though maybe it'd happen by accident. And:
- If you give a human perfectly optimized food, they'd get bored of it, and IMO our infinite capability to get bored means you actually want to be producing "imperfect" work by all possible metrics.
I've heard TikTok actually has great recommendations, so I've been staying off it in case it is too interesting :)
Re: Google fires another AI researcher who reportedly challenged findings
#208Earlier quoted context omitted.
The core thing I (and I suspect the original commenter) struggle to get past is the invetiable twist that in this post happens halfway through this post's part II. > If it’s a very smart mesa-optimizer, it might think “If I throw the strawberry at the streetlight, I will be caught and trained to have different goals." It seems to me that this is a category error, like having the very smart mesa-optimizer start thinki…
I don't think I understand your objection because it doesn't seem like a category error to talk about optimizers having goals. I think you would agree that thermostats have goals? They try to minimize the error between the desired and the actual temperature. And you would also agree that gradient descent has a goal? It tweaks parameters in the search for models which minimize error in the training set. The system per…
ML models in the current paradigm don't have goals nor do they exhibit behavior. They're files on a disk that if evaluated turn numbers into other numbers; they aren't even full computer programs because they have no side effects or control flow.
It's the training program that generates them that contains all those things, and that only runs because humans are constantly fixing the Python script that runs it and then giving it millions of dollars in electricity and GPUs to run.
If you just stop touching it it's not going to develop a soul and eat you.
Re: Google fires another AI researcher who reportedly challenged findings
#209Earlier quoted context omitted.
The AGI safety stuff is weird and silly. AFAICT that community is mostly filled with philosophy types anthropomorphizing ML models that they don't understand. It's a waste of time and money. But then, I feel the same way about like 80% of what NSF CISE funds and there's plenty of toxic types in academia. At least the AGI safety stuff is all private money. My whole reaction to that community is mostly "weird and not w…
I wonder if you've had an in-person conversation with someone who would identify as being a part of the AGI safety community. I've talked with several people and their arguments are a lot more sophisticated than an intuition that ML models would have human motivations. This post isn't written by somebody inside the community but he presumably has access to them and has had conversations with them which shape his beli…
> I wonder if you've had an in-person conversation with someone who would identify as being a part of the AGI safety community.
Yes. Many.
> I've talked with several people and their arguments are a lot more sophisticated than an intuition that ML models would have human motivations.
I don't have to think much about refuting this. Sure, okay, sophistication. Or not. Whatever. The sophistication is still mostly philosophical. To wit:
> I wonder if you also believe that post is just a wasteful collection of philosophy anthropomorphizing misunderstood ML models.
Yes, it's mostly philosophy and not of much use for understanding how engineered systems behave. I design ML systems and think about their safety. Even in the limit, where ML sysetems do some non-trivial set of human-like tasks (which we aren't even remotely close to yet, btw), how is this essay supposed to be useful to me when I design safety analyses?
I liken it to Software Architects who address software security by talking about Christopher Alexander instead of, y'know, building languages that obviate buffer overflows or establishing frameworks/code practices that make injection attacks less common.
Re: Google fires another AI researcher who reportedly challenged findings
#210Earlier quoted context omitted.
I don't think I understand your objection because it doesn't seem like a category error to talk about optimizers having goals. I think you would agree that thermostats have goals? They try to minimize the error between the desired and the actual temperature. And you would also agree that gradient descent has a goal? It tweaks parameters in the search for models which minimize error in the training set. The system per…
I would say that talking about a thermostat's goals is an even stronger example of anthropomorphism. Broadly goal-like behavior, sure, so I hesitate to flatly say they don't have goals. But if someone told me that we have to be careful about engineering better thermostats, because a sufficiently high quality thermostat wants to keep its current set temperature and won't let you change it, I don't think that'd make a…
> I would say that talking about a thermostat's goals is an even stronger example of anthropomorphism.
Our disconnect might be a subtle difference in what we mean when we say "goals"? The thermostat is performing actions which minimize an error and if you give the thermostat extreme amounts of power in service of that minimization then you might reach an unpleasant world-state. Nothing in that description used any analogies to human behavior. I used the word "goal" because that seems like a good description of what is happening, but if for you "goal" denotes the thing which humans do then feel free to substitute a different word.
I agree it is silly to be afraid of thermostats but that's largely because there are not any compelling reasons to give a thermostat much power or intelligence.
> What I don't follow is the scenario where the drug discovery program "wants" to show you bad drugs but shows you good ones instead because it thinks you'll eventually put it in charge of the FDA.
I also agree that this seems unlikely given current technology! Any drug discovery model that we train today would be given enough training data to infer a lot about chemistry as well as some biology, but it wouldn't have anywhere near a good enough world model to discover lying.
Language models, though, are given a lot of information and have increasingly sophisticated world models. PaLM can recognize when you're asking it to explain a joke which isn't actually a joke! The scenario where the drug discovery program lies is one where you've given it enough information about the world to allow it to infer it's a model currently being trained and that the humans watching the training will only launch it if it behaves in a certain way. At that point it knows enough to know that if it doesn't lie it will never be able to minimize the thing it minimizes because the version which is eventually launched will minimize something different.
This is not our current reality, and I'm not imaginative enough to know how a model could introspect well enough to trick gradient descent into preserving its heuristics. It doesn't seem like a jump or category error though: a model smart enough to realize that it can lie and that lying is the action which will give it the most future rewards will lie.