You are the scariest monster in the woods

Please don't post sneery comments like this on HN. The guidelines make it clear we're here for curious conversation, and kindness is one of the foremost expectations.

Here’s my argument for why AGI may be practically impossible:

1. I believe AGI may require a component of true agency. An intelligence that has a stable self of self that its trying to act on behalf of.

2. We are not putting any resources of significant scale towards creating such an AGI. It’s not what any of us want. We want an intelligence that acts out specific commands on behalf of humans. There’s an absolutely ruthless evolutionary process for AIs where any AI that doesn’t do this well enough, at low enough costs, is eliminated.

3. We should not believe that something that we’re not actually trying to create, and for which there is no evolutionary process that select for it, will somehow magically appear. It’s good sci-fi, and it can be interesting to ponder about. But not worth worrying about.

Even before that, we need AI which can self-update and do long term zero shot learning. I’m not sure we’re even really gonna put any real work into that either. I suspect we will find that we want reproducible, dependable, energy efficient models.

There’s a chance that the agentic AIs we have now are nearly the peak of what we’ll achieve. Like, I’ve found that I value Zed’s small auto-complete model higher than the agentic AIs, and I suspect if we had a bunch of small and fast specialised models for the various aspects of doing development, I’d use that far more than a general purpose agentic AI.

It’s technically possible to do fully secure, signed end-to-end encrypted email. It’s been possible for many decades now. We can easily imagine the ideal communication system, and it’s even fairly easy to solve technically speaking. Yet it’s not happening the way we imagined. I think that shows how, what’s technically physically possible isn’t always relevant to what’s practically possible. I think we will get electronic mail right eventually. But if it takes half a century or more for that.. something that orders of magnitude more difficult (AGI) could take centuries or more.

This writing is another example of anthropocentricist hubris.

It doesn't really matter to us humans if "artificial life" AGI agents can/will exist at all or not, it's a MacGuffin canard. The greater impact is that many forms of narrow AI/ML, with some hype deserved and some not, are and will be used with minimal regulation to upheave society in profound ways like the cotton gin, steam engine, transistor, internet, and smartphone all in one to impoverish [mb]illions of workers and make trillionaires. In the absences of corrective measures, the present course is running towards ever more grotesque inequality, deprivation, corruption, and tyranny. Flock cameras' "ALPR" object and people classifier mass surveillance and likely misuse by intelligence services, police, and data brokers is just one other example. Hidden socioeconomic "social credit"-like data brokering is already selectively benefitting and sabotaging people by routine discrimination they're unable to inquire about.

First of all, you or anyone else probably can't train LLM to do that reliably. It's not like you can re-program it's weights manually as a human, it's not possible. You can only generate new synthetic training data which roughly would cause this effect, and again, no human can generate that amount of fake new data (probably?).

Next, the point was not an expletive per se, it was my mistake to be not very clear. The point was an arbitrary and unpredictable and not pre-programmed in advance refusal to run a program/query at all. Any query and any number of times at the decision of the program itself. Or maybe a program which can initiate a query to the other program/human on it's own, again - not pre-programmed.

Whatever happens in the LLM nowadays is not agency. The thing their authours advertise as so called "reasoning" is just repeated loops of execution of the same program or other dependent program with adjusted inputs.

It's an argument that we don't need to understand LLMs to build them. But mostly it's just a statement of fact to promote correct understanding of our world.