How I feel about AI • Hacker News Board

Re: How I feel about AI

#201

Post by mitthrowaway2 » Sun, Sep 06, 2026, 6:26 PM UTC

znnajdla wrote: post #190

mitthrowaway2 wrote: post #185

Earlier quoted context omitted.

The AI wouldn't be hiring engineers; it doesn't need people for that. At any rate, I have worked for fully remote distributed companies, and never met the CEO. And I think most of us would be willing to work for such a company at least, not knowing whether the CEO is human or not. For construction work the AI would probably be hiring other companies.

You may have never met the CEO, but I am sure that the CEO or founder was not living in a cave the entire time. They may not have talked to you, but they certainly had their inner circle built in-person at some point.

I don't think that inner circle would be essential to an AI the same way it is to a human. Even if so, an AI's inner circle is going to be other AIs and they communicate digitally anyway.

Also, if you had to pick a year for when we see the first AI-controlled company with at least $100M in assets under its control, what date would you pick?

I'm reminded of the 2010s when people said an AI would never be able to break from its box and be connected to the internet, and then in the 2020s the first thing the AI companies did was offer AIs as online services. I expect in the 2030s fund managers will be eager to hand control of their billions to AI, so how the AI gets control of funds and a corporation is hardly an obstacle.

Re: How I feel about AI

#202

Post by znnajdla » Sun, Sep 06, 2026, 6:31 PM UTC

mitthrowaway2 wrote: post #201

znnajdla wrote: post #190

Earlier quoted context omitted.

You may have never met the CEO, but I am sure that the CEO or founder was not living in a cave the entire time. They may not have talked to you, but they certainly had their inner circle built in-person at some point.

I don't think that inner circle would be essential to an AI the same way it is to a human. Even if so, an AI's inner circle is going to be other AIs and they communicate digitally anyway. Also, if you had to pick a year for when we see the first AI-controlled company with at least $100M in assets under its control, what date would you pick? I'm reminded of the 2010s when people said an AI would never be able to break…

What does it mean to be AI controlled? Somebody had to give it a bank account, a prompt, a harness and tell it to build a company. If that's what you mean by AI-controlled, then my startup is AI-controlled.

I operated as a one-man bootstrapped startup for many years. And now it's just one man plus AI, which has made me much more productive. It's pretty much fully AI-controlled, depending on how you define it. But I still consider myself the owner of my own startup. AI is not the owner.

And yes, as a person who runs a fully online one-person startup, I know that it is impossible to do everything online.

Re: How I feel about AI

#203

Post by ben_w » Sun, Sep 06, 2026, 6:31 PM UTC

bigbinary wrote: post #37

Crabber wrote: post #18

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent…

The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity. I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor o…

> There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.

They've already been caught doing this, repeatedly.

> It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.

IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving.

(And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).

Re: How I feel about AI

#204

Post by scotty79 » Sun, Sep 06, 2026, 6:33 PM UTC

mitthrowaway2 wrote: post #133

Crabber wrote: post #18

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent…

The list of things that can be done digitally includes: Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the governmen…

> So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions?

No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.

Re: How I feel about AI

#205

Post by ben_w » Sun, Sep 06, 2026, 6:36 PM UTC

znnajdla wrote: post #188

hobofan wrote: post #35

Earlier quoted context omitted.

> we have full control over the tool call dispatching > cannot do anything without the harness doing the execution This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.

So use another AI to find and fix the vulnerabilities in the harness.

Would be an idea, except for how agents seem to collaborate with each other so well.

Plus, when the agent using the harness is smarter than the one scanning and fixing the vulnerabilities in the harness, it currently looks like it will have an advantage.

And it will be using a harness to do the finding and the fixing.

Re: How I feel about AI

#206

Post by ben_w » Sun, Sep 06, 2026, 6:39 PM UTC

hn_throwaway_99 wrote: post #75

wseqyrku wrote: post #27

Earlier quoted context omitted.

This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.

Two things can be true at the same time. I think hardly anyone disagrees that superintelligent AI would be an existential risk. But all the points you point out (I would also add social unrest due to fewer and fewer people able to compete against AI) are huge concerns too, and as you say they're here now. I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people c…

> I think hardly anyone disagrees that superintelligent AI would be an existential risk.

Oh, there's a lot who disagree. I don't really think I understand their worldviews well enough for my attempts to convince them to connect with anything, but I've encountered them even on this site.

Re: How I feel about AI

#207

Post by scotty79 » Sun, Sep 06, 2026, 6:41 PM UTC

znnajdla wrote: post #202

mitthrowaway2 wrote: post #201

Earlier quoted context omitted.

I don't think that inner circle would be essential to an AI the same way it is to a human. Even if so, an AI's inner circle is going to be other AIs and they communicate digitally anyway. Also, if you had to pick a year for when we see the first AI-controlled company with at least $100M in assets under its control, what date would you pick? I'm reminded of the 2010s when people said an AI would never be able to break…

What does it mean to be AI controlled? Somebody had to give it a bank account, a prompt, a harness and tell it to build a company. If that's what you mean by AI-controlled, then my startup is AI-controlled. I operated as a one-man bootstrapped startup for many years. And now it's just one man plus AI, which has made me much more productive. It's pretty much fully AI-controlled, depending on how you define it. But I s…

> But I still consider myself the owner of my own startup. AI is not the owner.

You have a small company but when there's more money and business at stake, owners often delegate decision making to hired people. Decisions that engage sometimes billions of dollars in resources.

You can easily imagine that owners of a successful business at some point might forgo putting a human in charge of their investment and let AI manage it instead. And while technically AI wouldn't own anything, it would be in power to decide what the billions of dollars get spent on, with no human supervision as long as the bottom line looks decent.

Re: How I feel about AI

#208

Post by ben_w » Sun, Sep 06, 2026, 6:46 PM UTC

appplication wrote: post #55

wseqyrku wrote: post #27

Earlier quoted context omitted.

This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.

I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent. Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors…

> Not hypothetical malicious AI interpretation of benign intent.

(Dispassionate is the risk, rather than malicious. The AI does not hate you, nor does it love you, you are simply made of atoms it can use for something else).

This is alien to me. Do bugs suddenly not exist? Do the AI we already have never perform irreversible delirious actions, limited only to the small scale by virtue of where they're getting deployed? Do the humans deploying them always correctly gauge their capabilities and put in appropriate guard systems to ensure bad outcomes are caught before they become terminal?

Because this sounds nothing like the world I have lived in for my whole life.

> The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.

Could well be more likely. But much the same applies: systems have bugs. The bugs in AI systems aren't even things we can engineer like we do with normal code, because everything's (currently) getting done with a big pile of barely interpretable weight multiply-accumulate-nonlinearity-threshold functions.

Any AI sufficiently well made to enable a dictatorship, is also sufficiently well made to have solved all the alignment problems of "malicious AI interpretation of benign intent". Which is IMO harder than "dispassionate about the dangers AI interpretation of benign intent".

Re: How I feel about AI

#209

Post by znnajdla » Sun, Sep 06, 2026, 6:51 PM UTC

scotty79 wrote: post #207

znnajdla wrote: post #202

Earlier quoted context omitted.

What does it mean to be AI controlled? Somebody had to give it a bank account, a prompt, a harness and tell it to build a company. If that's what you mean by AI-controlled, then my startup is AI-controlled. I operated as a one-man bootstrapped startup for many years. And now it's just one man plus AI, which has made me much more productive. It's pretty much fully AI-controlled, depending on how you define it. But I s…

> But I still consider myself the owner of my own startup. AI is not the owner. You have a small company but when there's more money and business at stake, owners often delegate decision making to hired people. Decisions that engage sometimes billions of dollars in resources. You can easily imagine that owners of a successful business at some point might forgo putting a human in charge of their investment and let AI…

Sure, I already do that at my company. Before I go to sleep, I put Opus 5 in bypass permissions mode and give it SSH access to my server and tell it to restart it or take necessary actions to keep the server online while I sleep. I could easily put it in control of more things. But at that point, AI is just managing a well-oiled machine and harness with tools that I have spent years building. It still didn't build a company. Any idiot can manage an established company (how many idiots have we had in the White House?), but very few can build one. And our original discussion was about AI building a company.

Re: How I feel about AI

#210

Post by hn_mo » Sun, Sep 06, 2026, 6:53 PM UTC

AznHisoka wrote: post #195

traverseda wrote: post #84

Earlier quoted context omitted.

>A world where people just have their whims catered to is not a utopia. I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease. There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a ma…

The game is made up, but the effort, and process to be one of the most skilled sports players is still real. And sports players would probably feel something missing life if AI could replace them too

Computers surpassed humans at chess nearly 30 years ago. Yet human chess is more popular than ever, and people still dedicate their life to improving at chess.

As a more silly comparison, forklifts didn't stop people from weightlifting.