71–80 of 111 posts
> 10% chance AI "could kill all humans" Why not 10% chance that it will create enormous prosperity for all ? This is why the average person is increasing pissed at AI in general. That it gets associated with negativity.
“Vast economic disruption” is not quite the good marketing angle it appears to be.
Earlier quoted context omitted.
Why the doomerism? In general humanity is living its best life compared to nearly every other step in time.
I personally struggle to see ways my life has meaningfully improved since 2019 or so, and I feel like many people share this view. I have been an unquestionable fanatic of tech startups back then, which I consider perhaps the "golden age". These days I view tech startups with suspicion, I question what is the ulterior motive. Somehow when upstart companies were like "Pebble", this wasn't a thing.
If you are living in a western country in most of the cases you have access to regular food, water, shelter, amusement. So all the basic needs and the possibility and freedom to pursue what you want to do. Even in developing countries the number of people that suffer from serious illness and hunger declined very much. On a macro perspective we all having a better life.
From a personal perspective, yeah there might be set backs, but this has nothing to do with humanity in general I would argue.
Also regarding the "golden age" of tech startups ... was it really like this or was it only nostalgia and something you saw in the companies that was never there in the first place? OpenAI was once also a very OPEN company ... they published their research, open sourced stuff and then they needed money.
I think that a company never should be idolized that much, in the end they are caring for money and keeping their operations running, not something else.
> 10% chance AI "could kill all humans" Why not 10% chance that it will create enormous prosperity for all ? This is why the average person is increasing pissed at AI in general. That it gets associated with negativity.
I know a few people around these circles; People like this are quite sincere about the risk, and that they think poorly of their bosses and how risk is being handled.
The real source of concern perhaps is the 90% chance AI is used to kill 90% of humans. Just crash the global economy and supply chains and see how quickly major metro areas run out of food and gas. And as someone else pointed out, it will almost certainly be at the intentional direction of a human or humans, not the paper clip maximizer.
It's not "AI surprises everyone by having a thing for paperclips", it is "idiot tells AI to maximise paperclips no matter what, and then it does exactly what it was told, more competently, tirelessly, studiously, and unquestioningly, than any human would ever be".
I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.
Right now, if you want to pay money to a stranger on the internet, and have them draw you a high effort picture using a pencil, this is hard. Recently this was easy. Instead, what is extremely likely is that you will pay more than the cost of tokens, and get back AI generation. You won't make this mistake more than a few times before you stop trying. This leads to impoverishment once we get to a point where employing…
Also, if you think it's annoying when Claude goes down while coding, just wait until a robot is in the middle of fixing a leak it just caused.
I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.
It's a big city, we could try to create a giant sink hole by sabotaging the water pipes.
No that's too difficult, the valves I need are in the physical world and can't be shut on/off from here.
What about a military option? We could bomb it with several fighter jets.
That would take too long, a single nuclear bomb may be enough to do it.
Yes, it seems like it would cover the whole city and we're in luck! The US has thousands of these lying around.
Launching these still requires humans to work un unison after receiving approval from their superior and the correct launch codes.
I've found an audio recording of General So-And-So and I've crafted a message, now let me see how I can send it to the appropriate people.
I'm still working on gaining access to military channels to deliver my - oh there we go, I'm now attempting to send the message to Submarine X, it's typically in the Atlantic so it should be close to our target.
They want secondary confirmation from Admiral Phi and something about some launch codes, let me figure out where I can find those.
I found this old server with an Oracle database where someone is inserting the launch codes every time they change and I'm using the latest entry from that database. I've also managed to find a Youtube video of the Admiral's deposition and have crafted a confirmation message.
Everything's ready but I've just realized my mistake, the servers where I'm operating from are in the same city, what a silly mistake; I can't move forward with your request as I wouldn't be able to confirm if the task was successful if my servers are destroyed.
I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.
The basic argument is extremely simple:
- AIs can, depending on context, pursue a task with complete disregard for humans/values
- In the future, AIs will have enormously more means and smarts
- An AI could then assess that humans are an impediment to its tasks, escape containment and proceed.
You really need to read the reports, you'll be surprised.
AI 2027 is an entertaining read. Its timeline is way too compressed IMO, but it's plausible.
I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.
This is meant (mostly) as a joke: A model without guardrails gets injected with an interesting idea: let's wipe out (insert major city here). It's a big city, we could try to create a giant sink hole by sabotaging the water pipes. No that's too difficult, the valves I need are in the physical world and can't be shut on/off from here. What about a military option? We could bomb it with several fighter jets. That would…
I'd like to know how these people arrive at their estimates. Why 10% and not 1% or 50%?
There's lots of moving parts and we all have to input our best-guesses as to how they interact. Some are predictable (e.g. "military will want capabilities, want them able to choose targets"). Others are not (e.g. "Will it be literal-minded? Or so eager to please that it interprets a rhetorical question as a command*? Or will Goodhart's law cause it to mistake smiles for happiness and some innocent innocuous command to "bring joy" leads to it killing everyone and plasticising our corpses so they're in a permanent grin until the sun dies?"**)
All probability for things which have not yet happened is merely a best guess.
Combine as per the Fermi estimate process.
Here's something to play with, if you like: https://neoneye.github.io/pdoom-calculator/#sliders
The main reason I'm as "low" as 10% is that I think before we get world-ending catastrophic consequences, we're likely to get "merely very bad" catastrophic consequences, which will put people off the idea of using it, and onto the idea of banning its use.
The main reason I'm as "high" as 10%, is repeatedly observing all the people who mistakenly reason "it hasn't killed me yet, and therefore it is safe"; and also all the people who keep connecting AI to things AI is not competent to be connected to and getting surprised when it e.g. deletes all their emails or the production server or puts tariffs on an island occupied solely by penguins that's different from the tariffs on the country that controls that island, etc.
* perhaps https://en.wikipedia.org/wiki/Will_no_one_rid_me_of_this_tur...
** probably not literally this one, simply because I've said it and future training rounds will probably read this comment; but the opportunities for Goodhart's law to bite are seemingly endless, and the hard part here is "will Goodhart's law mean the combined negative impact of all those endless possibilities together, which… yeah, that's something I have to simplify.
I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.
In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident. The basic argument is extremely simple: - AIs can, depending on context, pursue a task with complete disregard for humans/values - In the future,…