AI Labs Urged to Pump the Brakes in Open Letter

For a more detailed list of reasons, go read what AI alignment scientists think, since they have been working on how to align an AI in order to not "turn everything into paperclips" since the 1970s and it seems as though many are rather skeptical about us having any future if we keep up at this rate (spoiler: many believe the end of humanity will occur soon after the creation of any superintelligence): https://www.alignmentforum.org

My personal takeaway: continued development might mean the end of humans and maybe even our planet (if an ASI can deploy nanotech to convert everything into substrate). As it already stands, nobody knows why LLMs work as well as they do - they are already a black box. Sure, plenty of people can explain the math behind each of steps involved: training, matrices, transformers, inference, et al. but it's still a big black box which spits out "magic" answers. You can't just drop a breakpoint inside a model during inference to see what's going on, you will just get a long list of unintelligible floating point numbers at any step of the process.

Your question about regulation is valid... but something needs to be done. I feel like we're standing very close to one of the Big Filters.

Comparing AI/AGI/ASI to anything we have currently seen is probably pointless, they are worlds apart. Would you bother comparing a smartphone to a book? The rate of AI/ML progress is sufficient to measure in hours instead of months or years.

> "implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts"

Who deserves to be called an expert on this? Feels like Economics or something, where you have camps of thought advocating for themselves but little way of knowing who's right. Best to break things and develop antibodies whilst the stakes are still low.

In hindsight. Back in 2018 it wasn’t at all obvious they would be in the position they are today. Google released the Transformer paper just a year before and were commonly seen as being in the lead.

Part of the ethical consideration needs to be the opportunity cost a six month delay could cause.

As an example, let's say that GPT-N will cure cancer.

Over a six month period that's at least 5 million people dead if the date it arrives and cures cancers broadly was pushed by that long.

What about negotiating foreign policy treaties to prevent war, or identifying a way to reverse climate changes, or any number of other positive effects?

The fact the letter even positions 'should' such an advanced AI exist as a legitimate question I find pretty gross.

Should we hold back the progress of intelligent life in the universe out of the ego of humanity?

I get superintelligence is an unsettling idea.

But inherent to that name is an indication that there's a seat at the debate table that's currently vacant which might have important and interesting things to say on the subject, and anyone suggesting aborting it is a valid course of action (particularly given the myriad of existential threats we already face from fellow humans) I can only regard as being quite far from super.

And while I don't love OpenAI becoming increasingly closed, they really have done an excellent job with alignment that I very much doubt some of the signers of the letter would have.

So I'd much rather have the progress in AI continue to be led by a company pursuing caution than by another company that used those 6 months to catch up and take the lead with even less caution.

This is very much "out of the frying pan and into the fire" territory targeted at trying to stall OpenAI for competitive reasons and getting every outspoken person against AI for years in academia to jump onto signing it.