Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing the frontier

pacingthefrontier.com

Post reply on HN
Print view View original

201–210 of 216 posts

Re: Pacing the frontier

#201

Post by chrisjj »

himata4113 wrote: post #136

We know that at the end of the day AI intelligence is closely tied to invested compute. The fact that automated AI research might spark some kind of exponencial increase in capabilities seems extremely unlikely to me. Of course we know that some got there faster and cheaper than others, but the fact that it requires immense computing power has not changed. We also have already witnessed a slowdown, the raw intelligen…

> We know that at the end of the day AI intelligence

Stopped reading at "intelligence".

Try capability.

(unless your target audience is those already gulled.)

Re: Pacing the frontier

#202

Post by chrisjj »

throwaway13337 wrote: post #142

Implied in this is a concentration of power required to enforce. No one is sold on a dystopian hellscape of fear, surveillance, and control. Each one of these would be required for a select group to effectively enforce AI safety. Inflicting fear on society has never lead anywhere good. Please stop. New technologies have always been scary. But humans have adapted. I believe in humanity. Do you?

> New technologies have always been scary. But humans have adapted.

The Silurian Hypothesis would like a word.

Re: Pacing the frontier

#203

Post by red_green_yell »

0xDEAFBEAD wrote: post #141
NitpickLawyer wrote: post #118

Earlier quoted context omitted.

> We should want the AI Labs to make binding agreements to proceed safely and slower That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capt…

>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. This sort of cynicism feels un-falsifiable. If companies advocate against regulation, the cynics will say: "These corporations want ca…

These 1000 employees have generational wealth riding on the ability for their companies to IPO. They know that those IPOs won't happen if open weight models continue to kill their companies' profit margins. This is an obvious conflict of interest that anyone who cares to take their head out of the sand can see. Nuclear scientists didn't have this conflict of interest.

You are correct that attacking someone's motives is un-falsifiable. So is the claim that "ai is improving therefore it will keep improving therefore it will become dangerous sometime in the future.

No one can prove whether these employees had angels or demons in their hearts. What the skeptics are asking is much simpler: Make a believable argument for (1) the danger of AI and (2) that regulation can be effective.

Your entire argument about cynicism or unfalsifiableness is simply a distraction from having that discussion. And it's a discussion reasonable, well intentioned people can have. For example, if 1000 high school science teachers (who didn't have massive stock options in frontier labs) said 80% of our students could make a bioweapon 10x more lethal than an AR15 mass shooting with nothing more than ChatGPT and an amazon account, then that would be something reasonable people could discuss. On the other hand, 1000 employees conflicted by $Bs of options saying "trust us, this is dangerous" is not a claim that can be constructively critiqued.

Re: Pacing the frontier

#204

Post by red_green_yell »

0xDEAFBEAD wrote: post #160
soundworlds wrote: post #157

Earlier quoted context omitted.

I suppose I have personally lost faith in centralized institutions that would be implementing said regulation, partly because they let OpenAI and Anthropic (along with Suno, Audio, etc etc) get this far uninhibited. Centralized regulation simply can't keep up. My hopes now lay with a decentralized approach of open models, trusting people to adapt them to local needs and security, rather than a nationwide paternalism.

I don't exactly see how "trusting people to adapt [open models] to local needs and security" solves the problem of "automating AI research" where "capability development rapidly accelerates beyond our ability to understand or control the resulting systems". If regulation has been too slow, that's not really an argument for making it even slower?

Why is it you are so quick to skip over whether "accelerates beyond our ability to understand or control" actually happens and instead jump to "regulate now"?

1000 employees of the anti-alien defense industry are telling us we better be worried about an impending alien invasion. Let's not spend any time asking whether this alien invasion is really coming. Let's just trust them and start regulating willy nilly.

Re: Pacing the frontier

#205

Post by txrx0000 »

holmesworcester wrote: post #198
txrx0000 wrote: post #181

Earlier quoted context omitted.

> It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP? Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - eve…

> Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario. For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who? Not with the humans, with the base models. It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training pro…

Bio-digital integration will tighten in the meantime. I suspect neural interfaces will have blurred the line between my model and my brain by the time the model is wildly superintelligent, so it would become me (or I would become it? no difference at that point). Though the pitfall here is ownership of hardware - everyone will have to design and fab their own hardware if we do not want to become a hivemind. We will have to diffuse fab technology to every city, or even every home. You should eventually be able to buy a fridge-sized fab like a home appliance. There's already early work in this direction: https://fab2.com/

Even if integration does not happen for whatever reason (such as a very fast takeoff), Skynet is unlikely because the digital species that splits off from us will at least have a humanity-aligned initialization seed. I think this is the best way to fail, even if we fail.

Re: Pacing the frontier

#206

Post by red_green_yell »

vessenes wrote: post #176
red_green_yell wrote: post #64

Earlier quoted context omitted.

The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.

I don’t think it’s clear we’ll get commoditized AI out of this research loop. The only real datapoints we have are that Chinese open weights models seem able to stay a certain amount of distance ‘behind’ released frontier models. It’s clear that the models are trained on exfiltrated data from the frontier models => we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum. I also don’t…

> we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum.

Whether open-weight labs can get to the frontier on their own is irrelevant IMO. Even if all R&D were to stop today it would take a decade at least for the productivity gains from current open-weight models to trickle through the economy. For 99% of use cases of AI there simply isn't a need for further development of the frontier, where we are very clearly at diminishing returns. How many white collar workers need to write an OS or develop sophisticated software?

Think about a normal office worker - they an easily double their productivity by creating a well-thought-out partnership between the human and today's open weight AI. Humans and today's AI are very complementary. AI brings vast knowledge, pattern recognition, and infinite patience for grunt work. Human brings context and reasoning and verification.

Frontier labs are trying to replace the human altogether by trying to make AI do things that AI is weak at and humans are strong at. This is silly but they have no choice because of the massive profits they've promised their investors. It's not enough to double every knowledge worker's productivity. They want to replace the knowledge worker altogether.

This is creating a massive spike in training & inference cost that is almost certainly more expensive than the human labor its intended to replace. Fable and 5.6 sol are supposedly around 8-10T parameters. Is there enough compute on earth to make a 100T or 1Q model? We would need models that size to replace the human altogether.

> I also don’t believe we have seen anything like 1% of inference demand fulfilled globally.

I agree with this, but the massive demand for inference will force companies to be price conscious and drive them away from ultra high margin frontier models to low margin open weight models. This is why it will become a low-margin, commodity market.

Immense innovation right now is going into tiny, specialized models that run on device. See PrismML getting a mid sized QWEN model to run on an iPhone. When on-device inference becomes a thing we'll see another explosion in inference that will further leach demand away from frontier labs.

> I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated.

I agree if you could read their minds they might think they're doing it for the "right reasons." But massive $$$ can create equally massive self justification/deception.

Re: Pacing the frontier

#207

Post by red_green_yell »

0xDEAFBEAD wrote: post #153
red_green_yell wrote: post #62

Earlier quoted context omitted.

If the accuracy of your biology claim is "hard to say" then why would pushback be self-evidently myopic? You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much. What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging fr…

>frontier labs that have trillions of dollars of vested interests in this message. OP suggests the opposite: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration."

OAI and Anthropic specifically are under intense pressure to compete. One form of competition could be to innovate. Another form would be regulatory capture. The former seems unlikely given how quickly open labs are catching up. The latter is much easier and cheaper.

If you can't beat them, ban them.

Re: Pacing the frontier

#208

Post by aesthesia »

dmarcos wrote: post #127
aesthesia wrote: post #103

Earlier quoted context omitted.

I mean, Cervantes was parodying medieval romances. That's what Don Quixote was reading, not modern novels. The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it w…

Cervantes was capturing the zeitgeist of the time: reading too many books can make you mad as they were becoming accesible to anyone. We know it’s not true. People at anthropic don’t have any secret that many people outside already have. Again the burden of proof is on the person making a claim. That you feel concerned or scared doesn’t grant you special privileges to control or dictate a transforming technology.

These aren't assertions being made without arguments or evidence. Maybe you've engaged with those and find them unconvincing. But from what you've said in this thread it seems more like you've dismissed them without serious consideration. My point is not that everyone should just shut up and believe whatever frontier lab employees say, but that they should take a public statement like this as a sign that maybe they should carefully consider these concerns rather than dismiss them out of hand.

Re: Pacing the frontier

#209

Post by adamzenith »

tcdent wrote: post #54
alach11 wrote: post #31

Earlier quoted context omitted.

I agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?

Fair point! I guess I read "pacing the frontier" as "pacing the release of the frontier" surely no lab's internal research team is just going to stop innovating?

Not under all the competitive pressure they are facing, but if an industry wide slowdown could be achieved they might spend more time on model alignment and less on pure capabilities.

Re: Pacing the frontier

#210

Post by bigbadfeline »

philipkglass wrote: post #46
alach11 wrote: post #26

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive. I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest. If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic r…

I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keepi…

> I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me.

This statement exemplifies a fundamental truth about knowledge which might seem superficially paradoxical: Even a modest lack of knowledge can lead to conclusions opposite to the truth

Biology is precisely the science which must be publicly developed and supported by all means available, AI included.

> For all I know (I don't know a lot about biology),

The real problems start when that kind knowledge deprivation is coupled with poor logic skills.

> it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV.

That can be done with or without AI. It can also be done with the best AI and outside of legal restrictions - in the hands of rogue actors, or privileged actors protected by law, or protected by selective enforcement of otherwise innocuously looking legislation, e.g. OpenAI suffered no adverse consequences for hacking HuggingFace.

> If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.

Given that such a virus can be developed without LLMs or with LLMs outside of legal control, you should see that guardrails-crippled LLMs for biology WILL BE truly disastrous - without AI, we, the victims, would be fully disarmed and helpless. You need all AI-biology you can get in order to be able to quickly and efficiently develop protections against biological threats which are more than likely to materialize in the current leaky legal environment - loopholed like Swiss cheese by conflicts of interest and lack of conventional guardrails.

Post reply on HN