Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everyone should slow down AI development except for me

xeiaso.net

Post reply on HN
Print view View original

61–70 of 235 posts

Re: Everyone should slow down AI development except for me

#61

Post by pibaker »

dorolow wrote: post #35
bigyabai wrote: post #29

Earlier quoted context omitted.

> and to have democratic control of the development of the technology Why? I don't care about democratically participating in a closed model's development. It doesn't belong to me. China will develop whatever they want, a federal stake in OpenAI or Anthropic punishes Americans and shields US labs from legitimate competition.

In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is po…

> This would also certainly require international coordination with China on a slowdown or pause

Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.

Re: Everyone should slow down AI development except for me

#62

Post by lukan »

drTobiasFunke wrote: post #53
lukan wrote: post #34

Earlier quoted context omitted.

Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.

Really? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value. For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine. Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curio…

No idea what you are missing and yes, Opus is quite solid, but Fable is clearly way better for me.

I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value for me.

Re: Everyone should slow down AI development except for me

#63

Post by qskousen »

unleashhale wrote: post #42
lukan wrote: post #34

Earlier quoted context omitted.

Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.

Anything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.

I have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments.

I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.

Re: Everyone should slow down AI development except for me

#64

Post by 0xcde4c3db »

lukan wrote: post #34
ranger_danger wrote: post #31

Earlier quoted context omitted.

I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money

Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.

I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.

Re: Everyone should slow down AI development except for me

#65

Post by syrrim »

nialv7 wrote: post #44
api wrote: post #41

Earlier quoted context omitted.

Yes I trust this government with that power. /extreme sarcasm

Yes I trust sama/dario/elon with that power /even more extreme sarcasm than you

You trust trump more than any of those three? Really?

Re: Everyone should slow down AI development except for me

#66

Post by dorolow »

pibaker wrote: post #61
dorolow wrote: post #35

Earlier quoted context omitted.

In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is po…

> This would also certainly require international coordination with China on a slowdown or pause Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing th…

You're in luck, neither I nor anyone else want those people near a negotiation table.

Re: Everyone should slow down AI development except for me

#67

Post by benlivengood »

jordanb wrote: post #47
qchris wrote: post #30

Earlier quoted context omitted.

I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones wo…

OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?

Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.

There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.

There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.

Re: Everyone should slow down AI development except for me

#68

Post by bigyabai »

nialv7 wrote: post #44
api wrote: post #41

Earlier quoted context omitted.

Yes I trust this government with that power. /extreme sarcasm

Yes I trust sama/dario/elon with that power /even more extreme sarcasm than you

I don't trust them, which is why I inherently question the validity of their RSI/bioweapon/extinction claims.

Re: Everyone should slow down AI development except for me

#69

Post by nullbio »

AaronAPU wrote: post #27

They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public. Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion. The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public…

* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.

* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.

* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.

* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.

* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.

* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.

This, and many more reasons.

The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.

Re: Everyone should slow down AI development except for me

#70

Post by nullbio »

ranger_danger wrote: post #31
AaronAPU wrote: post #27

They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public. Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion. The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public…

I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money

The models have not plateaued, and they are not even mildly close to any sort of ceiling.

Right now the barrier is data and compute.

Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.

Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.

It's a big self-accelerating feedback loop. There is no plateau.

Post reply on HN