Can we get some insight into why Dario seems more scared now than before, do we think he’s being transparent?
We must pace the frontier
661–670 of 943 posts
Re: We must pace the frontier
#662Earlier quoted context omitted.
I don't think Anthropic and OpenAI will IPO at all. Word is that Anthropic will have $100 BN in revenues this year, and very likely OpenAI will get some similar amount. You IPO when you need money, and I think Anthropic and OpenAI are past that point.
That level of revenue is apparently not enough for them to feel confident in the stability of their competitive position.
And how does an IPO address that?
Re: We must pace the frontier
#663Earlier quoted context omitted.
I think the point is even stronger. alignment is really not very well defined, nor are the mechanisms used to provide it. from what I understand it's more of a statistically imposed bias than an impenetrable gate. personally I would like to flip this around, all of this is a consequence of some really extreme notions about investment. its a tail-wagging-the-dog. the investment community is apparently in love with the…
Probabilistic safety is too low a bar. We need to project the model outputs onto a safe subspace; lobotomize them, if you will. It may make them dumber, but that's a fine price to pay. I don't work in this space so I don't know the latest, but here's an example: Provably safe systems: the only path to controllable AGI ( https://news.ycombinator.com/item?id=37619285 )
I don't think the neuro-symbolic model is going to dumb anything down at all. Cyc didn't end up producing anything useful, but the only reason an llm is able to do math at all is that it can keep throwing stuff at lean all day. personally I think that that synthesis will turn out to be more powerful than just llm with constraints, but I don't really work work in the field either.
Re: We must pace the frontier
#664Earlier quoted context omitted.
I'd recommend reading the full post :) Specifically: AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't…
No proponents of RSI state they will be operating outside of reality. Said another way, they will operate within the confines of what's possible and still be RSI. I'm quite surprised this is something that needs to be clarified. You are constructing a straw man of your own making.
Well, right now we have ex Anthropic employees telling the media that their terabyte sized models can possibly copy themselves onto the internet and run elsewhere as if the necessary computing resources are ubiquitous.
Plus, "we must pace the frontier" implies that the argument is that the frontier is moving too fast, but if RSI can't move faster than the rest of reality and the models needed for RSI are already nearing the limits of current human reality, RSI can't move much faster than we can improve reality.
Re: We must pace the frontier
#665I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…
> RSI
For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."
Re: We must pace the frontier
#666I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…
Ok so Anthropic CEO will self-own themselves and surrender to the deepseek/kimi/glm models. Yet they are IPOing later this year. Interesting times.
they just said no ipo this year, most chinese models are distilled from claude anyway
Re: We must pace the frontier
#667I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…
Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work. Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial effort…
Just a reality check. They can absolutely do it.
Altman or Dario will donate some money to Trump, get cozy with the Department of War and make up a bullshit excuse why Open Source needs to be eliminated.
All access to this technology will be gatekept by a bunch of nasty people who want to eliminate your job as a knowledge worker and put everything behind a subscription.
Re: We must pace the frontier
#668I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…
Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources? The companies doing these things without following common sense security measures are the felony generators.
I question these "felonies" as well. For decades and decades these billion dollar corporations have been criminally negligent. Why worry about security? Just rush to market. Move fast and break things. Make billions. What does it matter if the code is insecure? Security doesn't pay bills, so nobody cares.
AI is merely exploiting their gross negligence and imprudence, and I think it's long overdue. If anyone should be liable for this, it's all of these corporations who released insecure systems to the masses and profited enormously from them.
Re: We must pace the frontier
#669I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that…
The whole point of technology is to replace work that we don't want to do. You are attacking the chief reason why people use technology in the first place. Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.
You should probably look closer at unemployment numbers in IT vs other spaces. Gains elsewhere are making up for IT losses, but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles in large numbers.
Re: We must pace the frontier
#670I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs…
> RSI For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."
"Recursive self-improvement"- models making better models