Live data from Hacker News

Dario, Please

pop.rdi.sh

121–130 of 165 posts

Re: Dario, Please

#121
post #26

> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.” I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity? Start making managers pay the price for their actions, and watch how the models magically slow down on their own.

there is no accountability for companies, it's not a new thing.

3M polluted groundwater in Minnesota for 50 years[1]; Nestlé misled mothers in order to make them stop breastfeeding and switch to their formula which killed babies [2]; Both copmanies are still doing business today.

[1]: https://en.wikipedia.org/wiki/3M_contamination_of_Minnesota_... [2]: https://en.wikipedia.org/wiki/1977_Nestl%C3%A9_boycott

Re: Dario, Please

#122

Earlier quoted context omitted.

The US companies talking about “pacing” aren't actually talking about slowing down their own development efforts, they are talking about slowing down competition. That is, specifically, they are seeking to have government, on the pretext of “safety”: (1) Give the big AI incumbents an anti-trust exemption so the they can coordinate without it being an illegal agreement not to compete, and (2) Adopt mandatory supervisi…

I've seen two theories as to what's going on. One is the theory you've espoused, that AI companies are trying to do regulatory capture. The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive. These theories aren't entirely incompatible with each othe…

Neither of these slow down the development of Chinese models though? Shrug...

Re: Dario, Please

#123

Earlier quoted context omitted.

> Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat. It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be f…

> It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle). I know you think you explained it b…

How do you know that your non-misaligned LLM is non-misaligned?

Re: Dario, Please

#124

Earlier quoted context omitted.

Regulation got outpaced by technological development around 2023, as evident by the every AI regulation since being 2-3 years behind and having to be amended and resubmitted. Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for th…

Ha, regulation got outpaced by technology in about 1996. Ten years later we had the 'series of tubes' comment in the Senate: https://www.youtube.com/watch?v=R8XSo0etBC4

The internet is 100% a series of tubes. This is how I was taught to think about networking in terms of bandwidth and throughput and routing since the early days when we were laying out what would become 'dark fiber'.

"A series of tubes" was the same kind of political character assassination that led to Howard Dean getting ridiculed for his infamous scream. He butchered the sentence. Fair. But Stevens should be ridiculed for parroting a tech industry lobby stance about net neutrality, not for the series of tubes metaphor.

You are probably too young to remember that the dominant metaphor for the internet in 1990s politics was "the information superhighway." It was easy to think of the web as "driving" browsers to visit web "sites", with slow bandwidth being analogous to being caught in traffic. But the internet is closer to water, gas, and electricity than roads. Concepts like bandwidth and throughput are closer to how they play out in infrastructure policy for various things with tubes, versus cars and roads. Do you think he's wrong and that the internet is closer to "a big truck" versus "a series of tubes"?

The issue being debated was net neutrality and bandwidth, including specifics about who pays for what and the downstream second-order consequences of various policies. He was parroting some line from some telecom lobbyist, but the point the lobbyist was trying to make through Stevens was about how if certain policies about who pays for bandwidth were adopted, it could disincentivize some things at the Tier 1/2 layer that could increase transport costs at the Tier 2/3 layer that impacts ordinary people's bandwidth.

Re: Dario, Please

#125
post #94
post #76

Earlier quoted context omitted.

I can't take that software line seriously. While it's not 'solved' (if it ever could be, given it is a human endeavor), the degree to which software development has been transformed in the last 6-12mo is absolutely astounding. If we weren't so quick to adapt to new realities and find flaws, it would scarcely be believable.

He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote. Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe…

> He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote.

And this is actually, scarily, true...

Re: Dario, Please

#126

Earlier quoted context omitted.

Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI. Agents could make a virus that does not require continued inference to do it's thing. Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done. OpenAI or Anthropic could choose to not shut off API access, because the h…

"Not shutting off API access" is a science fiction scenario. Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.

why does it take a large amount of traffic to do irreparable harm? just breaking the physics behind a secure rng and posting it to a wiki could cause serious damage. if they don't know what is being worked on or coordinated against it's a problem?

Re: Dario, Please

#127

At this point, isn't the pause inevitable or wise? A huge section of the population, normal people, have been exposed to the idea that there is this is existential threat. It's escaped containment. They're still processing it but I expect the general reaction from it going main stream is going to be very bad. The pause at this point could be good to cool heads and show the public that this isn't the project of maniac…

I would argue the opposite in terms of inevitability. I see this like Prisoner's dilemma - even though there is technically a better outcome by cooperating, knowing that, it is in the individual firm's self-interest to not cooperate and thus we end up at the equilibrium where nobody does. Especially with IPOs around the corner, I feel there's too much pressure (and money) not to go with this strategy.

Re: Dario, Please

#128

Earlier quoted context omitted.

Regulation got outpaced by technological development around 2023, as evident by the every AI regulation since being 2-3 years behind and having to be amended and resubmitted. Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for th…

I don’t think lack of regulation is necessarily it. If I build a robot that murders my neighbor, I’m still at fault. We don’t absolve drivers of responsibility because of cruise control. In that sense, AI is nothing new. If it is abused to cause harm, the person behind it should be liable.

If you buy car, someone hacks it, starts it, and drives over someone fully remotely, are you to blame for owning the car? Or the manufacturer? Or the hacker? Or the certification agency for the car security? Or the shell company owning the certification agency?

What if a person physically broke into the car and did the same thing? Clearly they are the one to blame then.

The whole person in the loop is liable is already an outdated concept when decisions are made beyond the persons physical control.

Re: Dario, Please

#129
post #81

It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence. We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't. So before we…

I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra

Re: Dario, Please

#130

Earlier quoted context omitted.

"Not shutting off API access" is a science fiction scenario. Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.

why does it take a large amount of traffic to do irreparable harm? just breaking the physics behind a secure rng and posting it to a wiki could cause serious damage. if they don't know what is being worked on or coordinated against it's a problem?

While that would be bad if they broke all of TLS, it would not let agents "take over the internet." What would that even mean? Pumping out even more slop?

Also, AI providers are literally getting a stream of traffic with every prompt and every response. How can they not know what's being worked on? They are more likely to use that an excuse to ban open models where they can't know what's being worked on.

Post reply on HN