Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We have a year to fix security everywhere

jyn.dev

Post reply on HN
Print view View original

211–220 of 370 posts

Re: We have a year to fix security everywhere

#211

Post by Gareth321 »

grenoire wrote: post #107
0xDEAFBEAD wrote: post #94

Earlier quoted context omitted.

There used to be a thing called "Moore's Law of Mad Science": "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those…

People reading into this may be thinking of single-person IQs, but it's probably measured in the millions.

Exactly. People cannot comprehend an IQ in the millions. But it's far crazier than this. Imagine millions of "people", each with IQs in the millions, all acting in unison at the speed of electrons. None of them can die. They all learn and improve indefinitely at incalculable speed.

Re: We have a year to fix security everywhere

#212

Post by vbezhenar »

Gareth321 wrote: post #207
0xDEAFBEAD wrote: post #94

Earlier quoted context omitted.

There used to be a thing called "Moore's Law of Mad Science": "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those…

I have always hypothesised that AI is the great filter from the Fermi paradox. Given current velocity, AI will offer us cheap and abundant energy designs in a decade. The thing with cheap and abundant energy is that it can be used for good and bad. If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world. And this is ju…

Your implication is not horrifying. Your implication is paradise. It's already horrifying enough to live in a world, where Putin and Trump can destroy our civilization with one button.

I'm not that optimistic, though. Either people will control AI; or people will be destroyed by AI. I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.

Re: We have a year to fix security everywhere

#213

Post by zahlman »

dgl wrote: post #26
simonw wrote: post #5

I don't think we even have a year. The current batch of LLMs are ferociously good at identifying vulnerabilities.

Even if they are good the vulnerabilities have to be there. There's lots of things turning up like Local Privilege Escalations (LPE) in Linux, but serious people didn't expect the kernel to be a boundary for a sophisticated attacker. A lot of the vulnerabilities LLMs are finding now are the "long tail" and affect only particular configurations, I would be surprised if e.g. a widely applicable RCE is found in Linux (b…

> but serious people didn't expect the kernel to be a boundary for a sophisticated attacker.

I think it has more to do with what's on each side of the boundary in practice, a la https://xkcd.com/1200/ .

Re: We have a year to fix security everywhere

#214

Post by FeepingCreature »

Planktonne wrote: post #137
forgotusername6 wrote: post #131

Earlier quoted context omitted.

The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.

This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.

"If unlikely thing happens, then an unrelated miracle will also happen" is not actually how logic works.

Re: We have a year to fix security everywhere

#215

Post by Planktonne »

0xDEAFBEAD wrote: post #168
Planktonne wrote: post #137

Earlier quoted context omitted.

This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.

>This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. By shutting down open-weight models? Why not just do it now then?

You don't need to shut down open-weight models to avoid SkyNet; it's not relevant at all.

Re: We have a year to fix security everywhere

#216

Post by Gareth321 »

sho wrote: post #14
simonw wrote: post #7

Earlier quoted context omitted.

The difference is memory bandwidth. The M5 Ultra that's coming out on 22nd September can do 1,200GB/s. The M5 Max you can buy today only has 614GB/s.

So, that gets us to about where nVidia was with Ampere in 2020. Let's hope the M7 catches us up with at least Hopper.

While true, the news here is the size of the unified RAM. Nvidia only exceeded 256GB RAM in the 2025 B300 - 288GB. The B300 alone (without the baseboard/PSU/chassis/wiring/CPUs/system RAM/etc) is at least 700% more expensive. This enables large language models on consumer hardware. 1200GB/s is plenty for many tasks.

Re: We have a year to fix security everywhere

#217

Post by vbezhenar »

sfn42 wrote: post #195
polishTar wrote: post #166

Earlier quoted context omitted.

I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote). We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a mal…

How do you envision a chatbot doing any of those things? You can't just program a super virus, nanobots, bombs etc. These things have to be created in the real world, you can't do it without involving a lot of humans using expensive tools and materials etc. So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged luna…

https://www.the-odin.com/crispr-kit/ CRISPR Bacteria Gene Editing Kit $129.00

Re: We have a year to fix security everywhere

#219

Post by FeepingCreature »

bamboozled wrote: post #158
olmo23 wrote: post #148

Earlier quoted context omitted.

This is because to build a pipe bomb you need difficult to source materials. This is not the case for other types of threats (cyber / bio). I personally have no need for an LLM which will readily explain how to cut up the genotype of smallpox into small chunks which can pass the screening at the bio-labs, and can be readily assembled into the real thing by a second year lab-student.

Ignorant question but won’t there be much smarter teams if people using LLMs to workout how to mitigate these threats. It seems like more of a problem if only a few people have access.

It seems like there would be a massive attacker bias in multiple ways. Defenders need consent, attacker does not. Defenders have to work with the human body, attackers only have to break it. Defenders have to stick to the law, which may prevent them from releasing anything at all, attackers do not. And so on. I would not surprised if the attacker's task is a hundred times easier here.

Re: We have a year to fix security everywhere

#220

Post by hnlmorg »

tokioyoyo wrote: post #115
hnlmorg wrote: post #70

Earlier quoted context omitted.

> The reason is simple - nothing really bad has happened that we can point at and say "ah, shit, let's all learn collectively". I know it sounds naive when I say it, but there hasn't been a significantly consequential hack, leak, destruction, or anything related to cybersecurity where it led for concerns of people. How consequential does a hack need to be? Troy has collected literally billions of stolen credentials.…

> Tens of millions of people have been directly compromised by ransomware (likely higher because that’s just the cases we know of) and you hear about state-sponsored hacks in the news all the time. With no consequences. Everyone just churns along. It might be detrimental to the business a little bit, but from my personal experience, there's more effort in creating DR processes, rather than preventing an attack, explo…

Like I said, even in actual engineering companies will take shortcuts. And then what happens is the government has to step in. But there’s no appetite for government involvement in the tech sector in the US. And everyone moans when the EU does.
Post reply on HN