Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We have a year to fix security everywhere

jyn.dev

Post reply on HN
Print view View original

201–210 of 368 posts

Re: We have a year to fix security everywhere

#201

Post by tyrabound »

0xDEAFBEAD wrote: post #94
kennywinker wrote: post #85

I'm confused why the worry about LLMs that will answer "how do I build a pipe bomb". That information is easily available other places. The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.

There used to be a thing called "Moore's Law of Mad Science": "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those…

The irony of that law is quite amazing, yet its irony will evade those who accept such a construct.

It’s an endless, positive irony spiral.

Re: We have a year to fix security everywhere

#202

Post by shelled »

I see a lot of people focused on servers and production environments, and of course that's needed, because that's business after all — but the personal computer seems to be absent from this discourse. Not everyone can buy a spare mac studio, and they might still need to install these tools on their personal computing devices, like their personal/home laptops. At that point, it's not even about whether a Claude Code, an OpenCode, or a Pi will steal/sniff personal data, but whether it can — though I think saying "it's a matter of 'when'" might be hyperbole. As of now, it's just: keep giving access and permissions or struggle while working, or create another user, or use Docker, run inside sandbox-exec, a VM, etc. As an end user, I am really scared. Someone who has been very disciplined and vehemently privacy- and security-conscious feels the ground below has just shifted.

OEM/OSes don't seem to have woken up to it yet. A mild proof is Apple's own special folder access reporting. When you go to Privacy & Security > Files & Folders, for a certain app, "Full Disk Access" is shown greyed out and mentioned in both cases — whether you had given Full Disk Access to that app or not. This directory-level permission UX is itself broken — there's Full Disk Access, and there's Files & Folders, and Full Disk Access gets shown in Files & Folders as well. This is, for lack of a better word, such an undesirable mess.

As of now I am debating between: creating a new user and just move everything work/learning to that user. Or just run all of it inside sandbox-exec (and maybe even block it from the shell if it tries to run outside it). Or use a tool that makes the latter easier and better. I even came across such a tool here on hn few weeks ago. agent-safehouse, yet to try it.

Re: We have a year to fix security everywhere

#203

Post by zahlman »

sfn42 wrote: post #195
polishTar wrote: post #166

Earlier quoted context omitted.

I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote). We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a mal…

How do you envision a chatbot doing any of those things? You can't just program a super virus, nanobots, bombs etc. These things have to be created in the real world, you can't do it without involving a lot of humans using expensive tools and materials etc. So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged luna…

Well, the risks that mere mortals can conceive generally involve control systems for dangerous equipment (I mean, equipment that can achieve dangerous effects) being connected to the Internet while having software vulnerabilities.

Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman.

Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.

Re: We have a year to fix security everywhere

#204

Post by zahlman »

ErroneousBosh wrote: post #140
sho wrote: post #6

> On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 seconds The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.

Also, LLMs never *write* code snippets, they just pirate them from somewhere else.

Can you show me an example of a time that you prompted an LLM to provide some code, it did so, and then you were able to track down an original source for the output?

Re: We have a year to fix security everywhere

#205

Post by b112 »

mark_something wrote: post #104
0xDEAFBEAD wrote: post #94

Earlier quoted context omitted.

There used to be a thing called "Moore's Law of Mad Science": "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those…

That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".

Even a person with a sky high IQ can't destroy the world easily.

A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same.

Gene sequences can ever be ordered online.

Re: We have a year to fix security everywhere

#206

Post by qsera »

0xDEAFBEAD wrote: post #119
mark_something wrote: post #104

Earlier quoted context omitted.

That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".

[flagged]

What about people who understands that there is no AI yet?

Re: We have a year to fix security everywhere

#207

Post by Gareth321 »

0xDEAFBEAD wrote: post #94
kennywinker wrote: post #85

I'm confused why the worry about LLMs that will answer "how do I build a pipe bomb". That information is easily available other places. The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.

There used to be a thing called "Moore's Law of Mad Science": "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those…

I have always hypothesised that AI is the great filter from the Fermi paradox. Given current velocity, AI will offer us cheap and abundant energy designs in a decade. The thing with cheap and abundant energy is that it can be used for good and bad. If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world. And this is just one of many ways to destroy the world with unlimited intelligence. Bio-weapons are arguably far more scary and likely.

I have come to the conclusion that we should not allow everyone access to unlimited intelligence. Some people genuinely want to do harm. Some are not responsible enough to handle that kind of power. Arguably, no one is. This leads me to an uncomfortable conclusion: we will destroy ourselves once AI advances to a sufficient degree. The only way to prevent this is to ensure AI is aligned with our best interests, *and prevents us from destroying ourselves.* The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.

Re: We have a year to fix security everywhere

#208

Post by vbezhenar »

kennywinker wrote: post #85

I'm confused why the worry about LLMs that will answer "how do I build a pipe bomb". That information is easily available other places. The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.

Any sufficiently determined individual can buy mac mini, put it under their bed, configure outside proxy via some random internet address and prompt "iterate on websites in the CT logs, one by one, try to find vulnerabilities, if you did - encrypt their data and blackmail them for this bitcoin address". And it'll work, day and night. Abliterated GLM 5.3 is much smarter than average software developer, they know a lot about information security, they can use any available exploits, they can find novel vulnerabilities and they won't say "no". This is dangerous for an average IT system which never encountered nothing worse than some wordpress GET requests.

It's not the end of the world. But future will be rough.

Re: We have a year to fix security everywhere

#209

Post by hypfer »

zahlman wrote: post #203
sfn42 wrote: post #195

Earlier quoted context omitted.

How do you envision a chatbot doing any of those things? You can't just program a super virus, nanobots, bombs etc. These things have to be created in the real world, you can't do it without involving a lot of humans using expensive tools and materials etc. So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged luna…

Well, the risks that mere mortals can conceive generally involve control systems for dangerous equipment (I mean, equipment that can achieve dangerous effects) being connected to the Internet while having software vulnerabilities. Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman. U…

> Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last

Of course we can do that. It's not an eternal being of light existing on the astral plane, but some code executing on someone's GPU.

It stops existing once you press Ctrl + C

Re: We have a year to fix security everywhere

#210

Post by MaxMatti »

XorNot wrote: post #118
mark_something wrote: post #104

Earlier quoted context omitted.

That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".

You can't design an infectious pathogen without testing it. It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).

If the goal is just destruction then letting it free in random public places is not that hard. Especially in countries where wearing a mask is not frowned upon.
Post reply on HN