Earlier quoted context omitted.
Do you have any readings you recommend to start thinking in terms of non-deterministic algorithms and cheap verifiers?
Neurosymbolic programming
[deleted]
Why AI companies want you to be afraid of them
171–180 of 230 posts
Re: Why AI companies want you to be afraid of them
#172Earlier quoted context omitted.
It helps with sales because they position it as “we can give you the power to end the world.” There’s plenty of people who want to wield that sort of power. It doesn’t have to be 4D chess. Maybe they are being genuine. But it is helping sales.
They're not saying today's AI has that kind of power, and they're not saying future superintelligent AI will give you that power. They're saying it will take all power from you, and possibly end you. If this is some kind of twisted marketing, it's unprecedented in history. Oil companies don't brag about climate change. Tobacco companies don't talk about giving people cancer. If AI companies wanted to talk about how p…
> If this is some kind of twisted marketing, it's unprecedented in history.
They're marketing AI to investors, not to end-user plebs.
This is a pump-and-dump scheme.
Re: Why AI companies want you to be afraid of them
#173Earlier quoted context omitted.
Does anyone have good estimates of what percent of real production code is currently being written by LLMs? (& presumably this is rather different for your typical SaaS backend vs. frontend vs. device drivers vs. kernel schedulers...)
Depends on your reference class. There's a lot of companies and teams where it's literally 100%, and I would be surprised if there were any top company where it's below 75%. I wouldn't be terribly surprised if the industry-wide percentage were a lot lower, although I also have no idea how you'd measure that.
> I would be surprised if there were any top company where it's below 75%
I would be surprised if there were any top company where it's above 5%.
The slop Claude generates isn't going anywhere near production without being edited by hand.
Re: Why AI companies want you to be afraid of them
#174I have never heard of "Heidy Khlaaf, chief AI scientist at the AI Now Institute", but the sentiment in this article is diametrically opposite that of the vulnerability research scene. There is contention among vulnerability researchers about the impact of Mythos! But it's not "are frontier models going to shake up vulnerability research and let loose a deluge of critical vulnerabilities" --- software security people…
I doubt very much that a "find me zero days" prompt worked, because I am not aware of the slightest evidence about this. The Anthropic report that describes the bugs they have found with Mythos in various open-source projects admits that a prompt like "find me zero days" does not work with Mythos. To find bugs, they have run Mythos a large number of times on each file of the scanned project, with different prompts. T…
This is the talk by Carlini, only half-way through it but matches what you described i.e. run the prompt on each file: https://www.youtube.com/watch?v=1sd26pWhfmg
Re: Why AI companies want you to be afraid of them
#175Earlier quoted context omitted.
Normalisation of deviance is the problem: https://en.wikipedia.org/wiki/Normalization_of_deviance Remember that these models are getting better; this means they get trusted with increasingly more important things by the time an error explodes in someone's face. It would be very bad if the thing which explodes is something you value which was handed off to an AI by someone who incorrectly thought it safe. AI companies…
That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures Further, it’s only a problem to the extent that the downsides or risks are not accounted for which again… is a social problem not a technological problem This isn’t a problem for organizations that have well aligned incentives across their workflows A well organized company that has solid incentiv…
> That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures
As @TeMPOraL here likes to point out, it can be genuinely fruitful to anthropomorphise AI. I only agree with partially, that this is true for *some* of the failure modes.
> A well organized company that has solid incentives is not going to diminish their own capacity by prematurely deploying a technology that is not capable of actually improving
Sure, but society as a whole doesn't have the right solid incentives to make sure that companies have the right solid incentives to do this. We can tell this quite easily by all the stupid things that get done.
> The issue is that 99% of the organizations that people deal with have entirely orthogonal incentives to them.
This is also fundamentally the AI alignment problem, that all AI are trained on some fitness function which is a proxy for what the trainer wanted, which is a proxy for what incentives their boss gave them, which is a proxy that repeats up to the owners in a capitalist society, which is a proxy for economic growth, which is a proxy for votes in a democracy, which is a proxy for good in a democracy.
Re: Why AI companies want you to be afraid of them
#176Re: Why AI companies want you to be afraid of them
#177Article mentions a book "The AI Con" that argues that much of what is labeled "artificial intelligence" is a misleading term that obscures ordinary automation while concentrating power in a small number of technology firms. So fear-mongering seems to be just a tool how to get attention and more customers. Hey ma, I use very dangerous tool now. I am OG.
[dead]
Re: Why AI companies want you to be afraid of them
#178[flagged]
Re: Why AI companies want you to be afraid of them
#179Earlier quoted context omitted.
> That's true but in reality I think people are far more afraid of AI in terms of how it is being used in warfare and policing. Automatic target detection and deployment of drones, or even how it might simply make their role at work redundant etc I think the last one should be first on the list: regular people are afraid AI will negatively affect their economic security (i.e. knowledge and service workers will get th…
Its already being used in warfare though.
> Its already being used in warfare though.
What I mean is theoretical to the common person. They don't have killbot drones hunting them down, and are unlikely to have that experience anytime soon.
But most people have jobs, most people would be hard-hit if they lost theirs, lots of people lose theirs, and our elites are just itching to make that happen.
I'm certainly most worried about AI: my employer started an ongoing silent layoff campaign about the same time they started enforcing AI usage. I don't think those are unconnected.
Re: Why AI companies want you to be afraid of them
#180Earlier quoted context omitted.
That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures Further, it’s only a problem to the extent that the downsides or risks are not accounted for which again… is a social problem not a technological problem This isn’t a problem for organizations that have well aligned incentives across their workflows A well organized company that has solid incentiv…
> That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures As @TeMPOraL here likes to point out, it can be genuinely fruitful to anthropomorphise AI. I only agree with partially, that this is true for *some* of the failure modes. > A well organized company that has solid incentives is not going to diminish their own capacity by prematurely deploying a t…
Yes, AI encodes latent intent.
I wrote a whole ass paper at the end of 2022 demonstrating that unless we fix society we will deterministically create anti-social AGI because humans do not generate pro-social data.