Live data from Hacker News

Andrej Karpathy: Software in the era of AI [video]

youtube.com

631–640 of 827 posts

Re: Andrej Karpathy: Software in the era of AI [video]

#631
post #369

It's going to be very interesting to see how things evolve in enterprise IT, especially but not exclusively in regulated industries. As more SaaS services are at least partly vibe coded, how are CIOs going to understand and mitigate risk? As more internal developers are using LLM-powered coding interfaces and become less clear on exactly how their resulting code works, how will that codebase be maintained and increme…

I've found that as LLMs improve, some of their bugs become increasingly slippery - I think of it as the uncanny valley of code. Put another way, when I cause bugs, they are often glaring (more typos, fewer logic mistakes). Plus, as the author it's often straightforward to debug since you already have a deep sense for how the code works - you lived through it. So far, using LLMs has downgraded my productivity. The bug…

Saw a recent talk where someone described AI as making errors, but not errors that a human would naturally make and are usually "plausible but wrong" answers. i.e. the errors that these AI's make are of a different nature than what a human would do. This is the danger - that reviews now are harder; I can't trust it as much as a person coding at present. The agent tools are a little better (Claude Code, Aider, etc) in that they can at least take build and test output but even then I've noticed it does things that are wrong but are "plausible and build fine".

I've noticed it in my day-to-day: an AI PR review is different than if I get the PR from a co-worker with different kinds of problems. Unfortunately the AI issues seem to be more of the subtle kind - the things if I'm not diligent could sneak into production code. It means reviews are more important, and I can't rely on previous experience of a co-worker and the typical quality of their PR's - every new PR is a different worker effectively.

Re: Andrej Karpathy: Software in the era of AI [video]

#632

This got me thinking about something… Isn’t an LLM basically a program that is impossible to virus scan and therefore can never be safely given access to any capable APIs? For example: I’m a nice guy and spend billions on training LLMs. They’re amazing and free and I hand out the actual models for you all to use however you want. But I’ve trained it very heavily on a specific phrase or UUID or some other activation k…

You may be interested in: https://www.anthropic.com/research/sleeper-agents-training-d... https://arxiv.org/abs/2404.13660

Re: Andrej Karpathy: Software in the era of AI [video]

#633

This got me thinking about something… Isn’t an LLM basically a program that is impossible to virus scan and therefore can never be safely given access to any capable APIs? For example: I’m a nice guy and spend billions on training LLMs. They’re amazing and free and I hand out the actual models for you all to use however you want. But I’ve trained it very heavily on a specific phrase or UUID or some other activation k…

Profit over security, outsource liability

Re: Andrej Karpathy: Software in the era of AI [video]

#634

Earlier quoted context omitted.

It’s very obvious this account was just created to promote your product…

I don't even have a product although I'd love people to work on something open source together. Also, I'm not nearly cool enough to earn a green username.

> This DevOps friction is exactly why I'm building an open-source "Firebase for LLMs."

i dont understand your earlier statement then

Re: Andrej Karpathy: Software in the era of AI [video]

#635
post #92

Earlier quoted context omitted.

Driving is not a general problem, though. Its a contextual landscape of fast-based reactions and predictions. Both are required, and done regularly by the human element. The exact nature of every reaction, and every prediction, change vastly within the context window. You need image processing just as much as you need scenario management, and they're orthoganol to each other, as one example. If you want a general tra…

It partially is. You have the specialized part of maneuvering a fast moving vehicle in physical world, trying to keep it under control at all times and never colliding with anything. Then you have the general part, which is navigating the human environment . That's lanes and traffic signs and road works and schoolbuses, that's kids on the road and badly parked trailers. Current breed of autonomous driving systems hav…

Yes, that’s exactly what I meant. I’d go even further and say the hard parts of driving are the parts where you are likely better off with a general model. And it’s not just signs, construction, police stopping traffic, etc. Even just basic navigation amongst traffic seems to require a general model of the other nearby drivers. It’s important to be able to model drivers’ intentions, and also to drive your own car in a predictable manner.

Re: Andrej Karpathy: Software in the era of AI [video]

#636

This got me thinking about something… Isn’t an LLM basically a program that is impossible to virus scan and therefore can never be safely given access to any capable APIs? For example: I’m a nice guy and spend billions on training LLMs. They’re amazing and free and I hand out the actual models for you all to use however you want. But I’ve trained it very heavily on a specific phrase or UUID or some other activation k…

This is what mechanistic interpretability studies are trying to achieve, and it's not yet realistically possible for a general case.

Re: Andrej Karpathy: Software in the era of AI [video]

#637

This got me thinking about something… Isn’t an LLM basically a program that is impossible to virus scan and therefore can never be safely given access to any capable APIs? For example: I’m a nice guy and spend billions on training LLMs. They’re amazing and free and I hand out the actual models for you all to use however you want. But I’ve trained it very heavily on a specific phrase or UUID or some other activation k…

You may be interested in: https://www.anthropic.com/research/sleeper-agents-training-d... https://arxiv.org/abs/2404.13660

Yes these look perfect! Thank you.

Re: Andrej Karpathy: Software in the era of AI [video]

#638

This got me thinking about something… Isn’t an LLM basically a program that is impossible to virus scan and therefore can never be safely given access to any capable APIs? For example: I’m a nice guy and spend billions on training LLMs. They’re amazing and free and I hand out the actual models for you all to use however you want. But I’ve trained it very heavily on a specific phrase or UUID or some other activation k…

I was just talking to somebody at work about a "Trusting Trust" style attack from LLMs. I will remain deeply suspicious of them

Re: Andrej Karpathy: Software in the era of AI [video]

#639
post #565
post #472

Earlier quoted context omitted.

[flagged]

> You think of economics like a 6 years old. If you continue to break the site guidelines, we'll end up having to ban you. If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.

He already had replies in his other multiple comments that stated the same thing.

Re: Andrej Karpathy: Software in the era of AI [video]

#640

This got me thinking about something… Isn’t an LLM basically a program that is impossible to virus scan and therefore can never be safely given access to any capable APIs? For example: I’m a nice guy and spend billions on training LLMs. They’re amazing and free and I hand out the actual models for you all to use however you want. But I’ve trained it very heavily on a specific phrase or UUID or some other activation k…

This is a good insight. There’s also a similar insight about compilers back in the days before AV.. we will have AV LLMs etc… basically reinvent everything for the new stack.
Post reply on HN