Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We found a division by zero bug in FFmpeg with a vibecoded fuzzer

code.ffmpeg.org

Post reply on HN
Print view View original

71–80 of 274 posts

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#71

Post by black_knight »

astrange wrote: post #67
black_knight wrote: post #59

Earlier quoted context omitted.

This is where I believe strong typing (like, Haskell-strong or stronger) and functional programming in general will be a win. The confidence I have that my fixes are localised when fixing Haskell code is infinitely stronger than fixing even Java, not speak about C, code.

Haskell's type system would not easily prevent this bug. It's not good at numeric/logic issues like that. When people say "Haskell makes it impossible to write bugs" they mean "Haskell has enums" (ADTs).

I am not claiming you cant write buggy code in Haskell! But following good functional style, your bug will more likely be compartmentalised, and fixing it will not break some other part of your program.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#72

Post by TheGoddessInari »

fouronnes3 wrote: post #64
black_knight wrote: post #59

Earlier quoted context omitted.

This is where I believe strong typing (like, Haskell-strong or stronger) and functional programming in general will be a win. The confidence I have that my fixes are localised when fixing Haskell code is infinitely stronger than fixing even Java, not speak about C, code.

What's stronger than Haskell?

Lean 4, Idris 2.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#74

Post by bilalq »

stefan_ wrote: post #50
harambae wrote: post #40

Earlier quoted context omitted.

It's mostly (not entirely, but mostly) finding security issues in old human-written code. It'll eventually start running out of those. From that standpoint, it's not a crazy setup security-wise. Maybe still crazy for development.

You can point AI at any AI produced code and ask it to review it, get back 10 bullet points and a few pages of prose. And the fun part is, you can do that over and over and over again!

This happens all the time. Yesterday, I ran into an especially egregious case.

I had Fable add a new subcommand to our internal CLI tool. I reviewed and tested it locally and had to suggest several fixes that I feel like I wouldn't have had to tell a human senior engineer to do. When it finally submitted the PR, I had it on a loop waiting a few minutes for comments on the PR, then assessing/addressing/replying-to/resolving them, and then repeating again until all AI reviewers were okay with it. It ended up going through dozens of revisions and ended up with 160 comments left on the PR.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#75

Post by ghaslt »

astrange wrote: post #68
fouronnes3 wrote: post #64

Earlier quoted context omitted.

What's stronger than Haskell?

Anything with ranged numeric types. Like everyone's favorite functional programming language, Ada.

This issue raises SIGFPE. Ada would raise Constraint_error, which is easier to catch than a signal, but still occurs at runtime.

You need range proofs to be 100% safe, and then you can as well use the regular type because invalid values will not occur.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#76

Post by MaxBarraclough »

ks2048 wrote: post #24

No doubt fuzzers (vibecoded or otherwise) can be powerful, but can't you just mark all "/" as potential divide by zero errors? I guess sometimes developers think they "know" some variable won't be zero, but unless it checked explicitly or by the compiler, that shouldn't be trusted.

If it's possible for program execution with some particular input to lead to a divide-by-zero, that's a bug, especially if the program is expected to be able to handle malformed inputs, or perhaps even deliberately malicious ones. It's not trivial to determine whether a program does this correctly. If it was, program analysis would be easy.

Division can 'go wrong' for certain inputs, but it's not just division. In C, signed integer addition, subtraction, and multiplication, all give undefined behaviour on overflow.

As 'Someone' already pointed out, it's not helpful to just flag all uses of the division operator, or of other potentially dangerous operators. Minimising false positives is one of the core challenges of program analysis.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#77

Post by _jackdk_ »

astrange wrote: post #67
black_knight wrote: post #59

Earlier quoted context omitted.

This is where I believe strong typing (like, Haskell-strong or stronger) and functional programming in general will be a win. The confidence I have that my fixes are localised when fixing Haskell code is infinitely stronger than fixing even Java, not speak about C, code.

Haskell's type system would not easily prevent this bug. It's not good at numeric/logic issues like that. When people say "Haskell makes it impossible to write bugs" they mean "Haskell has enums" (ADTs).

Liquid Haskell might require you to prove that the divisor is nonzero, but even in standard Haskell there's common idioms for ensuring that a list is non-empty (data NonEmpty a = a :| [a]) or that text is non-empty (newtype NonEmptyText = NonEmptyText Text, with non-exported constructor, helpers like make :: Text -> NonEmptyText, or more advanced tricks like https://exploring-better-ways.bellroy.com/haskell-koan-type-... ).

The big problem preventing this approach from working for numbers is that it's just so cumbersome there. Most of this is because all the arithmetic operators are bundled into a single Num typeclass, and `fromInteger :: Num a => Integer -> a` has a type that's impossible for a "non-zero number" wrapper to satisfy.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#78

Post by simonjuk »

dabinat wrote: post #2

It’s interesting how AI may both raise and lower the quality of software. It’s very easy to send an AI agent on an open-ended bug hunt, and if it wastes a bunch of time and effort and finds nothing, no big deal. Time is much more important for a human developer with a salary.

In my experience, there are two ways to use AI: speed or quality. Speed is where you give the AI a task to do and you review it; quality is where you write the code yourself and you get AI to review it. Both are valid for different situations.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#79

Post by adamddev1 »

nonethewiser wrote: post #34
dmix wrote: post #4

Earlier quoted context omitted.

Finding the bugs with LLMs is easy. Reviewing the output, cleaning it up, and making sure it doesn't break something else is the hard part.

No one can keep up with the volume of code AI produces. We wont stop using AI. We will use AI to check AI. Of course this is crazy, but it will also unlock pretty insane scaling and productivity and ultimately we will manage it on either end via requirements and tests.

> it will also unlock pretty insane scaling and productivity

Insane scaling of bloat, bugs, and technical debt I'd say.

> We will manage it on either end via requirements and tests

It is so crazy that this is being touted as a sane strategy. When I was a much worse programmer, I tried to write a big complicated string manipulation function to take two types of scripts in a language and add diacritics. I had the requirements very clear. I had the tests very clearly with all the edge cases. But I didn't have a good and clear picture of how to attack the problem which was quite novel for me. As I got closer to passing all the tests it got exponentially more unruly and confusing. And nearing the end I was frantically changing little bits here and there wincing and praying and hoping the tests would pass. "Please work! Come on!" Then when I got close enough, I could never ever think about touching that mess again.

I was a below average programmer then throwing myself at some novel problem I didn't understand. Throwing LLMs that produce below average code at novel problems and relying on tests and requirements is not where we want to go to make real progress.

(Years later after much learning and coding myself I was able to redo the function in a totally different way. This time I actually understood how to attack the strange problem and made something clean, clear, and robust that just worked. The tests then become a secondary guardrail, not the main force of correction.)

We are seeing such a massive regression from what we've learned over the years of CS.

Re: We found a division by zero bug in FFmpeg with a vibecoded fuzzer

#80

Post by shiandow »

adamddev1 wrote: post #79
nonethewiser wrote: post #34

Earlier quoted context omitted.

No one can keep up with the volume of code AI produces. We wont stop using AI. We will use AI to check AI. Of course this is crazy, but it will also unlock pretty insane scaling and productivity and ultimately we will manage it on either end via requirements and tests.

> it will also unlock pretty insane scaling and productivity Insane scaling of bloat, bugs, and technical debt I'd say. > We will manage it on either end via requirements and tests It is so crazy that this is being touted as a sane strategy. When I was a much worse programmer, I tried to write a big complicated string manipulation function to take two types of scripts in a language and add diacritics. I had the requi…

I think all code is technical debt in a way. Good code is a necessary evil, bad code is more evil than necessary.

Generating code automatically when you're not even quite sure what it is or even should be doing is insanity.

Post reply on HN