Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

cookieplmonster.github.io

Post reply on HN
Print view View original

301–310 of 315 posts

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#301

Post by smcl »

tialaramex wrote: post #294
smcl wrote: post #260

Earlier quoted context omitted.

I think the response to that would be: yes but the game would simply not have been made if it wasn't written in C++. That's not to say you couldn't or that you can't make something like GTA:SA in Rust in 2025 or in a safer different language in the early 2000s. It just would take a great deal more time and expense as you'd have needed to construct a lot of tooling and do a lot of training to ensure all of the employe…

We don't have enough information to claim it's the "right decision" only that this choice did work, not that other choices couldn't have been better. In video games you can go back and try another option but life isn't like that and so we can only suppose what might have happened.

Well what happened was that despite being based on an aging Renderware engine and programmed using a language with many potential footguns, the game was stable enough across multiple platforms, architectures and OSes that it was both a critical and commercial success.

I know what you’re saying - you can’t really know what might have been in an alternate reality. But in that alternate reality they’d have had to come up with something truly monumental to outdo themselves here.

I think you’re just being a wee bit picky about me using the words “the right decision”. If we’re honest with ourselves there probably wasn’t a Rust-like language in the conversation when they set out to build GTA3, Vice City or San Andreas so this is all kind of moot unless we're suggesting that Rockstar should have started out by building that language...

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#302

Post by LegionMammal978 »

AssKoala wrote: post #287
LegionMammal978 wrote: post #196

Earlier quoted context omitted.

From GP: > (with checks for things like this enabled) You can (and could) easily compile an optimized build with debug symbols to track down sources of issues, but catching a bug like this would likely take a dynamic checker like Valgrind or MSan, which do not allow for any optimizations if you want to avoid false negatives, and add even more overhead on top of that. (Valgrind with its full processor-level virtualiza…

This was a PS2 game and codebase. MSan didn’t exist at the time and valgrind doesn’t work on a ps2. Neither of those are necessary to find this bug as it could be found using a stomp allocator if you’re a developer on the project at the time.

How could a stomp allocator have possibly found this bug? The offending values are stored on the stack, in-bounds when written to, and again in-bounds when read from.

At no point is there an OOB access, just a failure to initialize stack variables. And to catch that, you'd need either MSan-style shadow state that didn't exist, thorough playtesting with fine-grained stack randomization, or some sort of poisoning that I don't think existed.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#303

Post by hoten »

zerd wrote: post #251
hoten wrote: post #165

Earlier quoted context omitted.

for real. The author clearly never heard of fmod

fmod takes in the order of 30+ cycles, probably more in year 2003 CPUs, vs 1 for cmp, 1 for sub, 1 for jmp.

Sure the lower bound is nicer here. But when the tradeoff includes an unlimited upper bound it's not a very attractive option.

I guess the most robust code handling both performance and unexpected input would be one iteration of this (leveraging the assumption that angles are either always within the bounds, or had one frame of going out of bounds by a small amount); followed by a fmod if that assumption is just totally off.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#304

Post by zerd »

ajross wrote: post #267
zerd wrote: post #249

Earlier quoted context omitted.

I tried to use Valgrind to catch pretty much this exact bug 20 years ago, and it was nigh impossible. If you call any 3rd party code it'll have flag tens of thousands of false positives that you have to sift through. And that was on a small game engine, I can't imagine running it on millions of lines of code.

Again, you don't valgrind a whole game. You valgrind your unit tests. Even in 2004 when this game was released, and even in the game industry, unit testing was a routine thing. And this particular bug was in code very amenable to lightweight unit testing.

That would be assuming they knew there was a bug in that particular part of the code, which they probably didn't, until Windows 11 24H2. And unfortunately Valgrind doesn't work on windows.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#305

Post by hoten »

masklinn wrote: post #272
nemothekid wrote: post #172

Earlier quoted context omitted.

I want to assume that the GTA developers did this hack because it was faster than floating point division on the Playstation 2 or something. But knowing they were able to they were able to blow up loading GTA5 by 5 minutes by just parsing json with sscanf, I don't have much hope.

They were not the only one to make that mistake e.g. rapidjson had to fix the same error, few people expect parsing one token out of sscanf to strlen the entire input (not only that but there are c++ APIs which call sscanf under the hood). The second error of deduplicating values by linear scanning an array was way more egregious.

The real, systemic error is that dozens(?) of engineers worked on that product, supposedly often testing the online component and experiencing that wait time first hand; and none thought "wait, parsing JSON doesn't take that long, computers are fast! what's going on?"

I think someone estimated that error cost them millions in revenue? I'm pretty sure a fraction of that could afford an engineer who knows how fast computers ought to be.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#306

Post by masklinn »

hoten wrote: post #305
masklinn wrote: post #272

Earlier quoted context omitted.

They were not the only one to make that mistake e.g. rapidjson had to fix the same error, few people expect parsing one token out of sscanf to strlen the entire input (not only that but there are c++ APIs which call sscanf under the hood). The second error of deduplicating values by linear scanning an array was way more egregious.

The real, systemic error is that dozens(?) of engineers worked on that product, supposedly often testing the online component and experiencing that wait time first hand; and none thought "wait, parsing JSON doesn't take that long, computers are fast! what's going on?" I think someone estimated that error cost them millions in revenue? I'm pretty sure a fraction of that could afford an engineer who knows how fast comp…

GTA was never my wheelhouse, but from what I gathered GTA Online didn't have that much support, and since it was only the initial loading time, and it would have increased over time as the shop content increased, and a very fast machine (e.g. a dev machine) would have had less of an issue, the engineers working on it were probably not that incentivised to dig into it.

Like, even though it's pretty critical to initial user experience initial loading time is generally what gets disregarded the most.

> I'm pretty sure a fraction of that could afford an engineer who knows how fast computers ought to be.

It can, if someone cares enough or realises it's an issue, and then someone is motivated enough to dig into it, or has the time to.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#307

Post by db48x »

smj-edison wrote: post #293
db48x wrote: post #242

Earlier quoted context omitted.

There are proposed cpu architectures that work that way, like the Mill https://millcomputing.com/ >. Where most cpus support multiple calling conventions the Mill enforces a single calling convention in hardware. There is a hardware `call` instruction that does all the work directly, along with a corresponding `ret` instruction for returning from a function call. It also uses its equivalent of the TLB to ensure that…

This is really interesting—how do stack references work in this design?

Technically I think you can read the “whole” stack; it’s only reads off of the ends of the stack as a whole that are prevented. However, note that the start of your current stack may not really be the start of the real stack.

Consider the case of a system call, such as `read`. You’re in user space and you have some stack frames on the stack as usual. You allocate a buffer on the stack (there’s a cpu instruction for that; it basically just extends your “turf¹” to include more of the stack page, and zeros it as mentioned) to hold the data you want to read. You then call `read` with the `call` instruction, including the address of the buffer and the buffer size as arguments. So far everything is very straight–forward.

But `read` is actually in a different protection domain; it’s part of the kernel. The CPU uses metadata previously set up by the kernel to turn this into a “portal call”. After the portal call your thread will be given a different protection domain. In principle this is the kernel’s protection domain, but in reality the kernel might split that up in many complicated ways. What is relevant here is that the turf of this protection domain has been modified to include this new stack frame. From the perspective of `read`, the stack has just started; there are no prior frames. The reality is that this stack frame is still part of the stack of the caller, it’s only the turf that has changed. Those prior stack frames still exist, but they are unreadable. Worse, the buffer is also unreadable; it’s located at an address that is not part of the kernel’s turf.

So obviously there needs to be another set of instructions for modifying turfs. The full set of obvious modifications are available, but the relevant one here is a temporary grant of read and/or write permissions to a function you are about to call. You would insert a `pass` instruction to pass along access to the buffer for the duration of the call. This access is automatically revoked after the call returns. (Ideally you wouldn’t actually have to do this manually for every portal call; instead you would call a non–portal `read` function in libc. This function’s job is to make the portal call, and whoever wrote it makes sure to include the `pass` instruction.)

¹ A turf is the set of addresses that a given thread running in a given protection domain can read and/or write.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#308

Post by userbinator »

voidspark wrote: post #224
devnullbrain wrote: post #216

Earlier quoted context omitted.

User has working software. User updates operating system. User has broken software. That's a problem for the party trying to sell operating system updates.

The software was fundamentally broken before the OS update. It was working by pure random chance with undefined behaviour. It’s a C++ issue, not an OS issue. The same code compiled for another OS would have different random results.

The problem is hostile pedants who neglect to consider the human element.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#309

Post by Thaxll »

ajross wrote: post #149
Thaxll wrote: post #138

Earlier quoted context omitted.

As if tools in early 2000's were any good...

Valgrind was released in 2002 to immediate celebration. It was available and surely known to the team. All they needed to do was write a unit test that loaded and instantiated those vehicle files and run it with "valgrind" in front of the command line.

Valgrind does not run on Windows.

Re: How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

#310

Post by pmarreck »

isatty wrote: post #145
pmarreck wrote: post #87

My takeaway, speaking as someone who leans towards functional programming and immutability, is "this is yet another example of a mutability problem that could never happen in a functional context" (so, for example, this bug would have never been created by Rust unless it was deeply misused)

The constant rust evangelism on this site is such a turn off from actually wanting to use the language.

I would have picked Elixir, but it's unsuitable for game dev (at least thus far)

I wouldn't consider it "Rust evangelism" as much as "not C/C++/any language that makes it trivially easy to write undefined-behavior bugs, evangelism".

I'd be just as much a fan of Roc, but they're not yet mature and actually in the middle of a compiler rewrite (as it so happens, from Rust to Zig, lol) https://www.roc-lang.org/

Post reply on HN