Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A nom parser for the Starcraft 2 Protocol Replay format

github.com

Post reply on HN
Print view View original

11–20 of 58 posts

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#11

Post by tornato7 »

This is a super cool project! It would be especially interesting to use on pro-level games, maybe you could even sell services to pros who want to get better. Like being able to say "your baneling run-bys are most cost-effective with 7 banelings around the ten minute mark" etc.

With a relatively larger dataset you could come up with some real interesting statistics on individual players performance, use of certain units, the success of various strategies and build orders, etc.

It would be real fun to try to predict the outcome of a game based on the first 3 minutes or something.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#12

Post by icza »

sebosp wrote: post #7
jsphweid wrote: post #4

I've also been impressed with the amount of data in SC2 replay files given the size, although I'm more interested in Broodwar (SC1) as a game. If I'm not mistaken Broodwar replay files contain a lot less information. They mainly contain player actions, i.e. you don't even know when a marine died -- the game has to literally be replayed to get that information.

Is there a general name to the way the data is compressed/serialized in SC2 Replays? Yesterday somebody was talking about other ways to serialize/pack data with borsh but not sure it's at all similar... Is the SC1 broodwar replay format similar with byte-alignment and bit "packing"? BTW I call the parsers "bitpacked" or "byte-aligned", but I'm not sure what the right name for them is...

SC2 replays are MPQ files, which is a proprietary format created and used by Blizzard. It's an archive that may contain multiple files stored with different compression and optionally encrypted. I wrote a lib to parse MPQ files that embodies SC2Replays: https://github.com/icza/mpq. I also wrote an SC2 replay parser that is more or less a port of the official s2protocol: https://github.com/icza/s2prot

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#13

Post by sebosp »

ktpsns wrote: post #6
jsphweid wrote: post #4

I've also been impressed with the amount of data in SC2 replay files given the size, although I'm more interested in Broodwar (SC1) as a game. If I'm not mistaken Broodwar replay files contain a lot less information. They mainly contain player actions, i.e. you don't even know when a marine died -- the game has to literally be replayed to get that information.

The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.

I fantasize if training an AI to watch all the replays of a map on YouTube can create a repro of the state machine logic that moves the units through the game, or to find the parts of a map that cannot be walked upon. In the replay it emits an event when the user wants a unit to move to x, y but then it has to path its way there, and I cannot predict (without their logic) how it would move there

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#14

Post by rcoveson »

icza wrote: post #10
ktpsns wrote: post #6

Earlier quoted context omitted.

The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.

Most of game corruption wasn't the result of faulty replay save / generation. Most of game corruption was the result of replays only recording user actions, and when the game engine was patched and rules were changed, re-playing the same user action could go wrong: imagine the cost of Probe getting increased from 50 minerals to 60. You load a replay that contains a "Build Probe" command when the user had 50 minerals.…

> imagine the cost of Probe getting increased from 50 minerals to 60

Yes, please.

- T, Z

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#15

Post by sebosp »

icza wrote: post #12
sebosp wrote: post #7

Earlier quoted context omitted.

Is there a general name to the way the data is compressed/serialized in SC2 Replays? Yesterday somebody was talking about other ways to serialize/pack data with borsh but not sure it's at all similar... Is the SC1 broodwar replay format similar with byte-alignment and bit "packing"? BTW I call the parsers "bitpacked" or "byte-aligned", but I'm not sure what the right name for them is...

SC2 replays are MPQ files, which is a proprietary format created and used by Blizzard. It's an archive that may contain multiple files stored with different compression and optionally encrypted. I wrote a lib to parse MPQ files that embodies SC2Replays: https://github.com/icza/mpq . I also wrote an SC2 replay parser that is more or less a port of the official s2protocol: https://github.com/icza/s2prot

I mean for example https://github.com/sebosp/s2protocol-rs/blob/755098fb86ab6b1... I hacked my way around the json protocol specification, an enum has n types, using log I can find the number of bits I need to read to uniquely identify each variant, that kind of serialization I wonder the name of.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#16

Post by IshKebab »

brundolf wrote: post #3
benmanns wrote: post #2

I may be the only one not familiar, but nom refers to https://github.com/rust-bakery/nom which looks like a pretty handy way to parse binary data in Rust.

It's a pretty handy way to parse any sequence of things in Rust! I'm using it to write a compiler for a toy language and it's been a delight

I would consider Chumsky for a programming language. It has recoverable parsing which is important for IDE support.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#17

Post by sebosp »

icza wrote: post #10
ktpsns wrote: post #6

Earlier quoted context omitted.

The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.

Most of game corruption wasn't the result of faulty replay save / generation. Most of game corruption was the result of replays only recording user actions, and when the game engine was patched and rules were changed, re-playing the same user action could go wrong: imagine the cost of Probe getting increased from 50 minerals to 60. You load a replay that contains a "Build Probe" command when the user had 50 minerals.…

This brings another question, if I watch an old replay, does it mean it has to have the versioned game engine stored in the computer? Does it mean with every update it gets increasingly bigger? Or is there a number of supported versions you could replay? I only started playing like 3 years ago...

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#18

Post by icza »

sebosp wrote: post #15
icza wrote: post #12

Earlier quoted context omitted.

SC2 replays are MPQ files, which is a proprietary format created and used by Blizzard. It's an archive that may contain multiple files stored with different compression and optionally encrypted. I wrote a lib to parse MPQ files that embodies SC2Replays: https://github.com/icza/mpq . I also wrote an SC2 replay parser that is more or less a port of the official s2protocol: https://github.com/icza/s2prot

I mean for example https://github.com/sebosp/s2protocol-rs/blob/755098fb86ab6b1... I hacked my way around the json protocol specification, an enum has n types, using log I can find the number of bits I need to read to uniquely identify each variant, that kind of serialization I wonder the name of.

I'm not sure about an official name, I also just call it "bitpacked".

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#19

Post by jsphweid »

rcoveson wrote: post #14
icza wrote: post #10

Earlier quoted context omitted.

Most of game corruption wasn't the result of faulty replay save / generation. Most of game corruption was the result of replays only recording user actions, and when the game engine was patched and rules were changed, re-playing the same user action could go wrong: imagine the cost of Probe getting increased from 50 minerals to 60. You load a replay that contains a "Build Probe" command when the user had 50 minerals.…

> imagine the cost of Probe getting increased from 50 minerals to 60 Yes, please. - T, Z

Artosis alt account?! :P

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#20

Post by sebosp »

ktpsns wrote: post #6
jsphweid wrote: post #4

I've also been impressed with the amount of data in SC2 replay files given the size, although I'm more interested in Broodwar (SC1) as a game. If I'm not mistaken Broodwar replay files contain a lot less information. They mainly contain player actions, i.e. you don't even know when a marine died -- the game has to literally be replayed to get that information.

The Broodwar replays were really cursed -- I remember that hilarious (perfectly reproducable) bugs of quickly replaying a game which resulted in a completely different replay then the actual game, apparently because the replay missed to integrate all replay information correctly and ended up in a different game state.

Btw such move instructions I visualize here with the amazing rerun.io https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO1...
Post reply on HN