Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A nom parser for the Starcraft 2 Protocol Replay format

github.com

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#41

Post by sebosp »

reaperman wrote: post #37
tornato7 wrote: post #11

This is a super cool project! It would be especially interesting to use on pro-level games, maybe you could even sell services to pros who want to get better. Like being able to say "your baneling run-bys are most cost-effective with 7 banelings around the ten minute mark" etc. With a relatively larger dataset you could come up with some real interesting statistics on individual players performance, use of certain un…

https://sc2replaystats.com is one service already doing that, at least to some degree. Some features are locked behind a subscription payment. There's definitely room for improvement -- but some of what you're looking for is "unsolved"...optimal strategies are unknown, and the best strategy for a professional player may not be the best strategy for you. You'd likely want to practice strategies which won't get you top…

Is there a way to get a replay pack with order of millions of games (maybe more?) ? I'd like that to be able to create meaningful Stats, proper data analysis, training, etc... Wouldn't mind paying a bit for that... So far for the exploratory data analysis I've done I only have my own games and 2 or 3 replays I've been sent that don't work...

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#42

Post by reaperman »

sebosp wrote: post #41
reaperman wrote: post #37

Earlier quoted context omitted.

https://sc2replaystats.com is one service already doing that, at least to some degree. Some features are locked behind a subscription payment. There's definitely room for improvement -- but some of what you're looking for is "unsolved"...optimal strategies are unknown, and the best strategy for a professional player may not be the best strategy for you. You'd likely want to practice strategies which won't get you top…

Is there a way to get a replay pack with order of millions of games (maybe more?) ? I'd like that to be able to create meaningful Stats, proper data analysis, training, etc... Wouldn't mind paying a bit for that... So far for the exploratory data analysis I've done I only have my own games and 2 or 3 replays I've been sent that don't work...

The only groups I know of with millions of games are sc2replaystats and Blizzard themselves.

SC2replaystats had 4,440,656 replays uploaded in 2020 and 3,390,117 in 2019 (the only years I can quickly find data for). Searching and downloading replays is now paywalled, probably to limit scraping.

> Wouldn't mind paying a bit for that...

Google probably paid Blizzard a good amount of money to help them develop AlphaStar and in the process got access to all the replays. But I doubt you'd be offering enough money to get executive-level attention for such a deal. Sc2replaystats might be more receptive to selling you a randomly selected subset of their total data, but you'd still have to offer enough cash to be clearly worth the time of going through the negotiation process. I doubt they'd sell you all of their replays, as that's essentially the business' moat.

Perhaps you could buy the sc2replaystats business though.

(Yes, I know none of these suggestions are practical for the vast majority of interested hackers)

----------

There are smaller free replay packs available with a quick Google search. I'm sure if you looked around or asked around either Harstem's SC2 discord server or the SC2AI discord server or SC2 subreddit, someone would be able to point you to a free vault offering on the order of 10,000's of replays.

However, the game is undergoing a period of large updates right now which are dramatically changing the details of the gameplay ... so really you'd want as many as possible starting from the most recent "balance patch" which went live just 6 days ago[0]. It will take the SC2 community quite a long time to establish known "best practices" for the current balance patch, and it's highly likely that it will be significantly modified again soon -- the most recent patch seems to be politically fragile at all levels of the game. Casual to professional players all have valid constructive criticism of it, so more changes soon are essentially inevitable.

0: https://news.blizzard.com/en-us/starcraft2/24009150/starcraf...

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#43

Post by celdon25 »

sebosp wrote: post #35
celdon25 wrote: post #27

I'm curious to see how well this works with Direct Strike (custom game mode for SC2)

I only had one replay with direct strike in it's name but I can't remember watching it, I thought it was just one more map... I know it was failing initially to decode but in the end I fixed a couple of bugs and then it was working with this file. Could you explain what is significant go direct strike? How is it different to a normal game?

It has an active competitive scene, lobbies fill up in seconds sometimes. There are also tools such as dsstats that attempt to parse the replays already, but are limited by technology (maybe your tool can help with the limitations).

If you search Direct Strike Starcraft on youtube you should be able to see the gameplay itself.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#44

Post by reaperman »

sebosp wrote: post #39
reaperman wrote: post #34

Earlier quoted context omitted.

SC2 patches to an older version if you load an old version. Then patches back to current version when you want to play again. It requires the game to restart -- and while SC2 has a feature where you can watch a live replay with your friends (by all joining the same game which is hosted as type "replay") -- that "watch replay with others" feature only works with replays which were created in the latest version because…

Wow that is interesting that it patches back and forth, super cool! How did you find out about this? I wonder now and then if one can "consume" the events on a live game somehow, like connect as a bot or sniff the events before they are encrypted through the network... Also curious about the drone kill counter mod/overlay that Harstem uses, do you know of the name of that? I can't figure it out but would love to enri…

> sniff the events before they are encrypted through the network

Uhh, this is confusing as written. But there are cheating hacks which allow you to see extra data that is sent to your client in-game, such as the entire map and your's and your opponents production queues. And while I'm not sure if this is what you were asking: "sniffing" events from other clients which haven't been sent to the blizzard server is not possible, because they're delivered to your client via the blizzard server (it's not P2P).

> Also curious about the drone kill counter mod/overlay that Harstem uses, do you know of the name of that?

It's a combination of two things: 1) A customized UI layout for replays called "GameHeart" and 2) a custom SC2 game mod called "GameHeart" -- you can use the UI layout for ladder games which weren't created as custom games with the GameHeart mod, but for that replay you'll miss out on features like the drone kill counter and income graph vs. time for which the data is generated by the game mod.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#45

Post by ZephyrBlu »

This is insanely cool! Very impressed you managed to implement a full parser in Rust.

I implemented a basic one in Rust a while back: https://github.com/ZephyrBlu/rust-parser

And a full one in Python with a few bells and whistles ages ago: https://github.com/ZephyrBlu/zephyrus-sc2-parser

Don't maintain either of them though :(, and the Rust one is super rough.

SC2 is a very interesting area for data analysis, but at the same time I found it very challenging. There is so much nuance and inconsistency across games it can be really hard to do accurately do things like categorize builds or measure build timings.

The area I ended up focusing on was builds, and I feel like I did some interesting stuff there: https://sc2.gg/reports/top-openings-2022/.

I found personal statistics less interesting than aggregate statistics. Even pro games are very volatile, ladder games even more so. Extremely hard to get reliable signal out of them if you're trying to track things across games. Even simple things like Collection Rate are poor indicators without significant categorization work (Matchup, build, opponent build, etc).

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#46

Post by ZephyrBlu »

sebosp wrote: post #41
reaperman wrote: post #37

Earlier quoted context omitted.

https://sc2replaystats.com is one service already doing that, at least to some degree. Some features are locked behind a subscription payment. There's definitely room for improvement -- but some of what you're looking for is "unsolved"...optimal strategies are unknown, and the best strategy for a professional player may not be the best strategy for you. You'd likely want to practice strategies which won't get you top…

Is there a way to get a replay pack with order of millions of games (maybe more?) ? I'd like that to be able to create meaningful Stats, proper data analysis, training, etc... Wouldn't mind paying a bit for that... So far for the exploratory data analysis I've done I only have my own games and 2 or 3 replays I've been sent that don't work...

I have a dataset of ~20k (?) pro replays from 2017-2022 if you'd like that. I don't think a dataset of millions of pro games exists. If you asked breath (Owner of sc2replaystats) nicely he might give you a dump of ladder games though.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#47

Post by ZephyrBlu »

tornato7 wrote: post #11

This is a super cool project! It would be especially interesting to use on pro-level games, maybe you could even sell services to pros who want to get better. Like being able to say "your baneling run-bys are most cost-effective with 7 banelings around the ten minute mark" etc. With a relatively larger dataset you could come up with some real interesting statistics on individual players performance, use of certain un…

Extremely difficult to create a useful product like this for pros. They are looking for very specific things that are highly dependent on matchup, their build, opponent build, who their opponent is, how the game is going so far, etc.

Feedback like "your baneling run-bys are most cost-effective with 7 banelings around the ten minute mark" is firstly unlikely to be accurate, and secondly unlikely to be useful to a pro because their play is so situational.

There also isn't enough data in the replay files to piece together the kind of specific information they would want.

SC2 has extremely high variance, even at the pro level. It's very hard to mine useful insights from game data. Even pro builds, which are supposedly standardized, have crazy variation: https://sc2.gg/reports/top-openings-2022/.

There are so many variations in games, and there are often adjustments on the fly or even sometimes mess ups.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#48

Post by jquery »

reaperman wrote: post #34
sebosp wrote: post #17

Earlier quoted context omitted.

This brings another question, if I watch an old replay, does it mean it has to have the versioned game engine stored in the computer? Does it mean with every update it gets increasingly bigger? Or is there a number of supported versions you could replay? I only started playing like 3 years ago...

SC2 patches to an older version if you load an old version. Then patches back to current version when you want to play again. It requires the game to restart -- and while SC2 has a feature where you can watch a live replay with your friends (by all joining the same game which is hosted as type "replay") -- that "watch replay with others" feature only works with replays which were created in the latest version because…

Can you link to any of those Harstem videos where he takes over the "losing" side?

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#49

Post by Kaszanas »

I am currently conducting research on esprots as a whole with the main target being StarCraft since around 2019. I was able to publish a nice dataset of pro replays, and resulting JSON files after using one of the Golang parsers written by Icza.

https://www.nature.com/articles/s41597-023-02510-7

Feel free to look into the tools on my GitHub (https://github.com/Kaszanas). Since this is mostly the topic of my PhD I guess I will be updating the dataset in the near future. You may want to try and test your parser against it.

further research for you would probably include running Logistic Regression on aggregated data from each of the replays to try and have a model that can discern between winners and losers and see which parameters are key in your data.

Example: https://www.researchgate.net/publication/363613604_Determina...

And even further embedding the games as timeseries data via various methods.

Re: Show HN: A nom parser for the Starcraft 2 Protocol Replay format

#50

Post by dleeftink »

On the data analytical site, I always wondered if complex game state patterns could be deconstructed using techniques from the JKU Visual Data Science Lab [0][1]. Here they demonstrate chess board win states from competitive data, and I wonder what the state relationships would look like for different (video) game matches and if new strategies could be derived from it.

[0]: https://youtu.be/yBCe8SqGwK8

[1]: https://jku-vds-lab.at/publications/2022_embedding_structure...

Post reply on HN