Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serialization for C# Games

chickensoft.games

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Serialization for C# Games

#41

Post by jayd16 »

I want to like this because it seems well done but I kind of grimace instead. It's not the library's fault.

Game engines have some form of serialization already (most of what a game engine does is load a serialized game state into memory, imo).

I've found its usually better to try to leverage those systems so you're not building multiple models objects and doing conversions between game engine and serialized types.

Engines often do a lot of (design)work to load things directly into memory in such a way that the game engine can use the inflated object immediately without a lot of parsing. It's nice to try to leverage that. Moreover less plugins is less complexity in the build process etc etc.

Those desires give me pause when looking at serialization plugins in the context of game engines.

Howeve, it's also not entirely feasible to only use the core engine systems in all cases. Often what's available at runtime for a game engine isn't always the same as build time. You might need to read this data outside of the engine and then you're really out of luck...life's so complicated.

Re: Serialization for C# Games

#42

Post by jolexxa »

wheybags wrote: post #29

In my experience, the pain of dealing with changes outweighs the pain of dealing with boilerplate, so it's better to explicitly write out save and load functions manually than rely on reflection. Also means you can do stuff like if(version<x) { load old thing + migrate} else {load new thing} very easily. And it's just code, not magic.

That's essentially what this system does — it identifies the models and their properties that you've marked as serializable at build-time using source generation, and then allows you to provide a type resolver and converter to System.Text.Json that lets you make upgrade-able models with logic like you just described.

The assist from the source generation helps reduce some of the boilerplate you need, but there's no escaping it ultimately.

Re: Serialization for C# Games

#43

Post by jolexxa »

kogir wrote: post #4

This seems to cover many common pain points, but I’ve written my fair share of .NET serializers and for anything I build now I’d just use protocol buffers. Robust support, handles versioning pretty well, and works cross platform. I’d like to know their reasons for making yet another serializer vs just using pb or thrift.

This is a good point. I don't think anyone wakes up wanting to make a new serializer. At this point, I was already pretty deep into making and releasing tools for my game projects so doing this didn't seem like such a stretch (although it actually ended up being one of the hardest things I've ever done).

A lot of small to mid-size games (which are the focus of the tools I provide) want to save data into JSON, whether it is to be mod-friendly or just somewhat human-friendly to the developer while working on the game. Not familiar with Thrift, but PB is obviously for binary data and has a focus on compactness and performance, which isn't the primary concern on my list of priorities for a serialization system. My primary concern for a serialization system is refactor-friendliness. I want to be able to rework type hierarchies without breaking existing save files, or get as close to that as possible.

I suppose you could say I'm only really introducing "half" of a serialization system: the heavy lifting is being split between the introspection generator (for writing metadata at compile time via source generation) and System.Text.Json (which handles a lot of the runtime logic for serializing/deserializing things).

Re: Serialization for C# Games

#44

Post by nitwit005 »

nearbuy wrote: post #16
isthiseasymode wrote: post #7

Naive question: is there a reason why SQLite wouldn’t work for something like this?

You could use it, but it's not really solving the same problem. For a game, you generally don't need the relational database features. You aren't doing queries. You just want to load an entire level into memory, or save an entire level. For the serialization and persistence aspect, I don't see an advantage of SQLite over just calling JsonSerializer.Serialize(). The author's system then adds a bunch of features like v…

While you don't need the relational features, some games do need the ability to make partial updates to make auto-save performant.

Do a search for something like "Minecraft save game size", and you'll see some people have multi-gigabyte saves. Similar issues crop up with some Paradox Interactive games.

Re: Serialization for C# Games

#45

Post by slaymaker1907 »

flohofwoe wrote: post #37
voxic11 wrote: post #36

Earlier quoted context omitted.

If your game engine is built on a data-first architecture like ECS then it can be pretty trivial to directly serialize your game state. I have had good luck with this using bitECS https://github.com/NateTheGreatt/bitECS/blob/master/docs/INT...

Agreed, when the data is already in a table format (instead of an "object spider web") the idea to automate serialization makes more sense, it essentially becomes a "database problem". I would still very carefully consider what data columns need to be persisted and which should be reconstructed, and I wouldn't try to come up with a too generic solution. For instance in some games it might not be necessary to save a r…

Depending on how large your save state is, it could be as simple as a function mapping from a list of game objects to the saveable object. That approach works really well with Redux on the web since you really don't want to save most things. Where things really get tricky is when you want to get fancy and support things like saving only the changed portion of the state.

Re: Serialization for C# Games

#46

Post by jolexxa »

Madmallard wrote: post #2

RunUO has an implementation of this and it's like 25 years old but still worked really well

I really like this implementation, but it's probably worth mentioning here that RunUO and other tools like it are solving the problem at a layer of abstraction beneath what I was introducing here.

The serialization system I am providing here actually leverages System.Text.Json for reading and writing data — it's more concerned with helping you represent version-able, upgrade-able data models that are also compatible with the hierarchical state machine implementation I use for managing game state.

Re: Serialization for C# Games

#47

Post by bob1029 »

freedomben wrote: post #40
bob1029 wrote: post #30

Earlier quoted context omitted.

https://www.sqlite.org/pragma.html#pragma_user_version I use SQLite for game state management. It's just like any other database scenario. I write migrators that check the user_version of the database. It's just a for loop from user_version to current version. The migrators themselves can be arbitrary methods that sometimes modify game state to bring it up to date. The most common scenario is adding a new property to…

I do the same thing, but I have increasingly found myself wanting to serialize json into columns because having a rigid schema can sometimes add a lot of friction. Experience has taught me though, that it's worth the extra effort to define a schema, because nine times out of 10, the flexible json will ossify into unexpected format that the code relies on anyway, but now the database doesn't help enforce integrity. I…

I am not against the JSON-in-columns hybrid path, but I have typically found it grows into a monster over time. In my experience, it caused performance problems more than anything else.

Re: Serialization for C# Games

#48

Post by slaymaker1907 »

interroboink wrote: post #3

Sometimes, when battling these issues, I wish the Smalltalk-style approach[1][2] was more popular/feasible. Basically, saving the entire state of the VM is a fundamental operation supported by the language. Only truly transient things like network connections require special effort. There are some echoes of this with things like Lua's Pluto/Eris, or serializable continuations in other languages (eg: Perl's Continuity…

You can surprisingly sort of do this in Java. Just create a lambda which will start the game at the current state when you call it.

Re: Serialization for C# Games

#49

Post by jolexxa »

flohofwoe wrote: post #23

If I learned one important lesson from writing savegame systems: don't directly serialize your entire game state on the "game object level" (e.g. don't create a savegame by running a serializer over your game object soup), instead decouple the saved data from your game logic internals, have a clear boundary between your game logic and the savegame-system, keep the saved state as minimal as possible, and reconstruct o…

All very good advice that I feel deeply. I think I fell into the honey trap some time ago, but I've made peace with that — the tools I'm making will probably do more good than any game I could finish making, at least for now.

Jokes aside, though, I do try to dog-food my tooling as much as possible. I maintain a Godot/C# 3d platformer game demo with full state preservation/restoration (https://github.com/chickensoft-games/GameDemo>) to demonstrate this.

By the time I've finished writing tests and docs for a tool, I've usually identified and fixed a bunch of usability pain points and come up with a happy path for myself and other developers — even if it's not 100% perfect.

I also have a bunch of unreleased game projects that spawned these projects, and even gave a talk on how this stuff came about (https://www.youtube.com/watch?v=fLBkGoOP4RI&t=1705s>) a few months ago if that's of interest to you or anyone else.

The requirements you mentioned in your comment cover selectively serializing state and decoupling saving/loading logic, and I could not agree more. While you can always abuse a serializer, I hope my demonstration in the game demo code shows how I've selectively saved only relevant pieces of game data and how they are decoupled and reconstructed across the scene tree.

Also probably worth mentioning the motivation behind all this — the serialization system here should hopefully enable you to easily refactor type hierarchies without having to maintain manual lists of derived types like System.Text.Json requires you to do when leveraging polymorphic deserialization.

Manually tracking types (presumably in another file, even) is such an error-prone thing to have to do when using hierarchical state machines where each state has its own class (like https://github.com/chickensoft-games/LogicBlocks>). States as classes is super common when following the state pattern and it is well supported with IDE refactoring tools since they're just classes. Basically this serialization system exists to help save complex, hierarchical state without all the headaches. While I was at it, I also introduced opinionated ways to handle versioning and upgrading because that's also always a headache.

Re: Serialization for C# Games

#50

Post by nearbuy »

nitwit005 wrote: post #44
nearbuy wrote: post #16

Earlier quoted context omitted.

You could use it, but it's not really solving the same problem. For a game, you generally don't need the relational database features. You aren't doing queries. You just want to load an entire level into memory, or save an entire level. For the serialization and persistence aspect, I don't see an advantage of SQLite over just calling JsonSerializer.Serialize(). The author's system then adds a bunch of features like v…

While you don't need the relational features, some games do need the ability to make partial updates to make auto-save performant. Do a search for something like "Minecraft save game size", and you'll see some people have multi-gigabyte saves. Similar issues crop up with some Paradox Interactive games.

Games are hugely varied. No doubt there are games out there for which SQLite is perfect. But I wouldn't use it for making partial updates in something like Minecraft.

It's not practical to store individual Minecraft blocks as table entries, so if you were using SQLite, you'd likely just store chunks (e.g. 16x16x16 blocks) as binary blobs. Then you'd rewrite entire chunks on save. It's not really taking advantage of what SQLite offers.

There are a lot of serializers and frameworks out there you could choose from, but even something as simple as just writing one map region per file and overwriting modified regions on save would be better than SQLite.

Post reply on HN