Writing a Memory Allocator for Fast Serialization
1–10 of 23 posts
Re: Writing a Memory Allocator for Fast Serialization
#2With libsrt C library [1][2] using one thread on Intel Core i5 3330 (maps based on Red-Black tree, not hash tables, i.e. no rehash required):
* libsrt i64-i64 map (equivalent to std::map : > 10M QPS
* libsrt string-string map (equivalent to std::map : > 1M QPS (> 2M QPS if key size [1] Repository: https://github.com/faragon/libsrt
[2] Benchmarks: https://github.com/faragon/libsrt/blob/master/doc/benchmarks...
Re: Writing a Memory Allocator for Fast Serialization
#3This is a similar approach to what Jonathan Blow demo'd recently in his Jai language. He uses relative offsets from the location of the pointer instead of from the base of the region though. The advantage there is that you don't need to pass around a reference to the current heap, the disadvantage is that copying/moving is harder but that can be mitigated wit compiler support.
Re: Writing a Memory Allocator for Fast Serialization
#4This is a similar approach to what Jonathan Blow demo'd recently in his Jai language. He uses relative offsets from the location of the pointer instead of from the base of the region though. The advantage there is that you don't need to pass around a reference to the current heap, the disadvantage is that copying/moving is harder but that can be mitigated wit compiler support.
Author here. I did think about creating a programming language specialized for serialization. Fortunately, using just C seems to be sufficient for building a POC. Another advantage for using C is it is easier to embed into other languages. OPIC is more library focused, which would benefit for integrating into other languages. Jai language seems to be a application (gaming) focused language, and the language abstraction makes developer faster to code is more important.
Re: Writing a Memory Allocator for Fast Serialization
#5I don’t understand the comment about not being able to use C++. In C++, you certainly have the same low-level control over your objects as you do in C. You gain a lot of syntax sugar and additional type safety.
If you just don’t wanna use C++, fine. But this looks like a great fit for it to me. Avoid the STL, turn off the features you don’t need, etc.
Re: Writing a Memory Allocator for Fast Serialization
#6What happens if the program unexpectedly terminates during an update to these persistent data structures? Is the mmap'ed file corrupted, or still in a usable state?
Re: Writing a Memory Allocator for Fast Serialization
#7What happens if the program unexpectedly terminates during an update to these persistent data structures? Is the mmap'ed file corrupted, or still in a usable state?
The operating system will dutifully write back to disk whatever bytes your program left on those memory-mapped pages at the moment it terminated. If your data structures were in an inconsistent state -- pointers to nowhere, say -- then your file will get in the same inconsistent state, i.e. corrupted.
Re: Writing a Memory Allocator for Fast Serialization
#8I don’t understand the comment about not being able to use C++. In C++, you certainly have the same low-level control over your objects as you do in C. You gain a lot of syntax sugar and additional type safety. If you just don’t wanna use C++, fine. But this looks like a great fit for it to me. Avoid the STL, turn off the features you don’t need, etc.
The problem is C++ brings in many extra pointers. For example, the vtable pointer used in virtual functions. All the pointers not converted to offset can be invalid in next process that deserializes the object.
Re: Writing a Memory Allocator for Fast Serialization
#9What happens if the program unexpectedly terminates during an update to these persistent data structures? Is the mmap'ed file corrupted, or still in a usable state?
I don't have a good answer to this question yet. For now I only create the heap in swap, write it to disk, and then use it as read only mmap. To ensure the written file is valid, one can write it to a temporal file first, once confirmed the file is written, then mv the file to desired file name and location. This works for immutable data, but is a big blocker for me to make OPIC work on mutable back store.
This problem is generally hard. See [Ensuring data reaches disk](https://lwn.net/Articles/457667/)
Re: Writing a Memory Allocator for Fast Serialization
#10I don’t understand the comment about not being able to use C++. In C++, you certainly have the same low-level control over your objects as you do in C. You gain a lot of syntax sugar and additional type safety. If you just don’t wanna use C++, fine. But this looks like a great fit for it to me. Avoid the STL, turn off the features you don’t need, etc.
The problem is C++ brings in many extra pointers. For example, the vtable pointer used in virtual functions. All the pointers not converted to offset can be invalid in next process that deserializes the object.
You don't have to use virtual functions though. You can just use plain old data structures.