Saving Millions by Dumping Java Serialization
quantcast.com
Saving Millions by Dumping Java Serialization
1–10 of 48 posts
Re: Saving Millions by Dumping Java Serialization
#2Author here, let me know if you have any questions/want more details.
Re: Saving Millions by Dumping Java Serialization
#3Author here, let me know if you have any questions/want more details.
So... what's quantcast?
Re: Saving Millions by Dumping Java Serialization
#4TLDR: we had shitty code, optimized it, now it runs well. No code examples, nothing.
Re: Saving Millions by Dumping Java Serialization
#5Author here, let me know if you have any questions/want more details.
So... what's quantcast?
We're a big data advertise and measure company based in San Francisco. We run online display ad campaigns for marketers across realtime bidding exchanges (RTB), such as those run by Google and AppNexus. We also provide a publisher product to give site owners insights into their audience. Stack Overflow's profile is at https://www.quantcast.com/stackoverflow.com.
Re: Saving Millions by Dumping Java Serialization
#6> Secondly, Java serialization produces very bulky outputs. Each serialization contains all of the data required to deserialize. When you’re writing billions of records at a time, recording the schema in every record massively increases your data size.
Sounds to me like you shouldn't be storing objects in your database.
Why not just write the data into tables, and then create new POJO's when necessary, using the selected data?
Re: Saving Millions by Dumping Java Serialization
#7> Secondly, Java serialization produces very bulky outputs. Each serialization contains all of the data required to deserialize. When you’re writing billions of records at a time, recording the schema in every record massively increases your data size. Sounds to me like you shouldn't be storing objects in your database. Why not just write the data into tables, and then create new POJO's when necessary, using the sele…
A standard database table isn't large enough to handle our large datasets. For example, the Hercules dataset was over 2 petabytes and even after optimization is almost 1 petabyte. Big data systems like Spark, Impala, Presto, etc. are designed to make the data look like a table, even though it is spread out into many files in a distributed filesystem. This is what we do. It's pretty common to reimplement some database features onto these big data file formats. In our case we have very fast indexes that let us quickly fetch data, similar to an index in a postgresql table.
Re: Saving Millions by Dumping Java Serialization
#8TLDR: we had shitty code, optimized it, now it runs well. No code examples, nothing.
If you want more details, we were packing a Row class into a base64 encoded string using an ObjectOutputStream. This is a fine thing for small scale serialization but sucks at scale, because of the reasons mentioned in the post.
Sorry we don't have code examples, but it's unclear how useful it'd be given that no one else uses our file format. If you want a bit more detail on how the format works. Each metadata contains a list of typed columns to define the schema of a given part. Our map-reduce framework has a bunch of internal logic that tries to justify the written Row class with the one the Mapper class is asking for. This allows us to do things like ingest different versions of a row with in the context of a single job.
I think questions of serialization at the scale are generally interesting, although ymmv. I know of one company using Avro, which doesn't let you cleanly update or track schema. They've ended up storing every schema in an HBase table and reserving the first 8 bytes to do a lookup into this table to know the row's schema.
Re: Saving Millions by Dumping Java Serialization
#9> Secondly, Java serialization produces very bulky outputs. Each serialization contains all of the data required to deserialize. When you’re writing billions of records at a time, recording the schema in every record massively increases your data size. Sounds to me like you shouldn't be storing objects in your database. Why not just write the data into tables, and then create new POJO's when necessary, using the sele…
A standard database table isn't large enough to handle our large datasets. For example, the Hercules dataset was over 2 petabytes and even after optimization is almost 1 petabyte. Big data systems like Spark, Impala, Presto, etc. are designed to make the data look like a table, even though it is spread out into many files in a distributed filesystem. This is what we do. It's pretty common to reimplement some database…
Well, you understand your system and requirements better than I, obviously, but...
A standard database table isn't large enough to handle our large datasets
As you already mentioned in your post, serialized objects are big - they contain all of their data, plus everything necessary to deserialize the object into something usable.
I imagine your objects have the standard amount of strings, characters, numbers, booleans, etc... why not just store those in the database and select them back out when needed? Less data in the database, and faster retrieval time since you skip serialization in both steps (storage and retrieval). Even if you have nested objects within nested objects, you can write-out a "flat" version of the data to a couple of joined tables surely.
On the other hand, serializing the object is probably more "simple" to implement and use... but then you get the classical tradeoff of performance vs. convenience.
Re: Saving Millions by Dumping Java Serialization
#10Author here, let me know if you have any questions/want more details.
Thanks for writing about your experiences. Why not use an existing serialization framework, such as Protobuf, instead of building something in-house?