Was using Thrift or Protobuf an option?
What is Thrift? Is it a service?
Saving Millions by Dumping Java Serialization
21–30 of 48 posts
Re: Saving Millions by Dumping Java Serialization
#22Some interesting benchmarks of various Java serialization libraries:
Re: Saving Millions by Dumping Java Serialization
#23Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.
Either way, it's serialization: object serialization or JSON serialization.
However, independently of how the data is represented (JSON or one of the many binary formats), the issue is to only encode/decode what you actually care about. From the little I understand about java object serialization, there's a lot of extra stuff that gets encoded, which may not be needed at all for the application at hand.
For an example of efficient serialization techniques, take a look at some of the MPEG formats (the older ones are easier to grok). They have a neat way of representing what is needed and dealing with optional data.
Re: Saving Millions by Dumping Java Serialization
#24Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.
JSON is a serialization format, just one that at least nods in the direction of human-readability. Formats which don't worry about human readability (e.g. Protobuf) can gain various degrees of efficiency.
Re: Saving Millions by Dumping Java Serialization
#25Earlier quoted context omitted.
A standard database table isn't large enough to handle our large datasets. For example, the Hercules dataset was over 2 petabytes and even after optimization is almost 1 petabyte. Big data systems like Spark, Impala, Presto, etc. are designed to make the data look like a table, even though it is spread out into many files in a distributed filesystem. This is what we do. It's pretty common to reimplement some database…
Well, you understand your system and requirements better than I, obviously, but... A standard database table isn't large enough to handle our large datasets ... isn't much of an answer as-to why you're storing objects in your database. As you already mentioned in your post, serialized objects are big - they contain all of their data, plus everything necessary to deserialize the object into something usable. I imagine…
What's "the database" that you have in mind?
Start out with the idea that you have hundreds of machines in your cluster, with 1000s of TB of data. Suppose the current data efficiency is on the order of 80% - that is, 80% of the 1000s of TB is the actual bytes of the data fields. What database do you have in mind to store this data, still on the order of 1000s of TB?
You say: a couple of joined tables. So you have hundreds of machines, and the tables are not all going to fit on one machine; they're going to be scattered across hundreds of machines each. How do you efficiently do a join across two distributed tables?
It's no picnic.
If each row in one table only has a few related rows in the other table, it's much, much better to store the related data inline. Locality is key; you want data in memory right now, not somewhere on disk across the network.
Re: Saving Millions by Dumping Java Serialization
#26Author here, let me know if you have any questions/want more details.
1) Can you provide any more details about how Rowfiles are structured and/or implemented? Specifically, how does it handle nested objects? Does it support `transient`? Do `writeObject` and/or `readObject` come into play?
2) Do you feel this is a generic enough solution that you would consider submitting it as a JSR?
Re: Saving Millions by Dumping Java Serialization
#27Earlier quoted context omitted.
I'd like to know this too. As a passerby, those seem to have solved serialization, so I'm curious why you need rowfiles instead of e.g. protobuf.
One reason on top of my head: Using such communication protocol would require changes to the other services consuming it.
So did switching to their homebrew serialization format -- in fact, most of the article is about how they managed the changes (which touched codebases at multiple sites in a fairly large organization).
Re: Saving Millions by Dumping Java Serialization
#28TLDR: we had shitty code, optimized it, now it runs well. No code examples, nothing.
If you want more details, we were packing a Row class into a base64 encoded string using an ObjectOutputStream. This is a fine thing for small scale serialization but sucks at scale, because of the reasons mentioned in the post. Sorry we don't have code examples, but it's unclear how useful it'd be given that no one else uses our file format. If you want a bit more detail on how the format works. Each metadata contai…
Avro can store the schema inline or out of line; with inline schemas, it's at the start of the file (embedded JSON), and it describes the schema for all the rows in that file. If you're working with Hive, the schema you put in the Hive metastore is cross-checked with each Avro file read; if any given Avro file doesn't contain a particular column, it just turns up as null for that subset of rows. Spark and Impala work similarly.
I agree serialization at scale is interesting. My particular interest right at this moment is in efficiently doing incremental updates of HDFS files (Parquet & Avro) from observing changes in MySQL tables - not completely trivial because some ETL with joins and unions is required to get data in the right shape.
Re: Saving Millions by Dumping Java Serialization
#29> Secondly, Java serialization produces very bulky outputs. Each serialization contains all of the data required to deserialize. When you’re writing billions of records at a time, recording the schema in every record massively increases your data size. Sounds to me like you shouldn't be storing objects in your database. Why not just write the data into tables, and then create new POJO's when necessary, using the sele…
The article explains that they did, in fact, promote the data into columns of the data store.
Re: Saving Millions by Dumping Java Serialization
#30Was using Thrift or Protobuf an option?
We started developing rowfiles around 2005. Thrift wasn't open sourced until 2007. I couldn't find a date for protobuf's release, but I don't think it was standard outside of google at that time. We use protobufs internally, and have a number of Rows whose field values are byte[]s containing protobufs. One big thing our rowfiles gives us is fast indexing. The only other big data format I know of that gives that is Kudu, which uses the same indexing scheme.