Was using Thrift or Protobuf an option?
Saving Millions by Dumping Java Serialization
11–20 of 48 posts
Re: Saving Millions by Dumping Java Serialization
#12TLDR: we had shitty code, optimized it, now it runs well. No code examples, nothing.
If you want more details, we were packing a Row class into a base64 encoded string using an ObjectOutputStream. This is a fine thing for small scale serialization but sucks at scale, because of the reasons mentioned in the post. Sorry we don't have code examples, but it's unclear how useful it'd be given that no one else uses our file format. If you want a bit more detail on how the format works. Each metadata contai…
What do you mean when you say Avro doesn't let you "cleanly update or track schema"?
From what I've read about Avro 1. It can transform data between two compatible schemas. 2. It can serialize/load schemas off the wire, so you can send the schema in the header.
If schema serialization causes too much overhead, you can set things up so you only send the schema version identifier, as long as the receiver can use that to get access to the full schema.
Re: Saving Millions by Dumping Java Serialization
#13Was using Thrift or Protobuf an option?
I'd like to know this too. As a passerby, those seem to have solved serialization, so I'm curious why you need rowfiles instead of e.g. protobuf.
Re: Saving Millions by Dumping Java Serialization
#14Can someone explain to an amateur why serialization is faster than say passing raw JSON?
It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.
Re: Saving Millions by Dumping Java Serialization
#15Was using Thrift or Protobuf an option?
I'd like to know this too. As a passerby, those seem to have solved serialization, so I'm curious why you need rowfiles instead of e.g. protobuf.
One reason on top of my head: Using such communication protocol would require changes to the other services consuming it.
Re: Saving Millions by Dumping Java Serialization
#16TLDR: we had shitty code, optimized it, now it runs well. No code examples, nothing.
"We had shitty code, optimized it, now it runs [maybe] better" is probably the only fixture in software development.
I'm not sure kind of code samples one would want in the context of an article as abstract as this one.
I did not take anything from the article, but had a lot of "been there, done that" moments when skimming it. The difference between the OP an me: The OP wrote about it so others can learn, something I never did.
Re: Saving Millions by Dumping Java Serialization
#17Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.
Serialization is the process of writing arbitrary data out into a blob of some sort (binary, text, whatever) that can be read in later and processed back into the original data, possibly not by the same system. This should be considered to include even the degenerate case of just writing the content of an expanse of RAM out, as that still raises issues related to serialization.
"JSON Serialization" and "Java Serialization" are two different things that can accomplish that goal. It sounds to me from your question that you think they have some fundamental difference, because your second paragraph implies you believe there is some sort of fundamental difference between Java serialization and JSON serialization, but there isn't. There is a whole host of non-fundamental differences that you always have to consider with a serialization format (speed, what can be represented, circular data structure handling, whether untrusted data can be used), but there's not a fundamental difference.
Re: Saving Millions by Dumping Java Serialization
#18Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.
JSON must also be serialized or deserialized. Parsing it is slow and hard and not cache friendly.
Protobuf has the benefit of being extremely compact and, in some important languages, fast and friendly to serialize and deserialize.
Re: Saving Millions by Dumping Java Serialization
#19Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.
[deleted]
Re: Saving Millions by Dumping Java Serialization
#20Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.
For one thing, all numbers can be written in binary, saving the lexing and conversion time. For another, strings can be written by first writing the length (in binary, of course), then writing the raw contents; there's no need to scan the input looking for the closing quote, handle backslash escapes, or do UTF-8 conversion.
That's probably most of the gain right there, but more things can be done along those lines.