Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving Millions by Dumping Java Serialization

quantcast.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Saving Millions by Dumping Java Serialization

#31

Post by jnewhouse »

cakoose wrote: post #12
jnewhouse wrote: post #8

Earlier quoted context omitted.

If you want more details, we were packing a Row class into a base64 encoded string using an ObjectOutputStream. This is a fine thing for small scale serialization but sucks at scale, because of the reasons mentioned in the post. Sorry we don't have code examples, but it's unclear how useful it'd be given that no one else uses our file format. If you want a bit more detail on how the format works. Each metadata contai…

What do you mean when you say Avro doesn't let you "cleanly update or track schema"? From what I've read about Avro 1. It can transform data between two compatible schemas. 2. It can serialize/load schemas off the wire, so you can send the schema in the header. If schema serialization causes too much overhead, you can set things up so you only send the schema version identifier, as long as the receiver can use that t…

I think what I'd heard about was likely a poorly implemented use of Avro. I haven't actually worked with it.

Re: Saving Millions by Dumping Java Serialization

#33

Post by MS_Buys_Upvotes »

felixgallo wrote: post #18
MS_Buys_Upvotes wrote: post #14

Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.

JSON must also be serialized or deserialized. Parsing it is slow and hard and not cache friendly. Protobuf has the benefit of being extremely compact and, in some important languages, fast and friendly to serialize and deserialize.

Thanks! You and the others are right: I didn't know JSON was serialized.

I can see why something in binary would be faster than structured text (think assembler vs Python).

Thanks again.

Re: Saving Millions by Dumping Java Serialization

#35

Post by chiph »

ScottBurson wrote: post #20
MS_Buys_Upvotes wrote: post #14

Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.

For one thing, all numbers can be written in binary, saving the lexing and conversion time. For another, strings can be written by first writing the length (in binary, of course), then writing the raw contents; there's no need to scan the input looking for the closing quote, handle backslash escapes, or do UTF-8 conversion. That's probably most of the gain right there, but more things can be done along those lines.

And no whitespace or curly braces taking up room, so the serialized data is smaller, and thus faster to transmit/store. Downside: Legibility? Future-proofing? Whats that?

Re: Saving Millions by Dumping Java Serialization

#36

Post by jnewhouse »

rst wrote: post #27
hrshtr wrote: post #15

Earlier quoted context omitted.

One reason on top of my head: Using such communication protocol would require changes to the other services consuming it.

So did switching to their homebrew serialization format -- in fact, most of the article is about how they managed the changes (which touched codebases at multiple sites in a fairly large organization).

Those switches all occurred at the pipeline level, leaving the map-reduce platform untouched. Switching our base logs to something like Parquet, Thrift or Protobuf would be a much larger project. We do support writing and reading Parquet to allow us to interface with other big data systems.

Re: Saving Millions by Dumping Java Serialization

#37

Post by coldtea »

MS_Buys_Upvotes wrote: post #14

Can someone explain to an amateur why serialization is faster than say passing raw JSON? It seems like parsing JSON would be faster than the serialize -> deserialize process but with the popularity of things like Protobuff it's clear that JSON is slower.

JSON is a serialization format itself. Even Javascript, which has JSON-like structures as native objects, has to serialize it (JSON.stringify) and deserialize it (JSON.parse) into text.

Re: Saving Millions by Dumping Java Serialization

#38

Post by coldtea »

chiph wrote: post #35
ScottBurson wrote: post #20

Earlier quoted context omitted.

For one thing, all numbers can be written in binary, saving the lexing and conversion time. For another, strings can be written by first writing the length (in binary, of course), then writing the raw contents; there's no need to scan the input looking for the closing quote, handle backslash escapes, or do UTF-8 conversion. That's probably most of the gain right there, but more things can be done along those lines.

And no whitespace or curly braces taking up room, so the serialized data is smaller, and thus faster to transmit/store. Downside: Legibility? Future-proofing? Whats that?

>Downside: Legibility? Future-proofing? Whats that?

There's nothing in this practice that is against future-proofing.

Legibility, yes, but those formats are not meant to be human readable.

Re: Saving Millions by Dumping Java Serialization

#39

Post by coldtea »

MS_Buys_Upvotes wrote: post #33
felixgallo wrote: post #18

Earlier quoted context omitted.

JSON must also be serialized or deserialized. Parsing it is slow and hard and not cache friendly. Protobuf has the benefit of being extremely compact and, in some important languages, fast and friendly to serialize and deserialize.

Thanks! You and the others are right: I didn't know JSON was serialized. I can see why something in binary would be faster than structured text (think assembler vs Python). Thanks again.

>I didn't know JSON was serialized.

Think of it like this: anytime you get stuff from the memory of your program (arrays, lists, strings, etc) and export it in a textual or binary format that can be exchanged between programs, moved over the network, saved to a file, etc, that's serialization.

Re: Saving Millions by Dumping Java Serialization

#40

Post by jnewhouse »

revscat wrote: post #26
jnewhouse wrote: post #2

Author here, let me know if you have any questions/want more details.

1) Can you provide any more details about how Rowfiles are structured and/or implemented? Specifically, how does it handle nested objects? Does it support `transient`? Do `writeObject` and/or `readObject` come into play? 2) Do you feel this is a generic enough solution that you would consider submitting it as a JSR?

It natively supports a limited set of Columns. Basically boxed primitives, java.util.Date, joda.time.DateTime, and arrays and double arrays of both boxed and unboxed versions of the preceding. The list of Columns being used is used to read and write to a byte buffer. The byte buffer is almost entirely the field's data, with one or two bytes describing how the subsequent field is encoded. Nested objects aren't handled out of the box, but there is the capability to define a UserRowField that allows for serialization/deserialization to bytes of any Serializable class. This gets used for our SQL map-reduce function a lot. The downside is that you need to have the UserRowField implementation in your classpath in order to read the Row, which is not generally the case.
Post reply on HN