Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Order of the JSON

blog.almaer.com

Post reply on HN
Print view View original

31–40 of 126 posts

Re: The Order of the JSON

#31

Post by j88439h84 »

AgentOrange1234 wrote: post #19
rurban wrote: post #17

Writers should produce sorted maps, readers need to accept unsorted maps. It's a security issue, not just convenience. With unsorted maps the internal hash seed can be exposed, together with timing information. Another famous omission from the specs.

I’m no security expert. How is “exposing the hash seed” a problem for the vast majority of applications? What timing information would be leaked and why would that be problem? On the other hand, accepting unsorted maps seems like it could introduce covert channels?

https://bugzilla.redhat.com/show_bug.cgi?id=750555

This led to hash randomization on by default since Python 3.3.

Re: The Order of the JSON

#32

Post by ken »

lacker wrote: post #6

I agree the system described is crazy. However, I find it frequently useful to use ordered JSON as a data format and I think it would be handy if more languages supported it. For one, it makes it a lot easier to write integration tests using a “golden file” of ideal output, because your program that outputs JSON now usually deterministically has one correct output. For two, it lets you hash a json-encoded object dete…

Ordering is part of the view, then, not the model. This is conflating different abstraction layers.

Similarly, sometimes it might be useful to store some data in a different character encoding, or maybe multi-character glyphs are stored in a different normalization form. (I can't tell "ü" from "ü" just by looking.) Or maybe your sample data has 1.0 but your program generates 1.000. There's a million ways that serialized structures can be functionally identical but quite different. Easy: don't compare raw bytes.

If you want functionally-equivalent data to be accepted, you just need an equality-tester (and hashing algorithm) that's agnostic to such issues. They're not hard to write.

Re: The Order of the JSON

#33

Post by eridius »

mlyle wrote: post #29
eridius wrote: post #26

Earlier quoted context omitted.

Please XOR your hashes instead of adding them! If you add them, you're losing bits on the low end. EDIT: No you're not. It feels like you should be, but with unsigned overflow, this actually works just fine. This assumes of course that you're using proper hashes that make use of the full domain of the output type (a proper hash will have a 50% chance of any arbitrary bit being flipped by any change to the input). But…

Can you please explain this assertion? ;) If I have a 32 bit current hash value-- for any possible 32 bit value I add, I get a different 32 bit value out. XORing is effectively adding each bit and throwing away the carry bit. Adding just cascades carries to the left.

You know what, you're right. I made a knee-jerk comment but I didn't think it through all the way. From any arbitrary unsigned 32-bit integer, every other unsigned 32-bit integer is reachable with a single addition. Therefore addition works just fine here.

It still feels wrong to say this, it feels like since adding will effectively shove bits off the high end and drop them on the floor that you're losing information, but I can't actually justify that feeling with reasoning.

Re: The Order of the JSON

#34

Post by mlyle »

eridius wrote: post #33
mlyle wrote: post #29

Earlier quoted context omitted.

Can you please explain this assertion? ;) If I have a 32 bit current hash value-- for any possible 32 bit value I add, I get a different 32 bit value out. XORing is effectively adding each bit and throwing away the carry bit. Adding just cascades carries to the left.

You know what, you're right. I made a knee-jerk comment but I didn't think it through all the way. From any arbitrary unsigned 32-bit integer, every other unsigned 32-bit integer is reachable with a single addition. Therefore addition works just fine here. It still feels wrong to say this, it feels like since adding will effectively shove bits off the high end and drop them on the floor that you're losing information…

Adding is actually considerably better. For high quality hashes XOR is just as good; but if there's any distributional problems at all in the hash, adding mixes stuff more.

(XORing is effectively adding with all of the carry information lost/falling off).

Re: The Order of the JSON

#35

Post by shortercode »

lilyball wrote: post #27

The tweet at the top of the article says > This so far out of the spec it makes my ankles hurt. This is not in fact out of spec. The JSON spec does not define semantics here but instead quite explicitly leaves it up to the JSON processor and data interchange spec for what to do about ordering of objects.

If the objects are not guaranteed to be in order, I would always consider them unordered. You could if both the producer and consumer are ordered, but as the OP noted this can cause problems later on and for no good reason. I'd use an array of key-value tuples if I wanted it to be ordered.

Re: The Order of the JSON

#36

Post by ajna91 »

bjoli wrote: post #30
Ididntdothis wrote: post #5

This sounds very familiar. A lot of companies are full of people who have no curiosity and no ability to think for themselves. I have seen it multiple times where someone claimed a change is impossible or takes insane effort. Then you have someone competent look at it and you have a solution in an hour. I think stuff likes this is the real price of not hiring really good people.

I had the same thing happen to me lately, but I was at the stupid end. I am not a programmer by profession, so it matters very little to me, but I had been struggling to produce a correct solution to a seemingly simple problem (writing a macro to allow definitions in expression context in r6rs scheme). My solution worked but had the side-effect rewriting obviously bad syntax into correct one and not in a good way. I…

A very humane telling of a humbling experience.

Re: The Order of the JSON

#37

Post by perfunctory »

> How does world not break due to technology more often

My own theory is that most (not all) of technology is there to support BS jobs and they are irrelevant for the normal functioning of society. It simply doesn't matter when (most) technology breaks.

Re: The Order of the JSON

#38

Post by trilila »

bjoli wrote: post #30
Ididntdothis wrote: post #5

This sounds very familiar. A lot of companies are full of people who have no curiosity and no ability to think for themselves. I have seen it multiple times where someone claimed a change is impossible or takes insane effort. Then you have someone competent look at it and you have a solution in an hour. I think stuff likes this is the real price of not hiring really good people.

I had the same thing happen to me lately, but I was at the stupid end. I am not a programmer by profession, so it matters very little to me, but I had been struggling to produce a correct solution to a seemingly simple problem (writing a macro to allow definitions in expression context in r6rs scheme). My solution worked but had the side-effect rewriting obviously bad syntax into correct one and not in a good way. I…

Great engineers learn from mistakes, and it seems like you did. Formal training or not, we all make mistakes, but indeed a good approach is to take a step back and analyse _any_ issue or complex problem before implementing it. Sometimes prototyping separate from the main codebase helps.

Re: The Order of the JSON

#39

Post by firefoxd »

Ha. Creating a layer on top of broken APIs is an thriving business. Every single carrier company has a broken API. That's why companies like aftership exist.

One example I struggled with for days recently was USPS. Not only they use xml in the url parameter, the order of the elements also matters. Unfortunately, the order in the documentation is incorrect.

Re: The Order of the JSON

#40

Post by dnautics »

firefoxd wrote: post #39

Ha. Creating a layer on top of broken APIs is an thriving business. Every single carrier company has a broken API. That's why companies like aftership exist. One example I struggled with for days recently was USPS. Not only they use xml in the url parameter, the order of the elements also matters. Unfortunately, the order in the documentation is incorrect.

Hell even Amazon's ec2 spec is confusing in some places (the boto python call parameter doesn't always match the restful call, which is less documented, leading to errors in some libraries).
Post reply on HN