Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Order of the JSON

blog.almaer.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: The Order of the JSON

#111

Post by coldtea »

jwilliams wrote: post #46

(I've used both DataPower and COBOL - I've got a healthy respect for robust, long-lived legacy systems). I must admit I was scratching my head on this. The JSON spec might not specify order, but the serialized JSON is ordered by nature. JWT needs it for example (and I'd assume many signature models). Or you might have a caching layer that needs it. Maybe unchecking this causes the legacy backend to get hammered? Ther…

>The JSON spec might not specify order, but the serialized JSON is ordered by nature. JWT needs it for example (and I'd assume many signature models). Or you might have a caching layer that needs it

Whether the file has an order is an implementation detail. There might be no static file at all, for example, it could be an endpoint returning you the JSON, and it could return different orders for the exact same items every time you call it. That would still be perfectly valid.

If you "need it", don't put your data into an object, put it in a list of objects.

If the underlying layers need "ordered objects" (in other words, ordered hashmaps) they don't really use JSON.

Re: The Order of the JSON

#112

Post by coldtea »

jwilliams wrote: post #67
moby_click wrote: post #62

Earlier quoted context omitted.

If you require serialized JSON in a specific order, you are not requiring JSON, but some dialect of JSON and that should be clearly communicated. If, for some reason, a client needed a special serialization, i'd ask them for a spec, because the JSON spec does not apply anymore, and be done with it.

I think my argument is a bit different -- sure, this is an interface (maybe to a COBOL copybook in this case?). And yes, it's not "JSON". Agree it's a leaky abstraction. Equally agree it should be documented. Often legacy systems have weirdness after seeing decades of edge-cases. Weirdness that makes them robust in all sorts of unlikely ways. Equally makes them poorly documented, leaky, opaque, and frustrating. But I…

>I think my argument is a bit different -- sure, this is an interface (maybe to a COBOL copybook in this case?). And yes, it's not "JSON".

If it's not JSON, it's undefined when it uses JSON tools and libraries, and anything goes.

If it's not JSON, document it as your own dialect, and use your own tools, and try to never talk to the outside world based on it with systems that treat it as JSON.

>But I certainly have a lot of pause with the sentiment of "some idiot checked the JSON order checkbox on DataPower". My first thought is instead "I wonder why someone thought this was necessary."

It might not be, it could be the BS default setting...

Re: The Order of the JSON

#113

Post by coldtea »

cat199 wrote: post #93
jwilliams wrote: post #67

Earlier quoted context omitted.

I think my argument is a bit different -- sure, this is an interface (maybe to a COBOL copybook in this case?). And yes, it's not "JSON". Agree it's a leaky abstraction. Equally agree it should be documented. Often legacy systems have weirdness after seeing decades of edge-cases. Weirdness that makes them robust in all sorts of unlikely ways. Equally makes them poorly documented, leaky, opaque, and frustrating. But I…

Money down 3 months later: 1) the gateway or more likely something behind it starts crashing for memory errors when someone is serializing some stupidly large array into it 2) using ordered vs non-ordered keys on input changes the behavior and fixes it 3) enabling this checkbox to prevent it fixes the issue and guards against it happening again 4) everyone has forgotten about this web weenies inquiry into the issue 5…

Money down:

1) Nothing happened, this already went down years ago, as the author writes.

2) The setting was only stopping the BS IBM system layer, wasn't really needed elsewhere.

3) The BS setting was probably even default, e.g. not consciously enabled by someone for the specific system.

>What is the point of this javadoc reference and how exactly does it relate to the issue at hand?

The point is that the IBM system who had this setting was using that BS, brain-dead, not-JSON, implementation to handle the order, when the setting was enabled.

Re: The Order of the JSON

#114

Post by coldtea »

jwilliams wrote: post #69
blattimwind wrote: post #47

Earlier quoted context omitted.

> JWT needs it for example (and I'd assume many signature models) You almost never need canonical representations for signing things. I would even say that if you need a canonical representation to sign your things, then that is a design smell of your cryptographic protocol.

I didn't mention canonicalization. My point is that serialized JSON is ordered - which I think is exactly the same property you're referring to.

That's an implementation detail. Serialized JSON can also be printed, but your systems shouldn't depend that JSON is always ink on paper.

Re: The Order of the JSON

#115

Post by quickthrower2 »

bjoli wrote: post #101
quickthrower2 wrote: post #55

Earlier quoted context omitted.

You had trouble writing a scheme macro? I think most pro software developers would struggle to! I don’t think this is dumb.

I do not have any problems writing macros (in fact, I recently finished my guile scheme version of rackets for loops) It is just that the macro I wrote did a lot more work than it had to, and led to weird syntax behaviour. I didn't delimit what I was trying to do. John over in #guile put me on the right track.

I didn’t mean to imply you had trouble writing all macros. But as someone who is trying to learn some c-lisp on the side I can see why they can get a lot more tricky than “regular” coding that one might do say in Java. So the point is you are probably doing something more advanced there.

Re: The Order of the JSON

#116

Post by atq2119 »

hyperpallium wrote: post #8
lacker wrote: post #6

I agree the system described is crazy. However, I find it frequently useful to use ordered JSON as a data format and I think it would be handy if more languages supported it. For one, it makes it a lot easier to write integration tests using a “golden file” of ideal output, because your program that outputs JSON now usually deterministically has one correct output. For two, it lets you hash a json-encoded object dete…

OK. I totally prefer ordered JSON, because it is so much easier to eyeball - to visually compare JSON with different ordered keys is quite a lot more difficult (O() complexity?) than if they are in the same order. It also enables diff to help see where the differences are (diagnosis, not just binary identical or not). And, in fact, I do use ordered JSON for comparison in testing, as you describe. However ... comparis…

Please note that using a sum of hashes almost certainly weakens any cryptographic guarantees you may expect from your hashes. Of course, that may be fine depending on your use case.

Re: The Order of the JSON

#117

Post by Spivak »

dtech wrote: post #87
Someone wrote: post #49

Next installment: how flipping an innocuous switch introduced a subtle bug that was silent for years and cost us millions. You can’t just flip that switch, run ”a client to post the JSON to that instance” , see that test “worked just fine” , and call it a day. The POST might just store it, for later processing to wreak havoc (say by ignoring a value that isn’t in the expected place in the JSON), or only rarely seen J…

How do you justify ever changing anything with that attitude? This change made parsing more lenient which is generally considered fine [1] and made the software actually follow the spec, since it specifies object entries are unordered. [1] https://en.wikipedia.org/wiki/Robustness_principle

The problem is that you have a complex interconnected system and you just flipped a switch that changed the entire protocol by which data was being transmitted.

OrderedJSON and JSON might seem compatible on the surface but are you sure you caught every little code path that might have been assuming OrderedJSON?

This isn't an issue of spec compliance or not because the data format wasn't JSON in the first place. OrderedJSON is not a subset of JSON even though it has the property that all OrderedJSON documents are syntactically valid JSON -- they represent different abstract values and JSON is the lossy interpretation.

Re: The Order of the JSON

#118

Post by Spivak »

Ididntdothis wrote: post #5

This sounds very familiar. A lot of companies are full of people who have no curiosity and no ability to think for themselves. I have seen it multiple times where someone claimed a change is impossible or takes insane effort. Then you have someone competent look at it and you have a solution in an hour. I think stuff likes this is the real price of not hiring really good people.

But this fix only (probably) works by accident. The problem is that flipping the switch makes the error go away in this system but ignores all the work of combing over the rest of the consumers of these documents to make sure they aren't subtly relying on the fact that the system was returning OrderedJSON instead of JSON.

Re: The Order of the JSON

#119

Post by Spivak »

coldtea wrote: post #111
jwilliams wrote: post #46

(I've used both DataPower and COBOL - I've got a healthy respect for robust, long-lived legacy systems). I must admit I was scratching my head on this. The JSON spec might not specify order, but the serialized JSON is ordered by nature. JWT needs it for example (and I'd assume many signature models). Or you might have a caching layer that needs it. Maybe unchecking this causes the legacy backend to get hammered? Ther…

> The JSON spec might not specify order, but the serialized JSON is ordered by nature. JWT needs it for example (and I'd assume many signature models). Or you might have a caching layer that needs it Whether the file has an order is an implementation detail. There might be no static file at all, for example, it could be an endpoint returning you the JSON, and it could return different orders for the exact same items…

This isn't really an issue of spec because these systems recognize a completely different protocol 'OrderedJSON' which interprets {} as an ordered list of key-value pairs instead of a set.

The mistake is assuming that two syntactically compatible protocols were actually the same protocol. uint =! int.

Re: The Order of the JSON

#120

Post by coldtea »

Spivak wrote: post #119
coldtea wrote: post #111

Earlier quoted context omitted.

> The JSON spec might not specify order, but the serialized JSON is ordered by nature. JWT needs it for example (and I'd assume many signature models). Or you might have a caching layer that needs it Whether the file has an order is an implementation detail. There might be no static file at all, for example, it could be an endpoint returning you the JSON, and it could return different orders for the exact same items…

This isn't really an issue of spec because these systems recognize a completely different protocol 'OrderedJSON' which interprets {} as an ordered list of key-value pairs instead of a set. The mistake is assuming that two syntactically compatible protocols were actually the same protocol. uint =! int.

Only there's no such protocol. It's some adhoc implementation within a single system (or a single IBM ecosystem).
Post reply on HN