If it has to be mangled to such an extent to do this, then it seems reasonable to assume JSON is the wrong format for the task. Better to rethink it from scratch instead of trying to put a square peg in a round hog.
[deleted]
Progressive JSON
11–20 of 244 posts
Re: Progressive JSON
#1299.9999%* of apps don't need anything nearly as 'fancy' as this, if resolving breadth-first is critical they can just make multiple calls (which can have very little overhead depending on how you do it).
* I made it up - and by extension, the status quo is 'correct'.
Re: Progressive JSON
#1399.9999%* of apps don't need anything nearly as 'fancy' as this, if resolving breadth-first is critical they can just make multiple calls (which can have very little overhead depending on how you do it). * I made it up - and by extension, the status quo is 'correct'.
We technically didn't need more than 640K either.
Having progressive or partial reads would dramatically speed up applications, especially as we move into an era of WASM on the frontend.
A proper binary encoded format like protobuf with support for partial reads and well defined streaming behavior for sub message payloads would be incredible.
It puts more work on the engineer, but the improvement to UX could be massive.
Re: Progressive JSON
#14I feel like in an ideal world, this would start in the DB: your query referencing objects and in what order to return them (so not just a bunch of wide rows, nor multiple separate queries) and as the data arrives, the back end could then pass it on to the client.
Many db has done that, 4d.com is one that comes to mind. It is kinda like socket.io + PostgreSQL + node/ruby/php (middleware layer) all in one. In db there is also concept of cursor, etc.
Seems like it is never about merit of technological design. As some CS professor put it, tech is more about fashion than tech now days. IMHO that is true, and often also comes down to the technological context surrounding the industry at the time, and now days if the code is open sourced/FOSS.
Re: Progressive JSON
#15I understand the GraphQL has fallen out of favour somewhat, but wasn’t it intended to solve for this?
Re: Progressive JSON
#16Very cool point, and it applies to any tree data in general. I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers. Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the v…
... It might be a pursuit worth making a small library for.
Re: Progressive JSON
#17This feels very similar to JSON API links[0]. This is a great way to implement handling resolving those links on the frontend though.
Re: Progressive JSON
#18Very cool point, and it applies to any tree data in general. I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers. Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the v…
You can even alternate between sending table and node chunks! This will effectively allow you to reveal the tree in any order including revealing children before parents as well as representing arbitrary graphs! Could lead to some interesting applications.
Re: Progressive JSON
#19This would be good.
I got really, really sick of XML, but one thing that XML parsers have always been good at, is realtime decoding of XML streams.
It is infuriating, waiting for a big-ass JSON file to completely download, before proceeding.
Also JSON parsers can be memory hogs (but not all of them).
Re: Progressive JSON
#20I understand the GraphQL has fallen out of favour somewhat, but wasn’t it intended to solve for this?
For serialization GraphQL uses ... JSON.
GraphQL could use Progressive JSON to serialize subscriptions.