Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

github.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#11

Post by wvh »

OzzyB wrote: post #8

My problem with serialization in Go isn't so much speed but marshalling/unmarshalling pain. I've been using these packages with some success: https://github.com/tidwall/sjson https://github.com/tidwall/gjson Not knocking this pkg, just thought I'd share xD

There are surprisingly many ways to do JSON in Go. There's reflection with the stdlib as well as keeping parts of it as "raw" byte slices; code generation with packages like ffjson and easyjson; practically hand-built with gojay; the approach of a state machine on a simple byte string like gjson and jsonparser; and then through templates like e.g. quicktemplate. The best approach really depends on what you want to do.

I think these sorts of challenges show that Go tends to be used in lower-level code than typical dynamic languages. I've combined three or four different approaches in several Go projects, which I couldn't imagine doing in a language like Python.

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#12

Post by soamv »

pelletier wrote: post #5
ilovecaching wrote: post #4

Earlier quoted context omitted.

Seconded, also an explanation of why it's faster on the landing page.

Tradeoffs are at https://github.com/segmentio/encoding/tree/master/json#trade... . Maybe we should move that upfront (or at least link to it for discovery).

Ah that answers it, thanks for posting that.

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#13

Post by loeg »

q3k wrote: post #3
soamv wrote: post #2

Looks neat, why not try to upstream it to the Go standard library if it's fully compatible?

Replacing a commonly used package like this likely wouldn't be without accidental compatibility breakage (in some edge cases), and as such I appreciate it not silently becoming the new default and potentially breaking software or corrupting data. This is especially true as JSON-the-standard (RFC7159) is horribly underspecified and as such it's difficult to even use some sort of acceptance suite to declare a replaceme…

Maybe there are ideas that can be used without changing the API surface at all? I don't think grandparent is proposing the Go developers just drop the thing in as-is without any evaluation of fitness or backwards compatibility. Part of the significance of this work is that (they claim) it is API-compatible with the stdlib version (modulo error messages, which can be addressed).

Edit: Then again, the authors advocate mildly against upstreaming it:

> For these reasons, we also don't believe that this code should be ported upsteam to the standard encoding/json package.

https://github.com/segmentio/encoding/tree/master/json#trade...

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#14

Post by iudqnolq »

bscphil wrote: post #7

One of the typical arguments in favor of small base languages is that frequently the best package will be written by a third party and become the "default", making the language bloated. For example, everyone uses Requests in Python instead of the built-in alternative. On the other hand, I think a big advantage of batteries-included languages that they define a standard implementation that other libraries can copy. Be…

Another good example is the python stdlib package json: ujson and cjson are drop in replacements that use c bindings to achieve much better performance. However, json is still very useful if you only need to parse/dump a reasonably small amount of data and don't want your users to need to build the c bindings.

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#15

Post by akira2501 »

q3k wrote: post #3
soamv wrote: post #2

Looks neat, why not try to upstream it to the Go standard library if it's fully compatible?

Replacing a commonly used package like this likely wouldn't be without accidental compatibility breakage (in some edge cases), and as such I appreciate it not silently becoming the new default and potentially breaking software or corrupting data. This is especially true as JSON-the-standard (RFC7159) is horribly underspecified and as such it's difficult to even use some sort of acceptance suite to declare a replaceme…

> This is especially true as JSON-the-standard (RFC7159) is horribly underspecified

Normally.. you would hope that the great test system built in to go would benefit stdlib upgrades like this and allow you do them more easily and with greater confidence.. but if what you're implementing suffers from this, then you pretty much stuck locking yourself into specific implementations rather than general specifications.

> but it's definitely in the Good Enough category

"Two steps forward, one step back" sums up how I feel about Go. It's still my favorite user-mode language, though.

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#17

Post by ploxiln »

kibwen wrote: post #9
bscphil wrote: post #7

One of the typical arguments in favor of small base languages is that frequently the best package will be written by a third party and become the "default", making the language bloated. For example, everyone uses Requests in Python instead of the built-in alternative. On the other hand, I think a big advantage of batteries-included languages that they define a standard implementation that other libraries can copy. Be…

> everyone uses Requests in Python I love Requests, but this statement overreaches. All of my Python these days is simple scripting, and in those simple cases the hassle of adding a dependency outweighs the utility that Requests provides over the standard library. And I'd say Requests is even a special case here in that it's widely known and acknowledged; while I wager you can find superior third-party replacements t…

yup, I've used stdlib urlopen in an assortment of scripts, it works, even with python2/python3 cross compatibility:

    if sys.version_info[0] == 2:
        from urllib2 import urlopen, Request  # python2
    else:
        from urllib.request import urlopen, Request  # python3

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#18

Post by slavik81 »

pelletier wrote: post #5
ilovecaching wrote: post #4

Earlier quoted context omitted.

Seconded, also an explanation of why it's faster on the landing page.

Tradeoffs are at https://github.com/segmentio/encoding/tree/master/json#trade... . Maybe we should move that upfront (or at least link to it for discovery).

That's just a file list on mobile. It's better to link to the README itself: https://github.com/segmentio/encoding/blob/master/json/READM...

Re: Segmentio/encoding: optimized drop-in replacement for Go's encoding/JSON package

#19

Post by CGamesPlay »

khc wrote: post #16

Now if there's a way to vendor built-in libraries...

Are you saying this so that you can exclude the built-in implementation from your binary? This is already done, the go compiler doesn't even look at the "built-in" implementation unless it's explicitly included in the code. You might have it included through a transitive dependency, but your dependencies can offer configuration like Gonic does. [0]

[0] https://github.com/gin-gonic/gin#build-with-jsoniter

Post reply on HN