Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I stopped using JSON for my APIs

aloisdeniel.com

Post reply on HN
Print view View original

101–110 of 245 posts

Re: Why I stopped using JSON for my APIs

#101

Post by MobiusHorizons »

almosthere wrote: post #92
morshu9001 wrote: post #44

Earlier quoted context omitted.

I've gone the all-JSON route many times, and pretty soon it starts getting annoying enough that I lament not using protos. I'm actually against static types in languages, but the API is one place they really matter (the other is the DB). Google made some unforced mistakes on proto usability/popularity though.

why are you against static types in languages? I once converted a fairly large JS codebase to TS and I found about 200 mismatching names/properties all over the place. Tons of properties we had nulls suddenly started getting values.

Sounds like this introduced behavior changes. How did you evaluate if the new behavior was desirable or not? I’ve definitely run into cases where the missing fields were load bearing in ways the types would not suggest, so I never take it for granted that type error in prod code = bug

Re: Why I stopped using JSON for my APIs

#102

Post by ablob »

klysm wrote: post #70
dfabulich wrote: post #68

> With JSON, you often send ambiguous or non-guaranteed data. You may encounter a missing field, an incorrect type, a typo in a key, or simply an undocumented structure. With Protobuf, that’s impossible. Everything starts with a .proto file that defines the structure of messages precisely. This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-pra…

It’s also conflating the serialization format with contracts

I feel like that's fine since both things go hand in hand anyway. And if choosing the JSON-format comes with a rather high amount of contract-breaches it might just be easier to switch that instead of fixing the contract.

Re: Why I stopped using JSON for my APIs

#103

Post by pornel »

JoshMock wrote: post #64

Protobuf is a great format with a lot of benefits, but it's missing one that I wish it could support: zero-copy. The ability to transport data between processes, services and languages with effectively zero time spent on serialization and deserialization. It appears possible in some cases but it's not universally the case. Which means that similar binary transport formats that do support zero-copy, like Cap'n Proto,…

It depends how you actually use the messages. Zero-copy can be slowing things down. Copying within L1 cache is ~free, but operating on needlessly dynamic or suboptimal data structures can add overheads everywhere they're used.

To actually literally avoid any copying, you'd have to directly use the messages in their on-the-wire format as your in-memory data representation. If you have to read them many times, the extra cost of dynamic getters can add up (the format may cost you extra pointer chasing, unnecessary dynamic offsets, redundant validation checks and conditional fallbacks for defaults, even if the wire format is relatively static and uncompressed). It can also be limiting, especially if you need to mutate variable-length data (it's easy to serialize when only appending).

In practice, you'll probably copy data once from your preferred in-memory data structures to the messages when constructing them. When you need to read messages multiple times at the receiving end, or merge with some other data, you'll probably copy them into dedicated native data structs too.

If you change the problem from zero-copy to one-copy, it opens up many other possibilities for optimization of (de)serialization, and doesn't keep your program tightly coupled to the serialization framework.

Re: Why I stopped using JSON for my APIs

#104

Post by adastra22 »

brabel wrote: post #15

Mandatory comment about ASN.1, a protocol from 1984, already did what Protobuf does, with more flexibility. Yes, it's a bit ugly but if you stick to the DER encoding it's really not worse than Protbuf at all. Check out the Wikipedia example: https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.

ASN.1 is a nightmare, and I would never use it for a greenfield project.

Re: Why I stopped using JSON for my APIs

#105

Post by peterkelly »

ablob wrote: post #102
klysm wrote: post #70

Earlier quoted context omitted.

It’s also conflating the serialization format with contracts

I feel like that's fine since both things go hand in hand anyway. And if choosing the JSON-format comes with a rather high amount of contract-breaches it might just be easier to switch that instead of fixing the contract.

Unless a violation of that contract can lead to a crash or security vulnerability...

Re: Why I stopped using JSON for my APIs

#106

Post by lanigone »

catchmeifyoucan wrote: post #13

I wonder if we can write an API w/ JSON the usual way and change the final packaging to send it over protobuf.

envoy (the proxy) can transcode RESTful APIs to internal grpc services and vice versa. you can even map like url params etc. to proto fields. it works well, even server streaming.

Re: Why I stopped using JSON for my APIs

#108

Post by cryptonector »

brabel wrote: post #15

Mandatory comment about ASN.1, a protocol from 1984, already did what Protobuf does, with more flexibility. Yes, it's a bit ugly but if you stick to the DER encoding it's really not worse than Protbuf at all. Check out the Wikipedia example: https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.

Thank you! Now I don't have to be the one saying this. Props if you use OER over DER. But since OP needs available tooling they might as well go to flatbuffers, which is much better than PB much like OER is much better than DER.

Re: Why I stopped using JSON for my APIs

#109

Post by cryptonector »

adastra22 wrote: post #104
brabel wrote: post #15

Mandatory comment about ASN.1, a protocol from 1984, already did what Protobuf does, with more flexibility. Yes, it's a bit ugly but if you stick to the DER encoding it's really not worse than Protbuf at all. Check out the Wikipedia example: https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.

ASN.1 is a nightmare, and I would never use it for a greenfield project.

I... wouldn't use it for a greenfield project either unless I got good at porting Luke Howard's Swift ASN.1 stack to whatever language I might be using that isn't C. For C I'd just use Heimdal's awesome ASN.1 compiler and be done. Even then I would be tempted to use flatbuffers instead, or else I'd have to go implement OER (a bunch of work I don't really care to do).

The problem with ASN.1 -- the only real problem with ASN.1, is lack of excellent tooling.

Re: Why I stopped using JSON for my APIs

#110

Post by cryptonector »

morshu9001 wrote: post #52
zzo38computer wrote: post #51

Earlier quoted context omitted.

I do not agree. Which parts are necessary depends on the application; there is not one good way to do for everyone (and Protobuf is too limited). You will need to implement the parts specific to your schema/application on each end, and if the format does not have the data types that you want then you must add them in a more messy way (especially when using JSON).

In what ASN1 application is protobuf spec too limited? I've used protobuf for tons of different things, it's always felt right. Though I understand certain encodings of ASN1 can have better performance for specific things.

Open types, constrained types, parameterized types, not needing tags, etc.
Post reply on HN