31–40 of 69 posts
That “compact JSON” format reminds me if the special protobufs JSON format that Google uses in their APIs that has very little public documentation. Does anyone happen to know why Google uses that, and to OP, were you inspired by that format?
Impressive. Some interop with established standards such as OpenAPI or gRPC would make it an easier sell for non-greenfield projects
Earlier quoted context omitted.
> Minor thing that bothered me disproportionately: the constant syntax in the docs (x = 600) doesn't match what the parser actually accepts (x: 600). You’re a better man than me. If the docs can’t even get the syntax right, that’s a hard no from me. Also, fwiw, you’ve got a few points wrong about protos. Inspecting the binary data is hard, but the tag numbers are present. You need the schema, but at least you can ide…
> Also, I disagree on the constructor front. Proto forces you to grapple with the reality that a field may be missing. In a production system, when adding a new field, there will be a point where that field isn’t present on only one side of the network call. The compiler isn’t saving you. I agree it's important for users to understand that newer fields won't be set when they deserialize old data -- whether that's wit…
Some callers may not need to update right away, or don’t need the new feature at all, and breaking the existing callers compilation is bad. If your caller is a different team, for example, and their CICD breaks because you added a field, that’s bad. Each place it’s used, you should think about how it’ll be handled, BUT ALSO, your system explicitly should gracefully handle the case where it’s not uniformly present. It’s an explicit goal of protos to support the use case where heterogeneous schema versions are used over the wire.
If a bug is introduced because the caller and handler use different versions, the compiler wasn’t going to save you anyways. That bug would have shown up when you deploy or update the client and server anyways - unless you atomically update both at once. You generally cannot guarantee that a client won’t use an outdated version of the schema, and if things break because of that, you didn’t guard it correctly. That’s a business logic failure not a compilation failure.
1. Dense json
Interesting idea. You can also just keep the compact binary if you just tag each payload with a schema id (see Avro). This also allows a generic reader to decode any binary format by reading the schema and then interpreting the binary payload, which is really useful. A secondary benefit is you never ever misinterpret a payload. I have seen bugs with protobufs misinterpreted since there is no connection handshake and interpretation is akin to 'cast'.
2. Compatibility checks
+100 there's not reason to allow breaking changes by default
3. Adding fields to a type: should you have to update all call sites?
I'm not so sure this is the right default. If I add a field to a core type used by 10 services, this requires rebuilding and deploying all of them.
4. enum looks great. what about backcompat when adding new enum fields? or sometimes when you need to 'upgrade' an atomic to an enum?
> Skir is a universal language for representing data types, constants, and RPC interfaces. Define your schema once in a .skir file and generate idiomatic, type-safe code in TypeScript, Python, Java, C++, and more. Maybe I'm missing some additional features but that's exactly what https://buf.build/plugins/typescript does for Protobuf already, with the advantage that you can just keep Protobuf and all the battle harde…
The entire original post, it seems, is dedicated to explaining why Skir is better than plain Protobuf, with examples of all the well-known pain points. If these are not persuasive for you, staying with Protobuf (or just JSON) should be a fine choice.
Earlier quoted context omitted.
The entire original post, it seems, is dedicated to explaining why Skir is better than plain Protobuf, with examples of all the well-known pain points. If these are not persuasive for you, staying with Protobuf (or just JSON) should be a fine choice.
[flagged]
Did you look at other formats like Avro, Ion etc? Some feedback: 1. Dense json Interesting idea. You can also just keep the compact binary if you just tag each payload with a schema id (see Avro). This also allows a generic reader to decode any binary format by reading the schema and then interpreting the binary payload, which is really useful. A secondary benefit is you never ever misinterpret a payload. I have seen…
0. Yes, I looked at Avro, Ion. I like Protobuf much better because I think using field numbers for field identity, meaning being able to rename fields freely, is a must.
1. Yes. Skir also supports that with binary format (you can serialize and deserialize a Skir schema to JSON, which then allows you to convert from binary format to readable JSON). It just requires to build many layers of extra tooling which can be painful. For example, if you store your data in some SQL engine X, you won't be able to quickly visualize your data with a simple SELECT statement, you need to build the tooling which will allow you to visualize the data. Now dense JSON is obviously not idea for this use case, because you don't see the field names, but for quick debugging I find it's "good enough".
3. I agree there are definitely cases where it can be painful, but I think the cases where it actually is helpful are more numerous. One thing worth noting is that you can "opt-out" of this feature by using `ClassName.partial(...)` instead of `ClassName()` at construction time. See for example `User.partial(...)` here: https://skir.build/docs/python#frozen-structs I mostly added this feature for unit tests, where you want to easily create some objects with only some fields set and not be bothered if new fields are added to the schema.
4. Good question. I guess you mean "forward compatibility": you add a new field to the enum, not all binaries are deployed at the same time, and some old binary encounters the new enum it doesn't know about? I do like Protobuf does: I default to the UNKNOWN enum. More on this: - https://skir.build/docs/schema-evolution#adding-variants-to-... - https://skir.build/docs/schema-evolution#default-behavior-dr... - https://skir.build/docs/protobuf#implicit-unknown-variant
Earlier quoted context omitted.
[flagged]
If you are fine enough with protobufs so that you're not actively looking for alternatives, maybe you should not spend the effort.
Copying from blog post [https://medium.com/@gepheum/i-spent-15-years-with-protobuf-t...]:
""" Should you switch from Protobuf?
Protobuf is battle-tested and excellent. If your team already runs on Protobuf and has large amounts of persisted protobuf data in databases or on disk, a full migration is often a major effort: you have to migrate both application code and stored data safely. In many cases, that cost is not worth it.
For new projects, though, the choice is open. That is where Skir can offer a meaningful long-term advantage on developer experience, schema evolution guardrails, and day-to-day ergonomics. """
Did you look at other formats like Avro, Ion etc? Some feedback: 1. Dense json Interesting idea. You can also just keep the compact binary if you just tag each payload with a schema id (see Avro). This also allows a generic reader to decode any binary format by reading the schema and then interpreting the binary payload, which is really useful. A secondary benefit is you never ever misinterpret a payload. I have seen…
Thanks for the feedback. 0. Yes, I looked at Avro, Ion. I like Protobuf much better because I think using field numbers for field identity, meaning being able to rename fields freely, is a must. 1. Yes. Skir also supports that with binary format (you can serialize and deserialize a Skir schema to JSON, which then allows you to convert from binary format to readable JSON). It just requires to build many layers of extr…
avro supports field renames though.
3. on second thought i believe you'd only have to deploy when you choose. the next build will force you to provide values (or opt into the default). so forcing inspection of construction sites seems good.