Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twirp: A new RPC framework for Go

blog.twitch.tv

Post reply on HN
Print view View original

21–30 of 99 posts

Re: Twirp: A new RPC framework for Go

#21

Post by puzzle »

buckhx wrote: post #9
gipp wrote: post #6

Earlier quoted context omitted.

On GCP, Cloud Endpoints proxies will transparently translate back and forth between protobufs and the canonical JSON, allowing either representation to be used. So if you're on GCP and don't care about vendor lock-in, that's a solution.

There is also the grpc-gateway project for JSON transcoding https://github.com/grpc-ecosystem/grpc-gateway

To be fair, if you chose gRPC for performance, but then end up using JSON for most of your traffic, perhaps you picked the wrong tool.

Re: Twirp: A new RPC framework for Go

#22

Post by justinko »

tschellenbach wrote: post #17
spenczar5 wrote: post #12

Hey everyone! I'm the OP and primary author of Twirp. I'm happy to answer any questions and hear feedback. You can also reach me directly, if you like, email is in my profile.

Thanks for posting! How did you work around issues with GRPC on AWS/ELB?

We had to stick to layer 4/TCP via an ELB (as opposed to an ALB).

Re: Twirp: A new RPC framework for Go

#23

Post by gipp »

puzzle wrote: post #21
buckhx wrote: post #9

Earlier quoted context omitted.

There is also the grpc-gateway project for JSON transcoding https://github.com/grpc-ecosystem/grpc-gateway

To be fair, if you chose gRPC for performance, but then end up using JSON for most of your traffic, perhaps you picked the wrong tool.

I don't think generally people want to have JSON accepted to use in their production workloads. More for development, testing, that kind of thing. Being able to just curl your service makes a huge difference.

Re: Twirp: A new RPC framework for Go

#24

Post by cimnine »

spenczar5 wrote: post #12

Hey everyone! I'm the OP and primary author of Twirp. I'm happy to answer any questions and hear feedback. You can also reach me directly, if you like, email is in my profile.

Did you ever consider Cap'N Proto [1] as well? How does it compare? It looks like it could easily be integrated as a third transport encoding besides JSON and ProtoBuf.

[1] https://capnproto.org/

Re: Twirp: A new RPC framework for Go

#25

Post by puzzle »

gipp wrote: post #23
puzzle wrote: post #21

Earlier quoted context omitted.

To be fair, if you chose gRPC for performance, but then end up using JSON for most of your traffic, perhaps you picked the wrong tool.

I don't think generally people want to have JSON accepted to use in their production workloads. More for development, testing, that kind of thing. Being able to just curl your service makes a huge difference.

It always starts that way, then people demand JSON everywhere ("why not?"), then they complain when things get too slow or when the OOMs begin to appear. :-)

Re: Twirp: A new RPC framework for Go

#27

Post by spenczar5 »

cimnine wrote: post #24
spenczar5 wrote: post #12

Hey everyone! I'm the OP and primary author of Twirp. I'm happy to answer any questions and hear feedback. You can also reach me directly, if you like, email is in my profile.

Did you ever consider Cap'N Proto [1] as well? How does it compare? It looks like it could easily be integrated as a third transport encoding besides JSON and ProtoBuf. [1] https://capnproto.org/

Yeah, we looked at it a bit. To be honest, the biggest blocker was that we wrote Twirp as a gRPC alternative, so we already had protobuf service definitions in some spots, and it seemed easiest to keep using the same thing.

Adding another transport encoding would be good, but its important that any Twirp server can support all of the encodings. We would need to be able to map a message defined in capnproto to a protobuf message definition, which didn't seem completely trivial when we looked at it, since capnproto uses its own IDL, I believe.

I don't know a ton about capnproto, though, and I'm open to learning more about it. We would just need to work under the constraints that JSON and Protobuf requests would need to still work.

Re: Twirp: A new RPC framework for Go

#28

Post by lobster_johnson »

This looks promising! We use the go-grpc SDK in conjunction with gogoprotobuf, and it's been a rocky road.

While the article identifies some operational issues (e.g. the reliance on HTTP/2), there are several considerable deficiencies with gRPC today, at least when using it with Go:

1. The JSON mapping (jsonpb.go) is clumsy at best, and by this I mean that it produces JSON that often doesn't look anything like how you'd hand-design your structures. "oneof" structs, for example, generate an additional level of indirection that you typically wouldn't have. Proto3's decision to forego Proto2's optional values (in Proto3 everything is optional) cause Go's zero value semantics to leak into gRPC [1]. (We had to fork jsonpb.go to fix some of these issues, but as far as I can tell, upstream is still very awkward.)

2. The Go code generator usually produces highly unidiomatic Go. "oneof" is yet again an offender here. The gogoprotobuf [2] project tries to fix some of go-grpc's deficiencies, but it's still not sufficient. Ideally you should be able to use the Proto structs directly, but our biggest gRPC project we basically gave up here, and decided to limit Proto usage to the server layer, with a translation layer in between that translates all the Proto structs to/from native structs. That keeps things clean, but it's pretty exhausting work, which lots of type switches (which are hampered by Go's lack of switch exhaustiveness checking; we use BurntSushi's go-sumtype [3] a lot, but I don't think it can work for Proto structs, as it requires that a struct also implements an interface).

3. Proto3 has very limited support for expressing "free-form" data. By this I mean if you need to express a Protobuf field that contains a structured set of data such as {"foo": {"bar": 42}}. For this, you have the extension google.protobuf.Value [4], which supports some basic primitives, but not all (no timestamps, for example) and cannot be used to serialize actual gRPC messages; you can't serialize {"foo": MyProtoMessage{...}}. Free-form structured data is important for systems that accept foreign data where the schema isn't known; for example, a system that indexes analytics data.

From what I can tell, though, Twirp doesn't "disrupt" gRPC as much as I'd like, since it appears to rely on the existing JSON mapping.

[1] https://github.com/gogo/protobuf/issues/218

[2] https://github.com/gogo/protobuf

[3] https://github.com/BurntSushi/go-sumtype

[4] https://developers.google.com/protocol-buffers/docs/referenc...

Re: Twirp: A new RPC framework for Go

#29

Post by joshuak »

I think the work the gRPC contributors are doing is great, including all the features. But I can't emphasize enough how important it is for projects like this that take great ideas to a new level by prioritizing simplicity. It's like one project brainstorms great ideas by not being too resistant to new ones (the "yes, and..." rule), while the other refines the ideas with a focus on simplicity to extract the greatest value for the cost.

Great work.

Post reply on HN