Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prototool – A Swiss Army Knife for Protocol Buffers

github.com

Post reply on HN
Print view View original

11–20 of 66 posts

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#11

Post by akhilcacharya »

puzzle wrote: post #5
throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…

Should you really be detailing the functionality of internal tools like this?

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#12

Post by vinkelhake »

zellyn wrote: post #6
throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release. Looks like some parts of it have escaped… https://github.com/eclesh/recordio

If you were interested in RecordIO, then this project might also be of interest to you: https://github.com/google/riegeli

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#14

Post by puzzle »

kodablah wrote: post #9
puzzle wrote: post #5

Earlier quoted context omitted.

Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…

> Kubernetes: canaries and automatic rollbacks Hot off the presses: https://cloudplatform.googleblog.com/2018/04/introducing-Kay... . Though you have to use Spinnaker.

That's an external controller, which is what most people are doing themselves these days, reinventing the wheel each time. Borg has long had an automatic rollback feature on updates, tuned through a few settings on top of the health check machinery. I'm in the camp believing that a basic implementation should be built-in, since health checks are already there. An implementation of this was started, but it has stalled. External controllers should be still allowed, for more advanced scenarios like rolling back upon detection of regressions in latency and similar.

Also, setting up Spinnaker is pretty much as complicated as Kubernetes itself. :-)

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#15

Post by grandinj »

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

You could add a TableEngine extension to H2 (h2database.com), pretty easily which would give you full SQL query functionality over such a file

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#16

Post by zellyn »

vinkelhake wrote: post #12
zellyn wrote: post #6

Earlier quoted context omitted.

When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release. Looks like some parts of it have escaped… https://github.com/eclesh/recordio

If you were interested in RecordIO, then this project might also be of interest to you: https://github.com/google/riegeli

Interesting. I wonder how different that is from RecordIO. Also, whether there'll be a Go implementation.

[Edit, after looking a bit.]

Pretty different. If I remember correctly, RecordIO is re-synchronizing, whereas Riegeli seems to break things up into 64KB chunks, splitting messages across chunks if necessary.

[Edit, after finding more information.]

Interesting… looks like Riegeli is intended to compress well, rather than just store sequentially. https://encode.ru/threads/2895-Riegeli-%E2%80%94-a-new-compr...

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#17

Post by QuercusMax »

akhilcacharya wrote: post #11
puzzle wrote: post #5

Earlier quoted context omitted.

Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…

Should you really be detailing the functionality of internal tools like this?

None of this is particularly secret. Any intern who worked at Google would have access to all this info.

(Source: I'm a SWE at an Alphabet company)

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#18

Post by throwaway84742 »

zellyn wrote: post #16
vinkelhake wrote: post #12

Earlier quoted context omitted.

If you were interested in RecordIO, then this project might also be of interest to you: https://github.com/google/riegeli

Interesting. I wonder how different that is from RecordIO. Also, whether there'll be a Go implementation. [Edit, after looking a bit.] Pretty different. If I remember correctly, RecordIO is re-synchronizing, whereas Riegeli seems to break things up into 64KB chunks, splitting messages across chunks if necessary. [Edit, after finding more information.] Interesting… looks like Riegeli is intended to compress well, rath…

IIRC (but memory has faded considerably), RecordIO also did support something to aid compression across records (rather than just offer per-record compression). There was some gnarly code in it to that effect where there could be a compressed subset of several records within the file. But I might be wrong.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#19

Post by dekhn »

zellyn wrote: post #6
throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release. Looks like some parts of it have escaped… https://github.com/eclesh/recordio

TFRecords are the closest thing to recordio that has Google support.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#20

Post by puzzle »

akhilcacharya wrote: post #11
puzzle wrote: post #5

Earlier quoted context omitted.

Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…

Should you really be detailing the functionality of internal tools like this?

Most of this is public (see the Slicer paper for dynamic sharding) or implemented in some other form by Envoy and other tools (dark launches, traffic visualization) or not that complicated (Kubernetes has manual rollbacks already and automating them is a natural extension). None of those are Spanner territory. Even LOAS is getting discussed and open sourced.
Post reply on HN