Prototool – A Swiss Army Knife for Protocol Buffers
1–10 of 66 posts
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#2In another decade or so the world might replicate half of the very nice internal tools Google has.
Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple aggregations. That was pretty handy. I really wish Google would open source some or most of this stuff. It’s not like keeping it closed source creates any kind of insurmountable competitive advantage, especially compared to the advantages that would accrue from broader adoption of protobufs.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#3In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
You might be interested in KSQL, SQL queries that run on Kafka streams. https://www.confluent.io/product/ksql/
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#4In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
You might be interested in KSQL, SQL queries that run on Kafka streams. https://www.confluent.io/product/ksql/
Nah. I’m interested in quickly querying on-disk data specifically, ie proto-based application logs and the like (another thing the world needs to adopt more broadly imo).
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#5In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
Other tools and features that don't exist outside:
- a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should.
- load balancing dashboards showing traffic between frontends and backends
- load balancer support for dynamic sharding
- gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!msg/chrom... (I think most of this is doable these days, but the initial setup requires a Linux system)
- Kubernetes: server-specific custom hyperlinks on dashboards (e.g. links to POD_IP:PORT/stats, /debug, etc. for each individual pod you are looking at)
- Kubernetes: multiple Docker images in the same container or pod. E.g. the first container could be your code, while the second one might be data or the JVM runtime, etc., without having to bundle them together or doing costly copies in init containers.
- Kubernetes: canaries and automatic rollbacks
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#6In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release.
Looks like some parts of it have escaped… https://github.com/eclesh/recordio
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#7Earlier quoted context omitted.
You might be interested in KSQL, SQL queries that run on Kafka streams. https://www.confluent.io/product/ksql/
Nah. I’m interested in quickly querying on-disk data specifically, ie proto-based application logs and the like (another thing the world needs to adopt more broadly imo).
Of note, Prototool has a binary-to-json command, so assuming your Protobuf files are in path/to/proto/files, and you had a newline-separated log file of Protobuf messages foo.bar.Baz, you could do:
cat /path/to/log.file | prototool binary-to-json path/to/proto/files foo.bar.Baz - | jq .search.term
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#8Says "Handle installation of protoc [...] behind the scenes in a platform-independent manner without any work on the part of the user", doesn't support Windows yet [0]. Granted, as pre-1.0 I should probably read the features as goals.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#9In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…
> Kubernetes: canaries and automatic rollbacks
Hot off the presses: https://cloudplatform.googleblog.com/2018/04/introducing-Kay.... Though you have to use Spinnaker.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#10In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…
> - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should.
Envoy can do this, via its shadowing feature. See the docs here: https://www.envoyproxy.io/docs/envoy/v1.6.0/api-v2/api/v2/ro....