I can't use something with a CoC.
Why not?
Prototool – A Swiss Army Knife for Protocol Buffers
21–30 of 66 posts
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#22Earlier quoted context omitted.
When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release. Looks like some parts of it have escaped… https://github.com/eclesh/recordio
If you were interested in RecordIO, then this project might also be of interest to you: https://github.com/google/riegeli
Pretty neat work. Especially the corruption detection/skipping and seek support. That part was super ugly in RecordIO proper, and relied on stars properly aligning and the absence of cosmic radiation. RecordIO being the default format for everything at Google, it's not something they can really fix though. Taking care of concatenation is a nice touch as well.
As someone who has spent quite a bit of time at Google working on a high performance file format (not RecordIO):
1. I'd also add LZ4 and/or Snappy for the cases where they are more Pareto-optimal (i.e. fast, network attached, remote storage, such as SSD Colossus, or its external proxy: SSD Persistent Disk).
2. IMO HighwayHash is overkill here, and the author should have used CRC32C instead. You don't particularly care about collisions in this case, you're detecting data corruption. CRC32C is perfect for that, and it's hardware accelerated in almost all recent Intel and ARM CPUs, and it's half the size on disk.
3. It'd be pretty cool to introduce some kind of metadata which would tell the user what type of message is encoded in the file. This is not something RecordIO has, but internal tools can guess most of the time because they have all the proto definitions at their disposal. There's no need to store it in every header, just the first one. I would advise against storing the full schema (that can get very gnarly in the presence of proto dependencies and extensions), but just have something lightweight, i.e. message name and perhaps SCM revision number or hash in the file header, so that the user (or the external system consuming the files) could somewhat reliably establish what the format is later on, when the proto definition drifts. Otherwise, this being a binary serialized file format, it's very easy to end up in a situation where you have some files from years ago and you no longer know how to read them. And yes, I'm aware that SCM hash can change if history is edited.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#23Earlier quoted context omitted.
Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…
> - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. Envoy can do this, via its shadowing feature. See the docs here: https://www.envoyproxy.io/docs/envoy/v1.6.0/api-v2/api/v2/ro... .
Great! I suspected it had to be in there, but it was hard to figure that through the docs. I didn't think of 'shadow' as a keyword, only 'mirror'. Thanks. One more reason to adopt Envoy and Istio.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#24We built https://github.com/GyroscopeHQ/grpcat at my company, which takes text-format protos as input and sends them to a gRPC endpoint. Looking at Prototool I think I should just merge the functionality into Prototool. This is cool!
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#25Re: Prototool – A Swiss Army Knife for Protocol Buffers
#26I can't use something with a CoC.
this seems like it's only relating to people wishing to contribute to prototool. also, it's uber, so it's nice/expected that they would have this sort of thing.
also the code is basically about not being a jerk to other people. seems like a low bar to meet.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#27In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
I would also love to see a protobuf/gRPC decoder for wireshark. Bonus: the ability to filter sniffed packets based on a field value.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#28Always wished there was a tool for protobuf which could test whether a changes to any .proto files were backwards compatible and if not raise an error
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#29In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…
You could add a TableEngine extension to H2 (h2database.com), pretty easily which would give you full SQL query functionality over such a file
Nope. Protos have repeated fields and can be hierarchical (that is, can contain other protos) and even recursive (that is, contain themselves, possibly as repeated fields). H2 is not going to work.
Re: Prototool – A Swiss Army Knife for Protocol Buffers
#30Earlier quoted context omitted.
You could add a TableEngine extension to H2 (h2database.com), pretty easily which would give you full SQL query functionality over such a file
Nope. Protos have repeated fields and can be hierarchical (that is, can contain other protos) and even recursive (that is, contain themselves, possibly as repeated fields). H2 is not going to work.
Yeah, a normal SQL model is not perfect because of the requirement that it look like a flat table i.e. fixed number of columns.
But H2 has ARRAY for repeated fields, and with some custom functions for decomposing other functions, you could get pretty far.
Just saying, not perfect, but could be useful without too much effort.