Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prototool – A Swiss Army Knife for Protocol Buffers

github.com

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#61

Post by reacharavindh »

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

Have limited protobuf knowledge.

Why not use SQLite[1] for storing this data? Storing structured data in binary format, and being able to run SQL queries on it, is already possible with SQLite right?

[1] - https://www.sqlite.org/appfileformat.html

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#62

Post by [deleted] »

reacharavindh wrote: post #61
throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

Have limited protobuf knowledge. Why not use SQLite[1] for storing this data? Storing structured data in binary format, and being able to run SQL queries on it, is already possible with SQLite right? [1] - https://www.sqlite.org/appfileformat.html

[deleted]

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#63

Post by endymi0n »

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

I‘m smelling the SQL case could be reasonably easily thrown together with PostgreSQL and a custom Foreign Data Wrapper based on protobuf-c (prior art: cstore_fdw by the Citus folks). Proto definitions then should compile rather cleanly to table definitions, at least one level down (PG isn‘t so good with nested structures).

The main thing stopping this endeavour is probably that to the best of my knowledge, there isn‘t any standardization in the Protobuf community about file formats serializing multiple of these together like RecordIO - that, and my C skills are pretty rusty by now :)

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#64

Post by erik_seaberg »

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

Sounds a lot like a Hive query over self-describing Avro files.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#65

Post by JyrkiAlakuijala »

dekhn wrote: post #19
zellyn wrote: post #6

Earlier quoted context omitted.

When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release. Looks like some parts of it have escaped… https://github.com/eclesh/recordio

TFRecords are the closest thing to recordio that has Google support.

Riegeli is the closest thing to RecordIO that has Google support.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#66

Post by dekhn »

JyrkiAlakuijala wrote: post #65
dekhn wrote: post #19

Earlier quoted context omitted.

TFRecords are the closest thing to recordio that has Google support.

Riegeli is the closest thing to RecordIO that has Google support.

I checked. Riegeli says the file format is not frozen. Therefore, while it may have support, I wouldn't recommend anybody use it.
Post reply on HN