Prototool – A Swiss Army Knife for Protocol Buffers • Hacker News Board

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#51

Post by [deleted] » Wed, Apr 11, 2018, 9:34 PM UTC

puzzle wrote: post #5

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…

[deleted]

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#52

Post by grizzles » Wed, Apr 11, 2018, 9:35 PM UTC

danby - grpc for the browser :: is looking for testers https://github.com/ericbets/danby There are two upcoming features. The first one is streaming support. The second is a callback API template that mirrors the grpc node API exactly. Or you will have the choice to stick with the current promise API. It's not a priority for us at the moment but adding a simple load balancer that distributed traffic randomly across a set of servers would be a ~5 line patch.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#53

Post by mabynogy » Wed, Apr 11, 2018, 9:59 PM UTC

zrail wrote: post #49

mabynogy wrote: post #45

Earlier quoted context omitted.

You should read it. It includes words that are politically connoted. Those words can easily be leveraged by people who are in that domain. It's not about common sense. You don't need to tell people to be polite (and you shouldn't). If someone isn't everybody will notice it.

> You don't need to tell people to be polite (and you shouldn't). If someone isn't everybody will notice it. Unless it's done in private. If someone is a jerk to me in private (over email or Slack or whatever), what do I do? If I'm contributing to a project without a CoC and someone is a jerk to me, I'm much more likely to just contribute to a different project, or start my own fork, or just stop contributing to open…

I always reply to offensive people. Someone who is offensive has no arguments. Saying him he is offensive is often enough. If there is a public (like here) people will understand who is insulting who.

You're looking for a way to be less implicated. It's because you're accustomed to live in systems based on laws and rules.

People aren't bad. Most are kind and they don't want to harm anonybody. The bad guys are very few (like 1% or 0.1%) and as they are few they should be handled case by case. Building a complicated system of rules for them will just bother the "not-bad" majority.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#54

Post by haberman » Wed, Apr 11, 2018, 10:30 PM UTC

zellyn wrote: post #6

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release. Looks like some parts of it have escaped… https://github.com/eclesh/recordio

I think the open-source equivalent of RecordIO is the leveldb log format:

https://github.com/google/leveldb/blob/master/doc/log_format...

https://github.com/google/leveldb/blob/master/db/log_reader....

https://github.com/google/leveldb/blob/master/db/log_writer....

I think the decision not to open-source RecordIO is likely related to legacy baggage that's baked into the format. The LevelDB format above avoids that.

It doesn't appear that the headers for this are public though.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#55

Post by lobster_johnson » Wed, Apr 11, 2018, 11:26 PM UTC

puzzle wrote: post #14

kodablah wrote: post #9

Earlier quoted context omitted.

> Kubernetes: canaries and automatic rollbacks Hot off the presses: https://cloudplatform.googleblog.com/2018/04/introducing-Kay... . Though you have to use Spinnaker.

That's an external controller, which is what most people are doing themselves these days, reinventing the wheel each time. Borg has long had an automatic rollback feature on updates, tuned through a few settings on top of the health check machinery. I'm in the camp believing that a basic implementation should be built-in, since health checks are already there. An implementation of this was started, but it has stalled…

How does Borg express updates? With Kubernetes only deployments have rollout/rollback, and you need something like Helm -- which is frankly not that great -- to handle groups of resources that are to be updated and versioned together, e.g. configmaps and services.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#56

Post by lobster_johnson » Wed, Apr 11, 2018, 11:40 PM UTC

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

How does RecordIO compare with Parquet and Arrow? Different use cases?

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#57

Post by puzzle » Thu, Apr 12, 2018, 12:09 AM UTC

lobster_johnson wrote: post #55

puzzle wrote: post #14

Earlier quoted context omitted.

That's an external controller, which is what most people are doing themselves these days, reinventing the wheel each time. Borg has long had an automatic rollback feature on updates, tuned through a few settings on top of the health check machinery. I'm in the camp believing that a basic implementation should be built-in, since health checks are already there. An implementation of this was started, but it has stalled…

How does Borg express updates? With Kubernetes only deployments have rollout/rollback, and you need something like Helm -- which is frankly not that great -- to handle groups of resources that are to be updated and versioned together, e.g. configmaps and services.

AFAIR, you can only update jobs (collections of single containers) and allocs (pods inside of which jobs will schedule and run: you need to state that job J lives inside alloc A, otherwise it will run in an anonymous, implied alloc). There aren't other objects you can manipulate, actually.

Configmaps don't really exist, although something similar is achieved with a job that has a second package holding just the data. This is why I think multi-image containers should be implemented, but also a reason why they haven't been yet: configmaps cover some use cases. When a job replica (task) gets updated or rolled back, both packages change in sync. On Kubernetes, you'd use version numbers in the configmap name (but you need to worry about garbage collecting unused ones).

Services live outside of Borg entirely. GSLB has its own push mechanisms and only consumes Borg's lists of containers that comprise a given Borg job.

From the paper:

> A user can change the properties of some or all of the tasks in a running job by pushing a new job configuration to Borg, and then instructing Borg to update the tasks to the new specification. This acts as a lightweight, non-atomic transaction that can easily be undone until it is closed (com- mitted). Updates are generally done in a rolling fashion, and a limit can be imposed on the number of task disruptions (reschedules or preemptions) an update causes; any changes that would cause more disruptions are skipped. Some task updates (e.g., pushing a new binary) will al- ways require the task to be restarted; some (e.g., increasing resource requirements or changing constraints) might make the task no longer fit on the machine, and cause it to be stopped and rescheduled; and some (e.g., changing priority) can always be done without restarting or moving the task.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#58

Post by smarterclayton » Thu, Apr 12, 2018, 12:18 AM UTC

puzzle wrote: post #5

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

Other tools and features that don't exist outside: - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should. - load balancing dashboards showing traffic between frontends and backends - load balancer support for dynamic sharding - gnubbyd under ChromeOS: https://groups.google.com/a/chromium.org/forum/m/#!m…

Image based volumes (second last to bullet) has long been blocked on the container runtime having a really clean way to enable and keep the container filesystems mounted. Definitely something I want to see fixed since otherwise you just end up doing hacky copies via emptydir.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#59

Post by pm90 » Thu, Apr 12, 2018, 3:58 AM UTC

puzzle wrote: post #14

kodablah wrote: post #9

Earlier quoted context omitted.

> Kubernetes: canaries and automatic rollbacks Hot off the presses: https://cloudplatform.googleblog.com/2018/04/introducing-Kay... . Though you have to use Spinnaker.

That's an external controller, which is what most people are doing themselves these days, reinventing the wheel each time. Borg has long had an automatic rollback feature on updates, tuned through a few settings on top of the health check machinery. I'm in the camp believing that a basic implementation should be built-in, since health checks are already there. An implementation of this was started, but it has stalled…

> I'm in the camp believing that a basic implementation should be built-in, since health checks are already there.

Agreed 100%. Although... in my mind k8s is still pretty young, and this would definitely be a great feature to have by default in the future.

Re: Prototool – A Swiss Army Knife for Protocol Buffers

#60

Post by throwaway84742 » Thu, Apr 12, 2018, 5:23 AM UTC

lobster_johnson wrote: post #56

throwaway84742 wrote: post #2

In another decade or so the world might replicate half of the very nice internal tools Google has. Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple…

How does RecordIO compare with Parquet and Arrow? Different use cases?

Don’t know about Arrow, but Parquet is a columnar format. Such formats can’t write record-by-record, they need a large number of records to shred into columns in order to realize their columnar benefits. In contrast, appending to RecordIO is little more than writing a binary string. The downside of RecordIO is that you can’t just read some fields in a message and not others. You have to deserialize the whole message. RecordIO is cheap to write and well suited for cases where reading the entire message is not that big a deal. Columnar formats are more suited for the cases where it’s ok to pay the relatively substantial up front encoding cost for vastly greater performance in analytical workloads. Advanced ones contain additional metadata (such as range and hash constraints, the former can be both per file and per block) which the analytical runtime will be able to take advantage of in order to avoid doing the work that doesn’t need to be done.