MySQL infrastructure testing automation at GitHub

#1

Post by samlambert » Thu, Jul 06, 2017, 4:53 PM UTC

MySQL infrastructure testing automation at GitHub
githubengineering.com

Re: MySQL infrastructure testing automation at GitHub

#2

Post by jivid » Thu, Jul 06, 2017, 5:52 PM UTC

Super interesting post. Would love to read more detail about their backup and restore infrastructure.

If Tom and/or Shlomi are reading this: you mention taking multiple logical backups per day. What benefit does this bring versus just having one per day and doing a point-in-time restore using binlogs? Is this just a tradeoff between time taken for a restore and storage you're willing to dedicate to backups?

Disclaimer: I work on Facebook's MySQL backup and restore system (https://code.facebook.com/posts/1007323976059780/continuous-...)

Re: MySQL infrastructure testing automation at GitHub

#3

Post by shlomi-noach » Thu, Jul 06, 2017, 6:16 PM UTC

jivid wrote: post #2

Super interesting post. Would love to read more detail about their backup and restore infrastructure. If Tom and/or Shlomi are reading this: you mention taking multiple logical backups per day. What benefit does this bring versus just having one per day and doing a point-in-time restore using binlogs? Is this just a tradeoff between time taken for a restore and storage you're willing to dedicate to backups? Disclaime…

@jivid the logical backups are done per-table, not per-server. Per-table logical backups are useful to the engineers owning the data. It makes it easy for them to restore data from a single table.

When an engineer loads logical backup data, it loads into a non-production private zone where the engineer has access to the data, and can then make informed decisions on whether there is need to re-apply data changes (due to bug, due to need to review historical data, etc.).

This of course has the advantage of quicker restores (only need a single table), and this happens to cover the vast majority of cases. This doesn't cover the case where we need to restore consistent data for two or more different tables.

Re: MySQL infrastructure testing automation at GitHub

#4

Post by ngrilly » Thu, Jul 06, 2017, 6:55 PM UTC

Great post! Do you use semi-synchronous or asynchronous replication? If you use asynchronous replication, when a server crashes and this triggers the automated failover, do you lose the last transactions?

Re: MySQL infrastructure testing automation at GitHub

#5

Post by ngrilly » Thu, Jul 06, 2017, 7:01 PM UTC

ngrilly wrote: post #4

Great post! Do you use semi-synchronous or asynchronous replication? If you use asynchronous replication, when a server crashes and this triggers the automated failover, do you lose the last transactions?

It looks like the answer to my question is here: https://githubengineering.com/orchestrator-github/

Re: MySQL infrastructure testing automation at GitHub

#6

Post by Thaxll » Thu, Jul 06, 2017, 7:31 PM UTC

What's the difference between gh-ost and the Percona tool? ( pt-online-schema-change ) Also did you try to use a recent version of MySQL that supports live migration?

Re: MySQL infrastructure testing automation at GitHub

#7

Post by samlambert » Thu, Jul 06, 2017, 7:37 PM UTC

Thaxll wrote: post #6

What's the difference between gh-ost and the Percona tool? ( pt-online-schema-change ) Also did you try to use a recent version of MySQL that supports live migration?

https://githubengineering.com/gh-ost-github-s-online-migrati...

Re: MySQL infrastructure testing automation at GitHub

#8

Post by sciurus » Thu, Jul 06, 2017, 8:21 PM UTC

ngrilly wrote: post #5

ngrilly wrote: post #4

Great post! Do you use semi-synchronous or asynchronous replication? If you use asynchronous replication, when a server crashes and this triggers the automated failover, do you lose the last transactions?

It looks like the answer to my question is here: https://githubengineering.com/orchestrator-github/

The author of Orchestrator also goes into questions like that at http://code.openark.org/blog/mysql/mysql-high-availability-t...

Tangentially, in that blog post I was impressed by the list of companies running orchestrator.

> orchestrator is actively maintained by GitHub. It manages automated failovers at GitHub. It manages automated failovers at Booking.com, one of the largest MySQL setups on this planet. It manages automated failovers as part of Vitess. These are some names I’m free to disclose, and browsing the issues shows a few more users running failovers in production. Otherwise, it is used for topology management and visualization in a large number of companies such as Square, Etsy, Sendgrid, Godaddy and more.

Re: MySQL infrastructure testing automation at GitHub

#9

Post by ngrilly » Thu, Jul 06, 2017, 9:13 PM UTC

sciurus wrote: post #8

ngrilly wrote: post #5

Earlier quoted context omitted.

It looks like the answer to my question is here: https://githubengineering.com/orchestrator-github/

The author of Orchestrator also goes into questions like that at http://code.openark.org/blog/mysql/mysql-high-availability-t... Tangentially, in that blog post I was impressed by the list of companies running orchestrator. > orchestrator is actively maintained by GitHub. It manages automated failovers at GitHub. It manages automated failovers at Booking.com, one of the largest MySQL setups on this planet. It manages…

Thanks for the link: it's a fantastic read!

Re: MySQL infrastructure testing automation at GitHub

#10

Post by andy_ppp » Thu, Jul 06, 2017, 11:04 PM UTC

I take it from this blog that any engineer at GitHub can get a copy of anyone's data. I find this vaguely terrifying, especially because Yahoo! was so careful internally about what an engineer could actually get access to. Most data like this had usernames/names/emails/other sensitive info redacted.

Can anyone at GitHub get a copy of my private repo(s) because I'm out if they can.

"Move fast and take a copy of the referrers table."