Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned from building a production database

maheshba.bitbucket.io

Post reply on HN
Print view View original

31–40 of 112 posts

Re: Things I learned from building a production database

#31

Post by mym1990 »

bertr4nd wrote: post #28

I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations?…

I think you assess your skills and what you bring, figure out the deficiencies, and then find people who can fill in other gaps(and hopefully have some overlapping skills as well). At the end of the day having half of the time and intellect is a great place to be, that is much better than having none of the time and intellect.

Re: Things I learned from building a production database

#32

Post by boulos »

titanomachy wrote: post #25
cmg wrote: post #12

Assuming I'm not the only one who's never heard of Delos or Cubby: https://engineering.fb.com/2019/06/06/data-center-engineerin...

Chubby, not cubby. And I also thought it was weird that the author referred to a piece of Google internal infrastructure to describe a piece of Facebook internal infrastructure... how many non-{X,G}ooglers know what Chubby is?

The paper describing Chubby was published in 2006 [1].

[1] https://research.google/pubs/pub27897/

Re: Things I learned from building a production database

#33

Post by bigodanktime »

bertr4nd wrote: post #28

I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations?…

I think you are assuming all these things are independent of one another. I think deep engagement with customers directs your problem statement. Build the smallest possible thing to test the idea you are proposing. If you are trying to publish than its a requirement to be reading papers in general, you need to be able to argue why this idea you are testing is orthogonal to other ideas being proposed. As time goes on you just scan important conferences in your field and get better at knowing which papers you need to skim versus deeply understand.

Edit: Also intelligence is an over-rated metric, persistence is far more important when it comes to these hard problems.

Re: Things I learned from building a production database

#34

Post by krschultz »

throwthrow564 wrote: post #27

> [15] Design APIs with migration to new implementations as a first-class consideration; custom migrations are huge time-sinks and sources of unreliability. Every major API should have a single CLI-driven call for switching implementations. I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.

We use the same principle on some of the frameworks I have worked on, it pushes you to make the API more explicit and keep the context localized. That makes codemods easier. This can sometimes lead to a more verbose / less magical API, but IMO at scale the tradeoff is worth it.

Re: Things I learned from building a production database

#35

Post by scv119 »

throwthrow564 wrote: post #27

> [15] Design APIs with migration to new implementations as a first-class consideration; custom migrations are huge time-sinks and sources of unreliability. Every major API should have a single CLI-driven call for switching implementations. I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.

The first engineer on the Delos team here. This design decision gave us an edge: we delivered the first production cluster in 8 months by layering over existing systems ; and we smoothly migrated all underlying implementations in the following 2 years without our customer aware of it (0 downtime). This design decision is well captured by the OSDI’20 Virtual Consensus in Delos paper.

Re: Things I learned from building a production database

#36

Post by throwthrow564 »

scv119 wrote: post #35
throwthrow564 wrote: post #27

> [15] Design APIs with migration to new implementations as a first-class consideration; custom migrations are huge time-sinks and sources of unreliability. Every major API should have a single CLI-driven call for switching implementations. I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.

The first engineer on the Delos team here. This design decision gave us an edge: we delivered the first production cluster in 8 months by layering over existing systems ; and we smoothly migrated all underlying implementations in the following 2 years without our customer aware of it (0 downtime). This design decision is well captured by the OSDI’20 Virtual Consensus in Delos paper.

Very interesting. How did you deal with the issue where the old API had some feature X that wasn't supported in the new API but some customers depended on feature X?

Re: Things I learned from building a production database

#38

Post by jiggawatts »

maheshba wrote: post #20
tyingq wrote: post #7

There was a comment asking what an "IC" was, but the comment was flagged and now dead. Guessing by the context, I'm pretty sure it means "individual contributor", or "person that is not a manager".

author of the post here: apologies for the acronyms! I code-switched from academia/research to industry via FB a few years back and just picked up a bunch of terms that I thought were universal in industry. IC does mean Individual Contributor or "person that is not a manager" -- I'll add a definition on top!

I wonder if it says anything about Facebook's culture that I've never heard of the term Individual Contributor before. Everywhere else, it's Team Member or something along those lines indicating that you're not on your own.

Re: Things I learned from building a production database

#40

Post by andrewmunsell »

nyanpasu64 wrote: post #37

Is a diff Facebook's equivalent of a feature branch or PR?

Isn't diff just a standard, generic term for a change difference? (Like in git: https://git-scm.com/docs/git-diff)

Someone correct me if I'm wrong, but if my memory is correct, I was under the impression that a pull request was a term coined in the early Github days specific to their PR feature.

Post reply on HN