Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned from building a production database

maheshba.bitbucket.io

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Things I learned from building a production database

#81

Post by polskibus »

ketzo wrote: post #70
physicles wrote: post #52

> [10] Make your project robust to re-orgs. A company management hierarchy is inherently fragile (a tree is a 1-connected graph, after all); socialize the project continuously with managers who might take over in the future. Do whatever it takes to make sure that manager churn does not result in unfair career outcomes for ICs. > [17] For storage systems, bias heavily in the beginning towards consistency and durabilit…

This is, to me, the definition of a senior software engineer. Awareness across every single part of the process of making software. Not expertise, necessarily, but enough knowledge to know what to be on the lookout for.

More of a team leader at least, non-IC. Some Senior ICs deliberately don't want any part in politics, just want a task and want to get it done (see for example advice on HN about avoiding meetings and politics at all costs).

Re: Things I learned from building a production database

#82

Post by vishnugupta »

alex64 wrote: post #76

This is a good list, but I'm trying to understand the following in better detail: > [12] Be conservative on APIs and liberal with implementations. > [18] Maintain multiple implementations in test for APIs; compare results between them. The cost is worth it (it will help with correctness, and also prevent leakage of implementation detail). The word implementation is thrown out a lot. What does implementation mean in t…

To take Stripe's example, the API is as documented here[1]. Essentially a documented contract between a platform provider and their clients.

Implementation is things like, as you noted, DB used, programming language, and so on.

By "Be conservative on APIs" they mean be careful what you promise to your clients as communicated and documented through APIs. Once promised it's almost impossible to go back; you will lose your clients' trust and piss them off. In case of Stripe they are committing to support idempotent requests[2] which is sort of a big deal. And I don't expect them to drop that support in any of their future APIs because I imagine hundreds of millions of $$ business would be relying on that idempotency support.

[1] https://stripe.com/docs/api

[2] https://stripe.com/docs/api/idempotent_requests

Re: Things I learned from building a production database

#83

Post by rad_gruchalski »

titanomachy wrote: post #25
cmg wrote: post #12

Assuming I'm not the only one who's never heard of Delos or Cubby: https://engineering.fb.com/2019/06/06/data-center-engineerin...

Chubby, not cubby. And I also thought it was weird that the author referred to a piece of Google internal infrastructure to describe a piece of Facebook internal infrastructure... how many non-{X,G}ooglers know what Chubby is?

ZooKeeper is modeled after Chubby.

Re: Things I learned from building a production database

#84

Post by rmbyrro »

bigodanktime wrote: post #33
bertr4nd wrote: post #28

I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations?…

I think you are assuming all these things are independent of one another. I think deep engagement with customers directs your problem statement. Build the smallest possible thing to test the idea you are proposing. If you are trying to publish than its a requirement to be reading papers in general, you need to be able to argue why this idea you are testing is orthogonal to other ideas being proposed. As time goes on…

I agree, persistence, discipline and ordering of daily activities.

It's very easy to get lost in day to day mess.

Re: Things I learned from building a production database

#85

Post by penguin_booze »

polskibus wrote: post #81
ketzo wrote: post #70

Earlier quoted context omitted.

This is, to me, the definition of a senior software engineer. Awareness across every single part of the process of making software. Not expertise, necessarily, but enough knowledge to know what to be on the lookout for.

More of a team leader at least, non-IC. Some Senior ICs deliberately don't want any part in politics, just want a task and want to get it done (see for example advice on HN about avoiding meetings and politics at all costs).

One ill effect of not deciding to participate is that you'll be subjected to rules that which you play no part in establishing. I think it's about the balance to strike between being well and content with what's going around, and needing to exercise control to change the environment.

Re: Things I learned from building a production database

#86

Post by scv119 »

alex64 wrote: post #76

This is a good list, but I'm trying to understand the following in better detail: > [12] Be conservative on APIs and liberal with implementations. > [18] Maintain multiple implementations in test for APIs; compare results between them. The cost is worth it (it will help with correctness, and also prevent leakage of implementation detail). The word implementation is thrown out a lot. What does implementation mean in t…

Co-creator of Delos here, Let me give a concrete example for [18].

We have a "StorageEngine" interface that wraps around RocksDB, i.e. "RocksDbStorageEngine" to provide key-value interfaces; however we are worried that someone would leak the implementation detail (RocksDB is so powerful) from the "StorageEngine" interface.

To solve the problem, we write another fully in memory storage engine implementation called "MemoryStorageEngine", which acts as a specification of the "StorageEngine" interface.

We run all storage engine tests against both implementations (rocksdb/memory), except for the durability part; we also configured the system to run over both storage engines.

By doing this:

- we can use the memory based one to enforce the behavior RocksDB based one by running unit tests against both implementations.

- no one can easily leak some implementation details of RocksDB from the StorageEngine interface; to do that, you need to first add those advanced feature into memory based StorageEngine as well!

This is just one example, and we have a list of such abstractions, such as MetadataStore, Logs etc. We created multiple implementation of each of these interfaces and ensure the system could run on any combination of those implementations.

Re: Things I learned from building a production database

#87

Post by scv119 »

_pastel wrote: post #47

> [16] Design as a team; implement as individuals. This is pithy. I'm curious - does anyone here follow this philosophy? How do you actually design as a team? A meeting for every single design decision? I'm used to the team finding general alignment on the problem in meetings, but then a single person coming up with the design, writing a doc, and circulating it for comments.

We wanted to ensure everyone on the team has explored the full design space, and everyone agreed that the chosen solution is the best we can think of, given the current information and constraints.

Once we have done that, everyone is aligned and it doesn't matter who to implement it; it will be implemented in the same way.

Re: Things I learned from building a production database

#88

Post by rmbyrro »

bottled_poe wrote: post #80
dalanmiller wrote: post #77

> [12] Be conservative on APIs and liberal with implementations. Curious how people interpret this? Is it: * Attempt to keep our API small while also... * thinking widely about the way it's used? Or how does this work in practice?

I’d interpret this to mean keeping you API specification basic to minimise backwards compatibility problems and reduce future refactoring difficulty. Don’t have special methods for unique cases, stick to restful principles where possible, keep data structures simple, etc.

I'd add give enough freedom for users to accomplish what they need from a small set of API abstractions.

Many times, this shifts burden to users. It's always more convenient to have one API which does exactly what you need.

But the idea here is to keep the API simple and stable in a phase where instability is already high.

Perhaps, after it gets mature, it'll make sense to incorporate common use cases into the API structure for convenience.

Re: Things I learned from building a production database

#89

Post by lysecret »

The main thing i learned in my first production db work is that you pay for every bit of complexity you introduce. As a noob you are somehow much more afraid to not "model the data right" so you tend to create much to many tables much to many n to m relationships you make everything a string a uuid a relationship, never use bools etc. But then you have to iterate and live with those systems and then you pay and every bit of complexity makes everything harder. To give you some simple examples:

1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results. 2. You just made this thing a relationship now every time you have to join.

In the end, if you can keep it simple even if you don't model the data "perfectly". But of course sometimes the complexity is really necessary. To know when is being a good Backend / Db Engineer.

Re: Things I learned from building a production database

#90

Post by faichai »

vishnugupta wrote: post #65
aurelius83 wrote: post #56

Earlier quoted context omitted.

Can you explain more what you mean by the kill-it phase? Why is everyone trying to kill it?

I have been part of platform and product teams. So I think I sort of know what the author means by "kill-it" phase. When new platform is being worked upon it's extremely difficult to get product teams to adopt. The PMs on the product teams view a new platform as a distraction because it seemingly doesn't help them launch new features. Worse it's taking the engineers who could be working on features to integrate with…

Well said! This has happened to me, I was introducing a new system and in conversation with a a primary customer team during the planning and initial dev constantly. They provided good feedback, and were supportive. When they realised we would stop supporting the old system, the attitude changed immediately and they did everything they could to derail the project.
Post reply on HN