Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned from building a production database

maheshba.bitbucket.io

Post reply on HN
Print view View original

101–110 of 112 posts

Re: Things I learned from building a production database

#101

Post by Kalimoto »

bertr4nd wrote: post #28

I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations?…

Experience.

It's not very hard what he did.

After being in the industry for 10 years you have seen probably most of what he mentioned.

At least I did.

Just be sure that you don't work on one project for 10 years.

Re: Things I learned from building a production database

#102

Post by softveda »

otras wrote: post #9
nzgrover wrote: post #6

Nowhere does it say what an "IC" is. Is it so hard to provided the Full Name(FN) and the abbrevation the first time you use it?

IC is a commonly used acronym (at least at some of the big tech companies) for Individual Contributor. For instance, when the author says "customer ICs", I imagine that means the internal engineers that would be using the database they created (individual contributors on other teams who would be customers/clients of this service). (Vouched for this flagged comment after seeing the other comment)

In Australia outside of IT roles, IC means In-Command i.e. a team lead and 2IC as 2nd In-Command or the person who becomes team lead if the team lead is on leave or resigns.

Re: Things I learned from building a production database

#103

Post by mellavora »

titanomachy wrote: post #25
cmg wrote: post #12

Assuming I'm not the only one who's never heard of Delos or Cubby: https://engineering.fb.com/2019/06/06/data-center-engineerin...

Chubby, not cubby. And I also thought it was weird that the author referred to a piece of Google internal infrastructure to describe a piece of Facebook internal infrastructure... how many non-{X,G}ooglers know what Chubby is?

The author says [40] Keep track of research in your space. Soon you’ll have a shorthand with your ICs that enables super-fast communication: “what if we try that thing from projectX? And combine it with the technique in projectY?”.

so by that learning it is not at all weird that they would refer to infra at an alternative company

Re: Things I learned from building a production database

#105

Post by ryathal »

e12e wrote: post #104

In this kind of structure, what role does managers fill? It sounds like ICs can manage themselves, and the project manager manages the project... What's left for managers to do?

Manage bureaucracy around people: leave, hiring, firing, and promoting. They also generally need to be involved in things that may be more project management depending on the authority project mangers have in organizations: procuring new tools, updating senior management, escalation for cross team issues/communication, filter out noise from external forces, etc.

Re: Things I learned from building a production database

#106

Post by jq-r »

Avalaxy wrote: post #17
otras wrote: post #9

Earlier quoted context omitted.

IC is a commonly used acronym (at least at some of the big tech companies) for Individual Contributor. For instance, when the author says "customer ICs", I imagine that means the internal engineers that would be using the database they created (individual contributors on other teams who would be customers/clients of this service). (Vouched for this flagged comment after seeing the other comment)

> IC is a commonly used acronym for Individual Contributor Its not though. Outside of hacker news I have literally never heard this term before, nor does it make any sense. If I'd start talking about "Individual Contributors" to my colleagues they would all be confused wth I'm talking about and wondering what's wrong with me. Who even coined this stupid term, and why do we pretend we need a new name for "engineers"?

Exactly. And this "IC" thing is a good example of something they call a negative externality. Negative externality in this sense is when you have to pay/are affected by something you don't want like polluted air.

The original author saved maybe couple of seconds to write "Individual Contributor" on the first occasion of the phrase, but made many people here spent many seconds trying to figure out (probably by searching the internet) on what that acronym means.

This is especially pronounced in some companies where the acronyms might be understandable just to a few members of some team, and when when this is communicated to the wider company, it can cause even a greater confusion (as you cannot search for it online) and waste of time for such a small "irrelevant" thing.

So in other words, it can be compressed a bit in: "why should I waste a little bit of my time, when I can waste everyone else's time". Or simply: it is disrespecting your audience.

The original article beside that is quite interesting though.

Re: Things I learned from building a production database

#107

Post by kcartlidge »

scv119 wrote: post #86
alex64 wrote: post #76

This is a good list, but I'm trying to understand the following in better detail: > [12] Be conservative on APIs and liberal with implementations. > [18] Maintain multiple implementations in test for APIs; compare results between them. The cost is worth it (it will help with correctness, and also prevent leakage of implementation detail). The word implementation is thrown out a lot. What does implementation mean in t…

Co-creator of Delos here, Let me give a concrete example for [18]. We have a "StorageEngine" interface that wraps around RocksDB, i.e. "RocksDbStorageEngine" to provide key-value interfaces; however we are worried that someone would leak the implementation detail (RocksDB is so powerful) from the "StorageEngine" interface. To solve the problem, we write another fully in memory storage engine implementation called "Me…

Great explanation :)

Re: Things I learned from building a production database

#108

Post by uvdn7 »

physicles wrote: post #52

> [10] Make your project robust to re-orgs. A company management hierarchy is inherently fragile (a tree is a 1-connected graph, after all); socialize the project continuously with managers who might take over in the future. Do whatever it takes to make sure that manager churn does not result in unfair career outcomes for ICs. > [17] For storage systems, bias heavily in the beginning towards consistency and durabilit…

The post and the author are both absolutely amazing.

But don’t you think the fact that our organizations require someone to be “full stack” organizationally to be a good tech lead is a failure of itself?

I wish organizations can support those talented engineers even though they might not have the best “people/soft skills”.

Re: Things I learned from building a production database

#109

Post by orthoxerox »

aurelius83 wrote: post #56
maheshba wrote: post #26

Earlier quoted context omitted.

That's a great point; I should definitely have included it in the list! I think new projects (especially ones that seek to innovate) have two phases: in the first phase, everyone's trying to kill it; in the second phase, everyone's trying to grow it. The kill-it phase is actually valuable for exactly the reason you outlined: it gives a small team the chance to create something highly coherent. (The grow-it phase is s…

Can you explain more what you mean by the kill-it phase? Why is everyone trying to kill it?

The natural reaction to "let's build something in-house" is almost invariably "let's try an OTS solution, surely one must already exist". Most successful internal projects require either an organizational culture of "we're at the forefront of technology" (but then you get "surely one must already exist in some other division" instead), a nuclear icebreaker of a champion that is willing to put their neck on the line, or a series of (un)fortunate events that leave your project as the only way out.

Re: Things I learned from building a production database

#110

Post by krubar »

turbocon wrote: post #5

I really think this is a great list however I think one of the major factors why this project succeeded isn't listed but is mentioned in the intro. > We hit production with a 3-person team in less than a year; and subsequently scaled the team to 30+ engineers spanning multiple sub-teams Starting with a small skilled team to build the core and actively plan for a larger team at some point is absolutely critical for pr…

I think this can be related to what Mel Conway stated in his famous "How Committees Invent?"

> Let us first examine the tendency to overpopulate a design effort. It is a natural temptation of the initial designer-the one whose preliminary design concepts influence the organization of the design effort-to delegate tasks when the apparent complexity of the system approaches his limits of comprehension. This is the turning point in the course of the design. Either he struggles to reduce the system to comprehensibility and wins, or else he loses control of it. The outcome is almost predictable if there is schedule pressure and a budget to be managed.

Post reply on HN