Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned from building a production database

maheshba.bitbucket.io

Post reply on HN
Print view View original

51–60 of 112 posts

Re: Things I learned from building a production database

#51

Post by physicles »

bertr4nd wrote: post #28

I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations?…

Yeah I know what you mean, I'm in awe of the author's ability to create the perfect environment for his team to succeed. Part of the key, I think, is that he was dedicated to caring about every factor that might affect success (customers, project management, design, and the other categories from the list), which left his team free to execute on the technical aspects.

I also think we might be underestimating how much one could do with one year, a relatively well-specified problem, and a team of three super motivated and intelligent coders.

Re: Things I learned from building a production database

#52

Post by physicles »

> [10] Make your project robust to re-orgs. A company management hierarchy is inherently fragile (a tree is a 1-connected graph, after all); socialize the project continuously with managers who might take over in the future. Do whatever it takes to make sure that manager churn does not result in unfair career outcomes for ICs.

> [17] For storage systems, bias heavily in the beginning towards consistency and durability rather than availability; these are harder to measure and harder to fix if broken. Because availability is easier to measure, there will be external pressure to prioritize it first; push back.

It's kind of crazy how well the author is able to move from the very top to the very bottom of the technical project management stack.

Re: Things I learned from building a production database

#53

Post by macintux »

scrubs wrote: post #29

Wow! One hell of a good list. And well explained. You bet I book marked it. Heck even the 1-connected comment was worth the read. Because it's true. I'm dealing with that right now. +5 and another reason I love HN. Now, if it you would have made it 43 items, a nice prime number, then I would be jealous to the point of angry (that's a riff on John Daily).

Speaking as a John Daily, but clearly not the one you're referencing: who dat?

Re: Things I learned from building a production database

#54

Post by Hermitian909 »

_pastel wrote: post #47

> [16] Design as a team; implement as individuals. This is pithy. I'm curious - does anyone here follow this philosophy? How do you actually design as a team? A meeting for every single design decision? I'm used to the team finding general alignment on the problem in meetings, but then a single person coming up with the design, writing a doc, and circulating it for comments.

Within a high level design there's often a lot of room for variations in the implementation such as code structure; though this freedom decreases significantly if you have a wide enough audience to need to consider Hyrum's Law.

Concretely, let's say I'm designing a term-rewriting system. The program will be inspecting an abstract syntax tree (AST) for certain patterns, and if it detects those patterns, rewriting them. (e.g. if we see the AST for "1 + 1" rewrite to "2"). I might write up a document on the architecture for the system, and a list of rewriting rules I need. How those rules are written can be left up to individuals on my team as long as it fits into the architecture, has the required functionality, and is of reasonable quality.

Re: Things I learned from building a production database

#55

Post by haimez »

scv119 wrote: post #45
throwthrow564 wrote: post #36

Earlier quoted context omitted.

Very interesting. How did you deal with the issue where the old API had some feature X that wasn't supported in the new API but some customers depended on feature X?

I guess that a hard question for user facing API: you need to support both the old and the new features during the migration. In our case, this design principle mostly applied to our internal APIs. Particularly, we designed our internal sub components (consensus protocol, particularly) in a way we can swap it without any downtime.

Right. So it seems you did a lift and shift over two years without changing the basic API surface. That’s ok, but stability of requirements throughout a rearchitecting project is a luxury and not the norm

Re: Things I learned from building a production database

#56

Post by aurelius83 »

maheshba wrote: post #26
turbocon wrote: post #5

I really think this is a great list however I think one of the major factors why this project succeeded isn't listed but is mentioned in the intro. > We hit production with a 3-person team in less than a year; and subsequently scaled the team to 30+ engineers spanning multiple sub-teams Starting with a small skilled team to build the core and actively plan for a larger team at some point is absolutely critical for pr…

That's a great point; I should definitely have included it in the list! I think new projects (especially ones that seek to innovate) have two phases: in the first phase, everyone's trying to kill it; in the second phase, everyone's trying to grow it. The kill-it phase is actually valuable for exactly the reason you outlined: it gives a small team the chance to create something highly coherent. (The grow-it phase is s…

Can you explain more what you mean by the kill-it phase? Why is everyone trying to kill it?

Re: Things I learned from building a production database

#57

Post by amznbyebyebye »

physicles wrote: post #51
bertr4nd wrote: post #28

I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations?…

Yeah I know what you mean, I'm in awe of the author's ability to create the perfect environment for his team to succeed. Part of the key, I think, is that he was dedicated to caring about every factor that might affect success (customers, project management, design, and the other categories from the list), which left his team free to execute on the technical aspects. I also think we might be underestimating how much…

Fwiw AWS regularly ships entire services in the span of a year with a small two pizza team.

Re: Things I learned from building a production database

#60

Post by [deleted] »

kzrdude wrote: post #19
otras wrote: post #9

Earlier quoted context omitted.

IC is a commonly used acronym (at least at some of the big tech companies) for Individual Contributor. For instance, when the author says "customer ICs", I imagine that means the internal engineers that would be using the database they created (individual contributors on other teams who would be customers/clients of this service). (Vouched for this flagged comment after seeing the other comment)

IC means integrated circuit. Obviously it can have many meanings, but.. maybe they are not obvious.

[deleted]
Post reply on HN