Live data from Hacker News

Things I learned from building a production database

maheshba.bitbucket.io

21–30 of 112 posts

Re: Things I learned from building a production database

#21
post #16
post #7

There was a comment asking what an "IC" was, but the comment was flagged and now dead. Guessing by the context, I'm pretty sure it means "individual contributor", or "person that is not a manager".

Where does the whole Individual Contributor terminology suddenly come from? I've suddenly seen it pop up many times over the last months and tbh the term doesn't make any sense. We just used to call them developers, engineers, or whatever.

I've been hearing the term in silicon valley for over a decade, it's not particularly new.

A lot of time in projects like these you end up getting shunted to tech leads, managers, etc, whose job it is to "take the coordination work" so that people writing code can "focus on implementation". The point of this advice is very specifically to bypass the coordinators and go directly to the people using your project in their code to make sure you understand their concerns, and don't play political telephone.

Re: Things I learned from building a production database

#22
post #11
post #7

There was a comment asking what an "IC" was, but the comment was flagged and now dead. Guessing by the context, I'm pretty sure it means "individual contributor", or "person that is not a manager".

"Interface directly with customer ICs" I've only ever seen it to mean "individual contributor", but then this would make it odd/bad advice. There should only be 1 "customer". Sure, that customer may have a whole team that actually needs the product, and may provide them for clarification or what not, but to avoid unrecorded/incorrect handshake agreements, specs, or pet requirements that have nothing to do with the in…

You should keep a record of what you're doing and why, but getting clear and accurate feedback from the real decision makers who have control over the technical success of your product is essential.

Re: Things I learned from building a production database

#23
post #16
post #7

There was a comment asking what an "IC" was, but the comment was flagged and now dead. Guessing by the context, I'm pretty sure it means "individual contributor", or "person that is not a manager".

Where does the whole Individual Contributor terminology suddenly come from? I've suddenly seen it pop up many times over the last months and tbh the term doesn't make any sense. We just used to call them developers, engineers, or whatever.

I've seen its popularity coincide with emphasis on non-managerial growth opportunities in large tech companies. Gitlab uses it for determining compensation, for example (the top IC is equivalent to a VP in their org).

https://about.gitlab.com/handbook/total-rewards/compensation...

Re: Things I learned from building a production database

#24
post #17
post #9

Earlier quoted context omitted.

IC is a commonly used acronym (at least at some of the big tech companies) for Individual Contributor. For instance, when the author says "customer ICs", I imagine that means the internal engineers that would be using the database they created (individual contributors on other teams who would be customers/clients of this service). (Vouched for this flagged comment after seeing the other comment)

> IC is a commonly used acronym for Individual Contributor Its not though. Outside of hacker news I have literally never heard this term before, nor does it make any sense. If I'd start talking about "Individual Contributors" to my colleagues they would all be confused wth I'm talking about and wondering what's wrong with me. Who even coined this stupid term, and why do we pretend we need a new name for "engineers"?

It has been around in industry for at least twenty years. Facebook is a bit unusual in using it as an actual job title, but it's by no means new. I remember talking about how DEC had a healthy IC track separate from going into management, and DEC hasn't even existed as a separate company since 1998.

Re: Things I learned from building a production database

#25
post #12

Assuming I'm not the only one who's never heard of Delos or Cubby: https://engineering.fb.com/2019/06/06/data-center-engineerin...

Chubby, not cubby. And I also thought it was weird that the author referred to a piece of Google internal infrastructure to describe a piece of Facebook internal infrastructure... how many non-{X,G}ooglers know what Chubby is?

Re: Things I learned from building a production database

#26
post #5

I really think this is a great list however I think one of the major factors why this project succeeded isn't listed but is mentioned in the intro. > We hit production with a 3-person team in less than a year; and subsequently scaled the team to 30+ engineers spanning multiple sub-teams Starting with a small skilled team to build the core and actively plan for a larger team at some point is absolutely critical for pr…

That's a great point; I should definitely have included it in the list! I think new projects (especially ones that seek to innovate) have two phases: in the first phase, everyone's trying to kill it; in the second phase, everyone's trying to grow it. The kill-it phase is actually valuable for exactly the reason you outlined: it gives a small team the chance to create something highly coherent. (The grow-it phase is sometimes worse than the kill-it phase, but that's an entirely different story).

Re: Things I learned from building a production database

#27
> [15] Design APIs with migration to new implementations as a first-class consideration; custom migrations are huge time-sinks and sources of unreliability. Every major API should have a single CLI-driven call for switching implementations.

I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.

Re: Things I learned from building a production database

#28
I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations? And oh by the way shipping this major product in under a year?

Re: Things I learned from building a production database

#29
Wow! One hell of a good list. And well explained. You bet I book marked it. Heck even the 1-connected comment was worth the read. Because it's true. I'm dealing with that right now. +5 and another reason I love HN. Now, if it you would have made it 43 items, a nice prime number, then I would be jealous to the point of angry (that's a riff on John Daily).

Re: Things I learned from building a production database

#30
"[1] Keep your customers happy" - It's hard to actually have a team that respects this lesson. Even if one or two people on the team believe in this rule, others won't, and if they have equal say in the final product, it means the customers will suffer. Dogfooding is really the only way to fight this. Become your own most demanding customer.

"[4] But realize that customers may not express what they really need; don’t take requirements at face-value, instead spend the time to understand their use case in detail. Read their code." - Yep - Users Are Liars. They will lie right to your face, about requirements, deadlines, use cases, functionality, costs, etc, for no reason at all. Never believe anything a user says. And obviously, never tell them you think this.

"[9] If you get good and/or aligned managers, be as understanding, supportive, and accommodating as you can. If you don’t get such managers…" - Become the shadow manager; Team Lead, Lead Developer, Delegate Architect, etc. Ask the manager if you can get "hands-on experience" with various tasks like: running ceremonies, taking meeting notes, coordinating work, planning, resolving blockers. Reach out to other teams in private to resolve problems without involving any management, and be careful that the people you reach out to don't widen scope to their managers. Hide problems and bubble up accomplishments to the bad manager. You'll get none of the credit, but the manager won't have the opportunity to make everyone's lives bad.

"[10] Make your project robust to re-orgs." - Of these, the most resilient I've seen are the ones that promise the moon, and then deftly delay everyone's realization that it doesn't live up to the hype until they're all deeply bought in. Don't make the mistake of trying to get an executive to buy in, because they'll be gone in 3 years and everyone will ditch your project.

"[15] Design APIs with migration to new implementations as a first-class consideration" - Yes, yes, yes!! Literally all technology is replaced eventually, most of it in 3-5 years. Plan the exit strategy at the outset.

"[21] Have the right number of abstractions" - The correct number of abstractions is the minimum possible. Every abstraction should be a solution to a problem. You have to define the problem that the abstraction was meant to solve, and then ask yourself if there is a simpler way to solve that problem.

Post reply on HN