Live data from Hacker News

Drunk Post: Things I've Learned as a Sr Engineer

old.reddit.com

491–500 of 510 posts

Re: Drunk Post: Things I've Learned as a Sr Engineer

#491

Earlier quoted context omitted.

Eh, that's a nice thing to say, but it only makes sense at certain scales, and no matter what, there's always a person that can break it. If any random person can break it, it's already broken. If any employee can break it, it's probably broken (there are very small scales where even this doesn't apply. Ever worked for a company with less than ten people? There's probably something any employee can break). If any emp…

There are lots of places where we require that no single person can break the system at least in a certain way. For example code review and LGTM ensures that a single individual can't just break the system by pushing bad code. Often there are other control planes that don't have the same requirement, but I think the idea that there must always be one person who can break the system isn't clearly true.

> For example code review and LGTM ensures that a single individual can't just break the system by pushing bad code.

There's always someone with rights to push code manually, or tell the system to push an older version of code which won't work anymore. Someone needs to install and administer the system that pushes the code, and even if they don't have direct access to push the code to where it eventually goes, someone's access credentials (or someone that controls the system that has access credentials) has access somewhere along the way.

But who controls that the code system is up and available even allow checkins? Can one person break that? What about who controls the power state of the systems the code gets checked in on? Is that also ensured not to be a single person? What about office access? What about the power main at your building? Is it really impossible for one person to cause problems there?

It might sound like I'm changing the goal posts, but that's sort of my point, these are all dependencies on each other. It's impossible to actually make it so one person can't cause any problems, because you can't eliminate all dependencies, and you can even accurately know what they all are. What you can do is focus on the likely ones, put whatever in place you can that's sane, but focus all the crazy effort you would have to do to track down the diminishing returns of trying to make failure impossible and start spending that time and effort on making recovery quick and easy.

Unfortunately, some work that goes into attempting to make sure any one person can't cause a problem might actually make that harder. Requiring someone to sign off on a commit to go live is great at 2 PM Tuesday, but not so great when it's required to fix something at 2 AM Sunday. This is the tightrope that needs to be walked, and also while even if you don't necessarily know about it, there probably is someone that has access to break something all by themselves, because they're who is called in to makes sure it can be fixed when the shit hits the fan and all those roadblocks to prevent problems need to be bypassed so the current problem can actually be fixed.

Any system that doesn't have some people like that at various levels persists in that state only until they have a problem and in the incident assessment someone needs to answer why a 5 minute fix took hours and the answer includes a lot of "we needed umpteen different people and only a fraction were available immediately".

Even at Google (which I see you work at from your profile), my guess is that people in the SRE stack can cause a very bad day for most apps. My guess is that even if the party line is that no one person can screw anything up, you probably don't have to ask to many SREs there before someone notes that it's more of an aspiration than a reality.

Sorry if that's a bit rambly. I know you weren't specifically countering what I was saying. I've just had a lot of years of sysadmin experience where it's pretty easy to see the gaps on a lot of these solutions where the face presented looks pretty secure.

Re: Drunk Post: Things I've Learned as a Sr Engineer

#492

Earlier quoted context omitted.

No, I've look enough I think. Maybe those better frameworks do exist though indeed, but I have no proof of that. Usually they just have the bare minimum of assert checks and call it a day. A testing framework should be able to mock and patch any class (or applicable) of the running instance of the program without modifying your code for example, tell me if a method was executed or not, intercepting all HTTP requests…

> mock and patch any class (or applicable) of the running instance of the program without modifying your code Why is it important for the sake of testing to be able to alter the runtime behaviour externally without changing the code? This is contrary to all TDD literature I've read which advises to make production code easy to test, e.g. , by coding against interfaces. After all, something being hard to test is exact…

> Why is it important for the sake of testing to be able to alter the runtime behaviour externally without changing the code?

Because that's additional cruft that you don't want when reading your code. Yeah I get it, you can pass dozen of abstract classes to each constructor for each thing you are mocking (or equivalent if it's not a class-based language) that you replace if needed, the only problem with that is that it's inconvenient, prone to mistakes (you can forget some) and makes the code ugly (you don't want to read testing code on the main code).

I've done it in multiple static languages and at best it's a workaround that you should not have to deal with. I should add that any additional barrier to writing tests like this one also reduces the likelyhood that your app is well tested since some developers of your team might not bother going as far as that.

> Spies are possible with e.g. the ReaderT pattern.

After looking online, I'm not sure how that works, that does not looks very convenient for sure. Is that possible with that to say something like "the method X of class Y was executed 2 times with the parameters Z" without changing your code?

> I don't know what this means. I looked at several articles describing some kind of factory pattern in TDD — all of which were horrifically verbose — and all I can glean from that is we are talking about mocking some function which generates objects.

Something like https://github.com/thoughtbot/factory_bot

> This is no different from mocking other system boundaries, which I have already addressed.

Changing the time should also affect any sub dependency of your code which is calling to get the current time, not only your code.

Re: Drunk Post: Things I've Learned as a Sr Engineer

#493
post #333

> Hacker News and r/Programming is only good to get general ideas and keep up-to-date, the comments are almost worthless That's a weird one. I don't know anything about that subreddit, but HN comments are frequently great. I submit stuff because I want there to be HN comments on it for me to read. I typically read the comments first and only bother opening the link if they were interesting.

You should check bitcoin early-days posts

Re: Drunk Post: Things I've Learned as a Sr Engineer

#494

Earlier quoted context omitted.

My experience of Hacker News comments is not positive. They tend to be very convincing but actually kinda bullshit (or just vacuous), which is arguably a lot worse than communities that are transparent.

That’s what Dan is saying - but once in awhile you find a pearl in an area you know enough about to confirm. Otherwise it’s best to assume everything you read here is full of it.

I was responding to the idea that it's better than everywhere else on the internet. That hasn't really been my experience. Discord is better, StackOverflow is better, even Reddit is often better if you're looking at specialist communities.

But to be honest I rarely find pearls here. Even many of the "pearls" in the article are just well-written articulations of stuff that is... kind of obvious. Some are inversions of ordinary wisdom for the sake of inversion. Only some carry new information.

I find the most valuable stuff here tends to be arguments where someone knows their shit but is going against the grain, and that person will usually be flagged into oblivion. But there aren't many places you see someone like that responding in context to the mainstream dogma.

---

Interestingly though, reading through it I explicitly remember a lot of the comments he quoted, e.g. the FedEx Airport one (which was really interesting). It's kind of crazy to think the site is small enough that we're all reading the same good stuff.

Re: Drunk Post: Things I've Learned as a Sr Engineer

#495

Earlier quoted context omitted.

> For example code review and LGTM ensures that a single individual can't just break the system by pushing bad code. Assuming code reviews are 100% effective at catching issues, particularly issues that cross multiple projects. Yes.

I'm making an (admittedly subtle) distinction here between complex mistakes, where something was missed, and simple mistakes/bad actors where someone used a privilege in a manner they shouldn't have. LGTM ensures that, for example, a single individual can't push a code change that drops the database. On the other hand, that same individual might be able to turn of the database in the AWS console.

> LGTM ensures that, for example, a single individual can't push a code change that drops the database.

Personally, I've seen LGTM let slip complex bugs in accounting code (admittedly, not great code) that went on to irreversibly corrupt hundreds of millions of accounting records.

Yes, it will catch "DROP DATABASE", but when it's still letting through major bugs that similarly require a full restore from backup... It seems functionally equivalent?

Given:

> There are lots of places where we require that no single person can break the system at least in a certain way.

I don't think code reviews are a solution. I mean, they're one of the better solutions I can think of, but they're not actually a solution.

Re: Drunk Post: Things I've Learned as a Sr Engineer

#496

Earlier quoted context omitted.

The issue does arise when you aren’t able to understand the problem space enough to realize that what you copied from stack overflow has a mistake or doesn’t fit the requirement you need (e.g. perhaps it doesn’t match your error handling architecture or so on). That said, stack overflow can be a great source and I’ve written plenty of code with a comment pointing to a SO link to further explain a pattern or snippet f…

"The issue does arise when you aren’t able to understand the problem space enough to realize that what you copied from stack overflow has a mistake or doesn’t fit the requirement you need " Yeah sure. A stupid programmer will remain a stupid programmer, even if he reaches a certain productivity by living off of stackoverflow ...

There is nothing wrong with copy pasting code from stack overflow.

I do see two kind of people doing that. One group learns from the code in order to become better, and can use it over and over to be more efficient. The other group doesn't care how it works and just wants to have a snippet that works.

The second group usually misses a curiosity, of which the effects show up in many more places than just copy pasting from stack overflow. They also tend to have a flatter learning curve. I don't want to generalize, but in this group you will encounter people who don't care about the difference between a list and a set, or think that code works when it compiles. In both cases the juniors know very little, but one grows and the other one doesn't (or less)

There is space for both in the world, but I prefer the first group in my team.

Re: Drunk Post: Things I've Learned as a Sr Engineer

#497

Earlier quoted context omitted.

I've been a "full-stack" developer at large tech companies, and my experience is there at least it means "frontend developer who can put together a basic API server". My fellow full-stack developers and I would spend most of our time building out frontends, which was generally regarded by others as challenging and specialized work, and maybe 20% of the time adding API endpoints to fetch or update some data, which was…

> my experience [full-stack] means "frontend developer who can put together a basic API server" This is 100% accurate in my experience too, and it's also true in the other way around: "full-stack" means backend developer who can put a basic SPA using React/Vue. From the frontend perspective: they call themselves full-stacks for knowing how to spin up a NodeJS HTTP Server powered by Express with MongoDB inserting JSON…

Full stack is not really a skill level qualifier - that's what junior, senior, principal, staff, etc. are for. It means you work in different areas and can't say "oh I don't do that work here" when someone gives you work in those areas. People call themselves front or back end engineers long before they've mastered it, and you don't have to wait for full stack either.

The front or backend bias you and your parent's comment talk about are team specific things so you don't give a hard front-end task to someone who is biased towards the backend. That still means they can take the less difficult tickets.

You can replace full stack for your example with "extremely good/talented/gifted engineer" and I'm almost sure it has nothing to do with the full stack label because it's the only example in a decade you've found.

But also the jobs that they get feed back into what they call themselves. So even if that's your example, it isn't how the business defines it, which IMO is ultimately why it's just a cost savings label to get people to work harder for similar amounts of pay as front or backend people.

Re: Drunk Post: Things I've Learned as a Sr Engineer

#498

Earlier quoted context omitted.

I'm an intermediate dev, masters degree, 9 years experience, non-FAANG, higher cost of living area (not SV, NYC or NVA), and work with obscure tech and proprietary tools. It sucks that I suck. Oh well.

Obscure tech and proprietary tools do translate… if you can translate them. Programming languages are all the same, so learn 3 or 4 new ones and discover that you can probably write in any language for an interview (then do some in relatively unfamiliar languages for kicks and giggles to practice). Tech is all the same. Take data in, poop data out. That’s the whole job. The formats and protocols change, but once you…

I already know Python, Java, Java for Android, Neoxam script, scripts (bash, bat), and Angular to some degree. I've also used JS, AngularJS, C++, C#, powershell, assembly (Intel), and COBOL in the past. So yeah, stuff translates and it isn't that hard to learn a new language (neoxam is probably the hardest since there is limited documentation and examples).

I don't have any interests in games or drivers. Those aren't applicable in my company either. I am currently working on an Angular site. I will host it on S3 with a Lambda and maybe SQS for a marketing email list. This is tech that we use at my job, and many other places.

Re: Drunk Post: Things I've Learned as a Sr Engineer

#500

Earlier quoted context omitted.

This is a very pessimistic long-term view, in my opinion. Life is extremely, infinitesimally short. If it's at all possible for you, you should try to spend as much of it as you can doing things you love. I know many people can't, but it's bleak to just give up and permanently settle, I think. (Especially if you don't currently have any dependents who rely on you; it changes the equation if you do.)

This is a very western view on life.

Maybe. I'm definitely not necessarily saying it in terms of work or career or anything like that. Just in general, I get a little depressed at the idea of just slogging through most of one's hours for most of one's days in existence.
Post reply on HN