Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incuriosity Will Kill Your Infrastructure

yellerapp.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Incuriosity Will Kill Your Infrastructure

#21

Post by trustfundbaby »

optimusclimb wrote: post #9

So what I want to know, is, as per all the recent agile/scrum discussions - how does the modern "do sprint planning/commit to a number of sprint points to do/tasks to work on/be the product manager's monkeys" align with, "you saw something that's probably representative of a major problem in your system, but stopping what you're doing to investigate it will kill your velocity, and make your team's statistics look bad…

I came here to make this point ... engineering organizations applying scrum or agile make no allowance for something like this. And while you might dig deeper, find a problem that indeed would have been a major crisis, and in some cases get to add that as a Product support ticket, after the fact, if you're lucky, the reality is that a lot of times you'll spend a lot of time investigating, learn that much more about your infrastructure/codebase but find its a small (but not insignificant problem) that gets thrown on the backlog, while the PM seethes at you for screwing up team velocity, and going "rogue" (working on a problem without telling him). Or, maybe, even worse ... you don't find anything, now you just wasted a day on nothing.

Of course you could say something in standup about it, but you and I both know you'd probably be gently admonished for wasting time on it and asked to go back to what you were doing. because ... sprint goals, quarterly objectives yada yada yada.

If you make doing this enough of a habit, it might show up in your one on ones and evaluations even. There goes your big annual raise.

In essence, you have to do your work AND go figure out these sorts of things on your own time. Thats how you find yourself putting in 50-60 hour weeks, but its okay because you're a "passionate" engineer.

Either way The business wins.

Re: Incuriosity Will Kill Your Infrastructure

#22

Post by adrianmacneil »

t__crayford wrote: post #16
jd007 wrote: post #4

Not related to the post, but after going to the main Yeller page, I noticed that at the bottom one of the features listed is "HTTPS Everywhere (we don't even allow HTTP over our API or website)", yet the site is not over SSL at all. In fact manually entering HTTPS in the URL shows that the certificate is not valid.

Hi (author/founder of Yeller) here. You're totally right, I need to change the wording there. The marketing site doesn't run over https - I'm bootstrapping, with relatively limited funds, and so can't properly afford the SSL costs for a CDN (my current one wants to charge $600 or so a month for serving ssl requests). The webapp and the api are all HTTPS only. I should change the wording on that page to reflect that.

I highly recommend CloudFlare. Their basic plan is completely free, and even comes with a free SSL cert.

Re: Incuriosity Will Kill Your Infrastructure

#23

Post by jfroma »

jes wrote: post #2

Embedded software medical device developer here. I once worked with a hardware engineer that would verbalize his thought process very explicitly, as we worked in the lab. He would say things like: "Ok, I'm about to let the target out of reset. I expect to see the I2C bus controller initiate a master read of address 0x80." Then he would do it, and look at the oscope, and see if his expectation was confirmed. If it was…

This is exactly my though process as a software developer. I work remotely so I dont speak out loud (most of time), but i think it helps too. It is similar to when you have been debugging something for hours and you figure the solution right away when you start explaining the problem to a coworker.

Re: Incuriosity Will Kill Your Infrastructure

#24

Post by vacri »

quanticle wrote: post #7
pjungwir wrote: post #5

Not just your infrastructure, your code base too. I've seen a lot of developers practice what I call "debugging by superstition," where they make random changes until it appears to work. I prefer to keep digging until I understand. Sometimes I make a hypothesis and test it, which superficially resembles debugging by superstition but is different. One benefit of experience is that you gain a better intuition about you…

>Sometimes I make a hypothesis and test it, which superficially resembles debugging by superstition but is different. As Adam Savage [1] says, "The only difference between science and screwing around is writing it down." I don't follow this process for every problem I encounter, but when I have a really intractable issue, where nothing I've thought of seems to work, I start a "lab notebook" (usually a few sheets of p…

As Adam Savage [1] says, "The only difference between science and screwing around is writing it down."

Not true at all, though publishing is an important step. Understanding what is going on is important. A startling revelation that the MythBusters guys have no understanding of statistics was when they invented their buttery-toast dropper. Doing a 'calibration' dry run (literal dry run!) with toast with one side marked with an X instead of butter, 7 out of 10 trials read one way. Adam remarked "This isn't random enough - it should be 5!".

The Mythbuster guys get 11/10 for curiosity and the spirit of investigation, but 4/10 for scientific rigour :)

Re: Incuriosity Will Kill Your Infrastructure

#25

Post by quanticle »

vacri wrote: post #24
quanticle wrote: post #7

Earlier quoted context omitted.

>Sometimes I make a hypothesis and test it, which superficially resembles debugging by superstition but is different. As Adam Savage [1] says, "The only difference between science and screwing around is writing it down." I don't follow this process for every problem I encounter, but when I have a really intractable issue, where nothing I've thought of seems to work, I start a "lab notebook" (usually a few sheets of p…

As Adam Savage [1] says, "The only difference between science and screwing around is writing it down." Not true at all, though publishing is an important step. Understanding what is going on is important. A startling revelation that the MythBusters guys have no understanding of statistics was when they invented their buttery-toast dropper. Doing a 'calibration' dry run (literal dry run!) with toast with one side mark…

That's not how I interpreted it at all. To me "writing it down" has nothing to do with publishing. "Writing it down" means that you systematically track the results of your experiments and then you use those results to update your hypotheses. If you don't write anything down, you're debugging by superstition.

Re: Incuriosity Will Kill Your Infrastructure

#26

Post by aptwebapps »

jfroma wrote: post #23
jes wrote: post #2

Embedded software medical device developer here. I once worked with a hardware engineer that would verbalize his thought process very explicitly, as we worked in the lab. He would say things like: "Ok, I'm about to let the target out of reset. I expect to see the I2C bus controller initiate a master read of address 0x80." Then he would do it, and look at the oscope, and see if his expectation was confirmed. If it was…

This is exactly my though process as a software developer. I work remotely so I dont speak out loud (most of time), but i think it helps too. It is similar to when you have been debugging something for hours and you figure the solution right away when you start explaining the problem to a coworker.

Pre-emptive rubber ducking. I'd like to try it, but I'm afraid it might be a nuisance for others.

Re: Incuriosity Will Kill Your Infrastructure

#27

Post by totally »

"Not once have I regretted spending unbounded amounts of time investigating something fishy"

While I agree with the gist here (heed warning signs, proactively preempt failure), there are literally hundreds of "fishy" things, many/most of them low impact, that I could investigate on a given day, and my time is bounded.

At the semi-formal dance of distributed systems, meandering investigation should be chaperoned by ruthless prioritization.

Re: Incuriosity Will Kill Your Infrastructure

#28

Post by vacri »

quanticle wrote: post #25
vacri wrote: post #24

Earlier quoted context omitted.

As Adam Savage [1] says, "The only difference between science and screwing around is writing it down." Not true at all, though publishing is an important step. Understanding what is going on is important. A startling revelation that the MythBusters guys have no understanding of statistics was when they invented their buttery-toast dropper. Doing a 'calibration' dry run (literal dry run!) with toast with one side mark…

That's not how I interpreted it at all. To me "writing it down" has nothing to do with publishing. "Writing it down" means that you systematically track the results of your experiments and then you use those results to update your hypotheses. If you don't write anything down, you're debugging by superstition.

I would not categorise that rather complex set of activities as 'only'. It's not just 'writing down', but analysing, predicting, and designing new experiments.

Re: Incuriosity Will Kill Your Infrastructure

#29

Post by LunaSea »

jes wrote: post #2

Embedded software medical device developer here. I once worked with a hardware engineer that would verbalize his thought process very explicitly, as we worked in the lab. He would say things like: "Ok, I'm about to let the target out of reset. I expect to see the I2C bus controller initiate a master read of address 0x80." Then he would do it, and look at the oscope, and see if his expectation was confirmed. If it was…

This is really interesting to me.

Do you have other stories or experiences to share about programming on "strict" systems like medical devices ?

Stuff like coding styles, rules, practices and the likes.

Re: Incuriosity Will Kill Your Infrastructure

#30

Post by isaacaggrey »

optimusclimb wrote: post #13
isaacaggrey wrote: post #12

Earlier quoted context omitted.

Any time a sprint commitment must be broken the decision rests on the team's product manager who is best (or _should_ be best) suited to understanding the tradeoff between sprint completion and a potential emergency bug situation. If the PM decides it's not worth it, then that's on them. It's worth noting, you may want to include a wide enough distribution so it's known that you noticed but PM didn't want you to work…

Yeah, I figured someone would respond that way. I have a feeling that the more process that lies around making such fixes...the more likely the system is likely to be under performing (and or broken) long term.

In Scrum, speed and efficiency are traded for predictability and consistency (whether or not those trade-offs are worth it is another conversation).

It's worth noting that I agree with your point and also don't believe Scrum is a panacea, but I am starting to understand its appeal from a business perspective.

Post reply on HN