Live data from Hacker News

Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

github.com

31–37 of 37 posts

Re: Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

#31

Whenever I see tools like this I always think "that wouldve been great at my old job where we didn't do post mortems" But nowadays I think if I can automate a runbook can I not just make the system heal itself automatically? If you have repeated problems with known solutions you should invest in toil reduction to stop having those repeated problems. What am I missing? I think I must be missing something because these…

A lot of on call teams lack the capability to do that automation, either because ops takes the pages and can't code (or can't code well enough) or because dev takes the pages and have no access or knowledge about the infra APIs they could use for self-healing.

These platforms can form a sort of "common ground" where dev can see the infra APIs and the "code" is simple enough for ops people that don't code to rig stuff up.

I don't think these platforms are built for the kind of places where being able to write a Python script to query logs from CloudFront are just table stakes for all ICs regardless of role.

Re: Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

#32

Whenever I see tools like this I always think "that wouldve been great at my old job where we didn't do post mortems" But nowadays I think if I can automate a runbook can I not just make the system heal itself automatically? If you have repeated problems with known solutions you should invest in toil reduction to stop having those repeated problems. What am I missing? I think I must be missing something because these…

Writing post-mortems is generally pretty kludgy. You might have a Slack bot that records the big picture items, but ideally, a post-mortem would include connections to the nitty-gritty details while maintaining a good high-level overview. The other thing most post-mortems miss is communicating the discovery process. You'll get a description of how an engineer suspected some problem, but you rarely get details as to how they validated it such that others can learn new techniques. At a previous job, I worked with a great sysadmin/devop who would go through a concise set of steps when debugging things. We all sat down as a team, and he showed us the commands he ran to confirm transport in different scenarios. It was an enlightening experience. I talked to him and other DevOps folks about Rundeck, and it was clear that the problem isn't whether something can be automated, but rather whether the variables involved are limited enough to be represented in code. When you do the math, the time it would take to write code to solve some issues is not worth the benefit.

Iterating on the manual work to better communicate and formalize the debugging process could fit well into the notebook paradigm. You can show the scripts and commands you're running to debug while still composing a quality post-mortem, as the incident is happening where things are fresh.

The other thing to consider is how often you get incidents and how quickly you need to get people up to speed. In a small org, devs can keep most things in their head and use docs, but when things get larger, you need to think about how you can offload systems and operational duties. If a team starts by iterating on operational tasks in Notebooks, you can hand those off to an operations team over time. A quality, small operations team can take on a lot work and free up dev time for optimizations or feature development. The key is that devs have a good workflow to hand off operational tasks that are often fuzzier than code.

The one gotcha with a hosted service IMO is that translating local scripts into hosted ones takes a lot of work. On my laptop, I'm on a VPN and can access things directly, where you need to figure out how to allow a 3rd party to connect to production backend systems. That can be a sticky problem that makes it hard to clarify the value.

Re: Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

#33
post #12

This is awesome, i've seen so many static runbooks (like confluence) and SREs will scan it once, not find what they need and then go wake up a senior dev. Pre-programmed scripts could go a long way in giving the SRE the ability to go that extra step, which could be vital to solving the problem faster.

Isn't that already possible via normal Python scripts? I've worked a couple places where dev had a "don't wake us up" script that was programmed to detect known and common issues and either fix it or offer recommendations on next steps (including a couple of code paths that led to a "page everyone, immediately and repeatedly").

From the SRE side, far and away the most common reason I end up paging devs is because the issue is somewhere deep inside the system and I lack that depth. I'm supporting half a dozen services and can't keep track of the churn that happens at high enough granularity. Eg I know the app's downstreams and most of its upstreams, but if there's an issue with a particular field in an API response I'm unlikely to know whether that field comes from our database, a downstream, summoned by voodoo, etc.

Still interesting to see, I'd love to be proven wrong.

Re: Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

#34
post #15

If it works like Jupyter, as a file that can be version controlled, and like Deepnote where multiple people can be viewing/working on it at the same time, my mind would be blown.

here, be blown away https://github.com/opral/monorepo/tree/main/lix solving version control for files like jupyter notebooks brings collaboration to those files without the need to give up files in favor of the cloud. playbooks could leverage lix in 1-2 years to build a file-based version of their tool

Wow, yeah. "Bringing backend features to files."

This feels a bit like that time we saw Etherpad playback for the first time. I'm just not sure if I've grokked the big picture yet.

https://news.ycombinator.com/item?id=495336

Re: Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

#35
post #34

Earlier quoted context omitted.

here, be blown away https://github.com/opral/monorepo/tree/main/lix solving version control for files like jupyter notebooks brings collaboration to those files without the need to give up files in favor of the cloud. playbooks could leverage lix in 1-2 years to build a file-based version of their tool

Wow, yeah. "Bringing backend features to files." This feels a bit like that time we saw Etherpad playback for the first time. I'm just not sure if I've grokked the big picture yet. https://news.ycombinator.com/item?id=495336

big picture is that cloud-based apps/saas is getting disrupted.

there is no value in a cloud-based solution that locks users and customers in if collaboration can be solved in the data (file) level. turns out that version control solves collaboration on the data level and is awesome to build apps.

Re: Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

#36

Reminds me of Rundeck and the time we were trying to build something similar. There are more modern take like fiberplane and moment.dev. Not sure about their adoption. At one point, we were building something like this on top of kubernetes. I think tech is the easy part here. Getting people to leave their existing workflows and use your product is hard. Secondly, difficult part of our journey was integrations. Until…

I was looking at using moment.dev for a very similar (internal) application but the lift of using TypeScript and how the whole tool worked was very daunting. Having a simple Jupyter notebook interface (in Python) is much more approachable for a devops background.

Re: Show HN: PlayBooks – Jupyter Notebooks style on-call investigation documents

#37
post #36

Reminds me of Rundeck and the time we were trying to build something similar. There are more modern take like fiberplane and moment.dev. Not sure about their adoption. At one point, we were building something like this on top of kubernetes. I think tech is the easy part here. Getting people to leave their existing workflows and use your product is hard. Secondly, difficult part of our journey was integrations. Until…

I was looking at using moment.dev for a very similar (internal) application but the lift of using TypeScript and how the whole tool worked was very daunting. Having a simple Jupyter notebook interface (in Python) is much more approachable for a devops background.

In my experience, getting devops and infrastructure engineers to use Jupyter notebook specifically for SRE stuff is hard. What is working for is, in our new pivot is, you have to meet where the engineers are at. It could be Jetbrain's tools, VSCode or terminal. Otherwise the lift is always too much. In my opinion, Jupyter way might be better but still not good enough to cross over.
Post reply on HN