Ask HN: How do you organize a platform team?

#1

Post by davideberdin » Tue, Sep 13, 2022, 12:46 PM UTC

I am a new manager of a large team (12 SREs) that are taking care of the Kubernetes platform in my company. This team is responsible for the provisioning pipelines (for both baremetal and AWS - no EKS is used), the Kubernetes controllers to integrate with other custom services, the observability stack, etc. The total fleet in use is around 6000 baremetal nodes and 1000 VMs in AWS spread over various DCs and regions. There are over 1500 developers actively using the Kubernetes clusters every day for a total of 2500 applications running in production.

The team spends a lot of time in operations as well as solving compliancy issues, vulnerability patching and customer support. The struggle I'm having is "how to drive focus" and avoid to die of operations. The team is large making the Scrum process ineffective. Every time I try to define teams and to split the people I realise that everything on the platform is so interconnected that the moment I would create 2 or 3 separate teams they would start being on top of each other.

What would you recommend to do?

Re: Ask HN: How do you organize a platform team?

#2

Post by asim » Tue, Sep 13, 2022, 1:08 PM UTC

I wouldn't split the team just because personally I think it'll weaken the concept of the "platform team". Ideally what you want is people with different priorities/focuses within the team. Some on low level ops, others on high level APIs or services. Identifying who's skills fit what and letting them do that is key.

The other thing is just to split the category of work into 3 things; p1 bug fixes / long term projects / support work. Each week just make a note of time allocation for each based on what's happening (sometimes a p1 fix can take up the whole week). Try to minimise the support burden by creating office hours and defining SLAs for the rest of the company.

Make sure your team is not getting buried in support work. What's going to help them is just being able to filter out what's an immediate priority versus pushing off to tomorrow or the day after. Don't let them get bogged down or pinged constantly. Try to make that request flow async.

And most importantly, give them the time to accomplish tasks they think are most important. They are deep in the trenches and know what's going to be p1 vs not. Trust in their ability to guide the outcomes.

Re: Ask HN: How do you organize a platform team?

#3

Post by jrowley » Tue, Sep 13, 2022, 1:11 PM UTC

This sounds challenging! First I’d recommend kanban as opposed to scrum for the team, as the team is so large it is probably hard for them to be really working on a single scrum board effectively.

2nd I’d make sure the team had space to do their operational duties, possibly on a rotation so that other members can focus more on platform development. It can be tempting to try to minimize the operational work, especially when coming from more of a product oriented team. But it is important to develop good process to support the recurring operational labor.

Finally within that team of 12 I’d identify the most senior devs and tech leads and work with them closely on these process changes. Try to understand their problems and ensure they understand the goals coming down from the VPs etc.

Sorry if this is basic / already in play. Good luck!

Re: Ask HN: How do you organize a platform team?

#4

Post by davideberdin » Tue, Sep 13, 2022, 1:16 PM UTC

asim wrote: post #2

I wouldn't split the team just because personally I think it'll weaken the concept of the "platform team". Ideally what you want is people with different priorities/focuses within the team. Some on low level ops, others on high level APIs or services. Identifying who's skills fit what and letting them do that is key. The other thing is just to split the category of work into 3 things; p1 bug fixes / long term project…

Thanks for the tips. I do appreciate them :)

`Don't let them get bogged down or pinged constantly. Try to make that request flow async.` - This is very true yet so hard. Slack is not helping at all with this issue. I will check with the team about office hours or maybe a triaging system to minimize the Support effort.

Re: Ask HN: How do you organize a platform team?

#5

Post by davideberdin » Tue, Sep 13, 2022, 1:19 PM UTC

jrowley wrote: post #3

This sounds challenging! First I’d recommend kanban as opposed to scrum for the team, as the team is so large it is probably hard for them to be really working on a single scrum board effectively. 2nd I’d make sure the team had space to do their operational duties, possibly on a rotation so that other members can focus more on platform development. It can be tempting to try to minimize the operational work, especiall…

Thanks for the tips! I appreciate them :)

I've been noodling for a while regarding Kanban. Where do you see the main advantages in using it?

Re: Ask HN: How do you organize a platform team?

#6

Post by asim » Tue, Sep 13, 2022, 1:28 PM UTC

davideberdin wrote: post #4

asim wrote: post #2

I wouldn't split the team just because personally I think it'll weaken the concept of the "platform team". Ideally what you want is people with different priorities/focuses within the team. Some on low level ops, others on high level APIs or services. Identifying who's skills fit what and letting them do that is key. The other thing is just to split the category of work into 3 things; p1 bug fixes / long term project…

Thanks for the tips. I do appreciate them :) `Don't let them get bogged down or pinged constantly. Try to make that request flow async.` - This is very true yet so hard. Slack is not helping at all with this issue. I will check with the team about office hours or maybe a triaging system to minimize the Support effort.

If you have a #platform team channel don't let that become the place the rest of the company comes and makes unfiltered requests. That's the channel for your team. It's really important to have that boundary, otherwise they don't have a safe space to communicate and have to resort to private channels. People should be able to see what the platform team discusses but not necessarily just start asking random support questions.

Also, try create a knowledge base for the rest of the company. If there's frequent issues that are already resolved which others could just search a page for or some runbook for things they can do themselves it's best to give them that "self serve" method.

Re: Ask HN: How do you organize a platform team?

#7

Post by davideberdin » Tue, Sep 13, 2022, 1:50 PM UTC

asim wrote: post #6

davideberdin wrote: post #4

Earlier quoted context omitted.

Thanks for the tips. I do appreciate them :) `Don't let them get bogged down or pinged constantly. Try to make that request flow async.` - This is very true yet so hard. Slack is not helping at all with this issue. I will check with the team about office hours or maybe a triaging system to minimize the Support effort.

If you have a #platform team channel don't let that become the place the rest of the company comes and makes unfiltered requests. That's the channel for your team. It's really important to have that boundary, otherwise they don't have a safe space to communicate and have to resort to private channels. People should be able to see what the platform team discusses but not necessarily just start asking random support qu…

Makes sense -- Knowledge Base is something I should definitely tackle. Thanks :)

Re: Ask HN: How do you organize a platform team?

#8

Post by stocktech » Tue, Sep 13, 2022, 1:56 PM UTC

Splitting teams with interconnected and related work, I do something like https://agilesquads.org/. For each sprint, we'll have a planning cadence where we scope 2 weeks of work e.g. Feature X and Feature Y. Each squad would get assigned a feature and they're able to focus on delivering that feature. The benefit of this over a real "split" is that you can have both "teams" working on the same roadmap/project/feature and/or change how many engineers are working towards a single feature. When you include "squad leads", it's also great for career development and leadership.

On product work, we do an oncall rotation of 1 engineer per week that triages issues and handles prod outages. This may solve for your 'help desk' type work.

Re: Ask HN: How do you organize a platform team?

#9

Post by davideberdin » Tue, Sep 13, 2022, 2:12 PM UTC

stocktech wrote: post #8

Splitting teams with interconnected and related work, I do something like https://agilesquads.org/ . For each sprint, we'll have a planning cadence where we scope 2 weeks of work e.g. Feature X and Feature Y. Each squad would get assigned a feature and they're able to focus on delivering that feature. The benefit of this over a real "split" is that you can have both "teams" working on the same roadmap/project/feature…

Thanks for sharing the link and your advices. I've never thought about squads as they are explained in the website.

Re: Ask HN: How do you organize a platform team?

#10

Post by danwee » Tue, Sep 13, 2022, 3:24 PM UTC

> The total fleet in use is around 6000 baremetal nodes and 1000 VMs in AWS spread over various DCs and regions. There are over 1500 developers actively using the Kubernetes clusters every day for a total of 2500 applications running in production.

Seems like they are quite successful already.

> The struggle I'm having is "how to drive focus" and avoid to die of operations. The team is large making the Scrum process ineffective.

Drive focus to where? As stated above, the platform team seems to have achieved what every platform team other there is struggling to achieve. What does "die of operations" mean?

And above all, who cares about Scrum?