Ask HN: Percent of time spent developing internal tools? • Hacker News Board

Re: Ask HN: Percent of time spent developing internal tools?

#11

Post by JangoSteve » Sat, Nov 10, 2012, 2:59 PM UTC

Of course it all depends on your actual business, income (i.e. how much time can you afford to spend on internal tools), needs, etc. Since you're talking about client-facing software, I'm going to assume you're in a software development company, which narrows it down a little.

For our company, we don't develop internal tools for any given problem until it's a real pain. You'll know when it's an actual pain (as opposed to just an idea), because you'll keep seeing it crop up in your day-to-day work, and if you talk about it with others on your team, they'll say, "Yeah! I know exactly what you mean, I was thinking about that the other day."

We wait until it's a real pain for a couple reasons:

1) We want to make sure we're not wasting our time; we're mostly engineers, and we all get real, unadulterated enjoyment from building things. It's only too easy for our enjoyment and enthusiasm for building a solution to fool us into thinking there's a problem. But usually, if everyone on the team is instantly excited about the prospect of solving some problem, that's usually a good indicator that it's a real problem.

2) The longer we can put off something, the more information we'll have when we go to solve it. In other words, we give ourselves time to figure out first how we want to work, and then find or build tools around that, rather than trying to fit our system to some tool.

The second reason is really important too, because when we sit down to really think about every facet of a problem, how it affects our work and how we have been putting up with it and working around it, we'll often realize the best solution isn't some new piece of software; it's merely leveraging some tool we already have slightly differently.

For one example, we started losing feedback and updates in email, because some person on the team would get left off the email (e.g. someone hit "reply" instead of "reply-all"), and we wouldn't realize it until some task went un-resolved for several days. We could have built or implemented some sort of support desk system and made it the clients' responsibility to make sure tickets are submitted and then integrated it with email. Or, we could just create a unique email alias for each client, which copies everyone involved. The latter took 2 minutes and improved communication threefold.

If you keep growing, you'll eventually reach a point where anything less than a full-blown software solution just isn't going to cut it. But by that time, you've seen how the team deals with the problem over time, and all the pitfalls of other smaller or less-custom solutions, and you'll be able to build the most awesome solution possible.

Re: Ask HN: Percent of time spent developing internal tools?

#12

Post by asher_ » Sat, Nov 10, 2012, 3:10 PM UTC

I see I need to add more specifics about my situation.

I work at a university developing research software (primarily - some is also external customer facing). My customers are internal customers, thus profitability and other business measures are not directly my concern.

The biggest issue I face with this is that my customers are non-technical people, so if I were to tell them that our infrastructure is now software defined and I have implemented analytics that will allow me to optimise the caching layer, they will most likely look at me with a puzzled expression and perhaps wonder what I'm actually doing since those things don't provide direct value to them.

The types of internal tools I am thinking of are things like; analytics, automated deployments, CI server etc.

One comment suggested I do a simple calculation like any other business decision. This is something I cannot for the life of me work out how to do. Internal tools will increase my efficiency in the future, so I guess at best I could estimate a payback time for some of these tools (like automation). Even so, how short a payback makes it worthwhile? For other tools, like analytics, how can this type of calculation be made?

Additionally, there are some things that are the "right way" to do things, such as software defined infrastructure vs managing it manually (which I currently do). Investing time in this won't have any immediate payback, but is nice to have in place in the case of a large failure when I need to recover quickly. These types of value calculations for me are not easy to make.

sourthyme, I like the idea of spending one day a week working on this stuff. Perhaps I need to have a conversation with my bosses/customers about this and try to explain to them the value that internal tooling will bring.

Thanks for the answers so far, please keep them coming.

Re: Ask HN: Percent of time spent developing internal tools?

#13

Post by asher_ » Sat, Nov 10, 2012, 3:15 PM UTC

JangoSteve wrote: post #11

Of course it all depends on your actual business, income (i.e. how much time can you afford to spend on internal tools), needs, etc. Since you're talking about client-facing software, I'm going to assume you're in a software development company, which narrows it down a little. For our company, we don't develop internal tools for any given problem until it's a real pain. You'll know when it's an actual pain (as oppose…

Thanks! I'm in agreement with you, I don't like spending a lot of time on tooling if it isn't needed. I like only very simple tools, as I find complex ones inevitably add friction in the workflow and thus end up unused.

Re: Ask HN: Percent of time spent developing internal tools?

#14

Post by taliesinb » Sat, Nov 10, 2012, 3:21 PM UTC

My answer at this very moment in time is around 1/3. Which seems a lot. Though these projects have been less about automating drudgery and more about unlocking new workflows and ways of understanding things.

When I first started working at my job I sneakily built a suite of tools for visually working with NL grammars (instead of just buckling down and writing the grammars). Although I don't maintain it anymore, it's now used by an entire team. It was great that my then-managers saw that I was on to something and let me obsess about it for a few weeks.

I recently spent a few weekends writing static code analysis tools to analyze our codebase. There is some pretty exciting stuff that can be done there. It hasn't paid off quite yet, though I know it will.

Another project I've been juggling with my 'official job' is a nice new logging system with all kinds of cool features. There are so many challenges about making a piece of infrastructure work technically and culturally across a large team, however. It's much easier to estimate how long the technical part will take.

I've also written a bunch of tools that just failed to find any real use. In retrospect, these were mostly vanity projects of some kind, whereas the projects that have worked have always been about scratching an itch that I can see other people have (whether or not they articulate it).

Re: Ask HN: Percent of time spent developing internal tools?

#15

Post by duquesne » Sat, Nov 10, 2012, 3:22 PM UTC

6ren wrote: post #4

Tools are automation. Rule of thumb: do it manually three times before automating it. That way, you'll have a fair idea of what it is supposed to do (which is the hardest part); you know whether it's predictable enough to be automated; and an how often it will come up. A programmer is someone who takes 5 hours to automate something that takes 5 minutes

Even that 5 hours / 5 minutes is worth it if you're going to be doing it enough.

Re: Ask HN: Percent of time spent developing internal tools?

#16

Post by sigsergv » Sat, Nov 10, 2012, 3:23 PM UTC

When you really need tools, couldn't survive without them and couldn't buy some 3rd-party so then you have to create some internal piece of software. And it's not a matter of time or resources, just accurately calculate will it help or not and decide.

Of course internal software MUST be developed using the same quality standards as your frontend program (VCS/tests/code review/etc).

Re: Ask HN: Percent of time spent developing internal tools?

#17

Post by chewxy » Sat, Nov 10, 2012, 3:26 PM UTC

My team and I spent a lot of time developing internal tools (and/or integrating them with other tools like CampaignMonitor and stuff).

Throughout our massive amounts of failures, we've learned some things (and some things the hard way): there are basics, and there are nice-to-haves. Get the basics right.

Our basics include an entirely zeroMQ based service-oriented architecture (after 8-9 different iterations over the past 4 years). We spent a lot of time writing a ZMQ based DHT, and service wrappers (kinda like zeroRPC[0]). Other basics include a standardized front-end A/B testing based off genetify[1], email service (with AWS' SES), paypal service, our fair matching services and many others.

The one thing to note is that these basics didn't come overnight though. It came through many many iterations of different products that failed. We merely picked up the components that worked best.

There were things that were nice to have like automated testing and deployment options that we considered. Even using tools like Puppet and/or Chef, we determined, were not quite a good use of our time, considering our release schedule isn't all that crazy, and deployments can be managed by one human being.

But our situation does not apply to you. You need to think about what fits your situation, and your styles. Maybe you are the kind that pushes a release to production servers every 5 minutes. Then yes, having an automated test and build system would be extremely useful.

Start with simple stuff like crons and tiny scripts. Let your internal tools evolve on their own. Don't jump in to write massive automation for your systems. Instead, do it in small parts, just enough to help you.

Hope this helps.

[0] http://zerorpc.dotcloud.com/

[1] http://genetify.com/demo/

Re: Ask HN: Percent of time spent developing internal tools?

#18

Post by tinco » Sat, Nov 10, 2012, 3:45 PM UTC

asher_ wrote: post #12

I see I need to add more specifics about my situation. I work at a university developing research software (primarily - some is also external customer facing). My customers are internal customers, thus profitability and other business measures are not directly my concern. The biggest issue I face with this is that my customers are non-technical people, so if I were to tell them that our infrastructure is now software…

In my opinion, you should not tell people who do not understand how analytics will improve your caching optimization that you are working on analytics. You should tell them you are working on improving the caching layer. If you sound confident that implementing analytics is required for this job to be done properly they will have to trust you in this, _you_ are the programmer.

You are correct in your assertion that when people ask you what you are doing, they want to know what value you are providing for them. So don't answer with technical details, you already know they're not interested in that :) Whenever a business stakeholder asks you what you are doing, trace back how what you are doing provides value for them and inform them of that.

Note that value does not have to be very direct. Caching makes your application faster, its business value is that the customers are happier when using the product. It is for the marketing department to decide wether that is worth it.

I would not bother too much with figuring out the exact numbers, they are not important. You should for yourself decide wether something is useful or not, if you are an experienced programmer I think this intuition will be right. Almost always, tooling will reduce your technical debt.

Tooling is not only automation, it is also process documentation.

Re: Ask HN: Percent of time spent developing internal tools?

#19

Post by tinco » Sat, Nov 10, 2012, 3:53 PM UTC

6ren wrote: post #4

Tools are automation. Rule of thumb: do it manually three times before automating it. That way, you'll have a fair idea of what it is supposed to do (which is the hardest part); you know whether it's predictable enough to be automated; and an how often it will come up. A programmer is someone who takes 5 hours to automate something that takes 5 minutes

There are three cases for a task that takes a regular human 5 minutes and a computer programmer 5 hours to automate:

1 The task is very complex, this means that there's a lot of room for human error. A well-tested and specified program will reduce this fragility and improve predictability.

2 The programmer is inexperienced, the programmer either is not a very good programmer (yet) or is using tools that he is not familiar with. The 5 hours mostly go into the learning process of the programmer, not in the task itself.

3 The task is simply something computers are not very good at, or hasn't been fully explored. For example speech recognition. The 5 hours go into exploring the domain, creating something reusable that the world could profit from.

In my opinion, it's always worth it, isn't that nice?

Re: Ask HN: Percent of time spent developing internal tools?

#20

Post by meaty » Sat, Nov 10, 2012, 3:54 PM UTC

I have actually automated myself out of a job before so be careful! Fortunately the company saw the value of this and contracted me on an adhoc basis.

I wad the unfortunate administrator for a large virtual mail system in the uk which was run off exim and flat files across 50 machines (one per client). This took 3 guys full time to manage as the machines were all over the uk and needed regular maintenance. I centralized everything and moved it to two virtual postfix machines backed with postgresql and wrote a web interface and set up scripts for backup and administrative operations, including integrated dns management.

Now, 12 years later, these guys are one of the larger hosting companies in the UK...

Automation almost always pays off, but not how you anticipate in some cases.