Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How to price a service that saves hundreds of engineering hours?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 89 posts

Ask HN: How to price a service that saves hundreds of engineering hours?

#1

Post by punkpeye »

I've built a Cypress test runner that completes all tests (no matter how many you've got) in less than a minute. In contrast, based on my experience, a medium-size Cypress test suite takes about 15 minutes to complete.

The only catch is that individual tests have to be under 30 seconds.

The idea came out of frustration running Cypress in my previous company, where it was taking 10 minutes+ to run all Cypress tests while parallelizing across 30 VMs and costing us in excess of USD 2k/month.

In order to achieve this, I have effectively built a new Cypress test runner from the ground up. It understands Cypress syntax, but otherwise have nothing in common with how Cypress works. The way it achieves this performance is by splitting each spec into individual tests and starting all of the tests at once, i.e. if you have 10 specs with 5 tests each, this program will start 50 VMs to run your tests.

I have two companies trialing this at the moment and the feedback has been incredibly positive, saying that it is saving _hundreds_ of engineering hours. I am trying to establish how to price this. The challenge is that this model is profitable at scale, but losing money if there is not high density of clients. This is because it costs me USD ~0.15 to run a VM for 1 hour and I need to spin up enough VMs to complete all tests, and I am charged in increments of an hour.

My thinking is to charge 2 cents per a test-minute, i.e. Using previous example of 10 specs with 5 tests each, it would cost USD 1 to run all tests once. If you run integration tests 100 times per day, that's USD 100/day.

This may sound much. However, if prior to using this you were waiting 15 minutes to run all tests, that is 14 minutes saved. If avg. engineer in your company is earning USD 60/hour, you are saving USD 14 by having engineers get immediate results rather than waiting for them. If positioned that way, it doesn't sound expensive.

I am currently targeting companies with 30-50 engineers (existing customers are series A and series B companies). At this size, they don't have crazy amount of tests, so I can deliver on the 1 minute promise, and they care a lot about moving fast.

What sounds reasonable?

Re: Ask HN: How to price a service that saves hundreds of engineering hours?

#2

Post by codingdave »

It is an interesting idea, but your approach to pricing seems off:

100/day is not reasonable, at least not for small/medium teams. Trying to do arithmetic on run times and hourly rates to convince people it is reasonable is not only overly simplistic, but actively distracts from the conversation you really want to have - how does your solution solve their problem? People buy when they believe you can solve a problem better than they could without you. So keep the conversation in that space and don't set up pricing that invites nitpicky arguments about hourly costs, etc.

Instead, I'd just price based on your own needs - figure out your actual operating cost per test-minute, tack on a reasonable profit margin and operating costs, and bundle up some packages that have a quota of test-minutes per month in sizes ranging from 20/month to 100/month. Make sure that you are profitable at every package, and see if you get takers at that price point. Then let prices increase naturally when they run out of minutes and need to ask for more.

Re: Ask HN: How to price a service that saves hundreds of engineering hours?

#4

Post by colechristensen »

Your minimum price is what it actually costs you, your maximum price is whatever you can get. You can convince customers to pay your price by demonstrating how much your product will save them and charging a discount on that amount.

Re: Ask HN: How to price a service that saves hundreds of engineering hours?

#6

Post by holistio »

I think if you find companies who do run integration tests 100 times per day, paying 2-3k per month to save around 500 hours per month does make sense.

You really have to work out your ideal customer, though. And find a way to scale it down - you can potentially use the capacity of more VMs spread across many smaller companies.

Maybe your business model is having a few clients pay you a couple grand a month, maybe it is having a couple hundred paying you 20-50 or so a month.

You do seem to have solved a problem, which is a great first step.

Re: Ask HN: How to price a service that saves hundreds of engineering hours?

#7

Post by issa »

Just to use your example, you would have a VERY hard time convincing me to spend $3,000 a month to speed up testing time. Running tests isn't exactly dead time. It gives engineers a break, they can think about other things while running them, get coffee, etc. There is DEFINITELY value there, but not $3k a month worth.

That said, pricing correctly is hard and requires a lot of experimentation. Growth and adoption might be more important at this stage.

Re: Ask HN: How to price a service that saves hundreds of engineering hours?

#8

Post by Sholmesy »

VoidWhisperer wrote: post #3

With Cypress, atleast in what I've heard of (and seen), flaky tests seem to be a larger waste of developer time than the time it takes to run the test suite overall. Does this help improve that?

My experience with flaky tests is that they seem to leave some state lying around, or are impacted by some randomness that wasn't accounted for in the test writing (lists in different order, DB state etc).

So I actually do think this would benefit that, as these tests are run "independantly", however that's just going to hide the problem of your bad/flaky tests, and you'll never solve the underlying problem.

Re: Ask HN: How to price a service that saves hundreds of engineering hours?

#9

Post by punkpeye »

VoidWhisperer wrote: post #3

With Cypress, atleast in what I've heard of (and seen), flaky tests seem to be a larger waste of developer time than the time it takes to run the test suite overall. Does this help improve that?

It sure does. The way Cypress works is that it injects their entire driver (test API) to the browser and then loads the tests themselves, also in the browser. It is a finicky process because they need to rewrite all JS and HTML (https://github.com/cypress-io/cypress/tree/develop/packages/...). It is slow because they have to proxy everything through HTTP 1.1 proxy (https://github.com/cypress-io/cypress/tree/develop/packages/...).

The thing that I built takes a different approach. It parses Cypress tests, interprets the commands that Cypress wants to run, and then uses Chrome DevTool protocol (CDP) to evaluate commands such as navigation, click, etc. If you are familiar with the JavaScript ecosystem, my tool effectively transforms tests to Playwright compatible tests and then uses custom orchestration logic to run them.

Re: Ask HN: How to price a service that saves hundreds of engineering hours?

#10

Post by sanjayio »

I think your pricing math is a tad off. I don't think it's harmful to use that for marketing, like you're saving engineering hours, but when you use it for pricing it falls apart. The reason is that you're selling to engineers, and we all know as CI is running, we aren't blocked. So, it isn't a 1:1 minute saved, although, I wouldn't conflate that with zero saved. In my experience a 1-5 minute test run isn't much different, but 3mins vs 30mins is a huge difference. I wouldn't force an engineering team to chisel 6mins to 2mins for example. I would roadmap how to reduce 30mins to less than 10mins. That being said, I think your equation needs more nuance to sell to an engineering manager or leader.
Post reply on HN