Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#11

Post by vii »

Real traffic exposes wrong assumptions in code, which cannot be caught with unit or integration testing. Awesome to automate this kind of testing. I encourage people to invest heavily in setting up artificial environments to replay historical data.

One benefit of the artificial environment is efficiency (can cover e.g. a week of historical data); this also requires mocking out time with simulation/replay time. The integration with data-platforms makes a big difference as it means data-scientists can help set up the right test scenarios.

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#12

Post by d_watt »

Using products like this in the pastI've run into a pretty simple issue:

- Request 1 is a post that generates a "todo" with a random id "5y22"

- Request 2 is a "get" for /todo/5y22.

That works in production, but on replay of the traffic:

- Request 1 generates a different random id, "86jj".

- Request 2 is still a replayed "get" for /todo/5y22, which is now 404.

How does your tooling handle this nondeterminism in replays?

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#13

Post by mleray »

2rsf wrote: post #10

Sounds great, for using this in a financial environment it will need an option for data anonymization, I'm not sure how can you identify what needs to anonymized without human interaction though.

You're spot on :). We got this feedback from one of our financial services alphas so we built a DLP rules engine to cover it. That wasn't enough. So we offered to integrate with Google DLP. Still no. So in the end we architected for a split-plane architecture (similar to DataBricks) so big customers can host their own data but we can manage the control stack. It's not something we're doing during the alpha but it's part of the plan. Would that work?

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#14

Post by kahrensdd »

d_watt wrote: post #12

Using products like this in the pastI've run into a pretty simple issue: - Request 1 is a post that generates a "todo" with a random id "5y22" - Request 2 is a "get" for /todo/5y22. That works in production, but on replay of the traffic: - Request 1 generates a different random id, "86jj". - Request 2 is still a replayed "get" for /todo/5y22, which is now 404. How does your tooling handle this nondeterminism in repla…

For one thing we look at both inbound and outbound traffic and we treat it separately. That use case looks different if we are trying to "test" TODO or if the TODO service is a backend that our app relies upon.

So if you mean that we want to "test" TODO, our analyzer looks for data in subsequent requests that was provided by the system-under-test (SUT) in previous responses. A common example of this is an HTTP cookie. The SUT gives us a session id through the Set-Cookie header response. So in a subsequent request we use the cookie from the app, not the one that was recorded. This has been done in general way to look for tokens.

Of course nobody is perfect so we'd love to see your real world app and test our algorithms against it. :)

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#15

Post by mrkurt »

kahrensdd wrote: post #9
mrkurt wrote: post #8

Can you run tests from different geos?

You can run it in your own environment. If you're running Kubernetes we provide an operator that orchestrates the test runs. If you are using docker we give you containers that you control with ENV VARs. Do you have more background on the multi geo use case?

We run a multi-geo service (Fly.io). Replicating user load on distributed apps is hard.

Containers with env vars are easy though!

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#16

Post by kahrensdd »

mrkurt wrote: post #15
kahrensdd wrote: post #9

Earlier quoted context omitted.

You can run it in your own environment. If you're running Kubernetes we provide an operator that orchestrates the test runs. If you are using docker we give you containers that you control with ENV VARs. Do you have more background on the multi geo use case?

We run a multi-geo service (Fly.io). Replicating user load on distributed apps is hard. Containers with env vars are easy though!

Yes I recently went through a similar use case with one of my alpha users. They wanted to run the reverse proxy and playback as docker containers spread through their environment. Will drop you an email with more info...

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#17

Post by atombender »

d_watt wrote: post #12

Using products like this in the pastI've run into a pretty simple issue: - Request 1 is a post that generates a "todo" with a random id "5y22" - Request 2 is a "get" for /todo/5y22. That works in production, but on replay of the traffic: - Request 1 generates a different random id, "86jj". - Request 2 is still a replayed "get" for /todo/5y22, which is now 404. How does your tooling handle this nondeterminism in repla…

I don't think that qualifies as non-determinism. This is just dependencies between operations.

Non-determinism would be, for example, something that's time-sensitive. If some result varies by time, then the only way to test it is to include time as a parameter. This can be complicated if the time variable plays into asynchronous updates (e.g. you want to test that a POST update worked, but it's actually eventually consistent). Caching (e.g. through Varnish or a CDN) would be another thing to make such tests much more complicated.

Another example is an API that has side effects. For example, a stock trading API might read real time quotes from another service. A stock trade then alters the next quote.

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#18

Post by nserrino »

This is cool, is the traffic curated in any way? Like if the database isn't initialized, do you start with create requests before moving on to GETs for those IDs? Also does this only support HTTP or does it support other protocols as well?

Re: Launch HN: Speedscale (YC S20) – Automatically create tests from actual traffic

#20

Post by mleray »

nserrino wrote: post #18

This is cool, is the traffic curated in any way? Like if the database isn't initialized, do you start with create requests before moving on to GETs for those IDs? Also does this only support HTTP or does it support other protocols as well?

Ultimately, the idea is to mock the database itself so we just return whatever the real database returned during the recording. We don't have to run create commands because we aren't actually managing a real database's internal state. We "only" need to accurately return the responses the database gives the system-under-test for a particular GET sequence. During the alpha we are limiting support to HTTP/s but protocols like MongoDB, redis, MySQL, etc are on the backlog. Until we have more database support we're asking alpha customers to deploy test data in a test database, which seems to be a fairly normal part of the CI process for big apps.
Post reply on HN