[deleted]
K6: Like unit testing, for performance
71–80 of 86 posts
Re: K6: Like unit testing, for performance
#72I used K6 in 1998. It was not a very good CPU but was cheap.
Re: K6: Like unit testing, for performance
#73It seems like the hard part is deciding when a change is suspicious enough to be investigated? Simple pass/fail tends not to work at scale because complicated systems tend to be noisy.
That's one of the main uses, yes. Although k6 gives you a lot of flexibility to decide what your pass/fail criteria are with its custom metrics ( https://k6.io/docs/using-k6/metrics#custom-metrics ), checks ( https://k6.io/docs/using-k6/checks ) and thresholds ( https://k6.io/docs/using-k6/thresholds ). See the example in https://github.com/loadimpact/k6/#checks-and-thresholds . If, at the end of the test run, some o…
Does K6 deal with the coordinated omission problem?
Gil Tene (Azul Systems) has argued convincingly [1] (slides [2]) that monitoring tools get latency measurement wrong because sudden spikes aren't represented correctly in timings because of averaging and the wrong use of percentiles.
He argues that percentiles simply aren't useful, because, statistically, most requests will experience >= 99.99-percentile response times. All percentiles lie; the only truly useful and realistic number is, in fact, the 100th percentile.
He also argues that the most revealing way to show latency is with a complete histogram.
I haven't used K6 yet, but I've been looking for a good load-testing tool, and intend to try it out.
[1] https://www.youtube.com/watch?v=lJ8ydIuPFeU
[2] https://www.azul.com/files/HowNotToMeasureLatency_LLSummit_N...
Re: K6: Like unit testing, for performance
#74Earlier quoted context omitted.
That's one of the main uses, yes. Although k6 gives you a lot of flexibility to decide what your pass/fail criteria are with its custom metrics ( https://k6.io/docs/using-k6/metrics#custom-metrics ), checks ( https://k6.io/docs/using-k6/checks ) and thresholds ( https://k6.io/docs/using-k6/thresholds ). See the example in https://github.com/loadimpact/k6/#checks-and-thresholds . If, at the end of the test run, some o…
Does K6 deal with the coordinated omission problem? Gil Tene (Azul Systems) has argued convincingly [1] (slides [2]) that monitoring tools get latency measurement wrong because sudden spikes aren't represented correctly in timings because of averaging and the wrong use of percentiles. He argues that percentiles simply aren't useful, because, statistically, most requests will experience >= 99.99-percentile response ti…
That's an interesting talk.
k6 allows the user to choose[1] which metrics are relevant for the particular test. By default, it displays max or p(100), p(95), p(90), min, med, and avg. User can specify other values such as p(99.995)
It's also possible to create completely custom metrics[2] to track whatever is relevant to the user.
k6 allows the user to change almost all aspects of execution, tracking, and reporting.
Re: K6: Like unit testing, for performance
#75Earlier quoted context omitted.
That's one of the main uses, yes. Although k6 gives you a lot of flexibility to decide what your pass/fail criteria are with its custom metrics ( https://k6.io/docs/using-k6/metrics#custom-metrics ), checks ( https://k6.io/docs/using-k6/checks ) and thresholds ( https://k6.io/docs/using-k6/thresholds ). See the example in https://github.com/loadimpact/k6/#checks-and-thresholds . If, at the end of the test run, some o…
Does K6 deal with the coordinated omission problem? Gil Tene (Azul Systems) has argued convincingly [1] (slides [2]) that monitoring tools get latency measurement wrong because sudden spikes aren't represented correctly in timings because of averaging and the wrong use of percentiles. He argues that percentiles simply aren't useful, because, statistically, most requests will experience >= 99.99-percentile response ti…
This used to be an issue with older k6 versions, but since v0.27.0 we have arrival-rate executors [1] that should address the biggest issue of coordinated omission. And we've always measured the max value of all metrics, even when they are not exported to some external output and we just show the end-of-test summary.
It has been a while since I watched these Gil Tene talks, so I might be missing something, but I think the only remaining task we have is to adopt something like his HDR histogram library [2]. And that's mostly for performance reasons, though we'll probably play around with the correction logic as well.
Re: K6: Like unit testing, for performance
#76Earlier quoted context omitted.
Does K6 deal with the coordinated omission problem? Gil Tene (Azul Systems) has argued convincingly [1] (slides [2]) that monitoring tools get latency measurement wrong because sudden spikes aren't represented correctly in timings because of averaging and the wrong use of percentiles. He argues that percentiles simply aren't useful, because, statistically, most requests will experience >= 99.99-percentile response ti…
This used to be an issue with older k6 versions, but since v0.27.0 we have arrival-rate executors [1] that should address the biggest issue of coordinated omission. And we've always measured the max value of all metrics, even when they are not exported to some external output and we just show the end-of-test summary. It has been a while since I watched these Gil Tene talks, so I might be missing something, but I thin…
That's perfect, thanks!
Re: K6: Like unit testing, for performance
#77In their documentation ( https://k6.io/docs/ ) they claim that > JavaScript is not generally well suited for high performance. To achieve maximum performance, the tool itself is written in Go, embedding a JavaScript runtime allowing for easy test scripting. How is it possible that pure go JavaScript interpreter (goja) with bindings for net/http and some reports would be faster than the same tool written in nodejs usi…
My guess would be that startup times to load all the JS libraries needed for a full tool would take quite a while. In build scripts, it's important that you can start up each task (k6 instance) very fast.
> would take quite a while
No, and definitely not when comparing with the time it takes to run load tests.
Re: K6: Like unit testing, for performance
#78Re: K6: Like unit testing, for performance
#79Re: K6: Like unit testing, for performance
#80I am one of the k6 developers. I've commented in a bunch of threads here, but if you have any questions, feel free to AMA :) I'll be around for the next ~1h and then probably later in the day as well.
Is there some way you can do whatever it is AWS CDK does under the hood to provide language bindings to other languages for writing tests? That'd be killer. I want to be able to write the tests in my backend language.