Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

K6: Like unit testing, for performance

github.com

Post reply on HN
Print view View original

21–30 of 86 posts

Re: K6: Like unit testing, for performance

#21

Post by na-- »

Ugohcet wrote: post #16

In their documentation ( https://k6.io/docs/ ) they claim that > JavaScript is not generally well suited for high performance. To achieve maximum performance, the tool itself is written in Go, embedding a JavaScript runtime allowing for easy test scripting. How is it possible that pure go JavaScript interpreter (goja) with bindings for net/http and some reports would be faster than the same tool written in nodejs usi…

I am not completely sure why the Go stdlib's HTTP client (which k6 uses) is faster than the NodeJS one. I think part of it is the fact that k6 spins up a separate JS runtime for each VU. goja is a much, _much_, slower JS interpreter than V8, but load tests are IO-bound, so that's usually not an issue. And you can spin up thousands of VUs in k6 (especially with --compatibility-mode=base), making full use of the load generator machine.

You can find some basic performance and other comparisons between load testing tools in this very long article of ours: https://k6.io/blog/comparing-best-open-source-load-testing-t...

And some advice for squeezing the maximum performance out of k6 in here: https://k6.io/docs/testing-guides/running-large-tests

Re: K6: Like unit testing, for performance

#22

Post by layla5alive »

slavik81 wrote: post #8

As a C++ developer who writes a lot of unit tests and cares a lot about performance, the tag line sounds appealing, but I can't tell whether this is something meant for me. Is this a general tool, or is it specific to web development? e.g. Would this be an appropriate tool for Blender to detect performance regressions in their rendering?

We (Insightful) are developing a tool for you! :) It's called Insightful PerfLab - it will work for native code (CPU and/or GPU) and is designed to help things like Blender detect performance regressions in rendering (we built it to help detect performance regressions in our game engine, and then realized it would likely be generally useful to other developers).

Re: K6: Like unit testing, for performance

#23

Post by na-- »

Ragib_Zaman wrote: post #10
slavik81 wrote: post #8

As a C++ developer who writes a lot of unit tests and cares a lot about performance, the tag line sounds appealing, but I can't tell whether this is something meant for me. Is this a general tool, or is it specific to web development? e.g. Would this be an appropriate tool for Blender to detect performance regressions in their rendering?

I think it's specific to web development, in particular it tests for performance under user load.

Yes, that's it. If the C++ application exposes some service, then k6 might be the tool to test how it behaves under load now (or in the future). The only protocols we currently support are HTTP(S), WebSockets and gRPC, though we plan to add a lot of others (raw TCP/UDP, various DBs, queues, messaging, DNS, etc.) in the future, and we've recently added a way to extend the k6 functionality by writing external Go modules: - https://github.com/loadimpact/k6/blob/master/release%20notes... - https://github.com/k6io/xk6

Re: K6: Like unit testing, for performance

#24

Post by na-- »

wiradikusuma wrote: post #7

I've used K6 as well. I like it bcoz it's code, but my (major) complaints are lack of auto-complete (intellisense) in IDE and debugging. Without those two, it's hard to do exploratory testing.

There is auto-completion, see https://k6.io/docs/misc/intellisense

Regarding debugging, that's unfortunately unlikely to come any time soon... We'd need support for that in the JS runtime we're using (https://godoc.org/github.com/dop251/goja) and then we'd need to figure out how to expose in in k6, while dealing with potentially hundreds of concurrent JS runtimes (VUs). Not impossible, just unlikely to land anytime soon...

Re: K6: Like unit testing, for performance

#25

Post by na-- »

multani wrote: post #13

I also discovered K6 a few days ago, it looked quite nice and could be a good replacement for our Gatling tests. We were initially looking for something slightly different though: we were interested to have perhaps less tests, but tests that would run much much more often (like every seconds or couple of seconds), in a continuous manner. Tue goal was to have something at the same time like a healtcheck (is it still w…

I might be missing something, but k6 should be able to completely cover all of your use cases? I am one of the k6 developers, can you share exactly what the missing piece was?

> tests that would run much much more often (like every seconds or couple of seconds), in a continuous manner.

You can do that, just use an arrival-rate executor that runs an iteration every second, with a test duration of 365 days or something like that :) See https://k6.io/docs/using-k6/scenarios/arrival-rate

> Tue goal was to have something at the same time like a healtcheck (is it still working), like a performance test (does it answer in a timely manner) and like a validation test (does it answer the right result - the endpoints we wanted to test do "complex calculations"). Our best answer so far was to wrap K6 in an infinite loop, but I wonder if there could be something smarter.

You can certainly wrap k6 in an infinite loop. Nothing wrong with that, though you can probably use the `scenarios` feature (with long `duration` values) to achieve it without wrapping k6: https://k6.io/docs/using-k6/scenarios

Re: K6: Like unit testing, for performance

#26

Post by na-- »

llarsson wrote: post #19
ignoramous wrote: post #14

K6 is licensed AGPL and so anyone who uses it needs to AGPL their code?

Define "uses". If you just run it in your own CI/CD pipeline: no. If you want to offer this as a SaaS: yes. If you want to extend this somehow: yes.

That's more or less it. Source: I am one of the k6 developers, and here's also an answer from our CEO on the topic: https://community.k6.io/t/k6-licensing-agplv3/395/2

Re: K6: Like unit testing, for performance

#27

Post by spockz »

What is the benefit of this tool over vegeta or wrk2? I do like that vegeta and wrk2 work with simple http requests.

If more complex flows and assertions on responses are required I go for Gatling. So maybe K6 is more comparable to gatling.

Re: K6: Like unit testing, for performance

#28

Post by ignoramous »

na-- wrote: post #26
llarsson wrote: post #19

Earlier quoted context omitted.

Define "uses". If you just run it in your own CI/CD pipeline: no. If you want to offer this as a SaaS: yes. If you want to extend this somehow: yes.

That's more or less it. Source: I am one of the k6 developers, and here's also an answer from our CEO on the topic: https://community.k6.io/t/k6-licensing-agplv3/395/2

The CEOs response doesn't inspire any confidence:

If you statically or dynamically link any part of the k6 codebase with some other code (a derivative work), the license’s copyleft virality is triggered and that other code also needs to be made available under AGPLv3. When it comes to using the k6 binary and interacting with it from another process or over a network it is not certain exactly how the copyleft virality would apply from what I’ve been told by lawyers with OSS license knowledge.

Re: K6: Like unit testing, for performance

#29

Post by na-- »

skybrian wrote: post #6

It seems like the hard part is deciding when a change is suspicious enough to be investigated? Simple pass/fail tends not to work at scale because complicated systems tend to be noisy.

That's one of the main uses, yes. Although k6 gives you a lot of flexibility to decide what your pass/fail criteria are with its custom metrics ( https://k6.io/docs/using-k6/metrics#custom-metrics ), checks (https://k6.io/docs/using-k6/checks) and thresholds ( https://k6.io/docs/using-k6/thresholds ).

See the example in https://github.com/loadimpact/k6/#checks-and-thresholds. If, at the end of the test run, some of these rules (encoded as `thresholds`) were unsatisfied, k6 will exit with a non-zero exit code (and thus fail your CI check): - The 95th percentile of all HTTP request durations should be be less than 500ms - The 99th percentile of all HTTP request durations that were tagged with `staticAsset:yes` should be less than 250ms - The failure rate of the checks should be less than 1%, although if it's more than 5% the test will abort immediately.

Re: K6: Like unit testing, for performance

#30

Post by rlonn »

spockz wrote: post #27

What is the benefit of this tool over vegeta or wrk2? I do like that vegeta and wrk2 work with simple http requests. If more complex flows and assertions on responses are required I go for Gatling. So maybe K6 is more comparable to gatling.

I wrote a blog article comparing the most popular open source tools out there - https://k6.io/blog/comparing-best-open-source-load-testing-t...

Disclaimer: I'm one of the people who created k6

Warning: The article is pretty long...

Post reply on HN