Was a big fan of using LoadImpact for years and started testing with K6 as LoadImpact deprecates. (I use them to benchmark WordPress Hosting company performance: https://reviewsignal.com/blog/2020/06/01/wordpress-hosting-p...) I have to say overall the experience was pretty good. I was able to write near identical scripts easily. The documentation was good. Overall pretty happy with K6 as a successor and stand alone. I've used a lot of load testing tools and it's definitely one of the best out there.
K6: Like unit testing, for performance
61–70 of 86 posts
Re: K6: Like unit testing, for performance
#62what is the max number of VUS you guys have run through it?
The most I started on a single machine was about 50.000 VUs. Back then, the limit was in available sockets. k6 now has support for multiple source IPs, so that limit has been effectively removed, and more VUs should be possible. On the `k6 cloud` side, we have executed 500k+ VUs. The most RPS I achieved with k6 was 4 791 928 (~4.8 million requests per second). That test lasted for 6 min and generated 1.5 billion requ…
Interesting. How did you collect metrics for reports when running that many VUS? In a recent attempt, influxdb quickly became the bottleneck for us when testing with around 5000 VUS. Is there some best practices outlined somewhere, or which you could outline here, on how to collect metrics for large runs?
Re: K6: Like unit testing, for performance
#63Earlier quoted context omitted.
The most I started on a single machine was about 50.000 VUs. Back then, the limit was in available sockets. k6 now has support for multiple source IPs, so that limit has been effectively removed, and more VUs should be possible. On the `k6 cloud` side, we have executed 500k+ VUs. The most RPS I achieved with k6 was 4 791 928 (~4.8 million requests per second). That test lasted for 6 min and generated 1.5 billion requ…
Interesting. How did you collect metrics for reports when running that many VUS? In a recent attempt, influxdb quickly became the bottleneck for us when testing with around 5000 VUS. Is there some best practices outlined somewhere, or which you could outline here, on how to collect metrics for large runs?
For that local test, I relied on the k6 CLI output.
Influxdb would most likely not be able to handle this amount of data. We are looking to add a TimescaleDB output to enable metric collection beyond what Influx is capable of. There's a working example here if you want to give it a shot ahead of the official release https://github.com/loadimpact/open-source-load-testing-stack
Local execution with cloud output is also possible (k6 run -o cloud), but that's not free.
To give you some idea of what k6 cloud can do, here are screenshots of a few large scale tests https://imgur.com/a/NtXsc1a
1. 500 000 VU test run with 1.5B requests
2. 12h soak test run with 5k VUs
3. 48h soak test, with 5k VUs
4. 20k VU local test (with -o cloud)
Re: K6: Like unit testing, for performance
#64This thread is finest example of why I like this community so much. Been using this tool for quite some time and I thought I figured out most of the stuff. Then I noticed this thread and saw there's even more useful options and functionalities. Thank you HN.
Re: K6: Like unit testing, for performance
#65Earlier quoted context omitted.
Happy to answer any questions you have regarding the license and why we chose it. Is your concern the copyleftness of the license?
Thanks. I must say, I prefer weak copylefts for my projects (MPLv2 with the incompatibility clause), so my concern isn't copyleft per se, but the virility of it. That said, I do have a couple questions: - Would a non-AGPLv3 open-source or source-available project using K6 be incompatible with AGPLv3 given they might "distribute" CI/CD along with their open-source or source-available code? - If a closed-source project…
I'm not a lawyer, so take that into account when reading my reply :)
> Thanks. I must say, I prefer weak copylefts for my projects (MPLv2 with the incompatibility clause), so my concern isn't copyleft per se, but the virility of it.
I need to read up some more on MPLv2, but from what I've read now it looks like a license that could fit k6 as well.
> - Would a non-AGPLv3 open-source or source-available project using K6 be incompatible with AGPLv3 given they might "distribute" CI/CD along with their open-source or source-available code?
No, using k6 as a tool in CI/CD would not be a violation of AGPLv3 or trigger the virality of the license in regards to the non-AGPLv3 open source or source-available code base. Any k6 test scripts you write would also not need to be licensed under AGPLv3. Gitlab built an integration with k6 which could be seen as a data point in agreement with this view: https://docs.gitlab.com/ee/user/project/merge_requests/load_...
> - If a closed-source project publishes results from K6, are they now in violation of AGPLv3?
If we're talking about publishing results from testing of the closed-source project, then no, that would not be in violation. There's however an undefined gray area around clause 13 "Remote Network Interaction" in AGPLv3 in terms of what is allowed when for example offering a SaaS product based on an AGPLv3 licensed component such as k6. Clause 13 has not been tried in court afaik. It could seem that clause 13 would provide some protection for a business like ours from other companies building commercial solutions on top of k6, but from what I've been told by lawyers we shouldn't count on that, so we're not (anymore).
We have had many internal discussions on this topic, as well as with other companies and individuals in the open source space; whether to go the route of a source-available license, effectively restricting commercialization possibilities of k6 by other companies, or going the open core route and restrict what we release as open source. Everytime we've had this discussion internally we've come back to open source being the right choice for us, given the type of product we build and where we think we can capture value.
We'll look into MPLv2.
Re: K6: Like unit testing, for performance
#66What's the term form "like unit testing, for performance"?
Re: K6: Like unit testing, for performance
#67K6 is licensed AGPL and so anyone who uses it needs to AGPL their code?
Member of team artillery.io here, my 2c on the subject. We chose MPLv2 specifically to address licensing concerns. With MPLv2, you can build on top of Artillery, you can build plugins and extensions for it, and integrate it into your systems without worrying about licensing. It's a well-understood license with very clear boundaries between your code and MPLv2-licensed dependencies (e.g. all Hashicorp tools use MPLv2)…
Agree that it's important to be aware of licensing implications of dependencies.
As I mentioned in reply to another comment, from a first look at MPLv2 it seems like it could be a good fit for k6 as well.
You said: > We chose MPLv2 specifically to address licensing concerns.
Is that something you've encountered while building Artillery? We've not met a lot of pushback from legal teams at companies using k6.
Re: K6: Like unit testing, for performance
#68> k6 is a modern load testing tool, building on Load Impact's years of experience in the load and performance testing industry. It provides a clean, approachable scripting API, local and cloud execution, and flexible configuration.
This sounds more like integration testing than unit testing. Something like Google's micro-benchmarking library [1] is more like unit testing.
Re: K6: Like unit testing, for performance
#69What's the term form "like unit testing, for performance"?
Micro-benchmarking?
Re: K6: Like unit testing, for performance
#70Earlier quoted context omitted.
Thanks. I must say, I prefer weak copylefts for my projects (MPLv2 with the incompatibility clause), so my concern isn't copyleft per se, but the virility of it. That said, I do have a couple questions: - Would a non-AGPLv3 open-source or source-available project using K6 be incompatible with AGPLv3 given they might "distribute" CI/CD along with their open-source or source-available code? - If a closed-source project…
I'm not a lawyer, so take that into account when reading my reply :) > Thanks. I must say, I prefer weak copylefts for my projects (MPLv2 with the incompatibility clause), so my concern isn't copyleft per se, but the virility of it. I need to read up some more on MPLv2, but from what I've read now it looks like a license that could fit k6 as well. > - Would a non-AGPLv3 open-source or source-available project using K…
Thanks a lot for being so thorough with the answers. (IANAL either but) Given the above, I think you'd be fine with AGPLv3.
MPLv2 and other related licenses such as Eclipse Public License v2, Erlang Public License, do have the advantage of being well understood and in some cases auto-approved for use at various enterprises and thus a good midway between MIT / Apache and GPLv3.
That said, you'd be right to lean more-copyleft (going Server Side Public License, for example) if K6 is a key product (and not a complementary product), though Bryan Cantrill thinks you might be better off closing up the source in that case: http://dtrace.org/blogs/bmc/2018/12/14/open-source-confronts...