K6

Summary

Over the years, HTTP driven benchmark testing has seen many contenders come and go. Tools in this space include siege, apache bench, bees with machine guns, and many more. These sorts of tests are not only great for load testing, but they also can be used to help find bottlenecks in web application stacks, stress monitoring and observability tools, and can be used for chaos and reliabiity testing.

Recently, one contender from a prominent observability company has risen above the rest to become one of the defacto load testing tools. Grafana’s K6 is an open source, extensible load testing tool. Find it on github if you’d like to browse the source. The tests are driven by javascript scripts and the k6 testing tool.

Installation

If you’d like to follow along with this article, first install the k6 binary locally, or pull the container. You can find the official installation docs upstream at https://k6.io/docs/getting-started/installation/

MacOS: brew install k6

Debian based:

sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6

RHEL based Linux:

sudo dnf install https://dl.k6.io/rpm/repo.rpm
sudo dnf install k6

Docker:

docker pull grafana/k6

Getting started

k6 works by loading javascript source containing an exported default function, then it runs that function. Let’s take a look at the given example for a single request. I’ve saved this locally as main.js and pointed it to a domain that I own. Using this file now, we can run a basic test by running:

k6 run main.js

Here’s what it looks like in action:

k6 run main.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: main.js
     output: -

  scenarios: (100.00%) 1 scenario, 20 max VUs, 3m30s max duration (incl. graceful stop):
           * default: Up to 20 looping VUs for 3m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)


running (3m00.8s), 00/20 VUs, 2035 complete and 0 interrupted iterations
default ✓ [======================================] 00/20 VUs  3m0s

     ✓ status is 200
     ✗ response body
      ↳  0% — ✓ 0 / ✗ 2035

     checks.........................: 50.00% ✓ 2035      ✗ 2035
     data_received..................: 54 MB  300 kB/s
     data_sent......................: 176 kB 971 B/s
     http_req_blocked...............: avg=1.18ms   min=0s      med=1µs     max=413.8ms  p(90)=1µs     p(95)=1µs     
     http_req_connecting............: avg=417.53µs min=0s      med=0s      max=54.12ms  p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=65.05ms  min=40.38ms med=61.58ms max=273.04ms p(90)=81.42ms p(95)=90.97ms 
       { expected_response:true }...: avg=65.05ms  min=40.38ms med=61.58ms max=273.04ms p(90)=81.42ms p(95)=90.97ms 
     http_req_failed................: 0.00%  ✓ 0         ✗ 2035
     http_req_receiving.............: avg=18.69ms  min=93µs    med=17.74ms max=189.26ms p(90)=30.99ms p(95)=43.45ms 
     http_req_sending...............: avg=175.78µs min=49µs    med=175µs   max=1.34ms   p(90)=246µs   p(95)=261.29µs
     http_req_tls_handshaking.......: avg=724.1µs  min=0s      med=0s      max=285.94ms p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=46.18ms  min=32.38ms med=41.85ms max=247.12ms p(90)=55.63ms p(95)=61.45ms 
   ✗ http_reqs......................: 2035   11.252972/s
     iteration_duration.............: avg=1.06s    min=1.04s   med=1.06s   max=1.5s     p(90)=1.08s   p(95)=1.09s   
     iterations.....................: 2035   11.252972/s
     vus............................: 1      min=1       max=20
     vus_max........................: 20     min=20      max=20

ERRO[0181] some thresholds have failed 

Let’s break down the parts of the single-request example to see what we can extrapolate from it:

Metrics

k6 comes with a few built in metrics types. In the single-request example, we can see the Counter metric in action:

export const requests = new Counter('http_reqs');

This requests object is built from the Counter object comes from the k6 library when we import it (import { Counter } from 'k6/metrics';). This library also gives us Gauge, Rate, and Trend. See the upstream documentation for this part of the k6 api here:

https://k6.io/docs/javascript-api/#k6-metrics

Test Options

The next portion of the single-request example showcases the test options. You can find an extensive guide to these options in the upstream documentation. Let’s take a look:

export const options = {
  stages: [
    { target: 20, duration: '1m' },
    { target: 15, duration: '1m' },
    { target: 0, duration: '1m' },
  ],
  thresholds: {
    http_reqs: ['count < 100'],
  },
};

First, we see an array of stages. A first stage starts with 20 virtual users (referred to as VUs) which runs for 1 minute, followed by a second stage of running 15 virtual users making requests for 1 minute, followed by a cooldown for 1 minute. To learn more about stages, see:

https://k6.io/docs/using-k6/k6-options/reference#stages

Next, we see a map of thresholds. Per the upstream documentation:

Thresholds are the pass/fail criteria that you define for your test metrics. If the performance of the system under test (SUT) does not meet the conditions of your threshold, the test will finish with a failed status.

In our single-request example, we set the http_reqs threshold, which is to say, we count the total http requests and only pass if its under 100. Kind of an odd threshold, but ok. We see this in the resulting report once the test runs:

   ✓ http_reqs......................: 42      11.853558/s

As we see, since the total count of http_reqs was less than 100, we pass the test. Ideally, your test probably checks that you were able to make more than x number of requests instead of less, but I digress.

There are a plethora of thresholds available for k6, so see more at the upstream documentation for thresholds here:

https://k6.io/docs/using-k6/thresholds/

The business

Finally, let’s analyze the function that defines the behavior of our test:

export default function () {
  // our HTTP request, note that we are saving the response to res, which can be accessed later

  const res = http.get('http://test.k6.io');


  sleep(1);

  const checkRes = check(res, {
    'status is 200': (r) => r.status === 200,
    'response body': (r) => r.body.indexOf('Feel free to browse') !== -1,
  });
}

This test makes it’s request to test.k6.io, then sleeps for 1 second, then checks the status and response body of the request. If the response code is 200, and the body contains the string Feel free to browse, then we have a successful request!

Review

So now that we hopefully understand the test breakdown, we should be able to shape this test into something meaningful for testing a single endpoint’s throughput. In the following example, we test that running 10 VUs for 1 minute with a 200ms sleep time should produce 10 * (60/.002) = 3000 total requests. In my case, I’m testing this blog, so these configurations are spefic to my usecase.

Here’s my test:

import http from 'k6/http';
import { sleep, check } from 'k6';
import { Counter } from 'k6/metrics';

export const requests = new Counter('http_reqs');

export const options = {
  stages: [
    { target: 10, duration: '1m' },
  ],
  thresholds: {
    http_reqs: ['count >= 3000'],
    http_req_duration: ['p(95) < 200'], // 95% of requests should be below 200ms
  },
};

export default function () {
  const res = http.get('https://hartje.io');

  sleep(.001);

  const checkRes = check(res, {
    'status is 200': (r) => r.status === 200,
    'response body': (r) => r.body.indexOf('posts') !== -1,
  });
}

And here are my results:

k6 run main.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: main.js
     output: -

  scenarios: (100.00%) 1 scenario, 10 max VUs, 1m30s max duration (incl. graceful stop):
           * default: Up to 10 looping VUs for 1m0s over 1 stages (gracefulRampDown: 30s, gracefulStop: 30s)


running (1m00.1s), 00/10 VUs, 6281 complete and 0 interrupted iterations
default ✓ [======================================] 00/10 VUs  1m0s

     ✓ status is 200
     ✓ response body

     checks.........................: 100.00% ✓ 12562      ✗ 0   
     data_received..................: 167 MB  2.8 MB/s
     data_sent......................: 512 kB  8.5 kB/s
     http_req_blocked...............: avg=130.89µs min=0s      med=1µs     max=104.85ms p(90)=1µs     p(95)=1µs    
     http_req_connecting............: avg=54.19µs  min=0s      med=0s      max=43ms     p(90)=0s      p(95)=0s     
   ✓ http_req_duration..............: avg=46.16ms  min=39.37ms med=44.73ms max=217.27ms p(90)=50.39ms p(95)=53.76ms
       { expected_response:true }...: avg=46.16ms  min=39.37ms med=44.73ms max=217.27ms p(90)=50.39ms p(95)=53.76ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 6281
     http_req_receiving.............: avg=6.62ms   min=84µs    med=6.87ms  max=174.64ms p(90)=9.4ms   p(95)=10.36ms
     http_req_sending...............: avg=88.94µs  min=30µs    med=73µs    max=3.41ms   p(90)=145µs   p(95)=189µs  
     http_req_tls_handshaking.......: avg=75.55µs  min=0s      med=0s      max=64.57ms  p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=39.45ms  min=32.39ms med=38.2ms  max=208.97ms p(90)=44.67ms p(95)=47.86ms
   ✓ http_reqs......................: 6281    104.587542/s
     iteration_duration.............: avg=47.79ms  min=40.77ms med=46.24ms max=218.61ms p(90)=51.95ms p(95)=55.47ms
     iterations.....................: 6281    104.587542/s
     vus............................: 9       min=1        max=9 
     vus_max........................: 10      min=10       max=10

Examples

Post random data to URL

import http from 'k6/http';
import { sleep, check } from 'k6';
import { Counter } from 'k6/metrics';
import { randomItem } from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';

export const requests = new Counter('http_reqs');

export const options = {
  stages: [
    { target: 1, duration: '1m' },
  ],
  thresholds: {
    http_reqs: ['count >= 3000'],
  },
};

export default function () {
  const payloads = [
    "foo",
    "bar",
    "baz",
    "qux"
  ]

  let data = {
    data: randomItem(payloads),
  }

  let res = http.post("http://address.tld:12345", JSON.stringify(data), {
    headers: { 'Content-Type': 'application/json' },
    timeout: '120s'
  });
  sleep(5);
  
  const checkRes = check(res, {
    'status is 200': (r) => r.status === 200,
  });
}

Reference

Thoughts on implementation

These days with the proliferation of CNCF tooling, higher throughputs for testing can be easily achieved by spreading these client tests across several gigabit internet connections by leveraging something like DigitalOcean’s Managed Kubernetes offering.

For this reason, it can be no surprise that k6 has already considered this and has created a test operator that makes this easy. You can learn more at:

https://k6.io/blog/running-distributed-tests-on-k8s/

By creating a configmap and CRD for the test, its easy to deploy your test cases to kubernetes and run at cloud scale and speed.

That’s all folks

I hope you’ve enjoyed this short sumary of k6. Good luck in your own testing adventures, and if you have any questions, drop them in the disqus forum below!

• ←Previous
• Next→