Benchmark Vault performance
Operating Vault in an efficient manner to support your use cases requires that you are able to accurately measure its performance. Ideally, you can benchmark and measure performance in environments which resemble production use cases to produce realistic results.
Challenge
You need to measure Vault performance in a meaningful way, and in an environment that also resembles that of your intended use case with respect to compute resources.
The Vault server under test should have the same auth methods and secrets engines enabled, along with example secrets, leases, and token data present to accurately simulate your use cases.
Solution
HashiCorp provides the open source utility vault-benchmark to help you measure Vault performance at a granular level using several of the available auth methods and secrets engines.
You can use vault-benchmark as a command line interface, as a Docker image, or as a Kubernetes workload to match the infrastructure you're using for Vault.
Personas
The end-to-end scenario and hands-on lab described in this tutorial involves one persona.
The persona is a Vault operator with privileged permissions to enable and disable auth methods and secrets engines. All the tasks in the hands-on scenario are performed as this persona.
Prerequisites
You need the following resources to complete the hands-on scenario based on whether you'll use the Vault community edition with CLI, Docker, Kubernetes, or HCP Vault Dedicated.
To complete the hands-on lab using CLI versions of Vault and vault-benchmark, you need the following:
Vault binary installed and in your system PATH.
vault-benchmark binary installed and in your system PATH.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
Versions used for this tutorial
This tutorial was last tested 11 Oct 2023 on macOS using the following software versions.
Lab setup
The setup for this tutorial's hands-on lab is different depending on whether you want to experiment with vault-benchmark in the CLI with a dev mode server, in the CLI with an Vault Dedicated server, in Docker, or in Kubernetes.
Create local hands-on lab home
You can create a temporary directory to hold all the content needed for this hands-on lab and then assign its path to an environment variable for later reference.
Open a terminal, and create the directory
/tmp/learn-vault-pgp
.Export the hands-on lab directory path as the value to the
HC_LEARN_LAB
environment variable.
Now choose the lab set up workflow that matches the environment you want to use for this hands-on lab.
Run a Vault dev mode server as a background process from your terminal session to follow the self-hosted Vault workflow in this hands-on lab.
Open a terminal and start a Vault dev server with
root
as the initial root token value.The Vault dev server defaults to running at
127.0.0.1:8200
. The server logs to the filevault-server.log
in the hands-on lab working directory, and gets automatically initialized and unsealed.Dev mode is not for production
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and all contents are lost when the Vault server process is stopped.
Export the environment variable for the
vault
CLI to address the Vault server.Export an environment variable for the
vault
CLI to authenticate with the Vault server.Check Vault status.
Example output:
The Vault server is ready for you to proceed with the hands-on lab.
Now that you've established a working lab, you're ready to explore vault-benchmark and its configuration.
Explore vault-benchmark
Your goal for this section is to explore vault-benchmark using a terminal session. Resources for diving deeper are provided at the section's conclusion.
Check your vault-benchmark version.
Example output:
You can get help to discover available commands.
Example output:
This hands-on lab focuses on your use of the
run
command. Get help for therun
command.
Configure benchmark
Your goal for this section is to configure vault-benchmark for a basic benchmark run.
Vault Benchmark is configured with a HashiCorp Configuration Language (HCL) file. You can use the example configuration shown in the Usage documentation in this hands-on lab.
Here is the entire example configuration file.
1 2 3 4 5 6 7 8 9 1011121314151617181920212223
Lines 1-5 are global parameters:
vault_addr
: the full URL plus port for the Vault server to benchmark.vault_token
: the literal token value for a token with capabilities to enable and manage secrets engines and auth methods. In this hands-on lab, the initial root token value is used, but you should not use a root token in production this way.vault_namespace
: the name of the Enterprise namespace to use for the benchmark.duration
: number of seconds to run the benchmark.cleanup
: whether to remove all resources created by the benchmark run.
Benchmark in production not recommended
You should benchmark Vault servers dedicated to the purpose, but not production servers. The benchmark can adversely affect performance of the server, and does not make guarantees about cleanup of artifacts created during the benchmark run.
Lines 7-15 and 17-23 represent the 2 tests making up this benchmark configuration. This test configuration runs two different tests, an approle_auth test and a kvv2_write test, with the percentage of requests split evenly between the two.
The first test beginning at line 7, is for AppRole auth method logins. Note that the first parameter is weight. You can think of this as a percentage of the entire workload. This means that the benchmark performs AppRole logins 50% of the time throughout the entire run.
You can configure a test in its config stanza. In this test, role_name specifies the auth method role name benchmark-role. Each token Vault issues after an AppRole login with the benchmark-role is configured with a token_ttl value of 2 minutes to specify the token's time-to-live (TTL). You can learn more about available parameters in the AppRole auth method benchmark documentation.
The second test beginning at line 17, is for Key/Value version 2 secrets engine writes. This test takes the remaining 50% of benchmark operations with its weight setting. It also uses numkvs to specify that 100 key/value secrets be written, and kvsize to limit each value to 100 bytes. Available parameters are documented in the KV v1 and KV v2 secret benchmark documentation
You can learn more about all the tests in the vault-benchmark test documentation.
Now that you've reviewed the example configuration, you need to change it a bit to function with the Vault cluster you're using.
In the lab setup section, you set the VAULT_ADDR
and VAULT_TOKEN
(and VAULT_NAMESPACE
for Vault Dedicated) environment variables for communicating to your Vault cluster.
Values from environment variables override anything specified in the vault-benchmark configuration file.
With this in mind, you can set the vault_addr
, vault_token
, and vault_namespace
values to an empty string. Their values come from the environment variables you set.
root namespace by default
If you're using a Vault cluster that is not in HCP, you might recall that you did not export a VAULT_NAMESPACE
environment variable during your lab setup.
vault-benchmark uses the root
namespace by default when there is no value set in the configuration file. Specifying it is unnecessary unless you're using HCP Vault Dedicated or a Vault Enterprise cluster with a namespace that is not root
.
These steps apply equally to self-hosted, Docker and Vault Dedicated.
Return to your terminal session and write the configuration file to
/tmp/learn-vault-benchmark/vault-benchmark-config.hcl
.
You've configured vault-benchmark for 2 tests, and they're now ready for you to run them.
Run benchmark
Your goal for this section is to run vault-benchmark with the configuration that you just created.
Run the benchmark.
Example output:
Review the output
The benchmark output is tabular by default. This example is from the self-hosted dev mode server, but the output will be similar for Vault in any environment.
12345678
The first 4 lines are log outputs from vault-benchmark which describe its current actions.
The 5th line shows the benchmark target URL, and the 6th line represents headings for the metric data.
The data metrics are as follows for the last 2 lines, which represent each of the 2 benchmark tests you configured.
op: the test name.
count: the number of tests completed during the benchmark duration.
rate: the true operations per second rate of all tests of this type.
throughput: the operations per second rate of all successful tests of this type.
mean: the mean time in milliseconds per operation.
95th%: the 95th percentile time in milliseconds per operation.
99th%: the 99th percentile time in milliseconds per operation.
successRatio: the percentage of successful tests of this type. When tests are not successful, you should consult the Vault operational and audit logs for details on the unsuccessful tests.
Tip
You can also output the benchmark results as JSON with the -report_mode=json
flag.
Cleanup
Follow the steps for the environment you used in the hands-on lab to clean up.
Stop the Vault dev mode server.
Remove the hands on lab directory.
Unset environment variables.
Remove cached Vault token.
Next steps
You learned the basics around the Vault Benchmark tool, including how to configure and run a benchmark. You also learned about the default benchmark output, and available documentation resources for Vault Benchmark.
To dive deeper into Vault performance, consider reviewing the performance tuning and production hardening documentation.