Vault usage metrics

Client calculation and sizing can be complex to compute when measuring Vault Enterprise usage. Vault provides telemetry metrics that can be monitored with solutions such as Prometheus and Grafana or Splunk; however, it requires additional configuration and tools.

Vault v1.6 introduced Vault Metrics dashboard in Web UI. This enhancements help to find active client count and historical client counts within Vault.

Policy requirements

To follow the steps in this tutorial, the following policies are required.

# To retrieve the usage metrics
path "sys/internal/counters/activity" {
  capabilities = ["read"]
}

# To read and update the usage metrics configuration
path "sys/internal/counters/config" {
  capabilities = ["read", "update"]
}

#---------------------------
# Vault Enterprise only
#---------------------------

# To view existing namespaces if any
path "sys/namespaces" {
  capabilities = ["list", "read", "update"]
}

# UI to show the namespace selector
path "sys/internal/ui/namespaces" {
  capabilities = ["read", "list", "update", "sudo"]
}

# UI to list existing mounts
path "sys/internal/ui/mounts" {
  capabilities = ["read", "sudo"]
}

# To read and update the usage metrics configuration for any namespace
path "+/sys/internal/counters/config" {
  capabilities = ["read", "update"]
}

# To retrieve the usage metrics for any namespace
path "+/sys/internal/counters/activity" {
  capabilities = ["read"]
}

Enable usage metrics

Usage metrics are a feature that is enabled by default for Vault Enterprise but disabled for Vault Community Edition.

To enable usage tracking for Vault Community Edition, use the sys/internal/counters/config endpoint. (This step is NOT necessary for Vault Enterprise.)

$ vault write sys/internal/counters/config enabled=enable

$ vault write sys/internal/counters/config enabled=enable retention_months=12

$ curl --header "X-Vault-Token: <TOKEN>" \
    --request POST \
    --data '{"enabled": "enable"}' \
    https://127.0.0.1:8200/v1/sys/internal/counters/config

$ curl --header "X-Vault-Token: <TOKEN>" \
    --request POST \
    --data '{"enabled": "enable", "retention_months": 12}' \
    https://127.0.0.1:8200/v1/sys/internal/counters/config

Usage metrics dashboard

Sign into the Vault UI, and select Client Count.

The default view for usage metrics is for the current billing period.

Part of what contributes to Vault pricing is client usage. The Current month and History tabs display three client usage metrics: Total clients, Entity clients, and Non-entity clients. Here is an example of the metrics shown for the history.

The data are grouped into 3 distinct categories:

• Running client total are the primary metric on which pricing is based. It is the sum of entity clients (or distinct entities) and non-entity clients.

• Entity clients (distinct entities) are representations of a particular user, client, or application that belongs to a defined Vault entity. If you are unfamiliar with Vault entities, refer to the Identity: Entities and Groups tutorial.

• Non-entity clients are clients without an entity attached. This is because some customers or workflows might avoid using entity-creating authentication methods and instead depend on token creation through the Vault API. Refer to understanding non-entity tokens to learn more.

  Note

  The non-entity client count excludes root tokens. Refer to the FAQs documentation which describes the difference between the entity clients and non-entity clients.

• Secrets sync clients are the number of external destinations Vault connects to sync the secrets. Refer to the documentation for more details.

• ACME clients are the ACME connections that authenticate under the same certificate identifier (domain) as the same certificate entity for client count calculations. Refer to the documentation for more details.

Vault Enterprise Namespaces

If you have namespaces, the entity clients and non-entity clients are also shown as graphs per namespace. The Attribution section also displays the top namespace where you can expect to find your most used namespaces with respect to client usage (Vault 1.9 or later). This allows you to detect which namespace had the most number of total clients in the given billing period.

You can export the metrics data by clicking on the Export attribution data button.

This downloads the usage metrics data on your local drive in comma separated values format (.csv) containing all client count usage metrics by namespace.

Auth mount attribution

The clients are also shown as graphs per auth mount. The Attribution section also displays the top auth mounts where you can expect to find your most used auth mounts with respect to client usage (Vault 1.10 or later). This allows you to detect which auth mount had the most number of total clients in the given billing period. You can filter for auth mounts within a namespace, or view auth mounts across namespaces. The auth mount attribution functionality can also be used if you are not using namespaces at all. You can export the metrics data by clicking on the Export attribution data button.

Querying usage metrics

Under the Filters, select the time period, and select custom end month. This brings up a dialog where you can specify the month and year to retrieve data for a different billing period.

It is important to keep in mind that Vault begins collecting data when the feature is enabled. For example, if the feature is enabled in March of 2021, a query cannot be made in the UI for data in January of 2021.

Vault will return metrics from March of 2020 through most recent full month. To specify a different ending month, select the down-arrow next to the current billing period.

You can select Current billing period or Custom end month to edit the period.

Filter by namespaces (Vault Enterprise)

Vault Enterprise namespaces provide multi-tenancy to isolate Vault usage. You may want to view the client usage metrics per namespace.

Under Filters, you can enter the target namespace in the text field next to the billing period.

When you put focus on the text field, it displays existing namespaces.

Querying usage metrics via CLI or API

You can query the usage count metrics using Vault CLI or API.

$ vault read -format=json  sys/internal/counters/activity | jq -r ".data"

{
  "by_namespace": [
    {
      "counts": {
        "clients": 2666,
        "distinct_entities": 1564,
        "entity_clients": 1564,
        "non_entity_clients": 1102,
        "non_entity_tokens": 1102
      },
      "mounts": [],
      "namespace_id": "ai0nK",
      "namespace_path": "education/certification/"
    },
    {
      "counts": {
        "clients": 2565,
        "distinct_entities": 1297,
        "entity_clients": 1297,
        "non_entity_clients": 1268,
        "non_entity_tokens": 1268
      },
      "mounts": [],
      "namespace_id": "QbHPY",
      "namespace_path": "education/training/security-edu/"
    },
    {
      "counts": {
        "clients": 2192,
        "distinct_entities": 1257,
        "entity_clients": 1257,
        "non_entity_clients": 935,
        "non_entity_tokens": 935
      },
      "mounts": [],
      "namespace_id": "RFdWg",
      "namespace_path": "education/training/security-edu/vault-edu/"
    },

    ...snip...

  ],
  "end_time": "2022-02-28T23:59:59Z",
  "months": [],
  "start_time": "2021-03-01T00:00:00Z",
  "total": {
    "clients": 13346,
    "distinct_entities": 6675,
    "entity_clients": 6675,
    "non_entity_clients": 6671,
    "non_entity_tokens": 6671
  }
}

$ vault read -format=json sys/internal/counters/activity \
     start_time=2021-09-01T00:00:00Z \
     end_time=2021-09-30T23:59:59Z  | jq -r ".data"

$ vault read -format=json sys/internal/counters/activity/monthly  | jq -r ".data"

{
   "by_namespace":[
      {
        "namespace_id":"0lHBL",
        "namespace_path":"finance/",
        "counts":{
           "distinct_entities":0,
           "non_entity_tokens":100,
           "clients":100
        }
      },
      {
        "namespace_id":"RxD81",
        "namespace_path":"marketing/",
        "counts":{
          "distinct_entities":5,
          "non_entity_tokens":15,
          "clients":20
        }
      },
      {
        "namespace_id":"root",
        "namespace_path":"",
        "counts":{
          "distinct_entities":85,
          "non_entity_tokens":15,
          "clients":100
        }
      }
    ],
    "distinct_entities": 90,
    "non_entity_tokens": 130,
    "clients": 220
}

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    $VAULT_ADDR/v1/sys/internal/counters/activity | jq -r ".data"

{
  "by_namespace": [
    {
      "namespace_id": "ai0nK",
      "namespace_path": "education/certification/",
      "counts": {
        "distinct_entities": 1564,
        "entity_clients": 1564,
        "non_entity_tokens": 1102,
        "non_entity_clients": 1102,
        "clients": 2666
      },
      "mounts": []
    },
    {
      "namespace_id": "QbHPY",
      "namespace_path": "education/training/security-edu/",
      "counts": {
        "distinct_entities": 1297,
        "entity_clients": 1297,
        "non_entity_tokens": 1268,
        "non_entity_clients": 1268,
        "clients": 2565
      },
      "mounts": []
    },
    {
      "namespace_id": "RFdWg",
      "namespace_path": "education/training/security-edu/vault-edu/",
      "counts": {
        "distinct_entities": 1257,
        "entity_clients": 1257,
        "non_entity_tokens": 935,
        "non_entity_clients": 935,
        "clients": 2192
      },
      "mounts": []
    },

    ...snip...

  ],
  "end_time": "2022-02-28T23:59:59Z",
  "months": [],
  "start_time": "2021-03-01T00:00:00Z",
  "total": {
    "distinct_entities": 6675,
    "entity_clients": 6675,
    "non_entity_tokens": 6671,
    "non_entity_clients": 6671,
    "clients": 13346
  }
}

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    "$VAULT_ADDR/v1/sys/internal/counters/activity?start_time=2021-09-01T00:00:00Z&end_time=2021-09-30T23:59:59Z" \
    | jq -r ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    $VAULT_ADDR/v1/sys/internal/counters/activity/monthly \
    | jq -r ".data"

{
   "by_namespace":[
      {
        "namespace_id":"0lHBL",
        "namespace_path":"finance/",
        "counts":{
           "distinct_entities":0,
           "non_entity_tokens":100,
           "clients":100
        }
      },
      {
        "namespace_id":"RxD81",
        "namespace_path":"marketing/",
        "counts":{
          "distinct_entities":5,
          "non_entity_tokens":15,
          "clients":20
        }
      },
      {
        "namespace_id":"root",
        "namespace_path":"",
        "counts":{
          "distinct_entities":85,
          "non_entity_tokens":15,
          "clients":100
        }
      }
    ],
    "distinct_entities": 90,
    "non_entity_tokens": 130,
    "clients": 220
}

Considerations for querying usage metrics

• Data is collected at the end of each month. In order to display the data for a month in particular, the entire month has to pass.

• In the heading, the UI will show the data that has been returned. For example, if you searched data from January 2021 but data is only available from March 2021, the heading will reflect that.

• When running Vault 1.10.x or earlier and if there is missing data somewhere in your search (e.g. data is available from January 2020 through October 2020, but April is missing), Vault will return metrics from May through October 2020, because that is the most recent contiguous set. It does not mean the other data is unavailable.

Export activity log using API

Vault's activity log is an internal system for tracking monthly client usage containing entity IDs or synthetic client IDs from non-entity tokens. Each client ID is only stored once per month. Previously, the activity log was only used internally to compute the count of clients.

As of Vault 1.11.0, the Activity Export API allows users to download the activity log based on the user-defined start time and end time.

A single client appears once in the output, and API uses the earliest activity in the time range for the client.

Example usage

Create your request payload specifying the start time and end time. Use RFC3339 timestamp or Unix epoch time to specify the start and end time. You can also specify the output format. Currently available format is CSV or JSON (csv or json).

The following request payload specifies the start time to January 1, 2022 and the end time to be June 1, 2022. The output should be in CSV format.

$ tee payload.json <<EOF
{
   "start_time": "1641024000",
   "end_time": "1654066800",
   "format": "csv"
}
EOF

Invoke the /sys/internal/counters/activity/export endpoint and store the output in a file, /tmp/activity-export.csv.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    $VAULT_ADDR/v1/sys/internal/counters/activity/export > /tmp/activity-export.csv

Refer to the Activity Export API documentation as well.

Version-specific client count metrics details

You should be aware that there are significant nuances in how client count metrics are computed between versions of Vault at or beyond version 1.9.0, and versions between 1.6.0 and 1.9.0.

Important details on these nuances are noted by version.

For additional details about how clients are computed, refer to What is a Client?.

Vault auditor tool

If you are running at least Vault version 1.3, you can use the vault-auditor tool to extract metrics from the server audit device logs. These metrics are typically useful for metering and billing, but also helpful for getting critical entity and token counts for troubleshooting scenarios.

Download vault-auditor binaries for macOS and Linux, and you can use the tool wherever you have access to the audit device log files.

Here is an example interaction with vault-auditor, beginning with help output and concluding with an example where the tool is used to parse a hypothetical directory of audit devices logs.

Get help for vault-auditor with the --help flag.

$ vault-auditor --help
Usage: vault-auditor [--version] [--help] <command> [<args>]

Available commands are:
    parse      Parse audit logs to find entity and token counts
    version    Prints the vault-auditor version

Currently, the tool features just two sub-commands, parse and version; check the help for the parse sub-command to learn more about how it works.

$ vault-auditor parse --help
Usage: vault-auditor parse [options] <directory>

  Parse one or more audit logs inside the given directory
  to count unique tokens.

      $ vault-auditor parse /tmp/audit-logs

Options:

  -prefix=<string>
      String that every line in the audit logs is prefixed with.
      This should only be necessary if the audit prefix contains a { character.

  -start-date=<string>
      Only count log lines on/after this date.
      RFC3339, YYYY-MM-DD, YYYY-MM, and Unix epoch time formats are supported.

  -end-date=<string>
      Only count log lines on/before this date.
      RFC3339, YYYY-MM-DD, YYYY-MM, and Unix epoch time formats are supported.

Note in the help output, that vault-auditor expects to be executed against a directory containing Vault audit device log files. It supports three optional flags for handling prefixed log entries and controlling the start and end ranges to operate on.

The most basic use of vault-auditor is to execute it and pass in the full path to a directory containing one or more Vault audit device log files. In this example, the logs are contained in an audit-device-logs directory

$ vault-auditor parse audit-logs

The parsing of logs will emit progress indicators and when successful, the output resembles this example.

vault-zus1-0  [=======================================]  100 %  0s
vault-zus1-1  [=======================================]  100 %  0s
vault-zus1-2  [=======================================]  100 %  0s
vault-zus1-3  [=======================================]  100 %  0s
vault-zus1-4  [=======================================]  100 %  0s


Distinct Entities: 7139
Non-Entity Tokens: 273
Total Clients: 7412
Total files processed: 5
Date range: 2020-10-02T08:36:29Z - 2020-10-07T15:57:50Z

There were a total of 5 logs processed from the audit-logs directory, with a total of 273 non-entity tokens identified, meaning tokens which were created directly from the Token auth method (such as root tokens). The tool also found 7139 distinct entities for which tokens were issued through a Vault auth method. The total clients count is a combination of distinct entities and non-entity tokens.

Lastly, the date range represented by the parsed logs is shown.

Limits of the auditor tool

The vault auditor tool was released along with Vault v1.6.0.

Since that time, some improvements were made to the Vault client count computation logic in newer versions of Vault. These changes are reflected only in the usage metrics API and web UI, but are not available in the auditor tool.

The following specific improvements are not reflected in the auditor output.

• Vault v1.8

  • Eliminated wrapped tokens and control groups from client count thereby reducing the non-entity token count. Prior to 1.8, the creation and usage of control groups and wrapping tokens incremented the client count via non-entity tokens every time the response was read in the case of a wrapping token, and every time a control group was created.
  • Changed logic of counting of non entity tokens on usage instead of at create time therefore resulting in more accurate client counts.

• Vault 1.9

  • Changed the non-entity token computation logic to de-duplicate non-entity tokens as described in the Version Specific Client Count Metrics Details section.
  • Changed the local auth mount computation logic such that local auth mounts count towards clients but not as non-entity tokens. Prior to v1.9, local auth mounts counted towards non-entity tokens.
  • New functionality to display clients per namespace (top 10) in the UI and the ability to export data for all namespaces. Prior to 1.9, there was no ability to view the split of clients per namespace on the UI, and there was no ability to export this data via the UI.

KMIP client metrics

To perform the tasks described in this section, you need to have a Vault Enterprise v1.7 or later with Advanced Data Protection module.

Vault Enterprise 1.7 introduced improvements to the audit device metrics to help users understand usage and scalability of the KMIP secrets engine for different workflows.

When you parse the Vault Enterprise audit logs using the vault-auditor tool, pass the -kmip-info flag to get the metrics specific to the KMIP secrets engine. The output will include the number of KMIP certificates, active KMIP certificates, and managed objects.

$ vault-auditor parse -kmip-info audit_log

Distinct Entities: 14
Non-Entity Tokens: 1
Total Clients: 15
Total files processed: 1
Date range: 2021-03-20T20:11:41Z - 2021-03-23T22:56:17Z
Distinct KMIP Certificates: 24
Distinct Active KMIP Certificates: 20
Distinct KMIP Managed Objects: 20

From the example output, you can observe that there are 24 distinct KMIP certificates issued by Vault, with 20 of those active during the date range of the audit device log, along with a total of 20 KMIP managed objects.

Recent changes

Vault 1.17

When you are running PKI secrets engine to issue and renew leaf server certificate with Automatic Certificate Management Environment (ACME) protocol, usage metrics dashboard reports the ACME clients separately for license utilization reporting.

Vault 1.16

Usage metric dashboard tracks the secrets sync client counts. When you use Vault's secret sync feature to automatically update secrets in external destination, Vault considers the destination as a secret sync client. Refer to the documentation for more details.

Vault 1.11

Vault 1.11 introduced Activity Export API allowing you to download the client count aggregated for the user-defined billing period. Refer to the Export activity log using API section for more detail.

Vault 1.10

Starting Vault 1.10, usage metrics dashboard refers to the non-entity tokens as non-entity clients, and unique entities as entity clients.

Previously, if there is missing data within your billing period (e.g. data is available from January 2021 through October 2021, but April is missing), Vault returned metrics from May through October 2021, because that is the most recent contiguous set. As of Vault 1.11.0, Vault returns all available data within the specified billing period.

Vault 1.9

Starting in Vault 1.9, Vault changed the non-entity token computation logic to deduplicate non-entity tokens. For non-entity tokens (where there is no entity to which tokens map) Vault uses the contents of the token to generate a unique client identifier, based on the namespace ID and policies. The clientID will prevent the same token from being duplicated in the overall client count. Non-entity token tracking is done on access instead of creation.

Help and reference

If you are running Vault v1.9 or later, refer to the version-specific client count metrics details section to learn about key differences in the computation of client usage metrics between versions. For Vault v1.3.x through v1.5.x, refer to the Vault auditor tool section on how to use the vault-auditor tool to extract the usage metrics.

In addition, refer to the following documentation:

• Client count documentation
• Client count FAQ
• What is a Client?
• Client Count API doc
• Vault v1.9 release notes