Define a backend for the secrets engine

13min
|
Vault
Video

In these tutorials, you will write a custom secrets engine against the authentication API of a fictional coffee-shop application using the HashiCorp Vault Plugin SDK.

You will learn how to create a new secrets backend, build a set of Vault roles, and create workflows to renew and revoke an API token using Vault.

You may build your own secrets engine to:

Manage secrets for an existing internal or proprietary tool
Extend the capabilities of an existing secrets engine (such as scheduled revocation of tokens) with a custom workflow

In this tutorial, you will set up your Vault secrets engine development environment and create a backend to store data, configuration, and secrets.

Step 1 defines a backend at /secrets-engine

To do this, you will:

Set up your development environment.
You will clone the HashiCups secrets engine repository. This contains many of the interfaces and objects you need to create a secrets engine.
Explore the plugin's entry point.
You will examine a file that allows Vault to start the secrets engine.
Explore the secrets engine's backend.
You will examine a scaffold that defines an empty schema and functions to store the secrets engine's information in a Vault backend.
Explore the client to access the demo application.
You will examine the client that encapsulates an API client for the secrets engine to use.
Explore the configuration for the secrets engine
You will examine the configuration attributes for the secrets engine, defined at the config path.
Update the client to access your target API
You will replace the code in the scaffold with a client to access the target API.
Update the backend to use your target API's client
You will replace the code in the scaffold to create a secrets engine backend that can access the target API.

Prerequisites

Golang 1.16+ installed and configured.
Vault 1.8+ CLI installed locally.

Set up your development environment

Clone the learn-vault-plugin-secrets-hashicups repository.

$ git clone https://github.com/hashicorp-education/learn-vault-plugin-secrets-hashicups

Change into the repository directory.

$ cd vault-plugin-secrets-hashicups

Note

If you are stuck in this tutorial, refer to the learn-vault-plugin-secrets-hashicups/solution directory.

Explore the plugin's entry point

Open cmd/vault-plugin-secrets-hashicups/main.go. The contents of the main function use the Vault SDK's plugin library to start the plugin and communicate with the Vault API.

As you examine the file, note the following:

VaultPluginTLSProvider: runs inside a plugin and gets the SSL certificate for Vault, returning a Vault TLS configuration object.
BackendFactoryFunc: provide your secrets engine's backend factory to BackendFactoryFunc. In this case, we pass the factory for the HashiCups backend framework.
hclog: use HashiCorp's logging package to surface errors from the plugin.

cmd/vault-plugin-secrets-hashicups/main.go

1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930package main

import (
    "os"

    "github.com/hashicorp/go-hclog"
    hashicups "github.com/hashicorp/vault-guides/plugins/vault-plugin-secrets-hashicups"
    "github.com/hashicorp/vault/api"
    "github.com/hashicorp/vault/sdk/plugin"
)

func main() {
    apiClientMeta := &api.PluginAPIClientMeta{}
    flags := apiClientMeta.FlagSet()
    flags.Parse(os.Args[1:])

    tlsConfig := apiClientMeta.GetTLSConfig()
    tlsProviderFunc := api.VaultPluginTLSProvider(tlsConfig)

    err := plugin.Serve(&plugin.ServeOpts{
        BackendFactoryFunc: hashicups.Factory,
        TLSProviderFunc:    tlsProviderFunc,
    })
    if err != nil {
        logger := hclog.New(&hclog.LoggerOptions{})

        logger.Error("plugin shutting down", "error", err)
        os.Exit(1)
    }
}

Note

Vault handles custom secrets engines through an external process and gRPC. Verified secrets engines are bundled as part of the Vault binary. Vault handles "builtin" secrets engines in memory.

Explore the secrets engine's backend

Open backend.go. The contents create a backend in Vault for the secrets engine to store data.

Note

The following details involve code that you must include in your own secrets engine. Edit the names of the structs and libraries so they reference your target API.

The first function, Factory, creates a new backend in Vault as logical.Backend. It should include a call to set up the backend.

backend.go

func Factory(ctx context.Context, conf *logical.BackendConfig) (logical.Backend, error) {
    b := backend()
    if err := b.Setup(ctx, conf); err != nil {
        return nil, err
    }
    return b, nil
}

The file contains a backend object for your secrets engine. In this case, you name it hashiCupsBackend. It should have three attributes:

framework.Backend: implements logical.Backend
lock: locking mechanisms for writing or changing secrets engine data
client: stores the client for the target API, HashiCups

backend.go

type hashiCupsBackend struct {
    *framework.Backend
    lock   sync.RWMutex
    client *hashiCupsClient
}

The file creates a method named backend that returns a hashiCupsBackend object. It contains a few important attributes, including:

Help: outputs help text for the backend.
PathsSpecial: sets any special paths for the secrets engine's API. Secrets engines usually reference two main fields, with optional fields typically used for auth methods.
- LocalStorage: For enterprise. These paths store data local to the Vault instance and will not be replicated, more important if you use Vault's Write-Ahead-Log. You would add a path for framework.WALPrefix. You do not need to set it for this secrets engine.
- SealWrapStorage: For enterprise. When using a seal, these paths should be wrapped with extra encryption. These will usually be set to:
  1. config: configuration of the secrets engine
  2. role/*: Vault role for the secrets engine
- Root: Optional. Paths that require a root token for access.
- Unauthenticated: Optional. Paths that can be accessed without authentication. Usually used for for auth methods on the /login endpoint.
Paths: add the secrets engine's API paths.
Secrets: add any secrets types, such as tokens or passwords
BackendType: use logical.TypeLogical for secrets engines
Invalidate: call the function to invalidate the secrets engine's configuration.

backend.go

func backend() *hashiCupsBackend {
    var b = hashiCupsBackend{}

    b.Backend = &framework.Backend{
        Help: strings.TrimSpace(backendHelp),
        PathsSpecial: &logical.Paths{
            LocalStorage: []string{},
            SealWrapStorage: []string{
                "config",
                "role/*",
            },
        },
        Paths:       framework.PathAppend(),
        Secrets:     []*framework.Secret{},
        BackendType: logical.TypeLogical,
        Invalidate:  b.invalidate,
    }
    return &b
}

The file includes two methods extending the backend type. Implement a reset method to lock the backend while you reset the target API client object.

backend.go

func (b *hashiCupsBackend) reset() {
    b.lock.Lock()
    defer b.lock.Unlock()
    b.client = nil
}

You also need to implement the invalidate method referenced in backend. It calls reset to reset the configuration and target API client object.

backend.go

12345func (b *hashiCupsBackend) invalidate(ctx context.Context, key string) {
    if key == "config" {
        b.reset()
    }
}

Examine the getClient method in backend.go. The method should take in the context and Vault storage API interface. You may notice that it does not return the demo application client, as per the function. You will update this method when you implement the target API client.

backend.go

1 2 3 4 5 6 7 8 9 101112131415func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    return nil, fmt.Errorf("need to return client")
}

At the bottom of backend.go, define a string with help text for the backend.

backend.go

const backendHelp = `
The HashiCups secrets backend dynamically generates user tokens.
After mounting this backend, credentials to manage HashiCups user tokens
must be configured with the "config/" endpoints.
`

Explore the client to access the demo application

Open client.go.

Note

The following details involve code that you must include in your own secrets engine. Edit the names of the structs and libraries so they reference your target API.

Import the Go SDK for HashiCups from a library named hashicups-client-go.

Note

When you create a secrets engine, you need to import the Go SDK for your target API. If your secrets engine does not have a Go SDK, you can write your own and import it for use in the secrets engine.

client.go

import (
    hashicups "github.com/hashicorp-demoapp/hashicups-client-go"
)

The file initializes an empty hashiCupsClient object that stores a reference to the client to interface with the HashiCups API.

client.go

1234567type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    return &hashiCupsClient{nil}, nil
}

newClient takes a Vault configuration object as an input, something you will define depending on the configuration for your secrets engine.

client.go

1234567type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    return &hashiCupsClient{nil}, nil
}

Explore the configuration for the secrets engine

Open path_config.go.

Note

The following details involve code that you must include in your own secrets engine. Edit the names of the structs and libraries so they reference your target API.

Define a constant for configStoragePath as config. You use the config path for secrets engines to define attributes like username, password, and API endpoint.

path_config.go

1 2 3 4 5 6 7 8 9 10111213package secretsengine

import (
    "context"
    "fmt"

    "github.com/hashicorp/vault/sdk/framework"
    "github.com/hashicorp/vault/sdk/logical"
)

const (
    configStoragePath = "config"
)

The file defines an object for hashiCupsConfig. You need this object in order to store configuration attributes related to the secrets engine.

path_config.go

type hashiCupsConfig struct {
    Username string `json:"username"`
    Password string `json:"password"`
    URL      string `json:"url"`
}

Note

For your own secrets engine, you can define the configuration with any attributes you need in order for your API's client to authenticate. The attributes can include identifiers for API resources.

Examine the getConfig method. You need to implement this method to pass the context and storage path for Vault to store the hashiCupsConfig into the config path for the secrets engine.

path_config.go

1 2 3 4 5 6 7 8 9 1011121314151617func getConfig(ctx context.Context, s logical.Storage) (*hashiCupsConfig, error) {
    entry, err := s.Get(ctx, configStoragePath)
    if err != nil {
        return nil, err
    }

    if entry == nil {
        return nil, nil
    }

    config := new(hashiCupsConfig)
    if err := entry.DecodeJSON(&config); err != nil {
        return nil, fmt.Errorf("error reading root configuration: %w", err)
    }

    return config, nil
}

Update the client to access your target API

Return to client.go. You must configure the client to access the target API, HashiCups.

Note

Replace the methods and structs in the scaffold with the embedded code examples.

Replace the newClient method in the scaffold. The method now checks that hashiCupsConfig has all of its attributes and has a configuration defined.

client.go

1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233package secretsengine

import (
    hashicups "github.com/hashicorp-demoapp/hashicups-client-go"
)

type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    if config == nil {
        return nil, errors.New("client configuration was nil")
    }

    if config.Username == "" {
        return nil, errors.New("client username was not defined")
    }

    if config.Password == "" {
        return nil, errors.New("client password was not defined")
    }

    if config.URL == "" {
        return nil, errors.New("client URL was not defined")
    }

    c, err := hashicups.NewClient(&config.URL, &config.Username, &config.Password)
    if err != nil {
        return nil, err
    }
    return &hashiCupsClient{c}, nil
}

Note

If your secrets engine has many attributes, you may only need to check for the required ones.

Using the hashiCupsConfig object you defined in path_config.go, initialize a new HashiCups client using the URL, username, and password. Return the HashiCups client in the hashiCupsClient object.

client.go

1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233package secretsengine

import (
    hashicups "github.com/hashicorp-demoapp/hashicups-client-go"
)

type hashiCupsClient struct {
    *hashicups.Client
}

func newClient(config *hashiCupsConfig) (*hashiCupsClient, error) {
    if config == nil {
        return nil, errors.New("client configuration was nil")
    }

    if config.Username == "" {
        return nil, errors.New("client username was not defined")
    }

    if config.Password == "" {
        return nil, errors.New("client password was not defined")
    }

    if config.URL == "" {
        return nil, errors.New("client URL was not defined")
    }

    c, err := hashicups.NewClient(&config.URL, &config.Username, &config.Password)
    if err != nil {
        return nil, err
    }
    return &hashiCupsClient{c}, nil
}

Note

In your own secrets engine, you need to initialize the client based on your target API's Go SDK.

Update the backend to use your target API's client

Return to backend.go. Recall that the getClient method returned an error by default. You need to add your client to this method in order for the secrets engine to access your target API!

Note

Replace the methods and structs in the scaffold with the embedded code examples.

Replace the getClient function in backend.go. You must retrieve your secrets engine's configuration using the getConfig function.

backend.go

1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    config, err := getConfig(ctx, s)
    if err != nil {
        return nil, err
    }

    if config == nil {
        config = new(hashiCupsConfig)
    }

    b.client, err = newClient(config)
    if err != nil {
        return nil, err
    }

    return b.client, nil
}

Vault will keep running the secrets engine and checking for a valid client. If you delete the configuration for the secrets engine or reset it, getClient returns a new hashiCupsConfig.

backend.go

1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    config, err := getConfig(ctx, s)
    if err != nil {
        return nil, err
    }

    if config == nil {
        config = new(hashiCupsConfig)
    }

    b.client, err = newClient(config)
    if err != nil {
        return nil, err
    }

    return b.client, nil
}

Otherwise, Vault will create a new client based on the configuration in the secrets engine. Set the client to the backend object and return it.

backend.go

1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829func (b *hashiCupsBackend) getClient(ctx context.Context, s logical.Storage) (*hashiCupsClient, error) {
    b.lock.RLock()
    unlockFunc := b.lock.RUnlock
    defer func() { unlockFunc() }()

    if b.client != nil {
        return b.client, nil
    }

    b.lock.RUnlock()
    b.lock.Lock()
    unlockFunc = b.lock.Unlock

    config, err := getConfig(ctx, s)
    if err != nil {
        return nil, err
    }

    if config == nil {
        config = new(hashiCupsConfig)
    }

    b.client, err = newClient(config)
    if err != nil {
        return nil, err
    }

    return b.client, nil
}

Next steps

Congratulations! You configured the secrets engine to retrieve attributes from its config path and create a new client to access your target API.

If you are stuck in this tutorial, refer to the plugins/vault-plugin-secrets-hashicups/solution directory.

To learn more about Vault plugins, refer to the Vault Plugin System Documentation.
Define your secrets engine's configuration in the next tutorial.

Collection Overview

Custom secrets engine

Define secrets engine configuration