Skip to main content

Secrets Management

Secrets Management provides a robust and flexible mechanism for applications to access sensitive information securely. It centralizes the configuration for how secrets are discovered and retrieved, abstracting the underlying storage details. This approach prevents hardcoding sensitive data directly into application code or committing it to version control, enhancing security and maintainability.

Core Capabilities​

The system's core capability is to define and load configuration parameters that dictate where an application should look for secrets. This includes:

  • Environment Variable-Based Secrets: Configure a specific prefix for environment variables that contain secrets. Applications can then read these environment variables at runtime.
  • File-Based Secrets: Define a default directory where secrets are stored as individual files. An optional file prefix can be specified to organize these files.
  • Flexible Configuration Loading: Automatically load these configuration settings from environment variables or a dedicated configuration file, allowing for dynamic adjustments without code changes.

Configuration​

The SecretsConfig object encapsulates all parameters necessary for locating secrets.

Environment Variable Prefix​

The env_prefix setting specifies the prefix used to identify environment variables containing secrets. For example, if env_prefix is _FSEC_, a secret named DATABASE_PASSWORD would be expected in an environment variable like _FSEC_DATABASE_PASSWORD.

The default value for env_prefix is _FSEC_. This can be overridden by setting the FLYTE_SECRETS_ENV_PREFIX environment variable or a corresponding entry in a configuration file.

Default Secrets Directory and File Prefix​

For secrets stored as files, the default_dir setting specifies the base directory where these files reside. Each secret is expected to be in its own file within this directory.

The file_prefix setting provides an optional prefix for these secret filenames. For instance, if default_dir is /etc/secrets and file_prefix is app_, a secret named API_KEY would be expected in the file /etc/secrets/app_API_KEY.

The default default_dir is /etc/secrets. This can be overridden using the FLYTE_SECRETS_DEFAULT_DIR environment variable or a configuration file. The file_prefix can be overridden via FLYTE_SECRETS_FILE_PREFIX.

Automatic Configuration Loading​

The SecretsConfig.auto() class method provides a convenient way to initialize the secrets configuration. It automatically reads the env_prefix, default_dir, and file_prefix settings from either system environment variables or a specified configuration file.

import os
# Assuming SecretsConfig is available in your application's secrets module
from your_app_secrets_module import SecretsConfig 

# Load configuration automatically
# This will check environment variables (e.g., FLYTE_SECRETS_ENV_PREFIX)
# and then a default config file if specified.
secrets_config = SecretsConfig.auto()

print(f"Environment variable prefix: {secrets_config.env_prefix}")
print(f"Default secrets directory: {secrets_config.default_dir}")
print(f"File prefix for secrets: {secrets_config.file_prefix}")

# To load from a specific configuration file (requires a ConfigFile utility):
# from your_app_config_module import ConfigFile 
# my_config_file = ConfigFile("/path/to/my_secrets_config.yaml")
# custom_secrets_config = SecretsConfig.auto(config_file=my_config_file)

This method ensures that the application's secret discovery mechanism is configured consistently across different deployment environments.

Accessing Secrets​

Once the SecretsConfig object is initialized, it provides the parameters for where to locate secrets. The application code then uses standard operating system functions to retrieve the actual secret values based on these configurations.

For example, to retrieve a database password:

import os
# Assuming SecretsConfig is available in your application's secrets module
from your_app_secrets_module import SecretsConfig

secrets_config = SecretsConfig.auto()

# Example for an environment variable secret
# Assuming a secret named 'DATABASE_PASSWORD' is set as _FSEC_DATABASE_PASSWORD
db_password_env = os.getenv(f"{secrets_config.env_prefix}DATABASE_PASSWORD")
if db_password_env:
    print(f"Database password from environment: {db_password_env}")
else:
    print("Database password environment variable not found.")

# Example for a file-based secret
# Assuming a secret named 'API_KEY' is stored in /etc/secrets/API_KEY
api_key_path = os.path.join(secrets_config.default_dir, f"{secrets_config.file_prefix}API_KEY")
try:
    with open(api_key_path, "r") as f:
        api_key_file = f.read().strip()
    print(f"API Key from file: {api_key_file}")
except FileNotFoundError:
    print(f"API Key file not found at {api_key_path}")
except Exception as e:
    print(f"Error reading API Key file: {e}")

Common Use Cases​

  • Database Credentials: Securely provide database usernames, passwords, and connection strings to applications without embedding them in code or configuration files.
  • API Keys and Tokens: Manage access tokens for third-party services (e.g., cloud APIs, payment gateways, external data sources).
  • Sensitive Configuration: Store any application-specific configuration that requires restricted access, such as encryption keys or private certificates.
  • Multi-Environment Deployments: Easily switch between development, staging, and production secrets by adjusting environment variables or configuration files without modifying application code.

Best Practices​

  • Never Hardcode Secrets: Always use a secrets management system to inject sensitive data at runtime.
  • Least Privilege: Ensure that applications only have access to the secrets they absolutely need.
  • Rotate Secrets Regularly: Periodically change secrets to minimize the impact of potential compromises.
  • Audit Access: Implement logging and auditing to track who accessed which secrets and when.
  • Encrypt Secrets at Rest and In Transit: While this system focuses on discovery, ensure that the underlying secret storage (e.g., Kubernetes Secrets, AWS Secrets Manager, Vault) encrypts secrets appropriately.
  • Avoid Logging Secrets: Ensure that secrets are never logged or exposed in application output.