Configuration & Environment

Configuration & Environment

Configuration & Environment provides a robust and secure mechanism for managing and retrieving sensitive configuration values, often referred to as secrets, at runtime. It ensures that credentials, API keys, and other confidential data are accessed consistently across different environments while abstracting away the underlying storage details.

Secret Resolution Mechanism

The system resolves secrets by checking multiple sources in a defined order, prioritizing environment variables for flexibility and security.

Environment Variable Resolution

The system first attempts to retrieve a secret from an environment variable. The expected environment variable name is constructed by concatenating a configurable prefix, the secret group, an optional group version, and the secret key, all in uppercase and separated by underscores.

For example, if the environment variable prefix is MYAPP_SECRETS_, a secret for group="database", key="password", and group_version="v1" would be sought in the environment variable MYAPP_SECRETS_DATABASE_V1_PASSWORD.

During local execution contexts, the system also checks for environment variables without the configured prefix, allowing for simpler local development setups.

File System Resolution

If a secret is not found in environment variables, the system then attempts to retrieve it from a file on the local file system. The expected file path is constructed from a configurable base directory, the secret group, an optional group version, and a file prefix concatenated with the secret key, all in lowercase.

For example, if the base directory is /etc/secrets, the file prefix is secret_, a secret for group="database", key="username", and group_version="v1" would be sought in the file /etc/secrets/database/v1/secret_username.

Resolution Order and Precedence

Secrets are resolved in the following order:

1. Environment variables (with prefix).
2. Environment variables (without prefix, only in local execution contexts).
3. File system.

Environment variables always take precedence over file-based secrets. If a secret is not found through any of these methods, a ValueError is raised.

Using SecretsManager

The SecretsManager class is the primary interface for interacting with the secret resolution system.

Initialization

Initialize SecretsManager with an optional SecretsConfig object to customize the default base directory, file prefix, and environment variable prefix. If no SecretsConfig is provided, SecretsManager.auto() is used to load default configuration.

from your_module import SecretsManager, SecretsConfig

# Using default configuration
sm = SecretsManager()

# Using custom configuration
custom_config = SecretsConfig(
    default_dir="/var/run/my_app/secrets",
    file_prefix="app_secret_",
    env_prefix="APP_CONF_"
)
custom_sm = SecretsManager(secrets_cfg=custom_config)

Retrieving Secrets

Use the get() method to retrieve a secret by its group and key. An optional group_version can be provided for more granular organization. The encode_mode parameter specifies how file-based secrets are read; use "r" for text and "rb" for binary data.

# Retrieve a database password
try:
    db_password = sm.get(group="database", key="password")
    print(f"Database Password: {db_password}")
except ValueError as e:
    print(f"Error retrieving database password: {e}")

# Retrieve an API key with a specific version
try:
    api_key_v2 = sm.get(group="api_service", key="auth_token", group_version="v2")
    print(f"API Key (v2): {api_key_v2}")
except ValueError as e:
    print(f"Error retrieving API key v2: {e}")

# Retrieve a binary certificate
try:
    certificate_data = sm.get(group="certs", key="tls_cert", encode_mode="rb")
    print(f"Certificate data (first 20 bytes): {certificate_data[:20]}")
except ValueError as e:
    print(f"Error retrieving certificate: {e}")

Attribute-Style Access

For convenience, secrets can also be accessed using attribute notation, which is particularly useful when dealing with a specific group of secrets. This method internally calls the get() method.

# Accessing a secret using attribute style
try:
    # This is equivalent to sm.get(group="api_service", key="api_key")
    api_key = sm.api_service.api_key
    print(f"API Service API Key: {api_key}")

    # Accessing a secret with a group version
    # This is equivalent to sm.get(group="database", group_version="prod", key="connection_string")
    prod_db_conn = sm.database.prod.connection_string
    print(f"Production DB Connection: {prod_db_conn}")
except ValueError as e:
    print(f"Error retrieving secret via attribute access: {e}")

Configuring Secrets

To make secrets available to the system, configure them either as environment variables or as files on the file system.

Environment Variables

Set an environment variable following the convention ENV_PREFIX_GROUP_GROUPVERSION_KEY (all uppercase).

# Example: Setting a database password
export MYAPP_SECRETS_DATABASE_PASSWORD="my_secure_db_password"

# Example: Setting an API key for a specific version
export MYAPP_SECRETS_API_SERVICE_V2_AUTH_TOKEN="some_long_api_token_v2"

File System

Create a file at the path BASE_DIR/group/group_version/FILE_PREFIXkey (all lowercase).

# Example: Creating a database username file
# Assuming BASE_DIR is /etc/secrets and FILE_PREFIX is secret_
mkdir -p /etc/secrets/database
echo "db_user" > /etc/secrets/database/secret_username

# Example: Creating a production database connection string file
mkdir -p /etc/secrets/database/prod
echo "host=prod.db.example.com;port=5432;..." > /etc/secrets/database/prod/secret_connection_string

Common Use Cases

• Database Credentials: Securely storing and retrieving sensitive database connection details like usernames, passwords, and connection strings.
• API Keys and Tokens: Managing access keys and tokens for external services, cloud APIs, and third-party integrations.
• Environment-Specific Configuration: Providing different values for development, staging, and production environments without modifying application code.
• Sensitive Application Settings: Any configuration value that should not be hardcoded or committed to source control, such as encryption keys or private certificates.

Best Practices and Considerations

• Granularity: Organize secrets into logical groups (e.g., database, api_service, aws) and use group_version for different versions or environments within a group.
• Security: Ensure that secret files on the file system have restrictive permissions to prevent unauthorized access. For example, chmod 400 for the secret files and chmod 700 for the directories.
• Version Control: Never commit actual secret values to your source code repository. Use environment variables or secure file systems managed by deployment tools.
• Local Development: Leverage the local execution override by setting environment variables without the configured prefix for easier testing and development.
• Binary Secrets: When dealing with non-textual secrets like certificates or keys, use encode_mode="rb" with the get() method to ensure correct binary reading.
• Error Handling: Always wrap secret retrieval calls in try-except ValueError blocks to gracefully handle cases where a secret might be missing.