Core Configuration Management

Core Configuration Management provides a robust and flexible system for managing application settings across various environments and operational contexts. It centralizes configuration parameters required for interactive sessions with the Flyte backend, serialization processes, and the runtime execution of tasks. This system prioritizes ease of use while offering granular control over how settings are defined and loaded.

Core Features

The configuration system is built around a hierarchical structure and supports multiple sources for defining parameters, ensuring adaptability and maintainability.

Centralized Config Object

The Config object serves as the primary container for all application settings. It aggregates various specialized configuration objects, such as PlatformConfig for backend connectivity, SecretsConfig for sensitive data, StatsConfig for metrics, and DataConfig for data storage interactions. This aggregation provides a single, unified interface for accessing all necessary parameters.

Example:

from flytekit.configuration import Config, PlatformConfig, DataConfig

# Accessing default configurations
default_config = Config()
print(f"Default platform endpoint: {default_config.platform.endpoint}")

# Overriding specific parameters programmatically
custom_data_config = DataConfig(s3=S3Config(endpoint="http://my-minio:9000"))
modified_config = default_config.with_params(data_config=custom_data_config)
print(f"Custom S3 endpoint: {modified_config.data_config.s3.endpoint}")

Multiple Configuration Sources and Precedence

Configuration values can originate from several sources, which are processed in a defined order of precedence:

1. Environment Variables: Values set as environment variables take the highest precedence. These are typically formatted as FLYTE_{SECTION}_{OPTION} (e.g., FLYTE_PLATFORM_ENDPOINT).
2. Configuration Files: If environment variables are not found, the system consults configuration files. Both INI-style (.ini, .config) and YAML-style (.yaml, .yml) formats are supported.
3. Default Values: If a parameter is not found in environment variables or configuration files, the system falls back to predefined default values.

This precedence model allows for flexible overrides, from global environment settings to specific file-based configurations, ensuring that the most specific setting is always applied.

Configuration Entry Definitions

Individual configuration parameters are defined using ConfigEntry objects, which encapsulate how a setting is read from different sources.

• LegacyConfigEntry: Used for INI-style configuration files and environment variables. It specifies the section and option within the file, and the expected type_ (e.g., str, int, bool, list). It also defines the corresponding environment variable name.
• YamlConfigEntry: Used for YAML-style configuration files. It uses a dot-delimited switch string (e.g., platform.endpoint) to navigate the YAML structure and specifies the config_value_type.

The ConfigEntry class combines these, allowing a single logical configuration parameter to be defined across different physical storage formats. It also supports optional transform functions for custom type conversions.

ConfigFile Utility

The ConfigFile utility abstracts the parsing logic for different configuration file formats. When initialized with a file path, it automatically detects whether the file is INI-style or YAML-style and loads its contents accordingly. This allows the higher-level Config object to interact with a unified interface for file-based settings.

Common Use Cases

Automatic Configuration Loading

For most applications, the Config.auto() class method provides a convenient way to load configurations. It automatically searches for configuration files in standard locations and applies the precedence rules (environment variables > config files > defaults) to construct a complete Config object.

Example:

from flytekit.configuration import Config

# Automatically load configuration from environment variables or default config files
app_config = Config.auto()
print(f"Loaded platform endpoint: {app_config.platform.endpoint}")

Connecting to a Flyte Backend

When deploying applications that interact with a specific Flyte backend, the Config.for_endpoint() method simplifies the setup. It allows you to specify the backend endpoint and whether the connection should be insecure, while still leveraging Config.auto() for other parameters.

Example:

from flytekit.configuration import Config

# Configure for a production Flyte endpoint
prod_config = Config.for_endpoint(endpoint="my-flyte-cluster.example.com:443")
print(f"Production endpoint: {prod_config.platform.endpoint}")
print(f"Is insecure: {prod_config.platform.insecure}")

# Configure for a local development endpoint with insecure connection
dev_config = Config.for_endpoint(endpoint="localhost:8080", insecure=True)
print(f"Development endpoint: {dev_config.platform.endpoint}")
print(f"Is insecure: {dev_config.platform.insecure}")

Local Sandbox Development

For local development and testing with a Flyte sandbox environment, the Config.for_sandbox() method provides a pre-configured Config object. This method sets up the platform endpoint to localhost:30080 and configures DataConfig for a local Minio instance, streamlining the setup for local sandbox interactions.

Example:

from flytekit.configuration import Config

# Get configuration for a local Flyte sandbox
sandbox_config = Config.for_sandbox()
print(f"Sandbox platform endpoint: {sandbox_config.platform.endpoint}")
print(f"Sandbox S3 endpoint: {sandbox_config.data_config.s3.endpoint}")

Programmatic Overrides

While environment variables and configuration files are excellent for externalizing settings, there are scenarios where programmatic overrides are necessary. The Config.with_params() method allows you to create a new Config instance with specific sub-configurations replaced or updated, without modifying the original object. This is useful for testing or dynamic configuration adjustments within an application.

Example:

from flytekit.configuration import Config, PlatformConfig

base_config = Config.auto()
print(f"Original endpoint: {base_config.platform.endpoint}")

# Create a new config with a different platform endpoint
test_platform = PlatformConfig(endpoint="test-cluster.example.com")
test_config = base_config.with_params(platform=test_platform)
print(f"Test endpoint: {test_config.platform.endpoint}")
print(f"Original config remains unchanged: {base_config.platform.endpoint}")

Important Considerations

• Security: When dealing with sensitive information, always use SecretsConfig and leverage secure environment variable injection or secrets management systems rather than hardcoding values in configuration files.
• Consistency: Maintain consistency in naming conventions for environment variables and configuration file keys to ensure predictable behavior across different deployment stages.
• Debugging: When troubleshooting configuration issues, remember the precedence order: environment variables override file settings, which override defaults. Inspect environment variables first, then the contents of your configuration files.
• Temporary Files: The Config object creates a temporary directory for local sandbox operations (local_sandbox_path). Ensure your environment has appropriate permissions for temporary file creation.