Skip to main content

Local Execution and Development Settings

Local Execution and Development Settings provide a robust mechanism for configuring the behavior of the system within a local development environment. These settings allow developers to customize aspects like caching, SDK resource discovery, and local file system interactions, ensuring a tailored and efficient development workflow without impacting deployed environments.

Core Features and Configuration​

The system manages local execution and development settings through distinct configuration sections, primarily local and sdk. These sections define various parameters that control how the system operates during development.

Local Caching Control​

The system manages local caching behavior through two primary settings: cache_enabled and cache_overwrite. These settings are defined within the local configuration section.

  • cache_enabled: A boolean flag that determines whether local caching is active. When True, the system stores and retrieves cached data locally, which can significantly speed up repeated operations.
  • cache_overwrite: A boolean flag that, when True, forces the system to regenerate and overwrite existing cached data, even if valid cached data is present. This is useful for ensuring the latest data is always used during development or for debugging caching issues.

These settings are accessible via the LocalConfig class, which provides an auto method to load configurations from a file.

Example Configuration:

local:
  cache_enabled: true
  cache_overwrite: false

Programmatic Access:

from your_module import LocalConfig # Assuming LocalConfig is in 'your_module'

# Automatically loads settings from the default configuration file
local_config = LocalConfig.auto()

if local_config.cache_enabled:
    print("Local caching is enabled.")
if local_config.cache_overwrite:
    print("Local cache overwrite is enabled.")

SDK Workflow Package Discovery​

The workflow_packages setting specifies a comma-delimited list of Python packages. SDK tools use this list to discover entities (e.g., workflows, tasks, models) for registration and execution purposes. This allows developers to organize their entities into specific packages and ensure the SDK can locate them during local development. This setting is part of the sdk configuration section.

Example Configuration:

sdk:
  workflow_packages: my_project.workflows,my_project.tasks,shared_library

Local Sandbox Management​

The local_sandbox setting defines the file system path where SDK tools place temporary files and artifacts during local executions and testing. This provides a dedicated, isolated environment for local development outputs. The SDK does not automatically clean up data within this directory, requiring manual management by the developer. This setting is part of the sdk configuration section.

Example Configuration:

sdk:
  local_sandbox: /tmp/my_sdk_sandbox

Runtime Logging Configuration​

The logging_level setting controls the default logging verbosity for the Python logging library. This integer value is set before user code runs, allowing developers to adjust the level of detail in logs during local execution. This is a runtime setting, meaning changes take effect immediately without requiring a recompile. This setting is part of the sdk configuration section.

Example Configuration:

sdk:
  logging_level: 20 # Corresponds to logging.INFO

Common logging levels include:

  • 10: DEBUG
  • 20: INFO
  • 30: WARNING
  • 40: ERROR
  • 50: CRITICAL

Common Use Cases​

  • Debugging and Development Iteration:
    • Temporarily disable cache_enabled and enable cache_overwrite to ensure that every execution uses fresh data, bypassing any cached results. This is crucial when debugging data processing logic or when underlying data sources change frequently.
    • Set logging_level to 10 (DEBUG) to obtain highly verbose output, aiding in tracing execution flow and identifying issues within custom code or SDK components.
  • Local Workflow Testing:
    • Configure workflow_packages to point to the specific Python packages containing your development workflows and tasks. This allows the local SDK to discover and execute these entities without needing to deploy them.
    • Direct local_sandbox to a project-specific temporary directory (e.g., ./.local_data) to keep all local execution artifacts organized and separate from other system files.
  • Performance Tuning and Resource Management:
    • Enable cache_enabled during performance testing of local workflows to simulate production-like caching behavior and measure the impact of caching on execution times.
    • Use local_sandbox to manage disk space by periodically clearing its contents, especially when dealing with large output files from local runs.

Important Considerations​

  • Configuration Source: Settings are typically loaded from a configuration file. The LocalConfig.auto() method demonstrates how these values are read and applied. Ensure your configuration files are correctly structured and accessible.
  • Sandbox Cleanup: The local_sandbox directory is not automatically cleaned up. Developers are responsible for managing its contents to prevent excessive disk usage.
  • Runtime vs. Compile-time: Most settings are applied at the start of an execution. The logging_level is a notable runtime setting, allowing dynamic adjustment of log verbosity.