Skip to main content

Tasks: The Building Blocks of Execution

Tasks are the fundamental building blocks of execution, encapsulating a single, atomic unit of work within the system. They define the executable logic, its inputs and outputs, and critical metadata that governs how the task runs on the platform. Tasks are designed to be reusable, independently executable, and form the core components of larger workflows.

The Task Abstraction​

The base Task class provides the foundational contract for all executable units. It defines the essential properties and behaviors that every task must possess, regardless of its underlying implementation or execution environment.

Key properties of the Task class include:

  • name: A unique identifier for the task instance.
  • task_type: A string that categorizes the task, often indicating the plugin or execution engine responsible for its execution (e.g., python-task, spark, sql).
  • interface: A TypedInterface that formally declares the task's inputs and outputs using Flyte's type system. This ensures type safety and enables static analysis.
  • metadata: An instance of TaskMetadata that specifies runtime behaviors such as caching, retries, and timeouts.
  • security_context: Defines the security configuration for the task's execution.
  • docs: Provides documentation for the task, including short and long descriptions.

The Task class also defines abstract methods that derived classes must implement:

  • dispatch_execute(ctx, input_literal_map): This method serves as the entry point for task execution, translating Flyte's literal-based inputs into a format suitable for the specific task implementation and invoking the core execution logic.
  • pre_execute(user_params): A hook executed before the main task logic, allowing for setup or context modification.
  • execute(**kwargs): The core method where the task's primary logic resides.

For local development and testing, the local_execute method allows tasks to be run directly with native Python values, simulating the remote execution environment.

Tasks can specify their execution environment through methods like get_container, get_k8s_pod, and get_sql. These methods return the necessary definitions (e.g., Docker container image, Kubernetes Pod specification, SQL query) for the Flyte backend to provision and run the task. The get_custom method allows for plugin-specific configuration.

Python-Native Tasks (PythonTask)​

The PythonTask class extends the base Task to provide a seamless way to define tasks using native Python functions. It is the most common way developers interact with tasks.

PythonTask introduces:

  • python_interface: A Python-native Interface that maps directly to the type hints of a Python function, simplifying input and output definition for Python developers.
  • task_config: An optional configuration object specific to the Python task's plugin or custom logic.
  • environment: A dictionary of environment variables to be set during task execution.
  • enable_deck / deck_fields: Controls the generation of Flyte Decks, which provide rich, interactive visualizations and summaries of task execution, including input, output, and custom content.

The PythonTask implements the abstract methods from the base Task class:

  • pre_execute(user_params): Allows modification of execution parameters before input conversion.
  • execute(**kwargs): This is where the decorated Python function's logic is invoked, receiving Python-native inputs.
  • post_execute(user_params, rval): A hook executed after the execute method, allowing for cleanup or modification of the task's return value before it is converted back to Flyte literals.

The dispatch_execute method in PythonTask orchestrates the entire execution flow: it calls pre_execute, converts LiteralMap inputs to Python native types, invokes execute with these native inputs, calls post_execute on the results, and finally converts the Python native outputs back to a LiteralMap. This method also handles caching logic and deck generation.

Task Metadata (TaskMetadata)​

TaskMetadata is a crucial component that defines the operational characteristics and runtime behavior of a task. It allows developers to fine-tune how tasks are executed, cached, and managed by the Flyte platform.

Key attributes of TaskMetadata include:

  • cache (bool): Enables or disables output caching for the task. When enabled, if a task is invoked with identical inputs, Flyte retrieves previously computed results instead of re-executing the task.
  • cache_version (str): A version string for the cache. Changing this version invalidates previous cached results, forcing a re-execution.
  • cache_ignore_input_vars (Tuple[str, ...]): Specifies input variables that should be excluded when calculating the cache key. This is useful for inputs that do not affect the task's output but might change frequently (e.g., timestamps).
  • cache_serialize (bool): When True and caching is enabled, identical concurrent executions of the task will be serialized, meaning only one instance runs, and others wait for its cached result.
  • retries (int): The number of times Flyte should retry the task if it fails. A value of 0 means no retries.
  • timeout (Optional[Union[datetime.timedelta, int]]): The maximum duration for a single execution of the task. If the task runs longer, it is terminated.
  • interruptible (Optional[bool]): Indicates if the task can be scheduled on preemptible or spot instances, potentially reducing costs but risking interruption.
  • deprecated (str): A message to mark the task as deprecated, providing guidance to users.
  • pod_template_name (Optional[str]): The name of an existing Kubernetes PodTemplate to use for the task's execution, allowing for advanced Kubernetes resource configuration.
  • generates_deck (bool): Indicates whether the task is expected to produce a Flyte Deck.
  • is_eager (bool): Marks the task for eager execution, potentially affecting scheduling.

Task Definition and Serialization (IDL Models)​

For remote execution on the Flyte platform, tasks are serialized into a language-agnostic Intermediate Definition Language (IDL). This ensures interoperability and consistent execution across different SDKs and backends.

  • TaskTemplate: This is the core IDL representation of a task. It encapsulates the task's id, type, metadata, interface, and the specific execution environment (e.g., container, k8s_pod, sql, or custom plugin data). When a Python task is registered, its Python-native definition is translated into a TaskTemplate.
  • TaskSpec: This combines a TaskTemplate with Documentation, forming the complete specification used for registering a task with the Flyte Admin service.
  • CompiledTask: A wrapper around TaskTemplate, used during the compilation of workflows.
  • TaskClosure: Encapsulates a CompiledTask, providing a self-contained definition of the task for storage and execution.
  • Task (IDL Model): The top-level IDL object representing a registered task, containing its id and closure.

Common Use Cases and Best Practices​

Tasks are versatile and form the backbone of any data or ML pipeline.

  • Data Processing Steps: Each step in an ETL (Extract, Transform, Load) pipeline can be a task. For example, a task to read data from S3, another to clean and transform it, and a final task to load it into a data warehouse.
  • Machine Learning Model Training: A task can encapsulate the entire model training process, from loading data to fitting a model and saving artifacts.
  • Inference and Prediction: Tasks can be used for real-time or batch inference, taking input data and producing predictions.
  • Custom Integrations: Tasks can integrate with external systems or specialized compute environments (e.g., Spark jobs, SQL queries, AWS SageMaker training jobs) by defining custom task_type and custom configurations.
  • Reusable Components: Define common operations as tasks (e.g., send_email_notification, upload_to_s3) to promote code reuse and maintainability across workflows.

Best Practices:

  • Clear Interfaces: Always define explicit and well-typed inputs and outputs for your tasks. This improves readability, enables static analysis, and prevents runtime errors.
  • Leverage Caching: For computationally expensive or frequently executed tasks with stable inputs, enable caching (metadata.cache=True) to significantly reduce execution time and resource consumption. Remember to manage cache_version effectively.
  • Configure Retries and Timeouts: Use metadata.retries and metadata.timeout to make your tasks robust against transient failures and prevent runaway executions.
  • Modular Design: Keep tasks focused on a single responsibility. Complex logic should be broken down into smaller, composable tasks.
  • Local Testing: Utilize the local_execute capability to thoroughly test your tasks in a local environment before deploying to the Flyte platform.
  • Documentation: Provide clear docs for your tasks to explain their purpose, inputs, and outputs, making them easier for others to understand and use.
  • Deck Generation: Use enable_deck and deck_fields to generate rich visualizations for task execution, aiding in debugging and understanding task behavior.