Gates and Manual Intervention

Gates and Manual Intervention provide mechanisms to introduce human interaction or timed pauses into automated workflows. This capability is crucial for scenarios requiring explicit approvals, data input from an operator, or controlled delays before proceeding with subsequent steps.

Core Features

The Gate class serves as a specialized workflow node designed to pause execution until a specific condition is met, which can be either a user action or a predefined time duration. It supports three primary modes of operation:

• Approval Gates: These gates halt workflow execution, awaiting explicit confirmation from a user to proceed. They are ideal for critical decision points where human oversight is mandatory.

  • An approval gate is typically created by passing an upstream item (e.g., the output of a previous task) to the Gate constructor, without specifying an input_type or sleep_duration. The local_execute method will then prompt for confirmation, displaying the value of the upstream item. If the user disapproves, a FlyteDisapprovalException is raised, halting the workflow.
  • Example: Approving a deployment based on the output of a test run.

• Input Gates: These gates pause the workflow to solicit specific data input from a user. The workflow resumes only after the required input, conforming to a specified type, is provided.

  • An input gate is instantiated by providing an input_type to the Gate constructor. The python_interface property will reflect this input type as the gate's output. During local execution, the local_execute method prompts the user to enter data of the specified type, which then becomes the output of the gate node.
  • Example: Requesting a user to provide a reason for a rollback or a configuration parameter.

• Sleep Gates: These gates introduce a controlled delay into the workflow, pausing execution for a specified duration before automatically proceeding. They do not require user interaction.

  • A sleep gate is created by setting the sleep_duration parameter in the Gate constructor. The local_execute method simulates this pause, and the workflow continues automatically after the duration elapses.
  • Example: Waiting for an external system to synchronize or for a scheduled maintenance window to pass.

Common Use Cases

• Human-in-the-Loop Workflows: Integrate human decision-making into automated processes, such as reviewing machine learning model predictions before deployment or validating sensitive data transformations.
• Staged Deployments and Releases: Implement approval gates between different deployment stages (e.g., dev -> staging -> production) to ensure manual verification and sign-off at each critical juncture.
• Data Curation and Validation: Pause a data pipeline to allow data stewards to inspect and approve data quality, correct anomalies, or provide missing metadata.
• Scheduled Maintenance and Synchronization: Use sleep gates to align workflow execution with external system schedules, ensuring that dependent services are ready or that specific time windows are respected.
• Error Handling and Recovery: Introduce an approval gate after an automated recovery attempt, allowing an operator to confirm the resolution or intervene manually if necessary.

Implementation Details and Considerations

When constructing a Gate instance, the name parameter is mandatory for identifying the gate within the workflow.

• Timeout: All gate types can specify a timeout duration. If the required user interaction (for approval or input gates) or the sleep duration is not completed within this timeout, the gate will fail. The default timeout is applied if not explicitly set.
• Local Execution: The local_execute method provides interactive prompts for input and approval gates during local development and testing, simulating the real-world interaction. For sleep gates, it logs the simulated sleep.
• Output Type: For input gates, the input_type directly determines the output type of the gate node, allowing subsequent nodes to consume the user-provided data. Approval and sleep gates typically do not produce a direct output value that is consumed by downstream nodes in the same way, though an approval gate can pass through the approved upstream item.
• Error Handling: Disapproval in an approval gate or failure to provide input for an input gate will raise an exception, stopping the workflow at that point. Developers should consider how to handle these exceptions in their workflow design.