Skip to main content

Flyte CLI Configuration and Core Usage

The Flyte CLI serves as the primary command-line interface for interacting with a Flyte cluster. It enables developers to manage projects and domains, register Flyte entities (workflows, tasks, launch plans), trigger executions, and monitor their progress directly from their local development environment or CI/CD pipelines.

Configuration​

The Flyte CLI requires configuration to connect to a Flyte cluster and authenticate. This configuration can be managed through a hierarchy of sources:

  • Configuration File: The primary method is a config.yaml file, typically located at ~/.flyte/config.yaml. This file defines the target Flyte cluster, authentication details, and default project/domain.

    admin:
      endpoint: dns:///flyte.example.com:80
      authType: Pkce # Or ClientCredentials, None
      insecure: false
    logger:
      level: 0
    console:
      endpoint: http://console.flyte.example.com
    storage:
      type: s3
      connection:
        access-key: YOUR_ACCESS_KEY
        secret-key: YOUR_SECRET_KEY
      container: flyte

    The admin.endpoint specifies the Flyte Admin service URL. authType dictates the authentication mechanism (e.g., Pkce for OAuth2, ClientCredentials for service accounts, or None for unauthenticated access).

  • Environment Variables: Configuration values can be overridden by environment variables prefixed with FLYTE_. For example, FLYTE_ADMIN_ENDPOINT=dns:///another.flyte.example.com:80 overrides the endpoint specified in config.yaml.

  • Command-line Flags: The most granular level of configuration is through command-line flags, which take precedence over both environment variables and the configuration file. For instance, flytectl get project --admin.endpoint dns:///temp.flyte.com:80.

Core Features​

  • Project and Domain Management: The CLI facilitates the creation and management of projects and domains within a Flyte cluster. Projects provide logical isolation for workflows, while domains offer environments (e.g., development, staging, production) within a project.

    # Create a new project
    flytectl create project --name my-new-project --id my-new-project-id --description "My awesome project"

    # Get details of a project
    flytectl get project my-new-project-id

    # Set default project and domain for subsequent commands
    flytectl config init --project my-new-project-id --domain development

  • Package Deployment (Registration): The CLI is essential for registering compiled Flyte entities (tasks, workflows, launch plans) with the Flyte Admin service. This process makes the code executable on the Flyte cluster.

    # Compile Flyte entities (assuming a Python project)
    pyflyte package --output-dir flyte-package

    # Register the compiled package
    flytectl register files --project my-new-project-id --domain development --archive flyte-package/*

    The pyflyte package command generates the necessary protobuf definitions and serialized Python code. flytectl register files then uploads these to the Flyte Admin.

  • Workflow and Task Execution: Developers can trigger executions of registered workflows or tasks directly from the CLI, optionally providing input parameters.

    # Launch a workflow execution
    flytectl create execution --project my-new-project-id --domain development --launchplan my_workflow.my_launch_plan --inputs '{"input_param": "value"}'

    # Relaunch a previous execution
    flytectl relaunch execution <execution-id>

    For complex inputs, it is often better to provide a JSON file using --inputs @inputs.json.

  • Monitoring and Inspection: The CLI provides commands to inspect the status of executions, retrieve logs, and view details of registered entities.

    # List active executions
    flytectl get execution --project my-new-project-id --domain development

    # Get details of a specific execution
    flytectl get execution <execution-id> --project my-new-project-id --domain development

    # Stream logs for an execution
    flytectl get execution <execution-id> --project my-new-project-id --domain development --logs

    # Get details of a registered workflow
    flytectl get workflow my_workflow --project my-new-project-id --domain development

  • Debugging and Troubleshooting: When an execution fails, the CLI helps in diagnosing issues by providing access to execution events, task logs, and error messages. The --logs flag is particularly useful for quickly identifying the source of failures.

Common Use Cases​

  • Local Development and Testing: Rapidly register and execute workflows during development cycles to test changes against a live Flyte cluster.
  • CI/CD Integration: Automate the deployment of Flyte workflows and tasks as part of a continuous integration and deployment pipeline. This ensures that new code versions are automatically registered and available for execution.
  • Ad-hoc Workflow Execution: Manually trigger specific workflows with custom inputs for experimentation, data backfills, or one-off operational tasks.
  • Operational Monitoring: Quickly check the status of running workflows, identify bottlenecks, or retrieve logs for failed executions without needing to access the Flyte UI.
  • Resource Management: Create and manage Flyte projects and domains, ensuring proper isolation and organization of data pipelines.

Limitations and Important Considerations​

  • Authentication: Ensure proper authentication is configured. Misconfigured authentication is a common source of "permission denied" errors.
  • Version Skew: While generally backward compatible, significant version differences between the flytectl CLI and the Flyte Admin service can lead to unexpected behavior. It is best practice to keep them in sync.
  • Large Inputs: For very large input parameters, consider using Flyte's data catalog or external storage solutions rather than passing them directly via the CLI, as command-line limits can be hit.
  • Idempotency: Registering the same workflow or task with the same version multiple times is idempotent; it will update the existing entity. However, be mindful of versioning strategies to avoid unintended overwrites.
  • Network Latency: Operations involving large file uploads (e.g., flytectl register files) can be affected by network latency and bandwidth.