Skip to main content

Map Tasks (Array Nodes)

Map Tasks (Array Nodes) facilitate the parallel or distributed processing of elements within a collection. They enable a single logical task to fan out into multiple independent sub-tasks, each operating on a distinct item from an input array. This pattern is essential for scaling operations that apply the same logic to many data points concurrently.

Core Features​

Map Tasks (Array Nodes) offer robust capabilities for managing distributed array processing:

  • Automatic Sub-task Creation: The system automatically generates a distinct sub-task, referred to as an Array Node, for each element in a specified input array. This eliminates the need for manual task definition for each item.
  • Independent Execution: Each Array Node executes independently, allowing for parallel processing and isolation of failures. A failure in one Array Node does not necessarily halt the entire Map Task, enabling robust processing of large datasets.
  • Result Aggregation: Mechanisms are provided to collect and consolidate the results from all executed Array Nodes into a single, coherent output. This often involves collecting individual Array Node outputs into a list or a structured data type.
  • Granular Error Handling: Individual Array Nodes can report their success or failure status. The Map Task can be configured to continue processing other Array Nodes even if some fail, or to aggregate error information for later analysis.
  • Dynamic Scaling: Map Tasks integrate with underlying execution environments to dynamically scale resources based on the number of Array Nodes requiring processing, optimizing resource utilization.

Common Use Cases​

Map Tasks (Array Nodes) are highly versatile and apply to various scenarios requiring parallel processing:

  • Batch Data Processing: Applying a transformation, validation, or enrichment function to every record in a large dataset. For example, processing millions of log entries, image files, or database records.
  • Parallel Computations: Running simulations or calculations for different parameter sets concurrently. This is common in scientific computing, financial modeling, and machine learning.
  • Hyperparameter Tuning: Evaluating multiple model configurations or hyperparameters in parallel to find the optimal settings for a machine learning model. Each configuration becomes an Array Node.
  • Distributed File Processing: Processing individual files within a directory or a distributed file system concurrently. Each file path can be an element in the input array.
  • API Call Fan-out: Making concurrent API calls to an external service, where each call processes a distinct piece of data from an input list.

Implementation and Integration​

Implementing a Map Task involves defining the input array and the processing logic for each Array Node. The system then orchestrates the execution.

Consider a scenario where a list of data items requires individual processing:

from my_task_framework import MapTask, TaskContext

def process_single_item(item: dict, context: TaskContext) -> dict:
    """
    Processes a single data item within an Array Node.
    This function receives an item from the input array and a TaskContext.
    """
    try:
        # Simulate a CPU-bound or I/O-bound operation
        processed_value = item["value"] * 2
        if item["id"] % 2 == 0:
            processed_value += 10 # Add extra logic for even IDs
        
        context.log_info(f"Array Node {item['id']}: Processed value {item['value']}")
        return {"id": item["id"], "processed_value": processed_value, "status": "success"}
    except Exception as e:
        context.log_error(f"Array Node {item['id']}: Error processing item: {e}")
        return {"id": item["id"], "status": "failed", "error": str(e)}

# Define the input array for the Map Task
input_data_array = [
    {"id": 1, "value": 5},
    {"id": 2, "value": 10},
    {"id": 3, "value": 15},
    {"id": 4, "value": 20},
    {"id": 5, "value": 25},
]

# Instantiate a Map Task.
# The 'map_function' is the core logic applied to each element.
# The 'input_array' provides the data that will be distributed across Array Nodes.
data_processing_task = MapTask(
    name="BatchDataProcessor",
    map_function=process_single_item,
    input_array=input_data_array
)

# Execute the Map Task.
# This initiates the creation and execution of Array Nodes.
# The 'execute' method returns the aggregated results from all Array Nodes.
results = data_processing_task.execute()

print("--- Map Task Execution Results ---")
for result in results:
    print(result)

# Post-processing: Analyze successful and failed Array Nodes
successful_nodes = [r for r in results if r.get("status") == "success"]
failed_nodes = [r for r in results if r.get("status") == "failed"]

print(f"\nSuccessfully processed {len(successful_nodes)} items.")
print(f"Failed to process {len(failed_nodes)} items.")

In this example, MapTask orchestrates the distribution of input_data_array elements to individual ArrayNode instances. The process_single_item function defines the logic executed by each Array Node. The TaskContext provides utilities like logging within the scope of an individual Array Node.

Limitations and Considerations​

While powerful, Map Tasks (Array Nodes) have specific considerations:

  • Overhead: Creating and managing a large number of very small Array Nodes can introduce significant orchestration overhead. For extremely fine-grained operations, consider batching multiple array elements into a single Array Node's processing scope to reduce this overhead.
  • State Management: Each Array Node should ideally be stateless or manage its state independently to avoid race conditions and ensure predictable behavior in a distributed environment. Shared mutable state across Array Nodes is generally discouraged.
  • Resource Contention: While Map Tasks enable parallelism, an excessive number of concurrently executing Array Nodes can exhaust system resources (CPU, memory, network I/O) if not properly managed by the underlying task scheduler.
  • Result Size: Aggregating results from millions of Array Nodes can lead to very large data structures. For such scenarios, consider streaming results, writing them directly to external storage, or using distributed data structures.

Best Practices​

  • Idempotent Processing: Design the Array Node's processing logic to be idempotent. This allows for safe retries of individual Array Nodes in case of transient failures without causing unintended side effects.
  • Robust Error Handling: Implement comprehensive error handling within the map_function to gracefully manage individual failures. Return structured error information from failed Array Nodes to facilitate debugging and recovery.
  • Monitoring and Logging: Implement detailed monitoring for individual Array Nodes to track progress, identify bottlenecks, and diagnose issues. Utilize the TaskContext for structured logging within each Array Node.
  • Input Data Partitioning: For extremely large input arrays, consider pre-partitioning the data into smaller, manageable chunks before feeding them to the Map Task. This can improve load balancing and fault tolerance.
  • Resource Allocation: Configure the underlying execution environment to provide appropriate resources (CPU, memory) for each Array Node based on its expected workload.