Skip to main content

Comet ML Experiment Tracking

Comet ML Experiment Tracking

Comet ML Experiment Tracking provides a robust platform for machine learning developers to track, compare, and optimize their experiments. It captures critical information about each run, enabling reproducibility, facilitating collaboration, and accelerating the development lifecycle of machine learning models. The primary purpose is to bring order and visibility to the iterative and often chaotic process of model development, ensuring that every experiment contributes to a deeper understanding and better outcomes.

Core Capabilities​

Comet ML Experiment Tracking offers a comprehensive suite of features designed to streamline the ML development workflow:

  • Automatic Experiment Logging: Automatically captures essential experiment metadata, including source code, git commit information, installed dependencies, hardware and environment details, and command-line arguments. This ensures that every experiment is fully reproducible without manual intervention.

    from comet_ml import Experiment

    # Initialize an Experiment object
    experiment = Experiment(project_name="my-ml-project")

    # All subsequent code execution within this scope is tracked
    # Parameters, metrics, and artifacts can be logged manually or automatically

  • Custom Logging and Artifact Management: Developers can log custom metrics, hyperparameters, text, images, audio, videos, confusion matrices, and other media types. The artifact management system allows for versioning and tracking of datasets, models, and other files, ensuring data provenance and model lineage.

    experiment.log_parameter("learning_rate", 0.001)
    experiment.log_metric("accuracy", 0.92)
    experiment.log_image("predictions.png", image_data)
    experiment.log_model("my_model", "path/to/model.pkl")

    # Log an artifact (e.g., a processed dataset)
    experiment.log_artifact("processed_data", "path/to/data.csv")

  • Experiment Visualization and Comparison: The web-based UI provides interactive dashboards to visualize experiment results, compare runs side-by-side, filter by parameters or metrics, and identify trends. This enables quick analysis of model performance across different configurations.

  • Model and Dataset Versioning: Beyond simple file storage, the platform provides a structured way to version control models and datasets as artifacts. This ensures that specific versions of models can be linked to the exact data they were trained on, enhancing reproducibility and auditability.

  • Hyperparameter Optimization: Integrates with built-in hyperparameter sweeping capabilities and external optimization frameworks. It allows developers to define search spaces and automatically run multiple experiments, tracking each trial's performance to find optimal hyperparameter configurations.

  • Reproducibility and Collaboration: Every logged experiment captures sufficient detail to reproduce the exact environment and execution. Experiments, dashboards, and reports can be easily shared with team members, fostering collaboration and knowledge transfer.

Common Use Cases​

Developers leverage Comet ML Experiment Tracking in various scenarios:

  • Model Iteration and Comparison: When experimenting with different model architectures, hyperparameter sets, or feature engineering techniques, the platform provides a centralized view to compare performance metrics, visualize training curves, and identify the most promising approaches.
  • Debugging and Performance Analysis: Tracking metrics like loss, accuracy, precision, and recall over time helps diagnose issues such as overfitting, underfitting, or training instability. Visualizations aid in pinpointing when and why performance degrades.
  • Automated Hyperparameter Sweeps: For tasks requiring extensive hyperparameter tuning, the platform automates the execution of multiple trials, logging each result and providing tools to analyze the impact of different parameter combinations.
  • Production Model Management: After training, models can be logged and versioned, forming a model registry. This facilitates tracking models from development to deployment, ensuring that the deployed model's lineage is clear and auditable.
  • Team Collaboration and Reporting: Teams use the platform to share experiment results, discuss findings, and generate reports for stakeholders, ensuring everyone has access to the latest model performance data and insights.

Common Integration Patterns​

The platform integrates seamlessly with popular machine learning frameworks, often requiring minimal code changes.

Basic Experiment Initialization​

The fundamental step involves creating an Experiment object. This object serves as the primary interface for logging data.

from comet_ml import Experiment

# For a new experiment
experiment = Experiment(
    api_key="YOUR_COMET_API_KEY",
    project_name="my-ml-project",
    workspace="my-workspace"
)

# To continue an existing experiment (e.g., after a crash or for distributed training)
# experiment = Experiment(previous_experiment="EXISTING_EXPERIMENT_ID")

# Log a simple message
experiment.log_text("Starting training process...")

Logging with TensorFlow/Keras​

The platform offers callbacks for automatic logging during TensorFlow/Keras training.

import tensorflow as tf
from comet_ml import Experiment
from comet_ml.integration.tensorflow import CometCallback

experiment = Experiment(project_name="tf-keras-project")
experiment.log_parameters({"epochs": 10, "batch_size": 32})

model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='relu', input_shape=(784,)),
    tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Use the CometCallback for automatic logging of metrics, parameters, and model graph
model.fit(x_train, y_train, epochs=experiment.get_parameter("epochs"),
          callbacks=[CometCallback(experiment)])

experiment.end()

Logging with PyTorch​

For PyTorch, manual logging within the training loop is common, or using a custom logger.

import torch
import torch.nn as nn
import torch.optim as optim
from comet_ml import Experiment

experiment = Experiment(project_name="pytorch-project")
experiment.log_parameters({"learning_rate": 0.01, "num_epochs": 5})

model = nn.Linear(10, 1)
optimizer = optim.SGD(model.parameters(), lr=experiment.get_parameter("learning_rate"))
criterion = nn.MSELoss()

for epoch in range(experiment.get_parameter("num_epochs")):
    # Simulate training step
    inputs = torch.randn(10, 10)
    targets = torch.randn(10, 1)
    outputs = model(inputs)
    loss = criterion(outputs, targets)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    # Log metrics manually
    experiment.log_metric("loss", loss.item(), step=epoch)
    experiment.log_metric("epoch", epoch, step=epoch)

experiment.end()

Logging with Scikit-learn​

Scikit-learn models can be logged by wrapping the training process and logging relevant metrics and the model itself.

from comet_ml import Experiment
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import numpy as np

experiment = Experiment(project_name="sklearn-project")

# Simulate data
X, y = np.random.rand(100, 10), np.random.randint(0, 2, 100)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Log hyperparameters
params = {"n_estimators": 100, "max_depth": 10, "random_state": 42}
experiment.log_parameters(params)

model = RandomForestClassifier(**params)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)

# Log metrics
experiment.log_metric("accuracy", accuracy)

# Log the trained model as an artifact
experiment.log_model("random_forest_model", model)

experiment.end()

Advanced Usage and Best Practices​

Offline Logging​

For environments with intermittent or no internet connectivity, the platform supports offline logging. Experiments are recorded locally and synchronized to the server once a connection is established. This is crucial for secure or air-gapped environments.

from comet_ml import Experiment

# Initialize in offline mode
experiment = Experiment(project_name="my-offline-project", offline_mode=True)

# All logging operations will store data locally
experiment.log_metric("loss", 0.1)

# To sync later, run: comet upload <path_to_offline_archive>

Environment Variables and Configuration​

API keys, project names, and workspace details are often managed through environment variables (COMET_API_KEY, COMET_PROJECT_NAME, COMET_WORKSPACE) or a .comet.config file. This avoids hardcoding sensitive information and simplifies configuration across different environments.

Performance Considerations​

While logging, the platform aims for minimal overhead. However, logging very large artifacts frequently or logging excessively granular metrics at high frequencies can introduce minor performance impacts.

  • Batch Logging: When logging many metrics or parameters, consider batching them using experiment.log_metrics(metrics_dict) or experiment.log_parameters(params_dict) rather than individual calls.
  • Asynchronous Logging: The underlying logging mechanism is largely asynchronous, minimizing blocking operations during training.
  • Artifact Size: Be mindful of the size and number of artifacts uploaded. For extremely large datasets, consider logging metadata and pointers rather than the entire dataset, or use the artifact system for versioning only critical subsets.

Known Limitations and Important Considerations​

  • API Key Security: Always protect your API key. Use environment variables or configuration files instead of embedding it directly in code, especially in shared repositories.
  • Experiment Lifecycle: Ensure experiment.end() is called to finalize an experiment and push any remaining buffered data. In scripts, this often happens automatically when the script exits, but explicit calls are good practice, especially in long-running processes or when handling exceptions.
  • Resource Usage: While generally efficient, running many concurrent experiments from a single machine might consume network bandwidth and local disk space for offline logs. Monitor resource usage in such scenarios.
  • Integration Specifics: While many frameworks have direct integrations, some custom training loops might require more manual logging to capture all desired information. Refer to specific integration guides for best practices.