Amazon SageMaker Pipelines: Streamlined Troubleshooting and Restarting Failed Pipeline Runs

Contributors: Yavuz Kulaber

Amazon SageMaker Pipelines is a powerful feature within the Amazon SageMaker service that simplifies the creation, automation, and management of machine learning (ML) workflows. With SageMaker Pipelines, you can define a sequence of steps, called a pipeline, which covers the entire ML lifecycle—from data preparation to model deployment. This capability supports scalable AI and ML workflows and cloud-native data science pipelines.

The challenge: handling pipeline failures

When building an ML pipeline, each step of the ML lifecycle is outlined, including preprocessing, model training, and evaluation. However, even after successfully completing the early stages, pipelines may fail at a later step. Typically, this failure requires resubmitting the job, causing the entire pipeline to restart from the beginning, which can be quite time-consuming in scientific informatics environments.

For example, imagine a scenario where the preprocessing and training stages are completed, but the pipeline fails during the evaluation step. If the earlier stages were computationally expensive or time-consuming, restarting the whole pipeline from scratch is highly inefficient.

The solution: efficiently resuming a failed pipeline

This guide will walk you through how to troubleshoot and restart a failed pipeline in Amazon SageMaker. Instead of restarting from the first step, you’ll learn how to resume the process from the exact point where the failure occurred. This approach aligns with best practices in data science and analytics services.

Steps to restart a failed pipeline

Locate the failed job

First, navigate to your AWS account:

• Open the AWS Management Console.
• Go to Amazon SageMaker > Processing Jobs.
• Look for the job that failed (for this example, let’s assume the failure occurred during the evaluation stage). You’ll find both the Preprocessing and Evaluation steps listed under Processing Jobs.

Obtain the source files

Click on the failed job. Under the job details, locate the Processing Inputs section and copy the S3 location of the .sourcedir.tar.gz file, which contains the necessary source files stored in secure cloud infrastructure.

Download and extract the files

Using your command line interface (CLI), download the file from the S3 bucket and extract its contents. You can use the following command:

mkdir troubleshooting # Name the folder as desired
cd troubleshooting # Go to the working directory

aws s3 cp s3://[your-S3-bucket]/[path-to-file] ./ # Copy the S3 URL

tar -xzf sourcedir.tar.gz # Extract the files (this may take some time)

Identify and fix the issue

Examine the error logs to identify the script causing the failure. In this case, the issue occurred in the evaluation step, so you’ll need to modify the evaluate.py script. Based on the error logs, make the necessary changes to fix the problem, following best practices in scalable data pipelines and scientific data management.

Repackage the files

After making the changes, repackage the updated files into a new archive. Run this command to create a fresh .tar.gz archive:

tar --exclude='sourcedir.tar.gz' -czf sourcedir.tar.gz *

Upload the updated archive

Next, upload the updated archive back to its original S3 location using this command:

aws s3 cp sourcedir.tar.gz s3://[your-S3-bucket]/[path-to-file]

Restart the pipeline

To resume the failed pipeline:

• Go to SageMaker Studio.
• Click the Home button and navigate to Pipelines.
• Locate the failed pipeline, click Retry, and the pipeline will continue from the step where it failed—supporting efficient cloud enablement and automation.

Conclusion: save time with targeted restarts

By following these steps, you can efficiently troubleshoot and resume failed Amazon SageMaker pipelines without having to re-run the steps that have already been successfully completed. This process not only saves time but also ensures smoother pipeline management for ML workflows running on scalable cloud platforms and enterprise data science environments.