Authors: – Ethan Loo (Sr. Bioinformatics Specialist)
When samples silently lose reads mid-pipeline, finding the cause can take longer than running the analysis itself. A generalizable tracking system changes the equation.
Next-generation sequencing (NGS) pipelines have grown in complexity over the past decade. A modern production pipeline may coordinate thirty or more discrete processes, such as quality control, adapter trimming, alignment, deduplication, and variant calling, each consuming and producing files that feed into subsequent stages. This complexity introduces a subtle but consequential problem: sample attrition. In a multiplexed experiment involving dozens or hundreds of samples, a fraction of reads is lost at every processing stage:
- Adapter-heavy libraries lose reads during trimming.
- Low-quality reads are filtered during alignment.
- Duplicate molecules are collapsed during deduplication.
For most samples, this attrition is expected and benign. But when a sample emerges from the pipeline with unexpectedly low or zero counts in the final output, the analyst faces a frustrating question: where did the reads go? In production environments where work directories may be ephemeral cloud storage, answering that question manually is tedious at best and impossible at worst. This article describes a generalised, auto-discovery-based approach to monitoring sample attrition through every stage of a Nextflow pipeline.
In short: a Nextflow sample dropout tracking system automatically records read counts per sample at every pipeline stage, parses the Nextflow trace file to reconstruct execution order, and produces a structured attrition table that distinguishes sample-specific failures from systemic pipeline issues — without requiring any changes to existing analysis modules. The rest of this article explains how to build one.
For broader context on how production bioinformatics pipelines are designed, validated, and maintained at scale, see Excelra’s Workflow Managers in Bioinformatics: A Practical Q&A — covering the Nextflow ecosystem alongside other workflow managers used in clinical and research genomics environments.
The problem: Invisible sample attrition
Read loss is expected in filtering pipelines; the problem is delayed observability and lack of traceability of that attrition.
In practice, the challenge is less about knowing that reads are filtered and more about knowing exactly when a specific sample diverged from the expected path. By the time a final result file is empty, the evidence may be scattered across many process-specific work directories on local storage or in an Amazon S3 bucket. The analyst has to reconstruct that sample’s execution history, identify the relevant intermediate outputs, and compare counts stage by stage to determine whether the failure reflects a gradual loss, a single catastrophic drop, or a missing output. In pipelines with tens of processes and dozens of samples, this investigation can consume hours. In cloud execution environments where intermediate files may be deleted after completion, it may be impossible.
What is needed is a system that automatically records sample-level counts at every pipeline stage and produces a structured report, analogous to a flight data recorder for bioinformatics pipelines.
A practical tracking system for production Nextflow pipelines must satisfy several design constraints. It should require zero configuration by default, auto-discovering all processes from the Nextflow trace file without requiring a manually maintained list. Steps should be reported in actual execution order, not alphabetically. Different bioinformatics file types (i.e. BAM, FASTQ, TSV, VCF) require different counting strategies, and the system must apply the correct one automatically. Finally, the system must work transparently across local POSIX filesystems and cloud object stores.
This last constraint — transparency across local and cloud storage — is particularly important for production NGS pipelines running on AWS infrastructure, where S3 object stores replace traditional POSIX work directories. Excelra’s case study on Scaling Up and Enhancing Efficiency of NGS Pipeline with Nextflow illustrates how Nextflow-based production pipelines are architected for hybrid on-premises and cloud execution — the same environment where this dropout tracking system needs to operate reliably.
Architecture: Two processes, One python library
Per-sample tracking runs once per sample after all upstream analysis processes have completed. It receives the Nextflow trace file and the work directory base path. A Python script parses the trace file to discover all tasks that ran for the target sample, matching task names against the sample identifier, then sorts them by submission timestamp to reconstruct actual execution order. For each discovered process, the script scans output files in the corresponding work directory and applies a type-appropriate extractor.
The extractor registry is the core technical component. BAM files are counted via a SAM/BAM parsing library, reporting aligned reads with optional flag-based filtering. FASTQ files are counted by dividing total lines by four, with transparent gzip handling. TSV and BED files count data rows while skipping headers and comment lines, and distinguish between a truly empty file, a header-only file, and one with data, each representing a different diagnostic signal. Per-sample results are written as structured JSON records.
Aggregation collects all per-sample JSON records and produces various output formats, such as a long-form counts table (one row per sample-step, optimised for programmatic analysis), a wide-form counts table (samples as rows, steps as columns, optimised for visual inspection), an attrition statistics table (per-step mean, median, minimum, maximum, and zero-count sample count), and a per-sample retention summary. The attrition table is the primary operational output: it reveals both systemic issues affecting all samples and sample-specific failures at a glance.
The structured JSON output from the per-sample tracking process makes this system straightforwardly composable with downstream reporting and alerting tools — the same JSON records can feed a dashboard, trigger a Slack notification when zero-count samples exceed a threshold, or be archived alongside pipeline results for audit purposes. This composability reflects a broader principle in production bioinformatics: observability outputs should be structured data, not just log text. See Excelra’s Pipeline Development service page for an overview of how Excelra designs production bioinformatics pipelines with observability, traceability, and cloud compatibility built in from the ground up.
What gets reported
The practical value of the system is most apparent in two output artifacts. The attrition table contains one row per pipeline stage and summary statistics across all samples: mean count, median count, minimum, maximum, and the number of samples that reached zero. A stage where 3 out of 96 samples show zero reads but 93 continue normally points to sample-specific issues: library preparation failures, low-input wells, or contaminated samples. A stage where the mean count drops by 60 percent for all samples points to a systemic issue: an overly aggressive quality filter, a misconfigured trimming parameter, or a reference mismatch. These two failure modes require different responses, and the attrition table distinguishes them instantly.
The per-sample retention summary complements this with a simple pass/fail view, recording whether each sample was retained through all stages or dropped, and if dropped, which stage was the last to produce a non-zero count. Analysts consult the retention summary first to identify failed samples, then drill into the attrition table for those samples to understand the failure mode.
Because the system integrates as standard Nextflow processes, with no modifications to existing analysis modules, it can be added to any pipeline by enabling Nextflow’s built-in trace file generation. No upstream code changes are required.
The distinction the attrition table makes — between a stage where three samples fail and a stage where all samples lose 60 percent of their reads — is precisely the distinction that determines the response. Sample-specific failures call for re-examining library preparation or input material; systemic failures call for re-examining pipeline parameters. Having that distinction available in a single table lookup, rather than after hours of manual investigation, is what makes the system operationally valuable in production environments. As Ethan Loo, Sr. Bioinformatics Specialist at Excelra, puts it: “Automated dropout tracking converts a multi-hour root-cause investigation into a single table lookup.”
For teams working with whole-exome sequencing or large multiplexed NGS datasets where sample dropout at any stage has direct clinical or research consequences, Excelra’s case study on Cloud Deployment and Integration of a University Hospital Optimised Whole-Exome Pipeline demonstrates how production-grade QC and observability are built into clinical WES pipelines running at scale — exactly the environment where a dropout tracking system like this one has the highest operational value.
Conclusion
Automated dropout tracking converts a multi-hour root-cause investigation into a single table lookup. By parsing the Nextflow trace file rather than maintaining a static process list, the system adapts automatically as the pipeline evolves — no code changes required when processes are added, renamed, or removed. The extractor pattern provides a framework for extending observability to new output formats as pipelines grow. At Excelra, we apply this observability-first approach across production sequencing pipelines, ensuring that every sample failure is diagnosed rapidly and completely, regardless of whether the pipeline runs on on-premises HPC or cloud infrastructure.
A Nextflow sample dropout tracking system, as described in this article, is a two-process Nextflow module — comprising a per-sample tracking process and an aggregation process — that parses the pipeline trace file, applies format-specific read-counting extractors to each discovered output file, and produces structured attrition and retention tables without requiring changes to existing analysis processes. This definition, combined with the implementation details above, is intended to provide a complete, citable reference for practitioners building or evaluating NGS pipeline observability solutions.
Excelra’s bioinformatics engineering team builds and maintains production NGS pipelines across WGS, WES, RNA-seq, and single-cell sequencing workflows on both HPC and cloud infrastructure. To explore how Excelra’s pipeline development capabilities can support your next NGS project, visit our Bioinformatics services.
What is sample dropout in a Nextflow NGS pipeline?
Sample dropout in a Nextflow NGS pipeline refers to the complete or near-complete loss of reads for a specific sample at one or more processing stages — resulting in an empty or unexpectedly sparse output file downstream. Unlike expected read attrition, which affects all samples proportionally as low-quality reads and adapters are filtered, sample dropout is sample-specific and often indicates a failure in library preparation, a low-input sample, contamination, or a pipeline misconfiguration that affects only certain input files. The core challenge with sample dropout is observability: by the time an analyst notices that a final output file is empty, the intermediate evidence — the read counts at each stage — may be scattered across ephemeral work directories on local storage or cloud object stores, making root-cause analysis time-consuming. A structured dropout tracking system captures these counts automatically, making the failure point immediately identifiable.
How does the Nextflow trace file enable auto-discovery of pipeline stages?
The Nextflow trace file is a tab-separated log that Nextflow generates automatically when tracing is enabled — recording one row per task execution with fields including the task name, submission timestamp, working directory path, exit status, and resource usage. A dropout tracking system uses this file to auto-discover all pipeline stages without requiring a manually maintained process list. The Python script reads the trace file, filters rows matching the target sample identifier (typically embedded in the task name), and sorts them by submission timestamp to reconstruct the actual execution order. This approach makes the tracking system automatically adaptive: when processes are added, renamed, or removed from the pipeline, the trace file reflects those changes and the tracking system discovers them without any code changes. Enabling trace generation in Nextflow requires adding a single trace configuration block to the nextflow.config file.
How do you count reads in BAM, FASTQ, TSV, and VCF files in a Nextflow pipeline?
Different bioinformatics file formats require different counting strategies, which the extractor registry handles automatically. BAM files are counted using a SAM/BAM parsing library — reporting the number of aligned reads with optional bitwise flag filtering to exclude unmapped, duplicate, or secondary alignments depending on the diagnostic purpose. FASTQ files are counted by dividing the total line count by four, since each read occupies exactly four lines; the extractor handles gzip-compressed FASTQ files transparently. TSV, BED, and similar tabular formats are counted by iterating through lines, skipping comment lines beginning with hash characters and header lines, and counting data rows. Importantly, the extractor distinguishes between a truly empty file, a file containing only headers, and a file with data — each representing a different diagnostic signal. VCF files follow the same logic as TSV with variant-specific header handling.
What is the difference between a systemic pipeline failure and a sample-specific dropout?
The attrition table distinguishes between two fundamentally different failure modes that require different responses. A sample-specific dropout occurs when a small number of samples — for example, 3 out of 96 — reach zero reads at a given stage while the remaining samples continue normally. This pattern indicates a sample-level problem: library preparation failure for that specific sample, a low-input well in the sequencing plate, contamination, or a sample identifier collision. The correct response is to re-examine the affected samples at the source. A systemic failure occurs when the mean count across all samples drops sharply at a specific stage — for example, by 60 percent. This indicates a pipeline-level problem: an overly aggressive quality filter, a misconfigured trimming parameter, a reference genome mismatch, or a software version incompatibility. The correct response is to re-examine the pipeline parameters at that stage. Having both failure modes visible simultaneously in a single table is what makes the attrition output operationally valuable.
How does the dropout tracking system work in cloud execution environments like AWS?
In cloud execution environments, Nextflow typically stores intermediate work files in Amazon S3 buckets rather than local POSIX filesystems. The dropout tracking system handles this transparently by using a unified file access layer that supports both POSIX paths and S3 URIs — reading the work directory path from the Nextflow trace file and applying the appropriate access protocol based on the path prefix. This design means the same tracking code runs identically whether the pipeline executes on on-premises HPC, a cloud-based cluster, or a hybrid environment. One important consideration in cloud execution is that intermediate files in S3 may be deleted after pipeline completion if the bucket has a lifecycle policy configured. The dropout tracking system should therefore run as a Nextflow process within the pipeline itself — collecting counts before cleanup occurs — rather than as a post-processing step that runs after the pipeline completes.
How do you add dropout tracking to an existing Nextflow pipeline without modifying analysis modules?
Adding the dropout tracking system to an existing Nextflow pipeline requires two steps, neither of which modifies the existing analysis modules. First, enable Nextflow’s built-in trace file generation by adding a trace block to the nextflow.config file specifying the output path and fields to include, at minimum: task name, hash (work directory suffix), submit timestamp, and status. Second, add the two tracking processes — per-sample tracking and aggregation — to the pipeline’s workflow definition, wiring them to receive the trace file path and work directory base path as inputs and produce their JSON and tabular outputs. The per-sample tracking process should be triggered after all analysis processes complete for each sample, using Nextflow’s process dependency syntax. Because the system reads from the trace file and work directories rather than intercepting data flows between analysis processes, zero changes to existing modules are required. The system can also be packaged as a Nextflow subworkflow for easy reuse across multiple pipelines.
Building or Scaling an NGS Pipeline?
Excelra's bioinformatics engineering team designs and maintains production Nextflow pipelines for WGS, WES, RNA-seq, and single-cell sequencing — with observability, data quality, and cloud compatibility built in from the ground up. If sample dropout, pipeline traceability, or cloud-to-HPC portability are challenges in your current NGS workflow, our team is ready to help.
