Overview
This case study details how a bioinformatics engineering team partnered with Excelra to implement end-to-end integration testing for a production Nextflow targeted sequencing pipeline using the nf-test framework. Supporting five aligners, two execution environments (SLURM + Singularity and AWS Batch + Docker), and multiplexed runs of 48–96 samples, the solution delivered infrastructure-aware test configurations, a layered 150+ assertion strategy, and full CI/CD integration — enabling automated regression detection on every pull request.
Our client
The pipeline development group maintained a production Nextflow pipeline for CRISPR editing quantification via targeted sequencing. The pipeline coordinated 40+ processes across alignment, post-alignment analysis, indel quantification, and reporting stages. Code changes were reviewed via pull requests but lacked automated verification that the pipeline produced correct end-to-end output after modification. Manual testing was time-intensive and inconsistently applied.
Client’s challenge
The team faced several key challenges in establishing automated testing:
- Integration boundary bugs. Per-process unit tests verified that individual tools produced valid output, but did not test the channel wiring, metadata propagation, and output format contracts between processes. Most production regressions occurred at these boundaries.
- Multi-environment execution. The pipeline ran on both SLURM (Singularity containers, POSIX work directories) and AWS Batch (Docker containers, S3 work directories). Testing locally with Docker did not validate SLURM-specific behaviour.
- Head node constraint. nf-test orchestrates Nextflow, which submits jobs to the scheduler. Submitting nf-test itself as a SLURM job created scheduler-on-scheduler conflicts. The correct invocation required running nf-test from the head node while jobs executed on worker nodes.
- Test data design. Full production datasets were too large for iterative testing. A representative but minimal test dataset that exercised all code paths needed to be designed.
- Output complexity. The pipeline produced 40+ output files per sample across multiple directory levels. Verifying correctness required asserting on a large number of outputs without creating a brittle test suite.
Client’s goals
The primary goals were to:
- Establish automated end-to-end regression testing for the pipeline
- Support infrastructure-aware test configurations for SLURM + Singularity
- Design a test suite that completed in under 20 minutes for rapid feedback
- Integrate test execution into the CI/CD workflow for pull request validation
Our Approach
We implemented a comprehensive nf-test suite with infrastructure-specific configurations and a layered assertion strategy for this bioinformatics pipeline engagement.
Key elements of the solution included:
- Representative test dataset. A four-sample samplesheet covering positive control, negative control, multi-replicate library, and edge-case (low read count) conditions. FASTQ files were downsampled to retain a few hundred reads per target region, reducing test data volume while exercising all code paths.
- Infrastructure-aware SLURM configuration. A dedicated test configuration file submitted jobs to the SLURM scheduler with reduced resource requests (fewer CPUs, less memory, shorter walltime) while executing in the real production environment with Singularity containers from the shared cache.
- Layered assertion strategy. Three tiers of assertions: (1) 150+ file existence checks verifying every expected output file across all test samples; (2) content assertions on critical outputs (positive control has non-zero indel counts, summary file contains all sample IDs, version file lists expected tools); (3) snapshot-based validation on two key summary files whose content should be deterministic.
- Test tagging. Tests tagged by execution environment (tag “slurm”) enabling selective execution. Quick-feedback tests tagged separately from comprehensive nightly tests.
- CI/CD integration. GitHub Actions workflow triggered on pull requests, executing against the SLURM profile, parsing nf-test output into a formatted test summary, and posting results to the PR.
We documented the head node constraint prominently in the test README, specifying the exact invocation sequence and warning against submitting nf-test as a batch job.
Our solution
The engagement delivered measurable outcomes:
- 783 lines of test infrastructure implemented across test definitions, configuration files, test data specifications, and documentation.
- 150+ assertions per test run covering file existence, content correctness, and snapshot stability.
- Under 15 minutes per test execution on the SLURM cluster with the downsampled test dataset.
- Automated PR validation established. Every pull request receives a formatted test summary showing pass/fail status and execution time.
- 2 regression bugs caught during initial deployment. The test suite identified a channel wiring error and a missing output file that had been introduced in a recent refactoring but not caught by manual review.
- Head node execution documented. Clear documentation prevented team members from encountering the scheduler-on-scheduler issue.
Conclusion
This engagement established that end-to-end integration testing for Nextflow pipelines is practical with nf-test, even for complex multi-aligner pipelines running on HPC infrastructure. The layered assertion strategy – file existence for breadth, content checks for depth, snapshots for stability – balances coverage with maintainability. The infrastructure-aware configuration pattern is transferable to any Nextflow pipeline operating on SLURM, AWS Batch, or both, and supports broader goals around scientific data management and pipeline reliability in production bioinformatics environments.
