Authors: – Sreejith Cherikkallil (Principal Architect, Scientific Informatics)
Clinical diagnostics teams depend on speed, precision, and compliance to deliver reliable genetic insights. Yet many laboratories still rely on fragmented spreadsheet-based workflows that slow down operations and increase the risk of manual errors. We recently worked with Baylor Genetics, a leading clinical diagnostics company specialising in next-generation sequencing (NGS), to modernise its Variant Interpretation, Classification, and Curation (VICC) workflow. Their legacy Excel-based system created operational bottlenecks, inconsistent audit trails, and challenges in maintaining regulatory compliance. Together, we transformed this process into a scalable web-based application designed to improve efficiency, reduce human intervention, and support more accurate clinical decision-making. This project demonstrated how digital transformation in genomics can directly improve workflow reliability while supporting long-term innovation in precision medicine.
The reliance on spreadsheet-based systems in clinical genomics is not unusual — many organisations built their NGS workflows incrementally on familiar tools before data volumes and regulatory requirements outpaced what those tools could reliably support. Excelra’s blog on Bioinformatics Solutions: Tools and Techniques for Data-Driven Discoveries provides broader context on how modern bioinformatics infrastructure — from pipeline development to data platform design — is reshaping how clinical genomics teams operate at scale.
Why Spreadsheet-Based diagnostics workflows create challenges
NGS-based diagnostics generate massive amounts of variant data that require continuous review, annotation, and validation. Baylor Genetics previously relied on Microsoft Excel files, macros, and manual review processes to manage this workflow. While the process was functional, it demanded significant manual effort from clinical geneticists and data reviewers.
The organisation faced several operational challenges:
- Data inconsistencies caused by manual updates and spreadsheet versioning
- Limited audit trail visibility for compliance reporting
- Delays in reviewing and annotating SNVs (Single Nucleotide Variants) and CNVs (Copy Number Variants)
- Increased risk of human error during data processing and interpretation
Spreadsheet-based systems can introduce substantial risks in regulated environments due to manual handling and lack of scalability. In genomics, where diagnostic accuracy directly impacts patient outcomes, these limitations can become critical.
The compliance and audit trail challenges that Baylor Genetics faced are common across clinical genomics organisations. Regulatory bodies increasingly require complete, timestamped records of every variant classification decision — a requirement that is structurally difficult to meet with Excel-based workflows, regardless of how carefully they are managed. Excelra’s Pipeline Development service addresses exactly this type of gap — designing bioinformatics infrastructure that builds compliance, traceability, and scalability into the workflow architecture from the ground up, rather than retrofitting it onto legacy systems.
How we built a scalable Three-Tiered solution
To address these challenges, we collaborated closely with Baylor Genetics to redesign the VICC workflow using a three-tiered architecture. Our goal was not simply to replace spreadsheets, but to create an integrated NGS data platform that could automate data ingestion, improve traceability, and support faster clinical review.
Our approach included the following steps:
- Understanding Existing Workflow: We conducted detailed interviews and workflow analyses to map out the client’s current process and identify pain points.
- Customized Solution Development: Based on the gathered insights, we developed a web application and a centralised SQL database to replace the Excel-based system. This solution was designed to handle data ingestion, pre-processing, and annotation with the internal variant library.
- Incremental Implementation: To ensure a smooth transition, we implemented the solution in phases. This allowed us to address any issues promptly and ensured minimal disruption to the client’s operations.
The transition from an Excel-based system to a web-based application involved several challenges, particularly related to the client’s existing infrastructure. The key challenges and our solutions were as follows:
- Data Migration: Migrating data from Excel files to a SQL database needed to be precise to maintain data integrity. We developed robust scripts and validation procedures to ensure all data was accurately transferred.
- Processing of Ingested Data: We utilised in-house Python scripts for processing ingested SNVs (Single Nucleotide Variants) and CNVs (Copy Number Variants), ensuring the data was handled efficiently and accurately.
- Logic Conversion: The macros, rules, and business logic used in the Excel files were provided by the client. We meticulously converted these into application logic, ensuring that the new system replicated the necessary functions without the limitations of Excel.
- User Adaptation: Shifting from a familiar Excel interface to a web application could be daunting for users. We provided comprehensive training sessions and user manuals to ensure a smooth adoption process.
To address the client’s compliance concerns, we incorporated detailed audit trail functionalities within the web application. This ensured that every action and interaction was logged and could be reviewed for regulatory compliance. Additionally, the automation of data processing and annotation significantly reduced human errors and improved overall operational efficiency.
Fig 1: High Level Workflow
Tier 1: Web application
- Project Management: Provided a centralised platform for managing the projects, facilitating seamless coordination among team members.
- Data Review and Annotation: Enabled clinical geneticists to review and annotate data efficiently through an intuitive user interface.
- Data Submission and Display: Simplified the submission process and provided real-time display of data and audit trail reports, ensuring transparency and traceability.
Tier 2: Data ingestion, processing, and curation
- JSON File Ingestion: Automated the ingestion of JSON files from the client’s infrastructure, reducing manual data entry and associated errors.
- Pre-processing and Curation: Utilised in-house scripts based on client-specific rules and logic to preprocess and curate the NGS variant data, ensuring consistency and accuracy.
- Message Generation: Generated messages for the Apache Event Hub to track workflow progress and log audit trail activities, enhancing oversight and compliance.
Tier 3: MS SQL database
- Centralised Storage: Stored variant data from test samples (CNVs and SNVs) and internal variant library in a structured SQL database, enabling efficient data retrieval and management.
- Workflow Tracking: Recorded messages for workflow tracking and audit trail reports in the database, providing a comprehensive overview of the entire NGS diagnostic process.
- Enhanced Data Annotation: Integrated with the internal variant library to provide accurate data annotation, improving the reliability of diagnostic results.
The transformation delivered measurable operational improvements for the client. By automating data processing and replacing manual Excel-based reviews, the new platform reduced workflow complexity and improved data accuracy. The centralised system also strengthened regulatory compliance by creating detailed audit trails for every workflow action. Teams gained better visibility into workflow progress, while automation reduced repetitive manual tasks for clinical reviewers.
The architecture pattern deployed for Baylor Genetics — a three-tier web application with automated data ingestion, Python-based processing, and a centralised SQL database — is directly transferable to other clinical genomics organisations facing similar NGS workflow modernisation challenges. For context on how similar digital transformation principles have been applied to compound registration in pharmaceutical R&D, see Excelra’s case study on Modernisation and Cloud Migration of Legacy R&D Compound Registration Platform — a parallel migration from legacy tooling to a scalable, cloud-native data platform with the same emphasis on data integrity, audit trails, and user adoption.
For clinical genomics teams evaluating how to structure the bioinformatics infrastructure that feeds into clinical NGS workflows, Excelra’s case study on Cloud Deployment and Integration of a University Hospital Optimised Whole-Exome Pipeline demonstrates how a production-grade whole-exome sequencing pipeline — with comparable data ingestion, processing, and variant annotation requirements — was deployed and validated in a regulated clinical environment.
Conclusion
Modern clinical diagnostics require more than accurate science; they require efficient, scalable, and compliant digital systems. Our collaboration with Baylor Genetics showed how replacing spreadsheet-based workflows with an integrated web platform can improve operational efficiency, strengthen audit readiness, and support more reliable diagnostic outcomes.
As genomics data volumes continue to rise, organisations that modernise their NGS workflows today will be better prepared to deliver faster and more precise patient care tomorrow. One important lesson from this project is that successful digital transformation is not only about technology implementation. It also requires understanding existing workflows, supporting user adoption, and building systems that align with regulatory and operational realities.
Excelra’s scientific informatics team supports organisations across the full spectrum of NGS workflow modernisation — from legacy system assessment and requirements gathering through web application development, data migration, bioinformatics pipeline design, and post-implementation support. To explore how Excelra’s capabilities can apply to your clinical genomics or diagnostic data engineering programme, visit our Scientific Informatics services page.
Why are Excel-based workflows a problem for NGS diagnostics?
Excel-based workflows create several structural problems in NGS diagnostics that become increasingly critical as data volumes and regulatory requirements grow. First, spreadsheet versioning is inherently unreliable — when multiple users update the same file, or when files are copied and modified separately, data inconsistencies accumulate that are difficult to detect and even harder to resolve retrospectively. Second, Excel cannot generate the kind of timestamped, user-attributed audit trails that regulatory bodies increasingly require for clinical variant classification decisions. Third, manual data entry at scale introduces transcription errors that can propagate through the entire interpretation workflow. Fourth, Excel macros and business logic are fragile — they break with version updates, are difficult to validate formally, and cannot easily be tested against new data types. In regulated environments such as clinical genetics and NGS diagnostics, these limitations represent both operational risk and compliance risk that compound as the organisation scales.
What is a three-tier web application architecture for clinical genomics?
A three-tier web application architecture separates a clinical genomics platform into three distinct layers, each with defined responsibilities. The presentation tier is the web application — the interface through which clinical geneticists review variants, annotate findings, manage projects, and generate audit trail reports. The application tier handles the business logic — automated data ingestion from upstream sources like JSON files from sequencing instruments, Python-based pre-processing and curation according to client-specific rules, and message generation for workflow tracking systems like Apache Event Hub. The data tier is the centralised SQL database — storing variant data from SNVs and CNVs, the internal variant library used for annotation, and the complete workflow and audit trail records. This separation of concerns makes the system easier to scale, maintain, and validate against regulatory requirements, because each layer can be updated or audited independently without disrupting the others.
What does SNV and CNV curation involve in a clinical NGS workflow?
SNV (Single Nucleotide Variant) and CNV (Copy Number Variant) curation are the core analytical steps in clinical NGS diagnostics, where raw variant calls from sequencing are systematically reviewed, classified, and annotated to support clinical reporting. SNV curation involves evaluating each detected nucleotide change against databases of known pathogenic, benign, and uncertain-significance variants, applying clinical classification frameworks such as ACMG guidelines, and annotating variants with supporting evidence from literature and population databases. CNV curation involves reviewing regions of copy number gain or loss detected by the sequencing assay, assessing their clinical significance based on size, gene content, and overlap with known pathogenic regions. In a manual, Excel-based workflow, this process is time-consuming, error-prone, and difficult to audit. A web-based platform automates the data ingestion and annotation steps, providing clinical reviewers with pre-processed, annotated variants and structured interfaces for classification — dramatically reducing review time and improving consistency.
How do you migrate variant data from Excel to a SQL database without losing data integrity?
Migrating variant data from Excel to a SQL database requires a structured approach that prioritises completeness, consistency, and validation at every step. The migration begins with a comprehensive data audit — cataloguing all Excel files, mapping their structure to the target database schema, and identifying inconsistencies such as multiple formats for the same variant identifier, missing values, or columns with mixed data types. Custom migration scripts — typically written in Python — are developed to extract, transform, and load data from each Excel file into the SQL database, applying standardisation rules during the transformation step. Validation procedures compare row counts, key field values, and derived metrics between source and target to verify that no data was lost or altered. For clinical genomics specifically, extra validation steps are applied to variant identifiers, classification values, and sample linkages — the fields most critical to diagnostic accuracy. A phased migration approach, beginning with historical data before cutting over live production workflows, allows issues to be identified and resolved before the new system is depended upon for clinical decisions.
What are the regulatory compliance requirements for clinical NGS variant workflows?
Clinical NGS variant classification workflows in the United States and Europe are subject to regulatory requirements from several bodies. In the US, CLIA (Clinical Laboratory Improvement Amendments) regulations govern the quality standards for laboratories performing clinical testing, including NGS-based genetic testing. The College of American Pathologists (CAP) accreditation programme provides specific guidance on NGS validation and quality management. The FDA regulates NGS-based in vitro diagnostic tests and has issued guidance on analytical validation for NGS. In Europe, IVDR (In Vitro Diagnostic Medical Devices Regulation) applies to diagnostic genomics testing. Across all frameworks, a common compliance requirement is a complete, traceable audit trail — a timestamped record of every classification decision, annotation action, and workflow event that can be reviewed by regulatory inspectors. Excel-based systems are inherently unable to generate audit trails of this quality, which is why web-based platforms with built-in logging and workflow tracking are becoming the standard for clinical NGS operations.
How long does it typically take to migrate a clinical NGS workflow from Excel to a web platform?
Migration timelines for clinical NGS workflows from Excel to a web-based platform depend on several factors: the volume and complexity of historical data to be migrated, the number of variant types and classification workflows involved, the complexity of existing business logic in macros and rules, the regulatory validation requirements of the organisation, and the number of users who need to be trained on the new system. A focused migration covering a single workflow — such as the VICC (Variant Interpretation, Classification, and Curation) workflow at Baylor Genetics — typically takes six to twelve months from requirements gathering through go-live when managed by an experienced implementation team. This timeline includes workflow analysis, application development, data migration, user acceptance testing, regulatory validation documentation, user training, and a phased cutover period where the old and new systems run in parallel. Larger organisations with multiple workflows, platforms, and sites should plan for twelve to twenty-four months for a comprehensive migration programme.
Ready to Modernise Your NGS Diagnostics Workflow?
Excelra's scientific informatics and data engineering teams help clinical genomics organisations replace fragmented, spreadsheet-based NGS workflows with scalable, compliant, and automated web platforms. Whether you are managing SNV/CNV variant curation, building clinical audit trail systems, or migrating legacy diagnostics infrastructure, our team is ready to collaborate.
