What makes U-Net a winner in Biomedical Image Segmentation Models- Excelra’s take

Contributors: Sushant Agarwal, Sherin Eapen, Mansi Upadhyay, Ashu Srivastav, Hita Garapati, Neha Kulkarni, Jitesh K. Pillai, Uzma Saeed, Puneet Saxena, Chandra Sekhar Pedamallu

Spatial barcoding-based transcriptomics

The spatial barcode-based transcriptomic approaches involve the sequencing of the RNA species at the whole transcriptome level. They offer unbiased and high-throughput analytical solutions. In this technology, the tissue sections are immobilized on the glass slides along with the reverse transcription primers with poly-T, which bind to the poly-A of the mRNA from the tissue sections. The primers also contain the spatial barcodes and unique molecular identifiers (UMIs) that represent the coordinates of each array.

When the tissue is permeabilized, the mRNA molecules in the tissue cells get diffused into microwells (100 μm in size) on the slides and get hybridized with primers. The reverse transcription reagents will be added to the tissue to synthesize the cDNA molecules, which are then visualized using the Cy3-labeled nucleotides. The tissue section is removed by enzymes and the cDNA molecules remain hybridized on the glass side (Stahl et al. 2016).

Visium is the most widely applied spatial omics technologies that can sequence tissues of 6mm × 6mm in size. Each spot on the chip can be measured at the resolution of ~100 µm containing about 2–10 cells. In 2019, this method got further developed by 10× Genomics and commercialized as “10× Genomics Visium”. Its application benefits lead to exploring the new functionality of the organelles and may enhance our understanding in the field of spatiotemporal molecular medicine (Figure 1).

The latest Visium HD technology offers high sensitivity and single-cell scale resolution by improving spot resolution to 2 x 2 um squares, providing even more granular insights into tissue composition and cellular interactions. This high-definition mapping is crucial for identifying rare cell types and understanding complex biological processes.

With advancements in computing infrastructure and big data availability, Artificial intelligence (AI) is benefitting and evolving rapidly, widening its scope in diverse domains that includes biopharma and healthcare industry. From diagnosing diseases to suggesting personalised treatment, with each success story, it has managed to gain confidence of the experts as well as large regulatory authorities such as FDA. Owing to rising concerns in oncology field, its potential is now being tested specially in the early detection of cancer.

Recently, medical imaging has become one of the prominent tools used in detection of cancer as it avoids the dependency on complicated biopsy procedures. While our earlier whitepaper details on various Biomedical image analysis tools used in Cancer detection and treatment along with over-arching benefits of AI/ML, this blog specifically focusses on U-Net biomedical image segmentation and how it’s playing a foundational role in the medical imaging world.

The U-Net architecture (Fig 1), originally proposed by Olaf Ranneberger and colleagues in 2015 for biomedical image segmentation, quickly became a cornerstone in the field of biomedical image segmentation models. Its primary appeal lies in its ability to produce precise, pixel-level segmentations from relatively small datasets, which is often the case in medical applications.

U-Net’s symmetric architecture, composed of an encoder-decoder structure with skip connections, allows it to capture both global context and fine-grained details simultaneously. This characteristic has positioned U-Net as a go-to model for various image segmentation tasks, not just in medicine but also in satellite imaging, object detection, and more.

Basic U-net architecture. The arrows represent the different operations, the blue boxes represent the feature map at each layer, and the gray boxes represent the cropped feature maps from the contracting path
Figure 1: Basic U-net architecture. The arrows represent the different operations, the blue boxes represent the feature map at each layer, and the gray boxes represent the cropped feature maps from the contracting path [1].

Figure 1: Basic U-net architecture. The arrows represent the different operations, the blue boxes represent the feature map at each layer, and the gray boxes represent the cropped feature maps from the contracting path [1].

Key Features of U-Net

1. Encoder-Decoder Architecture: U-Net follows a symmetric U-shaped architecture. The encoder path consists of repeated convolution and max-pooling layers, which reduce the spatial resolution while increasing feature depth. The decoder path mirrors the encoder, using up-sampling layers to restore the original resolution.
2. Skip Connections: The skip connections in U-Net link corresponding layers in the encoder and decoder paths, allowing the model to retain high-resolution features from earlier layers. This is crucial for improving segmentation accuracy, as it helps the model preserve spatial details that might otherwise be lost during down-sampling.
3. Fully Convolutional Network (FCN): U-Net is a fully convolutional network, meaning it does not contain any fully connected layers, allowing it to handle variable image sizes. This also reduces the number of parameters compared to models that use fully connected layers, making U-Net more efficient.
4. Data Augmentation: One of U-Net’s key strengths lies in its use of aggressive data augmentation strategies. By augmenting the training data with rotations, translations, and other transformations, U-Net can generalize well even when the training dataset is small, which is often the case in biomedical applications.

Advantages of U-Net

1. High Accuracy on Limited Data: U-Net’s ability to perform well with small, annotated datasets is one of its most significant advantages. The model’s architecture is designed to maximize the use of available data through data augmentation and careful feature map handling.
2. Precise Segmentation: The use of skip connections allows U-Net to produce highly detailed segmentations. This is particularly important in medical imaging, where accurately delineating boundaries (e.g., between tumors and healthy tissue) is critical.
3. Versatility: Although U-Net was originally designed for biomedical segmentation, its utility extends far beyond. It has been successfully applied in fields such as satellite imagery, environmental monitoring, and more, showcasing its versatility.
4. Fast Convergence: Due to its efficient architecture and the inclusion of skip connections, U-Net tends to converge quickly during training, often requiring fewer epochs than other models for similar tasks.

Application areas

U-Net has found applications in various domains where pixel-level precision is essential (Fig 2). Its impact on medical imaging is the most well-known, with use cases such as segmenting organs, tumors, or lesions from MRI, CT, or histopathology images. The model has also been applied to tasks in computer vision such as defect detection in manufacturing, autonomous driving (e.g., road segmentation), and even agriculture (e.g., plant disease detection).

Fig 2: Various image modalities that U-Net is capable of handling.

Variants of U-Net

When examining U-Net and its variants, it becomes clear that while the advancements in architectures like 3D U-Net and U-Net++ bring substantial improvements, they also raise certain concerns about complexity and generalization. These models, though more powerful, come with their own set of trade-offs, especially in terms of computational overhead and application-specific suitability.

3D U-Net: powerful but overly specialized?

The 3D U-Net (Fig 3) represents a substantial leap in volumetric segmentation, allowing deep learning models to process 3D data more effectively. Its architecture, a natural extension of the traditional U-Net, is meticulously designed to handle volumetric inputs through 3D convolutions and corresponding operations. However, the introduction of such architectural complexity might not always result in generalized improvements across all applications. While medical imaging, particularly tasks involving MRI and CT scans, clearly benefits from this enhancement, the specificity of 3D U-Net’s design could hinder its adaptability to more generic tasks. Its reliance on 3D convolutions and volumetric data makes it a niche model—highly effective but overly specialized.

References

• Stahl PL, Salmen F, Vickovic S, et al. Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353(6294):78–82 (2016).
• Rao A, Barkley D, Franca GS, Yanai I, Exploring tissue architecture using spatial transcriptomics, Nature 596 211–220 (2021).
• Garcia-Alonso L, Lorenzi V, Mazzeo CI, Alves-Lopes JP, Roberts K, Sancho-Serra C, Engelbert J, Mareckova M, Gruhn WH, Botting RA, Li T, Crespo B, van Dongen S, Kiselev VY, Prigmore E, Herbert M, Moffett A, Chedotal A, Bayraktar OA, Surani A, Haniffa M, Vento-Tormo R, Single-cell roadmap of human gonadal development, Nature 607 540–547 (2022).
• Liu C, Li R, Li Y, Lin X, Zhao K, Liu Q, Wang S, Yang X, Shi X, Ma Y, Pei C, Wang H, Bao W, Hui J, Yang T, Xu Z, Lai T, Berberoglu MA, Sahu SK, Esteban MA, Ma K, Fan G, Li Y, Liu S, Chen A, Xu X, Dong Z, Liu L, Spatiotemporal mapping of gene expression landscapes and developmental trajectories during zebrafish embryogenesis, Cell 57, 1284–1298 e1285 (2022).
• Maynard, K.R., Collado-Torres, L., Weber, L.M. et al. Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex, Nat Neurosci 24, 425–436 (2021).
• Satilmis B, Sahin TT, Cicek E, Akbulut S, Yilmaz S. Hepatocellular Carcinoma Tumor Microenvironment and Its Implications in Terms of Anti-tumor Immunity: Future Perspectives for New Therapeutics. J Gastrointest Cancer. Dec;52(4):1198-1205. (2021).
• Wu Y, Yang S, Ma J, Chen Z, Song G, Rao D, Cheng Y, Huang S, Liu Y, Jiang S, Liu J, Huang X, Wang X, Qiu S, Xu J, Xi R, Bai F, Zhou J, Fan J, Zhang X, Gao Q, Spatiotemporal immune landscape of colorectal cancer liver metastasis at single-cell level, Cancer Discov. 12 134–153 (2022).
• Zhao N, Zhang Y, Cheng R, Zhang D, Li F, Guo Y, Qiu Z, Dong X, Ban X, Sun B, Zhao X, Spatial maps of hepatocellular carcinoma transcriptomes highlight an unexplored landscape of heterogeneity and a novel gene signature for survival, Cancer Cell Int. 22 57 (2022).
• Zugazagoitia J, Gupta S, Liu Y, Fuhrman K, Gettinger S, Herbst RS, Schalper KA, Rimm DL, Biomarkers associated with beneficial PD-1 checkpoint blockade in non-small cell lung Cancer (NSCLC) identified using high-Plex digital spatial Profiling, Cancer Res. 26 4360– 4368 (2020).
• Gouin KH, Ing N, Plummer JT, Rosser CJ, Ben Cheikh B, Oh C, Chen SS, Chan KS, Furuya H, Tourtellotte WG, Knott SRV, Theodorescu D, An N-cadherin 2 expressing epithelial cell subpopulation predicts response to surgery, chemotherapy and immunotherapy in bladder cancer, Commun. 12 4906 (2021).