A Saturation Artifact Inpainting Algorithm for Cell Fluorescence Microscopic Images

doi:10.12068/j.issn.1005-3026.2024.07.002

Abstract

Abstract:

Fluorescence microscope images of cells contain abundant phenotypic features， which are used to study the absorption and transportation of substances in cells， as well as chemical distribution and localization. These analyses require high?quality cell images. However， saturation artifacts will cause serious loss of phenotypic features， which will affect morphological analysis and certain classification experiments. From the perspective of data post?processing， a two?stage cell image inpainting model is proposed based on generative adversarial networks to solve the loss of phenotypic features caused by saturation artifacts. The model can restore large areas of missing phenotypic characteristics. The effectiveness and reliability of the restored image are validated through four groups of experiments. The results indicate that the restored results effectively fill in the missing phenotypic features and enhance the image quality for analysis. Classification experiments， serving as a representation of cell morphology analysis experiments， are conducted on both the original and restored cell images. It is proved that the image after restoring saturated artifacts can improve the experimental accuracy based on cell morphology analysis.

Key words: fluorescence microscope images, saturation artifacts, image inpainting, deep learning, generative adversarial networks

CLC Number:

TP 391.7

Ji-hong LIU, Lü-heng ZHANG, Hai-xu YANG. A Saturation Artifact Inpainting Algorithm for Cell Fluorescence Microscopic Images[J]. Journal of Northeastern University(Natural Science), 2024, 45(7): 921-927.

Figures/Tables 7

Fig.1 Structure diagram of two stage inpainting model

Fig.2 Fluorescence microscopic images of cells

Fig.3 Example of TC-GAN restored results

Table 1 Indexes of masked and restored images

指标	数据组	均值
PSNR	伪影组	9.101
PSNR	修复组	25.948
SSIM	伪影组	0.725
SSIM	修复组	0.854
FID	伪影组	609.154
FID	修复组	50.345

Fig.4 Qualitative display of TC-GAN restored results

Table 2 Indexes of classification on SET3 and SET4

测试集	ACC	SEN	PRE
SET3	93.11	82.78	84.34
SET4	92.44	81.11	83.45

Table 3 Indexes of classification on SET5 and SET6

测试集	ACC	SEN	PRE
SET5	88.22	70.56	76.75
SET6	92.44	81.11	82.65

References 35

1	Bougen‑Zhukov N， Loh S Y， Lee H K.et al.Large‑scale image‑based screening and profiling of cellular phenotypes［J］.Cytometry：Part A，2017，91（2）：115-125.
2	Caicedo J C， Singh S， Carpenter A E.Applications in image‑based profiling of perturbations［J］.Current Opinion in Biotechnology，2016，39：134-142.
3	Ray K， Ma J， Oram M，et al.Single‑molecule and FRET fluorescence correlation spectroscopy analyses of phage DNA packaging：colocalization of packaged phage T4 DNA ends within the capsid［J］.Journal of Molecular Biology，2010，395（5）：1102-1113.
4	Beesley J， Sivakumaran H， Moradi Marjaneh M，et al. eQTL colocalization analyses identify NTN4 as a candidate breast cancer risk gene［J］.The American Journal of Human Genetics，2020，107（4）：778-787.
5	Sauvat A， Leduc M， Müller K，et al.ColocalizR：an open‑source application for cell‑based high‑throughput colocalization analysis［J］.Computers in Biology and Medicine，2019，107：227-234.
6	Ren W W， Isler H， Wolf M，et al.Smart toolkit for fluorescence tomography：simulation，reconstruction，and validation［J］.IEEE Transactions on Biomedical Engineering，2020，67（1）：16-26.
7	Wang Y， Maslov K， Kim C H，et al.Integrated photoacoustic and fluorescence confocal microscopy［J］.IEEE Transactions on Biomedical Engineering，2010，57（10）：2576-2578.
8	Ritter C， Wollmann T， Lee J Y，et al.Data fusion and smoothing for probabilistic tracking of viral structures in fluorescence microscopy images［J］.Medical Image Analysis，2021，73：102168.
9	Caicedo J C， Cooper S， Heigwer F，et al.Data‑analysis strategies for image‑based cell profiling［J］.Nature Methods，2017，14（9）：849-863.
10	Moen E， Bannon D， Kudo T，et al.Deep learning for cellular image analysis［J］.Nature Methods，2019，16（12）：1233-1246.
11	Bolte S， Cordelières F P.A guided tour into subcellular colocalization analysis in light microscopy［J］.Journal of Microscopy，2006，224（3）：213-232.
12	Aladeokin A C， Akiyama T， Kimura A，et al.Network‑guided analysis of hippocampal proteome identifies novel proteins that colocalize with A β in a mice model of early‑stage Alzheimer’s disease［J］.Neurobiology of Disease，2019，132：104603.
13	Zinchuk V， Zinchuk O， Okada T.Quantitative colocalization analysis of multicolor confocal immunofluorescence microscopy images：pushing pixels to explore biological phenomena［J］.Acta Histochemica et Cytochemica，2007，40（1）：101-111.
14	Li Y， Xu F， Zhang F，et al.DLBI：deep learning guided Bayesian inference for structure reconstruction of super‐resolution fluorescence microscopy［J］.Bioinformatics，2018，34（13）：i284-i294.
15	Ishii S， Lee S， Urakubo H，et al.Generative and discriminative model‑based approaches to microscopic image restoration and segmentation［J］.Microscopy，2020，69（2）：79-91.
16	Goodfellow I J， Pouget‑Abadie J， Mirza M，et al.Generative adversarial networks［J］.Advances in Neural Information Processing Systems，2014，3：2672-2680.
17	Tran N T， Tran V H， Nguyen N B，et al.On data augmentation for GAN training［J］.IEEE Transactions on Image Processing，2021，30：1882-1897.
18	Ren Y R， Li G， Liu S，et al.Deep spatial transformation for pose‑guided person image generation and animation［J］.IEEE Transactions on Image Processing，2020，29：8622-8635.
19	Kaneko T， Kameoka H， Hojo N，et al.Generative adversarial network‑based postfilter for statistical parametric speech synthesis［C］//2017 IEEE International Conference on Acoustics，Speech and Signal Processing （ICASSP）.New Orleans，2017：4910-4914.
20	Quan W Z， Zhang R S， Zhang Y，et al.Image inpainting with local and global refinement［J］.IEEE Transactions on Image Processing，2022，31：2405-2420.
21	Chen X J， Zhang C， Zhao J，et al.Weakly supervised neuron reconstruction from optical microscopy images with morphological priors［J］.IEEE Transactions on Medical Imaging，2021，40（11）：3205-3216.
22	Belthangady C， Royer L A.Applications，promises，and pitfalls of deep learning for fluorescence image reconstruction［J］.Nature Methods，2019，16（12）：1215-1225.
23	Boutros M， Heigwer F， Laufer C.Microscopy‑based high‑content screening［J］.Cell，2015，163（6）：1314-1325.
24	Nazeri K， Ng E， Joseph T，et al.EdgeConnect：generative image inpainting with adversarial edge learning［EB/OL］. ［2019-01-11］..
25	Canny J.A computational approach to edge detection［J］.IEEE Transactions on Pattern Analysis and Machine Intelligence，1986，8（6）：679-698.
26	Yu J H， Lin Z， Yang J M，et al.Generative image inpainting with contextual attention［C］//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City，2018：5505-5514.
27	Isola P， Zhu J Y， Zhou T H，et al.Image‑to‑image translation with conditional adversarial networks［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu，2017：5967-5976.
28	Johnson J， Alahi A， Li F F.Perceptual losses for real‑time style transfer and super‑resolution［C］//European Conference on Computer Vision.Amsterdam，2016：694-711.
29	Gatys L A， Ecker A S， Bethge M.Image style transfer using convolutional neural networks［C］//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016：2414-2423.
30	Russakovsky O， Deng J， Su H，et al.ImageNet large scale visual recognition challenge［J］.International Journal of Computer Vision，2015，115（3）：211-252.
31	Bray M A， Singh S， Han H，et al.Cell Painting，a high‑content image‑based assay for morphological profiling using multiplexed fluorescent dyes［J］.Nature Protocols，2016，11（9）：1757-1774.
32	Kingma D P， Ba J.Adam：a method for stochastic optimization［EB/OL］.［2017-01-30］..
33	Elharrouss O， Almaadeed N， Al‑Maadeed S，et al.Image inpainting：a review［J］.Neural Processing Letters，2020，51：2007-2028.
34	Heusel M， Ramsauer H， Unterthiner T，et al.GANs trained by a two time‑scale update rule converge to a local Nash equilibrium［EB/OL］.［2018-01-12］..
35	He K M， Zhang X Y， Ren S Q，et al.Deep residual learning for image recognition［C］//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016：770-778.

[1]	Hong-li LI, Hao-yu LIU, Rong-hua ZHANG, Yi CHENG. Emotional Classification Based on Multidimensional Feature Matrix and Improved Dense Connection Network [J]. Journal of Northeastern University(Natural Science), 2024, 45(7): 928-935.
[2]	Dong-hong HAN, Yan-ru KONG, Yi-meng ZHAN, Yuan LIU. Research on Emotion Recognition Method of Music Multimodal Data [J]. Journal of Northeastern University(Natural Science), 2024, 45(6): 776-785.
[3]	Wei-qi ZHANG, Hui-ming WANG. Interpretable Deep Learning Prediction Model for Compressive Strength of Concrete [J]. Journal of Northeastern University(Natural Science), 2024, 45(5): 738-744.
[4]	WEI Jian-yi， WU Jing-jing. Resource Allocation Algorithm in Industrial Internet of Things Based on Edge Computing [J]. Journal of Northeastern University(Natural Science), 2023, 44(8): 1072-1078.
[5]	LI Hai-yan， XIONG Li-chang， GUO Lei， LI Hai-jiang. Two-Stage Inpainting Algorithm Based on U-net Edge Generation and Hypergraphs Convolution [J]. Journal of Northeastern University(Natural Science), 2023, 44(3): 331-339.
[6]	YANG Xuan， HE Zhan-qi. Improved Two-layer BiLSTM Electrocardiosignal Segmentation Method [J]. Journal of Northeastern University(Natural Science), 2023, 44(12): 1705-1711.
[7]	JI Ce， WANG Xin， GENG Rong， LIANG Min-jun. LSTM-Based Channel Estimation Method in Time-Varying Channels [J]. Journal of Northeastern University(Natural Science), 2023, 44(11): 1521-1528.
[8]	CHEN Cheng， SHI Pei-xin， WANG Zhan-sheng， JIA Peng-jiao. Shield Load Prediction Method Based on Deep Learning with Multiattention Mechanism [J]. Journal of Northeastern University(Natural Science), 2023, 44(11): 1631-1638.
[9]	LI Juan-li， WEI Dai-liang， LI Bo， WEN Xiao. Improved SSD Rapid Separation Model of Coal Gangue Based on Deep Learning and Light-Weighting [J]. Journal of Northeastern University(Natural Science), 2023, 44(10): 1474-1480.
[10]	ZHAO Yong， JIAO Shi-hui， ZHAO Qian-bai. Hybrid Recognition Model of Microseismic Signals for Mining Based on Mel Spectrum and LSTM-DCNN [J]. Journal of Northeastern University(Natural Science), 2023, 44(10): 1481-1489.
[11]	ZHANG Xue-feng， WANG Zhao-yi. Automatic Lane Change Decision Model Based on Dueling Double Deep Q-network [J]. Journal of Northeastern University(Natural Science), 2023, 44(10): 1369-1376.
[12]	WANG Ying， WANG Ze-hao， LI Hong， HUANG Wen-jun. Named Entity Recognition in Threat Intelligence Domain Based on Deep Learning [J]. Journal of Northeastern University(Natural Science), 2023, 44(1): 33-39.
[13]	GU De-ying， LUO Yu-lun， LI Wen-chao. Traffic Target Detection in Complex Scenes Based on Improved YOLOv5 Algorithm [J]. Journal of Northeastern University(Natural Science), 2022, 43(8): 1073-1079.
[14]	DAI Yin， LIU Wei-bin， DONG Xin-yang， SONG Yu-meng. U-Net CSF Cells Segmentation Based on Attention Mechanism [J]. Journal of Northeastern University(Natural Science), 2022, 43(7): 944-950.
[15]	LI Hong-ru， REN Zi-yang， HUANG You-he， YU Xia. Recognition Method of Arrhythmia Based on Variable Weight Singular Spectrum Analysis [J]. Journal of Northeastern University(Natural Science), 2022, 43(3): 305-312.