Journal of Reproduction & Infertility

Journal of Reproduction & Infertility

A Homogeneous Ensemble of Robust Pre-defined Neural Network Enables Automated Annotation of Human Embryo Morphokinetics

Authors
Gunawan B Danardono 1
Alva Erwin 1
James Purnama 2
Nining Handayani 3
Arie A Polim 4
Arief Boediono 5
Ivan Sini 3
1 IRSI Research and Training Center, Jakarta, Indonesia; Faculty of Engineering and Information Technology, Swiss German University, Tangerang, Indonesia
2 Faculty of Engineering and Information Technology, Swiss German University, Tangerang, Indonesia
3 IRSI Research and Training Center, Jakarta, Indonesia; Morula IVF Jakarta Clinic, Jakarta, Indonesia
4 IRSI Research and Training Center, Jakarta, Indonesia; Morula IVF Jakarta Clinic, Jakarta, Indonesia; Department of Obstetrics and Gynecology, School of Medicine and Health Sciences, Atma Jaya Catholic University of Indonesia, Jak
5 IRSI Research and Training Center, Jakarta, Indonesia; Morula IVF Jakarta Clinic, Jakarta, Indonesia; Department of Anatomy, Physiology and Pharmacology, IPB University, Bogor, Indonesia
10.18502/jri.v23i4.10809
Abstract
Background: The purpose of the current study was to reduce the risk of human bias in assessing embryos by automatically annotating embryonic development based on their morphological changes at specified time-points with convolutional neural network (CNN) and artificial intelligence (AI). Methods: Time-lapse videos of embryo development were manually annotated by the embryologist and extracted for use as a supervised dataset, where the data were split into 14 unique classifications based on morphological differences. A compilation of homogeneous pre-trained CNN models obtained via TensorFlow Hub was tested with various hyperparameters on a controlled environment using transfer learning to create a new model. Subsequently, the performances of the AI models in correctly annotating embryo morphologies within the 14 designated classifications were compared with a collection of AI models with different built-in configurations so as to derive a model with the highest accuracy. Results: Eventually, an AI model with a specific configuration and an accuracy score of 67.68% was obtained, capable of predicting the embryo developmental stages (t1, t2, t3, t4, t5, t6, t7, t8, t9+, tCompaction, tM, tSB, tB, tEB). Conclusion: Currently, the technology and research of artificial intelligence and machine learning in the medical field have significantly and continuingly progressed in an effort to develop computer-assisted technology which could potentially increase the efficiency and accuracy of medical personnel’s performance. Nonetheless, building AI models with larger data is required to properly increase AI model reliability.
Keywords
Artificial intelligence
automation
computer
assisted image processing
Embryonic development
In vitro fertilization
Machine Learning
Neural Networks
  1. Kanakasabapathy MK, Thirumalaraju P, Bormann CL, Kandula H, Dimitriadis I, Souter I, et al. Development and evaluation of inexpensive automated deep learning-based imaging systems for embryology. Lab Chip. 2019;19(24):4139-45.
  2. Chavez-Badiola A, Flores-Saiffe-Farías A, Mendizabal-Ruiz G, Drakeley AJ, Cohen J. Embryo ranking intelligent classification algorithm (ERICA): artificial intelligence clinical assistant predicting embryo ploidy and implantation. Reprode Biomed Online. 2020;41(4):585-93.
  3. Lee CI, Su YR, Chen CH, Chang TA, Kuo EE, Zheng WL, et al. End-to-end deep learning for recognition of ploidy status using time-lapse videos. J Assist Reprod Genet. 2021;38(7):1655-63.
  4. Huang B, Tan W, Li Z, Jin L. An artificial intelligence model (euploid prediction algorithm) can predict embryo ploidy status based on time-lapse data. Reprod Biol Endocrinol. 2021;19(1):185.
  5. Kanakasabapathy MK, Thirumalaraju P, Bormann CL, Gupta R, Pooniwala R, Kandula H, et al. Deep learning mediated single time-point image-based prediction of embryo developmental outcome at the cleavage stage. arXiv preprint arXiv:2006.08346. 2020 May 21.
  6. Huang B, Zheng S, Ma B, Yang Y, Zhang S, Jin L. Using deep learning to predict the outcome of live birth from more than 10,000 embryo data. BMC Pregnancy Childbirth. 2022;22(1):36.
  7. Miyagi Y, Habara T, Hirata R, Hayashi N. Feasibility of artificial intelligence for predicting live birth without aneuploidy from a blastocyst image. Reprod Med Biol. 2019;18(2):204-11.
  8. Malmsten J, Zaninovic N, Zhan Q, Rosenwaks Z, Shan J. Automated cell stage predictions in early mouse and human embryos using convolutional neural networks. In2019 IEEE EMBS international conference on biomedical & health informatics (BHI) 2019 May 19 (pp. 1-4). IEEE.
  9. Liu Z, Huang B, Cui Y, Xu Y, Zhang B, Zhu L, et al. Multi-task deep learning with dynamic programming for embryo early development stage classification from time-lapse videos. IEEE Access. 2019;7:122153-63.
  10. Khan A, Gould S, Salzmann M. Deep convolutional neural networks for human embryonic cell counting. InEuropean conference on computer vision 2016 Oct 8 (pp. 339-348). Springer, Cham.
  11. Gingold JA, Ng NH, McAuley J, Lipton Z, Desai N. Predicting embryo morphokinetic annotations from time-lapse videos using convolutional neural networks. Fertil Steril. 2018;110(4):e220.
  12. Thirumalaraju P, Kanakasabapathy MK, Bormann CL, Gupta R, Pooniwala R, Kandula H, et al. Evaluation of deep convolutional neural networks in classifying human embryo images based on their morphological quality. Heliyon. 2021;7(2):e06298.
  13. VerMilyea M, Hall JM, Diakiw SM, Johnston A, Nguyen T, Perugini D, et al. Development of an artificial intelligence-based assessment model for prediction of embryo viability using static images captured by optical light microscopy during IVF. Hum Reprod. 2020;35(4):770-84.
  14. Chen TJ, Zheng WL, Liu CH, Huang I, Lai HH, Liu M. Using deep learning with large dataset of microscope images to develop an automated embryo grading system. Fertil Reprod. 2019;1(1):51-6.
  15. Louis CM, Erwin A, Galinium M. Automated annotation of in vtrio fertilization time-lapse using deep neural network [dissertation]: Swiss German University; 2020. 105 p
  16. Palermo GD, O’Neill CL, Chow S, Cheung S, Parrella A, Pereira N, et al. Intracytoplasmic sperm injection: state of the art in humans. Reproduction. 2017;154(6):F93-F110.
  17. Pribenszky C, Nilselid AM, Montag M. Time-lapse culture with morphokinetic embryo selection improves pregnancy and live birth chances and reduces early pregnancy loss: a meta-analysis. Reprod Biomed Online. 2017;35(5):511-20.
  18. Raudonis V, Paulauskaite-Taraseviciene A, Sutiene K, Jonaitis D. Towards the automation of early-stage human embryo development detection. Biomed Eng Online. 2019;18(1):120.
  19. O'Shea K, Nash R. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458. 2015 Nov 26.
  20. Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, et al. Recent advances in convolutional neural networks. Pattern Recognition. 2018;77:354-77.
  21. Albawi S, Mohammed TA, Al-Zawi S. Understanding of a convolutional neural network. In2017 international conference on engineering and technology (ICET) 2017 Aug 21 (pp. 1-6). Ieee.
  22. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467. 2016 Mar 14.
  23. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. {TensorFlow}: A system for {Large-Scale} machine learning. In12th USENIX symposium on operating systems design and implementation (OSDI 16) 2016 (pp. 265-283).
  24. Dillon JV, Langmore I, Tran D, Brevdo E, Vasudevan S, Moore D, Patton B, Alemi A, Hoffman M, Saurous RA. Tensorflow distributions. arXiv preprint arXiv:1711.10604. 2017 Nov 28.
  25. Chollet F, others. Keras [Internet]. San Francisco: GitHub; 2015 [cited 2021 Apr 16]. Available from: https://github.com/fchollet/keras
Journal of Reproduction & Infertility
Volume 23, Issue 4
Autumn 2022
Pages 250-256