Overlapping Cervical Nuclei Separation using Watershed Transformation and Elliptical Approach in Pap Smear Images

Izzati Muhimmah, Rahadian Kurniawan, Indrayanti Indrayanti


In this study, a robust method is proposed for accurately separating overlapping cell nuclei in cervical microscopic images. This method is based on watershed transformation and an elliptical approach. Since the watershed transformation process of taking the initial seed is done manually, the method was developed to obtain the initial seed automatically. Total initial seeds at this stage represents the number of nuclei that exist in the image of a pap smear, either overlapping or not. Furthermore, a method was developed based on an elliptical approach to obtain the area of each of the nuclei automatically. This method can successfully separate several (more than two) clustered cell nuclei. In addition, the proposed method was evaluated by experts and was proven to have better results than methods from previous studies in terms of accuracy and execution time. The proposed method can determine overlapping and non-overlapping boundaries of nuclei fast and accurately. The proposed method provides better decision-making on areas with overlapping nuclei and can help to improve the accuracy of image analysis and avoid information loss during the process of image segmentation.


elliptical approach; gynecological cytology; microscopic image analysis; overlapping nuclei; Pap smear; watershed-based segmentation.

Full Text:



Bruni, L., Barrionuevo-Rosas, L., Albero, G., Serrano, B., Mena, M., Gómez, D., Muñoz, J., Bosch, FX. & Sanjosé., S. De, Human Papillomavirus and Related Diseases in Indonesia. Summary Report 27 July 2017, ICO Information Centre on HPV and Cancer (HPV Information Centre, http://www.hpvcentre.net/statistics/reports/IDN.pdf, (15-Apr-2017).

Nauth, H.F., Gynecological Cytology, Stuttgart, Germany, Georg Theime Verlag, 2007.

Plissiti, M.E., Nikou, C. & Charchanti, A., Automated Detection of Cell Nuclei in Pap Smear Images Using Morphological Reconstruction and Clustering, IEEE Transactions on Information Technology in Biomedicine, 15(2), pp. 233-241, Mar. 2011.

Lin, C.H., Chan, Y.K. & Chen, C.C., Detection and Segmentation of Cervical Cell Cytoplast and Nucleus, International Journal of Imaging Systems and Technology, 19(3), pp. 260-270, Aug. 2009.

Yang-Mao, S.F., Chan, Y.K. & Chu, Y.P., Edge Enhancement Nucleus and Cytoplast Contour Detector of Cervical Smear Images, IEEE Transactions on Systems, Man, and Cybernetics, 38(2), pp. 353-366, Apr. 2008.

Bak, E., Najarian, K. & Brockway, J.P., Efficient Segmentation Framework of Cell Images in Noise Environments, IEEE International Conference of the Engineering in Medicine and Biology Society, pp. 1802-1805, 2004.

Garrido, A. & de la., Blanca, N.P., Applying Deformable Templates for Cell Image Segmentation, Pattern Recognition, 33 (5), pp. 821-832, May 2000.

Plissiti, M.E. & Nikou, C., Cervical Cell Classification Based Exclusively on Nucleus Features, International Conference on Image Analysis and Recognition, pp. 483-490, 2012.

Muhimmah, I., Kurniawan, R. & Indrayanti, Analysis of Features to Distinguish Epithelial Cells and Inflammatory Cells in Pap Smear Images, IEEE 2013 6th International Conference on BioMedical Engineering and Informatics, pp. 519-523, 2013.

Kurniawan, R., Sasmito, D.E.K. & Suryani, F., Feature Extraction and Selection for Classification of Epitel Cells with Inflammatory Cells on Pap Smear Imagery,, National Seminar on Information Technology Applications, pp. 23-28, 2013. (Text in Indonesian)

Huang, P.C., Chan, Y.K., Chan, P.C., Chen, Y.F., Chen, R.C. & Huang, Y.R., Quantitative Assessment of Pap Smear Cells by PC-based Cytopathologic Image Analysis System and Support Vector Machine, International Conference on Medical Biometrics, pp. 192-199, 2007.

Kim, K.B., Kim, S. & Sim, K.B., Nucleus Classification and Recognition of Uterine Cervical Pap-smears using Fuzzy Art Algorithm, Asia-Pacific Conference on Simulated Evolution and Learning, pp. 560-567, 2006.

Kanafiah, S.N.A.M., Jusman, Y., Isa, N.A.M. & Mohamed, Z., Radial-Based Cell Formation Algorithm for Separation of Overlapping Cells in Medical Microscopic Images, Procedia Computer Science, pp. 123-132, 2015.

Zimmer, C. & Olivo-Marin, J.C., Coupled Parametric Active Contours, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(11), pp. 1838-1842, Nov. 2005.

Zhang, B., Zimmer, C. & Olivo-Marin, J.C., Tracking Fluorescent Cells with Coupled Geometric Active Contours, IEEE International Symposium on Biomedical Imaging: Nano to Macro, pp. 476-479, 2004.

Wu, H.S., Gil, J. & Barba, J., Optimal Segmentation of Cell Images, Vision, Image and Signal Processing, 145(1), pp. 50-56, Feb. 1998.

Jung, C. & Kim, C., Segmenting Clustered Nuclei Using H-minima Transform-Based Marker Extraction and Contour Parameterization, IEEE Transaction on Biomedical Engineering, 57(10), pp. 2600-2604, Oct. 2010.

Plissiti, M.E., Louka, E. & Nikou, C., Splitting of Overlapping Nuclei Guided by Robust Combinations of Concavity Points, SPIE Medical Imaging, pp. 903431-903431, 2014.

Yang, X., Li, H. & Zhou, X., Nuclei Segmentation Using Marker-Controlled Watershed, Tracking Using Mean-Shift, and Kalman Filter in Time-Lapse Microscopy, IEEE Transactions on Circuits and Systems, 53(11), pp. 2405-2414, Nov. 2006.

Muhimmah, I. & Kurniawan, R., Shape-based Nuclei Area of Digitized Pap smear Images, International Conference on Digital Image Processing, pp. p. 83344J-83344J-5, 2012.

DOI: http://dx.doi.org/10.5614%2Fitbj.ict.res.appl.2017.11.3.1


  • There are currently no refbacks.

Contact Information:

ITB Journal Publisher, LPPM – ITB, 

Center for Research and Community Services (CRCS) Building Floor 7th, 
Jl. Ganesha No. 10 Bandung 40132, Indonesia,

Tel. +62-22-86010080,

Fax.: +62-22-86010051;

e-mail: jictra@lppm.itb.ac.id.