Image segmentation with prior guidance: application to cervical cytology images

Author: Ratna Saha

Saha, Ratna, 2020 Image segmentation with prior guidance: application to cervical cytology images, Flinders University, College of Science and Engineering

Terms of Use: This electronic version is (or will be) made publicly available by Flinders University in accordance with its open access policy for student theses. Copyright in this thesis remains with the author. You may use this material for uses permitted under the Copyright Act 1968. If you are the owner of any included third party copyright material and/or you believe that any material has been made available without permission of the copyright owner please contact with the details.


Cervical cancer is one of the most common type of gynecological cancer. In cervix tissues, pre-cancerous changes in cell morphology and structure develop over a span of 8 to 10 years. Pap smear is the most effective and popular screening worldwide for early detection of invasive cervical cancer. However, manual screening task of cytotechnologist is complex, time-intensive, tedious, and requires high level of expertise. Computer-aided screening of cervical smear images has the potential to assist the cytotechnologists and improve Pap test outcome, in a turn decrease cervical cancer morbidity and mortality rate. Nucleus features are crucial for characterization of cervical cytology images and diagnostic decisions. Hence, accurate segmentation of nuclei is the essential primary step towards computer-assisted cervical cell analysis. This study aims to improve cervical nuclei segmentation performance, and subsequently develop an abnormality detection or cervical cell classification framework exclusively based on nuclei features.

The main contributions of this thesis are the development of three novel nucleus segmentation techniques and a nuclei feature based classification framework. The first two segmentation frameworks are developed on pixel grid level and the last one on superpixel based image representation. Incorporation of prior knowledge in image segmentation techniques is useful in partitioning images with noise and low contrast. The proposed segmentation algorithms are designed to utilize prior guidance, for precise segmentation of cervical nuclei from overlapping Pap smear images.

Particularly, in the first contribution, a circular shape function is proposed at pixel level, utilizing spatial location of pixels. This shape function was integrated into fuzzy clustering technique. In the second contribution, circular shape guidance is included in efficient graph based segmentation algorithm. The adaptive shape guidance is designed at pixel level to include circularity measure of regions, in addition to image intensity information. In the third contribution, a gradient guided superpixel level merging framework is proposed. The novel merging criterion consists of pairwise regional contrast and gradient boundary guidance. Superpixels produced by SLIC and SRM segmentation techniques are employed in this merging framework. Lastly, a nuclei feature based classification framework is proposed for abnormality detection in cervical cells, where nuclei are segmented using all three proposed segmentation techniques.

The effectiveness of the proposed segmentation approaches is validated on ISBI 2014 and Herlev datasets, using Dice similarity coefficient, pixel and object based precision and recall as performance measures. The experimental results indicate that the proposed segmentation techniques can precisely segment nuclei from overlapping cervical cytology images, while keeping high level of precision and recall.

Performance of the proposed classification framework is validated on Herlev dataset, and evaluated using accuracy, sensitivity, specificity, AUC score, and Spearman rank order coefficient. Promising classification results of the proposed framework indicate that, in the presence of precisely segmented nuclei boundaries, it is possible to characterize cervical cells using only nuclei features.

Keywords: Nucleus segmentation, Cervical cell classification, Pap smear, Abnormality detection, Prior guided segmentation, Statistical region merging, Pairwise regional contrast, Gradient boundary, Circular shape prior, Graph based image segmentation, SLIC, Fuzzy clustering, Circular shape function

Subject: Computer Science thesis

Thesis type: Doctor of Philosophy
Completed: 2020
School: College of Science and Engineering
Supervisor: Dr Mariusz Bajger