Automatic medical image segmentation is an emerging field with upcoming new techniques that revolutionized how we view functional and pathological events in the body. Medical image segmentation is a very challenging problem and there is no standard segmentation technique that can automatically segment all types of three-dimensional medical images. Doctors and clinicians still prefer manual segmentation due to unreliability and unavailability of standard automatic segmentation techniques. Most of the segmentation algorithms are semi-automatic that require user interaction and are difficult for use in practical applications. Some of the algorithms that are automatic require very high resolution of images for segmentation. Moreover, segmentation algorithms for medical images are application specific and the algorithms developed for one application may not work for other type of application. There are a number of factors such as image noise, anatomy variation, disease type, intensity homogeneity, non-uniform object texture, image content, occlusion, input nature and special characteristics of image continuity that make the process of automatic segmentation more difficult and challenging. In this paper, we have categorized these challenges and have described their effects on commonly used segmentation algorithms using the criterion functions input type, dimensionality, anatomy variation, parameter tuning and need of user interaction.