What is a good approach in case of image classification problem. I am trying to classify two similar shapes - let's say a circle and an oval, in some case the confidence for the oval is very high for the circle input.

Have you looked at VNDetectContourDetection? Using this traditional computer vision approach might give you better results.

Tagged with: