Wrong corners detected from masks through Mask-R CNN

  mask, object-detection, python

I have an application where I trained a model using Mask-R CNN to identify windows and doors in images of houses. The model predicts objects in unseen images online and on these predictions we try to fit new designs of windows and doors using a custom built front end. To fit new designs onto the detected masks, we try to detect four corners of the said mask. These new calculated points would be the corners for the uploaded designs using a front end called as configurator.

Below is the code used to calculate the corners of all the instances (masks) predicted within an image.

import cv2
import numpy as np

r = results[0]  # r now contains all the masks predicted
mask = r['masks'].astype(np.float32)

for i in range(len(r['rois'])):
    if r['scores'][i] > 0.95:
        find_corners(mask[:, :, i], mask.shape)

def simplify_contour(contour, n_corners=4):
    Binary searches best `epsilon` value to force contour
        approximation contain exactly `n_corners` points.

    :param contour: OpenCV2 contour.
    :param n_corners: Number of corners (points) the contour must contain.

    :returns: Simplified contour in successful case. Otherwise returns initial contour.

    n_iter, max_iter = 0, 100
    lb, ub = 0., 1.

    while True:
        n_iter += 1
        if n_iter > max_iter:
            return contour

        k = (lb + ub)/2.
        eps = k*cv2.arcLength(contour, True)
        approx = cv2.approxPolyDP(contour, eps, True)

        if len(approx) > n_corners:
            lb = (lb + ub)/2.
        elif len(approx) < n_corners:
            ub = (lb + ub)/2.
            return approx

def find_corners(mask, size):
    Returns the 4 corners for the given door or window mask
    The points are normalized to be between 0..1
    The order must be
    d c
    a b
    -> [a, b, c, d]
    where the coordinate 0, 0 is the lower left corner.
    :param mask:  float32 mask image
    :param size: tuple (x, y) the original image size
    :return: [{x: float, y: float}] point array

    m = np.uint8(mask)
    # thresh = cv2.Canny(m, 0, 1)

    contours, h = cv2.findContours(m, 1, 2)

    for cnt in contours:
        approx = simplify_contour(cnt, 4)
        if len(approx) == 4:
            p = [x[0] for x in approx]
            c = order_by_rows(p)
            return [{'x': float(x[0]) / size[1], 'y': float(x[1]) / size[0]} for x in c]

    return None

def order_by_rows(points):
    Orders the given points counter clockwise around their center
    Only works because there are 4 entries
    :param points: 4 points
    :return:  4 points ordered
    c = sorted(points, key=cmp_to_key(compare))

    if c[0][0] > c[1][0]:
        tmp = c[0]
        c[0] = c[1]
        c[1] = tmp

    if c[2][0] < c[3][0]:
        tmp = c[2]
        c[2] = c[3]
        c[3] = tmp

    return c

def compare(a, b):
    if a[1] < b[1]:
        return -1
    if a[1] > b[1]:
        return 1
    return 0

The problem is that when the new designs are fitted onto these corner points, majority of them are a bit skewed, leaving the object of interest partially uncovered. Following are the images of the mask predicted and the image from the configurator front end when trying to re-design windows.

enter image description here

enter image description here

Source: Python Questions