Python – Solving PnP object_points/image_points shapes?

I’m trying to call solvePnP with the following parameters.

apriltag_object_points = np.array([(-1, -1, 0), (-1, 1, 0), (1, 1, 0), (1, -1, 0)], dtype=np.float)

camera_matrix_left = np.eye(3)

dist_left = np.zeros((5, 1))

image_points = np.array(detection.position, dtype=np.float)

cv2.solvePnPRansac(apriltag_object_points, image_points, camera_matrix_left, dist_left)

However, I get the following error:

OpenCV Error: Assertion failed (CV_IS_MAT(_src) && CV_IS_MAT(_dst) && (_src->rows == 1 || _src->cols == 1) && (_dst->rows == 1 || _dst-> cols == 1) && _src->cols + _src->rows - 1 == _dst->rows + _dst->cols - 1 && (CV_MAT_TYPE(_src->type) == CV_32FC2 || CV_MAT_TYPE(_src->type) == CV_64FC2) && (CV_MAT_TYPE(_dst->type) == CV_32FC2 || CV_MAT_TYPE(_dst->type) == CV_64FC2)) in cvUndistortPoints, file /Users/travis/miniconda3/conda-bld/opencv_1506476120161/work/opencv-3.3.0/modules/imgproc/src/ undistort.cpp, line 312
Traceback (most recent call last):
  File "/Users/me/Documents/Code/project/file.py", line 139, in <module>
    ret = cv2.solvePnPRansac(apriltag_object_points, image_points, camera_matrix_left, dist_left)
cv2.error: /Users/travis/miniconda3/conda-bld/opencv_1506476120161/work/opencv-3.3.0/modules/imgproc/src/undistort.cpp:312: error: (-215) CV_IS_MAT(_src) && CV_IS_ MAT(_dst) && (_src->rows == 1 || _src->cols == 1) && (_dst->rows == 1 || _dst->cols == 1) && _src->cols + _src->rows - 1 == _dst->rows + _dst-> cols - 1 && (CV_MAT_TYPE(_src->type) == CV_32FC2 || CV_MAT_TYPE(_src->type) == CV_64FC2) && (CV_MAT_TYPE(_dst->type) == CV_32FC2 || CV_MAT_TYPE(_dst->type) == CV_64FC2) in function cvUndistortPoints

It looks like there is a problem with the form of my arguments, but they seem to be okay….

apriltag_object_points.shape == (4, 3)
image_points.shape == (4, 2)

Do image points have to be homogeneous? Do I only need column vectors from hstack 1 to image_points?

The documentation error for this function is implemented in Python. solvePnP( ) and solvePnPRansac() Two states:

Parameters:

objectPoints – Array of object points in the object coordinate space, 3xN/Nx3 1-channel or 1xN/Nx1 3-channel, where N is the number of points. vector<Point3f> can also pass here

imagePoints – Array of corresponding image points, 2xN/Nx2 1-channel or 1xN/Nx1 2-channel, where N is the number of points. vector<Point2f> can also pass here

So, you

assume that you can use (N, 3) and (N, 2) arrays well as input. However, the error code states otherwise. Let’s break it down:

CV_IS_MAT(_src) && CV_IS_MAT(_dst) && (_src->rows == 1 || _src->cols == 1) && (_dst->rows == 1 || _dst->cols == 1) && _src->cols + _src- >rows – 1 == _dst->rows + _dst->cols – 1 && (CV_MAT_TYPE(_src->type) == CV_32FC2 || CV_MAT_TYPE(_src->type) == CV_64FC2) && (CV_MAT_TYPE(_dst->type) == CV_32FC2 || CV_MAT_TYPE(_dst->type) == CV_64FC2)

Ok, they are numpy arrays, so we can discard the first two. But in the next check, we saw something interesting :

(_src->rows == 1 || _src->cols == 1) && (_dst-> rows == 1 || _dst->cols == 1).

It is checking if there is only one row or column and dst in the src. In other words, it strictly expects your point of view in the second version declared in the docs; It wants a multi-channel point array. This means that the first coordinate is on one channel of the matrix, the second coordinate is on the next channel, and so on.

So if we just reshape your point as a multi-channel array:

>>> apriltag_object_points = apriltag_object_points.reshape(4,1,3)
>>> image_points = image_points.reshape(4,1,2)
>>> it_works, rvec, tvec, inliers = cv2.solvePnPRansac(apriltag_object_points, image_points, camera_matrix_left, dist_left)
>>> it_works
True

Effective!

Pro tip for future: Each function of OpenCV allows points to be formatted in (npoints, ncoords), and it also accepts their format AFAIK in (npoints, 1, ncoords). However, some functions only work with the latter format. So, if you use point coordinates in OpenCV in Python, it’s best to assume that your points should be in a multi-channel array, where the number of channels is the number of axes.