Java – How to draw a rectangle using MatOfKeyPoint for text detection | .java

How to draw a rectangle using MatOfKeyPoint for text detection | .java… here is a solution to the problem.

How to draw a rectangle using MatOfKeyPoint for text detection | .java

I’m using OpenCV4Android for real-time text detection and recognition. The identification part is all complete. However, I had to ask questions about text detection. I’m using MSER FeatureDetector to detect text.

This is the Live and Call Methods section:

public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
    carrierMat = inputFrame.gray();
    carrierMat = General.MSER(carrierMat);
    return carrierMat;
}

This is the basic MSER implementation:

private static FeatureDetector fd = FeatureDetector.create(FeatureDetector.MSER);
private static MatOfKeyPoint mokp = new MatOfKeyPoint();
private static Mat edges = new Mat();

public static Mat MSER(Mat mat) {
    for mask
    Imgproc.Canny(mat, edges, 400, 450);
    fd.detect(mat, mokp, edges);
    for drawing keypoints
    Features2d.drawKeypoints(mat, mokp, mat);
    return mat;
}

It works well for finding text with edge masks.

I want to draw a rectangle for the cluster like this:

enter image description here

Or this:

enter image description here

You can assume that my point is correct.

As you can see, the fd.detect() method returns a MatOfKeyPoint. So I tried this method of drawing rectangles :

public static Mat MSER_(Mat mat) {
    fd.detect(mat, mokp);
    KeyPoint[] refKp = mokp.toArray();
    Point[] refPts = new Point[refKp.length];

for (int i = 0; i < refKp.length; i++) {
        refPts[i] = refKp[i].pt;
    }
    MatOfPoint2f refMatPt = new MatOfPoint2f(refPts);
    MatOfPoint2f approxCurve = new MatOfPoint2f();

Processing on mMOP2f1 which is in type MatOfPoint2f
    double approxDistance = Imgproc.arcLength(refMatPt, true) * 0.02;
    Imgproc.approxPolyDP(refMatPt, approxCurve, approxDistance, true);

Convert back to MatOfPoint
    MatOfPoint points = new MatOfPoint(approxCurve.toArray());
     Get bounding rect
    Rect rect = Imgproc.boundingRect(points);
     draw enclosing rectangle (all same color, but you could use variable i to make them unique)
    Imgproc.rectangle(mat, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), Detect_Color_, 5);
    Features2d.drawKeypoints(mat, mokp, mat);
    return mat;
}

But when I tried the Imgproc.arcLength() method, it stopped abruptly. I provided a random approxDistance value for the Imgproc.approxPolyDP() method, e.g. 0.1, which doesn’t work very efficiently.

So how do you draw a rectangle for detected text?

Solution

I tested your code and ran into the exact same issue.
Now I still can’t find the problem.
But I found a project that uses both “MSER” and “morphology”.
You can find ithere

The structure of the project is very simple, the author will
Text detection in the “onCameraFrame” method is just like you.
I implemented the method of that project and it worked,
But the results are still not very good.

If you’re looking for a better text detection tool, here are two.

  1. Stroke Width Transform (SWT):
    A whole new way to find a text area. It’s fast and efficient. But it only works with C++ or Python. You can find some examples< a href="https://github.com/aperrau/DetectText" rel="noreferrer noopener nofollow">here

  2. Class-specific extremum regions using ERFilter-class: Advanced version of MSER. Unfortunately, it is only available in OpenCV 3.0.0-dev. You cannot use it in the current version of OpenCV4Android. The file ishere

Honestly, I’m new to this (2 months), but I hope this information helps you with your project.

(Updated: 2015/9/13)
I translated a C++ method from post.
It’s much better than the first GitHub project I mentioned.
Here is the code:

public void apply(Mat src, Mat dst) {
    if (dst != src) {
        src.copyTo(dst);
    }
    Mat img_gray,img_sobel, img_threshold, element;

img_gray=new Mat();
    Imgproc.cvtColor(src, img_gray, Imgproc.COLOR_RGB2GRAY);

img_sobel=new Mat();
    Imgproc.Sobel(img_gray, img_sobel, CvType.CV_8U, 1, 0, 3, 1, 0,Core.BORDER_DEFAULT);

img_threshold=new Mat();
    Imgproc.threshold(img_sobel, img_threshold, 0, 255, Imgproc.THRESH_OTSU+Imgproc.THRESH_BINARY);

element=new Mat();
    element = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(17, 3) );
    Imgproc.morphologyEx(img_threshold, img_threshold, Imgproc.MORPH_CLOSE, element);
    Does the trick
    List<MatOfPoint>  contours=new ArrayList<MatOfPoint>();
    Mat hierarchy = new Mat();
    Imgproc.findContours(img_threshold, contours, hierarchy, 0, 1);
    List<MatOfPoint> contours_poly=new ArrayList<MatOfPoint>(contours.size());
    contours_poly.addAll(contours);

MatOfPoint2f mMOP2f1,mMOP2f2;
    mMOP2f1=new MatOfPoint2f();
    mMOP2f2=new MatOfPoint2f();

for( int i = 0; i < contours.size(); i++ )

if (contours.get(i).toList().size()>100)
        { 
            contours.get(i).convertTo(mMOP2f1, CvType.CV_32FC2);
            Imgproc.approxPolyDP(mMOP2f1,mMOP2f2, 3, true );
            mMOP2f2.convertTo(contours_poly.get(i), CvType.CV_32S);
            Rect appRect=Imgproc.boundingRect(contours_poly.get(i));
            if (appRect.width>appRect.height) 
            {
                Imgproc.rectangle(dst, new Point(appRect.x,appRect.y) ,new Point(appRect.x+appRect.width,appRect.y+appRect.height), new Scalar(255,0,0));
            }
        }   

}

Related Problems and Solutions