与OpenCV的硬币模板匹配

Question

我正在进行一个项目,它将自动计算输入图像中的硬币值.到目前为止,我已经使用边缘检测和使用霍夫变换进行了一些预处理来分割硬币.

我的问题是如何从这里开始？我需要根据以前存储的一些功能对分割的图像进行一些模板匹配.我怎么能这样做呢.

我还读过一些叫做K-Nearest Neighbors的东西,我觉得这是我应该使用的东西.但我不太清楚如何使用它.

我遵循的研究文章:

• 硬币探测器

• 硬币识别

Answer 1

进行模式匹配的一种方法是使用cv :: matchTemplate.

这需要输入图像和作为模板的较小图像.它将模板与重叠图像区域进行比较,计算模板与重叠区域的相似性.有几种计算比较的方法.
此方法不直接支持比例或方向不变性.但是可以通过将候选者缩放到参考大小并通过针对几个旋转模板进行测试来克服这一点.

示出了该技术的详细示例以检测50c硬币的存在和位置.相同的程序可以应用于其他硬币.
将建立两个计划.一个是从50c硬币的大图像模板创建模板.另一个将输入这些模板以及带有硬币的图像作为输入,并将输出50c硬币被标记的图像.

模板制作者

#define TEMPLATE_IMG "50c.jpg"
#define ANGLE_STEP 30
int main()
{
    cv::Mat image = loadImage(TEMPLATE_IMG);
    cv::Mat mask = createMask( image );
    cv::Mat loc = locate( mask );
    cv::Mat imageCS;
    cv::Mat maskCS;
    centerAndScale( image, mask, loc, imageCS, maskCS);
    saveRotatedTemplates( imageCS, maskCS, ANGLE_STEP );
    return 0;
}

Here we load the image which will be used to construct our templates.
Segment it to create a mask.
Locate the center of masses of said mask.
And we rescale and copy that mask and the coin so that they ocupy a square of fixed size where the edges of the square are touching the circunference of mask and coin. That is, the side of the square has the same lenght in pixels as the diameter of the scaled mask or coin image.
Finally we save that scaled and centered image of the coin. And we save further copies of it rotated in fixed angle increments.

cv::Mat loadImage(const char* name)
{
    cv::Mat image;
    image = cv::imread(name);
    if ( image.data==NULL || image.channels()!=3 )
    {
        std::cout << name << " could not be read or is not correct." << std::endl;
        exit(1);
    }
    return image;
}

loadImage uses cv::imread to read the image. Verifies that data has been read and the image has three channels and returns the read image.

#define THRESHOLD_BLUE  130
#define THRESHOLD_TYPE_BLUE  cv::THRESH_BINARY_INV
#define THRESHOLD_GREEN 230
#define THRESHOLD_TYPE_GREEN cv::THRESH_BINARY_INV
#define THRESHOLD_RED   140
#define THRESHOLD_TYPE_RED   cv::THRESH_BINARY
#define CLOSE_ITERATIONS 5
cv::Mat createMask(const cv::Mat& image)
{
    cv::Mat channels[3];
    cv::split( image, channels);
    cv::Mat mask[3];
    cv::threshold( channels[0], mask[0], THRESHOLD_BLUE , 255, THRESHOLD_TYPE_BLUE );
    cv::threshold( channels[1], mask[1], THRESHOLD_GREEN, 255, THRESHOLD_TYPE_GREEN );
    cv::threshold( channels[2], mask[2], THRESHOLD_RED  , 255, THRESHOLD_TYPE_RED );
    cv::Mat compositeMask;
    cv::bitwise_and( mask[0], mask[1], compositeMask);
    cv::bitwise_and( compositeMask, mask[2], compositeMask);
    cv::morphologyEx(compositeMask, compositeMask, cv::MORPH_CLOSE,
            cv::Mat(), cv::Point(-1, -1), CLOSE_ITERATIONS );

    /// Next three lines only for debugging, may be removed
    cv::Mat filtered;
    image.copyTo( filtered, compositeMask );
    cv::imwrite( "filtered.jpg", filtered);

    return compositeMask;
}

createMask进行模板的分割.它将每个BGR通道二值化,对这三个二值化图像进行AND,并执行CLOSE形态操作以生成掩模.
三个调试行使用计算的掩码作为复制操作的掩码将原始图像复制为黑色图像.这有助于为阈值选择合适的值.

在这里,我们可以看到由创建的掩码过滤的50c图像 createMask

cv::Mat locate( const cv::Mat& mask )
{
  // Compute center and radius.
  cv::Moments moments = cv::moments( mask, true);
  float area = moments.m00;
  float radius = sqrt( area/M_PI );
  float xCentroid = moments.m10/moments.m00;
  float yCentroid = moments.m01/moments.m00;
  float m[1][3] = {{ xCentroid, yCentroid, radius}};
  return cv::Mat(1, 3, CV_32F, m);
}

locate计算面具的质心及其半径.以{x,y,radius}的形式在单行mat中返回这3个值.
它用于cv::moments计算多边形或栅格化形状的三阶之前的所有时刻.在我们的案例中是一个栅格化的形状.我们对所有这些时刻都不感兴趣.但其中三个在这里很有用.M00是面具的区域.质心可以从m00,m10和m01计算出来.

void centerAndScale(const cv::Mat& image, const cv::Mat& mask,
        const cv::Mat& characteristics,
        cv::Mat& imageCS, cv::Mat& maskCS)
{
    float radius = characteristics.at<float>(0,2);
    float xCenter = characteristics.at<float>(0,0);
    float yCenter = characteristics.at<float>(0,1);
    int diameter = round(radius*2);
    int xOrg = round(xCenter-radius);
    int yOrg = round(yCenter-radius);
    cv::Rect roiOrg = cv::Rect( xOrg, yOrg, diameter, diameter );
    cv::Mat roiImg = image(roiOrg);
    cv::Mat roiMask = mask(roiOrg);
    cv::Mat centered = cv::Mat::zeros( diameter, diameter, CV_8UC3);
    roiImg.copyTo( centered, roiMask);
    cv::imwrite( "centered.bmp", centered); // debug
    imageCS.create( TEMPLATE_SIZE, TEMPLATE_SIZE, CV_8UC3);
    cv::resize( centered, imageCS, cv::Size(TEMPLATE_SIZE,TEMPLATE_SIZE), 0, 0 );
    cv::imwrite( "scaled.bmp", imageCS); // debug

    roiMask.copyTo(centered);
    cv::resize( centered, maskCS, cv::Size(TEMPLATE_SIZE,TEMPLATE_SIZE), 0, 0 );
}

centerAndScale uses the centroid and radius computed by locate to get a region of interest of the input image and a region of interest of the mask such that the center of the such regions is also the center of the coin and mask and the side length of the regions are equal to the diameter of the coin/mask.
These regions are later scaled to a fixed TEMPLATE_SIZE. This scaled region will be our reference template. When later on in the matching program we want to check if a detected candidate coin is this coin we will also take a region of the candidate coin, center and scale that candidate coin in the same way before performing template matching. This way we achieve scale invariance.

void saveRotatedTemplates( const cv::Mat& image, const cv::Mat& mask, int stepAngle )
{
    char name[1000];
    cv::Mat rotated( TEMPLATE_SIZE, TEMPLATE_SIZE, CV_8UC3 );
    for ( int angle=0; angle<360; angle+=stepAngle )
    {
        cv::Point2f center( TEMPLATE_SIZE/2, TEMPLATE_SIZE/2);
        cv::Mat r = cv::getRotationMatrix2D(center, angle, 1.0);

        cv::warpAffine(image, rotated, r, cv::Size(TEMPLATE_SIZE, TEMPLATE_SIZE));
        sprintf( name, "template-%03d.bmp", angle);
        cv::imwrite( name, rotated );

        cv::warpAffine(mask, rotated, r, cv::Size(TEMPLATE_SIZE, TEMPLATE_SIZE));
        sprintf( name, "templateMask-%03d.bmp", angle);
        cv::imwrite( name, rotated );
    }
}

saveRotatedTemplates保存以前的计算模板.
但它保存了几个副本,每个副本都旋转一个角度,定义在ANGLE_STEP.这样做的目的是提供方向不变性.我们定义stepAngle越低,我们得到的方向不变性越好,但它也意味着更高的计算成本.

您可以在此处下载整个模板制作程序.
当使用ANGLE_STEP作为30运行时,我得到以下12个模板:

模板匹配.

#define INPUT_IMAGE "coins.jpg"
#define LABELED_IMAGE "coins_with50cLabeled.bmp"
#define LABEL "50c"
#define MATCH_THRESHOLD 0.065
#define ANGLE_STEP 30
int main()
{
    vector<cv::Mat> templates;
    loadTemplates( templates, ANGLE_STEP );
    cv::Mat image = loadImage( INPUT_IMAGE );
    cv::Mat mask = createMask( image );
    vector<Candidate> candidates;
    getCandidates( image, mask, candidates );
    saveCandidates( candidates ); // debug
    matchCandidates( templates, candidates );
    for (int n = 0; n < candidates.size( ); ++n)
        std::cout << candidates[n].score << std::endl;
    cv::Mat labeledImg = labelCoins( image, candidates, MATCH_THRESHOLD, false, LABEL );
    cv::imwrite( LABELED_IMAGE, labeledImg );
    return 0;
}

这里的目标是读取模板和要检查的图像,并确定与我们的模板匹配的硬币的位置.

首先,我们读入了我们在前一个程序中生成的所有模板图像的图像矢量.
然后我们读取要检查的图像.
然后我们使用与模板制作者完全相同的功能对要检查的图像进行二值化.
getCandidates找到要形成多边形的点组.这些多边形中的每一个都是硬币的候选者.并且所有这些都被重新缩放并且以与我们的模板大小相等的方形为中心,以便我们可以以不变的方式执行匹配.
我们保存为调试和调整目的而获得的候选图像.
matchCandidates将每个候选者与为每个最佳匹配结果存储的所有模板匹配.由于我们有多个方向的模板,因此可以提供方向的不变性.
打印每个候选人的分数,以便我们可以决定将50c硬币与非50c硬币分开的阈值.
labelCoins复制原始图像并在分数大于(或小于某些方法)的分数上绘制标签MATCH_THRESHOLD.
最后我们将结果保存在.BMP中

void loadTemplates(vector<cv::Mat>& templates, int angleStep)
{
    templates.clear( );
    for (int angle = 0; angle < 360; angle += angleStep)
    {
        char name[1000];
        sprintf( name, "template-%03d.bmp", angle );
        cv::Mat templateImg = cv::imread( name );
        if (templateImg.data == NULL)
        {
            std::cout << "Could not read " << name << std::endl;
            exit( 1 );
        }
        templates.push_back( templateImg );
    }
}

loadTemplates类似于loadImage.但它加载了几个图像而不是一个,并将它们存储在一个std::vector.

loadImage 与模板制作者完全相同.

createMask is also exactly the same as in the tempate maker. This time we apply it to the image with several coins. It should be noted that binarization thresholds were chosen to binarize the 50c and those will not work properly to binarize all the coins in the image. But that is of no consequence since the program objective is only to identify 50c coins. As long as those are properly segmented we are fine. It actually works in our favour if some coins are lost in this segmentation since we will save time evaluating them (as long as we only lose coins which are not 50c).

typedef struct Candidate
{
    cv::Mat image;
    float x;
    float y;
    float radius;
    float score;
} Candidate;

void getCandidates(const cv::Mat& image, const cv::Mat& mask,
        vector<Candidate>& candidates)
{
    vector<vector<cv::Point> > contours;
    vector<cv::Vec4i> hierarchy;
    /// Find contours
    cv::Mat maskCopy;
    mask.copyTo( maskCopy );
    cv::findContours( maskCopy, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, cv::Point( 0, 0 ) );
    cv::Mat maskCS;
    cv::Mat imageCS;
    cv::Scalar white = cv::Scalar( 255 );
    for (int nContour = 0; nContour < contours.size( ); ++nContour)
    {
        /// Draw contour
        cv::Mat drawing = cv::Mat::zeros( mask.size( ), CV_8UC1 );
        cv::drawContours( drawing, contours, nContour, white, -1, 8, hierarchy, 0, cv::Point( ) );

        // Compute center and radius and area.
        // Discard small areas.
        cv::Moments moments = cv::moments( drawing, true );
        float area = moments.m00;
        if (area < CANDIDATES_MIN_AREA)
            continue;
        Candidate candidate;
        candidate.radius = sqrt( area / M_PI );
        candidate.x = moments.m10 / moments.m00;
        candidate.y = moments.m01 / moments.m00;
        float m[1][3] = {
            { candidate.x, candidate.y, candidate.radius}
        };
        cv::Mat characteristics( 1, 3, CV_32F, m );
        centerAndScale( image, drawing, characteristics, imageCS, maskCS );
        imageCS.copyTo( candidate.image );
        candidates.push_back( candidate );
    }
}

The heart of getCandidates is cv::findContours which finds the contours of areas present in its input image. Which here is the mask previously computed.
findContours returns a vector of contours. Each contour itself being a vector of points which form the outer line of the detected polygon.
Each polygon delimites the region of each candidate coin.
For each contour we use cv::drawContours to draw the filled polygon over a black image.
With this drawn image we use the same procedure earlier explained to compute centroid and radius of the polygon.
And we use centerAndScale, the same function used in the template maker, to center and scale the image contained in that poligon in an image which will have the same size as our templates. This way we will later on be able to perform a proper matching even for coins from photos of different scales.
Each of these candidate coins is copied in a Candidate structure which contains :

• Candidate image
• x and y for centroid
• radius
• score

getCandidates computes all these values except for score.
After composing the candidate it is put in a vector of candidates which is the result we get from getCandidates.

These are the 4 candidates obtained :

void saveCandidates(const vector<Candidate>& candidates)
{
    for (int n = 0; n < candidates.size( ); ++n)
    {
        char name[1000];
        sprintf( name, "Candidate-%03d.bmp", n );
        cv::imwrite( name, candidates[n].image );
    }
}

saveCandidates saves the computed candidates for debugging purpouses. And also so that I may post those images here.

void matchCandidates(const vector<cv::Mat>& templates,
        vector<Candidate>& candidates)
{
    for (auto it = candidates.begin( ); it != candidates.end( ); ++it)
        matchCandidate( templates, *it );
}

matchCandidates just calls matchCandidate for each candidate. After completion we will have the score for all candidates computed.

void matchCandidate(const vector<cv::Mat>& templates, Candidate& candidate)
{
    /// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better
    candidate.score;
    if (MATCH_METHOD == CV_TM_SQDIFF || MATCH_METHOD == CV_TM_SQDIFF_NORMED)
        candidate.score = FLT_MAX;
    else
        candidate.score = 0;
    for (auto it = templates.begin( ); it != templates.end( ); ++it)
    {
        float score = singleTemplateMatch( *it, candidate.image );
        if (MATCH_METHOD == CV_TM_SQDIFF || MATCH_METHOD == CV_TM_SQDIFF_NORMED)
        {
            if (score < candidate.score)
                candidate.score = score;
        }
        else
        {
            if (score > candidate.score)
                candidate.score = score;
        }
    }
}

matchCandidate has as input a single candidate and all the templates. It's goal is to match each template against the candidate. That work is delegated to singleTemplateMatch.
We store the best score obtained, which for CV_TM_SQDIFF and CV_TM_SQDIFF_NORMED is the smallest one and for the other matching methods is the biggest one.

float singleTemplateMatch(const cv::Mat& templateImg, const cv::Mat& candidateImg)
{
    cv::Mat result( 1, 1, CV_8UC1 );
    cv::matchTemplate( candidateImg, templateImg, result, MATCH_METHOD );
    return result.at<float>( 0, 0 );
}

singleTemplateMatch peforms the matching.
cv::matchTemplate uses two imput images, the second smaller or equal in size to the first one.
The common use case is for a small template (2nd parameter) to be matched against a larger image (1st parameter) and the result is a bidimensional Mat of floats with the matching of the template along the image. Locating the maximun (or minimun depending on the method) of this Mat of floats we get the best candidate position for our template in the image of the 1st parameter.
But we are not interested in locating our template in the image, we already have the coordinates of our candidates.
What we want is to get a measure of similitude between our candidate and template. Which is why we use cv::matchTemplate in a way which is less usual; we do so with a 1st parameter image of size equal to the 2nd parameter template. In this situation the result is a Mat of size 1x1. And the single value in that Mat is our score of similitude (or dissimilitude).

for (int n = 0; n < candidates.size( ); ++n)
    std::cout << candidates[n].score << std::endl;

We print the scores obtained for each of our candidates.
In this table we can see the scores for each of the methods available for cv::matchTemplate. The best score is in green.

CCORR and CCOEFF give a wrong result, so those two are discarded. Of the remaining 4 methods the two SQDIFF methods are the ones with higher relative difference between the best match (which is a 50c) and the 2nd best (which is not a 50c). Which is why I have choosen them.
I have chosen SQDIFF_NORMED but there is no strong reason for that. In order to really chose a method we should test with a higher ammount of samples, not just one.
For this method a working threshold could be 0.065. Selection of a proper threshold also requires many samples.

bool selected(const Candidate& candidate, float threshold)
{
    /// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better
    if (MATCH_METHOD == CV_TM_SQDIFF || MATCH_METHOD == CV_TM_SQDIFF_NORMED)
        return candidate.score <= threshold;
    else
        return candidate.score>threshold;
}

void drawLabel(const Candidate& candidate, const char* label, cv::Mat image)
{
    int x = candidate.x - candidate.radius;
    int y = candidate.y;
    cv::Point point( x, y );
    cv::Scalar blue( 255, 128, 128 );
    cv::putText( image, label, point, CV_FONT_HERSHEY_SIMPLEX, 1.5f, blue, 2 );
}

cv::Mat labelCoins(const cv::Mat& image, const vector<Candidate>& candidates,
        float threshold, bool inverseThreshold, const char* label)
{
    cv::Mat imageLabeled;
    image.copyTo( imageLabeled );

    for (auto it = candidates.begin( ); it != candidates.end( ); ++it)
    {
        if (selected( *it, threshold ))
            drawLabel( *it, label, imageLabeled );
    }

    return imageLabeled;
}

labelCoins draws a label string at the location of candidates with a score bigger than ( or lesser than depending on the method) the threshold. And finally the result of labelCoins is saved with

cv::imwrite( LABELED_IMAGE, labeledImg );

The result being :

The whole code for the coin matcher can be downloaded here.

Is this a good method?

That is hard to tell.
The method is consistent. It correctly detects the 50c coin for the sample and input image provided.
But we have no idea if the method is robust because it has not been tested with a proper sample size. And even more important is to test it against samples which were not available when the program was being coded, that is the true measure of robustness when done with a large enough sample size.
I am rather confident in the method not having false positives from silver coins. But I am not so sure about other copper coins like the 20c. As we can see from the scores obtained the 20c coin gets a score very similar to the 50c.
在不同的光照条件下也会发生假阴性.如果我们能够控制照明条件,例如当我们设计机器拍摄硬币照片并计算它们时,哪些是可以而且应该避免的.

如果该方法起作用,则可以针对每种类型的硬币重复相同的方法,从而导致所有硬币的完全检测.