MatchGrid assertion with small grid

[FlyingLotus1983]

Using the following pipeline:

[
    {"op":"matchTemplate",
        "name":"match",
        "template":"template.png",
        "corr":"{{corr||0.8}}",
        "threshold":"{{threshold||0.6}}",
        "method":"{{method}}",
        "angle":"{{angle}}",
        "output":"{{output}}" },
    {"op":"matchGrid", 
        "name":"grid1", 
        "calibrate":"{{calibrate}}", 
        "scale":"{{scale}}",
        "sep":"[1.5,1.5]",
        "model":"match" },
    {"op":"drawRects", "name":"draw1", "radius":5, "model":"grid1", "color":"{{colorUndistort||[32,255,32]}}"},
    {"op":"undistort", "name":"undistort", "model":"grid1" },
    {"op":"drawRects", "name":"draw2", "radius":2, "model":"match", "color":"{{colorMatch||[32,32,255]}}"}
]

And the following grid / template:

I get the following assertion:

flyinglotus1983@lunix /media/sf_Computer_Vision/2015-08-22 $ firesight -p matchGrid.json -i endo-uncorrected.png -o output.png
firesight: /home/flyinglotus1983/git/FireSight/stages/calibrate.cpp:446: void GridMatcher::calcGridIndexes(): Assertion `0 <= c && c < gridIndexes.cols' failed.
Aborted

I've tried to look through the code, but I can't seem to find out what's causing it. We don't get this assertion with a larger grid. But I'm not sure what's causing it to fail with a smaller grid. The documentation and lack of comments in the code make this a bit confusing to follow. Any help would be appreciated.

[FlyingLotus1983]

If I remove the matchGrid where the assertion fails, and add a drawRects in, this is what i get:

matchGrid is returning lots of random points. This seems to have been caused by having a bad template that returned too many matches. The template was bad because it was copied and pasted from the grid which was also bad. I printed it out on a second printer, and this is what I get:

It does still fail the assertion though.

[firepick1]

Wow. That's a pretty fine grid.

MatchGrid is failing because it's starving on cell matches. The assert says "i have no idea where this cell is". For more data, try a larger grid taken with slight offsets between pictures. The algorithm has a key assumption that the cell positioning error is small with respect to the grid size. With the lens distortion I'm seeing, the cell positioning error is large and the algorithm can't quite figure out where the cells are.

MatchGrid serves two purposes:

1. perspective distance calculation (distance between cells, which is only weakly related to cell size)
2. lens distortion image generation

Sounds like you're trying to do #2. The original lens distortion algorithm in OpenCV relies on lots of checkerboard pictures and aggregates over them all. MatchGrid simulates that by using one picture and chopping it up into sections. Basically it cheats and you're discovering that. For more accuracy on the lens distortion calcs, I suggest coarser grid, but take five pictures (x0y0, x1y0,x0y1,x-1y0,x0,y-1) and average out the coefficients.

OpenCV lens correction is limited. It won't account for every single lens defect, and can only really model certain kinds of distortion. If the endolens has a weird unmodeled distortion, then OpenCV will try hard and give you uncertain results. With the RaspiCam I found myself using the middle 50% of the image and ignoring the periphery, which is where most of the distortion creeps in.

[firepick1]

The other thing I'm curious about is the template.
In the wiki I use:
https://github.com/firepick1/FireSight/blob/master/img/cross32.png

This is not an actual picture of anything but a perfect ideal grid intersection. The image was actually generated by FireSight itself. It's not a photograph. I found that template matching works best with idealized images instead of photographs provided the recognition goal is a simple geometric object. The use of photographs is only appropriate for highly complex templates (e.g., Mona Lisa) that have no simple geometric shape. Grids are pretty vanilla, hence cross32.png.

[FlyingLotus1983]

I've used the cross32.png plenty, and I have been using them interchangeably with the one I showed above. If I use it, the assert error doesn't go away.

[firepick1]

The cross32 wont affect the grid assert, since that happens after the
match. Your match template is succeeding, but the the grid algorithm is
failing because of tiny cells. :(

On Saturday, August 22, 2015, Neil Jansen [email protected] wrote:

I've used the cross32.png plenty, and I have been using them
interchangeably with the one I showed above. If I use it, the assert error
doesn't go away.

—
Reply to this email directly or view it on GitHub
#25 (comment).

Karl Lew
FIREPICK SERVICES LLC
[email protected]
www.firepick.org

[FlyingLotus1983]

I'd like to leave the issue open, we might not get to this immediately but I do think this is a valid open issue for firesight. I'm not sure if this is just a documentation fix or something that will drive some code changes. Either way I think this is going to be something needed for FPD in the future and I'd like to not have this information forgotten about.

[firepick1]

Code change would be major since the design assumption is orthogonal grid
with minor distortion.

On Saturday, August 22, 2015, Neil Jansen [email protected] wrote:

I'd like to leave the issue open, we might not get to this immediately but
I do think this is a valid open issue for firesight. I'm not sure if this
is just a documentation fix or something that will drive some code changes.
Either way I think this is going to be something needed for FPD in the
future and I'd like to not have this information forgotten about.

—
Reply to this email directly or view it on GitHub
#25 (comment).

Karl Lew
FIREPICK SERVICES LLC
[email protected]
www.firepick.org