Notes For OpenCV On Linux

OpenCV now uses GitHub.

Formerly they used CVS.

Change to the opencv source directory you just acquired and do the cmake dance.

I had trouble with the Eigen library not being found. I had to make a symlink like so.

I also had heroic struggles getting FFMPEG to be detected and incorporated properly. This is what that problem looks like.

This bug report is very much on target however, none of the "solutions" worked for me so far.

Maybe this approach will work: https://mindchasers.com/dev/ubuntu-opencv

I was able to get this to compile and run on Debian GNU/Linux 9 (stretch) with the system managed (binary package) install.

The compile command is very fussy and you have to include things just so and in the right order. For the sample program above, this used to work.

It does not work now. Things have been moved around and keeping track of this mess has become so impossible that is actually now easier because you can delegate to pkg-config. Here is the command that finally was able to compile the test program above.

Note that the order is critical. If you put the libs and includes first, it will have linker errors! This will show up as undefined reference. If you get that, try mixing up the order.

Then to compile your own programs using this, create a Makefile in your working directory that looks like this.

Create and Free Matrices

Query Matrix Properties

The best performing way to deal with matrices is to just use pointer arithmetic. The matrices are stacked so that X is filled first. Y is incremented when the first row of X is done.

It is often very effective to use ROI to isolate things to eliminate any extraneous operations on regions of a lack of interest. Here’s an example of using ImageROI to increment all of the pixels of a region. This will make a specified rectangle brighter and more white.

Another way to do this kind of thing is to use the widthStep property to map out a subregion (a hand crafted ROI of sorts). Sometimes doing this can be more efficient than using the ROI functions.

Need to do some operation on an array? Here are some of the possible functions available:

cvAbs, cvAbsDiff, cvAbsDiffS, cvAdd, cvAddS, cvAddWeighted, cvAvg, cvAvgSdv, cvCalcCovarMatrix, cvCmp, cvCmpS, cvConvertScale, cvConvertScaleAbs, cvCopy, cvCountNonZero, cvCrossProduct, cvCvtColor, cvDet, cvDiv, cvDotProduct, cvEigenVV, cvFlip, cvGEMM, cvGetCol, cvGetCols, cvGetDiag, cvGetDims, cvGetDimSize, cvGetRow, cvGetRows, cvGetSize, cvGetSubRect, cvInRange, cvInRangeS, cvInvert, cvMahalanobis, cvMax, cvMaxS, cvMerge, cvMin, cvMinS, cvMinMaxLoc, cvMul, cvNot, cvNorm, cvNormalize, cvOr, cvOrS, cvReduce, cvRepeat, cvSet, cvSetZero, cvSetIdentity, cvSolve, cvSplit, cvSub, cvSubS, cvSubRS, cvSum, cvSVD, cvSVBkSb, cvTrace, cvTranspose, cvXor, cvXorS, cvZero

For details on these, check the official reference.

When OpenCV needs to dynamically allocate memory it has an internal mechanism blandly called "memory storage" to facilitate this. Memory storages are linked lists of continuous memory blocks suited to efficient allocation and release. The functions used to create/destroy these entities are cvCreateMemStorage, cvReleaseMemStorage, cvClearMemStorage, and cvMemStorageAlloc. The default size is 64kB if not otherwise specified. The last function listed is a way to allocate the memory yourself and then assign that specific location to the memory storage object. Apparently explicit releasing of these things is essential if you really want to be comprehensive about clean up; other clean up functions don’t actually give the memory back to the system but merely make it ready for more of the same use.

On of the things that can be stored in a "memory storage" is a "sequence". This can be thought of as a deque in STL (but since OpenCV stubbornly does not like C++ they needed to do this internally - not that there’s anything wrong with that). Beyond this structure, sequences have pointers that can be used to assemble them into trees, lists, and other wacky structures.

To create a squence entity use cvCreateSeq. To clear a sequence use cvClearSeq but remember, to really free up the memory involved, you need to revisit cvClearMemStorage. To access an arbitrarily located item in a squence use cvGetSeqElem. Or if you just need to know where in a sequence an item is, cvSeqElemIdx (which is a somewhat inefficient thing to do). Sequences can also be copied whole or in slices with cvCloneSeq and cvSeqSlice (the former is a subset wrapper of the latter). Slices can be used to remove or insert elements with cvSeqRemoveSlice and cvSeqInsertSlice. Another way to do this an element at a time is with cvSeqInsert and cvSeqRemove. Performance on these random accesses to the middle may not be sufficient. There is also cvSeqSort the sequence. You get to provide the comparison function (type *CvCmpFunc). Reverse the sequence with cvSeqInvert.

Because they are really linked lists, it is easy to treat them as stack structures. The following functions are available to use sequences as stacks conveniently.

The function cvSetSeqBlockSize is a bit like inode tuning in that it is useful to set the memory block size that gets allocated when new sequence items are needed. This allows you to accommodate huge items in short lists or small items in long lists. The default is 1kB.

There are ways to convert a sequence to an array, namely the cvCvtSeqToArray function. To go the other way, check out cvMakeSeqHeaderForArray.

There are some fancy reading and writing functions to load and read sequences in bulk efficiently. The downside is that these must be initialized and then closed to do proper housekeeping. The functions involved are cvStartWriteSeq, cvStartAppendToSeq, cvEndWriteSeq, cvFlushSeqWriter, CV_WRITE_SEQ_ELEM for writing and cvStartReadSeq, cvGetSeqReaderPos, cvSetSeqReaderPos, CV_NEXT_SEQ_ELEM, CV_PREV_SEQ_ELEM, CV_READ_SEQ_ELEM, and CV_REV_READ_SEQ_ELEM for reading.

The primary GUI function is cvNamedWindow which opens a window and puts its name in the title bar. The title/name is used as a handle in subsequent references to the window.

It’s inverse is cvDestroyWindow. This also takes the human readable name you gave it. Windows can also be referenced by a void* window_handle, so if that shows up, don’t freak out.

If you’re in a bigger hurry to destroy a lot of junk, use cvDestroyAllWindows().

Some official documentation.

One important function that doesn’t seem to fit in the HighGUI library (but that’s where it’s found!) is cvConvertImage. This can do some color depth conversions and grayscale stuff. It can also flip images to reverse images (like loading an old slide backwards).

Here’s how to convert to gray scale which is a very common precursor to many operations.

If you read in an image using cv2.imread() you will get an BGR image, but if you read it in using matplotlib.image.imread() this will give you a RGB image.

void cvShowImage(const char* name, const CvArr* image);

See the examples for the complete process in action.

If cvWaitKey is sent a 0, then it will wait indefinitely instead of the specified number of ms.

Mouse features are supported but are done in the ordinary way with a callback.

In OpenCV, sliders are called "trackbars".

cvCreateTrackbar

Since HighGUI does not provide any kind of buttons it is common for trackbars that only have 2 positions.

Here is a complete example that worked with my PS Eye camera to record video capture to a file.

OpenCV has five types of image smoothing or blurring using the cvSmooth() function.

In OpenCV morphology refers to manipulations on an image based on some reference thing that reminds me of a "brush" in paint programs. There is dilation where the brush increases bright regions and there is erosion where the brush decreases them. The "brush" is called a kernel and can take square or elliptical form or some arbitrary user defined shape. There are all kinds of subtle variants. Look into the cvMorphologyEx function for more details.

Using the cvFloodFill() function parts of an image can be filled in as with normal image manipulation programs. A mask can also be supplied to restrict where the operation must spend resources and take effect. A seed point is provided as a place to start from. The function also takes a "lo" (their name) differential and an "up" differential. If the neighboring pixel is between the two it is colored. The pixels can all be compared to the seed or to neighbors. The result of what got filled can be returned as the mask.

The cvResize() function does what the name implies. Choose one of these interpolation functions.

This is a complex topic to be sure. For example, I keep running across the word "convolution" and looking it up I find, appropriately, "something that is very complicated and difficult to understand". But it also means "a twist or a curve" (as on the cerebrum of fancy mammals). None of those definitions are helpful yet. The idea of image pyramids is that a stack of derived images is created with decreasing resolution. This allows for operations to be performed at the top (cheap) level as a preliminary optimization and then extended to a focused region in the high resolution bottom level. Look into cvPyrSegmentation for details.

Some more about convolution. It seems to involve something that is done to all parts of an image. What that something specifically entails is defined by the "convolution kernel". The convolution kernel is just a grid of values with the "anchor point" located in the middle of the grid. This grid is superimposed, successively, over every point in the input image. The grid is aligned so that the kernel is on the iterated input image point. Then the values in the kernel are multiplied by the places on the input image they superimpose, those are added and that is set as the new value for the output image. Of course this implies that there is some trickery required at the edges. In other words if a 5x5 kernel was at (100,23) of a 100x100 the middle row of the kernel grid would be at (98,23), (99,23), (100,23), (101,23), and (102,23), but there is no 101 or 102. The details of how this gets resolved are found in the borderInterpolate function.

The cvThreshold() function takes a source and destination array (image, matrix, whatever) and checks the source against a comparison function and sets the corresponding output pixel accordingly.

Here are the types of threshold behaviors and the function that the destination pixel is set to.

Here s is each source pixel. M is the maximum value which seems to basically be just some number you’d like to use; you get to set it. t is the threshold value which you also get to set of course.

And if that is too easy for you, go crazy with cvAdaptiveThreshold() which can dynamically adjust t to be more in line with the local surroundings of s.

The Sobel derivative is a type of convolution that calculates the derivative (change in value over distance - but not really because it’s not continuous) of the image. This often has a directionality (e.g. change in X only). The point of this is to detect edges and other "features" in images. This is done with convolution and a kernel to reduce sensitivity to noise. If you want to use a 3x3 convolution kernel in your Sobel filtering, it is recommended to use CV_SCHARR which is a specific optimized value for that purpose.

Another similar function is cvLaplace(). The mathematical Laplace operator is a sum of second derivatives along the x and y axes. Again it’s not really a real derivative which indulges in the idea of an analogue universe. It’s the same kind of thing as the Sobel, but with slightly different results. Interestingly it can be combined with the Sobel filter for even more pragmatic and effective results in many normal edge detection situation.

Expanding on the previous techniques is the Canny algorithm. This does all kinds of mathematical black magic with derivatives, combining X and Y directions, and arrives at a very high quality edge detection. The Canny algorithm uses a hysteresis threshold to map contours of value that outline detected features. The Canny algorithm only works on greyscale (returns a 1 bit image). Check out the section on "contours" to find out more about how edge detection is done in practice.

The first Hough transform was a way to discriminate patterns from images. The Hough transform makes it possible to perform groupings of edge points into object candidates by performing an explicit voting procedure over a set of parameterized image objects. At frist the parameters were slope and intercept but this caused some mathematical problems (vertical lines=division by zero); polar coordinates are now used. The important thing is that if you need to find meaningful lines in your somewhat noisy image, look into this. Also circles and ellipses are theoretically possible.

Often you’ll have an original image and you’ll want a modified version of it. The cvRemap can map an image on to another. The target image is just the target, it is not some other scene that merges in some complex way. A classic simple example would be scaling an image up or down. If scaling up, it is easy to imagine gaps being present in the destination image. This is why the algorithm works from each position of the target image and back figures what from the source must go there. This involves a lot of interpolation, the specific nature of which can be specified (CV_INTER_NN, CV_INTER_LINEAR, etc). The mapping can be complex. If so it is described by map images which specify how the source is transformed. There is one map for X and one for Y. A great practical example of this function is for correcting camera lens distortions into a more useful form. I could also imagine sorting out some Oculus Rift feed with this.

When mutating the geometric form of images, there are two kinds of transformations, affine and perspective. Affine transforms basically produce parallelograms (from the four edges of the original). The points can be mushed in any way as long as the top and bottom remain parallel and the left and right side remain parallel. Perspective transformations, on the other hand, introduce a zoom or vanishing point and can produce trapezoids. In both straight lines remain straight. In the first case, cvWarpAffine will mutate the image into the desired form. As with remap, there is some heavy interpolation and how that’s done can be specified in the familiar ways. The details of how the transformation should go is encoded into a 2x3 matrix since the source and destination, both of which are necessarily rectangles, do not indicate that. To figure out what this matrix is, OpenCV has a nice function to compute it, cvGetAffineTransform. This function takes a source and destination image each containing exactly 3 points. The final parameter is the matrix which the function sets. If instead of mushing your image into squished shapes you are rotating them, there is another way to calculate the affine transform matrix. The function cv2DRotationMatrix does basically the same job as the cvGetAffineTransform function but with a center, angle, and scale input. These two can be combined for an image that is both rotated, scaled, and warped.

The cvWarpAffine works for "dense" images, i.e. an entire x,y grid of points. If the input is just a series of points, i.e. not all points in an x,y area, then this "sparse" set could be more efficiently transformed using the cvTransform function.

Perspective transformations have similar functions. cvWarpPerspective is the main one to transform dense maps. cvGetPerspectiveTransform helps to figure out what the mapping matrix is. Sparse perspective transforms can use cvPerspectiveTransform.

A nonintuitive mapping function is cvCartToPolar and cvPolarToCart. Why one would want to convert an image from Cartesian coordinates to polar or the other way is a bit tricky. It is useful in catching edge detection thresholds after other filtering is performed. Another similar but weirder one is cvLogPolar. This transforms (x,y) into (log radius,angle). If done just right, so the theory goes, this provides a kind of invariance to planar rotation and scaling which might be useful if trying to track an object. If the object scales, its transform will shift on the horizontal. If the object rotates, the transform will shift on the vertical. This might be useful, albeit complicated, for tracking a fixed size moving object from a fixed overhead camera.

OpenCV has cvDFT which is Discrete Fourier Transform. This is a "fast" (FFT) O(N log N) version. Apparently, DFT is often overkill and for practical situations a better function is likely to be cvDCT or the Discrete Cosine Transform`.

If an image is too dark or washed out, the variation between all the pixels is not ideal in that it does not use the full range of information expressible by the pixel value range. Plotting the histogram of such an image will show most pixels concentrating in a narrow band of value range. By using the cumulative distribution function, the histogram can be remapped and the image revalued to make sure that the resultant histogram uses more of the range. OpenCV has the cvEqualizeHist function for this. While I can see this being useful to improve the aesthetics of a rendered image for humans, I wonder if it could really impart any more information to the image in a way that would allow processing algorithms to actually do a better job. In other words, what’s the difference between equalizing the histogram and just having some feature recognition filter concentrate only on a limited range of values?

But histograms have quite a few clever uses. Histograms of colors, edge gradients and other attributes can be used to determine scene specific content. Hand gesture recognition is one application. They can detect transitions in videos. Maybe a way to train a system to cut out ads. Hmm.

OpenCV has a lot of functions to make working with histograms as easy and trouble-free as possible. Look for the data type CVHistogram and the constructor/destructor functions cvCreateHist, cvSetHistBinRanges, cvClearHist, and cvMakeHistHeaderForArray. The last one uses data you have already organized to bless it as a histogram that OpenCV can use.

Another way to generate less arbitrary and more mission oriented histograms, is to automatically create them from images. The cvCalcHist function can take an image and make a typical useful histogram out of it involving a variety of properties.

Once the histogram is complete, accessing the data (or the pointer to the data) can be achieved with functions such as cvQueryHistValue_nD and cvGetHistValue_nD.

Also consider cvNormalizeHist which will replace values with the portion of the total events that goes in that bin. So if you have 25 events and in bin x there are 5, the normalized form will replace the 5 with .2; everything should add up to 100% or 1 (though this can be changed with the factor argument to the function). Another valid approach which is subtly different is to normalize the colors (or whatever) in your source data so that the histogram is ready to be used with no further conditioning.

Another function to process the histogram is cvThreshHist that basically resets bins with very few members. Imagine bins with counts of 1,3,2,0,1,78,26,132,19,2,0,0,3,1. It is likely that thinking of this data as 0,0,0,0,0,78,26,132,19,0,0,0,0,0 would be more useful.

The cvCopyHist function does the obvious but in several subtle flavors involving either filling a pre-existing same size target histogram with the source or creating (allocating) a target from nothing.

The cvGetMinMaxHistValue gets minimum and maximum values obviously, but how exactly isn’t entirely clear. I believe that it returns the number of items in the bin with the most items and optionally the index where that bin is. I don’t know how the latter part of that handles ties.

Histograms can be compared with the cvCompareHist function. There are several possible criteria, or "methods", which can be used.

There is another histogram comparing technique called Earth Mover’s Distance. This basically treats the items in bins as dirt that must be moved and takes into account how much needs to be moved and how far away to compare two probability distributions or histograms. OpenCV has cvCaclEMD2 which is full of parameters allowing you to do fancy things such as specify your own distance and work metrics.

Make sure to use cvNormalizeHist before comparing because comparing unnormalized histograms is usually meaningless.

There is a technique called "back projection" which can determine how well data fit the distribution of a histogram model. For example, if you have a histogram of an object you can see if an image contains regions with a similar histogram using back projection. The function to consider is cvCalcBackProject. A related function is cvCalcBackProjectPatch which will check if an image contains sub regions that are well matched.

A similar thing is template matching which does similar things to the histogram functionality but without histograms per se. This uses cvMatchTemplate to take patches of image, say a thumbnail sized image of just an apple, and scan a larger image looking for likely similar regions. There are all kinds of similarity metrics as is typical and they all have subtle functionality and performance tradeoffs.

Contours are ways to manage features of images. They are stored as CvSeq type sequences (linked-lists deep down). Contours can be created from images filtered by cvCanny or cvThreshold, etc using the cvFindContours function. This function can get tricky in the same way a bucket fill tool can get tricky with respect to finding islands on lakes on islands in lakes, etc (see 69.793° N, 108.241° W for an example). OpenCV calls these things "contours" (islands) and "holes" (lakes). This all apparently does often get quite complex and OpenCV has a fancy data structure called a "contour tree" to organize these complex relationships. In this structure, the world’s continents (sticking with the geography metaphor) would have contours at the top of a tree and all lakes would be children. And islands on those lakes would be (contour) children of their respective lakes, ad infinitum. Back in the world of image processing, the islands often have lakes that match them exactly (like an atoll) because that’s how edge detection filters. This means that edges tend to have inner and outer edges themselves just as an atoll has an outer beach and an inner beach, but the atoll separates the lagoon from the sea.

The cvFindContours function expects an 8-bit single-channel image which, it is important to note, will be mangled during calculation (make a copy if that’s needed). This function will allocate the CvSeq structures necessary (and free them) but it’s a little confusing how to set it up. The firstContour parameter takes a pointer to a pointer that would point toward the first contour if it existed. But it doesn’t because the function spawns it. But that pointed to pointer is where you’ll find the head of the tree structure that results from the function. The return value is the total number of contours found. The mode and method parameters respectively specify what sort of operation should be calculated and exactly how if there are variants.

There are four different modes which basically specify the topology of the resulting tree of contours.

After figuring out how you want the contours to be organized internally, the next thing is to specify the technique used to compose the contours themselves. There are several and they’re pretty technical.

To get an idea of what these things are, this might be helpful. Described simply, chain code starts with coordinates of a boundary pixel and then is a stream (or chain) of directions one would need to travel to stay on the boundary. When the original point is arrived at, the region is defined. The specific case of Freeman chains is an 8 direction system with 0 at 12 o’clock going to 7 at 10:30. This encoding doesn’t have to be at the pixel level and can be quite rough making it a very efficient way to describe large areas of input images.

Besides cvFindContours, there is another way to do things. There is a cvStartFindContours which creates a "scanner" or a CvSequenceScanner object. You can iterate over the contours with cvFindNextContour. When finished, cvEndFindContour stops that process.

An important utility is to be able to draw contours which can be done with cvDrawContours. This allows all the normal stuff like line color and thickness as well as the levels of the tree to plot. A negative line thickness field (or set to CV_FILLED) will fill the contour allowing for solid shapes. This can be useful for making masks from vector paths (basically contours).

Here’s a Python example of drawing some simple shapes on an image.

Once the contour has been found, you can use cvApproxPoly on it to convert it to a contour with fewer points. This is basically a raster to vector operation even though the result is still a contour sequence. The algorithm works by finding maximally distant points on the original contour. Those are the first two of the final points. Then the line between them is checked to see where it is farthest from the contour. That point on the contour is added to to new approximated contour. This continues until the desired number of points is reached.

A related function is cvFindDominantPoints which seems very similar to what I call "cull shallow angles" in to2d. It differs in that it can look at several points away from just its immediate neighbor. It is the same basic idea though i.e. to do what the function name suggests. The method is selectable, but there is only one choice CV_DOMINANT_IPAN. This IPAN stuff is stupidly named and should be treated as a random label for this technique.

Now that you have simple or complex contour sequences, there are some calculations you can do that can be informative. There is cvArcLength and cvContourPerimeter. The former can provide lengths of just portions of the contour (use slices). The cvContourArea is similar and can also do a portion of the contour or you can set slice = CV_WHOLE_SEQ.

Another useful thing to do with a contour sequence is to get an even rougher idea of where it is by using cvBoundingRect. This is parallel to the X and Y axes. If you want the truly smallest rectangle regardless of orientation, check out cvMinAreaRect2 which returns a type CvBox2D (containing center x and y, size x and y, and angle).

In the same spirit as the bounding box functions, there is also cvMinEnclosingCircle. Related, but with a different approach is cvFitEllipse2. This does not ensure the resulting ellipse contains all points but rather does a kind of least squares fitting function.

After finding the contours and distilling them down to bounding boxes, you can check for collisions and the like. The questionably named cvMaxRect function takes 2 rectangles as input and returns a rect (actually a CvRect) which is the smallest rectangle that will enclose both.

Getting fancier is a function to calculate moments, cvContourMoments which returns into a special data type, CvMoments. It seem that function is actually a wrapper for the cvMoments function which can provide normalized moments too. This allows for comparing different sized objects in a consistent way. More specialty functions are cvGetCentralMoment, cvGetNormalizedCentralMoment, cvGetHuMoments (rotation invariant). More details about image moments. This kind of thing might be useful to build object recognition profiles, maybe something like OCR where each letter has a set of moments that identify it regardless of position, size or rotation. OpenCV has a function to compare shapes in this way (even calculating the moments on the way), cvMatchShapes.

Because this isn’t complex enough, there is another more detailed way to compare contours than the summary statistics like moments. Looking at the details of the contour paths themselves is done by constructing a data structure called CvContourTree which is not the same as the data structure that contains a (possibly linked list of a) set of contours. This is a tree that represents a single contour’s shape in a way that is more easily matched by hierarchical geometric features. This can all be thought of in black box terms as just another way to compare shapes based on geometry. The functions provided to do this are cvCreateContourTree, cvContourFromContourTree, and cvMatchContourTree.

Another way to summarize shapes for comparison/identification purposes is to calculate the convex hull and look for the differences, called the convexity defects. OpenCV has cvConvexHull2 and cvConvexityDefects to facilitate this. Also cvCheckContourConvexity can determine if a contour is already convex.

If hulls are not enough, there is yet another matching strategy called pairwise geometrical histograms. This basically takes the Freeman chain codes mentioned above and makes a histogram of the direction changes (a CCH, chain code histogram). See cvCalcPGH for the function to do this.

A simple way that objects can be detected in a scene is to subtract the pixels of a frame now from the values of pixels from a little while ago. The function cvAbsDiff does this helpfully dealing in absolute values so it doesn’t matter which way around you go. This function takes 2 input frames, one now, one before, and an output frame called "frameForeground". I’m not keen on the use of the word "foreground". Imagine pointing a camera out a window whose frame and curtains were in the shot. That window would be the foreground but the points of interest if something moved outside of the window are technically in the background. But just be aware that this is the terminology. When using the cvAbsDiff function, it’s usually sensible to cut off minor fluctuations which are generally noise and to set the rest to 255. Do this with cvThreshold.

There are much fancier ways that can help with things like blowing leaves on a tree in an outdoor scene. This wouldn’t want to be regarded as an object of interest in motion. One approach is to use averaging. OpenCV can develop a running average for the values of each pixel and when large deviations from that occur, do something special.

OpenCV has an accumulation function cvAcc that can help accumulate statistics about a series of pixel changes. This function basically adds up the value of the pixel which can be used with the total number of images to get the mean value. Another similar metric is to use cvRunningAvg. The cvSquareAcc can be useful in calculating the variance of pixels. Presumably this will be a metric of how wildly the values are differing which could be very relevant to motion detection. The cvMultiplyAcc is another one that can be used in such applications.

For complex moving backgrounds (windy trees), the ideal thing is to fit a distribution to the data that is present in the previous frames. Since this could imply using a lot of memory, a complex but efficient approach is to use the same kinds of tricks used by compression algorithms; check out codebooks. These focus on the important pixels more than the boring ones. The pro tip here is to use HSV or YUV and not RGB when doing fancy things like this.

A nice function is cvInpaint which can fill in small (thin really) details that have been messed up in an image. It reminds me of the clone tool in Gimp. So if you have a photo with some thin writing on it done in a paint program, for example, this function can really do a good job of blending it away. It seems ideal for camera artifacts and grainy footage.

The function is cvPyrMeanShiftFiltering and uses the pyramid structures. To me the results of this remind me of "posterization". But the technical description is something like this.

OpenCV has a function called cvGoodFeaturesToTrack() which uses the Shi Tomasi algorithm, computes the second derivatives using Sobel operators, calculates the required eigenvectors, (whew!) and simply, from our point of view, returns a list of points that should be pretty good for tracking. This list contains points that you hope to be able to find again in another frame of the video. For example, an edge can be moving parallel to its orientation and the motion may be undetectable. A good feature, like a corner is noticeable whenever it moves in any direction. Highlighting a few especially amenable points allows for "sparse", but sensible, optical flow mapping.

If for some reason you require more precision in feature detection than the grid of pixels would imply, explore cvFindCornerSubPix. This will do some fancy dot product trickery to try and isolate corners even more precisely than the bitmap would seem to allow. This seems applicable in calibration operations.

A good example of cvFindCornerSubPix that works with little fuss can be found here.

Lucas-Kanade (also "LK") is a sparse flow mapping technique. This was used in the worm motility project. Requires consistent brightness, small motions frame to frame, and "spacial coherence" (which I do not exactly understand).

OpenCV deals with Lucas-Kanade implementations in two ways. There is the cvCalcOpticalFlowLK function which just calculates the flow field where it can be calculated (0 where it can not be). And there is pyramid based processing with cvCalcOpticalFlowPyrLK. Basically you need to supply the points you want to track in featuresA (from cvGoodFeaturesToTrack usually) and call the function. When it returns, check the status array to see which points were actually tracked successfully and then check featuresB to see where they are now.

Mean shift is a technique for locating the maxima of a density function. It seems to work by specifying a window enclosing some points, calculating the center of mass of the points, recentering the window there, and iterating until the window no longer needs to move. Of course choosing this window wisely is a tricky detail. OpenCV has cvMeanShift to facilitate this process.

CAM is "continuously adaptive mean-shift. This allows for the window to be resized as necessary to accommodate things like a subject getting nearer and farther away from the camera. Its function is cvCamShift.

The features that these algorithms track are usually colors, but they can really track the distribution of any kind of feature. For example, my first thought was to use feature detection and track the distribution of "pointy bits" or straight edges. That may or may not work but it is theoretically possible with this framework.

This technique to track motion relies on an initial silhouette being specified. This could be done with chroma key or difference calculations with a stationary camera or some other obvious technique. You can also try some fancy technique like segmentation. A motion history image (mhi) is created by setting the value of the output to the current time stamp. Subsequent frames continue this and the older images leave a fading trail of previous location ghosts. The function cvUpdateMotionHistory helps with this.

By taking the gradient of this motion history map, perhaps by using Scharr or Sobel techniques, the motion vectors can be ascertained. OpenCV has cvCalcMotionGradient to get the gradients and cvCalcGlobalOrientation to find the overall vector of motion, i.e. the sum of the gradient vectors. Thinking of things like tracking a single rotation invariant object like a billiard ball from a stationary overhead camera, this is enough. But if you want to track many balls, you’ll need to segment the motion profile with something like cvSegmentMotion. This kind of complex technology could be useful for gesture recognition, for example.

A Kalman filter is a mathematical process that forms ever better predictive models based on continual (though possibly discrete) input updates. It can be thought of as a type of sensor fusion as it can handle multiple indicators about a state. For example, GPS and odometry can be used as inputs and the more stable of these will contribute more (or something like that). In motion tracking this can take many input indicators of motion like the motion vectors calculated as above and integrate them to form a more stable and correct impression of the actual motion of the entire object of interest (not just its corners in isolation).

OpenCV provides a CvKalman data structure. It is created and released with cvCreateKalman and cvReleaseKalman. The iterative process of the Kalman cycle is executed with the functions cvKalmanPredict and cvKalmanCorrect (those two functions provide the best two word description of this complex technique).

The Kalman filter assumes that the uncertainty in the feedback is Gaussian. That need not be the case. If there is a known bias, the probability distribution can be represented as a density map (more dense "particles" represent greater likelihood). This map with an arbitrarily complex probability profile can be given to the the cvCreateConDensation function which works much like the Kalman functions.There is a CvConDensation struct like the CvKalman. The tricky bit to using this is that this confidence map of particles needs to be continuously updated to reflect known conditions. There is no automatic way to do this.

Here is an example of a 10x10 identity matrix with 32bit floating point numbers in one channel (grey maybe).

Here is a way to access a value at the third row and 3rd column. Should be 1.00000.

If you want multichannel, the elements will be of Vec type so you’ll get channels at a certain point like this.

Here is an example that calculates the longest element in a Mat. Actually, what it calculates is the sum of the squares of each channel and not really doing a great job remembering that.

Ways to extract subsets of Mat objects. Note that these really create objects which point to the correct locations of the same actual data. To get a new manifestation of the actual data look into the copy function.

Note that "style" is not an OpenCV word per se. * thickness - pixels thick as an integer. Or cv::FILLED for closed shapes. * lineType - Either 4 or 8 or cv::LINE_AA. 4 and 8 are like Minecraft placement where you can either only move orthoganally (4 possibilities) or on diagonals (8 possibilities). AA is anti-aliasing.

cv::putText()

All fonts are based on Hershey vector fonts.

Also cv::FONT_HERSHEY_PLAIN | cv::FONT_HERSHEY_ITALIC

cv::getTextSize() - calculates how big text would be. Only in Y?

cv::Mat cv::imread( const string& filename, int flags);

Default flags is cv::IMREAD_COLOR

three 8-bit channels.

The cv::IMREAD_ANYDEPTH flag can allow greater than 8-bit channels. The flag cv::IMREAD_UNCHANGED combines ANYCOLOR and ANYDEPTH to try to exactly match whatever is in the file no matter what it is.

imread looks at the file itself and does not trust any filename conventions like jpegimagesendwith.jpg.

If the read fails there is no error but the resulting image will be empty. You can check with cv::Mat::empty()==true.

This takes the file name string and an image array and writes the file. The final parameter is optional and is used to control things like PNG compression (default 3) or JPEG quality (default 95).

This opens a video file.

This opens a camera device. The first is 0.

I think that a device of 200 is the first V4L (video4linux) device, 201, the 2nd, etc. 300 is the first firewire device. 500 is the first QT device. But 0 is the first of any found (cv::CAP_ANY).

You can check that it opened properly and is ready to go with this.

To read a frame.

The image will be empty and the return false if the read did not go well or you have come to the end of the file, i.e. last frame. There’s an alias to the >> but that seems more, not less, complex to me. Speaking of complex, the read operation can also be broken down into a cv::VideoCapture::grab and a cv::VideoCapture::retrieve sequence. This can be handy to "grab" from multiple cameras as close to simultaneously as possible, then do more time consuming decoding on the input. This allows triangulation and stereo analysis to minimize error.

The cv::VideoCapture::get() and cv::VideoCapture::set() functions allow access to the metadata contained in some video file formats. Handy things to query could be…

There are many others, not all of which are reliably available for every codec.

First you need a writer object. This gives you a chance to set what the output properties will be like.

Here are typical settings showing how to update the object too.,

This creates an MPEG-4 codec, with 30 fps, 640x480 size, expecting only color. The CV_FOURCC function handles bit packing to form the weird codec codes used in this business. As with the read the cv::VideoWriter::write() funciton has an analogue with << which I also think is overly complex. But good to recognize.

OpenCV also has ways to write various non image objects to files in key/value formats. This could be handy for parameter sets. Basically look into this sort of thing.

This will open an object fs which will write objects as YML. It works something like this.

Retrieve the data in these files with something like this. (Didn’t check it.)

I am pretty sure XML is supported too.

GUI windows pretty much are for showing images. OpenCV does not return a window object as one would expect. This is how windows the lifecycle of a window is handled.

Instead of handling the window with an object, you refer to it by its name string. I think this is designed to be awkward enough so that people don’t start building truly fancy things with OpenCV. The flags can be 0 which means let users resize or it can be cv::WINDOW_AUTOSIZE which conforms to a loaded image automatically.

When the image is applied to the window it gets its own buffer so that changes to the window’s base image are not reflected until a new call to imshow.

If and only if a window is successfully open, the waitKey function will wait for a UI keypress event. The delay is the time in milliseconds until it gives up waiting and just proceeds onto the next command (0 means do not time out). If the function does time out it returns a -1.

Here a simple example that shows the file provided as an argument until the user presses escape (character 27). The file name is used to refer to the window and is displayed in its title bar.

For proper operating systems there is also this mysterious command.

This, in theory, allows OpenCV to start a separate thread for a windoow (each extant window?) allowing it to respond better even while other things are going on.

Working with mouse events requires a callback. The function must look something like this prototype.

Here event and flags are codes refering to these actions and circumstances.

The x and y coordinates are where the mouse action happened. I believe it is in the coordinate system of the image matrix, not the entire user desktop, but I’m not 100% sure. I also don’t know why the events and flags seem to have redundant coverage of the mouse buttons. The param argument is a pointer that allows this callback to pass back any kind of thing as a pointer.

Once you have a mouse action to perform ready in the callback function, you need to register that callback to hook it up to the proper window.

To apply a NxN kernel to an image it’s clear that at certain edge pixels N/2+1 of the kernel will hang off the edge. This can be handled by cropping down the final image. Or you can pad out the original image with a synthetic border. There are many modes in OpenCV to do this.

void cv::copyMakeBorder( cv::InputArray srcimg, cv::OutputArray dstimg, int t_pad, int b_pad, int l_pad, int r_pad, int borderType, const cv::Scalar& value=cv::Scalar() );

The value is for constant (e.g. all black or white or greenscreen) borders. The borderType can be one of the following.

To selectively apply a filter based on some kind of single per pixel operation, the cv::threshold function can do that. It takes a source and destination array and the threshold values.

The threshValue can also be cv::THRESH_OTSU. This will make the threshold() function try to find the optimal threshold value. This maximizes the variance between the separated sets of pixels. This is a relatively slow function.

There is also the adaptive threshold function. This is the same technique that boosts the performance so much in findChessBoardCorners.

The threshold value is computed on the fly for each pixel based on a weighted average of a (blockSize² - Constant) region around it. The adaptiveMethod parameter is either "mean" or "Gaussian" (cv::ADAPTIVE_THRESH_MEAN_C or cv::ADAPTIVE_THRESH_GAUSSIAN_C). Gaussian gives wieghted values while mean is constant for that region.

There are at least 5 different kinds of smoothing operation in OpenCV. The simplist is cv::blur().

This just takes a kernelSize kernel and computes the mean value within it. This is the value assigned to the new image at the location the kernel was centered on. Since the kernel is rectangular this is a generalization of a "box filter". The cv::boxFilter() function is a more general case of cv::blur which allows for unnormalized mode and an explicit control over output depth (blur reasonably assumes normalizing and same depth).

Here is another similar kind of blur which uses the median of the kernel’s contents rather than the mean as with simple blur. It seems to me to lose most detail, but if you want major component segmentation, it migth be ideal.

Ths GaussianBlur is a much nicer looking function which produces results that are what you’d expect with wearing the wrong glasses. Detail can be preserved while noise can be muted.

The sigmaX and sigmaY are "Gaussian half-width" in each direction. What that means is vague but I assume it has to do with the shape of the Gaussian probability distribution used, perhaps related to the variance. If you leave it at 0, it will, in theory, used optimized code. Also 3, 5, and 7 square kernels get special hand tuned opmization boosts. See Image Border Padding for how to handle borderType.

The other type of smoothing is bilateral filtering. You can think of this as Gaussian smoothing that weighs similar pixels more highly than less similar ones, keeping high-contrast edges sharp.

The d is distance to consider (sort of like a kernel, but I think it really computes expensive distances). Some sources call this a "diameter" which is a different thing but I guess it has the same rough effect when adjusting it. The sigmaColor is like the sigmas for Gaussian blur. The sigmaSpace affects what will be included in the smoothing; the larger it is, the more extreme a discontinuity must be to be preserved.

The least fancy edge detection is the Sobel derivative.

The ddepth can change to output’s format (to CV_8U) for example) which is handy since your image of the edges doesn’t need all the color info of the original. The kernel size should be odd but not exceed 31. The order is the derivative order, presumably 1st derivative (1) for normal gradients. Second order (2) is also an option and even zero, though I don’t know what that means exactly in this context. Scale and offset are applied to values before calculation to help put the result in a usable (visible maybe) range.

The Sobel filter has better performance the closer the edge aligns with horizontal or vertical. To help with that, for example to improve the accuracy of histogram of oriented gradients approaches, you can use the special kernelSize parameter of cv::SCHARR. This will create a (fast) 3x3 kernel that is more balanced for edges in any oreintation.

The cv::Laplacian() function (not to be confused with the Laplacian pyramid) is basically the cv::Sobel() with order set to 2. This can be useful for edge detection. It looks like it favors strong distinctions and ignores milder gradient changes.

OpenCV seems to provide pretty comprehensive support for a technique called "image pyramids". This is basically a stack of reduced images. Some thoughts about it.

Functions involved in this.

Make the next downsample image to build your own pyramid.

Make them all in one go.

The cv::OutputArrayOfArrays seems to be like an STL vector<cv::OutputArray> or perhaps even vector<cv::Mat>. There is a cv::pyrUp which uses the Laplacian stuff.

An affine transform is one where a rectangle can be turned into any kind of parallelogram. This can be done with a 2x2 transformation matrix. The other kind of transform is perspective which requires a 3x3 matrix and can produce any 4 sided shape from a rectangle.