Tolerant Image Search

The search method (algorithm) allows to locate occurrences of the template image on the desktop and to return location of upper left corner of the match area(s). It may be efficiently used to verify that a certain component is displayed on the screen and eventually use its coordinates, for example to place a mouse click on a GUI component. For a complete specification of algorithm parameters see the Image Search specification.

The algorithm provides several levels of tolerance. The first one is the pass rate parameter provided with the command. The default value of 100% searches for an exact pixel match. Lower pass rate values then define relative number of pixels which may be different. For example when the template is 10x10 (=100pixels) and the pass rate is set to 95%, the method will search the desktop image for all areas which have at least 95 pixels equal to the template ones. Be aware that the method performance decreases with the pass rate.

The second level of tolerance relies on image transparency. If the template contains transparent (the Alpha byte is set to zero) or translucent (Alpha < 255) pixels, they are not compared during image search and they are automatically considered to match. This allows to design templates which express just a certain pattern, for example a fragment of text, with no dependency on the surrounding objects and background color.

T-Plan Robot Enterprise version 2.1 and higher supports automatic background transparency through the removebg, bgcolor and minalpha parameters. When the feature is on ( removebg=true), the image search algorithm strips all background color pixels from the template image and searches just for the solid color areas of the component. This allows to build search tasks for objects which may appear on various backgrounds. The background color defaults to the very first template image pixel which may be optionally overriden through the bgcolor parameter.

Image transparency may be also physically elaborated into the image with third party tools (such as Correl Draw on Windows or Gimp on Windows/Linux/Unix). To see an example watch the video below.

The last level of tolerance deliveredby v2.2 is called RGB variance tolerance (parameter tolerance). It is based on RGB distance. It is a number between 0 and 255 specifying how much the Red, Green and Blue components of a desktop pixel color may differ at a maximum to consider it to be equivalent to the corresponding template pixel. This value allows to deal with images whose pixels are changing slightly, for example as a result of blurring, highlighting or merging of the image with the background. This functionality is essential for Flash applications where decorative texts and even some images are not being rendered in a constant way. Be aware that the higher the tolerance value, the higher the probability of false matches is. In most scenarios the value should be in the [0, 100] range depending on the scope of color changes. If the tolerance parameter is not specified, it defaults to zero (unless the default value was changed in the image search preferences) and the algorithm compares pixels using exact color match which is compatible with previous versions. One of the videos below demonstrate how to use the RGB tolerance to deal with button highlighting.

Commands performing image search always return either 1 (meaning "not found") or 0 (meaning "at least one match was found"). This value is accessible through the _EXIT_CODEvariable after the command finishes. Number of match locations as well as their coordinates are then provided in form of context variables ( _SEARCH_MATCH_COUNT, _SEARCH_X, _SEARCH_Y).

To understand better the whole process of how to create a template image and create a command with image search watch the following video examples:

Image search example in the calculator script on Windows XP		Clicking on a component located through image search on Windows XP		Transparent template search example on Ubuntu Linux		RGB color tolerance example on Ubuntu Linux

1. Use image search to suspend script execution until a template image appears on the screen. If the image search keeps failing for more than 30 seconds, take a screen shot and terminate the script with exit code of 1.
   
   Waitfor match template="mytemplate.png" method="search" timeout=30s
if ({_EXIT_CODE} > 0) {
  Screenshot "failure.png" desc="Failed to find mytemplate.png"
  Exit 1
}
   
   
2. Search the desktop for the OK button and click on it. If the button is not found, terminate the script with exit code of 5.
   
   Compareto buttonOK.png method="search" onfail="Exit 5"
Mouse click to=x:{_SEARCH_X},y:{_SEARCH_Y}