Local Desktop Automation

Last update: 7 December 2015

Contents:
1. Introduction
2. Setting Up
3. Connection
4. Display Handling
5. Keyboard Handling
6. Starting Local Applications

1. Introduction

T-Plan Robot Enterprise version 3.2 introduced support of automation of applications displayed on the local desktop. This means that Robot and one or more Applications Under Test (AUT) run on the same local desktop. The automation is of course not limited to the AUT and it may perform any desktop action that a regular user can do.

Local desktop
environment

Advantages:

Single machine scenario with no need of another computer and/or OS.
It relies just on the local Java SE installation and it requires no other software to be installed. This allows to use Robot in non-intrusive scenarios where the target system may not be modified. In an extremely isolated scenario, the file based distributions of Robot and Java can be copied to an external media(CD, DVD, USB drive) and executed from there to keep the SUT free of any installation.
Support of multi-display environments (since v4.1).
Fast & easy operation.

Limitations:

Script recording is supported on MS Windows 2000 or newer (since version 3.5). Version 4.1 delivered support of Mac OS X and Linux.
Test script relying on the Local Desktop connection can not be started remotely through a terminal such as telnet or rsh. Use the VNC Server connection for such scenarios instead.

2. Setting Up

Local desktop automation is supported over two connection types:

The Local Desktop connection relies on the local Java SE installation. There are no other dependencies. To connect simply select the connection in the Login Dialog and hit Connect.
The local desktop can be also automated over a locally running VNC Server:

Install a supported VNC Server or enable the system built-in VNC access (Ubuntu, Mac OS X)
Select the VNC Server connection type in the Login Dialog and connect to localhost. If the server is configured to use a different port than the standard 5900 one use localhost:<port>, for example localhost:5901.

The Local Desktop connection relying on Java is the preferred one. It is in general more reliable than the VNC server and it can type non-English characters (see the keyboard support). The Local Desktop connection is also typically faster in terms of screen polling. The VNC server may provide better screen performance if it is attached directly to the video driver, such as for example UltraVNC with the Mirror Driver or TightVNC with the DFMirage driver.

Compatibility between the Local Desktop and VNC Server connections is not explicitly guaranteed. A test script designed against a local or remote VNC connection may or may not run correctly with the Local Desktop. The following factors apply:

The environments must match in terms of the OS and the desktop parameters (screen resolution, color depth). If the desktop parameters are different the template images may fail to compare and you will have to recreate them or add new alternatives using the Image Collections.
The Local Desktop connection supports a different set of keyboard keys. It can simulate a press of only those keys that are available in the current system keyboard layout. For example, if the computer uses the United Kingdom (GB) keyboard layout, the dollar key ('$') is not there. When Robot is instructed to press the key using the Press command it can't press it directly because the physical key associated with this character is not available. Robot in such a case uses the system clipboard to paste the character instead of pressing it on the keyboard. This will work well where the character is being typed as part of a text. It will however not work correctly where the AUT requires something like "Press the $ key to continue" and listens to the particular key press event.

Another difference is that while the VNC connection in general supports just characters from the ASCII 7-bit character set, the Local Desktop connection can transfer any national (UTF-8) characters supported by the system clipboard. It may for example type a Russian or Greek text. This allows to employ Robot for localization (l10n) or internationalization (i18n) testing.

3. Connection

To connect to the local desktop from a test script use the Connect command (TPR scripts) or the connect() method (Java test scripts). The argument URL must be in form of either "java://localhost" (Local Desktop connection type) or "rfb://localhost" (VNC Server).

For example, the following command/method call will connect you to the local desktop using the Local Desktop connection:

TPR test scripts: Connect java://localhost
Java test scripts: connect("java://localhost");

To connect to the local desktop using a VNC server running on the standard port of 5900:

TPR test scripts: Connect rfb://localhost
Java test scripts: connect("rfb://localhost");

The URL may be also used together with the -c/--connect CLI option to establish the connection on the Robot start up. On a successful connection Robot starts up with the Workflow chart displayed instead of the desktop viewer and the application title will indicate the connection (java://localhost or rfb://localhost:5900).

IMPORTANT: Do not change the OS keyboard layout while using the Local Desktop connection. It maps the layout when the connection is established and any later change may cause the test script press invalid keys. To remap the keyboard disconnect and reconnect. This may be performed even from a test script by a sequence of Disconnect and Connect commands (TPR format) or the disconnect() and connect() methods (Java format).

4. Display Handling

On single display environments the local desktop automation process differs from the remote one. When Robot needs to work with the local screen it must hide the GUI first. In the test script development phase it also applies to situations when the user chooses to perform image comparison, to take a screen shot, to define a screen location or a rectangle, to create a new template image or to perform Component Capture. Similarly when a script starts to execute the GUI gets hidden and it is not restored until the scripts finishes. To pause the script execution and restore the GUI move the mouse. The behavior can be changed through the Local Machine panel of the Preferences dialog. The options allow to set the mouse sensitivity for the script pausing purposes and the UI hiding behavior.

Robot v4.1.1 introduced full support of multi display environments. It is possible to automate a particular screen or all of them. When there are two or more displays the Login window displays a screen selector and a drop down with the options:

Screens are numbered starting from 1 where the first one is typically the primary one.

The most efficient set up is to automate one display and put the Robot GUI on the other one. This eliminates the need to hide the GUI on script executions and graphical operations.

To connect to a particular screen from a test script or from the CLI add the screen number to the connection URL as follows. When the number is omitted or is 0 the connection will automate all screens. This form is also suitable for the -c/--connect CLI connection. You may copy the URL of any of your recent connections from the Tools->CLI Wizard window. Another alternative is to give your connection a name in the Desktop->Connection Manager window and then use it instead of the URL.

TPR test scripts: Connect java://localhost?screen=1
Java test scripts: connect("java://localhost?screen=1");

IMPORTANT: For performance reasons the screen configuration is loaded just once on the Robot startup. To ensure correct operation do not change the screen layout or add/remove displays while Robot is running.

Since Robot 4.3.1 properties of the connected displays are exposed to test scripts through variables as follows:

The number of screens (displays) is available as _DISPLAY_COUNT
The [x, y, width, height] coordinates of each display is stored within the four variables of _DISPLAY_X_<n>, _DISPLAY_Y_<n>, _DISPLAY_W_<n> and _DISPLAY_H_<n> where <n> is the screen number. Numbering starts with one and the order is determined by the local system. If the connection is limited to a single display using the screen parameter its number is always 1 regardless of the ordinary number assigned to it by the system and displayed in the Login Dialog screen layout preview.
Image comparison commands such as CompareTo, Waitfor match/mismatch, Click and Drag populate a set of new variables to indicate the screen number the component or text was located on. These are _COMPARETO_DISPLAY_NO (display number of the topmost match with the number 1) and _COMPARETO_DISPLAY_NO_<i> where <i> is the ordinary number of the matching location. This allows to switch the behavior depending on the display involved. For example:

// Search for a component on all screens Compareto "mount.png" method= "search2" // Exit if not found if ({_EXIT_CODE} > 0) { Exit "1" } // Test the display the component is showing on if ({_DISPLAY_NO} == 1) { // Component on the first display -> do something } else { // Component on another screen -> do something else }

For a more generic description also see the table of implicit variables in the Var command specification.

5. Keyboard Handling

When the connection is being established it pops up a window showing the keyboard mapping progress. To ensure correct operation do not work with the desktop while it is displayed.

Version 4.1 introduced the stored keyboard maps (not applicable to the local VNC connection). Once the keyboard is mapped it gets saved to a file. The subsequent connections will reuse it as long as there is no change in the keyboard layout. This approach is faster and it avoids app focus issues on test suites executing a sequence of shorter test scripts.

The keyboard maps are by default stored to the user home folder. Should you need to manage the map(s) manually look for files with the tprlkm_ prefix and the .keymap extension. There's one file per each input language and keyboard layout. For example, the German keyboard layout will be stored as <home>/tprlkm_de_DE.keymap.
Robot will also accept any keyboard map file placed in its installation directory (the folder containing the robot.jar file). These maps must follow the above file name convention. They are preferred to the ones placed in the user home folder. This feature allows to create custom Robot based solutions.

Usage of stored maps is controlled through the localDesktop.useStoredKeymaps preference in the user configuration file. You may set it either in the Login window or in the Local Desktop panel of the Preferences window. Should you need to modify the option programmatically please read the Setting Preferences documentation.

6. Starting Local Applications

Version 3.5.2 extended the Exec command with the nowait and kill parameters to simplify automation of local applications over the Local Desktop connection. The basic pattern is that the script starts a local application, tests it and closes it at the script end. Example:

// Start myapp.exe and don't wait until it finishes Exec "myapp.exe"nowait="true"
...
// Test the application ...// Kill the application ('1' isordinary number of the process started with "Exec nowait=true") Exec kill="1"