T-Plan Robot Enterprise 4.4.7 Doc Collection

29/01/19

Automating Android Over The ADB

1. Introduction

2. Setting Up

2.1 PC Set Up

1. Download and install the Android SDK revision 18 or higher on your computer:
• If you have and older version of the SDK upgrade it to the latest one using the SDK Manager tool located in the SDK home folder.
2. MS Windows only: If you installed the full Android Studio add the folder containing adb.exe to the user or system path:
1. Open Windows Explorer (Windows+E)
   
2. Right click Computer and select Properties
   
3. Select "Advanced System Settings" and click the "Environment Variables" button 
4. Select the PATH user variable, click Edit and insert the path to the folder containing adb.exe to the beginning of the list. To separate the path from the original value use semicolon, ';'. The new value should look like:
    
   <SDK_path>\platform-tools;<originalPaths>
   
   
5. Save with OK.

IMPORTANT: As Java 9 introduced new security restrictions and disabled loading of Java libraries (JARs) on the fly it is necessary to update the class path:

1. Find out your Java version through one of the following options. Be aware that versions 6, 7 and 8 identify themselves as 1.6, 1.7 and 1.8.
   
• Open a command prompt (terminal) and execute:  java -version
• Start Robot and select Help->About
2. If your Java is version 9 or higher edit the Robot starting command and add full paths of the following libraries to the -classpath list:

MS Windows: <SDK_path>\tools\lib\sdklib.jar;<SDK_path>\tools\lib\guavalib.jar;<SDK_path>\tools\lib\ddmlib.jar;<SDK_path>\tools\lib\chimpchat.jar


Unix/Linux:    <SDK_path>\tools\lib\sdklib.jar:<SDK_path>\tools\lib\guavalib.jar:<SDK_path>\tools\lib\ddmlib.jar:<SDK_path>\tools\lib\chimpchat.jar


For example if your SDK is installed to C:\Android Studio\SDK the complete Robot start command must look like:

cd /Applications/TPlanRobot.app/Contents/Java
ln  <SDK_path>/tools/lib/sdklib.jar  sdklib.jar
ln  <SDK_path>/tools/lib/guavalib.jar  guavalib.jar
ln  <SDK_path>/tools/lib/ddmlib.jar  ddmlib.jar
ln  <SDK_path>/tools/lib/chimpchat.jar  chimpchat.jar


2.2 Android Device Set Up

1. Enable the "USB Debugging" option on your Android device:

• Android 4.1 and lower: Go to Menu->Settings->Applications->Development and set on the "USB Debugging" option.
• Android 4.2 and higher:
• Go to Menu->Settings
• Select the About menu in the More or General group. On some devices it is located under Settings->About->Software->Information->More.
• Scroll to Build Number and tap it 7 times until you see a message saying “You are now a developer” or ”Developer mode has been enabled”.

2. Go back to the Settings menu, select the Developer Options and turn on the slider at the top to enable the options:

• Set on USB Debugging.
• Set on OEM Unlocking  if it's present.
• Set the USB Configuration option to PTP (Picture Transfer Protocol). The device is unlikely to connect if this option is set to MTP (Media Transfer Protocol).
• It is recommended to set on the Stay Awake option to prevent screen locking while the device is connected to the PC.

2.3 Connecting To Robot

1. Connect the Android device through the USB cable to the computer. It is not recommended to connect the device after the connection is initiated from Robot because it leads to intermittent connection crashes.
   
2. Start T-Plan Robot Enterprise. If your system is a Linux/Unix one you will have to start it from the root account (sudo may not be sufficient).
   
3. In the Login Window set the "Connection Type" to "Android over ADB" and provide the path to the Android SDK. This path will be saved to the user preferences and it will get populated automatically onwards.
   
4. If you have two or more Android devices connected to the computer set the "Device Serial No." field to the serial number of your device. You may find it out through clicking of the "..." button or by executing "adb devices" from the platform-tools/ folder of your Android SDK. Alternatively leave the field on "default" to connect to the first detected device.
5. T-Plan Robot Enterprise version 4.2 or higher: Select the "Screen transfer method":
• The "ADB (default)" option is the slowest one but requires no additional application on the device. As the screen data is transferred over the USB cable the device doesn't have to be connected to the network.
• The "Screen Stream Mirroring" option will mirror the screen using the H.264 video stream. See the next paragraph for installation and configuration instructions.
  
6. Hit Connect to connect to the device. If the connection fails refer to the Troubleshooting chapter.
   

2.4 Screen Stream Mirror

1. Install the Screen Stream Mirroring application on the Android device. The trial version is OK for evaluation purposes (it displays ads and shuts down after a few minutes).
2. Start the app to verify it works. If you are connected to a network it must display an address like "rtsp://<IP_address>:5000/screen". If the displayed URL is like "http://localhost:5000/screen" or other than RTSP perform the following steps:
1. Tap the top left button 
2. Select Media Players
3. Switch back and make sure the URL has changed appropriately
   

3. No other steps are required if your device is connected to a network visible to Robot and you prefer the screen data transfer over TCP/IP. There's no need to start the app manually prior to connecting.
   
4. Should you want to avoid the network and set up the USB screen data transfer:
1. Start the app and tap the top left button to open the app menu. Choose Preferences.
2. Scroll down to the bottom and set on "Advanced settings".
3. Set on the "Stream through USB" option and tap the top left button to save the changes. The main screen must display the address of "rtsp://localhost:5000/screen".

2.5 Performance Considerations

• Some machines are equipped with both the USB 2.0 and 3.0 ports. Make sure that the device is plugged to a 3.0 one.
• When the screen mirror is used for video transfer the data travels over the local network. The network speed affects the performance.
• Decoding of the mirror stream relies on the CPU. The graphic card (GPU) plays a very minor role. Upgrade the CPU to boost the performance.
• The RAM is not a factor as long as there is enough of it. According to our performance stats Robot mirroring a full-HD Android device needs only 50-60 MB of RAM in the idle state. The memory may double or triple in peaks while executing a test script with image comparison where copies of the screen image are being processed. Unless the machine is overloaded or there is little free RAM it should not be a problem. To verify it please observe:
• Check the memory usage indicator at the right bottom corner of the Robot GUI. If it's in green and occasionally in yellow the program has enough RAM. If the usage is high please raise the limit as is described here.
• Start the Windows' Task Manager and check the CPU and RAM usage. If it's too high reduce the number of other programs and/or services running on the machine. 

If the video is slow or lags behind considerably:

• Stop Stream Screen Mirroring as well as any unnecessary applications running on the device and reconnect.
• Go to Edit->Preferences in the Robot menu. Select the Android Over ADB screen and raise the parameter called "Process only every <N> video frame" option. This will decrease the load and RAM required on the Robot machine. Animations and fast motion scenes will be blurred but still screens will retain the original image quality. Image comparison searching the screen for stable displayed components will not be affected.
• As the very last resort modify the video quality parameters in the Screen Mirroring Application. Be aware that a major decrease of the video quality is likely to break image comparison. Tap the gear wheel to the right from the "Mirroring available at:" label (see the above image) and choose "Video Preferences":
• The Resolution is set to 640p. Any resolution update will lead to the screen size change which will invalidate all component images created so far. Avoid changing of this parameter when you already have some automation in place.
  
• The Encoding bitrate is typically 2048 kb/s. You may decrease it to save resources but don't set less than 1024 kb/s.
• The Max framerate is typically set to 30 fps. This limit is rarely achieved in real environments. You may not see any effect unless you reduce it to 2-5 fps. It is recommended to leave the default value.
  
• To apply the preferences leave the screen using the top left "back" arrows. If mirroring is on it will restart to apply the changes. You will have to reconnect then.
  

3. Troubleshooting

• Should you experience any issues please set on the Debug mode to produce logs. These can be viewed in the console (command prompt) or the log file through the Help->Log Viewer window.
  
• When the device screen image is distorted or displays in wrong colors change the decoding method:
• The default "Android SDK" decoder relies on the Android SDK to decode the image data into the Java format. This approach is known to fail for some devices, for example Samsung Galaxy GT-I5500 which displays with a greenish screen.
  
• The "Robot" one decodes the data on its own. This method is slightly slower but seems to work for most failing devices.
• On unresolvable screen issues set on the "Create the screen image dump" option, make an attempt to connect to create the dump and then send the log file through the Help->Log Viewer window to T-Plan Ltd support. Then set off the option. The dump will allow us to reconstruct the image from the raw pixel data and inspect why the decoder fails.

• Disconnect the device from the USB cable and connect it again.
• Restart the ADB server manually:
• Open a console/command prompt and switch to the platform-tools folder under the Android SDK installation path.
• Execute:
  

adb kill-server
adb devices


• Make sure the device is listed in the output. If it is not there your environment is not configured properly:
• Make sure the USB Debugging and other options are enabled as is described in the Setting Up chapter
• Some devices may require specific USB drivers to enable communication with the Android Studio, for example Samsung Galaxy drivers. Search documentation provided by your device vendor for details. Restart your PC after driver installation and reconnect your device even if it is not required.
  
• Should everything fail retry with another USB cable.
  

4. Automation

4.1 Connecting To The Device

TPR test scripts:     Connect adb://MB104PY10519 screen=mirror
Java test scripts:    connect("adb://MB104PY10519", "screen", "mirror");

TPR test scripts:     Connect adb://MB104PY10519?screen=mirror
Java test scripts:    connect("adb://MB104PY10519?screen=mirror");

4.2 Dealing With Screen Rotation

Correct view		Incorrect view Search for components using the image comparison will not work and the touch screen events will be misplaced.

4.3 Touch Screen Automation

• To set on the UIAutomator driven mouse clicks globally select the "Enable click/tap over the UIAutomator API" user preference in the Edit->Preferences->Android Over ADB screen. As this gets saved to the user preferences file it will affect all Robot instances running on the local PC. The option may be tweaked for a single Robot instance through the -o/--option CLI option.
• To control the UIAutomator clicks temporarily from the script using Robot 4.0.1 or higher set the _UIACLICK variable to true ("enable") or false ("disable"). This mechanism overrides the above user preference. It has no impacts on other test scripts executing concurrently within the same Java process. For example, to enable it just for one particular click use:
  
  
  Var _UIACLICK=true
Mouse "click"
Var _UIACLICK=false
  

4.4 Keyboard Automation

• All printable ASCII 7-bit characters (codes 32-127) and part of the keys from the 0-31 range (Enter, Backspace, ...) are supported except for greater than (>), less than (<), double quote (") and circumflex (^) which don't have their counterparts in the Android SDK API.
• The only supported modifier keys are Shift and Alt. The Ctrl modifier is not supported because it doesn't have its counterpart in the Android SDK API.
• Some characters out of the ASCII 7-bit set can be generated through combinations of Alt+<character>. For example "Press Alt+Z" generates the Euro character on some devices. These composed keys are subject to the device, its language and keyboard settings and they must be explored per device.
• The PC keyboard action keys are mapped onto the native Android keys as follows. Support of these keys is subject to the particular Android device and some keys may not have any effect: