T-Plan Robot Enterprise 4.3.1 Doc Collection

25/09/17

Automating Android Over The ADB

1. Introduction

2. Setting Up

1. Download and install the Android SDK revision 18 or higher on your computer:
1. The Android command line tools release is sufficient for the connection to succeed.
2. For eventual debugging or CLI interaction with the device you'll need the adb tool which is only in the full Android Studio release.
   
3. If you have and older version of the SDK upgrade it to the latest one using the <SDK_path>/tools/android tool.
2. Enable the "USB Debugging" option on your Android device:

• Android 4.1 and lower: Go to Menu->Settings->Applications->Development and set on the "USB Debugging" option.
• Android 4.2 and higher:
• Go to Menu->Settings
• Select the About menu in the More or General group. On some devices it is located under Settings->About->Software->Information->More.
• Scroll to Build Number and tap it 7 times until you see a message saying “You are now a developer” or ”Developer mode has been enabled”.
• Go back to the Settings menu, select the Developer Options and turn on the slider at the top to enable the options. Then set on USB Debugging.

3. Connect the Android device through the USB cable to the computer. It is not recommended to connect the device after the connection is initiated from Robot because it leads to intermittent connection crashes.
   
4. MS Windows only: If you installed the full Android Studio add the folder containing adb.exe to the user or system path:
1. Open Windows Explorer (Windows+E)
   
2. Right click Computer and select Properties
   
3. Select "Advanced System Settings" and click the "Environment Variables" button 
4. Select the PATH user variable, click Edit and insert the path to the folder containing adb.exe to the beginning of the list. To separate the path from the original value use semicolon, ';'. The new value should look like:
    
   <SDK_path>\platform-tools;<originalPaths>
   
   
5. Save with OK.
   
5. Start T-Plan Robot Enterprise. If your system is a Linux/Unix one you will have to start it from the root account (sudo may not be sufficient).
   
6. In the Login Window, set the "Connection Type" to "Android over ADB" and provide the path to the Android SDK. This path will be saved to the user preferences and it will get populated automatically onwards.
   
7. If you have two or more Android devices connected to the computer, set the "Device Serial No." field to the serial number of your device. You may find it out through clicking of the "..." button or by executing "adb devices" from the platform-tools/ folder of your Android SDK. Alternatively leave the field on "default" to connect to the first detected device.
8. T-Plan Robot Enterprise version 4.2 or higher: Select the "Screen transfer method":
• The "ADB (default)" option is the slowest one but requires no additional application on the device. As the screen data is transferred over the USB cable the device doesn't have to be connected to the network.
• The "Screen Stream Mirroring" option will mirror the screen using the H.264 video stream. See the next paragraph for installation and configuration instructions.
  
9. Hit Connect to connect to the device.

1. Install the Screen Stream Mirroring application on the Android device. The trial version is OK for evaluation purposes (it displays ads and shuts down after a few minutes).
2. Start the app to verify it works. If you are connected to a network it must display an address like "rtsp://<IP_address>:5000/screen". If the displayed URL is like "http://localhost:5000/screen" or other than RTSP perform the following steps:
1. Tap the top left button 
2. Select Media Players
3. Switch back and make sure the URL has changed appropriately
   

3. No other steps are required if your device is connected to a network visible to Robot and you prefer the screen data transfer over TCP/IP. There's no need to start the app manually prior to connecting.
   
4. Should you want to avoid the network and set up the USB screen data transfer:
1. Start the app and tap the top left button to open the app menu. Choose Preferences.
2. Scroll down to the bottom and set on "Advanced settings".
3. Set on the "Stream through USB" option and tap the top left button to save the changes. The main screen must display the address of "rtsp://localhost:5000/screen".

• Some machines are equipped with both the USB 2.0 and 3.0 ports. Make sure that the device is plugged to a 3.0 one.
• When the screen mirror is used for video transfer the data travels over the local network. The network speed affects the performance.
• Decoding of the mirror stream relies on the CPU. The graphic card (GPU) plays a very minor role. Upgrade the CPU to boost the performance.
• The RAM is not a factor as long as there is enough of it. According to our performance stats Robot mirroring a full-HD Android device needs only 50-60 MB of RAM in the idle state. The memory may double or triple in peaks while executing a test script with image comparison where copies of the screen image are being processed. Unless the machine is overloaded or there is little free RAM it should not be a problem. To verify it please observe:
• Check the memory usage indicator at the right bottom corner of the Robot GUI. If it's in green and occasionally in yellow the program has enough RAM. If the usage is high please raise the limit as is described here.
• Start the Windows' Task Manager and check the CPU and RAM usage. If it's too high reduce the number of other programs and/or services running on the machine. 

If the video is slow or lags behind considerably:

• Stop Stream Screen Mirroring as well as any unnecessary applications running on the device and reconnect.
• Go to Edit->Preferences in the Robot menu. Select the Android Over ADB screen and raise the parameter called "Process only every <N> video frame" option. This will decrease the load and RAM required on the Robot machine. Animations and fast motion scenes will be blurred but still screens will retain the original image quality. Image comparison searching the screen for stable displayed components will not be affected.
• As the very last resort modify the video quality parameters in the Screen Mirroring Application. Be aware that a major decrease of the video quality is likely to break image comparison. Tap the gear wheel to the right from the "Mirroring available at:" label (see the above image) and choose "Video Preferences":
• The Resolution is set to 640p. Any resolution update will lead to the screen size change which will invalidate all component images created so far. Avoid changing of this parameter when you already have some automation in place.
  
• The Encoding bitrate is typically 2048 kb/s. You may decrease it to save resources but don't set less than 1024 kb/s.
• The Max framerate is typically set to 30 fps. This limit is rarely achieved in real environments. You may not see any effect unless you reduce it to 2-5 fps. It is recommended to leave the default value.
  
• Once set exit the preferences using the top left "back" arrows. If mirroring is on it will restart to apply the changes. You will have to reconnect then.
  

3. Troubleshooting

• Should you experience any issues please set on the Debug mode to produce logs. These can be viewed in the console (command prompt) or the log file through the Help->Log Viewer window.
  
• When the device screen image is distorted or displays in wrong colors change the decoding method:
• The default "Android SDK" decoder relies on the Android SDK to decode the image data into the Java format. This approach is known to fail for some devices, for example Samsung Galaxy GT-I5500 which displays with a greenish screen.
  
• The "Robot" one decodes the data on its own. This method is slightly slower but seems to work for most failing devices.
• On unresolvable screen issues set on the "Create the screen image dump" option, make an attempt to connect to create the dump and then send the log file through the Help->Log Viewer window to T-Plan support. Then set off the option. The dump will allow us to reconstruct the image from the raw pixel data and inspect why the decoder fails.

• Disconnect the device from the USB cable and connect it again.
• Restart the ADB server manually:
• Open a console/command prompt and switch to the platform-tools folder under the Android SDK installation path.
• Run:  adb kill-server
• Verify that the device is connected to ADB:  adb devices
  

4. Automation

4.1 Connecting To The Device

TPR test scripts:     Connect adb://MB104PY10519 screen=mirror
Java test scripts:    connect("adb://MB104PY10519", "screen", "mirror");

TPR test scripts:     Connect adb://MB104PY10519?screen=mirror
Java test scripts:    connect("adb://MB104PY10519?screen=mirror");

4.2 Dealing With Screen Rotation

Correct view		Incorrect view Search for components using the image comparison will not work and the touch screen events will be misplaced.

4.3 Touch Screen Automation

• To set on the UIAutomator driven mouse clicks globally select the "Enable click/tap over the UIAutomator API" user preference in the Edit->Preferences->Android Over ADB screen. As this gets saved to the user preferences file it will affect all Robot instances running on the local PC. The option may be tweaked for a single Robot instance through the -o/--option CLI option.
• To control the UIAutomator clicks temporarily from the script using Robot 4.0.1 or higher set the _UIACLICK variable to true ("enable") or false ("disable"). This mechanism overrides the above user preference. It has no impacts on other test scripts executing concurrently within the same Java process. For example, to enable it just for one particular click use:
  
  
  Var _UIACLICK=true
Mouse "click"
Var _UIACLICK=false
  

4.4 Keyboard Automation

• All printable ASCII 7-bit characters (codes 32-127) and part of the keys from the 0-31 range (Enter, Backspace, ...) are supported except for greater than (>), less than (<), double quote (") and circumflex (^) which don't have their counterparts in the Android SDK API.
• The only supported modifier keys are Shift and Alt. The Ctrl modifier is not supported because it doesn't have its counterpart in the Android SDK API.
• Some characters out of the ASCII 7-bit set can be generated through combinations of Alt+<character>. For example "Press Alt+Z" generates the Euro character on some devices. These composed keys are subject to the device, its language and keyboard settings and they must be explored per device.
• The PC keyboard action keys are mapped onto the native Android keys as follows. Support of these keys is subject to the particular Android device and some keys may not have any effect: