T-Plan Robot Enterprise 4.0.2 Doc Collection

10/06/15

iOS Mirror Automation

1. Introduction

T-Plan Robot Enterprise version 3.3 introduced a new connection type called "iOS Mirror". This allows Robot to test Apple iOS devices using our iOS 7&8 Plugin, or iOS 6 Server WiFi connection, where the device screen is displayed on the local desktop using Apple's 'AirPlay Mirroring' technology. This connection allows Robot to test iOS applications on non-jailbroken (non-rooted) devices, where before only a VNC Server connection was supported.
In environments where jailbroken devices are supported, the iOS Mirror connection can be used to supplement the VNC Server (Veency) operation, for increased performance and suitability. E.g. Game testing relying on the OpenGL technology.

It is recommended to use the T-Plan iOS 7&8 Plugin or iOS 6 Server, instead of VNC due to a number of advantages:

	iOS Mirror connection with T-Plan Plugin / Server	iOS Mirror connection with VNC Server (Veency)	VNC Server connection with VNC Server (Veency)
Supported since	Robot v3.5	Robot v3.3	Robot v2.0
Jailbroken device required	NO	YES	YES
Support of apps with OpenGL content	YES	NO	NO
Pinch/Zoom support	YES	NO	NO
UTF-8 character support	YES	NO*	NO*
Display orientation support	YES	LIMITED**	NO
Connections per machine	1***	1***	NO LIMIT***

* The RFB (VNC) protocol is limited to ISO 8859-1 characters.
** Only the portrait & counterclockwise landscape modes are supported.
*** One machine running Robot can automate only one iOS device over the iOS Mirror connection at a time because it relies on an exclusive access to the local desktop. The standard VNC Server connection doesn't apply any limit to the number of connections and one Robot machine may automate any number of iOS devices at a time up to the limit given by the license key as long as there are sufficient system resources (CPU, RAM).

Device Requirements:

• An Apple iOS device capable of screen mirroring. See the online iOS Compatibility Status page for the latest information. A connection to the network is also required (usually over WiFi).
  
• For the iOS 7&8 Plugin or iOS 6 Server driven testing, the device must have iOS 5.0 or higher. This option allows testing on Non-Jailbroken (non-rooted) devices.
  
• For the VNC driven testing, the device must be jailbroken and Veency must be installed and running on it. See the online iOS Compatibility Status page for the latest jailbreaking status.

Robot Machine Requirements:

• Operating system:
• Any MS Windows supporting window translucency (in general, Windows Vista and later).
• Mac OS X 10.7 (Mountain Lion) or higher (supported since Robot v3.4.1).
  
• AirServer, Reflector or X-Mirage (supported since Robot 4.0.1) installed and activated ("the mirror app"):
• MS Windows:
• AirServer is recommended for single device deployments because it seems to be more stable and faster than other products. It is not suitable for automation of 2+ devices on a single PC desktop because it displays all devices in a single window.
  
• X-Mirage or Reflector can be used for multiple device deployments because they display one window per device.
  
• Mac OS X:
• Reflector is the best option for Mac OS X. X-Mirage is another option.
  
• Though Robot will also recognize AirServer it is not suitable for automation for practical reasons. It starts by default in the full screen mode which is not supported by Robot. It also uses pop up tool and title bars which interfere in the screen.
  
• Java 1.7 or higher. The connection will be disabled when Robot runs on Java 1.6. It is highly recommended to use Java from Oracle which is also the only option for Mac OS X.
  
• It is recommended to use a display with a resolution sufficient to fit the device screen both in portrait and landscape modes (iPhone 4S screen size: 640x960, iPad 2: 1024x768). Environments which do not comply with this requirement may take advantage of the mirror app scaling (right click Reflector -> Scale -> "Half Size" or "Actual Size").
  

2. Robot Machine Configuration

MS Windows Set Up

• Install Java JDK 1.7 from Oracle and put it on the system path as is described in the Release Notes document.
• Install Robot v3.5 or later.
• Install AirServer, Reflector or X-Mirage (supported since Robot 4.0.1) and activate the product. The trial version is OK for evaluation purposes. Do not customize the target install folder to enable automatic mirror starting (since Robot 4.0.1).
  

Mac OS X Configuration

• Install Java JDK 1.7 from Oracle.
  
• Enable accessibility - Mac OS X version greater than 10.9.x (Mavericks):
1. Click the Apple menu and select System Preferences. Alternatively click the System Preferences icon in the dock.
2. Click the Security & Privacy icon, and select the Accessibility icon.
3. Unlike previous versions of OS X, which used a universal checkbox, the new functionality in Mavericks allows users to individually choose which apps can gain control of the system to perform their various scripted functions.
   If you do not see the "T-PlanRobot.app" (like in the screenshot below) or the "java" app (standalone Robot release), please attempt to run T-Plan Robot at least once, before it appears in this window.
   Note: It will be necessary to click the padlock at the lower left hand corner of the window, to authenticate as an administrative user, before you can select the check box next to the "T-PlanRobot.app" item.
   

• Enable accessibility - Mac OS X version less than 10.9.x (Mavericks):
  1. Click the Apple menu and select System Preferences. Alternatively click the System Preferences icon in the dock.
  2. Double click the blue Accessibility icon under the System group.
  3. Set on the Enable access for assistive devices check box at the bottom of the window.

• Install Robot v3.5 or later. Make sure to choose the distribution packaged as Mac OS X application for Oracle Java 1.7. Alternatively use the standalone cross platform release.
• Install Reflector or X-Mirage (supported since 4.0.1) and activate it.
  

AirServer / Reflector Set Up

While the mirror app is disconnected right click its icon in the system tray (Windows) or go to the Device menu (Mac OS). Select Preferences/Settings. Set the options as follows:

• Full screen - OFF (Reflector only)
  
• Always on top - ON (recommended)
  

• Connect the device to the same network the Robot machine is connected to (usually over the WiFi).
• Follow the AirServer / Reflector instructions.

• Reflector only: right click the app (Windows) or select the Device menu (Mac OS).  Set off the Show Frame option to make it display just a plain screen without the white/black iPhone hard cover image.
• Choose the mirror size (Scale/aspect) carefully and keep it constant. It is recommended to set the mirror size equal to the actual device screen one as long as it fits your screen. This keeps the image blurring caused by the lossy H.264 compression at a minimum.

Robot Set Up

• Start Robot and verify the following:
• Robot must be running on the default Graphite Look And Feel (LAF) scheme. The standard Oracle delivered LAFs such as Metal, Nimbus, CDE, GTK+ or Windows/Windows Classic will not work. Other ones supported by Robot may or may not work. To verify or change the LAF either click the More... link in the Login Dialog or navigate to Edit->Preferences and select the Appearance & Accessibility node.
• Open the Help->About dialog and verify that Robot runs on Java 1.7.
  

3. iOS Device Configuration

As detailed in the Introduction there are two basic high level options:

1. Non-Jailbroken (non-rooted) devices can be automated using our T-Plan Plugin or T-Plan Server tool with the iOS Mirror connection.
   Note: The iOS Mirror connection can be applied to any device supporting the AirPlay Mirroring technology (as per this table).
   
2. Jailbroken (rooted) devices can be automated using the Veency (Cydia) VNC Server connection.

A. Non-Jailbroken Devices

1. Device Configuration

It is recommended to set off the Auto Lock feature to prevent the device from going to sleep after the specified idle time:

• Tap the Settings icon in the iOS home screen.
• Go to General -> Auto Lock and select Never.

2. Configure Device with T-Plan iOS 7&8 Plugin or iOS 6 Server

The T-Plan iOS 6 Server enables testing of iOS applications on iOS version 5 & 6, using the iOS Mirror connection.

• This will give you control over the whole device OS.

The T-Plan iOS 7&8 Plugin enables testing of iOS applications on iOS version 7 and higher, using the iOS Mirror connection.

• This will give you control over the application under test.
  
• As Apple tightened the security in iOS version 7, it is not possible to control the touch screen and keyboard at the OS level any more. The only known way of automation is to add the automation capability (the T-Plan library) to the application under test (AUT). This will allow you to automate the application after it gets started.
  • I.e. Currently Robot cannot control the whole iOS, or any other functionality outside of the application, like it did using the iOS 6 Server method.

3. Start Airplay Mirroring

Automation of the Apple iOS device via T-Plan Robot Enterprise relies on Airplay Mirroring. Learn more about Airplay Mirroring.

• If you run Robot 4.0.1 or higher and the iOS application you are automating contains the T-Plan iOS 7 Plugin v0.7 or higher you don't need to do anything. Robot will start the mirroring of the iOS device to the PC screen automatically.
• For Robot 4.0 and/or the plugin v0.6 and lower you must start mirroring manually on the iOS device.
  

B. Jailbroken Devices Only

1. VNC Setup (Only for Jailbroken devices)

• Jailbreak the device.
• Install Veency from Cydia.
  
• Refer to the Release Notes document for Veency set up instructions.

2. Start Airplay Mirroring

Automation of the Apple iOS device via T-Plan Robot Enterprise relies on Airplay Mirroring. Learn more about Airplay Mirroring.

• If you run Robot 4.0.1 or higher and the iOS application you are automating contains the T-Plan iOS 7 & 8 Plugin v0.7 or higher you don't need to do anything. Robot will start the mirroring of the iOS device to the PC screen automatically.
• For Robot 4.0 and/or the plugin v0.6 and lower you must start mirroring manually on the iOS device.
  

4. Automation

Connection from Robot

• If you run Robot 4.0 or lower and/or the iOS application you are automating contains the T-Plan iOS 7&8 Plugin v0.6 or lower please make sure that you have invoked Airplay Mirroring on the device. Learn more about Airplay Mirroring.
• The device must be connected to the network over the WiFi.
  
• For iOS 5 & 6, check if the T-Plan Server is running on the device and start it eventually. When it asks you for access to the current location approve it. If you accidentally select "Don't Allow" please enable it under iOS Settings ->Privacy->Location Services.
• For iOS 7 and higher make sure that the application with the iOS 7&8 Plugin is running and displayed on the device screen. The connection will not work if the application is not running or if it has been pushed to the background.
• Start Robot, select the "iOS Mirror"connection type, provide the device IP address and the port and hit Connect. For the T-Plan iOS 6 Server or the iOS 7&8 Plugin connection use the IP address and the port of 5909 displayed in the server window. For the VNC Server connection use the device IP address with the port of 5901. If neither AirServer nor Reflector is currently showing the device screen, Robot will wait for it. On a successful connection Robot will display the mirrored screen in the desktop viewer.

• To connect to a device from a test script use the Connect command (TPR scripts) or the connect() method (Java test scripts). The argument URL must be in form of "apple://<address>:<port>". For example, the following lines will connect you to an Apple device with the IP of 192.168.100.1 running the T-Plan Server:

Automation of multiple devices from a single PC

Robot v4.0.1 introduced support of testing of 2 or more devices from a single PC. It is supported only for X-Mirage. The steps:

• All mirrors must be up and running prior to the Robot start.*
• Make sure to have a license with 2+ seats (one seat per each connection). Insufficient number of seats will make the exceeding scripts wait for a free connection (sequential execution).
• To achieve parallel testing execute a schedule with two or more parallel scripts in the CLI mode. Alternatively start multiple Robot instances in the GUI mode and initiate the testing as usual (one instance per one test script).
  

*If the mirrors are not running Robot will start them (Robot v4.0.1+ with T-Plan libs v1.0+). This will however make image comparison unreliable because start of each new mirror will make X-Mirage resize all other mirrors to fit the PC screen.

Troubleshooting

• Robot maintains itself as the topmost window with the mirror application positioned behind its desktop viewer. If the mirror application is not configured to stay always on top, Robot may fail to bring it into the desktop view after the connection. To overcome this click the mirror app icon in the Windows task bar to push it to the front and then click the Robot's title bar to lock the application in the viewer.
  
• Robot identifies the mirror application in the local OS by the window title. When the mirror app is visible but Robot fails to establish the connection saying "Waiting for AirServer/Reflector to start" go to Edit->Preferences in the main UI and select the "iOS Mirror" panel. There's a list of semicolon separated java.util.regex.Pattern regular expressions used to identify the mirror application title. Update it to match the mirror app window title. On Windows hover the mouse over the app in the Windows task bar to get the actual title. An alternative approach for both OSes is to enable the debug mode in the iOS Mirror panel, make an attempt to connect and check the log (Help->Log Viewer) for the list of displayed windows and their titles.
  
• If Robot fails to display just the device screen and you can see the application title bar and/or the surrounding frame or other components in the desktop viewer, go to the "iOS Mirror" panel and look for the inset preferences. They will allow you to adjust the view manually.
• Robot is not responsible for the AirServer or Reflector failures. Should you have any problems with these applications please contact their support desks.
  

Image comparison tips

Automation Tips

• The T-Plan iOS 6 Server is marked as a Navigation and VoIP application in order to be allowed to run in the background. To avoid draining of the device battery we recommend to keep the device connected to the power during automation. When the device is not being automated the server may be killed as follows:

1. Double press the Home button on the iOS device to open the list of running applications.
2. Scroll to the right (swipe to the left) if needed to locate the T-Plan Server application.
3. Long press the application icon until it starts shaking. Then press the "red minus" icon to kill it. Finally press Home to go back.

• Avoid running other applications which use the Always On Top window mode during automation. They can pop up unexpectedly and interfere with the Robot desktop view.
• Should you need to control the mirror app directly while it is locked in the Robot view on Windows, minimize the Robot application to the task bar. Robot will re-lock it after its window gets restored.
  
• The landscape mode is supported only when the device is rotated counterclockwise (the Home button on the iPhone is on the right). If the device is rotated clockwise no error will be reported but the mouse clicks and drags will not be placed correctly.
  
• Unlike in some VNC connections, the mouse pointer is not drawn into the local screen image which Robot works with. You don't have to make your scripts move the mouse pointer away to perform image comparison over a certain area.
  
• As automation is performed on the local desktop, one machine running Robot can be used to automate only one Apple device at a time.