Test.ai + Appium Element Finding Plugin Improvements

If you weren't at SeleniumConf 2018 in Chicago, you might have missed the talk by Jason Arbon (with a cameo by yours truly) on work I did in conjunction with test.ai to bring Artificial Intelligence to finding elements with Appium. Here's the video in case you want the full scoop:

In the last months, we've looked for ways to improve the performance and accuracy of the plugin, and there are two recent changes worth mentioning:

1. Images are now classified in their greyscale format rather than color RGB. The purpose for this intentional loss of information is twofold: first of all, users are typically able to recognize icons by their shape alone. Color is obviously important, but most icons can be reasonably classified with only reference to shape. Because of this, there's no real reason to include color information as it makes the learning space bigger. That is why, second, the original model was trained on greyscale images. Using greyscale images in the plugin thus improves prediction accuracy.
2. The plugin now takes advantage of TensorFlow's ability to classify multiple images in parallel. While the image classification step was never the long pole in the tent (so to speak), this improvement just about doubles the classification speed.

There are still ideas we have to make this plugin better, for example figuring out how to avoid getting every single element and its screenshot from the app using XPath, as this is incredibly time-consuming!

If you haven't yet tried out the plugin, be sure to read the Appium Pro article on the plugin to learn how to get started. And then, make sure you have the most recent version (2.1), which contains the improvements listed above. Happy testing!