Ways to identify UI elements in mobile apps

Problem: Many testers do not know how to identify UI elements in mobile apps.

In this post, I explain about different ways to find UI elements in Android’s calculator app. I also show you examples of using Appium’s API with the different locator strategies.


Why this post?

Identifying UI elements is a key component of writing UI automation. Most testers know how to inspect a web application and uniquely identify the UI elements that they want. However, inspecting and identifying UI elements in a mobile application is still somewhat of a dark art for most testers. Case in point – when I was learning Appium, it was hard to find a single article that showed me the various options I have to identify UI elements. This post aims to rectify this problem. In this post you will learn different ways to identify and interact with UI elements in a mobile application.


uiautomatorviewer: Mobile’s Firebug

If you automate web application testing, you are sure to have used either developer tools or Firebug. uiautomatorviewer is a GUI tool to scan and analyze the UI components of an Android app. uiautomatorviewer comes with the Android SDK. Please make sure to install Android SDK to follow along.

To launch uiautomatorviewer, pull up your command prompt, navigate to your $android-sdk/tools directory and issue the command uiautomatorviewer. Launch uiautomatorviewer and connect your Android device to your machine. Launch the calculator app. In uiautomatorviewer, click on the Device Screenshot button (top left) to take the snapshot of the calculator app. You can now inspect the layout hierarchy and view the properties of the individual UI components that are displayed on the device.

Screenshot of calculator app using uiautomatorviewer
Figure 1. Inspecting a mobile application

NOTE 1: For more details on connecting your Android device to your computer, please refer to the Step 2 of our previous blog.


Different ways of finding UI elements

Figure 1 is your path to happiness. You can use the properties of the elements found in the Node Detail table like text, class, resource-id, content-desc etc to find the UI components of the app. Here are a few locator strategies that map well with Appium’s API:
1. Finding element by xpath in mobile application
2. Finding mobile element by Android UIAutomator

1. Finding element by xpath in mobile applications
To write your xpath:
a. start with the class //class
b. [OPTIONAL] add attributes @attribute="blah"
E.g.: XPath to identify the number 7 in the calculator app is

//android.widget.Button[@text='7']

With Python and Appium, your corresponding call would be:

driver.find_element_by_xpath("//android.widget.Button[@text='7']")

2. Finding mobile elements by Android UIAutomator
Another way to identify UI elements in a mobile application is via UiSelector. To do so,
a. start with a new UiSelector object new UiSelector()
b. dot-add any of the other properties of the element .text()
E.g.: To identify the number 7 in the calculator app:

new UiSelector().text("7")

With Python and Appium, your corresponding call would be:

driver.find_element_by_android_uiautomator('new UiSelector().text("7")')

NOTE: This method uses properties such as text, content-desc, resorurce id etc to find elements. Please refer to this link for more details.

Here are a few shortcuts to identifying elements that are specific to Appium:
1. Via accessibility ID: On Android, you can use the string in content-desc.
E.g.: To identify the delete button in the calculator,

driver.find_element_by_accessibility_id('delete')

2. Finding element by Id: Corresponds to the resource-id field
E.g.: To identify the number 7 in the calculator app

driver.find_element_by_id("com.android.calculator2:id/digit7")

3. Finding element by class name
E.g.: To find the text/results box in the calculator app

driver.find_element_by_class_name("android.widget.EditText")

An example Appium + Python script with different locator strategies

I am providing a fully functioning Python script for you to practice your locator strategies. Challenge yourself to identify different elements in the calculator app and make corresponding changes to this script.

"""
Qxf2: Example script to run one test against calculator app
The test will show you how you can find UI elements by various methods like xpath, id, accessibility_id and android UIautomator
on a android calculator app
 
"""
import os
import unittest, time
from appium import webdriver
from time import sleep
 
class Android_Calculator(unittest.TestCase):
    "Class to run tests for android calculator"
    def setUp(self):
        "Setup for the test"
        desired_caps = {}
        desired_caps['platformName'] = 'Android'
        desired_caps['platformVersion'] = '4.4'
        desired_caps['deviceName'] = 'Moto E'
        # Get the package and activity name of the calculator to launch the calculator app
        desired_caps['appPackage'] = 'com.android.calculator2'
        desired_caps['appActivity'] = 'com.android.calculator2.Calculator'
        self.driver = webdriver.Remote('http://localhost:4723/wd/hub', desired_caps)
 
    def tearDown(self):
        "Tear down the test"
        self.driver.quit()
 
    def test_calculator(self):
        "Testing android calculator"
        self.driver.implicitly_wait(10)
        # Find the UI element using xpath  
        self.driver.find_element_by_xpath("//android.widget.Button[@text='7']").click()
        self.driver.find_element_by_xpath("//android.widget.Button[@resource-id='com.android.calculator2:id/mul']").click()
 
        # Find UI element using id
        self.driver.find_element_by_id("com.android.calculator2:id/digit7").click()
 
        # Find UI element using accessibility_id
        self.driver.find_element_by_accessibility_id('delete').click()
 
        # Find UI element using android UIautomator
        self.driver.find_element_by_android_uiautomator('new UiSelector().text("3")').click()
        self.driver.find_element_by_android_uiautomator('new UiSelector().resourceId("com.android.calculator2:id/equal")').click()
 
        # Find UI element using class name
        result=self.driver.find_element_by_class_name("android.widget.EditText")
        print result.get_attribute('text')
 
#---START OF SCRIPT
if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(Android_Calculator)
    unittest.TextTestRunner(verbosity=2).run(suite)

Happy inspecting! Happy locating!


Sign up for our monthly newsletter




Avinash Shetty
I am a software tester with over 10 years of experience in software testing. Currently I am working at Qxf2 Services Bangalore. As a student of the context-driven approach to software testing I feel there is a lot to learn out there which keeps me very excited. My work has helped me gain good experience in different areas of testing like CRM, Web, Mobile and Database testing. I have good knowledge of building test scripts using Automation tools like Selenium and Appium using Java and Python. Beside testing I am a “Sports Fanatic” and love watching and playing sports.

© 2013-2017, Avinash Shetty. All rights reserved.

38 Comments

  1. Kiran said:

    Hi Team,

    I have recently started working on Android and IOS App for Appium automation. Your article related to “Ways to Identify Elements” in Appium is excellent and 100% useful for new entrants into Appium.

    I have thoughts around this and couple of questions.
    a) Can we see same attributes for IOS App like you show in the “Calculator” Android App. I know IOS is different OS but any similarity which can be used to maintain single Object Repository for Automation across two different platforms. May be 100% elements not possible but at-least 75% if we can do , that would be fine. If you have any information around this , please share.
    b) As of now I have not configured Mac machine for automation , but curious to know how you maintain this Object Mapping in the Open Source Automation projects two different platforms (Android and IOS) which further needs to be integrated with automation framework, if you have any information please share.

    Regards,
    Kiran

    January 6, 2015
    Reply
    • Thanks for the compliments, Kiran.

      a) Short answer is most likely not. You may see some attributes for iOS apps that *may* just match . It will also depend on how your development team wrote the two applications. One side point: I have not re-checked recently, but about a year back (if I remember right), I got tripped up by Appium’s find_element_by_id method. There was some nuance (I do not remember now) with it behaving slightly different for iOS and Android. BTW, the latest recommended way to automate iOS is through Apple’s UI Automation framework: https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html

      b) I do not know. I have not had to maintain this kind of code before. Take the rest of the answer to be the best guess of someone who is very experienced with automation. Technically you could use Appium and then write your own mapping tool based on the platform. If you are disciplined and put all unique identifiers in one place, it will be easy to pick one set of unique identifiers at the time of initialization based on the platform. Using a page-object-like pattern should help you.

      Just based on your questions and immediate needs, it looks like you are in a position to arrive at an answer quicker than us. Whenever you solve your problem, can you please write it up either as a comment or (preferably) on your own blog. We would love to learn how you went about solving this problem.

      January 8, 2015
      Reply
  2. Priyan said:

    Good article nice one keep it up 🙂

    March 15, 2015
    Reply
  3. Amit Jain said:

    Please can you also give some examples using java

    March 23, 2015
    Reply
  4. John B said:

    Very Informative ! Nice Article ..

    April 1, 2015
    Reply
  5. John B said:

    Very Informative ! Nice Article

    April 1, 2015
    Reply
  6. Mohit said:

    Great knowledge supplier …Thanks

    April 30, 2015
    Reply
  7. Vaishali said:

    Hi ,

    Whats the command to upload he profile pic from gallery ?

    June 15, 2015
    Reply
    • Vrushali Toshniwal Vrushali Toshniwal said:

      Vaishali,

      Which application are you using? Did you try capturing the screenshot using UI Automator Viewer?

      June 25, 2015
      Reply
  8. Mallikarjun Reddy said:

    Hi Team,

    Am trying to configure Appium Grid,but not able to find relevant information in google.
    If u have worked on it plz share the stuff with me @mallikarjunareddy518@hotmail.com

    Regards,
    Mallikarjunareddy

    June 27, 2015
    Reply
    • Vrushali Toshniwal Vrushali Toshniwal said:

      Hi Mallikarjun,
      We do not have any relevant information on Appium Grid as we have not configured it yet.

      June 30, 2015
      Reply
      • Mallikarjun Reddy said:

        Oh it’s okay Vrushali, I got the solutions for it
        Anyhow thanks for the reply…..Doing great job, keep Up.

        July 15, 2015
        Reply
        • Vrushali Toshniwal Vrushali Toshniwal said:

          Mallikarjun,
          Do share the relevant information which helped you get the solution.

          July 21, 2015
          Reply
  9. lenguyen313.st@gmail.com said:

    Sir,does it work in Java. I do the same thing but it throw NoSuchElementException

    July 2, 2015
    Reply
  10. Rahul said:

    Nice article.

    Does Appium needs selenium for automating apps?
    Can we use appium with Coded UI ?

    Someone please help me on these topics.

    July 15, 2015
    Reply
    • Avinash Shetty Avinash Shetty said:

      Hi Rahul,

      Appium supports a subset of the Selenium WebDriver JSON Wire Protocol, and extend it so that you can specify mobile-targeted desired capabilities to run your test through Appium.

      I am not sure how to use appium on Coded UI.

      July 21, 2015
      Reply
  11. Anonymous said:

    Nice article

    August 3, 2015
    Reply
  12. Sam said:

    I found self.driver.find_element_by_android_uiautomator(‘new UiSelector().content-desc(“value of content description”)’).click() is not working with parameters content-desc or contentDesc ??? Can anybody tell how it works for the method find_element_by_android_uiautomator()

    August 25, 2015
    Reply
  13. Anonymous said:

    Whether the “Checkable, Checked, Clicked and enabled” node details functions can be implemented in appium mobile automation like resource-id, content – desc???? if yes please share the syntax for those functions??? Thanks in Advance

    September 10, 2015
    Reply
    • Avinash Shetty Avinash Shetty said:

      Hi,

      I am not able understand what you mean by “implemented in appium”. Do you want to identify the element using “Checkable, Checked, Clicked and enabled” or get their values? Please clarify.

      September 15, 2015
      Reply
  14. NithyaDharan said:

    Hi, Can anyone please tell, for android automation testing, which tools is best to use uiautomatorviewer or appium?

    October 30, 2015
    Reply
    • NithyaDharan,

      Both UIAutomator as well as Appium are good. The UIAutomatorViewer is a tool that lets you inspect elements of the app – so it complements both UIAutomator and Appium. UIAutomator is excellent for Android since it is officially supported by Android.

      At Qxf2, we prefer Appium, simply because the Appium of today reminds us of the Selenium from 2006-2007. We think Appium has the potential to become the Selenium of the mobile GUI automation space. Appium is still evolving and not perfect – but like the early Selenium, it works well enough across different platforms, supports multiple languages and (while brittle) lets you write automation code that works in even the complex cases.

      November 1, 2015
      Reply
  15. Anonymous said:

    please help to click item which got similar name and id , by using childuiselector in python

    March 31, 2016
    Reply
    • Avinash Shetty Avinash Shetty said:

      You can use any other property which is unique to the element and find that element. Using UiSelector
      a. start with a new UiSelector object new UiSelector()
      b. dot-add any of the other properties of the element
      E.g.: To identify the number 7 in the calculator app: new UiSelector().text(“7”)

      April 6, 2016
      Reply
  16. Anonymous said:

    Hi,

    I opened the UIautomatorViewer by double clicking on uiautomatorviewer.bat. Then I got Ui Automator Viewer. Then I click on Device Screenshot icon. But not able to inspect individual element on the page. Only whole page identifying.

    Could you please suggest how to proceed

    May 26, 2016
    Reply
      • Prathyuhsa said:

        Hi

        I am also having same issue, Only whole page identifying .with one app on which i can identify all elements through and this issue only comes in appium.
        So Could you please suggest whether there is some other reason fro this?

        August 16, 2016
        Reply
        • Shivahari P Shivahari P said:

          Hi Prathyusha,
          I am not able to understand your issue.
          My understanding is, You are not able to identify elements in Appium and you are able to identify them when you use another tool?
          If Yes, what other tool are you using?

          August 17, 2016
          Reply
  17. Krishna said:

    Nice article Avinash. This helped me a lot.

    Using UIautomator we can find many properties of an element in mobile apps but is there any way we can find out the id of an element using any tools?

    July 5, 2016
    Reply
    • Avinash Shetty Avinash Shetty said:

      Hi Krishna,
      Good to know the blog helped you.
      You can also use appium inspector to find the properties of an element on the page. However for android application the appium inspector didnt work well for me. It works very well for iOS though.

      July 12, 2016
      Reply
  18. kamalanathan v said:

    If any other tool available in market same as uiautomator tool?

    August 31, 2016
    Reply
    • Shivahari P Shivahari P said:

      Hi Kamalanathan,
      I am guessing that you are asking us if there is any other tool similar to uiautomatorviewer(since this post is about identifying elements) and not the uiautomator tool.
      Appium inspector is another tool that can be used to identify elements in mobile apps

      September 1, 2016
      Reply
  19. Gaurav said:

    Is there any way to validate that I am creating a right xpath using UI Automator like we can validate the xpath in consol on web without even implementing it in our code.

    March 21, 2017
    Reply
    • Shivahari P Shivahari P said:

      Hi Gaurav,
      I don’t think there is any tool that would let you do that, at the moment. We use UI Automator Viewer, Appium Inspector and XCUITest to identify UI elements for testing iOS and Android apps at Qxf2 and I am certain those three don’t let to validate your locators. Please let us know if you manage to find any tool that allows locator verification.

      March 22, 2017
      Reply
  20. Thanh said:

    Hello,
    Thank you for helpful post. But I am having a trouble with the question below:
    How can I press on a text field?
    I hope I receive your help.
    Thanks

    May 19, 2017
    Reply
    • Avinash Shetty Avinash Shetty said:

      Hi Thanh,
      Once you find the text element you can use send_keys() to send any keys to the text field.
      In case you want to click on the element use click() command.

      May 19, 2017
      Reply

Leave a Reply to Arunkumar Muralidharan Cancel reply

Your email address will not be published.