UTE Methodology

UTE is a highly-effective tool for usability testing web sites and web applications provided those using it adhere to the following industry standard testing methodology.

Steps:

1. Create a test plan
2. Develop pre-test questions
3. Create task scenarios
4. Develop post-test questions
5. Validate the test
6. Invite representative test participants
7. Run the test
8. Prepare test data for reporting
9. Generate test reports
10. Use the information collected to improve the website or application
11. Re-test the revised site or application to ensure that your changes had the desired effect

Detailed Steps:

1. Create a test plan
   • Determine what features of the site or application are to be tested
     • Identify high-priority features
     • Identify features you or others have concerns about
   • Determine which users should participate in the test
     • Identify distinct user populations
     • Determine the number of participants you plan to test
     • Determine how will participants be recruited
   • Determine who will conduct the test
   • Determine where you will conduct the test. Some alternatives are:
     • Testing lab
     • Conference room
     • User's office
     • Trade show floor
     • Remotely
   • Determine what testing approach will be used. Some alternatives are:
     • Moderated (facilitated)
     • Unmoderated (unfacilitated)
     • A mixture of both
   • Develop proposed task scenarios. Typical issues are:
     • How many scenarios should there be?
     • How do I create “good” scenarios?
     • How long should a test last?
2. Develop pre-test questions. Typical question types include:
   • Demographic questions
   • Survey questions
3. Create task scenarios
   • Insert pre-scenario questions
     • Often a good place to gather information about participant expectations regarding the difficulty of the scenario
   • Insert practice scenarios. These are useful to:
     • Familiarize participants with the site or application being tested
     • Familiarize participants with the UTE Runner interface
   • Create the task scenario text
   • Define the scenario’s starting point (generally a URL)
   • Select how the scenario is to be completed. UTE supports the following methods:
     • Participant reaches a target page
     • Participant answers a specific question
     • Participant indicates they have completed the task
     • Participant executes their first click
   • Where appropriate, specify the expected end page the participant should reach as he/she completes the task
   • Where appropriate, define the path or paths participants are expected to take in performing the task
   • Insert post-scenario questions
     • Often a good place to have the participant compare the reality of the scenario with pre-scenario expectations
     • Solicit participant comments about the scenario just completed
   • Other typical scenario-related issues include:
     • How many scenarios should there be?
     • How do I create “good” scenarios?
     • How long should a test last?
4. Develop post-test questions. Typically these are used to:
   • Solicit preference information
   • Gather participant comments about the overall site or application
   • Collect satisfaction data. A commonly used satisfaction questionnaire is SUS - the System Usability Scale
5. Validate the test
   • Solicit the help of representative participants
   • Watch for problem tasks. Common problems include:
     • Poorly written task scenarios
     • Questions with multiple, conflicting answers
     • Tasks for which information no longer exists on the site
     • Tasks which have leading or biasing scenario text
     • Tasks which have leading or biasing scenario names
   • Repair problem scenarios and validate again with a different set of representative participants
6. Invite representative test participants
7. Run the test
   • UTErunner must exist on the computer or computers to be used for testing
   • The test you have developed must exist on the computer or computer to be used for testing
8. Prepare test data for reporting
   • Gather UTE log files from all participants into a common folder location accessible to the UTEmanager
   • Disqualify participants, scenarios, and/or override correctness of answers as needed
9. Generate test reports
   • UTE automatically compiles data from test participants
   • Combine automatic reports with facilitator and/or observer observations
   • Identify usability problems
10. Use the information collected to improve the website or application
    • Prioritize problems
    • Develop solutions
    • Fix the problems
11. Re-test the revised site or application to ensure that your changes had the desired effect