Pratt Institute Libraries
Usability Testing - Eye tracking
Discover the Story
The Pratt Institute Libraries provide the service to both students and teachers at Pratt Institute. The Pratt’s Brooklyn Campus Library is located on Pratt’s main campus in Clinton Hill. The second library - The Pratt Manhattan Center (PMC), is located in Manhattan. Pratt’s students and teachers can freely use the resources made available in both libraries.
Together with my colleagues: Yiqiao (Karen) Li, Wenjun Zhou, and Armon Burton, we were asked to perform eye tracking usability study to provide feedback and recommendations on the map functionality of Pratt Institute Libraries catalog.
Duration of the project: 4 weeks, November - December 2019
Research methods: In-person moderated usability testing
Tools: Tobii Pro - eye-tracking software
My role: Within our team, each of us performed 2 in-person moderated usability tests. I was responsible for drafting tasks for these tests and the analysis of the obtained qualitative data. Additionally, I kept track of our progress and ensured that we kept our deadlines.
Research Origin
In November 2019, we had a meeting with a representative of the Pratt Institute Libraries, who asked us to research the usability of the map functionality of the catalogue.
Research objectives:
How users understand and interact with the map?
How users understand the role of Reference Desk?
Identify what can be improved and provide actionable recommendations.
Methodology
After the meeting with Pratt Institute Libraries representative, we knew what are our main research problems. During our group meeting, we realized that the best way to investigate them is to perform In-Person Moderated Usability Testing with a use of eye-tracking technology. Fortunately, our school has this technology available and we could learn how to use it. This section will present the way we took to achieve our results and create recommendations.
In-Person Moderated Usability Testing
In this method, participants are invited to the user testing lab to perform a set of tasks guided by the moderator. The moderator, with the help of a note taker, records users’ behavior and comments, and may ask additional questions during each task. Users are encouraged to speak aloud their thoughts. This method, though time-consuming, has one main advantage - researchers are directly on the spot and can follow up on user thoughts. However, a downside of this method is the so-called observer effect. When users are being observed, they may behave in a different manner than usual. Or they can look longer at the part of the screen they talk about, which affects the data received from Tobii, our eye-tracking technology. We have decided to use eye-tracking, as it allows us to receive bigger number of quantitative data and clearly see if interesting for us areas such as “Map it” button or floor number are noticed by users.
Test Design
We have decided to divide our usability testing into 3 parts: Pre-test, Observation and Post-test. In Pre-test we asked questions aiming to get additional data about our users such as their study year, their major, frequency of visiting Pratt Library website and going to the physical location of a library. We also asked test participants to tell us, which Pratt library they visit. These questions allowed us to have a better understanding of who are our test participants and what are they research habits.
Tasks
Go to: https://library.pratt.edu/ and search for the “Information Visualization” book by Colin Ware.
What information can you find on the website that will help you locate where this books is exactly?
[On Map page] - Now imagine you would like to retrieve this book personally at the library. Can you explain us step by step the process of retrieving the book?
2. You are a student and you are looking for a thesis written by Nicholas Dease. Please search for this thesis and tell us how would you retrieve it.
Participants
While developing the tasks, we considered who are our potential users. We decided that these are undergraduate and graduate Pratt students. Preferably, they should not wear glasses (to ensure that Tobii would catch the data correctly). We had 8 test participants: two undergraduate students, 6 graduate students. All of the students have been to the Brooklyn campus library, 3 of them to the Manhattan one. Two of our test participants have never used Pratt Library website. We have found our test participants by asking our friends and family members. The presence of the undergraduate students and students who did not use Pratt Library website before ensured that they represented the client’s desired users.
Pilot Testing
Before we invited our participants, we tested our methodology, task design and the software. We quickly learned that we need to combine what was our task 1 and task 2 into only one task. This worked better for users and was more natural for them. We have also learned that we need to cannot use Tobii’s functionality to record a page, but we need to record a screen. When the open website froze during the pilot test, we had to stop the test and start to do it again.
Usability Testing
All tests were performed within a week. Each participant signed the consent form and was briefed about what will happen during the session. Usually, all group members were participating in the usability testing, but they sat further from the test participant not to cause additional stress. During these 8 test sessions, we have learned that it is important to give time to our test participants to adjust to eye-tracking technology. Sometimes, we had to repeat the calibration of a device a few times to make sure that our data is correct. Mistakes in the data calibration were affecting the quality of the quantitative data. After each session, we have uploaded our notes and entered them into Google Spreadsheet to keep track of the data.
Data Analysis
Qualitative Data Analysis
Soon, after 8 sessions with participants we had seemingly endless string of data. I have decided to print it out, cut into pieces and manually categorise it. Thanks to it, the major themes emerged, which helped us to visualise the next steps of the data analysis. Each of us, summarised user comments under each theme and labeled it with: Content, Function and Visual category. This is how we achieved our Problem list.
The Content category has problems related to unclear or missing information, labeling, or descriptions
The Function category has problems related to missing functionalities or features
The Visual category has problems related to unclear or missing wayfinding arrows/path, icons, or signifiers
Based on the Problem list, we created a list of our findings, which were even more general.
Quantitative Data Analysis
Tobii, the eye tracking software, allows to create heatmaps and gazeplots. However, to create them, we had to manually fix the Times of Internet - timestamps on a video. Only because them, Tobii knew where user was switching between the pages and could create a heatmap/ gazeplot just for the particular page. The data from heatmap/gazeplot confirmed our first, draft findings from the qualitative data analysis.
Findings
Overall, 7 out of 8 users found map functionality useful and would use in future. However, the same number of users would like to improve wayfinding on a map. Based on usability testing, we can see, that users can achieve their aims using Pratt Libraries website, however, they might struggle with it. In our 5 Findings, we go into details regarding the problems they encounter. However, first things first – positive usability testing results!
Great news!
Overall, map is straightforward and useful. Users know how to find it and they can locate the crucial elements of the map, which helps them to achieve their aims.
100% users quickly found “Map it” button - all users naturally proceed to click on it
100% users recognised the crucial element of the map - either Pin marking the location of a book or a Reference Desk
100% users finalized successfully all tasks
Finding 1: Users have difficulties with wayfinding while using the maps
Users are not able to find Brooklyn campus name on the library map. Without it, they are unsure, which library is illustrated on a map. The vast majority of users had problem with locating entrances to both Pratt Institute Libraries. As the majority of them have visited both libraries, they used their knowledge to locate it. The floor number on the top of the page weren’t noticed.
The heatmap you see was generated, when test participant attempted to answer on a question how to find a way to the Reference Desk. It is visible, that the test participant was not able to locate the entrance to the Library (which is between Circulation Desk and Reference Desk).
Finding 2: Users ask to include the stack and book details on the map
Users noted that after clicking on the pin used to highlight the place where the searched item is located, they would expect to see more information. They would be especially interested in seeing the stack and shelf number. As one of the test participants noted, the map is sufficient for arriving to the destination point – in this case correct stack number. However, this user was not sure, how she would be able to find a book on a shelf.
Finding 3: Users are not familiar with the library terms used on the site
The role of the Reference desk is vague to users. Users do not receive enough information about the process of retrieving the thesis from the Reference desk. They are not sure what should they expect to happen at the Reference Desk.
Users are not sure how to find and use book call numbers. Users had a hard time finding the call number information. Additionally, they were not sure how to use them to retrieve the book.
Finding 4: Users are confused about the meaning and labeling of some of the icons
Users were not sure about the meaning of some of the icons. There is little consistency between Brooklyn Campus Library map and Pratt Manhattan Center. In Brooklyn Campus Library both Circulation and Reference Desks are described with labels. However, Circulation Desk at the Library of the Pratt Manhattan Center does not have the label. The icons used on both maps are the same. However, some icons are specific for the library at the Brooklyn Campus.
Finding 5: Users expressed an interest in being able to save the maps on their phone
Users were not sure how they can use the map in the retrieving searched items. They noticed, that they will not remember where the item was located without saving it on a phone or taking a picture of it.
Recommendations
Clarified and Easier Wayfinding
On our mock-up we tried to respond on our user needs. First of all, we clarified which library map it is. Additionally, we changed the place where floor number is located moving it from the top of the map to the spot next to the library name.
We added missing entrances to the library indicated with universal door sign and blue arrow. Afterwards, we added the path leading user to the searched item. We decided to grey out less relevant information such as book shelves to limit objects which might distract the user.
Instructional and Helpful
We decided to expand the section on the right of the map. First of all, we added the book title and authors’ name. This allows to reconfirm users, that they are in a correct place. Below, we described the way of finding an item. It is especially crucial in a case of items belonging to the Special Collections e.g. theses. Users rarely pick them up from the library and they are not familiar with a process. Detailed description helps them with it.
While performing usability testing, we have also learned, that users rarely click on a pin – in our mock-up, we have decided to explain the marker. Under it, we added information our users asked for: stack and call number. We would also like to propose to add shelf number, which would further help the users in their book search quest.
Meaningful and Relevant Icons
First of all, we made sure that we are using only the icons which are needed. Manhattan Center Library does not the same icons as a map of the library located in Brooklyn. Afterwards, we followed suggestion of one of our users and we created 2 new icons representing Reference and Circulation Desk. They both resemble humans and we believe, that it will give an additional clue to our users, what they can expect at both of the desks.
Allow Easier Offline Access
Some of our users were not sure how map checked on a computer can help them with finding a searched item directly in the library. Usually they managed by taking picture of a screen. We propose to add additional button allowing users to save an offline copy of a map on user's phone.
Reflection
As a team, we presented the results of our research to our client. This was an incredibly rewarding experience. Our recommendations were found actionable. And Nick Dease, librarian from the Pratt Institute Libraries said, that he will implement them immediately. We shared with him our presentation, problem list and highlights reel video, which you will also find available for download at the link below.
This was rather sprint project, we had barely month to create our final products. I would be interested in watching how our recommendations implemented and further investigate whether they were successful or not. Moreover, at our research we did not look into mobile use of a library maps, which opens a new research opportunity. Finally, we did not combine the results of our in-person moderated usability test session with other types of research such as Google Analytics or heatmaps generated by Hotjar.
If I could change something in the process, I would spend more time on analysing the quantitative data. While using Tobii, we had to manually create Times of Interest to mark moments, when users were changing the screen. That added additional difficulty to this task. I would love to learn how to create areas of interest (mark interesting areas on a screen), which would additional depth to the research.
To sum up, I believe, that we created good, actionable recommendations which can improve user experience and this is something, which for us - UX students is the most important.
Materials
This is the presentation which we delivered to our client. It contains brief description of our process, our findings and recommendations.
If you look for detailed description of the problems, we have identified - look no further!
In this video, our dear stars - our test participants in detail describe our findings - Enjoy!