For a Usability Studies course, we worked with AllenAI to test Semantic Scholar, an academic search engine that utilizes artificial intelligence to empower researchers to do great work. Our goals for the project were to evaluate the core search experience, catalogue/chart the most salient features of the site, understand how participants comprehend certain features, and establish usability benchmarks to inform future research. We had many research questions that fell into four domains:
Discoverability - What features do users discover, and when do they discover them?TOP
Comprehension - How do users understand the site and its features?
Effectiveness - Are users able to accomplish their goals?
Satisfaction - How do users feel about the site?
We met with our stakeholders to understand the goals for the project and to scope our research questions. We conducted a cognitive walkthrough of the product to identifying core tasks and uncover potential usability issues. We reviewed existing literature on search engines and familiarized ourselves with similar products.
Identifying stakeholder needs, cognitive walkthrough, secondary research
We recruited a balanced sample of representative users based on three key characteristics: academic field, work domain, and position level. Our participants were researchers in computer science and biomedical science, either in academia and in industry.
We ran 9 one-on-one usability tests that comprised of a pre-test interview, task-based scenarios, a post-test survey, and a post-test interview. We collected mixed methods data through audiovisual recording, custom designed note taking forms, and online surveys.
Structured interviewing, task-based scenarios, behavioral observation, surveying
We analyzed the data using affinity diagramming and statistical analysis. We organized, prioritized, and communicated usability issues both through a written report and an in person presentation at AllenAI.
Affinity diagramming, statistical analysis, reporting, presentationView Final PresentationView Research Report