Back to all projects

AllenAI Semantic Scholar Usability

Role: UX Researcher
Collaborators: David Mondello, Jane Winn
Type: Mixed-Methods Research, Usability Testing

Overview

For a Usability Studies course, we worked with AllenAI to test Semantic Scholar, an academic search engine that utilizes artificial intelligence to empower researchers to do great work. Our goals for the project were to evaluate the core search experience, catalogue/chart the most salient features of the site, understand how participants comprehend certain features, and establish usability benchmarks to inform future research. We had many research questions that fell into four domains:

Discoverability - What features do users discover, and when do they discover them?

Comprehension - How do users understand the site and its features?

Effectiveness - Are users able to accomplish their goals?

Satisfaction - How do users feel about the site?
TOP

Process

We met with our stakeholders to understand the goals for the project and to scope our research questions. We conducted a cognitive walkthrough of the product to identifying core tasks and uncover potential usability issues. We reviewed existing literature on search engines and familiarized ourselves with similar products.

Identifying stakeholder needs, cognitive walkthrough, secondary research

Finding our Users

We recruited a balanced sample of representative users based on three key characteristics: academic field, work domain, and position level. Our participants were researchers in computer science and biomedical science, either in academia and in industry.

Recruitment, scheduling

Usability Testing

We ran 9 one-on-one usability tests that comprised of a pre-test interview, task-based scenarios, a post-test survey, and a post-test interview. We collected mixed methods data through audiovisual recording, custom designed note taking forms, and online surveys.

Structured interviewing, task-based scenarios, behavioral observation, surveying

Data Analysis & Presenting Findings

We analyzed the data using affinity diagramming and statistical analysis. We organized, prioritized, and communicated usability issues both through a written report and an in person presentation at AllenAI.

Affinity diagramming, statistical analysis, reporting, presentation

View Final PresentationView Research Report