In 2017, I assembled a cross departmental team and created an agile process for fixing bugs and implementing feature requests in the Primo library search platform. While that team was successful in its goal of making incremental improvements to Primo, by the end of the first year we were thirsting for more. I understood the limits of what was possible with Primo, and recognized that many of the biggest obstacles users face in library research are due to the technical and legal complexities of the academic publishing landscape. I advocated for shifting our focus away from our search platform and towards the larger question of how users experience library discovery in the context of their work. We decided to begin reducing the volume of work put into Primo and create a project team to learn more about our users.
I assembled a cross functional team, and began with background research. We talked to internal stakeholders, researched similar projects at other libraries, and investigated the user inteview process itself. Since the library had never conducted user interviews, I led the group through background readings to understand what the process looked like and how to prepare for it.
We developed a list of research questions and I wrote an interview guide. The guide provided a list of interview questions as well as prompts and guidelines for follow-up questions. I structured the interview to be very open-ended. We wanted users to talk to us about their work and how the library fit into it. We also wanted to hear about the other tools and resources they use and how library research fits into the larger context of their work.
We have access to audio recording equipment through the library, but some project team members had never done recording before. I wrote documentation for the equipment and software and let the team through mock interviews so we could see how everything worked.
Since the scope of topics for discussion was potentially very wide ranging, our stakeholders made clear that we needed to interview a representative sample of users. I wrote a participant screening form for recruited users and we identified target numbers of disciplines, backgrounds, and experience levels across faculty, graduate, and undergraduate students.
Each academic department has a library liaison assigned, and we relied on these liaisons to help recruit faculty and graduate students. We advertised on our website, and had student workers form a street team to hand out flyers to students around campus.
In total, 70 students and faculty responded with interest. We used data from the screening form to select users. Our first major challenge came with scheduling. With students, we quickly learned that if you scheduled them any more than a day or two out, they would cancel or not show up. As for faculty, finding a time that worked for them proved much more difficult. We were able to increase the ease of scheduling by meeting at their office, but as the Fall semester neared its end, Faculty couldn’t prioritize meeting with us. We decided to put the project on hold for December and pick it up again in January.
In Universities, projects are frequently at the mercy of the academic calendar. So we were prepared for the risk and with frequent communication, stakeholders were prepared as well. We picked up interviewing again in January, and by early March had interviewed 19 students and faculty.
For each interview, one project team member would serve as lead interviewer. A second person would serve as note-taker, set up the audio equipment, and occasionally ask follow-up questions. Each member of the team shared the interview process fairly evenly. Each interview was scheduled for 30 minutes. Some ran under the allotted time, and many went well over the limit. The Boston College Libraries works with 3Play Media to caption videos for accessibility. We were able to work with the same company to transcribe our interview audio.
From the 19 interviews, we transcribed nearly 11 hours of audio. We coded and analyzed nearly 1400 passages from the transcriptions with NVIVO qualitative analysis software. During coding, we identified more than 180 different topics. We coded everything from mentions of specific tools and resources, to sentiment analysis, to descriptions of successes and failures during the research process.
With the help of NVIVO’s analysis tools, we were able to identify the topics with the most amount of frustration and commentary. From this we identified four key takeaways:
The most prevalent topic of conversation for users was frustration with access the full text of articles, particularly when searching for a known item. It was also the source of the most negative sentiment. Reviewing the interview data revealed that the complexity of interconnected systems doesn’t map well to users’ mental model. The primary mechanism for full text access, link resolvers, present users with frustrating and confusing choices, points of failure, and obstacles. Even advanced users found the options difficult to understand and harder to teach.
Users consistently noted a lack of confidence in choosing the correct terms for effective searching. Novice researchers tended to lack the subject vocabulary and search experience, faculty and graduate students have more domain knowledge yet struggle with specificity and exhaustiveness in their searches. We also learned that advanced researchers in some cases are completely unaware of discipline-specific resources at their disposal.
Among experienced researchers, one of the most prevalent needs was to stay current in their discipline. Many expressed the importance of reading the latest issues of core journals in their discipline, but lacking an easy way to do so. Using the library search platform to locate journals results in the same problems of disambiguation and access users encounter with known item searching. While the BC libraries do provide subject-based lists of journals, our focus has been on exhaustive representation. What we learned from users is that they need streamlined access to curated lists of only the most reputable journals in their field.
Another topic prevalent with novice and advanced users was the need for tools and strategies in support of organizing their work. The needs of individual researchers varies greatly. Faculty approached organization as a need for building knowledge and connecting ideas over time. But they frequently noted lack of awareness or limitations of the tools and strategies at their disposal. Advanced researchers also noted the importance of highlighting and annotation, and the gap between existing tools like Zotero and RefWorks and the need to organize selections from larger works. Graduate students encountered the problem as they embarked on their dissertations. For many it’s their first large scale research project, and they are ill equipped to deal with the volume of information. Many noted a process of trial and error over the course of their work that detracted from their academic progress.
With four key problems identified, I took the highest priority one, Full Text Access, and led a project to address it. From that project, I established a pattern of work that can be applied to the remaining areas and iterated on over time. Our interviews generated a wealth of data across a wide variety of topics of interest to the library. I documented our datasets, and together with access to NVIVO, there’s a great opportunity to use the data for other investigations.
I also documented the user interview process, from developing an interview guide, to recruiting and scheduling, to transcription and analysis. The library now has resources available for future user interview research.
