Case study: website A/B testing

This case study, from our Let’s Get Real action research program, examines how the library tackled the issue of presenting its history as well as presenting clear information to plan a visit. They used A/B testing – comparing two versions of a webpage against each other to see which works best – to develop an interactive timeline tool and worked with other teams within the museum.

About the participant:

Name: Liz McCarthy

Organisation: Bodleian Libraries

Jump to:

What was the research question behind your experiment? | Why was this important to your organisation? | What did you do to implement this? | What happened? | What were the personal challenges you faced when carrying out this experiment? | What did YOU learn? | What did YOUR ORGANISATION learn? | What next?

What was the research question behind your experiment?

How can we best deliver content about our history and who we are to our visitors,web and in person?

Sub-question: What do our visitors know about us/our history before they arrive, what do they learn, what do they want to learn and how?

Sub question: What stories can we tell to engage people with our mission and our history?

Why was this important to your organisation?

We had previously done very little to provide casual visitors with easily digestible information about our history and who we are (outside of booking a formal tour or buying a full-length book). As we review our content strategy and focus more on public engagement, it was important to make sure that we were accessible and clear about who we are, where we come from and what it is about our history that makes us special.

What did you do to implement this?

We initially ran user research both in person and online to ask visitors what they know and what they want to know.

We also updated our current About Us history page, which ran to 3,000 words. We a) reduced the length and b) created an interactive timeline that delivered content in easy to read chunks.

We ran A/B testing, showing users the old history section version and the new one.

We worked with our Venue Services team to overhaul the Tours and Venue Hire sections of our website, which are some of the ‘top visited’ content for members of the public. These sections are often the first experience visitors have with us, and they weren’t fit for purpose. The reworked version not only provides a more explicit path to tour tickets and hire info, but also provides clearer information about our historical spaces.

What happened?

An incredible level of support and collaboration from other teams! Our Venue Services team was enthusiastic about a redesign of their pages, and also about providing information about visitors that they have gathered after years of face-to-face interaction. Other teams were able to contribute information and stories for the timeline, and as we move forward we’ve had enthusiastic feedback on future plans.

What were the personal challenges you faced when carrying out this experiment?

Time. With all the best will in the world, we all have too much to do and too little time in which to do it. Colleagues were enthusiastic about helping, but I tried to be realistic about what they could actually fit in. It is always challenging in a large organisation to move big things quickly, so I focused on getting smaller elements done one bit at a time.

Too much choice! We have so much going on in our team, and in retrospect many of the things we’ve done over the past nine months could have been a great Let’s Get Real project.

What did YOU learn?

In addition to developing a better understanding of our public visitors, learning to use a few new technical tools and better analytical and testing skills, I felt what I really gained from the project was a better understanding of how to approach digital project management and the value of building a team that feels it can be creative and experimental.

What did YOUR ORGANISATION learn?

Technical and testing skills – we had issues with the timeline tool we wanted to use, which we were able to address but it took time and expertise from our developer. Once we had that sorted, we encountered issues with our Content Management Sysytem and A/B testing. We were able to fix these, but – again – it took time. However, now we’re set up to do more regular testing, and have experience of running tests.

We learned more than we thought we might about what our visitors know about us. We expected to see lots of Harry Potter references and ‘old books’ type statements, but in fact a huge number of the people coming into our spaces didn’t even know we were a library – they had just wandered in. The things people did know or wanted to know were quirky or experience-based – how to get in, amazement at seeing historic moments, seeing things ‘behind the scenes’.

We also did quite a lot of thinking about our ‘commercial’ public options (tours, venue hire, etc.) and how we present these to our visitors. We’re not at the stage yet where we can judge whether the improvements we made are having a financial impact, but user feedback is much better.

We learned that aren’t particularly consistent or considered in the language that we use to describe ourselves and what we do to the public. We have a clear mission as an academic library, but strategy around public engagement is newer. The project has raised questions that we’re now thinking about and trying to answer.

What next?

We’re still working on this experiment, but it has a number of different (and big!) project aspects of its own:

Run a social media campaign based on the timeline stories

Investigate audio guide or replacement options for visitor services

Explore content delivery on-site (mobile? kiosks? something else?)