Can usability be measured with web analytics?
Traditionally, usability is tested qualitatively by conducting interviews with a small number of potential users.
The interviews take place in a “lab” where the participants are given certain tasks to complete on the web site in question. While interacting with the site, a moderator observes their behavior and asks them to share their experiences.
While I’m a big fan of this kind of usability test, the method also has a number of drawbacks. The most important being that it takes a long time to carry out and doesn’t provide the kind of instant and continuous feedback we know from web analytics.
As such, I find it interesting to search for possible ways of using web analytics to measure usability. What we need, ideally, is some kind of easy-to-understand, yet robust, web metric that indicates how easy it is for the visitors to navigate our site.
Such web metric should not necessarily replace qualitative tests in a lab. Rather, it should help us to obtain early warnings and point to the parts of our website which are most in need of usability attention.
Moving beyond standard metrics
Measuring usability by means of web analytics is not as easy as one should perhaps think.
Traditional web metrics, to be sure, will not do the job: The number of page views per visit, the bounce rate, the number of visits per visitor, the average visit duration, etc. None of these metrics can tell us anything about navigational problems.
Although it is true that frustrated or confused visitors might end up producing many page views per visit, for example, highly engaged visitors might do the exact same.
Introducing Back Navigation Rate
So, the question remains: How can we identify usability problems simply by looking at click stream data? Is it possible at all?
Some years ago, my colleagues and I at Netminers set out to answer that question. We came up with the idea that if there is any way to identify frustrated or confused visitors it must be based on how often they go back and forth between the same pages.
A visitor who runs into a problem, we thought, is likely to stop and go back to make sure he or she didn’t miss something. This might even happen repeatedly since visitors are likely to double-check before abandoning the site altogether.
We therefore decided to develop a technique for tracking what we call “back navigations” and “turn pages”. Consider the following click stream.
The blue dots represent pages seen by a single visitor. The arrows indicate the direction of the click stream.
A is the entry page, while B and C are pages that lead to a particular interest point, D. Having looked at D, the visitor decides to go back to C, and further back to B. Finally the visitor makes a new move forward from B to E.
As can be seen from the figure, there are four forward navigations and two back navigations. We say in this case that the Back Navigation Rate is 50% (2 divided by 4). Notice also that D can be seen as a turning point in that the visitor decides to turn around here. We call D a “Turn Page”.
When is a high Back Navigation Rate bad?
Is it good or bad that the Back Navigation Rate is 50%? Well, it depends.
You cannot always say it’s a bad thing, because sometimes visitors just like to explore links by moving back and forth on the site. This is especially the case on the home page where visitors often explore many different links before selecting a specific page for closer inspection.
According to Netminers Index (i.e. a series of benchmarks based on all the data we collect across all of our clients), the Back Navigation Rate is in average 30%. Back navigations are therefore not as rare as you might first think.
Keep it low for your check-out procedure!
However, imagine now that you are looking exclusively at a check-out procedure or a similar conversion process. In this case you certainly do not want users to go back 30% of the times! A check-out procedure, to be sure, is supposed to be linear, not zigzag, and, in our experience, here the Back Navigation Rate definitely shouldn’t exceed 5%.
Turn Page Rate
But you can do more to identify problematic back-and-forth movements. Notice that in the click stream presented above D appear as a turn pages. This, however, happens only once.
Consider now a case where the visitor produces repeated turn pages.
In this case B and C are both turn pages, and both of them appear as such two times. We can capture the difference between the former and the latter click stream by what we in Netminers call Turn Page Rate. This rate is calculated as follows: First, you slice your data so that you only look at turn pages. Then you divide the number of page views by visits. This rate is 1 for D in the former click stream , but 2 for both B and C in the latter.
What do you think?
What is your opinion about Back Navigation Rate and Turn Page Rate? Can you think of any flaws of these metrics? Do you have suggestions for some alternative and perhaps better usability metrics? Please let me know!
May 22nd, 2008 at 11:41 pm
Hi
Interesting way of investigating the issue. My experience is that it is easy to see when the user´s are having usability problems by using CrazyEgg. The way of analyzing it is to start at the page with most traffic (off cause in many cases the frontpage) and then follow the path where most of the users go. In that way you kind of follow the users mind and in many cases see if “turn back” is happening.
Another obvious element is the users use of internal search. By tracking this you don’t just get a feeling of how big a problem navigation is, cause by tracking the specifik words, you also get knowledge on which areas you need to optimize on.
But the methods mentioned above is off cause not that easy to put in an algorithm but need some hands on and thinking…
The problems with your idea, as I see it, is that you dont know in what scale people use the backwards button when they are done reading, and by that meaning that the user maybe really wanted to read both A, B, C, D and after finishing reading in that direction they seek back to start to go for further investigation.
How can an algorithm track that?
May 23rd, 2008 at 11:49 am
Hi Jacob!
I see your point about using ‘video playback’ solutions to identify turn pages and back navigation. I can definitely see that these solutions could be an important supplement to more traditional web analytics solutions.
However, as you yourself indicate, these playback solutions are probably more suitable for qualitative or exploratory analysis. This is important too, of course, but what I was after in this post was a new type of web metric which we can use as a baseline or benchmark. For example, ‘Our goal is to keep the Back Navigation Rate below 5% for our check-out procedure’. Further, ‘No pages in the check-out procedure should have a Turn Page Rate above 1.5’.
I like your idea about looking at how the site’s search field is used. But, again, how do we turn it into something more hard-core? A simple Site Search Rate (visits who search divided by total visits) wouldn’t be enough, since search is usually a sign of engagement. What we need to do instead is to define a Search Abandonment Rate, which tells us how many visits enter a search string in the search field, but leave the site without clicking on a result. This, however, tells us more about the effectiveness of the site search engine than about the usability of the site as such.
Regarding your final point, I recognize that the Back Navigation Rate and Turn Page Rate are not perfect and as such they should never replace qualitative usability tests or online surveys. In fact, I recognize this much more than you might think!
Netminers (which I co-founded) is in fact based on the idea that traditional web metrics come short of solving usability issues. The reason is that you can never be sure only by looking at clickstreams who the visitors are, what they are looking for and whether or not they have a positive experience. Because of this shot-coming, we developed a built-in online survey technology (as early as in 2002) that automatically integrates responses with clickstream data (more about how to analyze this in future posts!). This importance of qualitative metrics is explained by Avinash Kaushik in a much better way that I could ever hope to do (see http://www.kaushik.net/avinash/2006/05/overview-importance-of-qualitative-metrics.html). He has also done a lot to promote what he calls the Trinity Approach, which combines quantitative and qualitative methods (see http://www.kaushik.net/avinash/2006/08/trinity-a-mindset-strategic-approach.html).
Having said all this, however, I still think Back Navigation Rate and Turn Page Rate are valuable. We may not be able to say ‘Why’ a Back Navigation Rate is high, for example, but I think that in certain situations, such as in the confined universe of the check-out procedure, something is wrong if the visitors move back 30% of the time. Also, I think, if a page (no matter where on the site) has a very high Turn Page Rate you would be well-advised to give it some attention. A very high Turn Page Rate for a page typically happens when visitors go back and forth between the same pages many times. Can that ever be desirable?
Thanks a lot for your input, it is very much appreciated!
July 17th, 2008 at 2:57 am
I think by bringing local search into the conversation, metrics for this type of analysis make more sense.
Local search, off-site search, really any situation where you know your goal is to get visitors to do some specific thing after they’ve done something specific. Using the local search example, where your goal is to provide relevant links based on visitor input, I think it would be quite valuable. I’m not sure you could come up with a goal, other than “less”, on a per-targeted-keyword basis… guess it depends!
Although it seems a stronger metric when applied to local search, or a situation where only the site being analyzed is involved, this metric is useful to me as a substitute for bounce rate in certain cases… bounce rate defined as based on an entry page (typical) causing additional off-site searches to be lumped into the same visit… though one would have to be careful of situations where the visitor simply prefers the prior engine (I can be a jerk like that).
July 17th, 2008 at 10:42 am
Hi Todd,
Thanks a lot for your input! I think you make a very important point: back navigations are in fact especially interesting in connection with search.
The funny thing is that after I wrote this post on usability metrics, I got involved in a project for one of our customers, where we, among other things, analyzed their back navigation rates broken down on content. We found that it was precisely their local search engine which produced the highest back navigation rates. The search engine allowed users to search for available hotels using certain conditions like geographical area, price level and facilities of the hotel. However, once the result page came up, the visitors had to go back and forth between all the different hotels in other to compare them with each other. On the one hand these back navigations were signs of engagement, but on the other they were also signs of “inefficient” visits were visitors had to see some of the pages many time in order to do what they wanted.
Our solution to this problem was 1. to include a “View next hotel” button at each hotel-details page so that visitors didn’t have to return to the result / overview page to select another hotel. 2. to provide more details on the overview page where all the different hotels which meet the visitors search criteria were listed.
We hope this will make it easier for the visitor to compare the hotels with each other and that they will have more “efficient” visits with fewer irritating back navigations. Our customer hasn’t implemented these recommendations yet, but in the end we hope, of course, that the conversion rate (online booking of hotels) will go up. It will be interesting to monitor
I think this case proves the importance of your point. Again, thanks a lot for sharing your views!