That’s an interesting hypothesis: Some non-respondents might reject the survey because they are simply too busy converting! This might cover up the fact that a sub-group of non-respondents could be very engaged, namely some of those 45% who, in my case study, do not bounce. This would explain why the sections “Online Booking” and “Find accommodation” (which both are strong indicators of booking intention) have more non-respondents than respondents. In total around 20% end their visit there. Notice, however, 20% of 45% is only 9%, meaning that the overall conclusion – that non-respondents are less engaged than respondents – probably still holds (provided that the bounce rate is more or less the same across all content sections).

Thanks very much for the tip!

How exciting to get a comment from somebody who also has experience with integrating online surveys and clickstream data!!

I’m not surprised, though, that this comment comes from a person working at Foviance. I have the greatest respect for Foviance and the company’s combined focus on usability and web analytics, which, I suppose, originates from the merger between The Usability Company and Web Abacus some years ago. I think usability is often ignored by the web analytics industry, so it’s really refreshing to hear that there are in fact some exceptions - apart from Netminers of course :-).

You say you had some difficulties finding differences between survey-based “persona” in terms of their actual behavior. I don’t know if this is because of the specific variables you used, but at Netminers we usually find striking, and very interesting, differences.

We typically pick one, primary variable, e.g. the question “Did you find what you were looking for”, and analyze how this correlates with viewed pages. Then we add supplementary variables to “construct” persona around the answers to the primary question: i.e. Yes, I found everything I was looking for; I found some of it, but not all; No, but I found some other interesting information; No, I didn’t find anything of value, etc.

This approach typically reveals interesting differences in terms of where on the website the most positive/negative experiences occur. This is great as a primer for more in-depth, qualitative usability studies, which are guided by, and can elaborate on, the statistical findings. We often recruit respondents for these in-depth studies on behalf of our qualitative usability partners.

Also, thanks for the tips on RFML technique. I’m glad to hear that you have found the same types of bias for survey data. There must be some truth to it, then!

Finally, I have a question for you: In this case study presented in this post there was an extremely high response rate for website visitors based in the UK. Do you have any idea why this might be the case? Could it be that online surveys are rarely used in the UK, meaning that fewer internet users have developed an online questionnaire aversion?

Thanks a lot for sharing you view!