I would like to open a debate. I would like to hear what you have to say about measuring the experience of good quality voice and videoconferencing versus actually experiencing that experience. In my business, there are a lot of people working long hours evolving the science of reportable statistics, i.e., measurements of network quality. And when it comes to bandwidth, throughput, latency, packet loss, and packet delivery ratios, everything is cool and the measurements are fairly strightforward. Where things get a bit crooked is when we try to use the same bases of measurement to report on user experiences.
There is one really big and good reason for this. Before we trained the users to talk about things per second and microsecond and millisecond, they were inclined to be more subjective in their assessments. You would hear invective like “VoIP sucks. Between the break-ups and the weird sounds and the echoes, I couldn’t understand half of what was said.” So we rushed in to correct them, of course. We taught them that it wasn’t the VoIP stuff but rather it was the network that was the problem. We taught them that some technical thing or other per second was too high (or too low) to support VoIP and videoconferencing over the Internet. We tried, and who can blame us, to get them to say that what they were expereincing was network latency and packet loss when what they were actually experiencing was garbled speech and pixelated images.
Lately, I have been reading about an initiative to adapt Mean Opinion Score (MOS) to make it applicable to video over IP and videoconferencing over IP. Personally, I think MOS is a trifle arbirtrary and even a bit funny. If you haven’t already, look it up and you’ll find that it is based on a user expereince sampling process that was only slightly more complex than walking around and asking “Can you here me now?” (Speaking of which, I think Verizon has it exactly right. They have eschewed the techni-blather in favor of a simple test of the only thing that matters to the consumer: Does it work?) And if MOS is a bit goofy to start with, what can we expect from a variation on MOS for measuring videoconference quality? (The day isn’t far off when the Verizon guy in the ads is going to be asking “Can you see me now?”)
I am as guilty of trying to push stats in place of experience as the next guy in this business. And I still do it. To some extent, it is simply the language we use when we communicate amongst ourselves in the network technologies community. The debate I want to have and the question I want to ask is about the best way to communicate the benefits of a good quality network with the users who are outside of our community. The reason I think this needs debating is because most users don’t actively care about the benefits of a good quality network. When things work as advertised, users have nothing to say. However, they care deeply about the bad things that happen when the online gamers create congestion at a router somewhere and it results in buckets of packets being dropped which causes their videoconference call to freeze up. In that case, they have plenty to say.
What do you have to say about measuring the users’ experience of a good quality network?
