Here’s a great and succint case study of how user research data can be easily misinterpreted, and a great example about why we should always be suspicious of statistics.
The marketing blog at FutureLab (which I do recommend) has a short post today entitled “Study Shows Fear of MySpace Predators is Overblown“.
The research paper (PDF) referenced in the post makes the following conclusions:
- Only 7% of those teens interviewed were ever approached by anyone with a sexual intent and nearly all of them simply ignored the person and blocked him from their page.
- Two-thirds of the parents were sure that there were many sexual predators on MySpace, while only one-third of the teenagers shared this concern.
- When asked about media coverage, 66% of the parents felt that it was either understated or close to the truth.
- Conversely, 58% of the teens felt it was vastly overblown.
At this point, several alarm bells should be going off in your head:
First, what parents or teens think of the media coverage is irrelevant. It’s a red herring. Perception of the media coverage has nothing to do to whether or not sexual predators are an actual problem.
Second, the very first statistic says that 7% of teens were approached by someone with sexual intent. That’s a huge number, really. If you just read the headline and didn’t read the actual body copy, you’d miss this. And if you read this number and didn’t think about it, you’d miss it. But 7% is a huge number!
Third, the study claims that nearly all of the teens who were approached were able to ignore or blocked the predator. If by “nearly all” they mean, say, 99 out of 100, that means that .07% of MySpace teens are approached by predators who are so aggressive that they cannot ignored or blocked. Assuming that there are, say, 10 million teens on MySpace (which is a low estimate, I’ll bet), that’s 7,000 incidents of aggressive predation. That’s a lot of serious incidents!
Finally, if you look at the end of the study’s press release, you will see that the study’s author is, in fact, writing a book about MySpace to “provide a candid view of the benefits and potential hazards of MySpace, and include helpful advice for parents to keep their children safe while allowing them to benefit from their virtual world experiences”.
The author’s conclusion may in fact be correct – MySpace may be perfectly safe. I’ll bet it is. It’s just that these numbers do not support that conclusion at all. Be alert!
Comments
3 responses to “User Research Smoke & Mirrors, Interlude: Data Interpreted Badly”
If the numbers are correct, MySpace is about 700% safer than the average American family.
Up to 45 percent of girls experience sexual abuse within the family (nuclear or extended) and between 25 and 35 percent of young boys — figures you can ratify in most journals dealing with child abuse, or by talking directly to abuse counselors and psychologists.
I lived with a psychologist expert in abuse and she came home more nights than not full of anger and frustration by here discovery of how rife the family is with abuse.
So maybe the point is MySpace, however bad it is, is light-years better than FamilySpace…which is the point of the matter.
Bob, I am skeptical of those numbers. Care to cite a source? For those numbers, what constitutes sexual abuse (I’ve heard of parents charged with abuse for taking pictures of their nude 3 year olds)? Over what duration does it cover (I imagine such numbers would normally cover a whole childhood)? I accept that such numbers are projections from officially reported numbers, since such abuse is wildly underreported, but how is that projection actually made? Does the person doing the calculation have an agenda to push? This is the meat of my gripe: throwing numbers around without context.
More importantly, the “700% safer” conclusion has no meaningful mathematical basis, as you are comparing wildly different contexts and measurements. If you compare the number of hours a child spends on MySpace versus the total number of hours spent with the family during childhood, for example, your 700% would probably drop down to less than 1% pretty quickly. That’s how meaningless making such pseudo-mathematical deductions is.
Misinterpreting data in this way also flies in the face of common sense: According to your logic, you might also conclude that a child would be safer if left alone in Times Square than they are at home in the average American household — this may seem to be true statistically, but does this mean that you should send your child alone out to Times Square instead of keeping them at home with your family?
And, using that same logic, if there is no sexual abuse going on in your own family (which is true for most families) then your child is infinitely more likely (actually, mathematically, they are guaranteed) to be abused on MySpace. The point is to be able to ascertain the danger of places outside of your control.
Christopher, interesting report if only for the complete lack of substance. A ‘convenience’ sample taken only in LA; a small sample at that; no reference to the purported ‘media hype’; reference to a ‘typical MySpace’ user – would love to know what proportion of users actually meet all of these criteria;