Upon finishing Everybody Lies by Seth Stephens-Davidowitz, I thought that it probably could be a condensed into a long magazine article. And I was right-- here is one by the author himself covering noteworthy highlights so I do not have to. Or if that’s too long, just the subtitle tells the story, "Big Data, New Data, And What The Internet Can Tell Us About Who We Really Are". I’m not saying the work involved to derive these insights was small, but really, you’re interested in the answers, right?
SSD (the author) basically believes that the hugeness of our single-point-of-failure internet giants provides enough statistical muscle to make some new and bold assertions about the world. Basically if a billion people search for something no one thought other people were searching for, that is interesting. I can’t imagine any reasonable person disagreeing with that.
Unfortunately, this new insight tends to show us that this hidden world is generally worse than we have supposed. Indeed, I think that this book contained the most depressing sentence I have ever read in my life: "If there is a genre of porn in which violence is perpetrated against a woman, my analysis of the data shows that it almost always appeals disproportionately to women." If you think I’m being overly sensitive about it consider the magnitudes, "Fully 25 percent of female searches for straight porn emphasize the pain and/or humiliation of the woman…" and "Five percent look for nonconsensual sex…" That’s one in twenty women looking to watch a rape on a popular porn site that actually forbids such content.
After that, the fact that people are horrendous racists almost seems banal. Oh, ya, people are horrendous racists it turns out. By the way. FYI.
The titillating and depressing content of this book is all over the internet. I’ll let you seek it out if you dig that sort of thing. I will simply say that the data is substantial and the methods seem credible. SSD wrote a book. Cool. Moving on.
SSD likes to think (probably correctly) that the insights of this kind of data can shed light on victims who normally do not have a voice, especially when they are being oppressed by more a more powerful group. The classic example is child abuse. This got me thinking about another vulnerable group who are viciously victimized by a powerful group who are good at preserving the secrecy of their horrific misdeeds. That brings me to my job where I spend some of my time trying to penetrate the shroud of secrecy erected by the pharmaceutical industry.
If you’ve been to the doctor in the last 100 years or so, what happened? More than likely he gave you some magic beans to eat. So here’s a funny question, how do you know those magic beans are not poison? Think about it—if a bunch of people who went to a doctor drop dead, well, that happens, right? So how do we know those magic beans are up to snuff (better than snuff!)? Most people will just say "uh because… doctors?" Some very stupid people will say "the free market". Some intelligent people will say, "Ah, the FDA!" Some might say "science" or "trials". But that actually does not tell you if the magic beans you’ve been prescribed are poison. All that can do is give you some hints about how to extrapolate a trial of hundreds (normally beset with corruption) to a blockbuster market of tens of millions. What could go wrong? So surely once 10 million people are taking the magic beans, there is some kind of review of how that’s actually working out. Right? There must be. Right?
Sorry. No.
I’ll save my work on the embarrassing (to the medical profession) FDA Adverse Effect Reporting System for another time. But churning through that stinking landfill of bad data looking for meaningful scraps has put an idea into my head. If SSD’s methods can detect flu outbreaks, etc. they should also be able to reveal what kind of problems people who use Atorvastatin (Lipitor) actually have. This is not unknown. Pfizer knows exactly how lethal this poison is just like Catholic priests know exactly how many boys in the parish are getting sodomized. What’s missing is some sunshine because these guys are really good at keeping it in the dark.
So if you’d like to walk away with some optimism after learning these depressing things, consider that knowing the truth can ultimately inspire positive changes. The techniques of this book do seem able to shine a light of truth, pleasant or not. Let’s hope that light does some good.