Your First Fake-News Detection Algorithm

There were two kind of interesting "fake news" stories this past week, illustrating the depth of the problem. One was easy, the Internet spread the rumor that CNN had broadcast a half hour of porn instead of Anthony Bourdain. Even the first reports said that only one person in Boston had seen it. Bourdain's pretty good, I figured more than one person would have tuned in and seen the porn if it was real, and when I first saw this story I concluded it was wrong. But it did get retweeted and forwarded thousands of times. Actually, my spidey-sense says that someone was doing an experiment, creating "fake fake news" to see how far it could go.

The second incident is harder to label. This time it's fake news about fake news. And here's the problem, it comes from a "reputable" source -- the Washington Post. They put out a story on Thursday that started like this:

The flood of “fake news” this election season got support from a sophisticated Russian propaganda campaign that created and spread misleading articles online with the goal of punishing Democrat Hillary Clinton, helping Republican Donald Trump and undermining faith in American democracy, say independent researchers who tracked the operation. Russian propaganda effort helped spread ‘fake news’ during election, experts say

Reading the story in one of the world's most legitimate sources, you would sip your coffee and say to your spouse, "Look what those Russians are doing." Most of us don't look up and go, "Huh, hon, it doesn't seem to say who these 'independent researchers' are."

Journalist Glenn Greenwald jumped all over this one (after Fortune noted the BS) with a story headlined, "
Washington Post Disgracefully Promotes a McCarthyite Blacklist From a New, Hidden, and Very Shady Group:

In casting the group behind this website as “experts,” the Post described PropOrNot simply as “a nonpartisan collection of researchers with foreign policy, military and technology backgrounds.” Not one individual at the organization is named. The executive director is quoted, but only on the condition of anonymity, which the Post said it was providing the group “to avoid being targeted by Russia’s legions of skilled hackers.”

In other words, the individuals behind this newly created group are publicly branding journalists and news outlets as tools of Russian propaganda – even calling on the FBI to investigate them for espionage – while cowardly hiding their own identities. The group promoted by the Post thus embodies the toxic essence of Joseph McCarthy but without the courage to attach their names to their blacklist. Echoing the Wisconsin Senator, the group refers to its lengthy collection of sites spouting Russian propaganda as “The List.”

Greenwald contacted the journalist, Craig Timberg, to ask him about the story, and got the response: "I’m sorry, I can’t comment about stories I’ve written for the Post."

If you are thinking about what kind of algorithm would be able to remove fake news from a "trending" feed or message stream, you would think that one of the major factors would be something like reputation of the publisher. The Washington Post says that this undocumented story about unnamed experts anonymously judging news sources as Russian propaganda was one of the most widely circulated political news articles on social media in the days after it came out -- and why not? It's The Post. Must be true. If it came from something like deplorablepatriots-dot-com your algorithm might reject it on reputation grounds, but The Post's name carries the story forward. You can read that story and never even wonder why they don't give any names (there are lots and lots of things wrong with this story -- go read Greenwald's takedown).

The CNN-porn story had no weight of reputation behind it. Somebody called "Solikearose" tweeted “Did anyone else with RCN in Boston see the hardcore porn that was broadcast by CNN by mistake?” BuzzFeed seems to have actually talked with the person, who swears she saw it, but nobody else did. As they write:

But soon enough, the story had gone viral, along with the notion that it was a widespread problem affecting “viewers in Boston.”

Dozens of news outlets including Variety, the New York Post, and Esquire ran with it. Their stories were shared thousands of times on Facebook and Twitter.

Some reports suggested that as many as 300,000 RCN subscribers may have been affected by the “epic blunder,” despite only a single person having complained about the problem.

CNN apologized and blamed RCN. the local Boston service provider, but then a little later tweeted, "Despite media reports to the contrary, RCN assures us that there was no interruption of CNN’s programming in the Boston area last night."

We don't know if that one person was hallucinating or pranking the world, or if her neighbor's cable signal leaked into hers or what, but she was the only one who saw any porn on CNN. The question then is why so many people picked up this story and re-posted it.

"Truth" is an impossible criterion. You can't tell by looking at a statement whether it is true or not. If you have ever been in a situation that was reported in the news, you will realize that most news has some fakery in it. It may be something left out, or an adjective that simplifies the interpretation while calling attention from more important aspects of the story, and often the details are simply inaccurate. Scientific findings are evaluated by peer review, which is really just some people saying the research looks okay to them -- science is an evolving body of successive approximations to truth. Knowledge itself is an evolving consensual schema, where what somebody "knew" a hundred years ago might have turned out to be absolutely false, just like what we know now. So your fake-news algorithm cannot evaluate truth directly, it has to look at source reputation, internal consistency and logical structure, it needs to compare details to reputable external sources.

For instance, the widely publicized "FBI Agent Suspected in Hillary Email Leaks Found Dead" story supposedly happened in a town that doesn't exist, quoted a Chief of Police who did not exist. The address given for the news publisher was a tree in a parking lot next to a vacant bank building. Your fake-news algorithm should pick up some of those things.

A fake-news detection algorithm might be tougher to develop than one that passes a Turing test. It would have to be able to recognize intent, inconsistency, shady sources, facts that do not appear to have widespread support, including new facts that are true and old facts that are not. This strains the capabilities of artificial intelligence but I bet something good will come out of learning to solve this problem.

In the meantime, these are things people should be doing for ourselves. We should be reading critically. We should look up sources and citations when we encounter them, at least before we forward the message on to others. Snopes.com is a great resource for catching "urban legends." Google a fact, see if it is an echo-chamber artifact or something that real news sources are carrying.