Need a reality check? On peer-review and domain knowledge

Entering the Subway by rosemarie_mckeon
Entering the Subway, a photo by rosemarie_mckeon on Flickr.

Last week I attended IDCC14 in San Francisco, where I was immersed in digital curation. Naturally, things like peer-review and domain knowledge/expertise were on my mind, as things to consider with research (data) publishing. Then I saw this story about a Twitter data scientist “hacking” BART. I immediately retweeted it with a remark about not understanding how BART works (which is true). Right now, particularly in the Bay Area, there are a lot of “hacks” to solve problems that aren’t actually problems. It’s just that the people who perceive the problems don’t have the full picture and “hack” the solution for them, which in the case of public transit is only a small segment of the users. Joe Eskenazi wrote a good column about this in SF Weekly that looks at both sides. (Disclosure: We play futsal together, or did until I broke my finger in a game. Miss you Kamikaze!)

Reading Haque’s paper on arXiv, it’s clear to me he’s got the math and science stuff down – it’s the transportation that he’s lacking. A common issue with data scientists is that they often have the analytic and technological skills, but lack the domain expertise. So they have to work with experts or learn enough to become an expert (which takes time). Even if he just ran some of these ideas past a transportation engineering or planning masters student, they could have helped him refine the “problem”. Haque seemingly wrote this in a vacuum, so when it saw the light of the internet the transportation folk just picked it apart based on the faulty assumptions of how commuter rail fares work. (Note: Everybody thinks they’re an expert on transportation because they use it. Sorry, you’re most likely not.)

This is where peer-review could have been a good thing. I looked at the paper on arXiv to see if it was published elsewhere, such as a journal or conference. arXiv is often used as an open access repository for pre-publication manuscripts. Haque’s paper was not (as of yet) published or submitted elsewhere, which means there’s been no obvious peer-review which explains a lot.

Peer-review would have pointed out the flaws in Haque’s methodology (assuming the reviewers had the expertise). Instead, he got the open peer-review of Twitter and lots of transportation professionals and advocates, many of whom are tired of tech workers “hacking” transportation in a way that doesn’t really help. (Seriously, fare evasion with the help of an app and surge pricing on transit? BART isn’t Uber!)

There is a lot broken with peer review, but this is one case where it could have helped. I really hope Haque can hook into the very passionate and knowledgeable transportation community here in the Bay Area and start “hacking” some real problems. Let’s do it!

One thought on “Need a reality check? On peer-review and domain knowledge

  1. Thanks for reading the paper.

    The paper, which is actually nothing more than a technical blog post, identifies a concern which is obviously up to the domain experts to evaluate. This concern is a simple computational check of balanced fares along different routes. Although BART is a transportation system one would hope that basic knowledge of economics and computation have been applied when ascertaining the fares. Neither transportation nor governance and politics are my areas of interest or expertise. And neither have I claimed to be. The paper dwells exclusively on computation.

    While “hack” is a popular media buzzword to grab the attention of readers who have not more than 5 seconds on a news title, the paper is far less sensational. Very few of the comments I have received so far indicate the commentator have actually read the paper. And then there are others who are interested in reading the paper but first need the confirmation that reviewers of reputed conferences or journals have evaluated the content as critically as humanly possible. An academic is often easy to identify through such a bias. Having finished a full PhD program I have seen both sides of reviewers, conferences and journals.

    For a weeklong evening project, I am grateful to have caught the attention of so many, specially nice posts like this. Thanks.

Leave a Reply