Peer Review Blog

Blogging as scholarly communication

2007-04-23T11:08:00.000+02:00

David Rosenthal has posted some very perceptive thoughts on blogging as scholarly communication, including some thoughts on peer review as it happens via blogs. He lists the following reasons why academics are likely to use blogging more and more for scholarly communication:

The process is much faster. A few hours to a few days to create a post, then a few hours of intensive review, then a day or two in which the importance of the reviewed work becomes evident as other blogs link to it. Stuart's comment came 9 hours into a process that accumulated 217 comments in 30 hours. Contrast this with the ponderous pace of traditional academic communication.
The process is much more transparent. The entire history of the review is visible to everyone, in a citable and searchable form. Contrast this with the confidentiality-laden process of traditional scholarship.
Priority is obvious. All contributions are time-stamped, so disputes can be resolved objectively and quickly. They're less likely to fester and give rise to suspicions that confidentiality has been violated.
The process is meritocratic. Participation is open to all, not restricted to those chosen by mysterious processes that hide agendas. Participants may or may not be pseudonymous but their credibility is based on the visible record. Participants put their reputation on the line every time they post. The credibility of the whole blog depends on the credibility and frequency of other blogs linking to it - in other words the same measures applied to traditional journals, but in real time with transparency.
Equally, the process is error-tolerant. Staniford says "recognition on all our parts that this kind of work will have more errors in any given piece of writing, and its the collaborative debate process that converges towards the truth." This tolerance is possible because the investment in each step is small, and corrections can be made quickly. Because the penalty for error is lower, participants can afford to take more creative risk.
The process is both cooperative and competitive. Everyone is striving to improve their reputation by contributing. Of course, some contributions are negative, but the blog platforms and norms are evolving to cope with this inevitable downside of openness.
Review can be both broad and deep. Staniford says "The ability for anyone in the world, with who knows what skill set and knowledge base, to suddenly show up ... is just an amazing thing". And the review is about the written text, not about the formal credentials of the reviewers.
Good reviewing is visibly rewarded. Participants make their reputations not just by posting, but by commenting on posts. Its as easy to assess the quality of a participant reviews as to assess their authorship; both are visible in the public record.

If your ideas are important enough...

2006-08-24T08:40:00.000+02:00

There is an interesting story at ACRLog about reclusive mathematicians getting the word out about their work (and winning prizes) despite never publishing in regular journals and never submitting their work for traditional peer review. The bottom line: "If your ideas are important enough and you get them out, people will pay attention to them, whether you publish in a high prestige peer-reviewed journal or not."

Journal peer review for blog posts

2006-01-15T10:28:00.000+02:00

Tony Hirst of the OUseful Info blog has a great idea for academic journals - rather than standing aloof from the world of blogging, accept that many blog posts do have value and identify the better ones by means of peer review.

Tony's scheme would work as follows: People submit links to blog posts (or to other stories) to the journal's site. These then get reviewed by the journal's reviewers who make very simple yes-no type judgements, and depending on the number of votes a link gets it floats to the top or sinks into obscurity.

The idea is basically like having lots of mini-Digg's for academia, with the difference that not just anybody would be able to vote for or against a link, but only people designated as peer reviewers. As Tony points out, for this to work a journal would need to have many reviewers who are also active web users. What's nice about it though, is that reviewers won't be expected to spend a lot of time writing reviewers' reports and all that stuff - just a quick read and a simple decision - so it should be much easier to find cooperative reviewers than for conventional academic review.

I sometimes wonder if this model (quick review by many peers) might not sometimes also be useful for conventional academic publication, which currently relies too heavily on slow review by a few peers. There is also a third form of review already in use of course - slow review by many peers, which is essentially what happens when academic publications get cited by others in the years or decades following publication. This last form of review is in the long run perhaps the most useful of all, but is unfortunately currently still very much hampered by commercial indexing and abstracting services that are more interested in making money from citation data than in facilitating the tracking of academic conversations over time.

I look forward to the day when all academic content is freely available online and when a publication's list of sources cited is routinely also accompanied by a "this article has been cited by" list.

Wikipedia for medicine

2006-01-15T09:35:00.000+02:00

Dean Giustini considers the possibility that a wikipedia for medicine would be better than current forms of peer review.

Transforming the mechanisms and the purposes of peer review

2006-01-07T15:30:00.000+02:00

The system of peer review as currently practiced is premised on a publishing economy of scarcity - only a certain number of books and journals can be economically sustained, and peer review is therefore necessary as one mechanism to ensure that only the best work is published. With the advent of electronic publishing, says Kathleen Fitzpatrick, there is "a vast transformation in both the mechanisms and the purposes of peer-review" -

"What if peer-review took place not prior to publication but on texts that have already been made public? What if that peer-review happened not anonymously, in back-channel communications with individuals other than a text’s author, but in the open, in direct communication between reader and author? Technologies ranging from commenting to, as John Holbo suggested in a recent post on The Valve, a more elaborated P2P system, could be made to serve many of the purposes that current peer-review systems serve (most importantly for institutional purposes, the separating of wheat and chaff), but would shift the process of peer-review from one that determines whether a manuscript should be published to one that determines how it should be received."

Katleen makes several other interesting points in her post. Also have a look at the comments.

Publishers reject booker prize winners

2006-01-03T11:57:00.000+02:00

Not quite peer review, but an example of ineffectual reviewing nevertheless: It seems the UK Sunday Times sent manuscripts of the opening chapters of V.S. Naipaul’s In a Free State and a novel by Stanley Middleton to 20 publishers and agents. Both authors are Booker prize winners and Naipaul also won the Nobel prize for literature. None of the publishers or agents recognized the books and of the 21 replies received, all but one were rejections. This sort of thing is good fun and demonstrates the subjectivity of review, but it also highlights another important issue. To quote the Sunday Times article by Jonathan Calvert and Will Iredale:

"Many of the agencies find it hard to cope with the volume of submissions. One said last week that she receives up to 50 manuscripts a day, but takes on a maximum of only six new writers a year."

This is the real sticking point for most forms of expert review - too few experts and too much material to be reviewed.