The editor-in-chief of the open-access journal Remote Sensing has resigned. An editorial in the journal explains the situation. The short version is that this journal is the one that let that Spencer & Braswell paper slip through peer review; that publication has fatal flaws throughout its assumptions and analysis. The paper was quickly and brutally criticized in the climate-blogosphere, but exaggerated and praised in the right-wing, and much of the mainstream, media. The journal was criticized for letting such a poor paper get published. The editor-in-chief, Wolfgang Wagner, has now had a chance to review the criticisms and the paper and the review process. He has come to the conclusion that the paper should not have been published. The reason seems to come down to poor selection of reviewers by the editor that was in charge of that paper. The reviews came back with little criticism, to which the authors responded, and that basically tied the editor's hands and the paper had to be accepted.
From my point of view, the failing was the editorial board's misunderstanding of the subject matter and clear mishandling of the review process. From the tone and content of the paper, it was clearly a contrarian point of view and had the flavor of ideological bias. I suspect that the reviewers were selected from the list of suggested reviewers supplied by the authors. In cases of controversial content, there needs to be at least one critical reviewer selected from the community involved, and I'm pretty sure none of the reviewers in this case were in the mainstream of the climate change community.
I'm glad that Wolfgang Wagner has stepped down. I don't know that he did anything wrong except let the other editors make some bad decisions, but this move does help to show that the journal doesn't want to build a reputation for publishing garbage, contrarian papers. Hopefully this will be a small blemish on the journal's reputation which won't mar the whole thing. It will be interesting to see if the next editor-in-chief conducts a review of the evidence and officially retracts the paper (it is unclear whether this journal has provisions for that, but they should consider it).
The original paper can be found from the journal's web page. The editorial is worth a read, and can be found in the current issue [LINK]. On a related note, a new GRL paper by Andy Dessler destroys the Spencer & Braswell argument in under 4 pages [LINK].
UPDATE: Lots of coverage of this story on the climate blogs. One worth seeing is Mooney's post on DeSmogBlog, which rehashes some similar scandals involving climate, intelligent design, and autism. [LINK]
Showing posts with label peer review. Show all posts
Showing posts with label peer review. Show all posts
2011-09-06
2011-07-26
A peer-review recipe
Eos is on a hot streak, at least compared to the usual Eos status quo. There's another useful article in the 12 July issue. It is an overview, or maybe a primer, about peer reviewing [LINK]. It is written by K.A. Nicholas and W. Gordon. I'm not sure what they do, Nicholas is at Lund U. in Sweden, and Gordon is at UT Austin. The article is a bit prescriptive for my taste, but I think it will be helpful reading for graduate students and others who are doing their first reviews. It's a good reminder for the rest of us about how we should think about some aspects of doing reviews.
One difference between my reviews and the ones outlined by the article is in how to treat fatally flawed manuscripts and majorly flawed ones. The article suggests identifying these errors and being done with the review. In my experience, there is a good chance that the manuscript will still get "accepted subject to major revisions," so this may be the only chance to really convey to the authors what they need to improve. (Otherwise, there's the risk of multiple rounds of reviews, which no one wants.) To that end, I always provide a full review, even if I am recommending rejection of the paper. I always hope that the editor will appreciate the effort, and also that the authors will get more constructive criticism to help improve a new version of the manuscript, or at least have a comprehensive set of comments to address in a revised version that will come back for another review.
On a side note, this is also helpful, at least in principle, in the "dialog" between the climate science community and the climate change denying community. There is a lot of mistrust of the peer-review process among "skeptics" (note they are not skeptical, really). The article in Eos is a simple explanation of how peer-reviews are written, and I think almost everyone will agree that the description is roughly what reviews look like. While sometimes reviews are more or less helpful, the thought behind them is basically the one described by the article. They are meant as constructive criticism, pointing out strengths and weaknesses of work, judging the contribution to the field, and recommending whether publication is reasonable or not. This is how I write reviews, and it is similar to reviews I receive. In climate science, reviews are usually anonymous, though there's always the option of signing a review so that the authors know who you are. The decision about how the submitted manuscript proceeds comes from the editor, not directly from the reviewers, so even if a paper receives an unfair review, the editor can take corrective action. When that doesn't happen, the authors can appeal to the editor, pointing out where the review has gone off track. The usual course of action would be for the editor to find another reviewer, who can then corroborate the criticism or back up the author. There really isn't anything secretive or mysterious about the process. There are obvious measures to avoid conflicts of interest, and procedures for interaction among the participants. So while there are interesting arguments about how to improve peer review (blinded reviews, e.g.), I think the current system is quite transparent and attempts to be fair to all involved. I hope that critics of climate science will read this article (and David Schultz's book, LINK) before they attack the peer review process in general.
2010-11-24
Invitations to contribute chapters
Yesterday I opened my email and found an invitation to contribute a chapter to a book on climate change. Huh? Yeah, reading through, it wasn't obviously random, they knew my name and my work, and they were asking my to write something related. Here's the email, with some info removed:
I've never heard of InTech or InTechWeb before, and I don't know Iva Lipovic, and I've never seen someone use MSc as a title before, so many red flags were waving. So to the interwebs! First stop is the website they sent me to, to check out this book, which seems to exist, and the web site is nice. I start to look at the InTech website, which is also nice. What doesn't look very good are the books themselves. I looked through the books on topics that I might know something about, in particular Climate Change and Variability [LINK]. I think I know one author in the entire book. The topics seems disjointed. The publisher does not appear to be InTech or InTechWeb, but Sciyo. Each chapter does seem to be like a real paper, but from some browsing, some of them seem to be low quality. There's a lack of editorial cohesion, in the sense that there are differences in formatting and style across the chapters. Something doesn't seem right. So now I start the Google search process.
The most useful information I've seen so far comes from an interview with Sciyo CEO
Aleksandar Lazinica by Richard Poynder [LINK]. From all appearances, this is a business model that takes advantage of the Open Access process, wherein research results are "open" to the public online, and authors pay a modest fee to the journal/publisher to cover costs. Open access is a legitimate publishing model, as evidenced by relatively high profile journals like PLoS One and ACP, but the future of this model is certainly far from settled. InTech, now called Sciyo, seems to be a mutation of the general Open Access publishing model. Instead of trying to attract high quality and high impact papers to specialized journals, they publish books for free online using "InTech" as the online publisher (there's no clear distinction between what is InTech and what is Sciyo, so I will use them interchangeably for the remainder of the post.) The catch is that they company appears to be centered on the idea of soliciting chapters from authors and charging them for publishing the chapters. The fee is not outrageous compared to standard journal fees, but this all starts to feel like a vanity press. This feeling seems confirmed by the description of the publication process on the InTech website, for example:
I've been trying to follow up my initial searches, but with limited success. There's a little entry on an Economist blog [LINK], and some of those comments are interesting. I think there's a fair comparison between the Sciyo publishing model and both Who's Who and those poetry "contests" that have been around forever. As far as the scientific enterprise goes, the Sciyo models is problematic. Since it does not provide reasonable peer review, the reader is left to determine the quality of the research (with no baseline, as opposed to traditionally reviewed papers where there is at least some credibility to start with). Another blog has a similar story to tell [LINK], and again several interesting comments from people invited to contribute chapters and even people who have done it. Still, the only thing left to conclude is that this is a pay-to-publish model with no peer-review and no evidence of any actual benefit from having these non-reviewed publications on one's CV.
On the plus side though, this could be an interesting model for people who want to publish a book, but want to dispense with some of the overhead. An industrious editor or two could conceivably use the Sciyo system as a platform to get a collection of papers into book-form for a modest price. The downside of being non-reviewed would remain, but could be overcome by having some big names in a given field attached (and by the editor weeding out the sub-par contributions from the "invited" contributions). The result could be a useful resource for some small field, since the books really are free to download. I imagine a group of specialists getting together to basically write a free online textbook for grad students, for example, giving an overview of recent results. This is just daydreaming though, until Sciyo or some other OA publisher decides to get serious about such projects; the current model would probably demand many more papers in any given volume in order to collect more publication fees.
My name is MSc Iva Lipovic and I am contacting you regarding a new InTech book project under the working title "Climate Change", ISBN: 978-953-307-419-1.
This book will be published by InTech - an Open Access publisher covering the fields of Science, Technology and Medicine.
You are invited to participate in this book project based on your paper "XXXXX", your publishing history and the quality of your research. However, we are not asking you to republish your work, but we would like you to prepare a new paper on one of the topics this book project covers.
Publication of the book is scheduled for 26 July, 2011. It will be abstracted and indexed in major online repositories and search engines. The book will also be available online and you will receive a hard copy via express delivery service.
Why should you participate?
- "Climate Change" covers your area of research
- Free online availability increases your paper's impact
- Each InTech book chapter is downloaded approximately 1000 times per month
- More citations of your work (research findings indicate that papers published under the Open Access model are likely to enjoy increased citation rates)
- You keep the copyright to your work
NEXT STEP: For further details about this book project please visit
http://www.intechweb.org/XXXXXXXX
On this page you can find a detailed description of the book project, its scope and topics, details of the publishing process and a registration form.
For further details about InTech and Open Access please visit:
- About InTech: http://www.intechweb.org/XXXXXXXX
- About Open Access: http://www.intechweb.org/XXXXXXXX
If you need more information about this book project, InTech or Open Access, please don't hesitate to contact me.
On behalf of InTech President, Dr. Aleksandar Lazinica,
MSc Iva Lipovic
Publishing Process Manager
---
InTech
Open Access Publisher
e-mail: lipovic@intechweb.org
Web: http://www.intechweb.org/
Phone: +385 (51) 686 165
Fax: +385 (51) 686 166
Vienna Office
Corporate Address
Zieglergasse 14
1070 Vienna
Austria, European Union
Rijeka Office
Publishing, Marketing and Finance
Janeza Trdine 9
51000 Rijeka
Croatia
Visit us in our Operations Centre in Rijeka!
I've never heard of InTech or InTechWeb before, and I don't know Iva Lipovic, and I've never seen someone use MSc as a title before, so many red flags were waving. So to the interwebs! First stop is the website they sent me to, to check out this book, which seems to exist, and the web site is nice. I start to look at the InTech website, which is also nice. What doesn't look very good are the books themselves. I looked through the books on topics that I might know something about, in particular Climate Change and Variability [LINK]. I think I know one author in the entire book. The topics seems disjointed. The publisher does not appear to be InTech or InTechWeb, but Sciyo. Each chapter does seem to be like a real paper, but from some browsing, some of them seem to be low quality. There's a lack of editorial cohesion, in the sense that there are differences in formatting and style across the chapters. Something doesn't seem right. So now I start the Google search process.
The most useful information I've seen so far comes from an interview with Sciyo CEO
Aleksandar Lazinica by Richard Poynder [LINK]. From all appearances, this is a business model that takes advantage of the Open Access process, wherein research results are "open" to the public online, and authors pay a modest fee to the journal/publisher to cover costs. Open access is a legitimate publishing model, as evidenced by relatively high profile journals like PLoS One and ACP, but the future of this model is certainly far from settled. InTech, now called Sciyo, seems to be a mutation of the general Open Access publishing model. Instead of trying to attract high quality and high impact papers to specialized journals, they publish books for free online using "InTech" as the online publisher (there's no clear distinction between what is InTech and what is Sciyo, so I will use them interchangeably for the remainder of the post.) The catch is that they company appears to be centered on the idea of soliciting chapters from authors and charging them for publishing the chapters. The fee is not outrageous compared to standard journal fees, but this all starts to feel like a vanity press. This feeling seems confirmed by the description of the publication process on the InTech website, for example:
In comparison with scientific journals, the book format is different in scope as well as in length. Furthermore, the book publishing process has to follow strict publishing deadlines. In order to accommodate these differences, we have developed a strict review process without compromising the quality of our publications.This is all just saying that the book editor is the only "review" of the content of the book, there is no external review of the science, and it is apparent that there is little or no copy editing. I can only conclude that Sciyo/InTech, in this current form, is a scam designed to publish as much as possible and collect publication fees along with whatever advertising revenue they can generate. Maybe this is recourse for those struggling to get a mediocre paper published without going through the hard work of making it acceptable to a mainstream journal? Isn't that what ArXiv.org is for?
The Subject Editor’s screening and the Editor’s review are the conditions of acceptance for publication. Subject Editors are permanent members of our Editorial Board and, given their scientific expertise in a specific field of research, they are responsible for sorting abstracts by scope and topics. Book Editors review the abstracts ans select resourceful research papers with a bearing on developments in the field. They have overall responsibility for the content of the publication, therefore they pay particular attention to originality, research methods, key results, and language.
Only abstracts that meet all scientific requirements are accepted. However, definitive acceptance is based on the final chapter review. Following the submission of full chapters, the Book Editor is in charge of the final quality check and every effort is made to ensure that manuscripts are reviewed efficiently and to a high quality.
I've been trying to follow up my initial searches, but with limited success. There's a little entry on an Economist blog [LINK], and some of those comments are interesting. I think there's a fair comparison between the Sciyo publishing model and both Who's Who and those poetry "contests" that have been around forever. As far as the scientific enterprise goes, the Sciyo models is problematic. Since it does not provide reasonable peer review, the reader is left to determine the quality of the research (with no baseline, as opposed to traditionally reviewed papers where there is at least some credibility to start with). Another blog has a similar story to tell [LINK], and again several interesting comments from people invited to contribute chapters and even people who have done it. Still, the only thing left to conclude is that this is a pay-to-publish model with no peer-review and no evidence of any actual benefit from having these non-reviewed publications on one's CV.
On the plus side though, this could be an interesting model for people who want to publish a book, but want to dispense with some of the overhead. An industrious editor or two could conceivably use the Sciyo system as a platform to get a collection of papers into book-form for a modest price. The downside of being non-reviewed would remain, but could be overcome by having some big names in a given field attached (and by the editor weeding out the sub-par contributions from the "invited" contributions). The result could be a useful resource for some small field, since the books really are free to download. I imagine a group of specialists getting together to basically write a free online textbook for grad students, for example, giving an overview of recent results. This is just daydreaming though, until Sciyo or some other OA publisher decides to get serious about such projects; the current model would probably demand many more papers in any given volume in order to collect more publication fees.
2008-08-13
Endangered Species Act to be undermined by new rules
This is outrageous. The Bush administration is trying to introduce some kind of executive rule that would allow federal agencies to conveniently ignore the impact of projects on endangered species. Not only that, but the new rule would apparently ban federal agencies from assessing the greenhouse gas emissions from projects. You need to go read the story, there's one at Yahoo!. There are some additional reactions at gristmill.
I suppose that some might believe this argument that the current system is bloated and that federal agencies really can do their own assessments. However, the idea of independent review is so fundamental to science, and is so successful, that to just throw it aside for an intrinsically flawed self-evaluation system seems ludicrous to me.
We'll see how this plays out. Whether my paranoid, leftist brain is overpowering my rational, pragmatic one will come clear in the coming weeks and months.
I suppose that some might believe this argument that the current system is bloated and that federal agencies really can do their own assessments. However, the idea of independent review is so fundamental to science, and is so successful, that to just throw it aside for an intrinsically flawed self-evaluation system seems ludicrous to me.
We'll see how this plays out. Whether my paranoid, leftist brain is overpowering my rational, pragmatic one will come clear in the coming weeks and months.
2008-01-22
Sex bias in peer review
I think this is an important issue, not just in terms of sex bias, but that the whole peer-review process could probably be improved.
This study (http://dx.doi.org/10.1016/j.tree.2007.07.008) by Amber Budden at U. Toronto, suggests that female authors are more successful in a double-blind peer review process rather than the more conventional single-blind review.
Just to be clear, double-blind means that the author doesn't know who the reviewers are and the reviewer does not see the name or affiliation of the author(s). Single-blind means the author submits the paper and the journal/editor finds reviewers (the author doesn't know who they are), but the reviewers see the author's name(s). Almost every scientific journal uses this single-blind approach; for no good reason.
From where I sit, there seems no good reason to maintain this single-blind review process. Not only does it possibly discriminate against women, I think there are likely many more negative effects. The most obvious one is that "prestigious" scientists, those who might have published a lot or have contributed seminal work in a field, seem more likely to get through the process less critically. This is part because they are good scientists, of course, but it can also be because there really are not enough reviewers to go around, and much more junior scientists (sometimes grad students) end up reviewing papers. It is intimidating as an inexperienced scientists to be critical of work by someone you know/respect/fear/want-to-work-with/etc. Also, many sub-disciplines are populated by a fairly small number of experts, who end up being asked to review each other's papers all the time. This can go either way: people are likely to be extra critical of rivals and less critical of friends.
Recently, there has been some open discussion of the review process (e.g., DOI: 10.1126/science.319.5859.32b and DOI: 10.1126/science.319.5859.32c), which is good. However, I haven't noticed any large-scale call for double-blind review. This is amazing, since double-blind studies are a foundation of modern science. I honestly can't think of a single reason that every journal should not immediately switch to double-blind reviews.
This study (http://dx.doi.org/10.1016/j.tree.2007.07.008) by Amber Budden at U. Toronto, suggests that female authors are more successful in a double-blind peer review process rather than the more conventional single-blind review.
Just to be clear, double-blind means that the author doesn't know who the reviewers are and the reviewer does not see the name or affiliation of the author(s). Single-blind means the author submits the paper and the journal/editor finds reviewers (the author doesn't know who they are), but the reviewers see the author's name(s). Almost every scientific journal uses this single-blind approach; for no good reason.
From where I sit, there seems no good reason to maintain this single-blind review process. Not only does it possibly discriminate against women, I think there are likely many more negative effects. The most obvious one is that "prestigious" scientists, those who might have published a lot or have contributed seminal work in a field, seem more likely to get through the process less critically. This is part because they are good scientists, of course, but it can also be because there really are not enough reviewers to go around, and much more junior scientists (sometimes grad students) end up reviewing papers. It is intimidating as an inexperienced scientists to be critical of work by someone you know/respect/fear/want-to-work-with/etc. Also, many sub-disciplines are populated by a fairly small number of experts, who end up being asked to review each other's papers all the time. This can go either way: people are likely to be extra critical of rivals and less critical of friends.
Recently, there has been some open discussion of the review process (e.g., DOI: 10.1126/science.319.5859.32b and DOI: 10.1126/science.319.5859.32c), which is good. However, I haven't noticed any large-scale call for double-blind review. This is amazing, since double-blind studies are a foundation of modern science. I honestly can't think of a single reason that every journal should not immediately switch to double-blind reviews.
Subscribe to:
Posts (Atom)