What quality controls are utilised by PLOS ONE when selecting reviewers? Who is deemed eligible?

In 2011, I expressed concern about the PLOS ONE business model and its associated review process. My worries were focused on the use of what some have called light or “lite” peer review, and the “pay-to-publish” system used by PLOS ONE (and now by many other publishers). My worries were subsequently recorded on the PLOS ONE Wikipedia page. 

Recent personal experience has increased my concern, and left me wondering about the way in which reviewers are recruited by PLOS ONE. 

On 12^th July, I received an email from a PLOS ONE academic editor inviting me to peer review a paper. I won’t say what the paper was entitled, or who the authors were, but it was on the topic of open access journals.

Since I am a blogger/journalist rather than an academic I was surprised to receive the invitation, and emailed PLOS ONE with the following question: “I have had an invitation to review the above paper. Can you point me to the rules on the eligibility of PLOS ONE reviewers?”

I received the following (I assume boilerplate) reply:

This did not address my question, so I also emailed the academic editor whose name had been at the bottom of the invitation. I am not going to name him, but I will say that he is based at Universidad de Las Palmas de Gran Canaria, “Was it you who sent me an invitation to review the above paper?” I asked. “If so, I am wondering why you invited me. Can you say?”

I received the following response:

Grunt work

Let me be quite clear at the outset: I had and have no interest whatsoever in reviewing this or any other scholarly work, not least because there is absolutely no incentive for me to devote my time to reviewing papers. Moreover, the one time I did agree to review anything for an academic journal (an editorial rather than a paper), my suggestions were all rejected on the grounds that “the author says he is too busy to make the changes you suggested.” Clearly I had not made very good use of my time!

But as I say, my first response on receiving the PLOS ONE invitation was to wonder whether it is inappropriate for non-academics to review scholarly papers.

With these thoughts in mind I tweeted the invitation under the strapline “PLOS ONE invites journalist to review scholarly paper”. Somewhat to my surprise, everyone who responded said that they saw no problem with my reviewing a scholarly paper on open access (although it could not presumably be defined as “peer” review). Their reasoning was that they are confident that I have the necessary expertise. And Roger Schonfeld commented, “I’d like to see expertise welcomed into the scholarly conversation without regard to academic affiliation.”

On reading these responses I recalled that some OA advocates maintain that an important benefit of OA is that it encourages members of the public to take a greater interest in science, and to even take part in the process themselves – by means of “citizen science”.

A few days later I read this in the Times Higher: “the open access movement in academic publishing, while far from perfect, means that more research papers than ever are available to those outside universities. Are we not entitled to expect graduates in the workforce and the general public to show the curiosity and commitment to engage with research findings relevant to them?”

It added: “Academics should not hide in the ivory tower, but neither should non-academics refuse to re-enter it after they graduate. Its doors remain wide open. Surely it is not too much to ask for policymakers, journalists, non-governmental organisations and industry researchers to make a greater effort to access its treasures?”

That all sounds laudable enough, but I wonder if the authors of the Times Higher article (two cultural studies researchers) would want their scholarly papers reviewed by members of the public.

As it happens, I tend these days to turn a somewhat sceptical ear to people when they start extolling the virtues of citizen science. As I put it in an interview last year, when researchers talk about the benefits of citizen science they usually have in mind a scenario in which the public’s role is simply to provide free labour for crowdsourcing projects. Citizens are despatched to count butterflies, bugs or birds, stare at images of galaxies on their computer, or “swab and send”, and then feed the data back to bona fide scientists in the lab.

“This activity is clearly valuable,” I suggested, “but seems to imply that the role of citizens in science is to do the grunt work, while the real science continues to be undertaken by professionals.”

Left wondering

That said, I was still intrigued as to how and why I had been invited to review a paper for PLOS ONE. So I persisted in my attempts to get an explanation, putting to PLOS ONE the following three questions:

·       Does PLOS ONE allow or not allow people to review a paper where they are not a member of a university or other research institution?

·       Does PLOS ONE have a published list of the criteria by which reviewers can be deemed to be eligible to act as a reviewer? If not, can you tell me what they are anyway?

·       On what grounds was I invited to review the paper in question, and why was it later decided that the invitation was a mistake?

It was not easy to get an explanation out of PLOS ONE, despite two tweets from the publisher saying that it was looking into the matter (here and here).

Eventually (on 22^nd July) I received the following email:

Academic Editors at PLOS ONE have great autonomy when selecting reviewers, and the decision as to whether an individual expert is eligible to act as a reviewer for a specific manuscript lies with the Academic Editor. We have been in touch with [the academic editor] and he has noted, as we mentioned, that the reasons why specific individuals are approached as reviewers are numerous and may vary depending on the submission. In relation to the invitation you received [the academic editor] has reiterated that this was issued in error, he apologized to you for this error when you first raised this to his attention.

From this I conclude that there is in fact no policy on reviewer eligibility at PLOS ONE, although it would have been nice if the publisher had directly acknowledged this.

So I am left wondering how PLOS ONE academic editors make their decisions when recruiting reviews. Presumably there will be quite a diversity of views on eligibility. Is such a loose arrangement really satisfactory?

Interestingly, a few days later Chealsye Bowley, a librarian at Florida Gulf Coast Universitytweeted about a related problem she had experienced with F1000Research. She had been informed that as she does not have a PhD she is not eligible to submit a paper to the publishing platform.

The good news is that F1000Research does have a published authorship policy. The better news is that the company responded to Chealsye’s concerns by proposing to amend the policy to cater for people like her (although some still view the proposed amendment as unsatisfactory).

Like Chealsye I too do not have a PhD. I did spend a year at Moscow State University in 1981 studying the “village prose” writers (e.g. Vasily Belov and Valentin Rasputin), but the interminable queues I had to stand in at the Lenin Library simply to request access to the books I needed – along with the constantly flowing Столичная vodka – proved too great a distraction. I never finished my thesis.

Not very encouraging

However, the fact that I do not have a PhD would not have been known by the PLOS ONE academic editor since I have no CV online. Presumably, therefore, he withdrew the invitation when he realised I was not a researcher. But should he not have established that before sending me the invitation? When I later asked him how the mistake had occurred, and how I had been targeted, he did not reply.

But there are some clues as to how I was selected, and I don’t find them very encouraging. Let me explain: 

The invitation I received (below) was sent to me at 12:08 on 12 July 2016.

When I looked at my blog statistics I noted that four minutes earlier someone from the University of Las Palmas De Gran Canaria had accessed one of the pages on my web site (this one).

This was the only page the visitor from Gran Canaria accessed that day, so I am inferring that the vetting process went something like this:

1.     The academic editor does a Google search on “open access” and sees my name in one of the hits.

2.     He clicks on the link in that list and is sent to a page on my site.

3.     He quickly scrolls down that page, copies the email address at the bottom, and sends off his invitation.

This suggests that the academic editor did not think to check whether I was a researcher. If so, it is a pretty telling mistake. After all, the banner on the top of the page he accessed on my site quite clearly states that I am an “independent journalist”. In addition, the email addresses he copied clearly describes my status – richard.poynder@journalist.co.uk.

If that’s right, then the mistake the academic editor made was to spend far too little time establishing the eligibility of the person he invited to review his open access paper – around four minutes in fact. This is of course speculation on my part as the academic editor stopped responding to my emails.

This leaves me with three further questions:

1.     Do PLOS ONE editors routinely only do four minutes research when looking for, and establishing the eligibility of, a reviewer, or was this editor in an uncharacteristic hurry?

2.     Does PLOS ONE have no quality controls in place to enable it to itself verify the appropriateness of reviewers being recruited in its name? Does it simply leave everything to the academic editor?

3.     If I had not questioned the invitation, but just sent in a review, any old review, and that review had met the needs of the academic editor, would anyone have known that a “mistake” had been made? After all, PLOS ONE does not operate open peer review – its guidelines state: “Reviewers are anonymous by default. Reviewers’ identities are not revealed to authors or to other reviewers unless reviewers specifically request to be identified by signing their names at the end of their comments.”

Serious issues

And if the situation at PLOS ONE is as I infer it to be then I think it raises some serious issues. As Danny Kingsley has pointed out, problems with peer reviewers are now a big, and growing, problem. “In August 2015 Springer was forced to retract 64 articles from 10 journals, ‘after editorial checks spotted fake email addresses, and subsequent internal investigations uncovered fabricated peer review reports’”, she wrote recently. “They concluded the peer review process had been ‘compromised’.”

Kingsley added, “In November 2014, BioMed Central uncovered a scam where they were forced to retract close to 50 papers because of fake peer review issues. This prompted BioMed Central to produce the blog ‘Who reviews the reviewers?’ and Nature writing a story on Publishing: the peer review scam.”

Is not PLOS ONE laying itself open to similar scrapes and scandals? (As it is, PLOS ONE has already featured on Retraction Watch a number of times).

It is also worth noting that in the Naturearticle that Kingsley cites (published in 2014) the authors point out that the editorial system used by PLOS ONE (Editorial Manager) is insecure, since it “actually sends out a password, without prompting, whenever it asks a user to sign in, for example to review a new manuscript.” Two years later this is still happening, as I can now attest.

All in all, I feel my experience suggests there are two, related, issues here. First, there appears to be inadequate checking of reviewers at PLOS ONE, and apparently no official policy on reviewer eligibility.

Second, in describing my invitation as a mistake, PLOS ONE would seem to assume that the involvement of citizens in science should, at best, be no more than as occasional passive readers of papers, and/or fodder for citizen science grunt work.

As I said, I have absolutely no wish to review papers myself, but I am confident that many citizens would be more than willing to contribute. And so long as they had the necessary expertise they would probably do a better job than many academics (I say that having read quite a few reviews in my time). Either way, I would suggest that establishing the expertise of a reviewer (whoever they might be) requires more than a quick Google search.

Moreover, if, as many maintain, the boundary between citizens and scientists should be more porous, is PLOS ONE sending out the right message to the public with incidents like this? I also suspect that had I been a researcher I would have received a more detailed and respectful response to my enquiries from PLOS ONE.

The Ivory Tower is dead, long live the Ivory Tower!

SocArXiv debuts, as SSRN acquisition comes under scrutiny

The arrival of a new preprint server for the social sciences called SocArXiv comes just a month after news that Elsevier is acquiring the Social Science Research Network (SSRN), a preprint repository and online community founded in 1994 by two researchers. 

Given the concern and disappointment expressed over the SSRN purchase by researchers, it is no surprise that the launch of SocArXiv has been very well received. Still smarting from Elsevier’s 2013 acquisition of Mendeley – another formerly independent service for managing and sharing scholarly papers – many (especially OA advocates) were appalled to hear that the publisher has bought a second OA asset. The reasons for this were encapsulated in a blog post by University of Iowa law professor Paul Gowder entitled “SSRN has been captured by the enemy of open knowledge”.

This concern has also attracted the attention of the Federal Trade Commission (FTC) which has launched a review of the SSRN purchase. The FTC is currently contacting many institutions and experts in scholarly publishing to assess the implications of the acquisition, presumably in order to decide whether it needs to intervene in some way.

Elsevier is understandably keen to downplay the interest the US government is showing in its latest acquisition. “The Federal Trade Commission is conducting a routine, informal review of our acquisition of the Social Sciences Research network,” vice president and head of global corporate relations at Elsevier Tom Reller emailed me. “Elsevier’s interest in SSRN is and has been about SSRNs’ ethos, a place where it is free to upload, and free to download. We are working cooperatively with the FTC, and believe that the review will conclude favourably.”

In other words, Elsevier does not believe the FTC’s interest in its purchase will lead to a formal investigation.

But however timely SocArXiv’s launch may be, the service is not a response to the SSRN acquisition, the director of the new service, and professor of sociology at the University of Maryland, Philip Cohen assured me. “We were already in planning before we heard about the SSRN purchase.”

That said, the fledgling service is clearly keen to ride the wave of discontent that Elsevier’s latest move has engendered: it has begun to host preprints by means of a temporary arrangement until the platform’s construction is completed.

So what is SocArXiv? As the name suggests, it is modelled on the physics preprint server arXiv, and describes itself as a free, open access, open source archive for social science research. Authors are able to upload their preprints to the service and make them freely available to all. The papers will be provided with permanent identifiers to allow them to be linked to the latest version, or to versions published elsewhere. They can also be made available under Creative Commons licences, and analytics data will be provided to show how often papers have been accessed.

Registration to the service will be free and open to all, regardless of academic affiliation. In addition, registered users will be able to comment on and discuss papers.

A branded service

However, unlike arXiv – which was conceived (and still largely continues to act) as a supplement to the traditional scholarly publishing system, it is hoped that SocArXiv will disrupt the traditional system, and help to eventually supplant it.

On the LSE blog, Cohen writes, “I hope that SocArXiv will enable us to save research from the journal system. Once it’s built, anyone will be able to use it to organise their own peer-review community, to select and publish papers (though not exclusively), to review and comment on each other’s work — and to discover, cite, value, and share research unimpeded.”

Or as Cohen put it to me: “In the end I believe we need to replace the current journal system. I hope SocArXiv helps us move in that direction.”

“I’m very optimistic about SocArXiv,” Gowder told me. “The platform doesn’t seem to be fully online yet (the upload process seems quite convoluted, for example), but I don’t see any reason why they can’t get wide usage among political science and law as well as sociology.”

It is important to note that SocArXiv is being built on an open source platform developed by the Center for Open Science (COS). “It is their platform which they offer free,” explains Cohen.

Brian Nosek

Founded in 2013, COS is a non-profit organisation based in Charlottesville, Virginia, and over the past three years it has developed a range of services designed “to increase openness, integrity, and reproducibility of scientific research”. At the heart of the project is the Open Science Framework (OSF) which, says COS executive director Brian Nosek, consists of two main components – a back-end application framework and a front-end view (interface). “The back-end framework is an open-source, general set of tools and services that can be used to support virtually any service supporting the research lifecycle.”

In other words, multiple different interfaces and services can be built on top of the OSF, and SocArXiv will be using one of those interfaces. “SocArXiv will simply be a branded service built on a generalised OSF pre-print service which will be offered so that anyone from any discipline can share research manuscripts with their community and receive feedback,” explains Matt Spitzer, COS Community Manager.

This is important because it means that SocArXiv is likely to be just the first of many such initiatives we can expect to emerge from OSF. “We are envisioning that services for other disciplines will be built on the same platform, and we are discussing this with several groups already,” says Spitzer. “The general service will be referred to simply as OSF Preprints, but as other groups use the platform for a specific subset, they can brand the site however they wish.”

Collective mistake

But perhaps the real significance of SocArXiv – and the OSF – is that it points to a growing reluctance within the research community to rely on legacy publishers, or indeed any for-profit organisation (as SSRN always was). In retrospect, says Gowder, “I think legal academia and the other disciplines that made heavy use of SSRN made a serious collective mistake in the first place, entrusting their repositories of open knowledge to a for-profit enterprise run in a non-transparent fashion.”

It also reminds us that many observers of the open access movement are having to conclude that the initial strategy and approach of the movement was misconceived, since it assumed that it would be sufficient to persuade legacy publishers to embrace open access. By doing so, it was believed, not only would research become more open, but the high and constantly rising costs associated with scholarly communication could be reduced.

In the event, open access is increasing costs rather than reducing them, and publishers are seeking alternative ways of enclosing publicly-funded research – to their own benefit, not the benefit of the research community.

Of course this should not have come as a surprise. Legacy publishers were always going to try to protect their revenues, and they were always going to seek to maintain control of the publishing process in order to do that. That is simply what for-profit companies do.

Indeed, seen from the perspective of Elsevier, the acquisitions of Mendeley and SSRN were both very smart moves. Speaking to me on a non-attributable basis, a member of the financial community who has followed the publishing industry for a number of years said that the acquisitions “demonstrate that Elsevier is following a very clever strategy.”

He explained: “By investing in businesses like Mendeley and SSRN they are getting closer and more intimately connected to researchers and academics. It is a very intelligent way to ‘future proof’ Elsevier from massive changes in how the economics of academic research dissemination work.”

He adds, “If OA happens, being close to authors will matter a great deal: the more interactions between authors and Elsevier, the more likely they are to submit their articles for publication on Elsevier journals. In addition, there may be pricing benefits if Elsevier happens to provide additional services to authors (for example, connectivity, dissemination, hosting and storage of underlying data, etc.)”

The problem is that while such a strategy might suit Elsevier, it is far from clear that it will benefit the research community. As Geoffrey Bilder has pointed out, the OA movement assumed it was sufficient to make scholarly material freely available. Today, however, it is discovering that the scholarly infrastructure is itself being enclosed in the process. Consequently, he argues, “All the things we have been doing to liberate content and data is under threat because organisations are developing tools and environments and systems where in order to access that data and content we have to go and interact within enclosed walls. If that continues then we will see many of the benefits that have been gained from liberating content and data undone”.

Some will doubtless deny that this is, or will ever become, the case with Mendeley and SSRN, but the fact is that both now belong to a large commercial organisation accustomed to enjoying very high profits. Elsevier is not going to give up those profits without a fight.

The important point is that what non-profit services like SocArXiv can offer that commercial sites like Academia.edu, ResearchGate, Mendeley and SSRN cannot offer is what SocArXiv’s web site describes as a “non-commercial, open-source alternative to this enclosure of the commons”.

It is no surprise therefore that many are concluding that it is time for the research community to wean itself off for-profit publishers. As Gowder put it on his blog, any scholarly publishing effort now needs to be “Organised as a non-profit or trust with legal obligations to offer its services for free and remain unowned by any for-profit entity (to protect it from future Elsevier buyouts).”

And this is why the COS platform is so interesting.

Two paths

Essentially, the research community now finds itself at a crossroads. If it takes one path it will likely remain in thrall to the so-called Academic Publishing Oligopoly, and have to continue paying over the odds for the services and infrastructure its needs to publish and share its research.

If it takes the second path it can hope to recover its ownership of scholarly communication, and in the process regain control of costs. It is this second path that is the long-term goal of SocArXiv (and presumably of COS). Commenting on the SSRN purchase Cohen says, “I regret the acquisition of anything by Elsevier. We should be moving in the direction of non-profit, open-source, and open-access publishing, not away from it.”

As if to underline this point, last Friday reports emerged that SSRN was unilaterally taking down PDFs from its service, citing copyright concerns. This led to many complaints on Twitter, with Gowder declaring, “I am de-publishing every single article from SSRN immediately. Please join me. Only a swift community response will deter these people.”

SSRN responded in a tweet: “We sent the wrong email – that’s it – nothing has changed in our policies. There isn’t some big conspiracy happening.”

“If they just sent the wrong e-mail, as they risibly claim over Twitter, what explains the string of surprise takedowns people are reporting?” responds Gowder. “Maybe they meant to send a nicer e-mail with the surprise takedowns? The problem is the surprise takedowns, not the tone of the e-mail.”

This incident has been covered at Techdirt here. Meanwhile, my several emails to SSRN CEO Gregg Gordon have so far gone unanswered.

 

Philip Cohen

But what will it take to regain ownership of scholarly communication? Above all, it will require the research community to share more – and to co-operate and partner more – in order to build the alternative infrastructure, an infrastructure open in every sense of the word. “The good news is that there are no real barriers between truly open research products,” says Cohen. “Different technology and communities can pursue different innovative strategies, but in the end the research materials that we disseminate can be integrated with anyone else’s. I like to say, ‘meet you at the metadata’”.

With this objective in mind, the SocArXiv metadata will be integrated with services like SHARE – the “free, open, data set about research and scholarly activities across their life cycle” currently being developed by a number of US-based organisations, including the Association of Research Librariesand the Association of American Universities.”

Here again COS looks to be key, because the OSF is designed to facilitate exactly this kind of sharing, and COS is committed to an open-source approach. It is also busy recruiting other universities to its platform. “We have a beta community of universities participating in OSF for Institutionsto provide a seamless experience for researchers into the OSF,” says Nosek. “Some partners are: Notre Dame, University of Southern California, University of Virginia, and UC Riverside.” [More here].

OSF can also add value to institutional repositories. “Most institutions have repositories already, but submission compliance is usually very low, many are not built for modern research outputs – code, 3D images, etc. – and they do little to surface valuable insight about active research,” says Spitzer. “In addition, many are exploring ways to connect their existing institutional repositories to the OSF and connect them to the research workflow, which they are not today.”

Importantly, OSF Preprints can also aggregate search results across all the sub groups using it.

Three challenges

However, the research community faces three significant challenges. First, it will be necessary to reallocate money currently being used to pay journal subscriptions to fund new non-profit open access initiatives. This need comes at a particularly difficult time as the OA movement has recently become enamoured with the so-called journal “flipping” strategy (i.e. flipping subscription journals to an OA model – see here and here). This would redirect subscription money to paying legacy (commercial) publishers for gold OA, not to pay for non-commercial OA initiatives.

Second, it will require additional funding to build the necessary alternative infrastructure. 

Third, history suggests that universities do not co-operate very well, and this is particularly true in the neoliberal environment they now have to exist in.

Cohen expresses his hopes in this way: “I would like to encourage academics to shift their labour contributions from for-profit and paywalled ventures toward non-profit and open-access institutions. At the same time, I believe the institutions that currently pay vast sums for journal subscriptions should shift their funds toward open-access efforts. Making this transition is one of the most important challenges for academia of our time.”

 

Paul Gowder

Gowder believes the onus is on research institutions more than individual researchers. “Universities are getting screwed almost as badly as ordinary people are by the current system, and the universities are the ones who have the most power to change it.”

What is needed, he says, is for “a big university consortium to sponsor prestigious open-access journals in every discipline?” The challenge he concedes, is that universities currently suffer from a “collective action problem”. But the promise is that “They’d save a lot of money if they could figure out how to stop paying journal subscriptions and instead fund open scholarly dissemination.”

But it is the second challenge that is the most interesting one, and it is this that COS and initiatives like SocArXiv are hoping to address. It is probably important here that the core technology work is being done by an external organisation, not universities themselves, since universities have shown themselves to be not very good at creating scholarly infrastructure. This is the reason for the relative failure of the institutional repository movement (certainly in terms of ease-of-use and interoperability), and it is presumably the reason why the University of Florida recently took the controversial decisionto (in effect) outsource its institutional repository to Elsevier.

We also know that research funders are generally reluctant to fund infrastructure projects. When they do it is inevitably short-term funding, and so the infrastructure that has been created tends to die a death when the money runs out.

Gowder is nevertheless optimistic. “It’s a lot cheaper to operate online services now than it was a decade ago. As open-source tools proliferate and get better and the costs of things like data storage and bandwidth go down, the cost picture for these services is likely to improve. 

Funding

So how is SocArXiv being funded? “We are a program at the University of Maryland,” says Cohen. “That means we can receive contributions or grants through the university, and the university will ensure that our spending is consistent with its institutional mission.”

As part of its funding efforts, therefore, SocArXiv is currently seeking donationson the University of Maryland web site. Cohen explains: “We hope to be funded by the institutions and actors that currently pay for academic publishing: universities and their publishing operations, research agencies, and foundations. The nuts and bolts of that are yet to be determined, of course. At present we benefit from the funding that COS already has, which will enable us to get up and running on their preprint server and establish the need for what we’re doing.”

He adds: “In addition to what COS offers through their basic platform, we hope to raise money to pay for them to add features that our community wants (like a peer review system). Then presumably such features or tools would be open to everyone too.”

This would seem to offer a potential virtuous circle of co-operative funding.

Once again, COS is key. Not only has the underlying technology work already been done by a non-profit organisation, but the organisation concerned has shown itself to be pretty successful at attracting money from funders. COS has received regular grants and donations since it was founded, and currently has funding from – amongst others – the Alfred P. Sloan Foundation, the National Institutes of Health (NIH), the National Science Foundation (NSF) and the Laura and John Arnold Foundation.

What most agree on is that the research community cannot rely on governments to protect them from commercial predation. Commenting on the FTC’s interest in the SSRN acquisition Gowder says, “I’m no antitrust expert, but it seems like the FTC is unlikely to take the kind of aggressive stance that you’d need to make some kind of anti-merger action stick here.”

The financial expert I spoke to concurs. “I cannot imagine the FTC will take action. Elsevier has a relatively limited share in social sciences, and its main businesses (journals and search engines) are not directly affected by the acquisition of SSRN. The key concern is whether the business model of SSRN will be changed by Elsevier over time – so the FTC may still intervene to extract a defined time period when Elsevier commits to some undertakings (for example, keeping free uploads and free downloads for five years, or something along those lines). I would put the probability of a formal investigation at 30% or less.”

In short, it would appear that the fate of scholarly communication lies in the hands of researchers and their institutions themselves, not governments.

The question is this: do they have the determination, the chops, and the wherewithal to take this on? Here the jury is still out. Will they, like the University of Florida, follow the path of least resistance? Or will they, as the University of Maryland seems minded to do, take the harder but more promising path?