A little bibliography on criteria for assessing the quality of scientific documents

Quality sign

As mentioned before I work on a literature review focusing on criteria for assessing the quality of scientific documents. As I found in the Web of Science, Scopus and Google Scholar very few literature on cognitive construction processes or relevant content-related parameters that make articles seem to be of high quality I asked in a recent posting for support by the readers of this blog.

One reader, Dr. Werner Dees, actually sent me very, very valuable tips. The complete list of the reviewed literature can be found below as text and downloaded as a BibTEx-file. Please give the file the extension .bib after the download, it ends now on .txt.

The articles I found most useful are Mårtensson, P., Fors, U., Wallin, S.-B., Zander, U., & Nilsson, G. H. (2016). and especially Bucholz (1995), the latter being a recommendation of Dr. Dees.

A lack of literature on qualitative aspects of scientific articles?

StarWorking on a literature review I am looking for publications on the perceived quality of scientific literature. I use the Web of Science, Scopus and Google Scholar as databases and find very, very few literature on cognitive construction processes or relevant content-related parameters that make some articles appear to be of high-quality and others to be inferior.

It is clear to me that quality is a multi-dimensional construct, the dimensions of which have to be operationalised themselves. Since the study should not be very time-consuming, it would suffice for me to find at least texts describing these dimensions. The operationalisation of these dimensions or the definition of indicators to describe them would not interest me in a first step.

I am/was also aware that the use of quantitative factors (e. g. citation-based parameters) are – in the opinion of many people mistakenly – used to operationalise quality in science. Nevertheless, I am surprised that apart from this quantitative information, almost only formal (e. g. design of the text through structuring) or syntactic (e. g. structure of sentences, formulation questions) are discussed as criteria of qualitative evaluation – whereas extremely rarely really substantive criteria relating to the content of publications.

After finishing my collection of literature I will publish my list of relevant articles here. If you would like to give me some literature tips, you do so by commenting, mailing or by other means. These tips will also be included in the list to be shared here – with reference to the contributor.

Foxconning Science: J. Sallaz on outsourcing in scientific publishing

factory hall asia photo
Photo by memn Foxconning Science: J. Sallaz on outsourcing in scientific publishing

While browsing I came across a very interesting article by Jeffrey J. Sallaz with the title „Your Paper Has Just Been Outsourced“.

The sociologist at the University of Arizona deals with outsourcing in publishing, as the title says. Sallaz draws parallels between the cheap production of overpriced mobile phones in Asian low wage countries and a similar practice of commercial science publishers: Outsourcing copy editing and layouting to low wage countries and selling the cheaply produced articles at top prices. It should be noted that copy editing and layouting are the only contributions of commercial publishers to the publication as a product.

Nonetheless, mobile phone manufacturers and publishers can sell cheaply produced works at overpriced prices, as customers tend to base the price they want to pay on the perceived prestige of the product rather than the real cost of its creation. Sallaz gives this finding a beautiful and catchy expression: Foxconning Science.

Altmetric.com tracks discussions about books listed in Amazon

Altmetric.com Logo
Altmetric.com Logo, downloaded from https://www.altmetric.com/about-us/logos/

The Altmetrics service Altmetric.com announced yesterday that it tracks now discussions about books listed on Amazon in social media and other not primarily scientific publications. According to altmetric.com’s posting this new data source produces „huge volumes of attention data – in just a few days Altmetric has found over 145,000 new mentions of 3,000 Amazon-listed books from Twitter, and over 20,000 mentions from other sources such as news, blogs and policy documents. Around 2 million mentions a year that relate directly to an Amazon record for a book are expected.“

Since the impact measurement for book publications is considered to be complicated and as citations are considered to be of little use for this purpose, Altmetric. com integrated an important impact source for books. It should be noted, however, that smaller publishers in particular are struggling with the use of Amazon as a sales platform due to its high service fees. Should Altmetric. com gain in importance, this could lead authors to opt out of publishing with smaller publishers if these don’t use Amazon as a sales platform.




Thesis on „Drivers and Barriers for Open Access Publishing“ published

Open Access photo
Photo by h_pampel Thesis on "Drivers and Barriers for Open Access Publishing" published

On August 12th a Phd Thesis entitled Drivers and Barriers for Open Access Publishing: From SOAP 2010 to WoS 2016 was published on Zenodo. The author Sergio Ruiz-Perez describes the publication in the abtract as follows:

„This PhD thesis follows up on previous studies aiming at finding out what a representative sample of researchers from all over the world and from all disciplines think about OA. We replicated the largest study of this type to date: the Study of Open Access Publishing run in 2010 (SOAP 2010).

We present a descriptive longitudinal study of active researcher’s opinions on open access publishing. We re-analysed a dataset from SOAP 2010 and we contacted authors publishing in scientific journals indexed in international databases (WoS 2016). We analysed the scientific community’s opinions on open access, in particular its evolution in the past 7 years. To do so, we used two different samples:

  • The SOAP project study (Dallmeier-Tiessen et al., 2011)
  • An ad-hoc sample obtained from the Web of Science database (WoS 2016) consisting of 15,235 unique responses

This PhD thesis was successfully defended on 24 July 2017 at the Facultad de Comunicacion y Documentacion from the University of Granada (Spain).

Please note that although the first few pages of this document are in Spanish all the rest is written in English.“

These are the biblographic details of Sergio Ruiz-Perez‚ thesis:

Sergio Ruiz-Perez. (2017, August 12). Drivers and Barriers for Open Access Publishing: From SOAP 2010 to WOS 2016. Zenodo. https://doi.org/10.5281/zenodo.842016

The author also published the supplementary data to his thesis as:

Sergio Ruiz-Perez. (2017). Drivers and Barriers for Open Access Publishing – WoS 2016 Dataset [Data set]. Zenodo. https://doi.org/10.5281/zenodo.842013

First copy costs of scientific articles – a short overview

While I was working on a longer text, I gathered – more or less as a by-product – information from studies and reports on the (first copy) costs of a scientific article. As a result, it can only be noted that the information available on these costs (and related costs as margins) are very different and really difficult to compare. The latter confirms my impression that there is a huge lack of transparency in the scientific publications market. The following table may give some insight in the information I found. It is also available via GitHub (https://github.com/scinoptica/article_costs.git), so please feel free to improve/ update the data or add new information.




British Academy








 Houghton et al.
 Van Noorden
First copy costs of an article, including profit margins 

1.127 £


PNAS: 3.700 $

Nature: 30.000 – 40.000 $

First copy costs of an article, without profit margins1.136 £ 

420 – 650 $


10 $

First copy costs, including profit margins per page 

360 $



Closed Access on average 20-30%,


Open Access (commercial) on average 15%

Cost of peer review (not included in first copy costs):1.194 £900 £ –
Methodsanalysis of literature and reportsexpert discussionempirical studyempirical studycase studyanalysis of literature and reportsanalysis of literature and reports, interviews
DisciplinesmixedSocial Sciences & HumanitiesmixedSocial Sciences & HumanitiesMachine Learningmixedmixed



British Academy for the Humanities and Social Sciences. (2007). Peer Review : the challenges for the humanities and social sciences. A British Academy Report. Retrieved from http://www.britac.ac.uk/policy/peer-review/contents.cfm

Dubini, P. (2012). PEER Economics : the effect of large scale deposit on scholarly research publishing. Retrieved from http://www.peerproject.eu/fileadmin/media/presentations/PEER_economics_29May12_Brussels-1.pdf

Houghton, J. W., Rasmussen, B., & Sheehan, P. (2010). Economic and Social Returns on Investment in Open Archiving Publicly Funded Research Outputs. Melbourne. Retrieved from http://www.arl.org/sparc/publications/papers/vuFRPAA/index.shtml

Research Information Network. (2008). Activities, costs and funding flows in the scholarly communications system (p. 88). Retrieved from http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/activities-costs-and-funding-flows-scholarly-commu
Shieber, S. (2012). An efficient journal. The Occasional Pamphlet. Retrieved June 05, 2012, from http://blogs.law.harvard.edu/pamphlet/2012/03/06/an-efficient-journal/

Van Noorden, R. (2013). Open access: The true cost of science publishing. Nature, 495(7442), 426–429. doi:10.1038/495426a

Waltham, M. (2010). Humanities and social science journals: a pilot study of eight US associations. Learned Publishing, 23(2), 136–143. doi:10.1087/20100209