Beyond journals and peer review: towards a more flexible ecosystem for scholarly communication

This article challenges the assumption that journals and peer review are essential for developing,evaluating and disseminating scientific and other academic knowledge. It suggests a more flexible ecosystem, and examines some of the possibilities this might facilitate. The market for academic outputs should be opened up by encouraging the separation of the dissemination service from the evaluation service. Publishing research in subject-specific journals encourages compartmentalising research into rigid categories. The dissemination of knowledge would be better served by an open access, web-based repository system encompassing all disciplines. There would then be a role for organisations to assess the items in this repository to help users find relevant, high-quality work. There could be a variety of such organisations which could enable reviews from peers to be supplemented with evaluation by non-peers from a variety of different perspectives: user reviews, statistical reviews, reviews from the perspective of different disciplines, and so on. This should reduce the inevitably conservative influence of relying on two or three peers, and make the evaluation system more critical, multi-dimensional and responsive to the requirements of different audience groups, changing circumstances, and new ideas. Non-peer review might make it easier to challenge dominant paradigms, and expanding the potential audience beyond a narrow group of peers might encourage the criterion of simplicity to be taken more seriously - which is essential if human knowledge is to continue to progress. Keywords: Academic journals, Growth of knowledge, Non-peer review, Paradigm change, Peer review,Scholarly communication, Science communication, Simplicity.


Introduction
Papers published in peer reviewed journals are a key part of the process by which academic knowledge is developed. Articles are checked through by "peers" to try to "ensure that the valid article is accepted, the messy article cleaned up, and the invalid article rejected" (Weller, 2001: p. xii). The goal of this process is to ensure that the papers published are of as high quality as possible, so that fellow researchers and other readers can have confidence in what they read.
According to Larsen and von Ins (2010) the number of peer reviewed journals "most likely is about 24,000". Authors of research articles wishing to publish in one of these journals, must first select their journal (usually not easy, given the choice), make sure their article conforms to the requirements of the chosen journal, send a copy to the editor who will decide if it is worth serious consideration, in which case he or she will send it out to two or three peer reviewers, and then make a decision based on the recommendations of these reviewers. This decision might be to publish the article as it is, to reject it, or to ask the authors to improve the article in line with the reviewers' recommendations and then reconsider it. The reviewers are "peers" in the sense that they are typically working in the same area, and may have submitted work to the journal in question. They are usually unpaid, anonymous volunteers who cannot be expected to respond immediately, so the whole process may take some time -in the worst cases, several years. The details of this process vary from journal to journal, and conventions in different disciplines differ, but the above outline is typical. However, this is a relatively recent development: "Peer review in its modern present form is only about 40 years old and is not standardized. ... a systematic peer review for Nature was only introduced in 1966 ... Proceedings of the National Academy of Sciences introduced peer review only a few years ago" (Larsen and von Ins, 2010).
There are differences between disciplines, but in general far more credence is placed on research published in peer reviewed journals than on other sources: researchers typically only take work published in peer reviewed journals seriously, students are routinely told that the only references that count are to peer reviewed journals, and when research results are given in news bulletins the name of the journal is often given as evidence that it is worth taking seriously. There is a sense in which research is the contents of peer reviewed journals, although this may obviously be recycled for a wider audience in books, TV programmes, educational courses and so on.
Although the peer review journal system is still the gold standard in most disciplines, it has attracted considerable criticism. The most commonly made points are: 1 Most journals are expensive for readers to access which, in practice, means that the readership is often restricted to those with university subscriptions to the journals. This has led to calls for open access journals, which are being increasingly heeded. The system does seem to be progressing in the direction of open access so that the products of research should be accessible to all. 2 The fact that there are typically only two or three volunteer reviewers chosen by the editor, may lead to the suspicion that "bad" papers may sometimes be published and "good" papers rejected. Empirical studies sometimes find little correspondence between the verdicts of different reviewers (see references below), so which papers get to be published may be little more than a lottery. This means that peer reviewed journals may not be an efficient quality monitoring system. 3 The demand for quality assessment of research has led to quality ranking of journals so that the journal a paper appears in can be used to assess the quality of the research reported in a paper. This has led to the feeling that the top ranking journals are unlikely to accept risky or innovative papers for fear that this may damage their status. This may have a detrimental effect on the growth of knowledge, and encourage researchers to write papers according a formula approved by the top journals. 4 It may take a long time to get a paper published, particularly if it fails to be accepted by the first choice journal and is submitted to a sequence of journals, each of which may take months, or even years, to make a decision (parallel submissions are not allowed). In a study of one author's submissions over a given time period, the average time between submission and appearance in print for the 39 articles which had been published was 677 days (nearly two years); he had a further 14 articles he was still trying to find an outlet for -on the date he did the analysis these articles had been waiting an average of 3.5 years since their initial submission and had been submitted to between 1 and 5 journals (Nosek and Bar-Anan, 2012). My experience 1 suggests that this is not untypical. This has obvious implications in terms of slowing down the growth of knowledge, as well as being frustrating for researchers and their audience.
These problems are serious. The Covid 19 crisis has provided an incentive to speed up the peer review process which has proved that this is often possible, but it has also highlighted the warning "not yet peer reviewed" about papers posted on repositories like medrxiv.org. This suggests, firstly, that papers on such repositories are not reviewed or checked by anyone, and secondly, that once a paper has been peer reviewed its conclusions are beyond doubt: both of these assumptions are far too simplistic.
In an earlier era, Einstein's papers on special and general relativity were accepted by the journals he submitted them to (although without peer review according to Adler, 2012), but many such revolutionary ideas are likely to have been rejected. If this had happened to Einstein's papers, physics may have progressed more slowly or along a different route. Or perhaps, someone else came up with ideas similar to Einstein's years before him? There is no way of knowing: counterfactual history is a tricky discipline. Gregor Mendel's now famous work on genetics was published in 1866 but was rarely cited and Darwin appears not to have known about it. Eventually the ideas were rediscovered in 1900.
The publication system had failed to adequately distribute a theory which subsequently turned out to be very useful. Many other useful ideas have almost certainly never seen the light of day. Although we obviously cannot point to these ideas, it is possible to model the possible impact of the inadequacies of the publication system: see Appendix 2 below.
Not surprisingly, there is an extensive literature on these problems and how they might be remedied: e.g. Schmidt and Görögh (2017), Kovanis et al (2017), Adler (2012), Nosek and Bar-Anan (2012), Priem and Hemminger (2012), Bornmann (2011), Weller (2001. Appendix 1 below contains sketches of a few stories which illustrate some of the problems of the current system. This literature provides a number of suggestions for making peer review quicker and more effectivee.g. reviews from the first choice journal being passed on to another journal if the paper in question is not accepted by the first choice journal, post publication reviews, open (published) reviews, and so on. However, most of these suggestions start from the premise that journals and peer review are essential.
The perfect system would deliver to every relevant person just those artefacts that are useful and of a sufficiently high quality, and do so immediately with no publication delay. Merely stating this makes it clear that this ideal is both impractical, and not clearly defined (how are usefulness and quality defined from the perspective of each person?). However, imperfections are easier to spot (see Appendix 1): it seems plausible that even a small improvement could make a large difference. Perhaps, for example, medical advances which have not yet happened might already be common practice with a more efficient system for distributing knowledge?
If there were a universally accepted checklist against which a new research article could be evaluated in an objective way, so that all reasonable people would reach the same conclusion, then the standard peer review system might be efficient and reliable. In practice, in most disciplines, this is far from the case: the process of reviewing a research paper requires subjective judgments and differences of opinion are almost inevitable. It is impossible to predict which research efforts the future will judge favourably: we can only guess. Given this, it seems, to me, almost inevitable that the present system will be arbitrary and, given the high stakes involved for the careers of academics, on occasions, possibly corrupt.
The current system depends on two key components: journalswhich are subject-specific, and peerreview. In the next two sections I discuss these two concepts and argue that they are both, to some extent, problematic. I then make two suggestions which could lead to a very different ecosystem with many advantages over the current one. The later sections of the article explore some of these advantages, and also at some of the barriers to change.
It is important to mention two preliminary issues. The first is the unit of research output. Traditionally this is the "paper" but there are strong arguments that this should be broadened. In the UK the term "research output" is used in documentation relating to research assessment. Some of the issues are outlined in Appendix 3 .
Second, academia encompasses may different disciplines and often different "tribes" within a discipline. This means that there is a great variety of types of research including, for example, randomized control trials in medicine, case studies in management, conceptual analysis in philosophy, and symbolic arguments in mathematics, to mention just a few. Words like "research", "scholarly", "scientific" and "rigorous" may seem natural in some contexts but not others, and may have different implications in different contexts. The use of the word "scholarly" in the title is intended to signal that my intention in this article is to provide a general analysis, not tied to a specific discipline.

Subject-specific journals
A few years ago journals were printed on paper, often by learned societies, and distributed to members of the society and libraries. Researchers in a particular discipline would browse through the appropriate journals to find material relevant to their research. They would probably have the most relevant ones in their office, and the university library could supply the rest. Gradually the number of journals increased, indexing services arose to help researchers cope with a much larger search domain, and online availability means that most articles are now instantly available. Google Scholar is now probably the most used indexing service, and the typical researcher's tactics have changed from browsing key journals to searching on Google Scholar or other databases. The journal a paper is published in matters little now to researchers or their audience, except in so far as a prestigious journal legitimises the research and persuades people to take it seriously.
However, subject specific journals persist. If I want to publish an academic article which wins the respect of the academic community I have to do so in a journal. This seemingly innocuous fact has many unfortunate consequences. I am only allowed to submit to one journal 2 at a time so if the first rejects it I may resubmit to a second, and so on -this can result in delays of years. In any commercial domain, this would be outlawed as a serious barrier to competition. My paper may be interdisciplinary and it may not be obvious which journal is appropriate. If my paper presents, say, statistical evidence on a medical matter, I may want my work checked by both the statisticians and the medics. There are a few journals which cater for a broad range of subjects -e.g. Sage Open spans "the full spectrum of the social and behavioral sciences and the humanities" -but these are the exception not the rule.
There is now little point in journals apart from their legitimising function which could easily be taken over by organizations with a more flexible remit, as explained in the next section. There is a detailed review of some of the problems of journals, attempts that have been made to remedy them, and some further suggestions in the literature cited in the Introduction above.

Peer review
Most of the published research on peer review is about its efficiency -does the current system provide valid and reliable assessments of research papers? The evidence here varies from discipline to discipline, but is often negative. Peer review may be unreliable in the sense that different reviewers may give different opinions, so taking just two or three reviewers makes the review process into a lottery -e.g. Kravitz et al (2010), Wood et al (2004), Peters and Ceci (1982). This is not surprising -two or three anonymous, unpaid and unaccountable reviewers, who are probably just experts in some of the areas relevant to a thorough critique of the paper, would not be expected, on a priori grounds, to produce definitive reviews. Even when reviewers agree there is, of course, no easy way of deciding whether they are right, or even of defining what "right" means.
This does not mean, of course, that disagreement between reviewers should never happen. In the absence of unambiguous criteria for evaluating research outputs, disagreements are inevitable. Indeed, the interaction between different viewpoints is likely to be essential for the healthy growth of knowledge, and a good dissemination system should facilitate such interactions.
There is another, more fundamental, problem with peer review: it is restricted to peers in the same discipline or sub-discipline. It is no longer considered acceptable that doctors, teachers and so on only review themselves: review by customers, and possibly other stakeholders, is now the norm. And, of course, producers of cars, haircuts, computers and meals have always been subject to market forces: the ultimate test is whether people are prepared to spend their money. The academics' argument in defence of their exception from the general rule of external review is that only peers are in a position to understand their work, and there is some justice in this. However a discipline which has no response mechanisms to the views of potential customers or audience, including researchers in other areas, may fail to generate the most relevant and interesting research. Small cliques of academics may review each other's work positively despite its lack of interest or relevance to anyone else. Sometimes this may work, and the argument that short term responsiveness to customer feedback may inhibit progress may be valid, but as a general principle it seems suspect. There is a strong case for at least a limited amount of non-peer review. This category might also include non-standard, perhaps paradigm changing, views, within a discipline -I discuss this in more detail below.

Two Suggestions
Systems like the journal publication system evolve as circumstances change. Many of the changes discussed here will happen inevitably, and in some cases are already happening, as the agents involved see and exploit various possibilities. Experimental journals and peer review platforms incorporating new ideas include Cureus (http://www.cureus.com/), InterJournal (http://www.interjournal.org/), PubPeer (https://pubpeer.com), Principia (http://www.principia.network) and the "alternative review tools and services" listed in Table 1 of Schmidt and Görögh (2017).
My intention here is to sketch, very briefly in the next two subsections, an alternative system based on two simple ideas. This could work in parallel with the existing system so that practices could evolve according the preferences of the various stakeholders. Some aspects of these suggestions are outlined and discussed in Nosek and Bar-Anan (2012), Priem and Hemminger (2012) and Schmidt and Görögh (2017) and doubtless other works too. Nosek and Bar-Anan (2012) use the term "utopia" in their title to indicate that their proposals are a goal to strive for, but that there are substantial barriers to change which they spell out in some detail. I will then go on to explore some of the possibilities these suggestions might facilitate.

All research outputs should be freely available on the web -which would effectively incorporate an open-access journal of everything
The obvious way of achieving this is to have a single global repository covering all disciplines (Wood, 2010). The importance of this covering all disciplines is neatly illustrated by the fact that two of the most interesting articles I found when researching this topic were published in journals specialising in areas which are largely irrelevant to the topic (Priem and Henninger, 2012, in a neuroscience journal; Nosek and Bar-Annan, 2012, in a psychology journal). At present there are repositories for papers which are freely available but these are mostly either restricted to particular ranges of subjects (e.g. arXiv.org: https://arxiv.org) or to a single institution. The closest to such a general repository is probably SSRN (www.ssrn.com) which started as the Social Science Research Network but has since broadened out and claims to be "the most interdisciplinary service of its kind, representing disciplines across the full research spectrum, including the applied sciences, health sciences, humanities, life sciences, physical sciences, and social sciences. SSRN is instrumental both as a starting point for students and faculty in the research process, and as a place to communicate their findings at an early stage, prior to publication in academic journals" (https://www.ssrn.com/index.cfm/en/ssrn-faq/ accessed on 3 December 2020).
There is an argument that subject specific repositories would be useful, because then people interested in, say, social enterprises, could go to the social enterprise repository. However, the difficulty is that there might be relevant material in general business repositories, or in repositories focusing on charities, and so on. Separate repositories for separate areas will lead to the same problem as that created by the 24,000 journals. The problem of finding relevant material can be solved by using web search facilities aided by labels or keywords.
There are further advantages to such a system. Documents on the web are far easier to obtain than oldstyle hard copies, and the page limit constraints of traditional paper journals should be a thing of the past -articles should no longer be turned down because there is no space in the journal and longer, more detailed articles would be feasible (Nosek and Bar-Anan, 2012). It is also far easier to publish updates and corrections where they are likely to be noticed, thus avoiding the problem that a printed journal article may continue to be accepted despite being discredited -see Appendix 1 for an example.

Evaluation of research outputs should be carried out by a range of organizations and individuals
The evaluation, or review or assessment, of research outputs could be carried out by organizations which would award some type of quality stamp to outputs. This review could go beyond peer review and might include reviews from various user perspectives, reviews of features like the readability of an article or the prerequisite knowledge necessary to understand it, as well as statistical reviews, research design reviews, and so on. Many aspects of this idea are discussed by Nosek and Bar-Anan (2012), and Priem and Henninger (2012).
This does happen currently, but only to a limited extent. Journals sometimes include comments on articles in earlier issues, there are various opportunities for post publication review either on an ad hoc basis, or via a post-publication review service (e.g. Mathematical Reviews) or an "overlay journal" for a repository such as the arXiv (http://www.ucl.ac.uk/ls/rioja/). Some web-based journals do encourage reviews from readers, and various styles of social media can be are being used in a variety of ways (see Priem and Hemminger, 2012).
How might this work in practice? The author would deposit his or her research output (an article or something else) in a repository on the web. He or she would then apply to one or more reviewing organizations to obtain a quality assessment. This might certify that the statistical analysis is correct, or that it reports rigorously conducted research according to some specified standard. They may insist on changes, in which case the changed document would also be posted on the web (and readers could see the changes made if they were interested). Then potential readers might search the web, or might search articles listed by specific reviewing organizations. Then, as well as having access to the research output, they would also have information about its quality on various dimensions from these reviewing organizations.
Some of these reviewing services might publish details of individual reviews; others might not. Conventionally peer reviewers are anonymous, but there are strong arguments in favour of publishing authors and details of reviews (Nosek and Bar-Anan, 2012: 236-8).
In a more open system, if readers find open reviews helpful, there is likely to be competitive pressure to provide them. Some reviewing services might stick with the current system of soliciting reviews from carefully selected experts, others might apply the principle of crowd sourcing and invite reviews from anyone.
These reviewing services might be provided by the learned societies which formerly published journals; their reputations would help to convince readers that the output is sound. If a problem is subsequently found, this might lead to the reviewing organisations amending their reviews. Like any information providing service in the public domain, these reviewing organisations would depend on their reputations with readers, and competition and market forces should make the system evolve in ways which are responsive to all parties. The reviews themselves may be subject to review, which may also help readers assess the value of reviewing organizations.

Some possibilities that may be facilitated by these two suggestions
There is, of course, no one body with the power to promote change. Change will come because various stakeholders see an opportunity and take it, but the current system has considerable inertia: I discuss the barriers to change below. In this section I look at some of the benefits which might be facilitated if the suggestions were adopted more widely.
I have divided the possible benefits into five categories: (1) improvements in the efficiency and effectiveness of the dissemination and evaluation system; (2) improvements in the quality and visibility of critiques of research; (3) encouraging reviews by people who are not peers, which may facilitate cross fertilisation with other disciplines and the possibility of seeding new paradigms of research; (4) providing a home for ideas that don't fit into existing journals ; and (5) the possibility of increasing the importance of simplicity as a criterion for evaluating research -which is, I think, critical for the growth of useful knowledge in years to come.

The advantages of a freer market: enhanced efficiency and effectiveness
The metaphor of a market 3 is helpful here. Transactions in typical economic markets are mediated by money, which is true to a limited extent in the market for academic knowledge: being acknowledged as an expert in a field may bring financial benefits, but it will also bring enhanced reputation and the feeling of having made an important contribution. Whatever the rewards, the basic principle of a freer market enabling more productive transactions should carry over. The effect of the suggestions above would be to make the market for scholarly research freer, and to create a market for reviews. Currently, the fact that authors of research articles can only submit to one journal at a time, and the fact that the journals themselves are the only real players in the review game, are restrictive, uncompetitive practices that would be outlawed in many markets in the formal economic system. Separating the review function from the distribution function is applying the principle that too much vertical integration hinders competition. Van de Sompel (2006) cites a 1997 article by Roosendaal and Geurts who take this principle further by distinguishing five separable functions of scholarly communication. Free markets are said to have numerous advantages in terms of the efficiency with which the needs and desires of customers (other academics, business and the public) are satisfied by the outputs from producers (researchers).
How this might play out in the market for academic knowledge is, of course, difficult to predict, but some possibilities are outlined below.
Currently readers may have to wait until a paper has been accepted and published in a journal. This can take between a month or so and several years. Under the new regime, the first draft would be available immediately, and the reviews are also likely to be available quicker than publication under the old regime for two reasons. First, reviews could be carried out in parallel, so the necessity to submit rejected papers to another journal would be avoided. Second, competition between reviewing organizations is likely to reduce delays.
More information would be available on the quality of research outputs from the reviews. At the moment the information available to readers is typically that the article has been passed by the peer review process of the journal. What this means depends on the rigour of the review process, about which readers can only guess. Some journals have a good reputation, but it is difficult for readers to know what this reputation is based on and whether it is deserved. In the new order, when reviewing organizations do nothing except review, it is unlikely that they would content themselves to a pass-fail rating, particularly when other reviewing organizations may be providing more information. As well as specifying the precise criteria on which a paper has been passed, they may also make further comments on strengths, weaknesses and possible extensions.
There might be a richer diversity of types of review. Because submission for review no longer excludes other options, it is likely that it would be easier for new entrants to the review market: this should lead to a richer diversity of review possibilities than currently. Different quality stamps for different criteria would be the likely result. A more nuanced category of evaluations would be feasible with the new system -perhaps including an "interesting but not proven" category, or a "good idea but needs work" category -so that papers which would have been rejected if the journal was publishing, and so legitimizing, the papers may be made available to readers. A more flexible and informative reviewing system might encourage competition between reviewing organizations on the quality of their judgments, and indeed on the definition of quality in a research area. Authors might still choose the safe, high ranking option, but they would also be able to submit to other possibilities. And, of course, a more varied set of review possibilities would encourage a more varied ecology of research outputs.
In an ideal world, authors would have benefit of services to edit their work, post it in an attractive and accessible form, and to include useful facilities for commenting, linking to data sets and so on. At the moment authors are at the mercy of their journals, but in the freer world envisaged here, there are likely to be organizations offering services like these.
The suggested more anarchic system should also reduce the possibility of mistakes -clearly erroneous papers being legitimized by publication in prestigious journals, and useful work failing to find an outlet (see Appendix 1).

A more critical approach to knowledge
The importance of being critical of ideas and theories is widely accepted. If a branch of knowledge is to progress, it is likely to be helpful to subject existing ideas to detailed critical scrutiny. For the philosopher Karl Popper (1972) testing scientific theories is crucial to the whole scientific enterprise 4 . There is a famous collection of articles from a meeting of philosophers of science (Lakatos and Musgrave, 1970) entitled "Criticism and the growth of knowledge": the title of the collection neatly summarises the spirit of many of the contributions.
Currently reviewing papers for journals is voluntary, unpaid and usually anonymous with comments and criticism not being published. In a freer market there might be opportunities for reviewing and evaluating research to have more status than it does currently, and the quality, extent and visibility of the evaluations might be enhanced. Arguably, too much effort now goes into producing research, and not enough into evaluating and improving it. The proposed market in reviewing might help to correct this imbalance.
There is also a strong argument that restricting the reviewing process to peers may not always be a good idea. This is the subject of the next section.

Non-peer review: other perspectives and paradigms
Like non-fiction and non-vegetarian food in India, non-peer review might turn out to be a broader and more important category of review than it may initially seem. The basic case for non-peer review is outlined above: peer review may degenerate into cliques of academics reviewing their own work by criteria that are of no relevance to anyone else. A bit of competition between different measures of academic quality might be useful.
There are different categories of non-peers. Review from the perspective of lay readers is one possibility. Other possibilities are review from the perspective of neighbouring or competing disciplines, review from the perspective of relevant techniques such as statistical techniques, and so on. There are many possibilities and it is not helpful to categorise them rigidly. Reviews from the perspective of another discipline should help to encourage the transfer of useful ideas across the boundaries of disciplines and sub-disciplines -something which the present system does not encourage. Some nonpeer reviews might legitimately be regarded as less rigorous than peer reviews, but others might apply more rigorous criteria.
One potential benefit of non-peer review is that it may make it easier to challenge the dominant paradigm in a discipline. The present peer review system is almost certain to have a conservative influence: it is not likely to be receptive to new ideas. In the terminology of Thomas Kuhn's The structure of scientific revolutions (1970), peer reviewers are likely to favour "normal science" and to be sceptical about ideas which depart from received wisdom. Stanford (2019) argues convincingly that "historical transformations [which include peer review] of the scientific enterprise have generated steadily mounting obstacles to evolutionary, transformative, or unorthodox scientific theorizing, but also that we have substantial independent evidence that the institutional apparatus of contemporary scientific inquiry fosters an exceedingly and increasingly theoretically conservative form of that inquiry." Luukkonen (2012) puts forward a similar argument in relation to research funding. Horrobin (1990) gives many examples in the biomedical sciences where peer review has resulted in the rejection of important articles and the "suppression of innovation". This conformist tendency is likely to be reinforced by journal ranking systems which may make journals fearful of risking their status by accepting risky work.
Normal science, or following the conventional route, is, of course, often a good idea. Science progresses by following conventions, procedures and assumptions which are seen to work, but a system which does not allow established ideas to be challenged runs the risk of getting stuck in a blind alley (see below for a discussion of this problem in relation to the covid 19 pandemic). Fostering new ideas is difficult, almost by definition, but the more anarchic system encompassing non-peer reviews proposed here seems more likely to achieve this than legitimising research just on the basis of reviews from two or three peers chosen by the editor of a journal. This might be a step towards Paul Feyerabend's "epistemological anarchy" (Preston, 2020), or the "disruptive science and technology" envisaged by Russell (2012).

Providing a home for ideas that don't fit existing journals
One of areas which interests me is the possibility of making knowledge simpler. (I will say a bit more about this in the next section because it is relevant to my present topic.) Having written an article on this theme, I then looked for a suitable journal to submit it to, but failed to find one. Simplifying knowledge did not seem to be a recognised research area. In the end I published it in an education journal (Wood, 2002b), and focussed on the educational aspect of my thesis. However, the argument is actually much broader so publishing it in an education journal seriously restricted its scope and its audience. Several years later I wrote a more general article and, as I had submitted several papers to arxiv.org which is "a free distribution service and an open-access archive for ... scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics," I tried submitting it to this repository.
However, their list of fields does not include simplifying knowledge so they duly rejected it. I subsequently submitted it to another repository, SSRN, which has a much broader remit and did accept it (Wood, 2016). I also submitted version 1 of the present paper to both repositories: arxiv.org accepted it but classified it as computer science, SSRN accepted it without giving it a subject classification. Classifying this paper as computer science seems unhelpful: the keywords below the abstract above give a better indication of its contents.
As time progresses fields of human knowledge will inevitably change: some fields will die out, new ones will emerge, and the character of many will evolve over time. Unlike the general repository suggested above, journals tied to specific fields must exert a brake on such changes.
Peer review in a new discipline is also problematic because there are, by definition, no peers. Again, the more anarchic system suggested here can only help, although creating a readership for, and acceptance of, new strands of thought is obviously a far bigger problem which cannot be solved by the two suggestions proposed here.

More emphasis on simplicity
Specialist disciplines evolve their own jargon and perspectives which workers in the field absorb as they learn their trade. The learning process may be cumulative in the sense that each step may require an understanding of previous steps. In extreme cases technical work at the leading edge of a discipline may be inaccessible to anyone outside the discipline. This creates two problems. First, reaching the leading edge may take a long time: it is possible to imagine that in years to come workers in some fields will need to spend so long reaching the leading edge that by the time they get there their cognitive powers are declining and they are thinking about retiring. This would mean that the discipline is stuck and cannot progress further. Second, inter-disciplinary collaborations and review of work by outsiders or non-peers of the kind I am arguing for here, is impossible. We will just have isolated islands of expertise, linked to nothing, and of no use to society in general.
We are obviously a long way from either scenario at the moment, but they do illustrate the potential problems. The obvious way of reducing the seriousness of these problems is to try to make ideastheories, models, jargon, etc -as simple as possible (but not, of course, simpler, as Einstein supposedly said), and as accessible to outsiders as possible. Editors of journals encourage authors to make the presentation of their ideas as simple as possible, but there is, in my experience, little to no emphasis on the simplicity of the ideas themselves. (This may, of course, not be true of all fields.) Sometimes, the opposite is the case: specialists may feel they need complex jargon and formalisms to enhance their status and keep outsiders out. In the long term, of course, this is self defeating; if simplicity is not taken seriously as a criterion for evaluating academic outputs we are likely to end up drowning in a sea of half understood technicalities. There is more on this theme in Wood (2002aWood ( , 2002bWood ( , 2016.
With luck, non-peer review, and less dependence on journals for specialist audiences, will encourage more emphasis on simplicity. Statistical analysis is one area which non-specialists often find difficult, with the result that misunderstandings are common: this is the subject of the next subsection.

The particular case of statistics
Statistics is an essential tool across a wide spectrum of fields. Unfortunately it is a complex discipline, both conceptually and mathematically, which means that many researchers, and their audiences, often have difficulty using and interpreting statistical concepts. The prime example is the concept of statistical significance, or p values. Conclusions from empirical research are often described as "statistically significant", which may be further qualified with a "p-value". There are two problems with this. First it is very widely misunderstood. The word "significant" seems to imply "big" or "important", but this is unrelated to its statistical meaning which is, in rough terms, "unlikely to be the result of chance". Furthermore, the word significant implies, unrealistically, that something is either proven or not proven. The second problem is that p-values typically do not answer the main question that people are likely to want answering: how likely is it that the conclusions from the research are valid in general terms? These problems have been acknowledged for decades (e.g. Cohen, 1994;Wasserstein, Schirm and Lazar, 2019) but to little effect: p values are still the gold standard in many disciplines.
Why do p-values persist? For many years statisticians, and workers in empirical scientific disciplines, have been steeped in the idea that significance tests and their associated p-values are the gold standard for statistical inference. Ideas this deeply entrenched are very difficult to dislodge. A more multidimensional system for evaluating research may help to loosen it, but the complexity of the framework within which statistical significance tests are embedded makes it difficult to see the escape route.
What is needed is a simpler framework which would make it easier to see how methods work and what the answers mean. I believe this is possible: my suggested alternative to p-values is outlined in Wood (2019). Simpler methods should make it easier to align techniques with what researchers actually want to know, as opposed to what the statisticians want to tell them.

Barriers to change
Many, but not all, of the suggestions proposed here are already happening to some degree. However, there are two big barriers to change: funding, and the inertia of the present system, particularly concerning reputational factors.

Funding
Old style journals are funded by selling the journals to readers and libraries, often at exorbitant prices. This regime is already starting to crumble with the advent of open access journals, so it is unlikely that the reviewing organizations would be able to charge their audience anything other than a nominal fee without something concrete like a journal to sell. The funding models for the new system could easily evolve from the various existing models for funding open access journals. One obvious possibility is that authors, or their institutions, would pay the reviewing organizations. However, it is likely that other possibilities would evolve. Nosek and Bar-Anan (2012) review some of the possibilities. It is worth bearing mind that the web does seem to foster large scale developments whose funding mechanism is not planned in advance: the absence of a clear business model is unlikely to discourage innovation in this area.

Inertia, especially to do with reputation and research assessment
The main barrier to change in the market for academic knowledge is probably that researchers feel that their reputations, and salaries, depend on publishing in the top journals. This gives the top journals a monopoly on prestigious research. In time, papers published in repositories on the web (like arXiv.org or http://www.ssrn.com/) and reviewed elsewhere may achieve the necessary status to make the journals unnecessary, but at the moment in most disciplines the stranglehold of the top journals is still strong.
There is a tendency for crude and rigid grading systems to be imposed in markets where customers may have difficulties in choosing between competing products. So schools and hospitals are graded with some being labelled failures, consumer reports by organizations such as Which? (http://www.which.co.uk/) give scores to products and label some best buys. Similarly the pressure to grade academic departments in order to decide which are deserving of funding has led to pressures to grade research papers for which journal grading (four star, three star etc) is the common proxy used. This is unfortunate from many points of view -including the healthy growth of knowledge. Inevitably the grades will ignore some important criteria. In the sciences there is even evidence that retraction rates, due to fraud or errors in the research, are higher in the higher ranked journals (Brembs & Munafo, 2013). Academic research deserves a more flexible and nuanced system for quality evaluation.
Despite this, the increasing pressure for open access to research papers may provide an escape route.
Research published in the so-called top journals is increasingly likely to be made freely available -either in the journals themselves, or by depositing copies in institutional or subject-based repositories. And, of course, anyone is free to publish reviews of the papers in these journals. At the moment this is not done to any great extent because of the assumption that publication in a good journal is the only type of review necessary. However, in the future, particularly if links to freely available copies of the papers in question are available, such reviews could easily be published to provide more information than the mere fact of publication in a prestigious journal.

Conclusions
I have discussed some suggestions to improve the academic knowledge market. At the moment, if I write an article on, say, the effectiveness of a new diet, I need to decide which peer reviewed journal to submit it to, make sure it conforms to the requirements of the chosen journal, send it off, wait for and react to the comments of the editor and reviewers, and then if it is rejected send it to another journal, and so on. This may take a long time, and the paper may get reviewed only from the perspective of the one journal -and it may be far from clear what sort of evaluation has been carried out. Under the alternative system proposed here, I would post the article, or other research output, on a public website. Then I might get it reviewed from the statistical point of view by one evaluation service, from the medical point of view by another, and from the social science point of view by a third. Another organization might produce a one paragraph summary in layman's terms of the status and credibility of my paper, assuming of course that it met their minimum standards -they might insist on, and draw on, reviews from the previous three bodies. Some quality stamps might depend on others: for example a medical reviewing service might insist on evaluation by a statistical reviewing service.
This system would have many advantages in terms of distributing articles and other artefacts quickly and efficiently, and responding to comments and requirements from a range of stakeholders -not just the "peers" in each discipline who control the present peer reviewed journals. It would also enhance the visibility and status of critiques of research. Instead of just knowing that a paper has been "accepted" by a journal, we would also know what comments and criticisms have been made and by whom.
In practice, at present, many research outputs are currently freely available on the web. However, many are not so available, and even if an open access copy is posted, the primary publication outlet is the journal. The suggestion in this paper is that authors should consider an open access repository as being their primary outlet. Such repositories do exist, but most are subject specific, and many like arXiv.org insist on putting articles into specific categories which is unhelpful for articles which straddle disciplines. Using labels, rather than hierarchical categories, would make such repositories more flexible.
The second suggestion, the development of reviewing organizations, has happened to only a very limited extent. This is a pity.
There are many potential benefits of such a system. Papers should be published quicker. Readers would be able to see how the paper has been evaluated on a range of criteria, including those of non-peers from outside the narrow confines of the academic tradition to which a paper belongs. The system should be more responsive to errors. Ideas that challenge dominant paradigms, or that don't fit into any existing subject specialism, should find it easier to gain a foothold. And the criterion of simplicity, which is arguably essential to progress, should have greater prominence. Peters and Ceci (1982) resubmitted 12 articles to the psychology journals which had published them 18 to 32 months previously, after changing the names of the authors and institutions and a few other minor details. Only three (25%) of the articles were recognised, and eight of the remaining nine were rejected by the same journals that had originally published them.
I first got into research on peer review when I asked a colleague who was organising a conference if I could have a look at the results of the peer reviews of the submitted papers (Wood et al, 2004). Each of the 58 papers was reviewed by two reviewers. If we divide the reviews into two categories -good and bad -50% (29) of the papers received one good and one bad review. This percentage represents the proportion of papers where the reviewers did not agree; ideally the conference organizer would have preferred this percentage to be much lower. Overall 69 of the 116 reviews were good, and the rest bad. If these good or bad evaluations had been awarded to the papers at random the disagreement rate would have been 48%. In other words, the disagreement rate between individual reviewers of the same paper was slightly greater than it would have been if the process were entirely random. The reviewing process was, almost literally, a lottery. According to the conference organiser, one of the participants in the conference was decidedly unhappy because his paper had been rejected following two bad reviews. If the conference organizer had done this analysis at the time he would surely have had less confidence in rejecting this paper.
The obvious response to this problem is to make the process more open (so that readers can see the basis and source of reviewers' judgments), and to publish, not reject, papers if there is any doubtwhich is facilitated by the lack space constraints on web-based systems.
The consequences of erroneous decisions Sometimes articles are published in prestigious journals and thus legitimised when they probably shouldn't have been. These range from accidental errors, to systematic bias and fraud. One high profile example concerns an article published in the medical journal The Lancet in 1998 that "lent support to the subsequently discredited theory that colitis and autism spectrum disorders could be caused by the combined measles, mumps and rubella (MMR) vaccine. ... Investigations by Sunday Times journalist Brian Deer revealed that Wakefield [the author] had multiple undeclared conflicts of interest, had manipulated evidence, and had broken other ethical codes. The Lancet paper was partially retracted in 2004 and fully retracted in 2010 .... In 2011, Deer provided further information on Wakefield's improper research practices to the British medical journal, BMJ, which in a signed editorial described the original paper as fraudulent. The scientific consensus is that no evidence links the vaccine to the development of autism, and that the vaccine's benefits greatly outweigh its risks" (http://en.wikipedia.org/wiki/MMR_vaccine_controversy accessed on 3 September 2013).
The problem of articles like this is exacerbated by the fact that corrections may take a long time to be published (more than 10 years in this case), and that readers may not notice them when they are published because the original article, and the paper version of the journal in which was published, cannot be changed. With less serious errors, corrections may not be published and the original error, legitimized by publication in a prestigious journal, may persist and even be cited as an example of how things should be done. Nosek and Bar-Anan (2012: 237) give an example of two published articles in a high profile journal which, they say, had critical errors in their analysis. One of these articles was cited by an editor of another journal as an example of how the analysis should be done. They comment that "it is very difficult to get such an error out of the present system." There is obviously no foolproof way of avoiding error in research, but if reviews are available on the web, and articles are updated as problems are found, readers would be much more likely to be aware of criticisms. If these reviews were crowd sourced, as well as relying on two or three reviewers selected by an editor, there is a much greater likelihood that errors like this would be found.
The opposite problem, of course, occurs when an article is rejected when it perhaps should have been published. Horrobin (1990) outlines several such examples from the biomedical sciences.
Problems arising in the covid 19 pandemic Much of the research on the virus was published quickly on repositories such as medRxiv (e.g. Lavezzo, 2020) but this came with health warnings like "not certified by peer review". This is problematic because peer review cannot be relied on to ensure an article is correct. The peer review process itself has been made much quicker for Covid related research, although I read Lavezzo (2020) two weeks after it was posted and there was no note about any review despite the urgency of understanding the coronavirus.
More fundamentally, Greenhalgh (2020) asks "Will COVID-19 be evidence-based medicine's nemesis?" in an editorial in PLOS Medicine. She contrast the "evidence-based medicine paradigm" with the "complex systems paradigm" and points out the rigidities of the former, with its emphasis on evaluating interventions by randomised controlled trials. The difficulty is that evidence which meets the high standards of the evidence based medicine paradigm may be difficult or impossible to obtain. For example she cited a report on the use of masks which was "criticised by epidemiologists for being 'nonsystematic' and for recommending policy action in the absence of a quantitative estimate of effect size from robust randomized controlled trials." This ignores the obvious fact that even if the evidence on the use of masks is below the highest standards, it may still be a good idea to recommend their use because of the strong possibility that they may help prevent infections.
According to an article in Nature (Lewis, 2020) entitled "Mounting evidence suggests coronavirus is airborne -but health advice has not caught up", theories about the transmission of the coronavirus have clung to the idea that aerosols (small droplets in the air) are not an important route of transmission despite evidence to the contrary, which means that the health advice given by the supposed experts is deficient and may have cost many lives.
In both cases, a more flexible and multi-dimensional approach to evaluating research might help to reduce the influence of unhelpful dominant paradigms.
noticed (p = 0.1), then the mean time to the next innovation which is noticed is 50 years, 45 years longer than it would have been if the M&Ss noticed every groundbreaking innovation.
This model is obviously very crude but it does illustrate the key importance of the two quantities involved. How can we improve the values of two key figures in the model? The probability of a piece groundbreaking research being noticed (p) can be decomposed into the probability of publication (pp) multiplied by the probability of it being noticed once it's been published (pnp). The first probability (pp) should be as close to one as possible: the problem here is that of ensuring that the review system works as efficiently as possible and does not exclude useful innovations. The traditional way of improving the second probability (pnp) is to reduce the size of the pool of published papers; the difficulty here is that this might lead to the exclusion of useful innovations. With a larger pool of papers for researchers to sort through, the question is how can we improve the probability of groundbreaking innovations being noticed? Two ideas occur to me. First, the abstract is critical. Many researchers may only read the abstract so this needs to be very carefully formulated. Second, there may be a case for reading a random sample of the literature: this would mean that different researchers would read different samples which should increase the diversity of the ideas being carried forward (in contrast to the present scenario where everyone tends to cite the same works).
The other variable is the frequency with which groundbreaking research is produced. I can envisage no appropriate, simple model to help understand this, but one reasonable assumption is that innovations are fostered by exposure to a variety of types of research which in turn is more likely with the open, anarchic system of dissemination I am proposing here. The censorship which is likely with the traditional system is likely to have the effect of inhibiting the development of a "long tail" (Anderson, 2006) of infrequently read, but potentially important, research articles.
Conventional papers have a publication date after which the paper is not changed. Updates and corrections and comments may appear in later issues of the journal, but readers may not see them. Web based media, however, can incorporate updates, different versions, corrections and so on. This is obviously particularly welcome when important flaws are found in the research -readers of the paper can then be alerted to this (see Appendix 1 for an example).
It is also worth pointing out the importance of named authors in academia. If you buy a car, or have a meal in a restaurant, you usually have no idea about the identities of the creative team behind the product. If you read some research, however, the author is an important and prominent piece of information. Academic pride and careers depend on authorship. This may lead to disputes, and to a failure to share data and ideas. If, in another parallel universe, authors of scholarly papers were anonymous figures in the background, would this make a difference to the progress of knowledge? It is impossible to know, but the success of Wikipedia suggests that something similar may be worth considering in academia (Rice, 2012). Obviously there are important issues concerning intellectual property, copyright and patents, and the motivation of researchers here.