Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Journal of Open Source Software: When collabora...

Arfon Smith
October 16, 2019

Journal of Open Source Software: When collaborative open source meets peer review

This is a presentation I gave at FORCE2019 (https://www.force11.org/meetings/force2019) about the Journal of Open Source Software.

In this presentation I try and explain how JOSS works, and how we're leveraging open source conventions to build a low cost, open access journal.

Arfon Smith

October 16, 2019
Tweet

More Decks by Arfon Smith

Other Decks in Research

Transcript

  1. Journal of Open Source Software Arfon M. Smith (@arfon), Tania

    Allard, Lorena A Barba, Katy Barnhart, Juanjo Bazán, Monica Bobra, Jed Brown, Jason Clark, Abigail Cabunoc Mayes, George Githinji, Olivia Guest, Roman Valls Guimera, Melissa Gymrek, Alex Hanna, Lindsey Heagy, Kathryn Huff, Mark A. Jensen, Daniel S. Katz, Vincent Knight, Thomas J. Leeper, Christopher R. Madan, Melissa Weber Mendonça, Kevin M. Moerman, Kyle Niemeyer, Lorena Pantano, Viviane Pons, Jack Poulson, Pjotr Prins, Karthik Ram, Marie E. Rognes, Ariel Rokem, Charlotte Soneson, Matthew Sottile, Yuan Tang, Tracy Teal, George K. Thiruvathukal, Kristen Thyng, Leonardo Uieda, Jake Vanderplas, Bruce E. Wilson, Yo Yehudi https://joss.theoj.org When collaborative open source meets peer review
  2. Software turns a theoretical model into quantitative predictions; software controls

    an experiment; and software extracts from raw data evidence supporting or rejecting a theory. Gaël Varoquaux – scikit-learn creator
  3. the skills required to be a successful scientific researcher are

    increasingly indistinguishable from the skills required to be successful in industry. Jake VanderPlas (Google)
  4. We must find a way to legitimize software as a

    form of scholarship. Phil Bourne, University of Virginia
  5. Challenges Technical Cultural Software isn’t easily citable ✔ ❌ Software

    citations aren’t allowed ❌ ✔ Software citations aren’t indexed ✔ ✔ Software isn’t peer reviewed ❌ ✔ Software can’t cite other software ✔ ❌
  6. 1. Find some way to fit software into current (paper/book-centric)

    system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions?
  7. Develop mechanisms for native software citation – see FORCE11 software

    (and data) citation working groups. Make your software ‘citable’, and make it easy for people to cite you. Lobby/lean-on/encourage indexers to count software citations. Find some way to fit software into the current system
  8. To show we’ve done our research. To credit the work

    of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?
  9. To show we’ve done our research. To credit the work

    of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?
  10. Develop mechanisms for citing all of the constituent parts of

    a research endeavour e.g. data, software, methods/ protocols, results… One potential solution: Transitive Credit (Katz, 2014, https://doi.org/10.5334/ jors.be) Evolving beyond one- dimensional credit model
  11. Smith et al WMAP PLANK Jones et al Smith et

    al astropy scikit-learn Transitive Credit Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2
  12. Smith et al WMAP astropy PLANK Jones et al Smith

    et al scikit-learn Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2 Whyte et al numpy scipy 0.5 0.25 0.25
  13. 1. Find some way to fit software into current (paper/book-centric)

    system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions? Really hard
  14. Individual change: Authors, reviewers, editors. Group/community change: Editorial teams, journals,

    publishers. Ecosystem change: Journals, publishers, indexers. Requires action across the whole scholarly ecosystem
  15. Software papers Gives us something easy to cite No changes

    required to existing infrastructure Publishing in existing journals raises profile of software within a community
  16. Software papers Writing another paper can be a ton of

    work Many journals don’t accept software papers For long-lived software packages, static authorship presents major issues Many papers about the same software may lead to citation dilution
  17. AAS Journals welcome articles which describe the design and function

    of software of relevance to research in astronomy and astrophysics. Such articles should contain a description of the software, its novel features and its intended use. Such articles need not include research results produced using the software, although including examples of applications can be helpful. There is no minimum length requirement for software articles.
  18. What if we made it as easy as possible to

    write and publish a software paper? Embracing the hack
  19. A developer friendly journal* for research software packages. Paper preparation

    (and submission) for well-documented software should take no more than an hour. The primary purpose of a JOSS paper is to enable citation credit to be given to authors of research software. * Other venues exist for publishing papers about software
  20. Short – generally under two pages. A summary describing the

    high-level functionality and purpose of the software for a diverse, non-specialist audience. A clear Statement of Need that illustrates the research purpose of the software. Acknowledgement of any financial support. Reference to related work, e.g. research results derived using the software. Not permitted to contain novel results. Paper format
  21. Be as conventional as possible when it comes to integrating

    with the scholarly ecosystem GOAL ORCID for logins. Deposit metadata and issue DOIs with Crossref. Archive papers (and reviews) with Portico. Archive reviewed software with Dryad/ figshare/Zenodo. Get indexed (Google Scholar, Scopus etc.)
  22. Follow best practices where they exist GOAL Open access –

    papers licensed CC-BY (authors retain copyright). Enforce the Open Source Definition. Clear ethics, ownerships, business model and governance statements. Rigorous, transparent peer review process. Register with community directories (e.g. DOAJ).
  23. Provide a valuable service to authors GOAL Minimize author effort

    beyond existing software documentation they already have. Offer a constructive peer review process by experts. Focus on improving submissions, not rejecting. Transparent, achievable review criteria. Actually review the software… Zero APCs, open access.
  24. Leverage the best parts of open source where possible GOAL

    Carry out reviews in a familiar environment (on GitHub). Foster a collaborative, welcoming community. Transparent editorial decision making process. Leverage open source tools where possible (e.g. Pandoc) Automate all the things! Use robots to automate repetitive or labour-intensive tasks where possible.
  25. Editor agreeing Managing editor asking editors if they can take

    submission Author suggesting reviewers Managing editor asking Whedon to assign editor
  26. Editor associating Zenodo software archive with paper Editor updating software

    version associated with review Editor tagging EiC team to accept
  27. Whedon produces final proofs of paper and Crossref metadata Managing

    editor asks Whedon to do a ‘dry run’ of accepting paper
  28. Whedon tweets link to paper Managing editor asks Whedon to

    accept paper …and confirms when DOI registered with Crossref
  29. 0 200 400 600 800 2016-07-01 2017-01-01 2017-07-01 2018-01-01 2018-07-01

    2019-01-01 2019-07-01 Year 1: 104 (8.6 papers/month) Year 2: 181 (15.3 papers/month) Year 3: 281 (23.4 papers/month) Year 4: 148 (30.3 papers/month)
  30. 1000th paper in ~June 2020 500 papers/year in 2021 0

    500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01
  31. Fixed costs we have to pay Annual Crossref membership: $275/year

    JOSS paper DOIs: $1/accepted paper JOSS website hosting: $19/month JOSS domain name registration: $10/year Portico fees: $250/year Total running costs: $1,063 $3.54/paper Assuming 300 papers published per year
  32. Including costs that we don’t currently pay Costs we now

    pay: $1,063/year Services for which we don’t currently pay: estimated at $20,600/year Volunteer editor services: 1.85 FTE effort or $370,000 (@$200,000/FTE) Reviewer time (not paid for in almost all journals, nor costed by JOSS) $1305/paper Assuming 300 papers published per year, excluding reviewer efforts
  33. But actually… Given that many journals don’t seem to pay

    editors much (if anything). If we include a modest editor stipend of $10,000 Total theoretical running costs: $31,663 $105/paper Assuming 300 papers published per year, excluding reviewer efforts
  34. Open Collaborations: a highly collaborative development process and are receptive

    to contributions of code, documentation, discussion, etc. from anyone who shows competent interest. Karl Fogel – Producing Open Source Software
  35. Collaborative open source Create more users through education. Encourage users

    to contribute through Liberal Contribution Policies. Enable more leaders through participatory governance.
  36. continuous Post-publication commenting Informal discussion of published research, independent of

    any formal peer review that may have already occurred Can be performed on third-party platforms, anyone can contribute, public Comments can be rude or of low quality, comments across multiple platforms lack inter-operability, low visibility, low uptake PubMed Commons, PeerJ, PLOS, BMJ Collaborative A combination of referees, editors and external readers participate in the assessment of scientific manuscripts through interactive comments, often to reach a consensus decision, and a single set of revisions Iterative, transparent, editors sign reports, can be integrated with formal process, deters low quality submissions Can be additionally time-consuming, discussion quality variable, peer pressure and influence can tilt the balance eLife, Frontiers series, Copernicus journals, BMJ Open Science Portable Authors can take referee reports to multiple consecutive venues, often administered by a third-party service Reduces redundancy or duplication, saves time Low uptake by authors, low acceptance by journals, high cost BioMed Central journals, NPRC, Rubriq, Peerage of Science, MECA Recommendation services Post-publication evaluation and recommendation of significant articles, often through a peer- nominated consortium Crowd-sourced literature discovery, time saving, “prestige” factor when inside a consortium Paid services (subscription only), time consuming on recommender side, exclusive F1000 Prime, CiteULike Decoupled post-publication (annotation services) Comments or highlights added directly to highlighted sections of the work. Added notes can be private or public Rapid, crowd-sourced and collaborative, cross-publisher, low threshold for entry Non-interoperable, multiple venues, effort duplication, relatively unused, genuine critiques reserved PubPeer, Hypothesis, PaperHive, PeerLibrary or deceptive publishing cast a shadow of doubt on the validity of versus careerism versus output measurement), and an academic Advantages and disadvantages of the different approaches to peer review.
  37. is published Optional open peer review As single blind peer

    review, except that the referees are given the option to make their review and their name public Increased transparency Gives an unclear pictures of the review process if not all reviews are made public PeerJ, Nature Communications Pre-publication open peer review Referees are identified to authors pre-publication, and if the article is published, the full peer review history together with the names of the associated referees is made public Transparency, increased integrity of reviews Fear: referees may decline to review, or be unwilling to come across too critically or positively The medical BMC-series journals, The BMJ Post-publication open peer review The referee reports and the names of the referees are always made public regardless of the outcome of their review Fast publication, transparent process Fear: referees may decline to review, or be unwilling to come across too critically or positively F1000Research, ScienceOpen, PubPub, Publons Peer review by endorsement (PRE) Pre-arranged and invited, with referees providing a “stamp of approval” on publications Transparent, cost- effective, rapid, accountable Low uptake, prone to selection bias, not viewed as credible RIO Journal comprising seven different traits of OPR: participation, identity, reports, interaction, platforms, pre-review manuscripts, and final- version commenting (Ross-Hellauer, 2017). The various parts of examined in more detail below. Each of these feed into the wider core issues in peer review of incentivizing engagement, providing appropriate recognition and certification, and quality Pros and cons of different approaches to anonymity in peer review.
  38. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  39. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  40. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  41. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  42. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  43. Scaling JOSS Most of our challenges are about scaling people

    processes: • AEiC/managing editor rotations. • More editors. • Term limits for editors (to avoid burnout). Technology improvements: • Smarter reviewer assignments. • Better monitoring tools for editors. • Tools to help authors prepare their submissions. 0 500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01
  44. Unexpected consequences of working openly Semi-regular emails from people annoyed

    they haven’t been asked to review yet. Generally need relatively small number of invites to identify reviewers (~2 invites per reviewer). Vanity software package ‘pile-on’. For high- profile open source projects, often have many reviewers volunteering.
  45. Some awesome things about working openly Zero privileged information in

    the system: Reviewer reports, editorial decisions available to all. Increase transparency: • Public declarations of potential conflicts. • Editorial decisions documented in the open. • Clear expectations of authors. Reduces complexity of infrastructure. People can link to their reviews.
  46. Zero privileged information in the system: So sometimes authors chase

    reviewers, editors etc. Good reviewers become well known quickly potentially leading to reviewer burnout. Potential cultural barriers to entry for some and negative dynamics for junior staff. Some not-so-awesome things about working openly