...mathematical model scores feature which is an objective measure of novelty for scientific work. https://pharmaceuticalmanufacturer.media/pharma-manufacturing-news/latest-pharmaceutical-manufacturing-news/desci-labs-launches-novelty-scores-for-scientific-manuscript/

The article says:

...evaluating the novelty of scientific manuscripts and grant applications takes centre stage in the scientific peer review process. The primary reason work is rejected by editors of high-impact journals or funding agencies is because referees think it is not novel enough. However, the current peer review process is subjective, slow, labour-intensive, and prone to bias and inaccuracy [...] The release of these novelty scores [...] means there is now an objective, automated measurement of one of the core parts of the peer review process.

As a general principle, I assume goodwill. With that in mind, it is with genuine, all due respect that I find this development to be deeply alarming.

First of all, how can there possibly be an “objective” measure of novelty?????

Secondly, while it's great to see on DeSci Labs's about page some laudable goals like enabling FAIRness, open science, developing open source software, and preserving scientific outputs (which I care deeply about), the same page also speaks of securing USD 6.5 million in “seed funding”, accelerating science, using “Web3” technology, and to “accelerate growth and enhance customer loyalty”. To me, this reeks of techno-solutionism and -accelerationism.

Third, the underlying math is published in Nature:

https://doi.org/10.1038/s41467-023-36741-4

To me, all three of the above speak volumes about the state of scientific research culture, and not in a good way... 😩

Contrast this with the excellent essay on “The Limits of Data” by C. Thi Nguyen recently shared with the Turing Way community by Shern Tee:

https://doi.org/10.58875/LUXD6515

Which reminds us:

...policymakers and data users should remember that not everything is as tractable to the methodologies of data. It is tempting to act as if data-based methods simply offer direct, objective, and unhindered access to the world—that if we follow the methods of data, we will banish all bias, subjectivity, and unclarity from the world. The power of data is vast scalability; the price is context. We need to wean ourselves off the pure-data diet, to balance the power of data-based methodologies with the context-sensitivity and flexibility of qualitative methods and local experts with deep but nonportable understanding. Data is powerful but incomplete; don’t let it entirely drown out other modes of understanding.

I hope the work on reforming academic research culture and #metaresearch could include diverse and skeptical voices in addition to simply developing new quantitative “metrics”.

Unless otherwise stated, all original content in this post is shared under the Creative Commons Attribution-ShareAlike 4.0 International license

Context

We've been doing many interviews over the past 1.5 years with different stakeholders in academia, and one of the most common barriers to changing behavior (such as doing more open research or changing research culture) is that “no one else is doing it and it doesn't benefit me”, but actually if everyone does it, then everyone benefits. Is solving such collective action problems something that has been studied in the academic context? If so, where and by whom?

I posed this question to the Turing Way and NASA TOPS Slack groups. Here's my attempt at collecting the responses so far.

Turing Way

So far, I haven't heard from someone who knows of research specifically about collective action problems in academia. But, a few theoretical frameworks were suggested as ways to examine the problem.

Agent-based modelling of individual vs collective behaviour

(from Shern Tee)

• You may find the Stanford Encyclopaedia of Philosophy entry useful: “Agent-Based Modeling in the Philosophy of Science” https://plato.stanford.edu/entries/agent-modeling-philscience/#TheoDiveInceStruScie_1
  • Unfortunately it doesn't directly answer the question of collective action problems. But (because I am a straitjacketed physicist) I find myself thinking about these situations as agent-based: a model simulation that shows agents doing things that are individually rational, but as a whole cause problems for science, is a demonstration of one possible model of collective action failure.
• This paper is a more concise overview of the above link: https://compass.onlinelibrary.wiley.com/doi/10.1111/phc3.12855
• An agent-based model of peer review, studying how scientists might want to trade-off work publishing papers with work reviewing papers: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6096663/
• This PhD thesis describes agent-based modelling of the academic publishing system — actors being journals and scientists: https://nova.newcastle.edu.au/vital/access/services/Download/uon:31339/ATTACHMENT01

This is interesting to me in the sense that I first of agent-based modelling in my intro ecology course during undergrad, but haven't considered it in the context of collective human behaviour.

Organisational theory

(from Liam McGee)

• Off the top of my head, and this might be a bit too general, but I quite like https://uk.sagepub.com/en-gb/eur/a-very-short-fairly-interesting-and-reasonably-cheap-book-about-studying-organizations/book276268 as a rapid intro to the competing models in organisational theory. Lots of references to explore. There's a book from the same series on “Studying Leadership”, which may also be of interest in approaching this problem.

Economic theories

(from Liam McGee)

• Other route might be down alternate economic theories (ones that seek to resolve the tragedy of the commons, like doughnut economics, or Ole Bjerg's stuff). Or Kahneman and Tversky from a Experimental Psych/CogSci perspective.

Religions

(from Liam McGee)

• Another interesting direction is to understand how the pro-social behaviour of various world religions works — a practical use case on that here: https://www.linkedin.com/pulse/church-cheese-frog-hugh-mason/ — comments on that article worth a dig too.

The Collective Action in Science Committee

(Julien Colomb) You may ask the people behind. https://freeourknowledge.org/committee/

Note:

I see that it proposes the model of “We will all do X (the ‘action’) when Y people have pledged (the ‘threshold’)”. This reminds me of the National Popular Vote Interstate Compact in the United States: https://en.wikipedia.org/wiki/National_Popular_Vote_Interstate_Compact

Thinking about common pool resources

(from Jonah Duckles)

A bit of a different tack on collective action, but still VERY related is Elanor Ostrom's work on Common Pool Resources, detailed in her book “Governing the Commons”. If you think about the work of an academic as working to advocate for and gather common pool resources (grant money) for themselves, I think it is an informative model for imagining a way that grant money could be considered less “contested” and more of a common pool of resources. The open science movement does kind of implicitly treat information as a common pool resource. Ostrom's work, I think, helps think about ways to build structures and systems around governing it for the benefit of many. A summary of “Governing the Commons” is her 8-point “Design principles illustrated by long-enduring Common Pool Resource (CPR) institutions” which is under the Research header on the Wikipedia page about her.

Thinking about “doers” and “thinkers”

(from Anne Lee Steele)

I think there are a few ways to approach this question: as sometimes the people doing the collection action & organising may not necessarily being the ones studying it, and vice versa. (Similarly for example: the work of community management is different from the act of studying communities!)

Regarding broader theories and ideas of social change at the individual level, the trans-theotical model (coming from medicine) is a very popular one: https://sphweb.bumc.bu.edu/otlt/MPH-Modules/SB/BehavioralChangeTheories/BehavioralChangeTheories6.html

There's also the studies of 'innovation diffusion' that talks about how systems change more broadly, studied by quite a few folks: https://en.wikipedia.org/wiki/Diffusion_of_innovations#Process

Regarding the tension between 'doers' and 'thinkers' (which of course is not necessarily cut and dry), it might be helpful to think through a few examples:

Organisers of collective action (for example – there are so many!): * https://movementecology.org.uk/ * https://scienceforthepeople.org/ * https://techworkerscoalition.org/

Studies of collective action:

• The Logic of Collective Action: Public Goods and the Theory of Groups by Mancur Olson is one I've heard cited quite a bit
• Elinor Ostrom (as @Jonah Duckles mentioned!) also wrote about collective action theory: https://academic.oup.com/edited-volume/28345/chapter-abstract/215160451?redirectedFrom=fulltext
• Institutional ethnography has been used to understand the roles, rituals and practices of all sorts of different environments, including academic spaces: https://blogs.lse.ac.uk/highereducation/2023/11/17/are-we-proper-institutional-ethnographers/
• More broadly, I've also seen how some studies of neoliberalism in academic institutions affect collectivising practices – stumbled upon this interesting piece: https://discovery.dundee.ac.uk/en/publications/revealing-the-manifestations-of-neoliberalism-in-academia-academi

Hope this helps!

Note:

Interestingly, the innovation diffusion model by Rogers is cited in the Center for Open Science's theory for behaviour change: https://www.annualreviews.org/content/journals/10.1146/annurev-psych-020821-114157#f3

NASA TOPS

Similar to the Turing Way responses, nothing specific to academia here. But there's a very interesting one about learning from climate action suggested by Jamaica Jones:

This is such an interesting question! I am not sure if it's exactly what you are looking for, but you might find Sheila Jasanoff's work to be informative. She contributed a chapter to a book called Human Choice and Climate Change that may be relevant. I also found another climate change-focused citation that seems similarly aligned: the article is called “Doing What Others Do: Norms, Science, and Collective Action on Global Warming”, by Bolsen et al.

Here's the Bolsen et al. paper: https://web.archive.org/web/20240522100857/http://eprints.lse.ac.uk/64670/1/Leeper_Doing what others do_2016.pdf

For the book Human Choice and Climate Change, it's available to borrow online from the Internet Archive:

https://archive.org/details/humanchoiceclima0001unse

It reminds me of my past life studying environmental sciences and learning about the concept of collective action problems and the tragedy of the commons. I wonder if anyone's done research on how to take lessons solving collective action problems in one domain (e.g. climate action) and applying them to another (e.g. academia)?

#metaresearch #ideas

Unless otherwise stated, all original content in this post is shared under the Creative Commons Attribution-ShareAlike 4.0 International license