Metascience working notes

How does funding work in research now? Are vast improvements possible?

Is science funding inefficient? Are there ways to fix?

Introduction

Where is today’s Einstein? This feeling of science slowing has some merit.
- Economist Bloom et al. (2020) finds inputs are rising dramatically, while outputs remain stagnant.
  - Are Ideas Getting Harder to Find?
  - Science Is Getting Less Bang for Its Buck
- This suggests ideas are getting more expensive to find.
Yet scientists these days spend ___ of their time writing grants, even while every minute becomes more precious.
- Science is undoubtedly important, producing X% of our economic development. This essay looks for how we may better allocate resources for such a crucial project.

Screening issues. Are the transaction costs of administration too high?

Grant-writing
- Currently, scientists spend too much time administering or marketing instead of research. von Hippel and von Hippel 2015 finds average proposal takes 116 PI hours and the sheer quantity of grants written translates to more funding.
- Increased competition actually reduces efficiency because the cost of writing proposals outweighs gains of better selection (Gross and Bergstrom 2019)
- Counter: Ayoubi et al. (2019) shows the process of competing for grants produces the benefit, but the grant itself does little.
Peer review
- Argument 1: prestige provides a signal of reputation. Peer review bridges information gap b/w funder and researcher.
  - Can central authority even actually predict value of proposal?
  - Danielle Li 2017 finds yes, reviewers with direct domain experience do predict which grants produce higher impact publications
  - But she also finds expertise tied to bias, so reducing bias may sacrifice review quality.
- Argument 2: above certain threshold it’s all random (depends who’s on the panel that day)
  - Canonical Cole, Cole, and Simon (1981) finds it’s by chance.
  - Pier et al 2018 is modern confirmation of this; finds the performance of a grant depends mostly on the reviewer. There is practically no relationship between grant and rating.
- Argument 3: peer review kills bad science but also weird science (which is necessary)
  - Wang, Veugelers, and Stephan (2017)
  - Conformity; unwillingness to support an unconventional idea in front of peers.
  - Which brings us to second main issue.

Incentives and risk

Argument 1: NIH has model of public funding which requires specific deliverables on short-term 3-year contracts to prevent slack off or not producing value for taxpayers.
Argument 2: But science is heavy-tailed.
- Azoulay’s canonical HHMI vs NIH paper finds long horizons get far more breakthroughs. [detail specific results]
Counters:
- HHMI picks talented people who would win anyways.
  - On the other hand, even doing any rigorous statistics on this topic is by definition capturing the bulk of the curve you’re trying to escape.
  - So we have to interpret this with caution. Perhaps just as heuristics or intuition pumps.
- Scientists rationally follow their incentives and work on safe topics (principal agent problem)
- Nintil’s analysis
- Argument 3: Nnature itself is getting harder to discover. Changing funding mechanism won’t help save us.

Potential fixes

What have people tried, and are they sufficient? What other ideas?
- Modified lottery
- Fang and Casadevall 2016: Research Funding: the Case for a Modified Lottery - PubMed
- Tried before: Liu et al. 2020 at The Health Research Council of New Zealand
- Peer review only filters out the bottom bad stuff. Everything above enters a lottery.
  - This removes marginal costs (95->98% best)
  - Don’t need to revise grants so much to appease the reviewers.
  - Removes bias to prestige.
- But counterarguments:
  - Submit enough times and you get funding?
  - Removes the signal of grants, so unis stop using it as metric and disrupts labor market for scientists.
People-funding (HHMI)
Program manager model from Azoulay 2019 (ARPA)
- Give one manager the ability to fund ideas.
Fund based on variance.
Fast Grants
- Time is money (people saved from earlier treatment).
NIH Pioneer award has been tried

Notes on sources

A vision of Metascience - Michael Nielson and Kanjun (N-Q)

[TODO: writeup response]

The trouble in comparing different approaches to science funding - Michael Nielson

Outlier effects - Performance in science is plausibly dominated by outlier discoveries, not typical discoveries, making many common statistics misleading.

For Thiel’s Founders Fund, the top return is more than sum of all others. If you try to use heavy data-based approach, with large samples of all great companies, you’re studying the bulk of the curve you want to get away from.
- E.g. You might learn a great management technique but that prevents truly outlier performance. Anecdotal outlier success could help more. But that’s also probably overfitting.
Science isn’t VC’s entirely: you can’t neatly quantify the value of discovery
- Discoveries aren’t fungible like commodities. Can’t measure in units of milli-CRISPRs (say), and pile up three dozen 30 milli-CRISPR discoveries to get General Relativity.
- It’s like art. Can you RCT poetry? Greatness = importance x perfection?
- Doesn’t mean we give up; certainly better social processes out there. But must contend with outlier-vs-typical issue. Using outlier curves as intuition pumps for goals in science funding
Typical - shift median or 80th percentile to the right, even at cost of less outliers.
Variance - increase variance even if worsens typical outcome, may value polarizing grants or perform failure audit (require failure above some treshold?)
Outlier - maximize for the very few extremes These curves are not based on reasonable assumption, but serve as heuristics that can be extremely helpful
Like economics. We don’t have detailed model of the entire economy, but simple models reveal important ideas – ideas like comparative advantage, marginal cost of production, price elasticity, the relationship between supply and demand, and so on.

Non-stationarity

Science is like an ecosystem that adapts the more healthy it is to new successes.
In VC you try to look for hints from future; not very rigorous
- Venture capital has largely resisted the quantification that has revolutionized modern finance. In lieu of mathematical modeling, venture capitalists tend to subscribe to pieces of folk wisdom around their investing activities. How, when, and in what a VC invests all tend to have only the thinnest veneer of theoretical kor empirical justification. - Abe Othman, the head of data science at AngelList
Funding/incentive schemes can change the underlying distributions.
Who shows up is itself determined by incentive scheme (e.g. anonymized, or “golden ticket” for any reviewer to give green light)

Conclusion:

AGZM is more a prototype than a finished approach, but it’s worth doubling down, not giving up. Can we imagine and create a world in which science rapidly and routinely greatly improves its social processes?

Limits and Possibilities of Metascience - Nintil

Intro:

Nielson & Kanjun’s vision (above) is to explore an enormous design space for social processes, leading to a far more structurally diverse set of environments for doing science; and ultimately enabling crucial types of work difficult or impossible within existing environments.
But there’s another optic from which all we have done is interesting but ultimately unhelpful theorizing that will not lead to any meaningful change.
- “All that matters is to fund good people doing good work. Trying to think deeply about the way science works will lead to minor improvements, making all that effort a waste of time. Just find good scientists and give them money and time, everything else is a footnote to this central dogma of science funding.”

Some notes/takeaways (not comprehensive rn)

Metascience is hard to study because 1. costly to experiment 2. hard to compare
- Replication in more areas like physics is one that can help.
The design space for new institutions and reforms may not be as large as we think.
- Judicial systems (law) and restaurants and evolution converge because of inherent constraints. Many aspects of PhDs now are here for a reason.
Research and execution (building new institutions, entreprenuership) must be tightly coupled.

This is meta-meta-science.. Getting a bit far.

Nintil - Applied positive meta-science

Categorizing problems in “fix science”
Includes a flowchart and an appendix with list of proposals
Helpful to get a sense of the space

The rise and fall of peer review

Peer review only catches ~a quarter of errors; three studies:
- ScienceDirect
- JAMA (journal)
- Sage Journals
- and misses really important issues like “the paper claims to be a randomized controlled trial but it isn’t” and “when you look at the graphs, it’s pretty clear there’s no effect” and “the authors draw conclusions that are totally unsupported by the data.”
This gives the stamp of approval on totally fraudulent papers, causing real harm.
- The debunked theory that vaccine causes autism comes from a paper (in the Lancet, as peer-reviewd and prestigious as it gets) and it stayed there for 12 years until public criticism got it retracted. That’s a lot of unprotected kids.
Nobody really cares about peer review anyways. They just turn around and submit elsewhere.
So do we make it stricter? That might seem to catch more errors, but exacerbates existing issues.
- Scientists will be even more pressured to appeal to reviewers, writing their papers like boring 100-page legal contracts nobody wants to read.
- And already very expensive and labor-intensive. Longer wait times.
- More importantly, researchers won’t even think about non-popular ideas because they know it won’t pass. But that’s how real innovation happens.
  - Copernicus would have gotten rejected by geocentric peer-reviewers feeling proud that they’re preventing misinformation.
  - Would black authors have been able to publish anything back then? (and this kind of thing is still an issue)
Solution might be to make things more open.
- Progress in science is defined by the best of our ideas, not the worst ones.
- We don’t need to squash every wrong paper, just give them the right disclaimer (that nobody has really checked)
- If we st op evaluating papers based on big names or prestige of journals, researchers no longer push to publish wrong stuff.
- Benchmark papers on reality.
  - How well does it work in experiments? In practice?
  - That’s why industry produces so much innovation; they’re pushed to deliver, not publish.