Scientific Bang for the Buck January 5, 2008Posted by dorigo in computers, mathematics, news, physics, politics, science.
A concept worth a preprint, specifically Bruce Knuteson’s “A Quantitative Measure of Experimental Scientific Merit“, physics.data-an/0712.3572v1. And certainly a preprint worth a look, if only for making up one’s mind on the scientific merit of working at MIT. It came out on Christmas day on the ArXiv.
Jokes aside, I found the paper quite entertaining, and at times indeed surprising. While I find Bruce’s approach to the problem of assessing the scientific merit of a proposed experiment or analysis rather dangerous, and the explicit formulation of priors for the probability of discovering new physics in this or that experiment vaguely reactionary, I admit the paper brings home a point, which is however its premise rather than its thesis: review committees, as well as search committees, move in the dark. I am still in doubt on whether the exercise of endlessly debating over priors is a valid substitute to good-old preconceptions and biases.
Bruce is quite up-front from the very beginning in stating what is the main purpose of his study:
“In the context of determining which research program to pursue, review committees often must decide the relative scientific merits of proposed experiments. Within large experiments, deciding which analyses to emphasize requires similar decisions”.
Which gets me to raise the first objection – or rather a comment: It is remarkably radical to talk about “which analyses to emphasize”. I find that the concept, in fact, is a bit a too business-like way of doing physics in a large experiment. At the Tevatron we certainly need to emphasize the top mass measurement, the B mixing, and the Higgs searches these days, but we do not need a computation of entropy decrease to know it; emphasizing other analyses (which means, please note, de-emphasizing others) because of some pre-arranged prior (the estimated probability that a gluino is there, for instance) smells of a covert way of depriving scientists working in the collaboration of their wonderful inventiveness, of their freedom to be guided by their nose, by their intuition.
It is not a chance, it seems, that Knuteson is one of the authors of a complex automated machinery for new physics searches, a device producing hundreds of histograms of kinematical variables describing any combination of physics objects (high-Pt electrons and muons, jets, missing Et, photons, etcetera) in search for discrepancies with the standard model: is number-crunching winning its battle with scientific minds as much as it has won the chess challenge with our best grandmasters ?
The paper starts with a definition of the surprise content of the result of an experiment. It does so by using information theory, arriving at the wanted measure of the merit of an experimental result as the entropy decrease in the state of knowledge relative to the particular physics question investigated. Here is the synopsis of the discussion up to Section II, in Knuteson’s words:
“The essential thesis of this article is summarized in two sentences.
- The appropriate quantification of scientific merit of a proposed experiment or analysis (before it is performed and its outcome is known) is the reduction in information entropy the experiment or analysis is expected to provide […].
- The appropriate quantification of scientific merit of an experiment or analysis after the result is known is the information gained from the result […].”
Fair enough: if one knew what is the chance of the Tevatron discovering new physics in Run II, or the LHC finding something beyond the Higgs, one could certainly be able to tell how well the money was spent in building those experiments. Using the reduction in information entropy is a principled way to quantify the appropriateness of the investments.
But here, in fact, comes the nice part: the paper goes on to delve with the question by specifically working out priors. In Section III, Knuteson uses priors derived in the Appendix to estimate the “scientific bang for the buck” (SBFB) of existing experiments, and even that of past experiments discovering the Psi, the W and Z bosons, and so on. One learns that the probability of the Tevatron Run II finding new physics is 20%, and that the probability that the LHC will see something new is 90%.
Using those numbers and the cost of the experiments, the SBFB of the LHC is computed at a mere 0.001, while the Tevatron stands a giant at 5.0! Also worth noting is the specific search for single top production at the Tevatron, which – due to the low surprise factor – has a SBFB of 0.00001. Ironically, in the same table Knuteson includes the SBFB of the experiment of flipping a coin: the SBFB of the experiment is zero, not that different from the global search for new physics at the LHC!, although, to be fair, zero and 0.001 are indeed quite different when you take the logarithm.
As far as completed experiments go, one learns instead that the tau discovery stands at a SBFB of 5.0, soundly beating runner-up J/psi discovery at 0.2, with the top quark discovery at an amateurish 0.0004. The table is long, and you can search for your favorite HEP result, and judge for yourself on whether the Nobel Prize to Rubbia was’t indeed a bit hasty.
In earnest, the summary of Bruce’s paper is very direct in clarifying the rather limited scope of the proposed quantification method:
“Use of information content or information gain to evaluate the scientific merit of experiments requires the estimation of the probabilities of qualitatively different outcomes, and the reader may object that the problem of quantifying an experiment’s scientific merit has simply been reformulated in terms of the estimation of the probabilities of possible experimental outcomes. At worst, this reformulation significantly changes and focuses the discussion. The fact that there is not a well-developed literature to point to for the justification of these a priori probabilities emphasizes the fact that until now the importance of these probabilities has not been properly recognized […]”
However, he argues that
“The reader may object to the very idea of constructing an explicit figure of merit […] Such a reader misses the point that this is done (implicitly, if not explicitly) every time a decision of resource allocation is made. It is surely in the field’s best interest for such evaluations to be made in the sharpest, most open, most quantifiable, and scientifically best motivated framework possible”.
Which, to my biased ears, sounds like, “come on, we all know that the allocation of funding to science is made by fools, so let’s give ‘em some only partially random numbers to base their decisions upon and we will contain the damage”.
I do not mean to criticize the paper too much. It is a quite principled and tidy study of the problem. I think one cannot do much better in terms of finding a suitable figure of merit than what Knuteson did. I disagree with the very concept, though. But maybe I am too old-fashioned and I miss the point: scientific funds are not allocated wisely. On that, I think, we all agree.
Update: being away on vacation obviously does not help one staying in touch with what happens elsewhere on the web. I only now got aware of two other posts on this same topic: one at Superweak and one at Collider Blog. Backreaction also discusses it shortly.
Update 2: a detailed discussion of the statistical aspects of Knuteson’s paper is also available at Deep Thoughts and Silly Things.