On the significance of mass bumps January 29, 2007Posted by dorigo in computers, internet, mathematics, personal, physics, politics, science.
The excess of di-tau events found by CDF while searching for higgs boson decays in their dataset spurred some discussion around the blogosphere on what is the real significance of the observed bump (see for instance the comments column in http://www.math.columbia.edu/~woit/wordpress/?p=509 Peter Woit’s blog, or the one in http://motls.blogspot.com/2007/01/mssm-higgs-at-160-gev-21-sigma-signal.html Motl’s blog).
Since a precise evaluation of the real significance is arguably still to come, we are in the happy situation of letting our opinions, gut feelings, and experience have their say on the matter. And my personal opinion on the issue is that the significance of the bump is quite small, meaning that the probability that the background created an upward bump-like fluctuation with respect to its predicted shape, of the magnitude observed, is quite high.
The one-line explanation of my belief is that if you do not know in advance where a signal could show up in a wide spectrum, and you later observe a possible bump in a particular point of the spectrum, the significance of that bump is much smaller than a mere event count could estimate (such a measure is usually S = (E-B)/sqrt(B), where B is the number of background events below the bump and E is the excess over background, and S is the significance), because small probabilities add themselves linearly, and local statistical fluctuations are thus more and more probable as the number of places where they can occur increases.
A back-of-the-envelope calculation is easy: if the signal spans a fraction f (say, 0.1) of the considered distribution, and if the excess has a probability p (say, 0.01) of occurring where it has been observed, the real probability to observe one such fluctuation anywhere is p/f – in our example, 0.1, or 10%. Of course, this easy example makes all sorts of assumptions: that one can define f precisely, that the background is flat, etcetera.
A precise way to compute a probability for a fluctuation to occur and mimic a signal can only be done with a so-called toy Monte Carlo technique – a powerful tool only possible thanks to our friend, CPU. Note that this is no joke: when I started my career as a particle physicist, toy Monte Carlo techniques were seldom used, because a meaningful answer usually required too much computer power. Now these things can run in background on your PC while you play DOOM…
So let me tell you what is a toy Monte Carlo technique, and how would we use it to estimate the real significance of a bump. But in order to make the story more interesting, let me introduce a real case of a bump search, one I did happen to analyze a few years ago. It is the story of the scalar quark bump in CDF.
The picture above shows the mass of muon pairs collected by CDF in Run I through a suitable di-muon trigger, in the range from 6 to 9 GeV. There are 52,000 events in the plot. A fit is overlaid on the data points (opposite-charged muon pairs, shown by black points with error bars), and it suggests the presence of a narrow resonance sitting at about 7.2 GeV of mass (ignore the other blip at 7.7 GeV). Is it a real signal ?
Mind you, the question is not academic. The standard model predicts that there are no vector meson states that could decay to two muons in that mass range, so finding one would be a small revolution. What’s more, there are models of new physics which could accommodate a scalar bottom quark, of mass approximately equal to 3.6 GeV: such a particle would easily bind to its antiparticle to create a 7.2 GeV vector meson, and the observed rate of its decays to muon pairs would be roughly in the right ballpark, considering its p-wave production and the predicted width [C.Nappi, PRD25, 84(1981)].
But wait, there is even more. A quantity called the R ratio, which is basically a number which tells you how many elementary quarks have a mass lower than a given value, always showed a disturbance (the R mini-crisis) in the region where a scalar quark of 3.6 GeV could give its contribution.
Here is the R ratio, compiled from many different measurements by several experiments. If x=sqrt(s) is the center-of-mass energy of an electron-positron collision, R(x) is defined as the ratio between the rate of hadronic events -ones when pions, kaons and other quark-composed particles are produced- and events with just two muons in the final state. R tells us how many possible quark-antiquark pairs can be produced in a e+e- collision, and it rises every time the threshold for production of a new quark-antiquark pair is crossed: thus, you can see that above 3 GeV R is increased by the allowed charm production (the charm quark has a mass of 1.5 GeV). A 3.6 GeV scalar quark would contribute at 7.2 GeV and above. If you observe closely, you will notice that there indeed is some disturbance at values x equal to 7-8 GeV… The disturbance has been explained without the intervention of scalar quarks, but the hint is suggestive.
Anyway, back to the CDF bump. Not convinced the hypothesis of a scalar quark bump is interesting ? Well, there is still more: in CDF, the very same people who found this suspicious bump, had previously obtained some intriguing hints at the production of a scalar bottom quark of roughly that same mass, in a totally different dataset!
Enter Tommaso. At that time, I was in the committee which was reviewing, and carefully double-checking, the weird results obtained by this group – headed by Paolo Giromini, a very bright research director from Frascati. We already had our hands full with the analyses the Frascati group had produced earlier – which were by themselves controversial as no other analysis had been in CDF before – and when the dimuon bump appeared on the scene, the situation went utterly out of control.
Individual collaborators who had taken the pains to re-do the analysis by themselves were finding inconsistent results; others were fitting the spectrum with alternate methodologies and found that the signal had a much narrower width – 19 MeV – than the one fit by the Frascati group, implying a fluctuation of the background (that is because our experimental resolution on the mass of an object decaying to two muons in that energy range was about 40 MeV, and a bump significantly narrower could not be a real particle signal). Still others were digging out fixed-targed data from past experiments, to see whether a bump could be ruled out by a close examination of the spectra.
In the meantime, every Friday afternoon Paolo Giromini showed impatience, and threatened to publish the data without waiting for an approval by the Collaboration (he used to say he had his finger on the “submit” key of the ArXiV web interface), and many in CDF took him quite seriously. After all, he had been patient for four years already…
It was some sort of a Maelstrom.
[to be continued]