jump to navigation

Internal reviewers November 4, 2006

Posted by dorigo in news, personal, physics, science.
trackback

The publication process of a physics analysis in CDF has always been a very complex and lengthy business.

You present your progress at physics meetings regularly, document your work in internal notes, and then have to go to a “full status report” where you discuss the whole thing in detail. After that is done, you can ask for a “preblessing”, where you do the same but have documented in advance in detail every important aspect of your work in another internal note.

At the preblessing, people can (and usually will) present you with a host of nasty questions and will ask for countless checks. Before you can go for a “blessing” talk you need to address the questions in a web page created for the purpose. Then you finally go for blessing of the results you want to make public, and yet other objections may be raised.

And that is not even the beginning: once you’ve blessed a result, if you want to write an article on a scientific paper, you will need internal referees (called “godparents”) that will scrutinize the analysis in yet more detail. After maybe six to ten months, you will then be allowed to release a paper draft to the collaboration, which will be then able to request all sorts of modifications, corrections, checks. Even the last grad student is allowed to bug you for as long as he sees fit (very democratic, I must say).

Once you answer all criticism you put together a second draft, the godparents agree to release it, and hell breaks loose once again on you. Then, finally, you give a paper seminar, and send the paper out for publication… Where a new review process starts anew.

All this is well known and I have discussed it here in the past. I think it is way too long and cautious a process, and it does not help the advancement of science, only defends the “reputation” of those in the collaboration who think they have one to save.

Anyway… This process has recently become a bit more complex – but apparently with the aim of expedite the blessing – by introducing yet another player in th game: the “internal reviewers”, who are not really godparents. They enter the fray before the status report, and suggest a course of action, review in detail everything, allegedly to make sure the blessing process goes smoothly, but maybe making things even harder for those who carry on the analysis…

We will see how it goes this time, with the Z->bb signal now almost ready to be blessed for the winter conferences: today,  a fresh new “internal review” committee has been appointed to bring us to bless our result, and they are ready to jump at us… I hope they will help us rather than hindering our progress!

Comments

1. Pietro Vischia - November 4, 2006

Hi Tommaso!

Could you elaborate on the publication process of a physical analysis at CMS? (I say CMS and not LHC, because I suppose the publication process being part of the experiment (CMS) rather than the entire structure (LHC): correct me if I am mistaken)

Cheers
Pietro

2. Andrea Giammanco - November 4, 2006

I didn’t know that it was mandatory, in CDF, to summarize results on a webpage before approval. Sometimes I stumbled upon these pages, while looking for documentation, but I thought that they were written voluntarily.
Since I always found them very useful in order to catch the main analysis features and results, I think this method could be suggested to the CMS management. What do you think about that?
As you certainly know, in CMS the authors just have to write the internal note, an announcement is done on the CMS internal webpage (for registered users only), and the whole collaboration has 20 days to send comments, suggestions and criticisms to the authors. In my experience, it’s really rare that somebody bothers to comment an internal note… quite surprising, since we are 2000! The reason, in my opinion, is not a general shyness, but simply that in most cases only the referees read the whole note (and sometimes not even they do, eheh), because, come on, why should you take the time to read more than the abstract of a 20-30 pages draft about something completely unrelated to your activity, when your time is already completely filled with your duties? So, even if you would like to, you usually don’t (if you have enough spare time to do that, you will probably prefer to use that time to do something more pleasant).
But a webpage, if reasonably well done, can provide to the reader enough information to see if there are major flaws, requiring less effort from his side. (This will not prevent the small minority of very interested people to go through the complete internal note, if they want to check the very technical details. And maybe after being attracted by a simple webpage, more people will be induced to do so.)

3. Alejandro Rivero - November 7, 2006

An outsider question: is there, between experimenters, an standard handbook for statistics? I wonder if a 5-sigma in a metodology could be, say, a 4.7-sigma with the same data.

4. dorigo - November 7, 2006

Hi Pietro,

I think Andrea more or less answered your question above…

Hi Andrea,

I think a webpage is a very user-friendly way of conveying information in a quick way, and provide a summary of all the documentation, usable plots (for conferences etc.), and ultimately broadcast the result. Yes, that should be done in CMS too IMO.

Anyway, do not be so sure that people will not bother commenting on internal notes in CMS, once these are based on real data. At that point, competition as well as the severe eye of the naysayer will pitch in with a quite different result than that you’ve noticed on MC notes…

Cheers,
T.

5. dorigo - November 7, 2006

Hello Alejandro,
there is no such thing as a wholly agreed-upon definition of 5-sigma effects, unfortunately, AFAIK. And indeed, what some quote as a 5-sigma effect, others call it a smaller figure (the opposite, for some reason, is less frequent😉 . One reason behind this is the difference between frequentist and bayesian approaches to the computation of the probability of the observed effect.
There is however a growing consensus on some techniques which get used more often than others when setting limits, for instance the CL_s method used by LEP II for the Higgs search. You can find a summary of that in the “statistics” pages of the PDG.

Cheers,
T.

6. Andrea Giammanco - November 7, 2006

Hmmm, probably you are correct. But my experience was the same also in ALEPH, with real data…
So, I correct your correction: people is very happy to comment (and try to demolish) your analysis if it is on a “hot” topic, not if you work on some secondary topic. (And of course MC studies are secondary by definition!)


Sorry comments are closed for this entry

%d bloggers like this: