jump to navigation

A first for me – just rejected a paper June 30, 2008

Posted by dorigo in personal, physics, science.
Tags: ,
trackback

I am not new to the job of refereeing scientific papers, but most of my experience comes from internal reviews. That mandate is called godparenting in CDF; it consists in carefully checking an analysis even before a paper draft exists, and then babysitting the process ending in the paper being submitted to the publisher. It is a long journey at times, and in more than one occasion it required me to even do real work to double check some results. I have been a godparent of a total of eight papers, and I was always on the pushing side (as per the true mandate of CDF, which expects godparents to help speed up the process once they are convinced on the soundness of results): I never tried to hinder publication of the work of my peers. And I claim that the fact that godparents are not anonymous did not have any weight in my decisions…

Instead, the refereeing of papers written by other collaborations is something I have started doing only in recent times. Here, one is doing things anonymously, and in principle there is room for showing one’s own true pickiness or malignity. I am not malign with other people’s efforts, but boy, I sure can be picky. And this time I was – but I think I took the right decision.

I just sent back to the publisher my comments on a paper recently submitted for publication by a large collaboration, which has published more than a hundred papers before this one. I cannot disclose the name of the experiment nor the topic of the paper, unfortunately. I can still say what was wrong with it, which in the end forced me to decide against publication if no further work will be done by the authors.

I found the paper rather sloppy in several aspects. Lack of precision, vagueness of some statements, numbers lacking uncertainty estimates, carelessness in describing experimental techniques. But these are minor sins; what really upset me, however, was to see in one of the main figures which accompanied the paper a systematic disagreement between data and Monte Carlo which was not commented in the text nor sized up in any way. Not a small effect: something big, which certainly propagates to the main results of the work. It is as if you were candidly showing you did not understand the composition of your data, but still wanted to extract results from it.

I can well imagine the reason for the inaccuracy. These days many people involved in experiments that have stopped running are ramping up their involvement in the LHC experiments, so they cannot spend their time on keeping the level of publications of their former experiment as high as they would like. They still want to “get the paper out”: unfinished business is as bad in high-energy physics as it is in any other human activity, from landing a plane to transplanting a liver. However, nothing is an excuse for lack of accuracy in a scientific paper, in my opinion. If something is below some threshold, why publish it ?

Sure, the main author could be waiting for his Ph.D. and he or she needs the publication as we need the air we breathe – but that does not move me: as scientists we have to produce analyses of the highest quality we can, and demanding a high level is the only way we can oppose a trivialization of our discipline. Besides, I think it will require just a couple more weeks to the authors of the paper I reviewed to figure out the source of the systematic effect I pointed out, and produce a much improved draft.

Still, the question is nagging me. Wasn’t I the one in favor of the most open diffusion of scientific information ? Open diffusion certainly means less control on accuracy. So why did I reject the paper ?

I believe I did so because I know that the reputation of the collaboration which signed it is high, and I have thus just enforced some “quality control” on its output, having been given a chance to do it. I have no doubt the paper will come out, hopefully improved, in a near future.

Comments

1. goffredo - June 30, 2008

Hi Tommaso. You rejected the paper because scientific information is good for placing bench-marks, hence worthy for publication, if it has been reviewed by peers and is found to meet certain quality standards. Instead the open diffusion of scientific information, without peer review, is, at its best, interesting, didactic for outsiders and even stimulating for experts, but not a substitute for standard reviewing and publishing. You know it, most of us know it and serious philosophers and sociologists of science know it. Then there bla-bla sociologists, philosophers and psuedo-intellectual-know-it-alls that bullshit about what science was, is or should be in a changing world.

2. Marianne - June 30, 2008

Many investigators hire scientific writers to do the writing. This is because they don’t know how to write, or analyze themselves. They have no writing ability at all. They have always delegated this task, so that they cannot do it on their own. Then, if their scientific writer is missing, who normally makes all the needed corrections, and catches all the errors, they decide to do without, and submit it, not knowing how really bad they are.

It is possible, in this case, that there was no one there to fulfill this role at the time, and it slipped through.

marianne
scientific writer

3. Tony Smith - June 30, 2008

Tommaso,
“… a systematic disagreement between data and Monte Carlo which was not commented in the text nor sized up in any way …”
is indeed something serious that should be corrected before publication,
and
rejecting until that is done is actually doing the authors a favor, because a publication with that flaw reflects adversely on the authors, and, if they are members of a collaboration, on the entire collaboration.

Although you said that you did it “anonymously”, the fact that you had the courage to put it on your blog means that you are really not “anonymous” to the authors (actually, given the way rumors etc get around, you probably were not really anonymous anyway).

Back decades ago when I did journal refereeing, I never acted anonymously. If I rejected a paper, I always sent an email to the authors saying why I rejected it and describing in some detail how they could get it into acceptable shape.

I am happy to see you doing similarly, and I hope the authors appreciate that you did them a favor.

Tony Smith

4. dorigo - June 30, 2008

Gosh Tony,

I really hope I remain anonymous🙂 – in any case, my review was very thorough and I do not think anybody will take it personally. First of all, I did not wait for the deadline (had ten more days to review the manuscript but sent in my comments today), and second, I described in detail the checks that I thought were necessary. Of course, it might just be that those checks were already made. In that case, they just need to rewrite the text, which is now a bit too vague.

Cheers,
T.

5. changcho - June 30, 2008

Good job – you do what you need to do to keep sloppy science from being published. I’ve only peer-reviewed one paper so far (I’m quite new at this), but I recommended publication with minor edits.

6. metal - June 30, 2008

The discipline is already trivialized

7. Nick - July 1, 2008

Ooooof. I remember my physics professor talking about all the crap he got sent and how he is at least nice enough to send back a response telling the person what they did wrong. Not saying what you rejected was “crap”, but it just reminded me.

I’m seeing that it’s pretty easy to make mistakes in physics, and one of the most important things you do is stay accurate in everything you do.

Later,
Nick

P.S. on a side note, you should make a post about “free energy”. I know it’s not possible (unless it’s more along the lines of “grabbing” the vast amounts of energy in the universe thought already to exist), but it would be interesting to hear your thoughts on it. I came across stuff like zero point energy and the Casimir effect which both looked pretty interesting.

8. dorigo - July 1, 2008

Hi Nick,

well, this is official refereeing for a journal, and the paper comes from a large collaboration of physicists, many smarter than I am. So I would hesitate to liken my experience to your professor’s…

In any case yes, accuracy is of paramount importance. It is the focus of the refereeing, rather than assessing whether something is “right” or “wrong”, which nobody can usually determine anyway from just the paper (one usually finds mistakes only in the source code!).

As for free energy -better call it dark energy- I have a guest post on the matter, click on “guest posts” on the tab above. It was written by somebody who knows the topic more than I do.

Cheers,
T.

9. goffredo - July 1, 2008

Hi Tony. I whole heartedly admire you for saying that you did not remain anonymous and instead sent emails to the authors. I do think that we should keep personal interactions at a minimum so I wouldn’t make personal contacts the norm, but I admire you anyway.

Jeff

p.s. don’t people go to conferences to communicate results and THERE get all the flak and even personal attacks from all types of “peers”, both from serious and talented ones AND also from complete jerks? Of course if one instead submits a paper straight to a journal then the work should be well prepared. In the particular case of the paper refereed by Tommaso I imagine that the disaggrement of data with the simulations would have been noticed by most in the audience of a serious conference. Indeed I expect that the authors, anticipating that, would not have even gone to such a conference. They might have chosen a foolish conference instead.

10. dorigo - July 1, 2008

Hi Jeff,

yes, science cannot do much progress without peer review, especially in these days (see the Say of the Week of this week!) where one can find support for whatever crazy idea one has, even the flatness of the earth or similar outlandish thoughts.

Hi Marianne,

usually, in our field there is no need for a science writer. We have our standards, but they have little to do with the form and the flow, and much to do with contents.

Hello Changcho,

in fact, I would have been happier to recommend publication. The good thing, at least, is that (save some intuition by the authors if they visit here) I remain anonymous to them, otherwise bad feelings would be easy to foster.

Jeff, yes, the disagreement of data and sum of contributing processes was clear, but I asked the authors to work on it more mainly because of the lack of information… The fact is, one got the impression that the disagreement (a largish effect) was having a sizable impact on the measurement, but they did not quantify it in any way, nor commented it. That is what really caused me to reject the current version of the manuscript…

Cheers all,
T.

11. Nick - July 3, 2008

P.S. Wasn’t trying to compare my story to this, it just reminded me of it. Sorry if it sounded like I was calling those people stupid, as I’m guessing they definitely aren’t

12. dorigo - July 4, 2008

Yes, they aren’t. They are just overburdened with new experiments… The paper refers to an experiment which is now decommissioned.

Cheers,
T.


Sorry comments are closed for this entry

%d bloggers like this: