jump to navigation

Idiot’s guide to handling a multi-million-$ experiment May 21, 2007

Posted by dorigo in computers, humor, personal, physics, science.
trackback

Every time I run a Scientific Coordinator (SciCo for the insiders) shift at the CDF control room, I have this funny feeling of being unqualified for the task.

Sure, I studied particle physics. Of course, I know every bit of this detector rather well – I have been around for 15 years now. And quite naturally, I have studied the documentation required to perform SciCo shifts, I have gone through the emergency procedures, the shot setup checklist, and the silicon monitoring instructions. And besides, the responsibilities of a SciCo make a short list: he or she has to answer phone calls, write entries in the e-log to document what happens, direct the shift crew to the most efficient data taking, phone experts if they are needed, and little else.

Nonetheless, an experiment like CDF is not only a giant device, which cost thousands of years of manpower and hundreds of million dollars to build, but also a very complex system, and it is difficult to fully understand what is going on when something stops working the way it should. Even deciding which expert to call is sometimes not trivial – even worse, there are cases (especially during night shifts such as the one I am running presently) when one has to decide whether to wake somebody up or not for a consultancy on some issue which could be serious or irrelevant.

So, I feel incompetent – and I may indeed be: the readers of this blog have a proof I even get confused with basic facts about electric circuits. But I have cooked up a recipe to shrug off that funny feeling: make a tour of the tens of monitors, crates, and panels which adobe our control room and the adjoining Level-2 counting room, in search for hints such as the ones I document below. They are visual proofs that it is considered normal to be ignorant on some of the details of how the control devices work, and human to make stupid mistakes. So here we go:

The picture above shows a monitor dedicated to controlling the status of the silicon detector. Green boxes are modules which are in a normal state, yellow boxes show modules which are within tolerances but still ok, and black boxes modules which are off or dead. Pink would mean that a module has gone out of range and requires to be reset. What is reassuring here is that a color copy of the “normal” status of the detector components has been posted above the screen, such that you do not need to know much to see if things are going as they should or if there is something to worry about.

Just underneath the keyboard of the monitor shown in the previous picture, a diagram covered with transparent plastic has been stuck. It contains a flowchart that my eight-year-old child could easily follow. Also note the fine print: “NB: You expect a big jump in the bias current after you just biased a ladder (remember Ohm’s Law?“.

Above, the panel with the SVX abort led. The black button serves the purpose of silencing the alarm if it goes off (it produces a nasty sound). Here, a notice has been stuck to warn the incautious operator not to mess with the button: “Do NOT (push to acknowledge) without explicit permission from the CDF operations manager“. Just in case.

Above you can see the “s*** happens” button for the protection of our precious silicon detector, and on top of it, the cooling alarms for the SVX, ISL, and L00 subsystems. If the cooling fails, these detectors are at risk. The corresponding green led will turn off, and a siren will sound. However, when I took the picture somebody had stuck a color copy of the same panel on top of the ISL and L00 part, because those two detectors have been turned off these days (to solve a cooling problem). Just in case a newbye got frantic by seeing the black leds…

Also note the detailed notes explaining the difference between the different alarms, quite in “emergency procedures for dummies” style.

This is my favourite. The main control panel has a poster attached, to make sure passers-by know they should not mess around too much.

Just another sticker with miscellaneous instructions. We live of these things.

The style of messages attached here and there is often friendly: the one above explains things as your grandpa would have.

The panel here contains buttons and pulls that I hope I will never have to mess with. The one on the left is the halon release in the collision hall, to be used only if a real fire develops there: it totally floods the hall with fire extinguishing stuff. On the right, the other safety buttons have been protected with a “dummy-safe” plexiglass cage: that way, if you accidentally trip in front of the panel, you run fewer risks of causing some trouble…

Comments

1. jeff - May 21, 2007

Ah che nostaglia! I miss it! But you need to be young to handle the frustrations. I fear my heart couldn’t put up with the stress and consequent high blood pressure.

2. Alexander W. Janssen - May 21, 2007

Oh my, I laughed really hard when I read you had a “L00 subsystem”. That brings a totally new meaning to the phrase “when the sh*t hits the fan”…🙂

I was working in a network operations center of a phone company some years ago, doing night shifts; we aso had our surveillance systems and in former times even an alarm-printer: It just belonged to my duties to visit the alarm-printer every 30 minutes, checking if a huge “CRITICAL ALERT” A3-sized page was printed out. And I loved the huge alarm-panels with that damn-loud bell.

And well, our escalation-plans were (and still are) a huge mess, a flowchart as big as A1.
However, we didn’t have scary alerts like “radiation alert” or something, the worst thing which could happen was a total call-outage (only emergency-calls possible) or EVEN WORSE, a total billing-outage. That’s the worst-case which scares the hell out of mobile-phone companies🙂

Luckily nowadays everthing is more silent and more computerized, but I miss those blinkenlights… But I ain’t no NOC-bitch anymore, I swaped the job in for something more engineering-orientated.

Cheers, Alex.

3. Kea - May 21, 2007

Boy, you mean they put shameless blogging addicts in positions of responsibility there? Thanks for the great insight into how it all works! I think I can appreciate the impossibility of not being able to understand how everything works. It reminds me of the Search and Rescue coordination exercises for avalanche incidents, where everybody gets handed a little card explaining that you can’t pee in the snow (it ruins the dogs’ buriedpeople-detection software).

4. carlbrannen - May 22, 2007

This is great. Re fires. When I arrived at UCI’s neutrino group in the summer of 1982, there had just been a fire at Los Alamos that had been undetected for some time. My first project was a temperature monitor that could kep track of 16 or 32 temperatures. It was also used for the system that purified a gas used in the system, so it was supposed to be approximately correct for 1 degree C from liquid nitrogen (when purifying gas) to molten lead (when recharging the filter, maybe it was a molecular seive) . I loved building hardware.

5. dorigo - May 22, 2007

Jeff, I did not know you had a problem with high blood pressure. I think CDF in Run II is not so risky for you however. Shifts are mostly uneventful…

Cheers,
T.

6. dorigo - May 22, 2007

Alex, a critical alert printed out and left on the printer ? Wow, even I could think a more effective alert system. But good for you you don’t run night shifts anymore… They suck one’s life away.

Cheers,
T.

7. dorigo - May 22, 2007

Hi Kea,

well, yes, things are organized in a way that even an idiot can do a decent job… Despite that, last night I did pee in the snow. I failed to call an expert during the night for a problem that required immediate attention (my crew and I were misled by a particular configuration we’d been told to run with). It turned out that he would have been unable to do anything until the next morning, so all was good… But my deficient procedure could have cost our experiment a whole night worth of data. Now, since running the experiment costs O(10M$)/year, that loss could be estimated at some 10k$….

Cheers,
T.

8. dorigo - May 22, 2007

Carl, how did the thermometer work ? It seems a pretty wild range from liquid nitrogen to liquid lead. A combination of different systems ?

Cheers,
T.

9. carlbrannen - May 22, 2007

dorigo,

Thermocouples are amazing things. They are extremely repeatable, until their electrical contact is broken. But there are three problems. First, they have low output so you must pay proper attention to common mode rejection ratio when you amplify them. Second, they are quite nonlinear so you have to put in a correction table. (I used a single chip microcomputer.) Third, their voltage depends on the ambient temperature.

I had been told that I could rely on ambient temperature being exactly 72 F. This turned out not to be the case so at the last minute I had to add an “artificial ice” circuit. This is a circuit that gives a voltage that tracks the voltage of a thermocouple that would be put into an ice bath. The voltage from this circuit is subtracted from the other voltages to reference them to ice = 0 degrees C.

I designed and built a CAMAC card so that the big computer system there could read out the voltages directly. I must have been quite the masochist back then. If I did it again I would let the software on the big computer do the ice subtraction and nonlinearity corrections.

One of the things I love doing is soldering things in such a way that they last forever. There are various secrets to this. A big one is how you trim the insulation on your wires. Another is routing most wires so they do not move, and the remaining wires are of the proper type and have “strain relief”.

The experiment was at Los Alamos. Herb Chen was the PI. He died some 20 years ago, which makes me feel quite old. I think that the gas was purified so it could be used in flash chambers and that the experiment was E225, but it’s been a long time.

My main learning experience from this was to never again believe circuits and figures that are taken from magazines, but instead to always understand the theory and verify calculations. I had copied an artificial ice circuit and it turned out to have its amplification too high by 1000x. It must have taken me several days to figure out why my circuit was always stuck high or low.

10. Kea - May 23, 2007

voltage depends on the ambient temperature

Hee, hee. We used them back in the heyday of high-Tc superconductivity back in (?) 1987. I used to know a lot more about electronics back then, but I can’t say I remember much!

11. franco - May 23, 2007

you studied particle physics, sure, but you did not
grasp much.
sure, you have been around CDF for 15 years, but
your legacy is in your blogs, certainly not in PRD papers.
CDF may be a giant detector, you are a midget
with a wet mouth. Get real

12. dorigo - May 23, 2007

Kea, what is the electronics you were involved with ? I thought you were a die-hard theoretician with feet on mountainous terrain and head on the clouds😉

Cheers,
T.

13. dorigo - May 23, 2007

Hi Carl,

fascinating story. What I like of it is that it transpires that you do love working with electronics hardware… I have little experience in that field, but I do love soldering. I ran a week-long shift here in CDF during the Run II upgrade years. It involved soldering thousands of contacts for some calorimeter front-end boards. I do mean thousands – I think at least a dozen. And yes, combing the wires in the right way is a key to ensuring that the job is well done…

I am growing interested in your CV… Are you a technician or an engineer ?

Cheers,
T.

14. dorigo - May 23, 2007

Dear Franco,

I agree with much of what you said.

I did grasp something out of particle physics books, but I forgot most of it. What remains, as you know, is culture, plus the knowledge of where to fetch information if I need it.

And yes, my legacy is (currently) more in my blogs than in PRD papers, although I did write a few papers which will be around for a while more.

But no, I do not see myself as a midget – I think there are several midgets around in our field, but I am sharper than you seem to think. Only, I sometimes like to fly low.

Finally, I do have a wet mouth… But I fail to understand why that appears to disturb you.

Cheers,
T.

15. There is a cable on the floor! « A Quantum Diaries Survivor - May 23, 2007

[…] trip on the cable, or stomp it, or who knows what… So, quite in line with the spirit of the Idiot’s guide I discussed the other day ,  they designed a “professor-proof” protection, consisting of yellow tape strung […]

16. carlbrannen - May 24, 2007

Dorigo, if you looked at what I did in school it would be mathematician. I put food on the table by engineering. I was a grad student in physics for about 2 years. It looked like it would be impossible to get a job in theoretical physics, and I didn’t see any reason to do experimental physics when I could do very similar things in industry and be better paid.

An electronics engineering design is a set of instructions for taking a pile of parts and to combine them into something that meets a specification. The problem is that the parts have very wide tolerances. Because of this it is frequently very easy to get one prototype to work, but in production the variances cause high failure rates (or the prototype fails at different temperatures, for instance).

The solution is a correct design, which must accept the tolerances of the parts and stay within the desired specification of the design. This is a mathematics problem, a mapping problem, and it is a fun one to work on. If you understand it, you can go far in industry.

17. franco - May 24, 2007

Tommaso

that is what is bothering me.
You may be smart, and sharp, but you don’t make
a dent in the progress of our field. You are out of it,
and reinvent yourself in your blog . I appreciate the
therapeutic value – hard to confess being a failure
to your superego – but it makes me hangry that you
use the taxpayers money, that pays your salary,
for nothing more than venting banalities on these
pages.

18. dorigo - May 24, 2007

Carl, you did look more like a mathematician than an engineer to me, but I understand that one thing is one’s vocation and another is how one earns one’s daily bread.

Cheers,
T.

19. dorigo - May 24, 2007

I answered Franco privately, but I write down here the main lines of my mail:

– I am not a general, but I am a good soldier, and I did contribute to a few important things in HEP in the past. Most notably, the Z->bb signal, and the tools for searching for Higgs bosons.
– My blog has many functions, one of which may well be the one Franco mentions.
– Being paid by INFN himself, Franco knows my salary is very low and is probably as ashamed as I am that Italy is in this situation with basic research. Anyway, I spend 12 hours a day in front of a computer screen, and am paid for 7. I do not see much of a waste of taxpayers money in using an hour a day of my time for running a blog.
– These pages do contain banalities, but they do fill a gap for people interested to know what is going on in experimental HEP. Ultimately, the response to it is the justification for running it.

Cheers,
T.

20. carlbrannen - May 24, 2007

One of the things that Herb Chen told me was that he was very proud that neutrinos had no use, especially military use. I was reminded of this today when somebody, perhaps his ghost, found one of my websites by searching for “neutrino” and “weaponry”.

But the purpose behind neutrino physics is not practicality. Society pays for this stuff because of curiousity. I really don’t see how cutting off acadmics from writing blogs that the public reads would be in the interest of either physics, physicists, or the public.

21. alvin - May 24, 2007

wow !
what a crappy answer you gave to Franco.
15 years in CDF, and that is whot you got : a Z->b bbar signal (never published) and tools to search for the Higgs (and what are they, out of curiosity ?).
How can I believe that you spend 12 hours a day
in front of a computers? Dario told me that you arrive at work at 10 am and leave at 4 pm. Beside,
Cdf is an “as complicate experiment as a space station”. How did you built it if you always stare at a computer ? May be your colleagues did,
those who don’t write blogs and master the art of hep better than that of self-aggrandizing

22. dorigo - May 25, 2007

As discussed in another thread, the person above (alvin, aka franco, aka JoAnn, aka aldo) is Paolo Giromini, a research director from Frascati. He has been using multiple fake names (and correct email addresses) to attack me. I will leave the comments up here, they comment themselves.
T.

23. LAURA - May 26, 2007

I do not believe you.
Where is the attack ?
You are always so melodramatic: I lost my
power cord, I waste my life on night shifts,
I have to hop on flights all the time,
there is a cable on the floor…
You received harsh comments
and, surprisingly, you agreed with most of them.
So, does it matter where they came from ?
By the way, if you can’t prove this accusation,
you may get into a lot of troubles.
You hope he is a good spirit – he may be if he
is for real. Why are you complaining at all, anyway ?
If you pretend to be what you are not – a scientist-
why he cannot pretend to be JoAnn, alvin, aldo, franco…
I got lost – your blog has become a zoo.
He is probably teaching you a lesson by diminishing himself.
Put aside your stupid ego and say thank you !

24. dorigo - May 26, 2007

Dear Paolo Giromini, aka Laura,Aldo,Franco,Alvin, JoAnn, Debora, peter, etc:

every comment you leave here is sent in carbon copy as a message to my mailbox, with a specification of the IP address they come from. I can tell you which ones you sent from the nfras04 pc (IP 131.225.232.160) in your office, and which ones from the comcast account (IP 67.167.243.175) you have at home. If you need a crash course on anonymity on the web, look elsewhere though, I am not the most informed person. But I can read nfras04.fnal.gov (for other readers here, nfras04 is a machine of the Frascati cluster at Fermilab).

So be careful, it is you who used fake identities and are liable to be sued by the people you impersonated. Just to mention a few: JoAnn Hewett, Aldo Menzione, Franco Cervelli, Alvin Laasanen. All respected scientists who I think would not like to know the use you do of their names and (real) email addresses.

Sorry, but you force me to disclose it by saying I cannot prove the accusation… Every good game does not last long.

Cheers,
T.

25. carlbrannen - May 27, 2007

Dorigo, I just remembered another Herb Chen story and thought I’d put it here before senility takes it away from me.

One of my fellow grad students was given the task of tapping a series of holes in a pieces of aluminum. Instead of using the hand tap provided, he chucked the tap into a power drill and promptly broke it off. I grew up in a shop sort of environment and thought this was hilarious.

When I told this story to Dr. Chen, he didn’t laugh. Instead, he told me that when he was a grad student he had done the same thing. Ooops.

26. dorigo - May 27, 2007

Hi Carl,

well you can’t expect that novices will become experts by just trial. They also need error sometimes.

When I was building the CMX extension for the CDF upgrade in 2000, I had to drill some 300 1/4″ holes in large slabs of 1/8″ steel.

Since the slabs wouldn’t fit in any machine, the work was done with a hand drill… And I did break quite a few bits to finish the job, but it was just impossible to avoid id. I think by the time the job was done I was quite a good driller of steel…

Cheers,
T.


Sorry comments are closed for this entry

%d bloggers like this: