Idiot’s guide to handling a multi-million-$ experiment May 21, 2007Posted by dorigo in computers, humor, personal, physics, science.
Every time I run a Scientific Coordinator (SciCo for the insiders) shift at the CDF control room, I have this funny feeling of being unqualified for the task.
Sure, I studied particle physics. Of course, I know every bit of this detector rather well – I have been around for 15 years now. And quite naturally, I have studied the documentation required to perform SciCo shifts, I have gone through the emergency procedures, the shot setup checklist, and the silicon monitoring instructions. And besides, the responsibilities of a SciCo make a short list: he or she has to answer phone calls, write entries in the e-log to document what happens, direct the shift crew to the most efficient data taking, phone experts if they are needed, and little else.
Nonetheless, an experiment like CDF is not only a giant device, which cost thousands of years of manpower and hundreds of million dollars to build, but also a very complex system, and it is difficult to fully understand what is going on when something stops working the way it should. Even deciding which expert to call is sometimes not trivial – even worse, there are cases (especially during night shifts such as the one I am running presently) when one has to decide whether to wake somebody up or not for a consultancy on some issue which could be serious or irrelevant.
So, I feel incompetent – and I may indeed be: the readers of this blog have a proof I even get confused with basic facts about electric circuits. But I have cooked up a recipe to shrug off that funny feeling: make a tour of the tens of monitors, crates, and panels which adobe our control room and the adjoining Level-2 counting room, in search for hints such as the ones I document below. They are visual proofs that it is considered normal to be ignorant on some of the details of how the control devices work, and human to make stupid mistakes. So here we go:
The picture above shows a monitor dedicated to controlling the status of the silicon detector. Green boxes are modules which are in a normal state, yellow boxes show modules which are within tolerances but still ok, and black boxes modules which are off or dead. Pink would mean that a module has gone out of range and requires to be reset. What is reassuring here is that a color copy of the “normal” status of the detector components has been posted above the screen, such that you do not need to know much to see if things are going as they should or if there is something to worry about.
Just underneath the keyboard of the monitor shown in the previous picture, a diagram covered with transparent plastic has been stuck. It contains a flowchart that my eight-year-old child could easily follow. Also note the fine print: “NB: You expect a big jump in the bias current after you just biased a ladder (remember Ohm’s Law?“.
Above, the panel with the SVX abort led. The black button serves the purpose of silencing the alarm if it goes off (it produces a nasty sound). Here, a notice has been stuck to warn the incautious operator not to mess with the button: “Do NOT (push to acknowledge) without explicit permission from the CDF operations manager“. Just in case.
Above you can see the “s*** happens” button for the protection of our precious silicon detector, and on top of it, the cooling alarms for the SVX, ISL, and L00 subsystems. If the cooling fails, these detectors are at risk. The corresponding green led will turn off, and a siren will sound. However, when I took the picture somebody had stuck a color copy of the same panel on top of the ISL and L00 part, because those two detectors have been turned off these days (to solve a cooling problem). Just in case a newbye got frantic by seeing the black leds…
Also note the detailed notes explaining the difference between the different alarms, quite in “emergency procedures for dummies” style.
This is my favourite. The main control panel has a poster attached, to make sure passers-by know they should not mess around too much.
Just another sticker with miscellaneous instructions. We live of these things.
The style of messages attached here and there is often friendly: the one above explains things as your grandpa would have.
The panel here contains buttons and pulls that I hope I will never have to mess with. The one on the left is the halon release in the collision hall, to be used only if a real fire develops there: it totally floods the hall with fire extinguishing stuff. On the right, the other safety buttons have been protected with a “dummy-safe” plexiglass cage: that way, if you accidentally trip in front of the panel, you run fewer risks of causing some trouble…