The current situation has at least two consequences:
1 – any “mistakes” (even if not “usually” existing) will not be found by independent analysis;
2 – some NN will be elevated to Holy Writ, to be followed absolutely no matter how wrong they may be.

The situation is not confined to physics.
See for example a 31 July 2007 web article by Katherine Burton and Richard Teitelbaum at
infoproc.blogspot.com/2007/07/algorithm-wars.html
that said in part:
“… Renaissance … founded by Simons in 1988, is a quantitative manager that uses mathematical and statistical models to buy and sell securities, options, futures, currencies and commodities. It oversees $36.8 billion for clients …[and it]… sued Belopolsky and Volfbeyn in December 2003, accusing them of misappropriating Renaissance’s trade secrets by taking them to another firm, New York-based Millennium Partners …
Volfbeyn said that he was instructed by his superiors to devise a way to “defraud investors trading through the Portfolio System for Institutional Trading, or POSIT,” an electronic order-matching system operated by Investment Technology Group Inc. …
Volfbeyn told superiors at Renaissance that he believed the POSIT strategy violated securities laws and refused to build the algorithm, according to the court document. The project was reassigned to another employee and eventually Renaissance implemented the POSIT strategy, according to the document. …”.

In short, it seems to me that the use of trade secrecy to hide the way hedge funds function is very much like the use of authoritarian arrogance by physicists to hide the way NN code works,
and that in both cases the only real reason for hiding the facts is to protect the powerful.

Tony Smith

Cheers,
T.

just wanted to say thanks for this excellent post.

As the noose begins to tighten around the far-extended necks of the “powers that be”.

And there’ll be dancin’
dancin’ in the streets

Cheers,
T.

Andreas

Cheers,
T.

We used to be able to get very interesting (i.e. goofy) results by messing around with the input space of NN models for doing data mining. I always felt there were numerically “stiff” areas all over the “classification spaces”. I am sure what you talk about in learning how to use them was learning where they actually were applicable.

We used to take the output function weights and try to understand them by varying them, and looking at what happened to the classification, and then messing around with the inputs to give a clue as to how to get the variation we were applying. Is that what you were doing with your models? I would think that this is one of the perfect uses for NN’s as you can kind of nail down the variables and you don’t have the classic “all the pictures with the tank in it were when it was shady” junk.

sure I know you’re one of the best. I think the fact we give little importance to details of how NNs work has its roots on the fact that we all spent a large fraction of our life checking the results the first NNs used in HEP were giving meaningful results. I remember one time, in the mid nineties, when it was all about “how many events did you use for the training?”, “What is the estimated uncertainty?”, “How did you evaluate that ?” “How can you be sure?”, and the like. Students working with NNs used to be grilled on skewers.

Now, nobody asks those questions any more – it’s not fun: there are no mistakes to spot – usually!

Cheers,
T.

Kea, no, no fairy fields yet. I start to think we could well end up with nothing in our hands

Cheers,
T.

(*) I’m not throwing the first stone, I always did exactly the same. But I know for sure that I’m not alone, and by far not the worst

Cheers,
T.

yes, the NN they use seems to have some peculiar behavior, which I think is under control anyway. Its output is not the usual distribution from 0 to 1 with a spike at 0 for background and a spike at 1 for signal, but it seems to be doing a reasonable job anyway… The spikes are due, according to the paper, to “regions of phase space which are cut out for the training by the use of kinematical pre-cuts”. In fact, they have different NN configurations (different inputs and optimizations) as the cut level and test higgs mass vary….

Hi Thomas,

yes, the discrepancy with expectations is an effect due to the fact that at 120 GeV they observe 31 events while they expect only 20.8+-1.7 events (with an expectation for Mh=120 of 0.32 events from H->WW). If I do the simple math of extracting a xs from those numbers, I get (10.2+-5.5)/0.32=(32+-17)xSM, which means that a 95% CL stands at about x60SM (I’m making these computations by heart, so bear with me for inaccuracy). Larger, but not too different from what they get by taking into account the actual distribution of the NN output, after folding in systematics (I read x47SM in the plot). That means their binning in NN output does help, as it should (the NN output is still a discriminant after all cuts, as is evident in the plots).

Cheers,
T.