A Quantum Diaries Survivor

Imagine you have n background models B_i, with i=1…n -all equally reasonable- for a mass distribution, and the data contains mostly background but includes a small amount of an additional signal of unknown mass M.  

You take each background model B_i in turn and fit the data to the sum of background plus a signal template which depends on the unknown value of M. Each fit will return a preferred value  M_i, its statistical error S_i, and a chisquared X_i, which is a measure of how well the data is explained by the hypothesis of signal of mass M plus i-th background.

An example is given below, where the various values of X_i are plotted on the y axis, at a value proportional to the mass M_i on the x axis (it is the infamous b-jet energy scale - a ratio between the measured and true mass value of a jet-decaying resonance): 

Now, how do you choose which is the right value M_i of M among the n possibilities, if all the fits are reasonably good? And how do you assign a systematical error to your measurement due to the arbitrariness of the background model ?

One solution would be to take the weighted sum of the n values  M_i, where the weights are the factors exp(-X_i). I have seen that done somewhere. But for some reason I do not like that too much. It looks as if it sort of underestimates the systematical error due to the arbitrariness of the background model.

My skin feeling would say you need to give each background model equal chances of representing nature, and thus you cannot rule any of them out by looking at the combined behavior of the other models. Following this line of reasoning brings me to think that defining the parameter estimate as the middle point of the range spanned by the M_i and the systematic error on the parameter as some sort of “maximum spread” is the way to go.

Any thoughts on the subject ? Examples of how the matter is dealt with in specific measurements ? Your contribution is welcome.