Probability: Identifying Pascal Scams

From Unenumerated:

Beware of what I call Pascal's scams: movements or belief systems that ask you to hope for or worry about very improbable outcomes that could have very large positive or negative consequences. (The name comes of course from the infinite-reward Wager proposed by Pascal: these days the large-but-finite versions are far more pernicious).  Naive expected value reasoning implies that they are worth the effort: if the odds are 1 in 1,000 that I could win $1 billion, and I am risk and time neutral, then I should expend up to nearly $1 million dollars worth of effort to gain this boon. The problems with these beliefs tend to be at least threefold, all stemming from the general uncertainty, i.e. the poor information or lack of information, from which we abstracted the low probability estimate in the first place: because in the messy real world the low probability estimate is almost always due to low or poor evidence rather than being a lottery with well-defined odds:

(1) there is usually no feasible way to distinguish between the very improbable (say, 1 in 1,000) and the extremely improbable (e.g., one in a billion). Poor evidence leads to what James Franklin calls "low-weight probabilities", which lack robustness to new evidence. When the evidence is poor, and thus robustness of probabilities is lacking, then it is likely that "a small amount of further evidence would substantially change the probability. "  This new evidence is as likely to decrease the probability by a factor of X as increase it by a factor of X, and the poorer the original evidence, the greater X is.  (Indeed, given the nature of human imagination and bias, it is more likely to decrease it, for reasons described below)....MORE