tag:blogger.com,1999:blog-8060573803177342872024-03-14T18:01:00.424+00:00It's A Stat LifeThoughts on uses and abuses of statistics in the news, plus a few educational pieces.Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-806057380317734287.post-5134059034006595462015-12-09T09:41:00.003+00:002015-12-09T09:41:30.377+00:00B is for Bayesian InferenceJust a note to say that I guest blogged for the University of Oxford's <a href="https://www.maths.ox.ac.uk/about-us/engagement/oxford-mathematics-alphabet/b-bayesian-inference">Mathematics Alphabet</a> this week, on B is for Bayesian Inference. Do check it out!Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-4680123209878979112014-10-16T10:00:00.000+01:002014-10-16T10:00:06.088+01:00Ebola huntingHere's a quick prediction for what happens now that <a href="http://www.bbc.co.uk/news/uk-29616724">the UK has started screening people for Ebola</a> at Heathrow (soon to be extended to other airports and terminals). They'll find a lot of people with minor stomach bugs of some sort.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwBhgzyhlCqPs3z6Jk7mOchEvfykcYHWmmEvbuUn4LAbU8rSFQVLbntAUQk56m5xFMU7y8yc9YsMMpDq1GM61iYNBI3uWyYsMOqDergf-4TNEIBz4itWqvYlXLcT4XQ2DUqzHdZ1plp64/s1600/Ebola_virus_virion.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Ebola virus" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwBhgzyhlCqPs3z6Jk7mOchEvfykcYHWmmEvbuUn4LAbU8rSFQVLbntAUQk56m5xFMU7y8yc9YsMMpDq1GM61iYNBI3uWyYsMOqDergf-4TNEIBz4itWqvYlXLcT4XQ2DUqzHdZ1plp64/s1600/Ebola_virus_virion.jpg" height="183" title="" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Image: CDC</td></tr>
</tbody></table>
<br />
<a name='more'></a><br />
There are <a href="http://en.wikipedia.org/wiki/Ebola_virus_epidemic_in_West_Africa#Timeline_of_cases_and_deaths">about 150</a> new Ebola cases reported each day at the moment, so even if we're generous and assume the true figure is 300, and that symptoms last for 2 weeks, only about 4,000 people in total have it at any given time. The combined population of the worst affected areas (Guinea, Liberia and Sierra Leone) is about 20 million, so maybe approximately 1 in every 5,000 people are affected. Fewer than this will be in a state fit to travel (I imagine they would have to be in the early stages of the disease).<br />
<br />
Gastroenteritis is endemic pretty much everywhere, and is worse in the developing world because of poor sanitation and sewage treatment. There are around <a href="http://en.wikipedia.org/wiki/Gastroenteritis#Epidemiology">3 to 5 billion cases</a> per year, so on the order of half the people in the world get it each year (in reality, young children are more vulnerable than adults). If it lasts for 4 days on average (it can be much worse, of course), that suggests that on a given day about 1 in 200 people would have it. <br />
<br />
From this a fairly generous estimate is that someone screening will see at least 25 people with gastroenteritis for every person who has Ebola. Since many of the symptoms are pretty similar, I predict that, if anything happens at all, a lot of people will get sent to hospital without much cause.<br />
<br />
Note that, in reality, international travellers are much richer than most residents of the west African countries affected; consequently they have better access to sanitation and healthcare when in those countries, and will have a substantially lower risk of infection with Ebola as a result (though this also applies to gastroenteritis).<br />
<br />
The moral is that <a href="http://www.badscience.net/2006/12/crystal-balls-and-positive-predictive-values/">hunting for rare things</a> is a pretty thankless task.<br />
<div>
<br /></div>
Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-20125123506544717692014-10-15T09:05:00.001+01:002014-10-15T09:05:19.342+01:00What price a preposition?The headline to <a href="http://www.theguardian.com/politics/2014/oct/14/ukip-30-seats-2015-general-election">this article</a> made me aware of a pretty big ambiguity in its contents: there's a big difference between saying that "in 30 seats UKIP are likely to win", and "UKIP are likely to win 30 seats". <div>
<br /></div>
<div>
To see this, consider the statement: "Chelsea are likely to win every game they play this season". Interpreted one way, you're just saying that Chelsea are favourites in each game they play (plausible). Another meaning would be that Chelsea are likely to win <i>all</i> the games they play, which is exceptionally unlikely (no-one has ever come close).</div>
<div>
<br /></div>
<div>
Of course the results in different seats in general elections are strongly correlated, much more so than football matches. But I'd still like to know what exactly the Guardian means.<br />
<br /></div>
Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-76564315619992429552014-10-12T23:35:00.002+01:002014-10-14T17:26:42.027+01:00Can a gamble ever be right or wrong?A couple of weeks back I was feeling fairly smug, having put a bit of money on a 'No' vote in the Scottish referendum. I then placed a few quid on UKIP to win the <a href="http://en.wikipedia.org/wiki/Heywood_and_Middleton_by-election,_2014">Heywood and Middleton by-election</a> last Thursday, which they didn't. But the latter result was quite close, so I felt that somehow it had been a 'good bet' to have made. So what makes a person right or wrong to have placed a bet in the first place? (Mathematically that is, leave your morals at your home page.)<br />
<br />
<a name='more'></a><br />
As a statistician I'd say that winning or losing a particular bet does not in itself indicate whether or not the bet was <i>good value</i>. A bet is good value if you expect (on average) to make money from it. For example, suppose someone offers to let you bet that England will win their next match at odds of 50% (2 in decimal odds, evens in traditional parlance); the bet is good value if <i>you</i> think the chance of that happening is greater than 50%. If the true odds are 75%, then you expect to make<br />
$$<br />
0.75 \times £1 \times \left(\frac{1}{0.5}-1 \right) - £1 \times 0.25 = £0.50<br />
$$<br />
for every £1 you stake. Sounds good! But after the match is over, you're left with either double your money or nothing. <br />
<br />
I got odds of about 5.5 (18% or 9 to 2) on UKIP winning in Heywood. This seemed like a bit of a long-shot, so when I saw that the election had indeed been so close this seemed like evidence that the true chance of them winning <i>had been</i> higher than 18%. This is obviously slightly illogical, since by the time I had that information the chance of them winning had dropped to zero.<br />
<br />
So is there an objective way to say whether a particular bet was good value or not? For a one-off event like the Scottish referendum, or even the by-election, the answer is pretty clearly no. There are many expert <i>opinions</i> available about which outcomes were plausible (from both before and after the event) but these are ultimately subjective.<br />
<br />
Purely <a href="http://itsastatlife.blogspot.co.uk/2012/05/what-is-probability.html">subjective probabilities</a> are of limited practical use in this context: if I 'expect' to make money, but I'm also an idiot, then this doesn't help me much. Such probabilities are most useful when they have some sensible long-run interpretation: e.g. if I roll a die 6,000 times about 1,000 times it will show a six: hence we'd say the probability is $\frac{1}{6}$. Referenda have no such interpretation without resorting to rather contorted abstract ideas, because there aren't any other events which are the same (or even vaguely similar, usually) to a particular plebiscite: you can try to compare it to other elections, but you'll never know whether this one was somehow 'different'. But surely probabilities for these events aren't totally meaningless, are they?<br />
<br />
For a probability forecaster like a weather service or a bookie, you can check whether they are 'calibrated': take a large number of predictions of (say) 60% rain, and check whether approximately 60% of them were followed by rain. The events don't even have to be related: you can throw in any events given a 60% probability and see if they're jointly calibrated. <br />
<br />
But for any individual prediction, there isn't much you can say about its quality from purely statistical evidence. What about <a href="http://www.bbc.co.uk/sport/0/football/28968148">a pundit</a>? Well I'd say they only add value if you'll make money by betting on his or her predictions: i.e. he or she is telling you something that the bookie's odds don't. If they just tell you the most likely outcome this isn't very useful: in a sporting fixture it's usually clear who the favourite is.<br />
<br />
For a gambler of course, the proof is in the winning: a good gambler will make money <i>eventually</i>, and a bad one will lose it: so far I'm up. My next wager is that the Conservatives will win the most seats at the next UK general election... I'd be quite happy to lose that bet.<br />
<br />
(If you're interested in my reasoning for the bets above: I bet on the Scottish Referendum just after a YouGov opinion poll showed 'Yes' ahead, guessing that people would over interpret this single poll: I got odds of 1.4 (71% or 2 to 5). I bet on the Heywood and Middleton by-election after reading that Labour were nervous about the result: the odds would have been much higher if I'd waited until nearer the day.)Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-5192777639723644082014-01-26T21:15:00.002+00:002014-01-26T21:41:08.007+00:00Frog kissing. Or, why do we never learn?A friend pointed me to <a href="http://jezebel.com/expect-to-kiss-an-average-of-15-frogs-before-you-get-a-1493288810">this story</a> about the average number of men (or rather frogs) a woman has to date before finding their 'Prince Charming' (apparently it's 15). Let's leave aside for now how they came up with the number: I've no doubt the methodology conforms to the most rigorous standards that we've all come to expect from the internet. I thought that this paragraph might give some people hope:<br />
<blockquote class="tr_bq">
If you've made out with 11 men, great news! A new survey suggests you're only 4 away from Prince Charming. On average, anyway.</blockquote>
It's now my duty to crush that hope.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtJLMhW1pSuDI6sQ0EjrHvDD17U_FdeIf9VYswxOW_QvEh56ii01GNGu6ynJHaTBvtNoKQDPO-DtyDgTdwDfwxZ5ks3J6Rxc7Aisa2m8AgM_750LWXqS9sSQMjFsUpqSXLiGlH8MsrF9c/s1600/HRH_Prince_Charles_79.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="A prince" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtJLMhW1pSuDI6sQ0EjrHvDD17U_FdeIf9VYswxOW_QvEh56ii01GNGu6ynJHaTBvtNoKQDPO-DtyDgTdwDfwxZ5ks3J6Rxc7Aisa2m8AgM_750LWXqS9sSQMjFsUpqSXLiGlH8MsrF9c/s1600/HRH_Prince_Charles_79.jpg" height="320" title="" width="228" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Distinguishing princes and frogs can be <br />
difficult at first glance [Photo: Allen Warren]</td></tr>
</tbody></table>
<br />
The bad news is that (even if the rest of it is correct) the survey absolutely doesn't say that. If you've been in 11 disastrous (but possibly ultimately affirming) relationships, you've guaranteed that the total number you end up in is at least 11. But it could have been fewer than that (it happens apparently), so the possibility that you stayed with your childhood sweetheart is included in the average which arrives at 15. Once you exclude that possibility, the average will only increase.<br />
<a name='more'></a><br />
It's the same with life expectancy: every year you live, your <i>total</i> life expectancy will increase simply because you didn't die that year (way to go!). When you're young and healthy your chances of dying this year aren't high, so it's no great surprise when you survive; your life expectancy therefore remains roughly the same.<br />
<br />
If you're 25, male, and live in England and Wales, your remaining life expectancy is 54.71 years, as estimated by the ONS (<a href="http://www.ons.gov.uk/ons/publications/re-reference-tables.html?edition=tcm%3A77-325699">data from 2010-12</a>), for a total of 79.71 glorious years. If you're 26, your remaining life expectancy is 53.74, giving a slightly increased total of 79.94. This becomes much more pronounced for older people: if you're 85 and female, you have a remaining life expectancy of 6.84 years, for a total of 91.84, but by the time you're 86, this has increased to 92.36, a gain of more than half a year, and 10 years more than at birth.<br />
<br />
In fact, it's quite possible for the <i>remaining</i> life expectancy to increase, let alone the total. Imagine a population in which half the people die as infants (under 5), but the remainder live to be 75. When born, your life expectancy is about $\frac{1}{2} \times 5 + \frac{1}{2} \times 75 = 40$: not great. But if you make it to age 5, you're guaranteed to live another 70 years. Sadly, high infant mortality means that in some parts of the world this is not an abstract concept.<br />
<br />
<h4>
Kermit's Revenge</h4>
One particularly interesting case is if the distribution is <i>memoryless</i>. Let's get back to the dating: suppose your approach is to just meet people at random, learn nothing from each experience, and you don't become more or less picky over time. So we assume that, each time, you have a fixed probability $p = \frac{1}{15}$ of ending up in a relationship that works. <br />
<br />
The chance that it takes exactly $k$ relationships before you find a non-amphibian is $(1-p)^{k-1} p$, because the first $k-1$ have to all be unsuccessful, and the $k$th one a charm. If you can sum a geometric series (hopefully my first year students are reading this), then you can show that the total number of people you have to meet before settling down is, on average, $\frac{1}{p} = 15$. <br />
<br />
But if you've already dated 11 people? Well, we can use conditional probability: the chance of you having to date exactly $11+k$ frogs and princes is<br />
$$<br />
(1-p)^{11+k-1} p,<br />
$$<br />
and the chance that you find 11 people without meeting a keeper is $(1-p)^{11}$. So the conditional probability of having to date exactly $k$ <i>more </i>guys, <i>given</i> that you've already put up with 11 toads, is<br />
$$<br />
\frac{P(X = k+11)}{P(X > 11)} = \frac{(1-p)^{11+k-1} p}{(1-p)^{11}} = (1-p)^{k-1} p.<br />
$$<br />
But this is just the chance of having to meet $k$ guys in the first place! This is the memoryless property. It doesn't matter how many times you've tried and failed, you're still at square one until the prince comes along.<br />
<br />
Of course, we might like to think that we learn from our mistakes, grow as people, and make better decisions as we get older, but the empirical evidence available to me doesn't bear this out.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-65585883957545195082013-11-22T09:00:00.000+00:002013-11-27T09:17:38.366+00:00Just Instrument It(With apologies to macro-economists.)<br />
<br />
The amount of time you spend in education predicts your earnings quite strongly, and it's generally agreed (<a href="http://www.telegraph.co.uk/culture/tvandradio/simon-cowell/10452341/Simon-Cowell-is-stupid-for-suggesting-children-dont-have-to-work-hard-at-school-Michael-Gove-has-said.html">Simon Cowell aside</a>) that if you want to do well in life, staying in education for longer is a good idea.<br />
<br />
But how much effect does it have? We could look at a survey of people's incomes and group them by education level, but this doesn't give a <i>causal effect</i>. It might tell us that people who have a masters degree earn more during their lifetime, on average, than those who don't. This could be because people from wealthy backgrounds can afford the tuition for a masters degree, and also have pals in the city who can help them get a big salary afterwards. Or perhaps people who do masters degrees work harder than the rest. We can't easily tell the difference: this problem is called confounding.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia1QNq8tvhzqrlyCLUs9nHAL6X9mBlm67SaXfrVQn-WhD8hiFQfoER9_TmfEz4nfNmKtPgA7l8N3OUSFp96_SSYWU2JvzkSl0Z4styPwE9CjPckoNQyxO3zSDNYSxqCFOQv9c3gOeONVA/s1600/IV1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="182" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia1QNq8tvhzqrlyCLUs9nHAL6X9mBlm67SaXfrVQn-WhD8hiFQfoER9_TmfEz4nfNmKtPgA7l8N3OUSFp96_SSYWU2JvzkSl0Z4styPwE9CjPckoNQyxO3zSDNYSxqCFOQv9c3gOeONVA/s400/IV1.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">We can't tell whether time spent in education level <i>causes</i> earnings <br />
to increase, or there's a third factor which affects both.</td></tr>
</tbody></table>
<br />
<a name='more'></a><br />
Suppose Sarah, 16, is considering whether to drop out of school, or stay for another year. She doesn't care about whether other people who choose to stay in school turn out to be rich, what she wants to know is: if she stays, how much more will she earn than if she leaves (and will it cover the counselling?). If the apparent benefit of education is really down to having rich friends, then Sarah staying in school won't necessarily help at all. Knowing the best course specifically for Sarah is pretty difficult, but we might be able to answer this: if an <i>average</i> person stays in school an extra year, how much more will they earn than if they leave?<br />
<br />
To answer this we can simply do a randomized control trial: take 2,000 people, toss a coin for each, and force those who get heads to stay to 17, and the rest to go straight into work, and see who fares best 25 years later. Regrettably indentured labour (or education) of subjects is not an option open to researchers in social policy, so it is not possible to perform this experiment.<br />
<br />
<h4>
Instruments</h4>
<br />
So what can we do? Well, what we really want is to find a <i>third</i> quantity which affects whether or not people decide to stay in school, but isn't related to later income. This is a bit like outsourcing our coin tosses to a third party who doesn't suffer from our ethical qualms. We call this quantity an <b>instrument</b>.<br />
<br />
What sort of quantity would work? Two researchers from the National Bureau of Economic Research, Joshua Angrist and Alan Krueger, thought they'd found one: the time of the year in which you're born. In many US states it's a legal requirement to be educated until a certain age, say 16. If you're born in September, this means that at the very beginning of 10th grade, you can drop out; but if your birthday is in August, you have to wait right until the end of the grade to leave, so you get an extra year's schooling.<br />
<br />
Looking at US census data, <a href="http://qje.oxfordjournals.org/content/106/4/979.abstract">Angrist and Krueger</a> (1991) found a very weak ($R^2 \approx 0.0001$) but significant association between the time of year individuals were born, and the number of years education they received; the later in the school year you're born, the more education you get. But we can think of this as just a coin toss: the time of year in which you're born is not determined by your family background or other factors relevant to your earnings, it just happens.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNTMgu_f9l6U_DNARxBlyKWNjuBGG7PrGAd3DtTLlPvAGHQyIfwzchcLy4zWmJDBxFArwsynYDv-kx07sJ3eB_UBER0F_7o0nb_Gb4pHNHr-udK0LIe6ZYrUeHSOsdrXdi_NQftq6RHCc/s1600/IV2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="132" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNTMgu_f9l6U_DNARxBlyKWNjuBGG7PrGAd3DtTLlPvAGHQyIfwzchcLy4zWmJDBxFArwsynYDv-kx07sJ3eB_UBER0F_7o0nb_Gb4pHNHr-udK0LIe6ZYrUeHSOsdrXdi_NQftq6RHCc/s400/IV2.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">We assume that the time of your birth is <b>not</b> affected by your background, <br />
but it does have a small effect on how long you stay in school for. </td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
So if we find that people born in September earn less than the group born in August, then the difference must be because of the extra year of education (some of them) received! <br />
<br />
There are four key <b>assumptions</b> we need to make this work:<br />
<ol>
<li>the time of birth (the <b>instrument</b>) is not confounded with education (the <b>intermediate variable</b>) or earnings (the <b>outcome</b>): in other words, the instrument is, for all intents and purposes, randomized and not affected by, for example, your background;</li>
<li>the time of birth doesn't <i>directly</i> affect earnings, except through the intermediate variable: so being born later might mean that you get an extra year of schooling, which in turn means you earn more, but the time of birth can't affect earnings in any other way;</li>
<li>the time of birth has an effect on education: in our case, being born later increases the amount of schooling you get.</li>
<li>the effects are <b>linear</b> (as in linear regression).</li>
</ol>
Then because we assume that the time of birth is randomized (or at least, not confounded), any correlation between birth and education (say $\alpha$) <i>is</i> causal (it's as if we did a randomized trial), and also the correlation between birth and earnings (say $\beta$) <i>is</i> causal.<br />
<br />
Furthermore, because we believe that all the effect of birth on earnings is <i>through</i> education, this means that the causal effect of birth on earnings ($\beta$) should just be the combination (i.e. the product) of the causal effect of birth on education ($\alpha$), and the causal effect of education on earnings (which is what we want to know, say $\rho$). That is: $\beta = \alpha \times \rho$, where we can measure $\beta$ and $\alpha$ from our <i>observational</i> experiment. Then we just use:<br />
$$<br />
\rho = \frac{\beta}{\alpha}<br />
$$<br />
and hey presto! $\rho$ is the causal effect we wanted to find (and will be different in general from the ordinary correlation between education and earnings). This is the method of <b>instrumental variables</b> (IV), and has been around <a href="http://faculty.arts.ubc.ca/ftrebbi/research/ST_JEP_2003.pdf">since the 1920s</a>.<br />
<br />
Note the importance of assumption 3 here: if the instrument and the intermediate effect are not related then $\alpha = 0$, and <a href="http://math-fail.com/2010/06/never-divide-by-zero-ever.html">division by zero</a> leads to unpleasant stomach cramps and a skin rash.<br />
<br />
<h4>
"Dude, that is weak"</h4>
<br />
So Angrist and Krueger divided the effect of birth on earnings by the effect of birth on education, and obtained an estimate of the <i>causal </i>effect of an extra year's schooling on earnings. (I'm simplifying quite a lot here.) They find that additional compulsory years of schooling does lead to higher wages later in life, because of the additional education received. The exact estimates vary for different cohorts, but is of the order of an 8% increase in wages for each additional year of education. Which all seems plausible, and in fact it turns out to be similar to the observed correlation.<br />
<br />
Unfortunately, this approach carries a fatal flaw in this context, as shown in this rather nice 1995 <a href="http://typo3-8442.rrz.uni-koeln.de/fileadmin/wiso_fak/cmr/pdf/Jaeger_Publication_List/jasav90n430.pdf">JASA paper</a> by Bound, Jaeger and Baker. The problem is related to the dividing-by-zero warning I gave earlier. A <i>perfect</i> instrument will control the intermediate variable exactly, just like an actual randomized trial; in this case we don't need to use instrumental variables methods. If the instrument doesn't affect the intermediate <i>at all</i>, then we can't do anything, which is why we need assumption 3. <br />
<br />
But if the instrument only <i>very weakly</i> affects the intermediate (and therefore the outcome), it's almost as bad as not affecting it at all: we have to divide a tiny number ($\beta$), by another tiny number ($\alpha$), and hope we get a reasonable result. As every computer scientist knows this is very bad, because it means that if we get our estimate for $\alpha$ or $\beta$ wrong by even a tiny amount, then the division will give completely the wrong answer.<br />
<br />
This means that if the model is mis-specified by even a tiny amount, then the instrumental variables estimator will be totally misleading. This is the problem of <b>weak instruments</b>. <br />
<br />
We're assuming that the effects are linear (so each extra month's schooling leads to the same number of extra dollars earned, for example) and at best this is only an approximation to the truth. The world just isn't very linear in practice. Worse, assumptions 1 and 2 are essentially untestable in the IV set up, so (without other information) it's quite possible that our model <i>is</i> mis-specified.<br />
<br />
<h4>
Still Instrumental</h4>
Although one should be very careful about drawing strong conclusions from studies using instruments, they are still very useful, and a devilishly clever idea. Many questions in economics just can't be empirically tested by randomized trials: can you imagine asking Mark Carney to flip a coin before deciding whether or not to raise interest rates, just so we can see the causal effect over time? Instead we must resort to instruments or other <b>natural experiments</b>.<br />
<br />
Single studies using instruments are pretty unreliable; but they can still contribute to a wider 'body of proof' about something: we're pretty confident that staying in school increases your earnings, because it's been demonstrated in lots of studies in different countries and time periods. It is a robust finding.<br />
<br />
There is another rather promising application of instrumental variables, one which has gained quite a bit of attention recently. <b>Mendelian randomization</b> uses our genes as instruments, and has been used to prove, for example, that alcohol raises your blood pressure. But I'll save that delight for another time.<br />
<br />
<h4>
Reference</h4>
<a href="https://sites.google.com/site/andrewjohnstoneconomics/my-advice/instrumental-variables">This nice page</a> by Andrew Johnston was useful.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-86141653908654510492013-11-14T09:00:00.000+00:002013-11-14T09:19:33.357+00:00But is it causal? Defining causalityAs I alluded to in my last post, defining what it means for $X$ to <i>cause</i> $Y$ is no simple task. It is not an idea that can be defined in purely probabilistic terms, because it says something about the mechanisms underlying the system we are studying, and what will happen if we interfere with that system in some way.<br />
<br />
Consider the example given at the end of the <a href="http://itsastatlife.blogspot.co.uk/2013/11/if-correlation-isnt-causation-then-what.html">last post</a>. The <a href="http://www.dailymail.co.uk/health/article-2427724/How-short-nap-raise-risk-diabetes-high-blood-pressure-high-cholesterol.html">headline</a> was:<br />
<blockquote class="tr_bq">
<span style="font-size: large;">How a short nap can raise the risk of diabetes</span></blockquote>
The implication of this is that the risk of diabetes increases <i>because of</i> the nap. But what does this mean? <br />
<a name='more'></a><br />
Let's start with a more clear cut example. Suppose I drop a glass bottle onto a concrete floor and it breaks: we might say that the bottle broke <i>because</i> I dropped it. In doing so we pack in a couple of ideas:<br />
<br />
<ol>
<li>I dropped the bottle, and it broke; </li>
<li>if I hadn't dropped the bottle, then it wouldn't have broken.</li>
</ol>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNO0-18uv__HUe2EbToWt-o_6NCpUu5gwldGD10Qi4m8JMcZD8TNGYVZoObv04SQxdi8-ztkGbo1tXWugk7w6VDTk5GdS8YytE6bd-yH298ZWkdp54I-2LZdgqvjzafxajR9u_q1qysM4/s1600/494584099_507ff665ed.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Oh cruel, cruel world" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNO0-18uv__HUe2EbToWt-o_6NCpUu5gwldGD10Qi4m8JMcZD8TNGYVZoObv04SQxdi8-ztkGbo1tXWugk7w6VDTk5GdS8YytE6bd-yH298ZWkdp54I-2LZdgqvjzafxajR9u_q1qysM4/s1600/494584099_507ff665ed.jpg" /></a></td></tr>
<tr align="right"><td class="tr-caption"><span style="font-size: xx-small;"><span style="font-size: x-small;">Oh cruel, cruel world.</span> (Photo by <a href="http://www.flickr.com/photos/hoanglongvn/">LongPHAM</a>, Creative Commons)</span></td></tr>
</tbody></table>
The only difference between the two scenarios is whether or not I dropped the bottle, and yet in one the bottle breaks and in the other it doesn't. Therefore the action of dropping the bottle directly caused it to break.<br />
<br />
<a href="http://en.wikipedia.org/wiki/David_Hume#Causation">Unless you're a philosopher</a> all this seems fairly uncontroversial, because we have a very good understanding of the actual mechanism by which the bottle smashes: gravity pulls the bottle down so that it hits the floor, and in doing so the bottle's kinetic energy is transferred to the brittle glass. We can be pretty confident when we say that <i>if I hadn't dropped the bottle</i>, it wouldn't have broken, even though this is an event we did not observe, because we can model the physics and have a lot of experience to show that bottles are pretty stable when left to their own devices. If they weren't then the Jesus College wine cellar would be an awful mess.<br />
<br />
Now a trickier example. Suppose that at the end of term I get a cold; I might say that "it's because I've been working hard and not sleeping enough, my immune system is a bit weaker". If we follow the previous example, then I must mean that "If I hadn't been working so hard, then I wouldn't have caught a cold." This is far from clear: people sometimes get colds even if they haven't been working hard. <br />
<br />
It might well be the case that working hard and not sleeping enough makes it <i>more likely</i> that one will get a cold (let's assume for the sake of argument that it does). But it's also possible that I would still have become ill even if I'd taken it easier: perhaps I caught it from my housemate, so less work means more time at home potentially exposed to the virus. On the other hand, I might have caught the cold from a student during a particular meeting, so if I'd had a different meeting at that time I might have worked just as hard and not got a cold. It therefore seems much less clear what we mean by the idea that me working hard and not sleeping has <i>caused</i> me to become ill.<br />
<br />
<h4>
Probabilistic Ideas of Causality</h4>
The important point in the previous example is the idea that working hard makes it more likely that one would get a cold. We could base a notion of causality on this: say that $A$<b> is a cause of </b>$B$ if the probability of $B$ occurring is greater when $A$ happens than when $A$ doesn't happen. <br />
<br />
In the case of the bottle, dropping it increased the probability of it breaking from essentially zero (no chance of breaking) to essentially one (will definitely break). In this sense it is an extreme example. On the other hand, working hard might increase my chances of getting a cold, but it probably only by a very small amount (an increase from 0.1 to 0.15 would be a very high estimate in my view). In this case then, there's still a reasonable chance I would have caught a cold anyway, so it seems a very strong statement to have said that I became ill <i>because</i> I was working hard.<br />
<br />
In a large group of people though, the idea becomes clearer. Suppose I take 1000 people, and give them all a relaxing couple of months, with a sensible workload. I might expect approximately 100 of them to get colds; on the other hand, if I make them all work very hard for those two months, my estimate suggests that around 150 will get colds. This notion of causality says that more people who work hard will get colds than if those same people took it easy. <br />
<br />
Applied to the Daily Mail's headline, the implication is that if we stopped people from taking naps, then fewer of them would get diabetes. This is a much stronger statement than simply noting the association between the two. If, as seems rather more plausible, having an underlying illness makes you more likely to feel tired and therefore to nap, then this acting to stop people napping would not have the desired effect at all. They'd just be more tired, and not have any impact whatever on their chances of developing diabetes.<br />
<br />
The notion common to most ideas of probabilistic causality is that the likelihood of an effect, $Y$, is changed when I <i>intervene</i> to change the cause, $X$. By just passively observing, we might see some association between $X$ and $Y$; but if $X$<i> causes </i>$Y$, then when I perform an action which somehow changes the value of $X$, it will also change the probability of $Y$ occurring. <br />
<br />
Smoking <i>causes</i> lung cancer, in the sense that if everyone stopped smoking we would see less lung cancer. On the other hand, if I gave everyone a vaccine which stopped people from developing lung cancer, it wouldn't stop people from smoking (perhaps the reverse, in fact!).<br /><br />
<h4>Potential Outcomes</h4>
The above ideas all refer to probabilities and chances which occur at the population level: I can't say whether or not my specific cold is actually caused by working hard, only that more people have colds when the work hard in a general sense. So can we define a notion of causality at an individual level?<br />
<br />
Imagine that two worlds 'exist': the world we observed, in which I worked hard and then got a cold, and a second<b> counterfactual </b>world which was essentially the same, except that I took it easy. Let $Y_{\text{hard}}$ and $Y_{\text{easy}}$ be variables which denote whether or not I get sick under each scenario (1 for ill, 0 for not ill): we know $Y_{\text{hard}} = 1$ because I actually did work hard, and I did get a cold. On the other hand we don't know anything about $Y_{\text{easy}}$: this is an outcome associated with the counterfactual world in which I took it easy, so we can't observe it.<br />
<br />
We can consider the pair of values $(Y_{\text{hard}}, Y_{\text{easy}})$ as <b>potential outcomes</b> for an individual, just by assuming that the value of $Y_{\text{easy}}$ <i>exists</i> and is well defined (even though it is not observable, even in principle. We can use the values of this pair of numbers to divide the population into groups of people based on how they individually respond in the two different worlds. This idea (due to Donald Rubin) is both powerful and controversial, and deserves a future post of its own; I will say no more now for the sake of brevity, other than that the idea is mathematically very useful, whether you find it completely natural or philosophically challenging.<br />
<br />
<h4>Testing Causal Questions</h4>
Now that we've defined some ideas of probabilistic causality, the real question is this: how can we answer causal questions? As I've already suggested, the 'gold standard' method is to do a randomised trial. We take a large group of people, and randomly divide them into two groups. The first group are all treated with $X=1$ and the second with $X=0$, but otherwise they are treated the same; if the number of people with $Y=1$ differs between the two groups (in a statistically and scientifically significant way), then there is a causal effect of $X$ on $Y$ in the sense defined above.<br />
<br />
If we <i>can't</i> do this for some reason, whether practical or ethical, then life is rather harder. Next week I will discuss one popular approach to this, known as <b>instrumental variables</b>.
<br />
Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com2tag:blogger.com,1999:blog-806057380317734287.post-71894170981451544832013-11-08T10:46:00.001+00:002013-11-08T16:34:27.874+00:00If correlation isn't causation, then what is?I've started to get a little tired of writing entirely about media shenanigans, and in all likelihood so, dear readers, have you tired of reading about them. So today I'm going to provide one of the 'educational pieces' alluded to in this blog's description; specifically I'm going to start talking about <i>causal inference</i>, which is the driving force behind the research I'm lucky enough to be paid to do.<br />
<br />
We hear a lot in statistics classes (if you're inclined to go to them) and places like this blog, that "<a href="http://xkcd.com/552/">correlation is not causation</a>." Packaged within this pithy but stern warning is some very sound advice: just because you observe a relationship between two things doesn't tell you anything about the mechanism (if any) which creates that relationship.
<br />
<a name='more'></a><br />
For example, you might not be surprised to learn that Guardian readers are, on average, <a href="http://www.guardian.co.uk/advertising/demographic-profile-of-guardian-readers">better educated</a> than the general UK population. You are unlikely to conclude from this that reading the Guardian makes you better educated, because this is absurd (unless you have a rather narrow view of what 'educated' means). Perhaps instead you imagine that better educated people are (generally) more interested in the things the Guardian has to say, or perhaps that people from a middle-class background are both more likely to be well educated and to read the Guardian. <br />
<br />
We often perform this sort of inference unconsciously; in the real world we are confronted with patterns and associations all the time and, usually being unable to experiment, we must infer the mechanism just from observing. In some respects we are remarkably good at this, but this is often because we have learned a lot about the world around us through experience.<br />
<br />
It can also trip us up: for centuries in the Western world it was believed that bad smells caused disease (think of malaria's etymology). This does not seem unreasonable, since there is a strong association between foul smelling material and illness, but the conclusion is quite incorrect. In particular, it suggests that the appropriate remedy is to cover up the bad smell with a good one, rather than washing one's hands more often. It wasn't until the late 19th century when Louis Pasteur and Robert Koch showed that the bad smell and the diseases were <i>both</i> caused by bacteria, that our modern understanding developed.<br />
<br />
To help us understand both the importance and the difficulty of causal inference, I will present a little piece on the history of understanding the relationship between smoking and lung cancer.<br />
<br />
<h4>
Fisher's Cancer Gene</h4>
<br />
Sir Ronald Aylmer <a href="http://en.wikipedia.org/wiki/Ronald_Fisher">Fisher</a> was a highly distinguished and brilliant statistician, and is responsible for the foundation of many modern statistical methods; if you've studied statistics you'll already know this, because he has <a href="http://en.wikipedia.org/wiki/Fisher_information">quite a few</a> <a href="http://en.wikipedia.org/wiki/Fisher%27s_principle">important things</a> <a href="http://en.wikipedia.org/wiki/Fisher%27s_exact_test">named after him</a>. He was also a difficult and stubborn man who entertained feuds with <a href="http://en.wikipedia.org/wiki/Jerzy_Neyman">other statisticians</a>, and who rejected one of the greatest triumphs of modern epidemiology: proving that smoking causes lung cancer.<br />
<br />
Fisher was himself a heavy smoker, and this may have influenced his views. By the 1950s, large observational data sets were available to show that people who smoked had a much higher risk of developing lung cancer. Given what we now know, and the seemingly obvious potential problems caused by putting foreign substances into the body, it seems difficult to understand how such seemingly strong evidence could be ignored (on the other hand, people used to sell <a href="http://en.wikipedia.org/wiki/Doramad_Radioactive_Toothpaste">radioactive toothpaste</a>).<br />
<br />
We will represent the idea that some quantity $X$ influences another $Y$ in a causal way using an arrow: $X \rightarrow Y$ (I will try to define this more precisely in another post, though doing so is not straightforward: just ask a philosopher). The modern consensus view, then, is represented by this diagram:<br />
<br />
$$\text{Smoking} \longrightarrow \text{Cancer}$$<br />
However, Fisher argued that it was not possible to rule out other explanations for the <i>association</i> between smoking and lung cancer. Was it not possible, for example, that people who are already in the process of developing lung cancer (though unaware of it) might be more likely to use cigarettes as a palliative treatment, in order to relieve some related irritation of the lungs? In such a case the proper conclusion would be that the cancer causes the smoking.<br />
$$\text{Smoking} \longleftarrow \text{Cancer}$$<br />
Or perhaps there is some other factor which we have not considered, and which both makes smoking and lung cancer more likely. Social factors are quite plausible in this context: perhaps people who live in the city are both more likely to smoke and also to inhale factory soot. However, such hypotheses can be checked by gathering additional data. Fisher himself proposed a genetic explanation: perhaps there is a gene which provides its carriers with a strong craving to smoke, and which also increases their risk of developing cancer. That is to say:<br />
$$\text{Smoking} \longleftarrow \text{Gene} \longrightarrow \text{Cancer}$$<br />
The key difference between the three hypotheses is this: if we forced or convinced everyone to stop smoking, in the first example it would reduce the number of deaths from lung cancer, whereas in the other two it would make no difference whatever because the cause is something else. (This gives a hint of how one might try to define causality more formally.)<br />
<br />
Logically Fisher is correct, an observed association can be explained in any of these ways. In order to be certain that smoking causes lung cancer, we would need to do a randomised controlled trial: simply take 1000 people, divide them into two groups at random. Force one group to do 40-a-day, ban the rest from smoking at all, and wait 20 years to see what happens. Since any other mechanism which might cause participants to smoke (or not) has been broken, any remaining association <i>must</i> be causal. Sadly though, this experiment didn't make it past the ethics committee.<br />
<br />
So how was Fisher proved wrong? Ultimately it came down to the sheer weight of evidence: Fisher's genetic explanation is highly implausible, because it would require two genetic effects (one on cancer and one on smoking) to be absurdly strong; genetic effects can also be controlled for (to some extent) by studying people who are related. The ethics committees of the past <i>did</i> allow animal experiments, which demonstrated that smoking in (for example) beagles certainly does cause lung cancer. And external circumstances may be used as a proxy for randomised trials: a higher cigarette tax reduces the amount by which people smoke, but shouldn't have any other effect on lung cancer (this is called an 'instrument', beloved of economists in search of causal conclusions).<br />
<br />
<h4>
The last great hope for causality</h4>
<br />
Causal inference is difficult, but it is not impossible (even without randomised trials). It is certainly try that any causal conclusions drawn from observational studies should be treated with great caution (here's a <a href="http://www.dailymail.co.uk/health/article-2427724/How-short-nap-raise-risk-diabetes-high-blood-pressure-high-cholesterol.html">particularly terrible example</a> from our old friend the DM, I couldn't resist). But the defeatist (or rather obstructive) attitude of Fisher is not the answer. Causal inference is a fast developing field which shows that it <i>is</i> possible to obtain evidence of causal effects even without ideal study designs, and there is a huge potential benefit for many fields in discovering how.<br />
<br />
I hope to share some of those methods with you in future posts.<br />
<br />
<br />
<br />
<b>Reference </b><br />
<br />
Stolley, P.D<span style="font-family: inherit;">. - <a href="http://aje.oxfordjournals.org/content/133/5/416">When Genius Errs: R. A. Fisher and the Lung Cancer Controversy</a>, <i>Am. J. Epidemiol.</i> 133 (5): 416-425, 1991.</span><br />
<br />
<span style="font-family: inherit;"><cite><span class="slug-pages"> </span>
</cite>
</span>Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-33564586049022565912013-09-13T12:54:00.001+01:002013-09-13T12:55:50.559+01:00The Monty Hall ProblemIt's great to see pieces on familiar mathematics puzzles in the mass media, so I was pleased to see <a href="http://www.bbc.co.uk/news/magazine-24045598">this article</a> about the Monty Hall problem on the BBC News website. However, I'm moved to write a little piece about this, because I think it uses a rather carelessly analogy.<br />
<br />
A brief summary of the problem. You're in a game show, and there are three doors; behind one of the doors is a prize, behind the other two is nothing (or possibly worse, a goat). The procedure is as follows <i>every time the game is played</i>:<br />
<ol>
<li>you choose a door, say number 1 (but don't see what's behind it);</li>
<li>the game show host, <i>who knows where the prize is</i>, opens one of the other two doors, say number 2, and reveals nothing;</li>
<li>the host gives you the opportunity to either stick with your choice or door 1, or move to door 3;</li>
<li>your choice is revealed.</li>
</ol>
<br />
The question is - should you stick, or switch, or does it make no difference?<br />
<br />
The BBC video (with Marcus du Sautoy and Alan Davies) is very clear, and so is most of the article. However the beginning includes a reference to Deal or No Deal: the reason I don't like this is that in Deal or No Deal <i>the banker doesn't know where the money is</i>. As we will see, this point is absolutely critical to getting the right answer.<br />
<br />
<br />
<a name='more'></a><br />
I won't go into too much detail here (there are lots of ways to explain this, have a look at the article), but the answer is that you should switch: to see this, note that the probability you chose the prize door in step 1 is clearly 1/3; in step 2, at some level you learn nothing, because the same thing always happens (the host opens a door to reveal no prize). So the chance that you have the correct door is <b>still 1/3</b>, and therefore the chance that the remaining door has the prize must be 2/3. It's not easy to understand at first, but it's all to do with the inevitability of what happens in step 2.<br />
<br />
Suppose instead that the host <i>doesn't know</i> where the prize is, and just opens a door at random. There's a 1/3 chance you originally chose the prize, in which case the host will definitely reveal nothing. But there's a 2/3 chance you do not select the prize door, in which case there is a 50% chance the host will choose it instead. If he does this, one assumes that the game is over and you lose. <br />
<br />
Let's call the prize door A, and the other B and C, and write AB to mean that you choose door A, and the host chooses door B. If you and the host both work at random, there are 6 equally likely possibilities:<br />
<br />
AB, AC (should stick)<br />
BA, CA (already lost)<br />
BC, CB (should switch)<br />
<br />
There's a 1/3 chance you lose immediately, but <b>if</b> you get to step 3, then in half of the instances you should switch, and half you should stick. In other words, it doesn't make any difference to you chances of winning, it's 50-50.<br />
<br />
In the original scenario though, the host <i>never picks door A</i>, so BA and CA are not allowed. <br />
What happens in step 2 is completely inevitable, whereas if the host is as ignorant as you, there's a chance you'll learn exactly where the car is.<br />
Instead, if you choose door B, then the host will definitely pick C, so BC 'absorbs' all the probability from BA; similarly CB from CA. Now both BC and CB are twice as likely as each of AB and AC. <br />
<br />
A friend of mine found the following generalisation helpful for understanding the problem. Suppose there are 100 doors, and still only one prize. You pick a door, and the host then opens 98 of the remaining 99 doors to reveal nothing. Again, this outcome was inevitable, so there's still a 1% chance the door you chose hides the prize, and if you switch there's a 99% probability you find the prize.<br />
<br />
(Thanks to Tim Cannings for suggesting this, and Aeron Buchanan for the generalisation.)<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com2tag:blogger.com,1999:blog-806057380317734287.post-23447943270852435102013-06-05T09:00:00.000+01:002013-06-05T09:07:00.167+01:00Twins, or In which newspapers mess up odds calculations (again)<a href="http://www.bbc.co.uk/news/uk-22765462">This BBC piece</a> interviews a couple who've been 'blessed' with three sets of twins (rather you than me). The caption states that<br />
<blockquote class="tr_bq">
Doctors told them the chances of having three sets of twins was 500,000-1.</blockquote>
<div>
Doctors that don't know much about genetics, perhaps. <br />
<br />
<a name='more'></a>Our old friend the Daily Mail managed to imply that this happened for the first time in Britain last year:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGCmbnBvV7T439uQ33NphYnW9LtgyJJ-hXNf0_Pm_eV64Xwyi1wSOas769j-Ju8DhMFiM5HJPJV4ZJAsStfe2PY3QRDYAZk8z2LtF6Y6hCsOKLhjrmR_qKIfDbXp-RQIZVxFmjPYPlAPM/s1600/DM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGCmbnBvV7T439uQ33NphYnW9LtgyJJ-hXNf0_Pm_eV64Xwyi1wSOas769j-Ju8DhMFiM5HJPJV4ZJAsStfe2PY3QRDYAZk8z2LtF6Y6hCsOKLhjrmR_qKIfDbXp-RQIZVxFmjPYPlAPM/s1600/DM.png" /></a><br />
<br />
Obviously they don't read <a href="http://www.telegraph.co.uk/news/uknews/1359583/Mother-has-her-third-set-of-twins.html">the Telegraph</a>, which has an example from 2001. (The DM <a href="http://www.dailymail.co.uk/news/article-2211310/Mother-British-woman-birth-twins-THREE-times-shes-got-children-that.html">article has since been changed</a> but they forgot to change the title of the web-page, which is why it appears like this on Google; the correction is not acknowledged, naturally.) And if three sets of twins are unlucky, feel for <a href="http://www.bbc.co.uk/news/world-africa-12280109">this poor woman</a>, who has six!<br />
<br />
The Telegraph helpfully details the mathematics involved in their calculation. For natural births, in white European women, approximately 1 in 80 result in twins (or triplets, or more, if one is even luckier). Therefore if you take three births at random, there's about a one in $80 \times 80 \times 80 = 512,000$ chance they'll all be twins.<br />
<br />
Except, obviously, these births were not selected 'at random' because they're all from the same mother. Which is sort of the point. <br />
<br />
Like many things, there are genetic factors which predispose people to have twins, generally (as far as my research tells me) because the mother is prone to hyper-ovulate. So if you pick a mother who has <i>already</i> had twins, they're more likely to be someone prone to hyper-ovulation, and therefore it's more likely that they'll have a second set. If they've already had two sets, well you can be pretty darn sure they hyper-ovulate. <br />
<br />
Other factors are also important. If you're of Nigerian ancestry, then your chances of having twins in the first place are considerably higher. Older mothers are also more likely to have multiple births.<br />
<br />
I haven't been able to find anything terribly concrete quantifying the observed proportions of multiple sets of twins, but it seems to be about <a href="http://www.twinparents.com/articles/twin-pregnancy/twins-again-what-are-chances.html">four times as likely</a> that you'll have a second set of twins if you've already had one. This reduces our highly unlikely 1 in 500,000 to a mere 1 in $80 \times 20 \times 20 = 32,000$.<br />
<br />
Quite a lot of families are out there who have gone through three pregnancies. Third births within marriages and civil partnerships in England and Wales (outside marriage the order isn't recorded) represented <a href="http://www.ons.gov.uk/ons/publications/re-reference-tables.html?edition=tcm%3A77-230704">about 7.5%</a> of the 720,000 in 2010. That's about 50,000 or so per year, so we can expect one of these stories to come around about every... oh nine months or so?<br />
<br />
Actually that's probably not true, 'cos if people with two sets of twins already have any sense they'll start being a little bit more careful... <br />
<br /></div>
Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com1tag:blogger.com,1999:blog-806057380317734287.post-32760115059033160812013-04-19T12:47:00.002+01:002013-04-19T12:47:41.577+01:00Extreme ComparisonsThis is just a fairly short note on what has been covered in other cases elsewhere. The <a href="http://www.bbc.co.uk/news/health-22214065">BBC reported today</a> that Tameside in Greater Manchester is the "UK's heart disease capital". This is on the basis that the rate of deaths from heart disease between 2009 and 2011 was higher than anywhere else, at 132 per 100,000 people. The data come from the British Heart Foundation, and I couldn't immediately see how to get at it.<br />
<br />
The article mades a particular point of noting that this rate is three times higher than that of Kensington and Chelsea. This sounds very dramatic, but it really isn't, so don't go moving just yet (even if you can afford a house in Kensington).<br />
<br />
<a name='more'></a><br /><br />
The problem with this approach is pretty simple. If you take two different groups of people, even if the people in these groups are actually essentially interchangeable, different things will happen to them just by chance. For example, some heavy smokers live to be 90, just because they're lucky (but <a href="http://understandinguncertainty.org/node/58">not many</a>).<br />
<br />
If you take a large collection of different groups (all the UK's <a href="http://en.wikipedia.org/wiki/Local_government_in_the_United_Kingdom">local authorities</a>, in this case) this effect becomes quite strong. Let's suppose that people in those groups are essentially the same, say some statistically literate but morally dubious government has decided to randomly assign people to live in particular areas. Just by chance, some groups will have more cases than others.<br />
<br />
I tried this with a little simulation - take 300 large-ish equally sized groups with an average of 150 deaths each; by chance the number of deaths in each group would typically range between 116 and 187, and variations as large as between 100 in the lowest group and 200 in the highest wouldn't be too surprising. The effect becomes more dramatic if some of the groups are much smaller than others, as is the case with local authorities.<br />
<br />
Exactly the same phenomenon is <a href="http://understandinguncertainty.org/three-fold-variation-uk-bowel-cancer-death-rates">seen here</a> for bowel cancer rates. It's explained very well with a funnel plot.<br />
<br />
Of course the groups of people in different local authorities <i>aren't</i> the same, even on average. They'll have different age profiles, for example, different lifestyles, income levels, access to healthcare and so on. Some of these should be controlled for (depending on the point you're trying to make), some perhaps shouldn't. But comparing extremes as the BBC (egged on by the BHF) does tells us very little indeed.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-37650834104992453262013-03-22T14:27:00.000+00:002013-03-22T14:27:09.512+00:00Smoking rates: even the 'good guys' shouldn't be trustedThe Guardian reports today that the number of children under 16 (more precisely aged 11-15) taking up smoking has has <a href="http://www.guardian.co.uk/society/2013/mar/22/number-children-smoking-rises-year">risen by 50,000 in a single year</a>. This is a slightly irritating headline simply because the number is without context, but the sub-headline is more helpful: from 157,000 to 207,000, which is quite dramatic: a 32% increase.<br />
<br />
First obvious question - is this statistically significant? Well, let's have a look at the research quoted by the Guardian, which comes from <a href="http://www.cancerresearchuk.org/cancer-info/cancerstats/types/lung/smoking/#children">Cancer Research UK</a>. Their Figure 6.10 immediately arouses suspicion. There does appear to be an uptick in the figures from 2010 to 2011, but they remain below the 2009 figure! The overall trend in the numbers over the past 10 years is clearly downwards. So either we believe that the number of children taking up smoking fell and then rose dramatically in consecutive years, or else we might just be witnessing a bit of noise in our data. <br />
<br />
<a name='more'></a><br />
<h4>
Surveys</h4>
<br />
So first, what exactly is the number actually measuring? Well it's the number of students who become regular or occasional smokers from one year to the next. This is measured by comparing the number of such regular or occasional smokers (for example) aged 12 in 2010, with the number aged 13 in 2011.<br />
<br />
But the researchers cannot, in practice, go around asking every 11-15 year old whether or not they smoke. Instead they commission a survey, or find the results of an existing survey, in this case the snappily titled <i>Smoking, drinking and drug use among young people in England in 2011</i>, carried out by a branch of the NHS. You can see it all <a href="http://www.ic.nhs.uk/catalogue/PUB06921">here</a>.<br />
<br />
The survey gives an estimate of the <i>proportion</i> of children smoking in England, which the statisticians at Cancer Research UK multiplied by the ONS's mid-year population estimates <i>for the UK</i> to arrive at a total number. This is, in my view, a mistake, because we end up combining the statistical error in the original survey with the error in the ONS's population estimates (though this should be fairly small). Also: why multiply population estimates for the UK by proportions for just England? Simple: the UK has more people, so the numbers sound bigger that way.<br />
<br />
The original research says that the proportion of regular smokers amongst 11-15 year-olds in the past 5 years is 6%, 6%, 6%, 5% and 5%. Their Figure 3.2 backs this up. All in all, not very dramatic. No mention is made in the original report of any increase in smoking rates; in fact rates amongst school children have fallen dramatically in the past 20 years, with the number of regular smokers having been around 10% for boys and 15% for girls in the mid 1990s.<br />
<br />
The problem is that it's in Cancer Research UK's interest to make this all sound as bad as possible so that ever more action is warranted by the authorities. In fact they seem to have gone to quite a lot of trouble to construct a figure which will get them a headline.<br />
<br />
Their motives may be laudable, but they have an agenda just like everyone else, and it does not necessarily involve being totally honest. So keep your skeptical hats on at all times!<br />
<br />
<h4>
<b>Appendix on Batch Effects</b></h4>
<br />
Here we cover some detail about the surveying method. <br />
<br />
The survey covered 6,519 school pupils, but clustered into 219 schools in England. Ideally the authors would have written down a list of every 11-15 year-old in the country, and then picked several thousand of them from the list at random, and then given the lucky selection the survey. But such a list isn't readily available in the UK, and it's very difficult (and probably illegal) to create one.<br />
<br />
Much easier is to make a list of every high school in the country, and pick a few hundred at random. Then you can go to each school, and either survey every student, or use the school's list of pupils to sample some of them at random. This is much simpler, and if done correctly should (very approximately) mean that every student in the country is still equally likely to be selected for the survey.<br />
<br />
This means that the proportion of students in the survey who say they smoke should be an <i>unbiased</i> estimate of the proportion of <i>all</i> students who would answer the same way. Of course, if I ran the survey again and asked a different 6,519 students, i would get slightly different answers by change; but because the design is unbiased, I should get the 'correct' answer <i>on average</i>.<br />
<br />
There's a disadvantage to this method though, which is that 6,519 students from 219 schools doesn't give you as much information as 6,519 students picked randomly from a list of all students. This is because students in the same schools are more likely to be similar to one another; they have similar backgrounds and come from the same area, they're all from the inner city or rural Cumbria, for example. This within-school correlation (a <i>batch effect</i>) can be estimated from the data and corrected for, which the authors of the original research do. More info in <a href="http://www.guardian.co.uk/science/2006/apr/01/badscience.uknews">this article</a>, on a similar example.<br />
<br />
No attempt was made by the Guardian's article (or by CRUK as far as I can tell) to assess whether the difference they observe might be due to sampling variation.<br />
<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com1tag:blogger.com,1999:blog-806057380317734287.post-38737083758341446062012-10-29T17:12:00.002+00:002012-10-29T17:39:04.971+00:00Independence, independence, independenceMultiplying probabilities without any thought is a dangerous game. Just look at <a href="http://www.rollingstone.com/politics/news/global-warmings-terrifying-new-math-20120719">this piece</a> in Rolling Stone, referenced in <a href="http://www.guardian.co.uk/commentisfree/2012/oct/28/tory-retreat-on-climate-change-senseless?CMP=twt_fd">this Guardian article</a>.<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
...warmest May on record for the Northern Hemisphere – the 327th consecutive month in which the temperature of the entire globe exceeded the 20th-century average, the odds of which occurring by simple chance were $3.7 \times 10^{-99}$, a number considerably larger than the number of stars in the universe.</blockquote>
</blockquote>
<br />
<a name='more'></a>How does this number arise? The chance of a random variable exceeding its median is one-half (that's the definition of a median). If we treat each month as some independent random variables, then the chance of 327 of then all exceeding their medians is $0.5 \times 0.5 \times\ldots $(327 times)$\ldots\times 0.5 = 0.5^{327}$, which is indeed $3.7 \times 10^{-99}$; very very unlikely.<br />
<br />
Of course, we don't <i>know</i> what the median of these random variables is, we simply use a historical average (from the 20th Century) to estimate it.<br />
<br />
Since the probability we got was astronomically small, we deduce that one of the assumptions we used to calculate it was false; in Rolling Stone and the Guardian's case, they assume that medians were not correctly estimated, because the world has got warmer over time. This seems likely to be true, given all the scientific evidence we have for climate change.<br />
<br />
But in reality the independence assumption is certainly false <i>as well</i>. The temperatures in consecutive months will certainly be correlated with one another; for example, my limited understanding of <a href="http://en.wikipedia.org/wiki/El_Ni%C3%B1o%E2%80%93Southern_Oscillation">El Nino</a> is that it can lead to warmer weather. So, in fact, the probability calculated above is completely meaningless.<br />
<br />
Suppose instead, for example, that conditional on January being warmer than average, the chance that February is also warmer than average is $0.75$, and the same for other consecutive months. The probability becomes $0.75^{327}$, which is a mere $5.9 \times 10^{-42}$. This is still astronomically unlikely of course, but the difference in magnitude between the two numbers is more astronomical, which suggests that it wasn't sensible to present the probability in the first place.<br />
<br />
It's tempting to try and turn the very complicated but compelling evidence for something like climate change and turn it into a single number or graph in order to communicate a message. That's a very worthy aim, but make sure the number you choose is even vaguely correct.<br />
<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-77567133055798828742012-10-16T08:20:00.002+01:002012-10-16T08:21:09.096+01:00Horse Racing and CoincidenceJust penned this to the BBC about why their <a href="http://www.bbc.co.uk/sport/0/horse-racing/19954827">piece on a jockey winning seven races</a> in a single day quotes the wrong odds. <br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
In your article on Richard Hughes winning seven races in a single day, you quote the odds of this event as being 10,168-1. Whilst undoubtedly a fantastic achievement, these odds are incorrect, since they they ignore the fact that Mr Hughes raced in eight races that day. The chances of him winning 7 out of 8 races (or more) is about 1,257-1, which is a bit more modest. In particular, it seems fairly unlikely that someone would place a bet on the rider winning these particular seven races, and not the eighth.<br />
<a name='more'></a></blockquote>
<blockquote class="tr_bq">
If the piece's author or anyone else want to talk more about how these odds are calculated, or why this sort of thing is important, I'd be very happy to chat about it.</blockquote>
<blockquote class="tr_bq">
Best wishes,</blockquote>
<blockquote class="tr_bq">
Robin Evans</blockquote>
</blockquote>
I think this is the right approach to take with these mistakes, and I hope it doesn't come across as condescending. With a bit of luck they'll acknowledge the mistake and get in contact.<br />
<br />
While we're on the subject, calculating multiple odds can seem tricky at first. The quickest way to do it is this: if the odds an event are 'a to b' (usually written a-b or a/b), then the probability of the event is b/(a + b). For example, 3/2 means 2/(3 + 2) = 0.4, so we'd expect 40% of events with these odds to actually occur (or slightly less if the bookies are taking a cut!).<br />
<br />
To work out the chances of several events happening, and assuming these are independent (reasonable in this case, since the odds would be updated after taking into account what had happened in previous races), we multiply the probabilities. In this case the seven races had odds of 13/8, 5/2, 7/1, 4/1, 5/2, 7/4 and 15/8, and multiplying these gives 0.000098, which as the BBC say is about 10,168/1. <br />
<br />
However there was an eighth race, in which Hughes was the 2/1 favourite, and in which he 'only' came third. So we need to consider the probabilities of him winning seven races, <i>and</i> failing to win the eighth race; of course there are eight ways in which he could do this, because we would have found it equally amazing if he'd won any of the seven races, and we throw in the chance of him winning all eight, since that would be seriously impressive.<br />
<br />
The sum of all these possibilities comes to about 1257/1 (R code working below). This is still impressive, but an order of magnitude different. There's a lot of race meetings every year, so I'm surprised that this event hasn't happened since 1996 - perhaps jockeys don't often race eight horses in one day. Or maybe their the odds on their horses are usually a bit longer.<br />
<br />
There are <a href="http://www.parentdish.co.uk/2010/10/14/couples-three-children-share-the-same-birthday/">plenty</a> of <a href="http://www.dailymail.co.uk/news/article-1247872/Woman-finds-double-egg-yolks-box-beating-trillion-odds.html">other</a>, <a href="http://en.wikipedia.org/wiki/Sally_Clark">examples</a> where people straightforwardly fail to calculate odds correctly, making something seem strange, spooky or suspicious, when it's merely mundane or tragic. It would be great if the BBC could avoid contributing to this malaise!<br />
<br />
<h4>
R Code</h4>
<br />
<span style="font-family: Courier New, Courier, monospace;">p = 1/(c(13/8, 5/2, 7/1, 4/1, 5/2, 7/4, 15/8, 2/1)+1)</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">out = prod(p)</span><br />
<span style="font-family: Courier New, Courier, monospace;">for (i in 1:8) out = out + prod(p[-i])*(1-p[i])</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">out</span>Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com1tag:blogger.com,1999:blog-806057380317734287.post-25025979802041187882012-10-11T15:22:00.002+01:002012-10-11T15:22:58.714+01:00Fruit, Vegetables, Health and HappinessWhilst perusing the Daily Mail today (my excuse is that it's next to the espresso machine) I saw this interesting <a href="http://www.dailymail.co.uk/health/article-2215737/Forget-day-Now-scientists-say-youll-healthier-happier-eating-SEVEN-daily-portions-fruit-veg.html">health advice</a>:<br />
<blockquote class="tr_bq" style="font-family: inherit;">
Forget five a day: Now scientists say you'll be healthier and happier eating seven daily portions of fruit and veg.
</blockquote>
<br />
The phrase "scientists say" is always a red light. <br />
<a name='more'></a>The article says that there's been a study which shows that people who eat more fruit and vegetables are happier than everyone else. It runs through all the classically bad newspaper science reporting clichés. The old wisdom has been turned on its head, the government has been wrong all these years, those crazy scientists have gone and done it again! For example, it says:<br />
<blockquote class="tr_bq">
But now scientists claim if we upped it to seven, we’d also be far happier.</blockquote>
There is a clear <i>causal </i>implication: <i>if </i>we change our diet to increase our intake of fruit and veg, we will <i>become</i> happier.<br />
<br />
Here's a preprint of <a href="http://www.andrewoswald.com/docs/October2FruitAndVeg2012BlanchOswaldStewartBrown.pdf">the paper referred</a>, titled <i>Is Psychological Well-being Linked to the Consumption of Fruit and Vegetables? </i>The abstract contains the important sentence which is omitted from the newspaper article:<br />
<blockquote class="tr_bq">
Reverse causality and problems of confounding remain possible.</blockquote>
They remain very possible indeed! All the paper does is look at some associational studies, where they ask people about their fruit and veg intake and their happiness. No randomised studies were carried out<br />
<br />
It's well documented that poorer people eat less fruit and fewer vegetables, and it also seems pretty likely that your income and wealth have a substantial effect on your happiness, however measured. Thus income is potentially a very strong confounder. Needless to say the authors do try controlling for variables like this (and smoking status, BMI, etc), and still find a correlation.<br />
<br />
Just as important is the problem of reverse causation: if I'm feeling bad about myself, do I reach for an apple? Well in my case, it's more likely to be the chocolate or the Pralines and Cream. So it may be that happiness has an effect on fruit and vegetable intake.<br />
<br />
Another difficulty is that these studies rely upon people remembering how much fruit and veg they've been eating, and their mood at the time of asking might affect their memory.<br />
<br />
In addition, the effect size is not that large. Taking all my estimates from Table 1 in the paper (based on Scottish data), people who eat 8 or more portions per day have a mean life satisfaction 0.27 higher than those who eat none at all; this is measured on a scale from 0 to 10. So it certainly isn't correct to say that scientists claim you'll be <i>far</i> happier if you eat a few more grapes.<br />
<br />
There is quite a lot of variation in the estimates, so for example eating 5-6 portions appears to result in a higher increase over eating none (0.23), than does eating 6-7 (0.17). It seems to me that a better idea would be not to treat the intake as a discrete variable taking lying in these different categories, but as something continuous and fit a single regression line; this would avoid these problems and allow the whole sample to do the work.<br />
<br />
Lastly the DM question whether the health advice given by the government should be changed on the basis of this paper. It mentions that various other countries suggest a higher intake of fruit and veg, though they omit to mention that the definition of 'fruit and veg' and a 'portion' are different in each case. <br />
<br />
Needless to say nothing should (or will) be changed because of this one paper, but it's worth pointing out that health advice is partly based upon its actual effect on the population - if the Ministry of Health started saying that you should eat ten portions of fruit a day, people might dismiss the advice as unrealistic, and ignore it altogether. There are ethical questions about how untruthful one can be in this case, but the presentation of such advice is certainly important.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com1tag:blogger.com,1999:blog-806057380317734287.post-80359576060823864372012-10-02T11:01:00.000+01:002012-10-02T11:01:12.718+01:00Trust your instinctsAs Obi Wan almost said: "Let go your conscious self and act on instinct. Your news articles, government agencies and other organisations can deceive you, don't trust them."<br />
<br />
My brother Andrew (<a href="https://twitter.com/andysstudy">@andysstudy</a>) has been studiously following this advice since 1977, and sent me this tweet:<br />
<blockquote class="tr_bq">
See below. There are 73m children (u15) in EU I find it hard to believe 1:73 goes missing every year? http://www.euronews.com/2012/10/01/has-anyone-seen-my-child/</blockquote>
<a name='more'></a>The link is to a Euronews article about the problem of children disappearing in the EU. The killer statistical sentences in the article are these:<br />
<blockquote class="tr_bq">
An estimated one million children go missing every year in the European Union. These include runaways, criminal abductions, those abducted by a parent, the lost or injured, as well as missing unaccompanied migrant children.</blockquote>
As Andrew says, this seems like an extraordinarily large number. In a typical large high school like the one I went to, it would mean about 4 of the 300 students in each year group going missing annually.<br />
<br />
So where does the number come from? Fortunately not much detective work was required on my part: <a href="http://ec.europa.eu/justice/fundamental-rights/files/conference_missing_children_final_report_en.doc.pdf">this document</a>, a summary report of a European conference on missing children from May 2012 states, as one of its conclusions, that<br />
<blockquote class="tr_bq">
5. The Commission will use a working figure of 1 per cent of children in the EU (one million children per year) who go missing, pending more reliable data becoming available.</blockquote>
In other words, the number is made up. It's really that simple - in the absence of any data on the subject, they created a 'working figure' instead. So the Euronews' sentence should be "It is guessed that one million children go missing...". <br />
<br />
Children disappearing, whether runaways, abductees, or victims of violence undoubtedly represent a terrible problem afflicting our societies, and any number would represent a great deal of untold misery for those children, their parents and families. Which, in my view, makes it ever more critical that such information is factually correct. I can only assume that the organisers of the conference have concluded that a headline grabbing number is important enough to justify making one up.<br />
<br />
There is a seductive quality to choosing a small (and therefore plausibly conservative sounding) fraction, 1 percent, applying it to a very large number, 100 million, and getting a still large number, which can be used to shock people into action.<br />
<br />
A similar example comes from Norman Myers' 1979 book, <i>The Sinking Ark</i>:<br />
<br />
<blockquote class="tr_bq">
Let us suppose that, as a consequence of this man-handling of natural environments, the final one-quarter of this century witnesses the elimination of l million species--a far from unlikely prospect. This would work out, during the course of 25 years, at an average extinction rate of 40,000 species per year, or rather over l00 species per day.</blockquote>
So, again, one apparently plausible figure is invented to justify a shocking one: 100 species going extinct per day. Species extinction is a very real problem, but this sort of approach doesn't do anyone any credit, and only provides ammunition to those who spout guff about scientists (such as Prof. Myers) being involved in some sort of conspiracy to exaggerate environmental problems in order to get more grant money.<br />
<br />
We deserve better.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-25787387699647945462012-09-13T23:42:00.003+01:002012-09-13T23:53:35.460+01:00Road safety - how not to reason from dataMy apologies for going quiet over the summer. I've been in China, and they're not too big on blogs there (blogger is blocked).<br />
<br />
Just a short post here, on something fairly obvious. Here's an extract from post number 51 by Cambridge News' <a href="http://www.cambridge-news.co.uk/Home/Blogs/Perfect-cycling-weather.htm">Cycling Blog</a> on speed limits.
<br />
<blockquote class="tr_bq">
In Great Britian in 2011, 7 people were killed on a road with a 20 mph limit. 636 killed in a 30 mph limit. 289 people were seriously injured in a 20 mph limit, 13,168 in a 30 mph limit... Yes, these are large numbers. But it is the proportions that matter here... So that is proof then. Lower speed limits means fewer people killed.</blockquote>
[In case you're wondering, there's no irony in the last sentence. At all.] I assume you're all thinking the same as me by now.<br />
<a name='more'></a>
<ol>
<li>There are a lot more 30mph roads (by length) than 20mph ones.</li>
<li>30mph roads are often busier than 20mph roads, so there's more things to collide with each other.</li>
</ol>
<div>
After a cursory glance for official proof of the first point I came up empty-handed, but I imagine you'll believe me.</div>
<div>
<br /></div>
<div>
There are a lot of good reasons to think that 20mph roads are safer than 30mph ones, at least if the speed limits are observed, just from simple physics. A car travelling at 30mph has 2.25 times as much kinetic energy as one travelling at 20mph. As a driver reacts before braking, the car travels 50% further; as it starts to brake, this will happen more slowly the faster the car is going to start with. I am personally in favour of 20mph limits - they're a lot more pleasant to cycle on, because I can actually keep up with the traffic. They certainly feel safer, because drivers are less likely to try and overtake me inappropriately. But what's listed above isn't proof of anything.</div>
<div>
<br /></div>
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com3tag:blogger.com,1999:blog-806057380317734287.post-21880031905434500172012-07-20T09:38:00.001+01:002012-09-13T23:52:24.393+01:00Polygraphs and Sex OffendersJust a short post on the <a href="http://www.bbc.co.uk/news/uk-18916405">news story</a> that a pilot scheme which administers polygraph tests ('lie detector tests') to sex offenders on probation has been deemed successful, and the Ministry of Justice plans to roll it out nationwide.<br />
<br />
The <a href="http://www.justice.gov.uk/downloads/publications/research-and-analysis/moj-research/evaluation-of-mandatory-polygraph-pilot.pdf">MoJ's research</a> seems reasonable enough: the pilot scheme took place in the Midlands, and comparison groups of offenders were selected from other areas. The assignment was <i>not</i> randomised, which is unfortunately all too common, but the comparison group was similar to the treatment group on the basis of the most obvious covariates (age, original offence, risk or reoffending, criminal history, etc.). Based on a quick-ish reading of the paper it seems fairly solid.<br />
<a name='more'></a><br />
The main finding was that the treated group were far more likely to make <i>clinically significant disclosures</i> (CSDs) than the comparison group; CSDs include relevant information about recent activity, sex life, and sexual fantasies. The offenders tended to make these additional disclosures when taking the polygraph test, often just before the test. This suggests that the <i>perception</i> of being under scrutiny for deception caused the offenders to disclose; in particular the study design can give no indication of whether the polygraph tests actually detected deception in the offenders.<br />
<br />
My limited understanding of the research around polygraph testing (<strike>I couldn't find a good scientific review, but if anyone knows of one, let me know</strike> see below for a scientific review; here's an interesting <a href="http://www.tandfonline.com/doi/abs/10.1080/14789940412331337353">historical review</a> [£]) is that they are fairly effective at detecting deception, but that if the test taker has been trained to beat the test they become almost useless. My main concern (scientific, rather than ethical) about using testing is that it can lull operators into a false sense of security.<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
Some offender managers stated that ... when polygraph results showed <span style="background-color: white;">no deception, ... it reassured them of the offender’s honesty ...</span></blockquote>
</blockquote>
This is particularly worrying as the offender mangers (and indeed their fellow criminals) were wont to describe some sex offenders as 'devious', which suggests that they will be particularly likely to try to manipulate the test. Lastly I note that the study did not consider reoffending rates, which is presumably the outcome we would most like to see reduced.<br />
<br />
UPDATE: Tweeter <a href="http://twitter.com/hjnock">@hjnock</a> pointed me to <a href="http://www.bps.org.uk/sites/default/files/documents/polygraphic_deception_detection_-_a_review_of_the_current_scientific_status_and_fields_of_application.pdf">this review</a> by the British Psychological Society which gives a great summary of the evidence for relating to polygraphs. I think it justifies my fairly vague comment above!Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com2tag:blogger.com,1999:blog-806057380317734287.post-71246449488496040072012-07-18T11:24:00.000+01:002012-07-18T11:27:09.664+01:00Reading too much into one numberThe BBC points out that deaths from road accidents in the UK <a href="http://www.bbc.co.uk/news/uk-politics-18881049">increased last year</a> for the first time in 10 years. In total 1,901 people were killed during 2011, which is 51 more than in 2010. This is certainly not good.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXxLUq5q9ZcTwByv1-tsX2Aj7Odu5kxJ7crMHAiCcFpGyr_0OALkgF27Vid4ggKLXO_TIJbMtzoI_5lbvbx3gO2J1Hioo3ShRnp7rDR4hofHQSx2Eq6E5Rxf_Igu5Pzic2V5YnB_LF6gk/s1600/Screen+shot+2012-07-18+at+11.04.43.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXxLUq5q9ZcTwByv1-tsX2Aj7Odu5kxJ7crMHAiCcFpGyr_0OALkgF27Vid4ggKLXO_TIJbMtzoI_5lbvbx3gO2J1Hioo3ShRnp7rDR4hofHQSx2Eq6E5Rxf_Igu5Pzic2V5YnB_LF6gk/s400/Screen+shot+2012-07-18+at+11.04.43.png" width="400" /></a></div>
<br />
<br />
But is it surprising, in a statistical sense? <br />
<a name='more'></a>Well if the roads were just as safe (or just as dangerous) in 2011 as in 2010, we still wouldn't expect the number of deaths to be exactly the same in the two years: other factors such as the weather will have an effect. The <a href="http://www.publications.parliament.uk/pa/cm201213/cmselect/cmtran/506/50602.htm">Transport Select Committee's report</a> concludes by stating:<br />
<blockquote class="tr_bq">
<blockquote class="tr_bq">
In the response to this report, we recommend that the <span style="background-color: white;">Government outlines why it thinks road deaths increased in 2011.</span></blockquote>
</blockquote>
A possible (though unlikely) response is: "we were just unlucky this year".<br />
<br />
<h3>
Poisson Distributions</h3>
A crude way to capture the variability is with a <a href="http://en.wikipedia.org/wiki/Poisson_distribution">Poisson distribution</a>. The essence of a Poisson is that it counts how many events of some kind take place (like road accidents) whilst assuming that each event is independent of all the others. In other words, the fact that one accident has occurred (or hasn't occurred) doesn't make the roads any more or less dangerous for everyone else during the rest of the year. You don't think "oh well, there was an accident in my town today, so that's our quota for the month - I'll be safe now" - this would obviously be ridiculous.<br />
<br />
The only other piece of information we need is the intensity of accidents - how many deaths do we expect per year <i>on average</i>, possibly over many years of equally dangerous roads. Again, crudely, we can estimate this as the deaths of accidents in 2010, which was 1,850. So if in an average year we get 1,850 deaths, how often would we see 1,901 or more? Then answer turns out to be 12%, or about one in every eight years. In other words, this is not very surprising.<br />
<br />
What about the assumptions we made? In reality accident deaths are not independent, mainly because several deaths can occur all at once. Last year 7 people were killed in a particularly awful <a href="http://en.wikipedia.org/wiki/2011_M5_motorway_crash">accident on the M5</a>. This sort of clustering of events causes <i>overdispersion</i>, and means that the variability in the number of deaths will actually be greater than my simple model implied. Hence, 12% is likely to be an underestimate.<br />
<br />
Could it work the other way around? Perhaps after seeing the M5 crash, everyone drives a bit more carefully, and there are fewer accidents (this would lead to <i>underdispersion</i>, naturally). This is possible, but the effect is unlikely to be as strong as the effect of several people dying all at once. To be certain, we could test this idea with appropriate data.<br />
<br />
<h3>
Remarkable success</h3>
<div>
The reduction in the number of road deaths (particularly for vehicle occupants) in the last seven years has been remarkable (see graph); I can think of few other areas in which such a dramatic success can be demonstrated. The huge improvement in the safety standards of vehicles are the main reason, but better roads and higher petrol prices (so people drive more slowly) are also likely to have helped. </div>
<div>
<br /></div>
<div>
It's clear that the downward trend seen between 2006 and 2010 could not continue forever, so we shouldn't get too worked up about a small one year increase. If it rises again in the next couple of years, I'd be more concerned. I also can't understand why the committee find it "shocking" that road accidents are the biggest killer among young people. Something has to be the leading cause of death in this group; would they rather it was homicide? Or suicide? Or drug abuse?</div>
<div>
<br /></div>
<div>
Having said all this, the increase in the number of people killed or seriously injured looks much more significant (2% increase to 25,023), and might be harder to explain away as statistical noise. This post isn't intended to be an excuse for complacency in the Transport Department, just another cautionary tale about reading too much into small amounts of data.</div>
<div>
<br /></div>Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com1tag:blogger.com,1999:blog-806057380317734287.post-28623934671696901012012-07-03T15:50:00.000+01:002012-07-03T16:30:17.149+01:00Sigma-tisedIs it just me, or is everyone completely unable to explain what a confidence interval is? <br />
<br />
The Higgs Boson is back in the news again, <a href="http://www.nature.com/news/physicists-find-new-particle-but-is-it-the-higgs-1.10932">here's a Nature News article</a> discussing Cern's latest discovery, which is that<br />
<blockquote class="tr_bq">
<span style="background-color: white;">The data contained “a significant excess" of collision events at a mass of around 125 gigaelectronvolts...</span></blockquote>
<blockquote class="tr_bq">
<span style="background-color: white;">Physicists have maintained that they will not announce the discovery of the Higgs until the signal surpasses 5 sigma, meaning that it has just a 0.00006% chance of being wrong. The ATLAS and CMS experiments are each seeing signals between 4.5 and 5 sigma, just a whisker away from a solid discovery claim.</span></blockquote>
This is <b>wrong</b>.<br />
<a name='more'></a>I also found incorrect explanations by <a href="http://web.mit.edu/newsoffice/2012/explained-sigma-0209.html">MIT</a>, the <a href="http://www.guardian.co.uk/science/shortcuts/2012/jul/02/higgs-boson-found-definite-maybe">Guardian</a>, and the <a href="http://www.nytimes.com/2012/07/03/science/physicists-inch-closer-to-proof-that-higgs-boson-particle-exists.html?_r=2">New York Times</a>, the latter of which (discussing a different discovery) said<br />
<blockquote class="tr_bq">
The odds that the Fermilab bump were due to chance were only one in 550.</blockquote>
<span style="background-color: white;">Now I'm not a physicist, but my understanding of what Nature is </span><i style="background-color: white;">trying</i><span style="background-color: white;"> to say is something like "</span><b style="background-color: white;">if there were no particle</b><span style="background-color: white;"> of a mass around 125 GeV, then the chance of a 5-sigma event is</span><span style="background-color: white;"> </span><span style="background-color: white;">0.00006%."</span><br />
<span style="background-color: white;"><br /></span><br />
<h3>
Confidence for beginners</h3>
<span style="background-color: white;">The difference sounds quite subtle, but we'll see with some examples why it matters. Think of the physics experiment as a very accurate diagnostic test - <b>if</b> the particle doesn't exist, it's very unlikely your experiment will give you a false positive. It seems then to follow that <b>if</b> you get a positive result, it must be very unlikely that the particle doesn't exist. This is both 'intuitively' true and fatally flawed, because it really depends upon the base rate, or how likely you thought it was that the particle existed before you did the experiment.</span><br />
<br />
<span style="background-color: white;">Let's think of another example (I gave this one in a <a href="http://itsastatlife.blogspot.co.uk/2012/06/how-should-statistics-be-taught-some.html">previous post</a>). You go to the doctor, and he gives you a diagnostic test for a disease. The test is very accurate, let's say 99%; this means that <b>if you have the disease</b>, 99% of the time the test will say so, and <b>if you don't have the disease</b>, 99% of the time the test will come up negative. The test comes up positive - do you have the disease? (or rather what is the probability that you do?)</span><br />
<br />
The point is that it will depend upon how likely you were to have the disease in the first place, that is before you took the test. Pregnancy tests are very accurate, but if I took one and it came up positive I'd be pretty darn sure it was a false positive (not that pregnancy is a disease, *cough*). If the disease is rare (say 1 in 1000 people have it), and you had no particular reason to think you might have the disease, then it's still about 10 times more likely that the test was wrong than that you have the disease. In other words, the probability you have the disease is only about 1 in 11. So no need to panic.<br />
<br />
Reasoning wrongly as above is known as the base rate fallacy, and the maths can be formalised with <a href="http://en.wikipedia.org/wiki/Bayes'_theorem">Bayes formula</a>.<br />
<br />
<h3>
Other issues</h3>
Going back to the Higgs Boson example, things are a bit different. Physicists actually had good reason to believe that Higgs exists before the Large Hadron Collider actually started work, so we might put the prior probability of its existence at, say, 10%. (Perhaps one could do a study of how many Physicists' predictions turn out to be true.) Then if we observed a 5-sigma event we could apply Bayes' rule and find the posterior probability of the particle's non-existence, which would be about <span style="background-color: white;">0.0005% (one zero fewer than before).</span><br />
<br />
<span style="background-color: white;">Big deal you might say, this still seems like pretty solid evidence. There is a further problem, which is that these probabilities assume the experimenters have dealt with all possible other explanations for the evidence, critically including all sources of systematic error in their measurements. I'm sure they've set the experiment up very carefully indeed, but the probability of an error of this kind is still likely to be much higher than </span><span style="background-color: white;">0.0005% (I'd guess between 1% and 10%), so we really shouldn't say that the probability of an error is </span><span style="background-color: white;">0.00006%.</span><br />
<br />
<span style="background-color: white;">More generally, medical studies are generally performed to a standard of 5% confidence. This means that for a positive result, roughly speaking, <b>if the drug is useless</b> (or worse) there less than a 5% chance of seeing results as good (or better) than we did. This isn't perfect, so it should be obvious that some positive results will later turn out to be wrong, and so they are. You might think it would be precisely 5% of results, if you hadn't read the rest of my post.</span><br />
<br />
<span style="background-color: white;">In fact, <b>most</b> such studies <a href="http://theness.com/neurologicablog/index.php/are-most-medical-studies-wrong/">are false positives</a>. This is partly because of experimenter bias, but it's also just because most drugs put forward for trial don't work. Hence the relative rate of false positives to true positives is quite high. Things are almost certainly much worse in the social sciences.</span><br />
<span style="background-color: white;"><br /></span>Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-80483364012090475572012-07-02T13:27:00.002+01:002012-07-02T13:27:50.425+01:00Data Overload?Today I guest blogged about the government's open data plans on the Understanding Uncertainty website. <a href="http://understandinguncertainty.org/data-overload">Check it out here</a>.Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-11845651565974243182012-06-24T15:35:00.000+01:002012-06-29T22:41:59.732+01:00National Traits and the Ecological Fallacy<a href="http://www.dailymail.co.uk/sciencetech/article-2163771/Why-criminals-believe-heaven-Study-finds-crime-rates-vary-according-religious-beliefs.html">This article</a> caught my eye, headlined "Why criminals believe in heaven." [The Daily Mail's 'Science' and 'Health' pages are an unlimited resource for writing blog posts about bad statistics, but I promise to try not to <i>only</i> pick on them in future.]<br />
<br />
The <a href="http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0039048">original paper is here</a> (predictably the DM don't bother to link to it), and <a href="http://uonews.uoregon.edu/archive/news-release/2012/6/belief-hell-according-international-data-associated-reduced-crime">here is the university's press release</a> which has been pretty slavishly copied to create the article. The summary is, the researchers combined crime data from one source (the United Nations Office on Drugs and Crime), with survey data on people's attitudes to religion and other things (from World Values Surveys and European Value Surveys). They found that the belief in heaven and hell is a strong predictor of crime rates at a country-wide level, whilst rates of particular religious beliefs were not.<br />
<br />
<a name='more'></a><span style="background-color: white;">My main problem with the press release, and consequently the newspaper article, is this sentence:</span><br />
<blockquote class="tr_bq">
The finding surfaced from a comprehensive analysis of 26 years of data involving 143,197 people in 67 countries.</blockquote>
The implication of this seems to be that the research involves following 143,000 people for 26 years, and seeing both whether they are religious and whether they commit crimes. In fact the number refers to how many people were asked about their religion. This is a nice large sample size, so they probably have pretty accurate information about religiosity in the relevant countries.<br />
<br />
However the crime data is taken from a completely different source! The UN numbers are based on national statistics, which are, of course, collected in completely different ways in different places, and at varying levels of accuracy. I would be very wary of comparing rates of rape or burglary between countries based on this methodology, let alone with variables from other surveys.<br />
<br />
The second point which occurs to me is that the crime data effectively covers <i>all</i> the people in these countries, so one might as well say that we have "data involving [several billion] people in 67 countries." In fact there isn't really any data on individuals which answers the question being posed, so our sample size is, at best, 67. The sentence quoted above misleads us into thinking the evidence is much stronger than it really is.<br />
<br />
Anyway, if you believe the numbers, you might legitimately conclude that rates of belief in hell are correlated with crime rates amongst the countries surveyed. This is shown in the paper's graph, reproduced here:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRTfWGZ1p1DqW_zDPIWHU-IB1WgWheJaEkhrMaBDd7_eV4RVKJDgLfiLpw8b99e01DlyZiffQpAfFTZbMf2_N4JHnsK-BFQw9QwOKPhgEZAJi1FbJKu4SbZISRwMN5n7d0TaSq1BXXqR4/s1600/journal.pone.0039048.g001.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="323" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRTfWGZ1p1DqW_zDPIWHU-IB1WgWheJaEkhrMaBDd7_eV4RVKJDgLfiLpw8b99e01DlyZiffQpAfFTZbMf2_N4JHnsK-BFQw9QwOKPhgEZAJi1FbJKu4SbZISRwMN5n7d0TaSq1BXXqR4/s400/journal.pone.0039048.g001.png" width="400" /></a></div>
<br />
[Note to all those who present data: don't try to label all 67 points on a graph like this, it gets very confusing.]<br />
<br />
What we would like, but do not have, is information at the <i>individual level</i> about whether those people who commit crimes and those who don't believe in hell are the same people. It is quite possible that if you group individuals in some way (such as by country), and only look at the group averages, you get completely different relationships between variables. This is called the <i><a href="http://en.wikipedia.org/wiki/Ecological_fallacy">ecological fallacy</a></i>. <br />
<br />
To see this, imagine what would happen if the US was treated as 50 separate states, instead of one country, and that all the states occupied pretty much the same point on the graph (the US is the yellow dot at about (11%, 0.8)); this would increase the slope of the best fit line, making the relationship seem more important. Conversely, suppose we did this to Tanzania (the blue dot on the left), dividing it into its 26 regions; then the relationship would seem a lot weaker. Combining the Latin American countries would also make the relationship weaker, because many of them have both high crime rates and high belief in heaven relative compared to hell.<br />
<br />
Now it might seem very arbitrary and silly to divide Tanzania up in this way, or to combine Latin America, but the point is that countries are somewhat arbitrary units in the context of the relationship between religion and crime too. Some countries are very homogenous, some very diverse; some groups of countries are very culturally similar, and at other times in history would have been considered a single country (e.g. <a href="http://en.wikipedia.org/wiki/Union_between_Sweden_and_Norway">Norway and Sweden</a>); some countries are much more populous than others (China vs. Guatemala).<br />
<br />
Even if we could see a relationship at the level of individuals, this would be much stronger evidence than the country-wide averages, but would still only be a correlation. It could be that people stop believing in hell if they see lots of crime, or if they see people committing crimes and getting away with it. It could be that other variables such as prosperity or unmeasured cultural similarities are important. The authors try to account for these possibilities, but there's only so much you can do.<br />
<br />
The original paper carefully mentions both these problems:<br />
<blockquote class="tr_bq">
[I]t will be important to examine these real-world effects at the level of the individual. The present findings tie rates of belief at the societal level to national crime rates... It is also possible, however, that an intervening variable or variables are at work at the societal level. ... The direct causal explanation is most closely in line with the experimental findings, but it could well be that both the direct and indirect mechanisms are at work. To assess individual-level effects simultaneously with societal-level effects, it will be necessary to collect data with both national crime rates and individual tendencies toward immoral behavior.</blockquote>
The press release, however, only mentions the causation problem and not the ecological one, and the newspaper articles mention neither. The Daily Mail's article largely avoids the trap of mentioning individual level phrases (apart from the headline). But because it contains no caveat that (i) country-wide averages may not reflect individual level traits, or (ii) correlation does not imply causation, the overall impression given is that if an individual believes in hell this causes him to be less likely to commit crimes.<br />
<br />
Other headlines I've seen include "People who believe in redemption commit more crimes" (Real Bollywood), "Belief In Hell Lowers Crime Rate, According To International Study" (Huffington Post) and "Belief in Hell Keeps Crime Rates Down, According to New Study" (Christian Post), all of which screw up on one or both of these points.<br />
<br />
To be clear, I don't have a problem with doing this kind of analysis, but it only constitutes <b>very weak evidence</b> of a direct relationship between crime and belief in hell. It's probably not worthy of a newspaper article at all, but the desire of universities for self-publicity is strong, and therefore the pressure on researchers to get press attention is strong too.<br />
<br />
<b>Reference:</b><br />
Shariff, A and Rhemtulla, M. - Divergent Effects of Beliefs in Heaven and Hell on National Crime Rates, <i>PLoS ONE</i> 7(6), 2012.Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-26015671067731514482012-06-23T22:42:00.000+01:002012-07-03T16:31:55.518+01:00Review: The Geek ManifestoThis week I've been reading Mark Henderson's book <i>The Geek Manifesto: Why Science Matters</i>, which discusses the role of science and its geeky proponents in public life. Henderson's principal thesis is that politicians and scientists have been wilfully ignorant of each other for too long, and that it is time for science, its methods, and its practitioners to take a more central place in politics and policy. This includes statistics, of course, which is why it seemed relevant to the blog.<br />
<span style="background-color: white;"></span><br />
<a name='more'></a>The central examples are chosen to illustrate the challenges science and scientists face in the public sphere: Simon Singh's <a href="http://en.wikipedia.org/wiki/BCA_v._Singh" style="background-color: white;">libel troubles</a><span style="background-color: white;"> for pointing out that chiropractic medicine has no useful effect for problems like asthma and colic, in spite of the way it is marketed by some practitioners; </span><a href="http://en.wikipedia.org/wiki/David_Nutt#Dismissal" style="background-color: white;">David Nutt's dismissal</a><span style="background-color: white;"> as head of the Advisory Council on the Misuse of Drugs, essentially for doing his job and daring to say that the drugs classification system does not reflect the relative harm of the substances being classified; and the coalition government's plan to reduce the science budget by as much as a third. </span><br />
<br />
But what is most important about these incidents is not so much the serious problems they reveal about the way our country is governed, but the impressive reaction they elicited from large numbers of people in our society who <i>do</i> recognise the importance of science and proper evidence. The campaign to support Singh contributed greatly to the current push for libel reform, which is working its way through parliament as I write this. The backlash against the sacking of Nutt took the government and then Home Secretary Alan Johnson by surprise, as they had assumed it would just be a routine sacking going largely unnoticed. And the planned budget cuts spawned a movement named <a href="http://scienceisvital.org.uk/" style="background-color: white;">Science is Vital</a> which passionately argued for the benefits of government support for research, and won a significant, though <a href="http://www.guardian.co.uk/science/2010/oct/19/spending-review-science-budget-spared">imperfect, reprieve</a> (the non-capital science spend was frozen for five years).<br />
<br />
The common theme according to Henderson is the rise of a sort of Geek Power movement, which he believes has the potential to transform the way that science is understood by the general public, and can help make truly evidence-based policy integral to all governments and parties, current and future. The book covers areas including health, the environment, justice and education. <span style="background-color: white;">This is an important work, and it would be wonderful to believe that some of the ideas Henderson recommends might bear fruit. There is some evidence that progress is being made: a copy is being sent to every MP. A good start would be for the government to act on </span><a href="http://www.cabinetoffice.gov.uk/resource-library/test-learn-adapt-developing-public-policy-randomised-controlled-trials" style="background-color: white;">this report</a><span style="background-color: white;"> on randomised trials.</span><br />
<br />
I'm very much of the view that as voters we get the politicians we deserve, and Henderson clearly agrees, urging geeks to make our views known to our elected representatives. It is too easy for an MP to oppose a mobile phone mast near a school or support a homeopathic hospital if only one side ever makes a fuss. If we expect our leaders to use evidence to make policy, then we have to make it costly for them not to do so. They may think that it shows weakness to admit that the evidence shows a policy has failed, but we need to convince them that this is a sign of strength. Something similar goes for newspapers, of course.
<br />
<br />
Henderson is very careful to make the distinction between scientific advice to government, which lays out the evidence and facts as best understood, and the legitimate political choice which then follows. What needs to be avoided at all costs is politicians being able to claim to be acting on (possibly non-existent) evidence, when in fact the motive is purely political. The upgrading of Cannabis from Class C to Class B is a good example of this.<br />
<br />
My only real criticism of the book is that science itself seems to be portrayed in a rather idealised way, a noble pursuit which sits above ordinary human endeavour. <span style="background-color: white;">This is best illustrated by an anecdote of Richard Dawkins' which is retold:</span><br />
<blockquote class="tr_bq">
'There was an elderly professor<span style="background-color: white;"> in my department who had been passionately keen on a particular theory for ... a number of years, and one day an American visiting researcher came and he completely and utterly disproved our old man's hypothesis. The old man strode to the front, shook his hand and said, "My dear fellow, I wish to thank you, I have been wrong these fifteen years."'</span></blockquote>
This describes perfectly how scientific research should be, but scientists are, in fact, human, and this story is not representative of how people usually behave when their pet theories are demolished (in my modest experience).<br />
<br />
I don't suppose that Henderson, though not a scientist himself but a former science editor at <i>The Times</i>, is actually under any such illusion about researchers, r<span style="background-color: white;">ather the style seems to be a product of his (very successful) efforts to persuade the reader. My own feeling is that this attempt to show science in the best possible light involves perpetuating a slightly unrealistic view of how it is really carried out. It's a bit like giving Robin Cook's resignation speech as an example of how politics is done.</span><br />
<br />
The book is well written<span style="background-color: white;">, as you might expect</span><span style="background-color: white;">, and I imagine it would be enjoyed by anyone who is interested in science, and possibly those who might have thought science rather dull until now (like a high school chemistry class). </span><span style="background-color: white;">The content is not just a series of examples aimed at those who will already agree that science is great, but an inspiring call to arms for researchers and their allies to stand up and defend what's important to them. A worthy aim, and I have to say it's worked on me. See you on the barricades. </span><br />
<br />
<i>The Geek Manifesto: Why Science Matters</i> by Mark Henderson is available now in hardback and as an e-book, published by Bantam Press. Some free extracts are found <a href="http://geekmanifesto.wordpress.com/2012/06/06/read-extracts-from-the-geek-manifesto-for-free/">here</a>.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-35359680850694852672012-06-17T19:39:00.000+01:002012-07-03T16:31:23.329+01:00How should statistics be taught? Some thoughts.Inspired by Timothy Gowers' recent post on <a href="http://gowers.wordpress.com/2012/06/08/how-should-mathematics-be-taught-to-non-mathematicians/">how mathematics should be taught to non-mathematicians</a>, I thought it might be prudent to ask how statistics should be taught. If you need to be motivated as to how important teaching statistics is, watch <a href="http://www.ted.com/talks/arthur_benjamin_s_formula_for_changing_math_education.html">Arthur Benjamin's short TED talk</a>.<br />
<span style="background-color: white;"></span><br />
<a name='more'></a>My own experience of statistics up to GCSE was generally one of boredom: in the main it seemed to involve drawing histograms and line graphs, with a peculiar level of attention applied to getting the axes perfectly right, and choosing a suitable title. This is not to say that labelling axes is unimportant, but it's not exciting. More useful, and also more fun, is to explain why bad graphics are such a problem. Here is pretty <a href="http://www.psdgraphics.com/psd-icons/3d-pie-chart/" style="background-color: white;">bog standardly horrific example</a><span style="background-color: white;"> of a 3D pie chart. It wouldn't be hard to explain to school children why not to use this sort of presentation, and perhaps it would eventually filter up into the upper echelons of corporate management (I live in hope).</span><br />
<br />
School probability, perhaps understandably, involved simply counting events (there are 3 blue socks and 4 red socks in a drawer, I pick 2 at random, what's the probability they're both blue?), with the examples tending to be rather implausible and dry. This sort of thing is again important, but the examples can easily be jazzed up a bit; much like socks.<br />
<br />
At A-level things perked up a bit: hypothesis tests actually made me feel that I could do useful things with statistics - I even did a bit of coursework on UK election results. Box and whisker plots and finding the median of a bunch of numbers was somewhat less thrilling.<br />
<br />
The view I've come to form recently is that applied statistics should, at all levels, teach two essential things, let's call them the <i>cornerstones</i>.<br />
<br />
<ol>
<li><b>'Basic' probabilistic and statistical intuition.</b> People just aren't innately very good at dealing with probability and risk, even those of us who are trained to do so. Many statistics courses do cover these issues, but it has to be drilled into people again and again. Here I'm thinking of the base rate fallacy, prosecutors fallacy, regression to the mean, etc; don't worry if you don't know what these terms mean yet, but see some of the examples below.</li>
<li><b>Construction of statistical methods.</b> The thing I don't like about most applied statistics courses (including ones I have led myself) is that they involve teaching a long list of hypothesis tests or other methods which are useful in certain scenarios, and then providing some contrived examples to which we can apply them. But what if the situation you actually find yourself in doesn't lend itself to a t-test, an <a href="http://en.wikipedia.org/wiki/ANOVA">ANOVA</a> test, or a generalised linear model?</li>
</ol>
<div>
For example, to my mind it isn't so important that a student remembers what <a href="http://en.wikipedia.org/wiki/Cook's_distance">Cook's distance</a> or leverage is, as long as they understand what statistical inference is, and therefore why it might not be a good idea for your estimate to be very unstable when a few data points are removed. After all, that lesson applies in almost any statistical model, whilst Cook's distances do not.</div>
<div>
<br /></div>
<h3>
A Case Study Approach</h3>
<div>
I think that an approach based on the study of real (or at least believable) examples could help to ensure that the cornerstones are the main message of the course, rather than the less important technical details. Of course, the suitable level of rigour in a statistics course need not be constrained in any way by the use of these sorts of examples, and will depend strongly upon the intended audience. Horses for courses, as they say.</div>
<div>
<br /></div>
<div>
Below are some interesting questions that could be answered with probability and statistics, and a short explanation of how they relate to the cornerstones. Please do let me know what you think!</div>
<div>
<br /></div>
<div>
<i>1. Are men taller on average than women?</i></div>
<div>
<br /></div>
<div>
Leaving aside questions of gender identity, the point here is that we can't measure every man and woman to get a definitive answer. We might therefore think about sampling some men and some women, in the hope that the average of the sample is related to the average of the population. If, for example, men are taller than women in our sample, we would also need to consider whether this could have happened by chance. All this motivates the law of large numbers and the central limit theorem; these are weighty theorems, and it can be hard to communicate both their content and importance. For less advanced students, it can be used to give an intuitive idea of statistical significance. The key point is <b>not</b> to just teach students how to use a two-sample t-test.</div>
<div>
<br /></div>
<div>
<i>2. How would you go about determining the number of people in Cambridge who like to watch basketball? How about in China?</i></div>
<br />
This problem also uses the idea that sampling is somehow relevant to the population. A bigger question is how on Earth one could obtain a random sample of people from China, and what a polling company might do to try to compensate for this difficulty. The binomial distribution is obviously relevant here, but again we don't just want to do 'inference by formula'.<br />
<div>
<br /></div>
<div>
<i>3. A doctor gives you a test for a rare disease: approximately 1 in 1000 people have the disease. The test is very accurate; it is correct 99% of the time. If the test is positive, what is the probability you have the disease?</i></div>
<div>
<br /></div>
<div>
This is the simplest sort of base rate fallacy; the test is so accurate that it seems almost certain you have the disease. In reality, assuming that you have no particular reason to think you had the disease before the test (e.g. symptoms), it is still much more likely that the test is a false positive (probability about 91%) than that you actually have the disease (about 9%).</div>
<div>
<br /></div>
<div>
<i>4. A council collates the number of road accidents on 500 similar stretches of road. It finds that 40 of these have had at least 7 accidents in the last 5 years, whilst the average is just 4. It decides to put a speed camera at these 40 locations. 5 years later, the average number of deaths at the camera locations is just 4. Did the speed cameras do their job?</i></div>
<div>
<br /></div>
<div>
This is a dressed-up 'regression to the mean' example. Imagine that really there's no difference between the roads; some will have more accidents in the first 5 year period by chance, and then because they're not actually more dangerous, the number of accidents will probably be lower in the second period. So we can't tell whether the speed cameras work or not from this sort of data. The important thing is to consider what sort of experiment the council could use to answer the question. [I don't have anything against speed cameras, by the way, but it's a realistic sort of example.] </div>
<br />
<i>5. A PSA test is used to diagnose prostate cancer in men. The measured PSA level is generally higher if the volume of a tumour is higher. How should the test be used?</i><br />
<br />
This is much more complicated, and takes into account the relationship between PSA and tumour size (linear regression?), what PSA looks like in healthy patients, the relative problems of false positives and false negatives, etc.<br />
<br />
The last two examples are taken from Professor Sir Timothy's post:<br />
<div>
<br /></div>
<i>6. In September 2009 <a href="http://www.telegraph.co.uk/news/newstopics/howaboutthat/6202593/Bulgarian-lottery-picks-same-numbers-in-straight-draws.html">the same six numbers were chosen in two consecutive draws of the Bulgarian State Lottery</a>. Was this conclusive evidence that the draws were manipulated?</i><br />
<br />
We see silly newspaper stories about this kind of 'coincidence' all the time, and it's not always obvious how big a coincidence it is. These problems are quite challenging (<a href="http://www.bbc.co.uk/news/magazine-16118149">see also this</a>).
<br />
<div>
<br /></div>
<div>
<i>7. In 1999 a solicitor named Sally Clark was convicted for the murder of
her two sons... Roy Meadow, a paediatrician, argued for the prosecution as
follows. The probability of a cot death is approximately 1 in 8500. So
the probability of two cot deaths is roughly the square of this, or 1 in
about 73 million. Therefore it was overwhelmingly likely that the
deaths were not due to natural causes. Is this argument valid?</i></div>
<br />
A sad and infamous example of the prosecutor's fallacy, the Texas sharp shooter fallacy, and various other statistical crimes, with real and terrible consequences. Much has been said about this already, so I won't add more here.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com0tag:blogger.com,1999:blog-806057380317734287.post-39004872417151821402012-06-02T12:39:00.000+01:002012-06-29T22:43:21.547+01:00Newsflash: everyone being healthier reduces deathsPeople like me often complain about the accuracy of scientific journalism, and the selective nature of the reports we see in the media; indeed, there's plenty to complain about. However it's perfectly possible to misrepresent a story simply by choosing to place a particular spin upon it, usually one which makes the story seem more political than the original research. It's easy to blame the newspapers for this, but often the problem is compounded by academic press releases, which are themselves designed to catch the eye of media outlets.<br />
<a name='more'></a><br />
<span style="background-color: white;">A good example of this comes from </span><a href="http://www.bbc.co.uk/news/health-18262608" style="background-color: white;">a story this week</a><span style="background-color: white;"> about alcohol intake, which reports that, essentially, if everyone drank only one half of one unit of alcohol per day (i.e. 5 ml of pure alcohol per day), this would save thousands of lives per year. Many news outlets picked this up, and it even found its way onto Radio 4's </span><i style="background-color: white;">The News Quiz</i><span style="background-color: white;">, where the idea that people should only drink about 125 ml of beer per day was thoroughly ridiculed. The Daily Mail's headline, for example, began "</span><a href="http://www.dailymail.co.uk/health/article-2152424/Dont-drink-quarter-pint-DAY-Oxford-study-claims-slashing-official-alcohol-limit-save-4-500-lives-year.html" style="background-color: white;">Don't drink more than a quater of a pint a day</a><span style="background-color: white;">", followed by "Oxford study claims slashing the official alcohol limit would save 4,500 lives a year."</span><br />
<br />
You can get the original paper <a href="http://press.psprings.co.uk/Open/may/bmjopen957.pdf">here</a> (open access, happily), and I recommend a look to see the contrast with what you might have been exposed to in the news. If you're not a scientist, or imagine that scientific papers must be completely incomprehensible then I think you'll be pleasantly surprised; it's perfectly readable, largely free from unnecessary jargon and technical terms, and unlike many reports you might encounter in the worlds of business and politics, admirably concise (7 pages, including tables and figures). <br />
<br />
Notice the difference between the newspaper headlines and the title of the paper: "What is the optimal level of population alcohol consumption for chronic disease prevention in England?" Aside from being rather less exciting sounding, note that this makes no mention of strategies for <i>individuals</i>, nor about telling people what to do.<br />
<br />
The results summary tells us that "reducing the median consumption of alcohol to 5 ml per day would avert or delay 4,579 (2,544 to 6,590) deaths per year" [note to the authors, <a href="http://en.wikipedia.org/wiki/Significant_figure">rounding</a> is your friend]. There are three things about this sentence which strike me. First, unlike the news stories, there is a 95% credible interval associated with the estimate of the number of "lives saved"; it could be as low as 2,500, or as high as 6,600, which is a pretty wide margin. This doesn't need further statistical explanation here, suffice to say that there is a good deal of uncertainty about how much effect alcohol has upon different aspects of people's health. Second, we're talking about the median alcohol intake, not everyone's intake, which is not the same; under this scenario half the population drink 5 ml per day or more, and half drink 5 ml or less. Lastly, it mentions deaths "averted or delayed".<br />
<br />
Now you might wonder what averting our own death might mean, as opposed to delaying it, which is seems more realistic. Sadly the paper doesn't explain precisely, but here's my best guess: the paper is based upon a series of simulations under different scenarios of how much people drink, using estimates of the effects of alcohol on various diseases. Then they count the number of deaths each year under the various scenarios; 1,000 deaths averted or delayed simply means that the number of deaths counted in the simulation was 1,000 fewer than under the status quo. Now, from an individual point of view, it could just be that you die the next year instead; or you could live another 20. The study doesn't consider this, because their interest is from a public health perspective.<br />
<br />
<h3>
Risk and reward</h3>
This is perfectly reasonable from the government's point of view, and in a purely economic sense: if someone lives for a year longer before they get liver cirrhosis or some other unpleasant chronic condition, then that's one more year before the public purse has to shell out for your treatment. Standard economic theory applies a discount to a cost in the future as compared to something you have to pay for now. In particular, it only requires a pretty small proportion of people to live a bit longer for this to turn into a significant amount of cash saved. And as a bonus your citizens live happier, longer lives, etc.<br />
<br />
From an <i>individual's</i> perspective, however, the chance of benefiting from this reduced risk of death may be extremely small, and when presented with the facts it would be perfectly rational to decide that you'd rather enjoy a few drinks (or a <a href="http://www.youtube.com/watch?v=a1PtQ67urG4">bacon sandwich</a>) than have a slightly longer life expectancy.<br />
<br />
In any case the articles in the news imply that scientists are telling us all to drink less, rather than just reporting that drinking less would might be good for us. In particular the Daily Mail's article is pretty explicit in saying that "scientists say" that we should do such and such, and even claims that "changing the guidance" would save the 4,500 lives, rather than people actually drinking less. The paper itself says explicitly otherwise [my emphasis].<br />
<blockquote class="tr_bq">
The recommendations and public messages...that would be required to achieve this...level of consumption are <b>beyond the scope of this work</b>... Public health behavioural recommendations should ideally be based on the best available evidence for optimising population health outcomes.</blockquote>
<h3>
Who should we shout at?</h3>
Now to my favourite part: assigning blame. In this case, the BMJ's own <a href="http://group.bmj.com/group/media/latest-news/cutting-daily-alcohol-intake-to-half-a-unit-5-g-could-save-4600-lives-a-year">press release</a> is at least partly at fault, and reads pretty similarly to the BBC article. The credible interval has been removed, even though it is a very important measure of uncertainty inherent in the study, and, in this humble statistician's view, is perfectly capable of being understood by the non-scientific general public. That the level of uncertainty in scientific studies tends not to be reported contributes to the general impression that science always provides precise answers to questions; this causes problems when people read about studies with conflicting results, and leads to the contradictory impression that scientists don't know what they're talking about, and that one may as well ignore them. Naturally, neither of these things are true.<br />
<br />
As an example of how this translates to media-world, the Mail says [my emphasis again]<br />
<blockquote class="tr_bq">
The new advice <b>flies in the face of previous studies</b>, which have shown that drinking alcohol in moderation reduces the risk of dying from heart disease. </blockquote>
This is completely untrue. The paper is itself a meta-analysis of other results combined with a simulation, and does not involve a new study in the sense of following drinkers and seeing what they die from. The model used takes account of the fact that alcohol appears to be protective against heart disease, but finds this to be outweighed by other risks.<br />
<br />
I'm uncertain as to who writes these sorts of press releases. There's plenty of pressure on scientists to get their research noticed by the general public, and the lead scientist of this study is quoted in some detail by the Mail. Getting this kind of attention depends upon newspapers and other media outlets picking up these stories, which in turn depends upon those outlets persuading their customers that the story is worth reading about. Implying that it's telling them all to stop drinking is a good way to grab someone's attention.<br />
<br />
Some people take the view that it's better for scientists to hide behind a wall of solid fact, especially on issues such as climate change and evolution, because the idea that some scientists disagree on any aspect (however trivial in comparison with the might of the whole idea) might suggest that it's somehow equally valid to assume climate change is not down to carbon dioxide, or to believe in intelligent design. I think that this is likely to be highly counterproductive, and that it would be far better for scientists and the media to give a more realistic idea of what science, and being a scientist, is really like. The BMJ improving their press releases would be a good place to start.<br />
<br />Robinhttp://www.blogger.com/profile/05296526563838815366noreply@blogger.com2