Predictive Versus Descriptive Observations

The detailed quantification and numerical dissection of sports started in the US long before it started here in Australia, one result of which has been a modern-day ability to produce statistics so obscure and specific that their half-life is measured in seconds - until the next play, the next pitch, the next batter, the next down, or even just the next statistic. It might well be the case that the Cubs are 7-0 when leading by 3 or more at the Bottom of the 6th when facing a left-handed pitcher at home but what, really, does that tell us about their chances in the current game? Not much, if anything, I'd suggest.

Such observations, while fun to marvel at if only for the ingenuity of their conception and the breadth and volume of the raw data that fuels them, can delude the casual fan into a misunderstanding of the nature of a truly analytic approach. For an observation like this - a statistic or rule - to be useful in an analytic sense, it needs to be reliably related to some outcome measure of interest, such as who wins, by how much, how many points are likely to be scored, and so on. The more tortured and specific an observation, the less likely it is to serve this function.

So it was interesting to me in the latter stages of the 2015 AFL competition to see fans and pundits trotting out statistics about, for example, the comparative success of Minor Premiers, playing at home in Preliminary Finals, as if this information somehow trumped the fact that the revealed form of the year's actual Minor Premiers, Fremantle, suggested that they were a much weaker Minor Premier than just about any we'd seen in recent seasons. Knowing how a particular team has performed in a particular season with mostly the same players will always be more informative about their chances in a given contest than any number of statistics about the performance of other teams in other seasons whose main similarity extends only to where they happened to finish on the ladder.

To talk like a Bayesian for a minute (which, I'll be honest, feels a little fraudulent for me to do, trained as I was almost solely in the Frequentist tradition) it's about building an informed prior based on an appropriate weighting of relevant empirical evidence. Not every relationship derived from history deserves equal weight - some probably don't even deserve a non-zero one. 

I've been struggling to come up with a useful way of thinking about these different types of empirical evidence presented as observations and, inspired by the useful distinction highlighted in this paper, landed on the terms that I've used in the title of this blog, namely Predictive Observations and Descriptive Observations.

A Descriptive Observation is one that draws a comparison between some current situation and similar situations from the past - for example, other teams that have finished in the same ladder position or played on the same ground. Similarity can be narrowly or broadly defined here, but is key to determining how informative the Observation is about the current situation. Every Observation is a Descriptive Observation, but they are Predictive Observations too only when the nature and level of the similarity is such that the Observation improves our ability to make predictions about the current situation. A Descriptive Observation can be interesting and apparently unlikely, but still have little or no predictive utility.

Once we've satisfied ourself that some Observations have predictive value, we can order them based on the degree to which they improve our ability to make predictions. In statistical modelling terms we can think of Observations as variables, and draw a parallel with the notion of variable importance. That line of thinking alerts us to the fact that there is no single measure of importance, and that the notion can be conceptualised in a number of slightly different ways. All the approaches and measures, however, are grounded in the idea that importance relates to how much better we can predict with the variable (Observation) than without it.

Ultimately then, whether or not a Descriptive Observation is also a Predictive one is a purely empirical matter, but there are some heuristics we can employ to help us make a rough assessment of whether or not an Observation is predictive and, if so, how much:

• How many defining characteristics does the Observation employ? Does it limit the time frame, the location, the time of day or day of week, or otherwise seek to narrow the scope of the Observation? If so, do these limitations serve mostly to enhance the applicability to the current situation or do they, instead, serve mostly to enhance the impressiveness of the Observation?
• How different are the outcomes as you progressively remove or adjust these limitations? If the Observation is, say, that home teams win 80% of the time when leading at Quarter time on Friday nights, what's the equivalent statistic for Saturdays? Ask yourself, what it is about Fridays that might make it a causal component of the Observation.
• How similar are the situations encapsulated by the Observation to the current situation? Here it's important to ignore merely superficial similarities and to focus solely on similarities that seem likely to confer some predictive value. Make explicit in your own mind plausible causal connections between the situational elements in the Observation and the eventual outcome.
• In particular, be careful of apparent similarities that assume constancy across long periods of time. If, for example, some generally weak team has won 15 of its last 20 encounters against some strong team at home, feel free to ignore that statistic when your realise that the weak team has lost the last 27 games against all-comers and that none of the players from those 15 previous victories over the stronger team still plays for the weaker team. The team bearing the same name from 5 years ago is barely, if at all, meaningfully "similar" to the current team. In a related vein, the proposition that a team has a "hoodoo" ground also requires close scrutiny. Why would this be the case? Does the shape or nature of the ground not suit their style of play? If so, has this really been the case for 10 or 20 years while the "hoodoo" has been taking hold?
• Also, be cautious about broad labels that blur meaningful distinctions, for example "favourites", "teams finishing 6th", "Victorian teams". Are all teams finishing 6th sufficiently similar to be treated as interchangeable? Probably not. Almost always, Descriptive Observations using these broad terms will barely classify as Predictive Observations.

You might have other heuristics that could be added to this list. If so, please let me know and I'll include them. Any general thoughts or feedback is also welcomed.

In the meantime, cast a critical eye over the Descriptive Observations that you encounter and see how many of them you think would qualify as Predictive ones too.