Crime is down. Crime is up. What's going on?

Crime in England and Wales fell by seven per cent in the 12 months to December 2014. Or rather, crime in England and Wales rose by two per cent in the 12 months to December 2014.

The latest crime data, published yesterday by the Office for National Statistics, supports both claims. Here's a quick guide to what's going on and why you should take all claims about rises and falls in crime with a pinch of salt.

The Crime Survey for England and Wales

The official crime data is collated in two different ways. First there is the Crime Survey for England and Wales. As the name suggests, it is a survey-based dataset, collated through face-to-face interviews with members of the public (currently around 32,000 householders over the age of 16).

The researchers ask those they interview a range of questions about their experience (or not) of victimisation in the previous twelve months. The incidents covered include violence, robbery and theft, burglary and vehicle crime, and criminal damage.

All the responses are then pulled together and used to estimate all the incidents that took place in England and Wales in that twelve month period. In other words, the researchers assume that the incidents recalled by the thirty two thousand people they interviewed are representative of those of the fifty seven million people living in England and Wales. Put another way, the recalled victimisation experiences of 0.6 percent of the population of England and Wales are used as a basis to estimate the victimisation experiences of the entire population of England and Wales.

This isn't as crazy as it sounds. The sample of 32,000 interviewees is a big one (the voting intention polls politicians and commentators are dissecting with such enthusiasm at the moment are typically of only 1,000 to 2,000 people). Sample size matters because it reduces the likelihood of error, as anyone who has shopped around for a bargain will know. If you are looking for a new television and 'Cheap TVs' quotes you £159 you don't necessarily know whether it's a bargain or not. So you increase the sample size by shopping around. If 'TVs Today' quotes you £179 and 'TVs R Us' offers it at £169, you'll be more confident that you've found a bargain. The same approach lies behind the various shopping websites.

Although an estimate of victimisation experiences based on less than one percent of the population sounds a bit dodgy, the fact that it's based on the recollections of 32,000 people, rather than 2,000 or 200 people, makes it more likely that their experiences are representative of the population. The Crime Survey still offers estimates however. The 6,949,000 incidents it reports have not all been individually counted.

The Crime Survey is also selective in the types of victimisation it covers. It doesn't cover homicide for instance. You can't ask someone if they have been murdered in the previous twelve months. Sexual assaults are also left out, on the grounds that there is not a big enough sample size to be confident about estimates. The victimisation experiences of some groups - for instance young people, women, the homeless and those living in poorer areas - are, for a variety of reasons, only partially covered.

So the Crime Survey is not a measure of 'total crime', 'overall crime', 'all crime' or any other variant of such a claim.

If a politician, journalist or commentator tries to tell you differently, they either don't understand the statistics, or they do and they are misleading you.

Police recorded crime data

The second way of measuring crime is through the data compiled by the police: so-called police recorded crime data.

These data have recently come under sustained criticism for being unreliable, because of the police tendency to fiddle the figures. As Professor Tim Hope pointed out on this site last year, the most surprising thing about the realisation that the police fiddle crime figures is that it should have come as any surprise at all.

The problem of police manipulation of crime data is not, however, the real issue when it comes to the question of whether they offer an accurate representation of crime trends. To understand why, let's think of an example from another policy area.

Imagine that you wanted to get a clear overview of the health problems in your local area. Imagine too that you lived near a hospital with a busy accident and emergency department. You go down to the A and E department on a Friday evening to see who turns up. You're aware of the problem of sample size (you recently got a bargain TV for £159 because you shopped around) so you go to your A and E department for three successive Friday evenings.

After compiling all your data you find that the health problems in your local area are largely related to:

• drunk young men who have got into fights
• people with a bad back who can't get in to see their GP
• DIY enthusiasts who have injured themselves while putting up the wallpaper 
• elderly people with breathing difficulties
• children who have managed to get their heads, toes and thumbs stuck in a variety of household appliances.

What your data will miss is all those other conditions - cancer, liver disease, mobility problems, depression etc - that don't generally end up in A and E.

Police crime data are rather like this. They are based on incidents reported by individuals to the police, and on incidents the police themselves record during the course of their activities. For this reason they are better at capturing some types of crime (such as burglary and vehicle theft) than others (domestic violence and sexual assault for instance). In principle police data covers all types of behaviour currently defined as 'crime'. In practice they do not do so because much behaviour defined as crime never comes to the attention of the police.

Even if every incident currently defined as crime was accurately recorded by the police, the police recorded crime data would still not represent all crime committed in a given period. For one thing, the police recorded crime data only list so-called 'notifiable' offences ('notifiable' in the sense that the police are required to notify the Home Office about them). Notifiable offences are a subset of all behaviours open to being defined as criminal: in essence all those offences that could result in a trial by jury, rather than by a magistrate, along with a miscellaneous set of other offences.

If this all sounds rather arbitrary, if not somewhat confusing, that's because it is. The important point is that the 'notifiable offences' that appear in the police crime data are far from being a comprehensive list of all offences the police might come across.

Another thing to remember about police recorded crime data is that they are a record of allegations made (by the public) and/or conclusions drawn by the police. I might report to the police that I was assaulted in the pub following an argument with another customer whose pint I spilled. The police might duly record the incident, which duly makes its way into the notifiable offences in the police recorded crime dataset. Whether my allegation is ever fully substantiated, or an individual ever convicted of that assault, is another matter. The incident recorded relates to my allegation, not a successful prosecution.

There are any number of additional complications, covered in excruciating detail in the Home Office counting rules. The guidance on recording crimes of violence against the person, for instance, stretches to 87 pages.

Cutting through all this complication to the essential point, we can say that police recorded crime data tells us something about incidents the police come across during their day-to-day activities. They tell us only a little - sometimes nothing at all - about the many day-to-day behaviours and confrontations that could, in principle, be defined as a crime. They tell us nothing at all about behaviours and incidents formally defined as a crime as a result of someone being found guilty in a court.

Different data, different results

The Crime Survey and police recorded crime data measure different phenomena in difficult ways with varying degrees of accuracy and usefulness. Neither dataset on their own, or taken together, offers an accurate guide to all behaviours open to definition as crime, or crime 'overall'. Indeed, as I have previously argued, talk of 'overall crime' is best reserved for crime involving overalls.

This isn't going to stop politicians and commentators using particular crime datasets to make particular claims about crime. Nor will it stop the rather futile and tedious discussion over small percentage point changes in crime statistics that tell us little about the lived experience of members of the public.

But being aware of the limitations of crime statistics, and the problems associated with the often grandiose claims made about them, is an important step in the direction of a more informed public debate.