Don’t Let Numbers Fool You: Common Statistical Tricks You Should Know About
As promised, here’s my fourth post inspired by the recent Fraser Institute report on taxes paid by Canadian families.
I can’t stand seeing people fall simple numbers tricks. And while I realize that I don’t have the time to argue with everyone who is wrong on the Internet, I try to make it a point to call attention to questionable data manipulation practices and conclusions that don’t follow from the actual data, especially when these are passed for objective statistical analysis in the media.
The latest Fraser Institute report on the Canadian Consumer Tax Index provides a number of examples of common statistical tricks that everyone should know about and watch out for every time they read about a report quoting statistics on spending. Whether the article in question is about total spending on taxes, government spending on public services such as education or health care, or government debt levels, there are three important questions you need to ask yourself.
1: Are the numbers adjusted for inflation?
Failing to adjust for inflation makes changes in spending over time appear much bigger than they really are. The larger the time period being considered, the bigger the impact of forgetting to adjust for inflation.
For example, the Fraser Institute’s researchers do not account for inflation when they calculate that the average family’s tax bill increased by 1,624% over the last half century. The Consumer Price Index shows that inflation increase by 629% between 1961 and 2009, suggesting that a dollar in 1961 was equivalent in spending power to $7.29 in 2009 (thanks to Purple Library Guy for pointing out that there was a mistake here in the original post). Adjusting for inflation, we find that the average family’s tax bill increased by 137% not 1,624% since 1961.
2: Are actual spending numbers compared, or is spending expressed as a share of income?
Families and governments alike spend more money when they have higher incomes, so focusing on the raw spending numbers will produce a larger increase over time.
For example, the Fraser Institute researchers focus on the average family’s tax bill alone, without comparing it to family’s average income, which produces misleading results. Even if the tax system had not changed at all since 1961 and families still paid 33.5% of their incomes on taxes, the average tax bill would grow as income increases over time. With an average income of $69,175 the average family tax bill would have to be $23,174 just to maintain the same effective tax rate as in 1961. This, according to the Fraser Institute methodology, represents a 1,284% tax bill increase.
This trick is commonly used when looking at government spending on whatever program you’d like to see cut (healthcare spending rising out of control, anyone?). It can also be used when looking at the increase of government debt over time. Again, the longer the time period you look at, the larger the distortion will be.
The savvy reader now knows that personal spending has to be expressed as percentage of income and government spending or debt as a percentage of GDP to make comparisons over time meaningful.
3: Does the conclusion depend critically on the starting or ending year picked for the comparison?
Picking a starting year when spending was particularly low will make the change over time appear bigger. This one is a little bit harder to check on, because it requires you to look at the actual data, which is not always easily available.
For example, the Fraser Institute chose to start their analysis in 1961, a year when taxes only represented 33.5% of family income. The next two years that the report provides data for, 1969 and 1974 show considerably higher tax shares of income: 39% and 43.4%, respectively. Starting the analysis in any of these two years will dramatically change the results. If they compared taxes as a share of income in 1969 and 2009, they would conclude that the share of income going to taxes increased by 7% over the last 40 years. If they compared taxes as a share of income in 1974 and 2009, they’d find an actual decrease in the share of family income going to taxes.
Note that these tricks allow unscrupulous researchers to manipulate numbers in order to produce a particular effect without having to alter the underlying numbers in any way.
The only reason why economists can pull this kind of stuff off is because the busy and untrained journalists and general public do not challenge them. We need to increase basic statistical literacy so that the audience would be able to easily see through these misleading tricks.
In the meantime, however, we need economists and researchers to show integrity and oppose the use of such data manipulation and call it when they see it.
Typo.
“The Consumer Price Index shows that inflation increase by 629% between 1961 and 2009, suggesting that a dollar in 1961 was equivalent in spending power to $627 in 2009.”
I believe you need a decimal point there; would 629% not imply $1 –> $6.29 rather than $629?
Good article. I do wonder about point 2, however. Yes, it is frequently used as a trick. But if the economy genuinely grows per capita over time, if productivity in some sense genuinely increases, is it not reasonable to expect any given task to be performable using a smaller percentage of economic output? In other words, shouldn’t productivity growth imply an economy capable of doing more things? Of course, productivity growth isn’t equal across sectors and types of activity, so there’s no real way to calculate an expectation for such effects. They may also be swamped by other factors such as aging population, and one might expect gradual definition creep, where just what is comprised by some service gradually comes to be more comprehensive without changing its name. But, to take an extreme case, say you had a government service that provided a calculator for every man, woman and child in Canada. One would expect that in 1975 that would take up a much bigger share of GDP than in 2010. Pensions and such would be a very different story, of course.
Now personally, I would tend to prefer using extras from such effects to increase services rather than cut back taxation. But I’m still wary of a blanket expectation that a given level of services across the board should require the same percentage of economic output as economic output grows.
All that said, sure, outfits like the Fraser Institute aren’t making careful analyses of that sort of effect, they’re just slanting the heck out of things.
Good eye, Purple Library Guy, my inflation calculation was wrong so I fixed it. With 629% inflation, $1 in 1961 equals $7.29 in 2009 (calculated as $1*CPIin2009/CPIin1961). Note that going from $1 to $7.29 represents a 629% increase.
On your second point, I agree that over longer periods of time (like fifty years, which is what we’re looking at here) productivity growth should make things relatively cheaper. This is what we see with shelter, food and clothing taking up a smaller fraction of family’s income in 2009 than in 1961 – something the Fraser Institute shows (although they don’t spend much time interpreting it). This is also why the costs of basics take up a smaller share of family income in richer countries than in poorer countries.
However, we can choose to buy better quality or higher quantities of some goods & services as we get richer. The public services and direct transfers that our taxes pay for today are very different from the ones Canadians paid for in 1961. Yes, some things have gotten worse – EI eligibility and probably even benefits being a good example – but others are much better – look at OAS or CCP, which have drastically reduced poverty among Canadian seniors.
Therefore, there’s no a priori reason for taxes to fall as a share of family’s income, especially when we collect many taxes as a share of all earned income or all consumer spending. Instead, the government takes the pool of taxes and distributes them to provide new or enhanced services over time.
Good Post! I see similar slights of hand all the time in the the comparative capitalism literature where comparisons are regularly made on the basis of decades rather than Peak to Peak or trough to trough comparisons. Sometimes it is innocent and sometimes it is about loading the dice. My suspicion with the Fraser, given they refuse public debate, is that it is rarely innocent.