Don’t Let Numbers Fool You: Common Statistical Tricks You Should Know About
As promised, here’s my fourth post inspired by the recent Fraser Institute report on taxes paid by Canadian families.
I can’t stand seeing people fall simple numbers tricks. And while I realize that I don’t have the time to argue with everyone who is wrong on the Internet, I try to make it a point to call attention to questionable data manipulation practices and conclusions that don’t follow from the actual data, especially when these are passed for objective statistical analysis in the media.
The latest Fraser Institute report on the Canadian Consumer Tax Index provides a number of examples of common statistical tricks that everyone should know about and watch out for every time they read about a report quoting statistics on spending. Whether the article in question is about total spending on taxes, government spending on public services such as education or health care, or government debt levels, there are three important questions you need to ask yourself.
1: Are the numbers adjusted for inflation?
Failing to adjust for inflation makes changes in spending over time appear much bigger than they really are. The larger the time period being considered, the bigger the impact of forgetting to adjust for inflation.
For example, the Fraser Institute’s researchers do not account for inflation when they calculate that the average family’s tax bill increased by 1,624% over the last half century. The Consumer Price Index shows that inflation increase by 629% between 1961 and 2009, suggesting that a dollar in 1961 was equivalent in spending power to $7.29 in 2009 (thanks to Purple Library Guy for pointing out that there was a mistake here in the original post). Adjusting for inflation, we find that the average family’s tax bill increased by 137% not 1,624% since 1961.
2: Are actual spending numbers compared, or is spending expressed as a share of income?
Families and governments alike spend more money when they have higher incomes, so focusing on the raw spending numbers will produce a larger increase over time.
For example, the Fraser Institute researchers focus on the average family’s tax bill alone, without comparing it to family’s average income, which produces misleading results. Even if the tax system had not changed at all since 1961 and families still paid 33.5% of their incomes on taxes, the average tax bill would grow as income increases over time. With an average income of $69,175 the average family tax bill would have to be $23,174 just to maintain the same effective tax rate as in 1961. This, according to the Fraser Institute methodology, represents a 1,284% tax bill increase.
This trick is commonly used when looking at government spending on whatever program you’d like to see cut (healthcare spending rising out of control, anyone?). It can also be used when looking at the increase of government debt over time. Again, the longer the time period you look at, the larger the distortion will be.
The savvy reader now knows that personal spending has to be expressed as percentage of income and government spending or debt as a percentage of GDP to make comparisons over time meaningful.
3: Does the conclusion depend critically on the starting or ending year picked for the comparison?
Picking a starting year when spending was particularly low will make the change over time appear bigger. This one is a little bit harder to check on, because it requires you to look at the actual data, which is not always easily available.
For example, the Fraser Institute chose to start their analysis in 1961, a year when taxes only represented 33.5% of family income. The next two years that the report provides data for, 1969 and 1974 show considerably higher tax shares of income: 39% and 43.4%, respectively. Starting the analysis in any of these two years will dramatically change the results. If they compared taxes as a share of income in 1969 and 2009, they would conclude that the share of income going to taxes increased by 7% over the last 40 years. If they compared taxes as a share of income in 1974 and 2009, they’d find an actual decrease in the share of family income going to taxes.
Note that these tricks allow unscrupulous researchers to manipulate numbers in order to produce a particular effect without having to alter the underlying numbers in any way.
The only reason why economists can pull this kind of stuff off is because the busy and untrained journalists and general public do not challenge them. We need to increase basic statistical literacy so that the audience would be able to easily see through these misleading tricks.
In the meantime, however, we need economists and researchers to show integrity and oppose the use of such data manipulation and call it when they see it.