Data, data, everywhere: but
So, the National Household Survey’s Portrait of Canada’s labour force is out, and I can’t help but think of Donald Rumsfeld’s known unknowns. We know that we don’t know anything about those who didn’t respond to the survey, or how they might be different from those that did. We also know that there are some discrepancies in terms of labour force data even at the national level when comparing the NHS to the LFS.
We know that the sample size for the NHS was 4.5 million households, with a 68.6% response rate. The LFS only samples 56,000 households, but it is mandatory and conducted by specialized interviewers. And, finally, the LFS excludes persons living on reserve & in Aboriginal communities, and full time members of the Canadian Forces, but the NHS does not. So which one gives us better labour market information?
| Â | LFS (Unadjusted)
(May 15 – 21, 2011) |
NHS
(May 1 – 7, 2011) |
Difference |
| Employment | 17,478,700 | 16,595,035 | -883,665 |
| Employment rate | 62.5% | 60.9% | -1.6% |
| Unemployment | 1,410,000 | 1,395,045 | -14,955 |
| Unemployment rate | 7.5% | 7.8% | 0.3% |
| Not in labour force | 9,060,300 | 9,269,445 | 209,145 |
| Participation rate | 67.6% | 66.0% | -1.6% |
So the NHS estimates nearly 900,000 fewer jobs in May 2011 than the seasonally unadjusted data from the LFS, which results in a significantly lower employment rate. Has the labour market been doing worse than we think? Has the recovery been even more sluggish than the LFS numbers have led us to believe?
Which survey is more reliable right now isn’t even the biggest concern. If we are concerned about the NHS not reaching groups based on their education level, language, immigrant status, or more, then the accuracy of weighting for the LFS is bound to become less reliable as well. In 2011 the LFS was re weighted and revised based on the 2006 census, in 2016 the LFS will be reweighted and revised based on the 2011 NHS. Sam Boshra at Economic Justice covers this issue well here.
It’s okay to know what we don’t know – but I’m worried about thinking that we know something that “just ain’t so”.
 Updated June 27th: Data collection period added.
Angela,
It might be interesting to do that same analysis above but include the CV for both the LFS and NHS estimates to see if given the margin of error they overlap. I think it was just last month that the LFS started making their SE much more prominent, although you could always dig it up. Presumably the margin of error for the LFS is much much higher than the NHS.
Also I wonder if there are some timing issues comparing these two datasets like when the NHS actually sampled compared to the correct LFS month. Maybe you’ve got that covered though.
David, I just realized I didn’t cover the issue of timing in the post. The period for the NHS was May 1 – 7, 2011, and the period for the LFS data shown was May 15 – 21, 2011.
I agree it would be interesting to see if the error bars overlap. The monthly LFS is notoriously unreliable, but the error on employment is usually somewhere around plus or minus 50,000, nowhere near the difference found here.
I just checked, and the SE that is now published with the LFS is the standard error of the change, not of the estimate.
Using the CVs in the Guide to the Labour Force Survey (http://www.statcan.gc.ca/pub/71-543-g/71-543-g2012001-eng.pdf), there is a CV of 1% for any monthly estimate at the national level over 1.159 million. So at the 95% confidence level the employment estimate could be plus or minus 350,000 (likely a very generous estimate). The NHS CV would have to be at least 1.6% for there to be any overlap.