Measuring Quality in a Census, Part 2

Print This Post Print This Post

Written by: Director Robert Groves

An earlier post, “Quality in a Census, Part 1,” provided basic concepts of quality in population counts based on a census. This post gives the basic sets of information we use to measure those concepts, to answer the question, “How good is the 2010 Census?”

There are three common approaches to measuring quality. First is a process-oriented approach. That measures whether the steps used to create the product contained features to improve its performance. For example, we have learned that it is more likely that a successful space shuttle launch will occur when there are redundant sub-systems and test-firings of propulsion systems, and would label such as of “higher quality” than comparable shuttle designs that omit them. Another common method is to compare outcomes of different alternatives. For example, if your watch says 8:58 and the clock on the wall says 9:01, you feel safer thinking that it’s about 9 than if the clock said 9:42. Another common approach is to imagine the ideal and focus on how close the actual is to the ideal. For example, airlines use the percentage of their flights leaving on-time as a quality indicator.

For the 2010 Census, we have two of these kinds of indicators:

a. Design and process quality indicators (use of bilingual forms, use of replacement forms, additions and deletions to the address list, participation rates during nonresponse followup, percentage of cases yielding counts of persons within the household). These indicators are starting to become available in preliminary forms.

b. Comparison of the 2010 Census to other approaches (demographic estimates of population counts based on birth, death, immigration, and emigration records; use of a large sample survey matched to census records). These indicators won’t be available for some time.

We do have a notion of the “ideal” census – everyone residing in the US enumerated once and only once and assigned to their appropriate residential location. However, we don’t know what that ideal would actually achieve for counts at all levels of geography, for different demographic groups. Thus, the third type of indicator we don’t really possess.

I’ll talk about the two kinds of indicators over coming posts.

Please submit any questions pertaining to this post to

This entry was posted in Quality Assurance. Bookmark the permalink.

13 Responses to Measuring Quality in a Census, Part 2

  1. I read this article heading “Measuring Quality in Census” and it made me laugh. This census will be so far off if it was conducted the same around the country as it was in my neighborhood. I am not at all opposed to the census and was happy to contribute and fill out the paperwork the first time. I had one census worker show up at my door with a hand written packet and she asked me to fill it out. I was happy to do it and I sent it back in. A few days later another person stopped at my house to drop off another packet. This one had the address printed on it. I explained to the gentleman that I had already sent one in but he said that I still need to fill out his packet and send it in. I laughed and said that I would do it and just deduct 3 people from the census count everytime I see it. A few weeks pass and someone shows up at my door and says that they need to gather out information. I again explain that I have now sent in two packets and I think that this is now becoming a waste of time and resources. She was not going to leave without the information so it was given to her. I figured that would be the last of the visits. I guess I was wrong as I just had another census worker at my door a few minutes ago. I explained the whole thing to her and she said that she was told she needed to stop at my address and has to collect the information. I told her how frustrated I was with the whole system and then gave her the information again. All of this to ask the question how are we really measuring quality in the census? It seems to me that we are wasting money and peoples time. I now understand why people are against the census and refuse to take part in it. I started to wonder if the census workers are paid for each home that they collect information from because that is how it is starting to look to me.

  2. JC says:

    As an enumerator who has taken this task ‘very seriously’, I find that is not always the case with upper levels. I don’t know if this attitude comes from their managers or higher.
    Many of my co-enumerators felt the same as myself. Most of us believe in the process of the census, as well as the importance.
    We are now winding down the 2nd phase/VDC, after only 11 days. I personally have had numerous vacancies turn out to be occupied. With this in mind, I would suggest that the Bureau thoroughly recheck the V and D’s.
    Realizing that the maps provided were at times inaccurate, with streets having the same number with letters behind them, there were a miriad amount of reasons for these mistakes and oversights. When working with the ‘human factor’, things like this happen.
    But . . . the key here is to count people as ACCURATELY as possible and that should be our MAIN AND ONLY GOAL. Not worrying about being under budget,time constraints, not taking the ‘next person on the list’ to be in FSO or CL positions, whether they are capable or not, but to aim for QAULITY, QUALITY, QUALITY.
    I am grateful for the experience and of course, could suggest more efficient ways and actions, but overall thanks to the conscienctious enumerators out doing the most difficult work, people have been counted who wouldn’t have been.

  3. jhc says:

    I agree with JC about a lot of people that will fall through the cracks if not given a lot more time on vacants & deletes! Some areas of concern I have found that may be missed:
    -New home/apartment construction & conversions ( I have found a lot of these in my AA’s that never had map spots and enumerators passed over because they did not have EQ’s)
    -Vacant houses/apartments that were occupied April 1st but people moved and landlords/owners could not be reached or refused to give any information.
    -Housing in and around college towns where off campus students moved out in May before we could call on them
    -City housing in depressed areas that enumerators wrote off because of potential violence
    -Seasonal home owners/rentors that were never counted anywhere
    MY SOLUTION TO GET AN ACCURATE COUNT IS TO ASK ENUMERATORS THAT WORKED THESE AREAS FOR THEIR INPUT SO THEY CAN BE REVIEWED! Some office probe looking at EQ’s or some computer looking at EQ’s will not get you all the areas of concern.

  4. Bob Knows says:

    The census taker rang our doorbell, and then began pounding on the door with his fist before we could get to the front of the home. When we opened the door he became belligerent and gruff.
    The Constitution only authorizes Congress to require a count or “enumeration” of the people. All that other “survey” stuff is beyond Constitutional authority. We answered the questions politely about how many people resided at our location on April 1st, and then politely declined to provide information that the government will use for whatever other unknown purposes. The census taker became even more gruff and rudely exclaimed that he could get information from our neighbors and drove off.
    In the 1930s a previous Fascist regime made lists of people by category, Polish, Jews, Arians, etc. We do not trust the new American Fascist regime any more than the last bunch of fascists. What are you planning to do with the lists of people by race and ethnicity? What has the current regime been recruiting and training “Internment Camp Guards” for on the National Guard eimployment web site? Why has the government become the enemies of free men?
    We gladly participate in the census as much as it is authorized and required by our Constitution, to enumerate (count) the people. We decline to participate when a fascist state goes way beyond the Constitution to push racist and illegal (unconstitutional) programs.

  5. Brandon says:

    My post is being censored.
    My employment was extended through august… I have not received any communication from the Bureau and had to contact my Crew Leader to find out I would no longer be working with the Bureau due to random selection of participants of the next phase in my area.
    It would have helped if i had known back in June instead of getting a false extension…
    So I could file for unemployment and start looking for more work.

  6. DLS says:

    Actually, it’s not beyond Constitutional authority. Judges have found, again and again, that these additional questions are constitutional. See
    With respect to your comparing the census to fascist lists of individuals, I’d like to remind you that all data are held completely confidential. No one outside of the census can see your answers (although they will be public in 80 years or so). The national guard, IRS, postal service, even the president can’t see your answers. There are fines of up to hundreds of thousands of dollars that will be assessed for individuals (Census employees) who violate census confidentiality. They really do take it very seriously.

  7. J says:

    I have had much the same experience.
    First it was slow down, make it last and I was practically begging for more cases, then it was hurry up and finish. Keeping in mind the expected one done per hour on average. Which would have been fine if they gave enough out to do that. And I was hearing all kinds of rumors about some weird competition between the crew leaders to finish first. It was not supposed to be a race.
    Now in this second phase, we get a 5 hour training class and they say there is a lot of work, and the one per hour is out the door because it is harder to find a respondent on VDC, and here it is a week later and they have no more work for me. And I got the 3rd degree because I had a few hours more than cases. Oh, but they don’t want to take my badge and bag yet because something might come up and I am good.
    I too took it seriously. I wanted the right answers. I got stuck in some competitive issues with people worried about their own selves making it into another phase rather than getting it right. I never got anywhere near the promised 20-30 hours we were hired to do. I was honest and thourough and I feel disillusioned to say the least.

  8. Greco says:

    How and why did the VDC workload get cut from 15 million to 8 million in the course of a week? (Both numbers come straight from Dr. Groves.)
    Does anyone up there realize that tens of thousands of snowbirds will be counted in the wrong states or not at all because we’re not checking on three-quarters of our ‘vacants’? Sorry, Ohio. Too bad, Michigan. Better luck next time, Wisconsin.
    If it’s any comfort, we were on time and under budget.

  9. myrl bohn says:

    We have gotten 4 calls from the census wanting to double check our information the first 3 we hung up on thinking they were scammers. this last one I ask for her information an check the phone # it was leget I called back with my case number they wanted an interveiw of 10 minutes. I refused, we filled out the form returned it in due time. What is the problem! Why are you wasting our time, “money”. The government should be able to hire someone or group to complete this operation proficiently. my understanding was if we complied with the form in the first place we would not get a call in person or by phone. Please do it right,

  10. @myrl Please help us ensure the quality of the 2010 Census by responding to follow-up calls about the data collected on your form.
    We are currently following up with a small number of households to clarify answers about the number of people living at an address, based on answers they provided on the form.
    Please, if you are one of the small percent of homes visited during our quality assurance process for re-interview or verification, take a few minutes to help us ensure the quality of the 2010 Census.
    We are charged with counting everyone, once, and only once, and in the right place. We check and double check to make sure we get it right.
    Learn more:

  11. Dex Panthenol says:

    Sam, no doubt you are well-meaning and honest in your response to Myrl but that message is NOT getting across to the residents who are being irritated at apparently meaningless re-visits and phone calls after they have already cooperated.
    As a VDC enumerator I had an opportunity to speak to a multi-unit landlord (to use as a proxy) to re-verify a vacant apartment last week after several trips to the empty unit. (And zero useful information from the neighbors across the hall: “Four months ago? You’re asking if I know the exact date when those people moved out back in March or April? I didn’t even know their names.”)
    “Well,” the landlord said, “I thought I had seen all of you guys by now but you’re a new one.” When we were finished – he did cooperate – he asked (sarcastically) when (not if) we would be back again. I said we had been told that we were down to the last couple of days so maybe we wouldn’t be bothering him yet again.
    “I’ve heard that for the last two months.” was his parting remark.
    Maybe it’s time (or way past time) for the Bureau to take out some wide-coverage general-media advertising to let folks know why we keep trying to “get it right”, as you put it, and that they might hear from us again. Or not, if that’s finally true.

  12. Door Knocker says:

    During NRFU we were told to produce 1.7 CI’s per hour. In VDC it was 2.5 per day. No mention of quality, just quantity.
    If your daily D-308 didn’t have enough cases that were accepted to be near the target number it was always pointed out, especially toward the end of NRFU when we were putting 90% of the effort into closing the last 10% of the hardcore cases.
    At crew meetings we were sometimes shown spreadsheets so that each crew could see where they stood against other crews in the LCO.
    We had the same changing stories from week to week about how many binders were remaining, sometimes told “Lot’s left so hurry up and finish what you’ve got so we can get them”, then next week after we had turned in everything in a rush, “Oh, all gone”.
    I guess it’s over now. EQ’s are all gone for VDC and CL has gotten no word about the final QA-3 phase.

  13. Eugene G. Kovach says:

    T he 2010 Census Quality Survey is the most idiotic and senseless waste of resources that I have yet seen. I received the request for completion of the on-line questionnaire, which asked highly intrusive and identifiable questions, but the site was so poorly managed that waiting times for completion were ridiculous. At final copmpletion, I was advised that the heavy traffic required me to start over at a later time. This next effort, 2 days later, asked for the PIN assigned when I first logged on – which of course I had used and then promptly forgottten. End result: No response, close to an hour of wasted effort by me, I’m not dwelling on the cost of mailing letters telling me I was going to get a letter…
    All in all, whoever designed this should be fired.

Leave a Reply

Your email address will not be published. Required fields are marked *