Measuring Quality in a Census, Part 3

Print This Post Print This Post

Written by: Director Robert Groves

An earlier post, “Quality in a Census Part 2,” noted that we have two basic tools to evaluate a census – process-oriented indicators and comparisons to other methods of estimating the population size.

This is a post about some of the process-oriented indicators. I’ll talk about what operations and features of the census might be relevant to answering “how good is it?”

A recent initiative of the American Association for Public Opinion Research urges survey organizations to become more transparent about their process indicators; we at the Census Bureau support such transparency as a way to permit more open evaluation of methods.

As we finish the nonresponse follow up stage, we’re starting to get some indicators of how everything is going thus far.

All the indicators are preliminary at this writing and will change somewhat as our final operations are completed. But here’s how things are looking right now:

1. We used a short form only for all the approximately 135 million housing units. We finished the mailout/mailback phase with a 72 percent versus 69 percent participation rate in 2000 (combined short form and long form 2000 rate). The 2000 short form rate was the same as this year, the 72 percent figure I’ve cited in earlier posts.

2. For about 13 million units in areas with 20% or more Spanish-speakers, we send out a bilingual form; our preliminary analysis suggests that it increased the participation rate in those areas by about 2 percentage points over the English-only form.

3. For about 40 million units, disproportionately in hard-to-enumerate areas, we sent out a replacement form a couple of weeks after the first mailed form. It worked to increase the participation rate in these areas. The result was that we have less variation in participation rates in 2010 than in 2000.

4. We used new questions to identify households with dynamic membership and then recontacted them (about 7.5 million in total) to make sure we didn’t miscount them (in 2000 we did check out large households in this manner – about 2.5 million). We don’t yet know how many problems were resolved by this effort.

5. We updated the address list multiple times using different sources. As a result we had fewer “deadwood” listings (we deleted 4 million during our visits vs. 6 million in 2000). We also added fewer cases to the list when we did our field work. (This last point is a more ambiguous result, which could have arisen either because of a better address list or less diligent field work.)

6. We designed a more efficient assignment process in the nonresponse followup stage, so the miles driven per interview is less than in 2000; we are under budget on the nonresponse followup stage.

7. Despite this, reaching the nonresponse followup cases and getting their cooperation was harder this time; after failing in six tries to contact and interview units, we had to get counts of residents using informed neighbors and building managers relatively more frequently (currently about 5 percentage points more such reporting in 2010 versus 2000).

8. The percentage of occupied units that yielded counts of persons, one way or another, may be very slightly lower this year (about 98.0% in 2010 vs. 99.5% in 2000). We think both this finding and 7. above mirror the lower participation rates in surveys more broadly.

9. We’ve implemented a reinterview process whereby a portion of essentially every enumerator’s work was redone and checked against their original results. (In 2000 only about 75% of the enumerators’ work was subject to the reinterview.)

10. We found a smaller proportion of the enumerators failing to meet our quality standards than we did in 2000. (This, too, has multiple interpretations; we used much more consistent computer-assisted rules for determining violations than was done in 2000.)

11. We found many more vacant units when we went out for Nonresponse Followup than was true in 2000 (about 14.3 million vs. 9.9 million in 2000); that makes sense, given the widely publicized foreclosure rates. However, we need to know the April 1 residency status of units now vacant, so they pose challenges to us in our nonresponse followup.

12. Finally, in my professional experience with large data collection activities, problems during the data collection phase lead to missed deadlines and overruns of the budget. For this Census every operation since the Fall of 2009 has been on schedule and cumulatively we’re significantly under budget.

As these indicators are revealing themselves, some look better than the experience in 2000; some, not, as you can see above.

We’ll gradually be refining these results as our final quality assurance operations (the Vacant/Delete check, and Field Verification) take place. I’ll report them when we have them, especially those that show any changes from our initial insights.

Please submit any questions pertaining to this post to

This entry was posted in Quality Assurance. Bookmark the permalink.

20 Responses to Measuring Quality in a Census, Part 3

  1. says:

    when will you ever pay me for my services For the Riverside, CA office build out You guys owe $360,000.00 plus interest Paul A. Morales

  2. D@R!$ says:

    Sincerely… Puerto Rico…the operations were so awfully done, without quality, legacy and it isn’t the people out there, it is the people who work with Census 2010. Im telling this, because I work in an LCO. PRAO is the mirror of a bad administration and illegal. So, hopefully i have to say…good luck with the mission!

  3. Enumerator in the Field... says:

    Wow, sounds like the Census Bureau needs to send a team to the PR LCO to check on it’s operation.

  4. jhc says:

    My area is in a county of over 1,000,000 population that covers about 40 square miles. One LCO covers this area.
    Vacants so far from my working field quality assurance in about 8 AA’s I have covered is running about 70% negative. In other words I am finding people living April 1st in EQ’s that had previously had the status of “vacant” or “delete”!!! I know this may not be a representative sample but this is what I have uncovered so far. Most of the cases are in apartments where access may have been denied along with poor areas where workers safety may have challenged their effort. If you want to capture these vacants/deletes I would suggest a buddy system to those areas.

  5. chad says:

    I have a question that ties in with this post –
    I worked as an enumerator for NRFU and am now working as an enumerator for NRFU VDC, as I am driving from AA to AA I hear lifelock commercials that essentially accuse the Census operation of fraud and mismanagement of PII, mainly on Rush Limbaugh’s show, but also on a few others. Limbaugh has a 20,000,000 member audience but I have never heard anyone from the Census Bureau refuting any of his claims.
    I think pretty hard about that when I then have to go from house to house to find a proxy for a vacant because people won’t give me a name and phone number because they are scared we are going to lose or sell their information, or because they listened to Glenn Beck when he told them that all they have to provide is a population count and it is unconstitutional for us to ask for anything else. I know that over the course of the two operations at least 20 people have refused information because of that reason. It is especially irritating when the enumerator manual says that a respondent has the right to refuse any information but then we are sent right back out in the field with with the EQ because a respondent has refused a phone number when the chances are that the proxy won’t be able to provide the same level of information. (For that matter how are the Amish getting counted if no EQ can be completed without a phone number)

  6. Tom says:

    I recieved no form in the mail. When I called Regional Office I was told they were out of forms, had to wait for a visit. Called congressman, no luck. Now, no visit. I have called both the Regional and Field offices several times over last month, still no visit.
    Am I being excluded intentionally?

  7. Case worker says:

    Dr. Groves:
    I have talked to a few of my friends that are not in my state that are also part of the NRFU-VDC and some confusion does exist on EQ’s we are given to call on.
    The EQ’s in this operation I have seen so far fall into:
    1-Vacants(previous NRFU operation)
    2-Deletes(previous NRFU operation)
    3-New EQ’s (no case record of NRFU operation)
    4-Completed case ID’s(no NRFU previous because they had been received)
    We are asked to treat them as a new case and find the status as of April 1st along with verification of proper address. This seems to be a very important operation with us finding a lot of people that were not counted.
    The only area of concern is #4 on why or how these were selected? Were they
    -never received
    -not filled out properly
    -random selection for quality
    -recorded improperly
    Just a few question.

  8. Tom,
    You may call 1-866-872-6868 and provide your information over the phone through July 30. Note that this summer, we will visit addresses that were added to our address list too late to receive a form, so it is possible you may still be visited by a census taker.

  9. Justin boots says:

    nice work and very considerate. keep up!

  10. noyb says:

    I received a D-26 note, after receiving multiple postcards in the mail, reminded at the subway stop what to do, and TWO copies of the census form, the first of which I returned to the Census Bureau On Time, in a United States Postal Service mailbox.
    What I want to know is why every bill I mail in to ConEd, and every tax form to the State and City, and every DVD I mail to Netflix arrives, sometimes the next day to the recipient. And yet, the Census Bureau LOSES my form, and those of other friends I know? If this form is So Very important to you, and you have a quantitative idea of how many forms will be returned based on a list of residences for any given location, why would you lose the forms sent back to you? Is it an intentional inefficiency to create more work for the underemployed census workers?
    I strongly suggest a DIGITAL ONLINE VERSION of the form so that the data for those of us who are computer savvy will not be lost, will be automatically tracked, and you can save TAXPAYER MONEY and hire fewer census workers (who generally are coming to the doors of people who are at work anyway, or who do not answer their door for strangers)…. and Fire the people who lost my form to begin with.
    I am truly not interested in my tax dollars going to such an inefficient operation as this. This is 2010, not 1930. Evolve.

  11. Concerned Enumerator in the Field... says:

    Well we were just 2.5 weeks into the NRFU-VDC operation and we received a call that the LCO had a shake-up and all the AA Binders and EQ’s needed to be turned in TODAY and sent back to the LCO and we still have Cases (locations)that need to be visited!!!! So what’s up? Will we get called back to complete the Cases that are still open, what do I tell a resident when they call me in the next few day’s because I left a Notice-of-Visit on thier door?? Sure hope I hear back from my Crew Leader soon…..

  12. Concerned Enumerator in the Field... says:

    Good News, I got a call to pick-up my AA Binders and EQ’s…. so I’m back in the field. Not sure yet what happened, but I’m glad to be back at work on the NRFU VDC operation. I want to see the 2010 Census to the end. I sure might be interested in doing this again for the 2020 Census.

  13. Greco says:

    Dr. Groves, I wish I could tell if you know the real situation in the field. But if quality was as important to the regional offices as you would like us to believe, the LCO managers’ mailboxes wouldn’t be full of ‘horse-race’ e-mails telling us which offices and regions are ‘winning.’
    I doubt there’s been one single day since NRFU began that our managers haven’t gotten a pep talk or a warning accompanied by a spreadsheet ranked by % complete. I doubt there’s been a single day that an area manager hasn’t put the fear of God and unemployment into an ‘underperforming’ office.
    Quality doesn’t get a tenth the emphasis that speed does. And coming in under budget and ahead of schedule means nothing if the data is as riddled by holes and errors as this year’s will be.

  14. Ezekias says:

    The reinterview idea seems like a good one. You have to have tried tracing your ancestry to understand the importance of getting yourself counted in the Census every ten years, whatever you have to do. Hopefully, you are fortunate enough to have had ancestors who participated in their Census, so that you are able to research your family tree effectively. Otherwise, your job will be more difficult. Think of your descendants and how they may in the future attempt to research your family tree.

  15. #1Enumerator says:

    I have absolutely loved being an enumerator!! Will the Census Bureau be conducting an evaluation using focus groups and other qualitative and quantitative measures to survey enumerators? I believe our feedback from field work would be valuable for 2020!

  16. Pilose says:

    I, too, was a NRFU and VDC enumerator. I had a surprising number of people tell me that the questions were unconstitutional, that the President is a communist and the Census questions are part of some plot, that “the government doesn’t need my phone number (or age or race or whether I have a mortgage)”, and one woman who told me that she was “not supposed to talk to me” because she “heard it on the news”. I tried to explain to her that the news programs were probably telling her that she IS supposed to talk to me but she insisted. LOUDLY.
    She closed the door in my face after telling me that she had to “talk to my lawyer” before she could talk to me.
    I returned a number of times (and, often, could hear people inside or spotted them peeking out through the blinds) but no luck. Eventually I used a proxy and got minimal data but likely not what the Bureau had hoped for.
    Chad’s comment (above) about countering some of the anti-Census radio blather might have been worth a try. It felt awfully alone out there.
    PS. My crew leader’s response was to “just try again” because the LCO would not accept refusals until we had been turned away at least 3 times. Maybe this level of support from our management has something to do with the Director’s points #7 and #10 (above).

  17. Pilose says:


  18. Door Knocker says:

    YES! But include the worker bees, not just the bosses.

  19. Field Enumerator... says:

    Yes, for sure don’t forget the enumerators in the field who walked and visited every location.

  20. Field Enumerator... says:

    Well, I got a call on Friday night that they needed me for a added operation, called NRFU Residual Followup starting Aug 12th thru Aug 25th, going to check on the EQ’s that were coded POPULATION UNKNOWN- 99. Might be difficult talking to people again…but I’m up for the challenge. Thanks 2010 Census for calling on me, I must be doing a good/quality job.

Leave a Reply

Your email address will not be published. Required fields are marked *