Stumbled across this site recently. Argh.

http://honestygap.org/what-is-the-honesty-gap/

The site is well designed, professional, and the statistics and analysis look convincing. And they get their NAEP score analysis almost exactly wrong. The purpose of the site is to argue for this claim:

“Frequently, states’ testing and reporting processes have yielded significantly different results than the data collected and reported by the National Assessment of Educational Progress (NAEP). The discrepancy between NAEP, the Nation’s Report Card, and a state’s claim is what can be described as an “honesty gap.”

They go on to provide state by state evidence for this supposed Honesty Gap:

The implication is that states are lying about student proficiency because their proficiency rates are so much lower than the NAEP proficiency rates. BUT this analysis, and this website, doesn’t include a crucial detail (and I suspect they know about this detail and they purposely fail to include it): the way the NAEP developers use the term “proficient” is VERY different from the way that term is used on the state achievement tests they are comparing NAEP scores to. Here’s a summary (from this Washington Post article) about this important difference:

“Oddly, NAEP’s definition of proficiency has little or nothing to do with proficiency as most people understand the term. NAEP experts think of NAEP’s standard as “aspirational.” In 2001, two experts associated with NAEP’s National Assessment Governing Board (Mary Lynne Bourque, staff to the governing board, and Susan Loomis, a member of the governing board) made it clear that:

‘[T]he proficient achievement level does not refer to “at grade” performance. Nor is performance at the Proficient level synonymous with ‘proficiency’ in the subject. That is, students who may be considered proficient in a subject, given the common usage of the term, might not satisfy the requirements for performance at the NAEP achievement level.'”

So the “Honesty Gap” argument falls apart before it begins: you can’t compare “proficiency rates” between the NAEP test and state tests, because the term “proficient” is defined very differently. I’ve seen this same argument on other sites (often promoting charter schools) and it’s just wrong wrong wrong. The only Honesty Gap the site argues for is their own dishonest use of NAEP data. Knock it off, please.