Background and Summary
Based
partly on tradition and partly on the requirements of state
and federal legislation, including that of No Child Left
Behind, achievement testing in mathematics and reading is
administered in nearly every state. They report on every
district and every school within their public education
systems. Separately, at the federal level, there is the
National Assessment of Educational Progress (NAEP)- also
known as the Nation’s Report Card. Important
characteristics of the NAEP include:
•
A long track record of 39 years, establishing itself as the
defacto national standard for achievement.
• The NAEP achievement level of
“proficient” defines what it is to be at
“grade level.”
• Content standards tested by the NAEP are drawn from
the National Assessment Governing Board.
• NAEP tests children in 4th
and
8th
grades
and reports proficiencies at the state level.
• In 12th
grade
NAEP tests and reports only at the national level.
• The use of statistical sampling techniques prevents
reporting at district and school level.
The tests administered by the states also claim to
determine which students have become
“proficient” in the tested subject areas.
Generally, the state administered achievement tests:
•
Examine sufficient numbers of children to report
statistically valid results for schools and districts.
• But measure against content standards and use
cut-scores inconsistent with the NAEP.
• Most states significantly inflate the number of
children deemed “proficient” compared to the
NAEP.
• Inflation is always accompanied by differential
inflation in which disadvantaged groups suffer markedly
higher levels of inflation than the state average. The
cynic views this as optimally applying the "cosmetic" where
it is needed the most to cover up the most dysfunctional
groups. Others, bent on helping such children, seek
realistic proficiency measures to show their dire needs.
• Some grossly inflate their proficiencies to as much
as 300% or more above the NAEP results. The median state
inflates at about 100%.
• In three New England states, where the NECAP test is
used, a number of problems have been identified. We believe
the NECAP has too many flaws. Rather than overhauling this
assessment regimes we think replacing it makes more sense.
Click here to
learn more- including a speech about its dysfunctional
nature.
For stakeholders at the school and district levels, the
well-respected NAEP results are not available and the state
generated results are unreliable, due to the inflation. To
address this lack of useful information:
•
We developed a conversion formula (mapping procedure) to
estimate what the NAEP proficiencies would have been in
these localities. It essentially removes the inflation from
the state reported scores.
• Error measurements of this procedure in simulated
environments have shown good accuracy.
• Where actual results are known independently we have
verified that they are reasonably accurate.
• The method’s errors are measured to be of the
same order as the NAEP reported sampling errors.
• In terms of proficiency percentages, the measured
errors have rarely exceeded 5%.
By applying these mapping procedures we have constructed
report cards for every public school and every school
district within the states of Oklahoma, New Hampshire, New
Mexico, Pennsylvania, Rhode Island, Tennessee and Vermont.
Additionally we have studied Ventura County in California,
Bristol County in Massachusetts, and Hudson County in New
Jersey. Our findings include:
•
In the worst performing schools, less than 5% of children
are proficient in both reading and math.
• Among the “best” performing schools, it
is rare to find more than 70% of the students proficient.
• Mathematics proficiency percentages are generally
lower than those for reading during the high school years.
Our
evaluation of the analysis conducted thus far suggests:
•
In all jurisdictions studied,
social promotion is the key dysfunction leading
to the low proficiencies.
• Age based group instruction hinders the ability to
properly “place” students consistent with skill
levels.
• Strict retention policies, such as used in India,
could correct and eliminate social promotion, but we think
such methods are too draconian.
• However, replacing group instruction with self-paced
(probably online) instruction would be effective.
• Reform the math curriculum first and extend it to
all math courses. A few such efforts are underway,
including our own Stellar Schools
project.
Asora®
Report Card Services:
We are
ready to work with interested clients who seek to obtain
more realistic estimates of student proficiencies in the
K-12 public schools within their states or in other client
defined regions.
To conduct its analysis, Asora Consulting uses as its input
data:
•
Recently published results from the Nation’s Report
Card- also known as the National Assessment of Educational
Progress (NAEP), which it obtains from trusted data
repositories or from the U.S. Department of Education
directly.
•
Recently published results from the state’s
achievement test as reported by the states directly or
through the Council of Chief State School Officers' data
repository at
SchoolDataDirect.
District Proficiencies
After
applying our mapping procedure to the input data, we can
generate estimates as to how students would have performed
on the Nation’s Report Card on a district-by-district
basis. We generally provide estimates at the primary school
level (4th
grade),
the middle school level (8th
grade)
and at a high school grade level (which varies from state
to state according to the level tested by the state’s
examination). Results are presented in terms of the
percentages of children measured to be proficient or
better. According to the definition of proficiency used by
the NAEP it also denotes the percentage of students who are
at or above grade level. Given the fact that proficiency
levels are nearly always “inflated” by the
state administered examinations, the proficiency estimates
we generate are low and troubling. Former President
Bush’s remark about educators being engaged in
“the soft bigotry of low expectations” is
consistent with these estimates.
School
Proficiencies
In a
similar manner we can produce estimates for individual
schools. These estimates are useful in showing the degree
to which schools are dysfunctional. It is not uncommon to
find schools with proficiency percentages well less than
5%. We believe that a properly operated school will
maintain student proficiencies above 90%. However, among
the public schools we have checked none has ever reached
that level.
Private
School Proficiency Estimates
In most
states, private school students do not take the
state’s achievement tests. Where they do, we can
generate NAEP scale estimates for them in the same way we
generated the public school estimates. For those schools
that do not take the state’s achievement tests but
take the SAT or ACT tests we can make a rough estimate of
the 12th
grade
proficiency levels of those schools. For example, we
performed such an analysis in Rhode Island and found only
two private schools for which we estimate proficiency
percentages above 90%. Many private schools do not release
their SAT or ACT scores. In those cases we are unable to
make estimates.
Commissioned
Reports
Depending
on the client’s needs and wishes we can provide
reports based on the performance estimates generated. The
report could range in length and complexity from:
• A
short description of the findings with selective
presentation of the estimates.
• To a more in-depth analysis of the findings with a
complete presentation of the estimates generated.
We will work with clients to produce reports suitable for
their needs.
Clients will hold the copyright to reports generated for
them and will be permitted to revise and extend them
subject to the requirement that Asora Consulting will be
cited for its contributions.
Some
Examples
We have
analyzed student proficiencies in Oklahoma, New Hampshire,
New Mexico, Pennsylvania, Rhode Island, Tennessee, and
Vermont where we provided NAEP estimates for every public
school in these states. We also looked at a small number of
private schools for which SAT scores were available.
Many of the reports resulting from these studies can be
downloaded from our Reform Reports
page.
Methodology
The
derivations of our mapping method can be reviewed in the
report ELQ-Mappings.docx
and its
accompanying spreadsheet ELQ-Derivation.xlsx
which
are also downloadable from our Reform Reports
page.
__________________
If you are interested in reviewing the "big picture" of
Asora's full breadth of activities, click here to reach our
home page.