Sunday, August 16, 2015

FCAT Reading Scores - Only two of the five failure factories actually fail

The Times' story has focused on FCAT reading and math scores as a metric of school performance.  There's a variety of reasons why I, and many others, would argue that exam scores aren't always the best metric of student or school performance, but that's beyond the scope of this blog.  If that's the benchmark the Times' has set, then I'll run with it.

The Times has suggested that the five elementary schools in question are failing their predominantly African American student populations as measured through test scores that are below acceptable levels, and just looking at the FCAT pass performance of these schools gives you an idea why.  Math and reading scores are noticeably lower than for other elementary schools in the district, as well as others in the state.

Here's where things get tricky, though.  We know that each of these schools is characterized by two distinct trends: 1) higher than average percentage of Black students, and; 2) communities with higher than average levels of poverty.  It so happens that we also know that both of these factors are statistically related to lower test performance.  Let's not get into the reasons why right now, but rather agree that this is a universal trend across the United States, and not a problem specific to Pinellas or St. Petersburg.

We would expect a school comprised of a larger than average percentage of Black students whose families earn less than the mean U.S. household income to have FCAT scores below the average for all schools.  That's not to say we expect less of these students, or should ever feel comfortable with this, but that's the nature of the beast.  Statistics can't show racial preference, they can only show what is.

Fortunately, we have a way of statistically controlling for the effects of race and income as they relate to exam performance.  In fact, we can even predict where a school's exam scores would be expected to be based solely on the proportion of Black and impoverished students taking the exam.  We do this through what we call multivariate regression modeling.

The Times claims that the five schools in question are failing their students at a disproportionately large rate.  For this to happen, we'd need to calculate what we should expect the school's FCAT passage rate to be based on factors outside the control of the school.  Schools that perform below this prediction may in fact be contributing to decline, whereas schools performing at or above their predicted exam passage rate likely aren't doing anything to harm students - they may even be helping them overcome these outside factors.

To test this, I looked at the percentage of 4th graders earning a 3 or above on FCAT reading for 2014 for each elementary in Pinellas County, minus charter schools and those with too few students tested to include.  All told, this resulted in 74 elementary schools included in the sample.

The purpose of my regression analysis was to predict FCAT reading scores based on the percentage of students taking the exam who were Black, on a free or reduced lunch program (our proxy for income), or who were ESE excluding gifted (exceptional student education - think special education/learning disabled).  If you're going to make an apples to apples comparison across schools, you have to hold each of these three factors constant.  Otherwise you're asking some schools to start the race thirty second after everyone else, while still expecting them to finish with the pack.

Here are the results of the regression model:


Let me help you interpret this if you're not one who regularly reads regression outputs.  First, all three factors tested were statistically related to the percentage of 4th graders earning a 3 or above on the FCAT reading exam at a 95% confidence level.  That is, we're incredibly certain that the race, income and special education status of a school's 4th graders influences their scores.

The actual impact of each of these factors can be seen in the column labeled "B".  This is the beta coefficient, which tells us what happens to the percentage of students earning a 3 or above when we adjust the independent variable by 1-point.  That can be a bit tricky to wrap our heads around with percentages, so let's just move the decimal point one place to the left and say the following:

  • As the percentage of Black fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 2%
  • As the percentage of free or reduced lunch fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 8%
  • As the percentage of ESE fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 3%
What this tells us isn't a surprise - race matters less than income or special education student status, but all three still matter independently of one another.  So how do we predict based on this output?  Simple - we use the regression equation, which multiplies the actual value for each school by their respective beta coefficient, adds those up (plus the constant), and comes up with a predicted dependent variable value (% earning 3 or above).  So let's run those equations and see what we get.

% FCAT Reading 3 or Above

Based on these results, only two of the five "failure factories" performed worse than we would have otherwise expected controlling for the communities in which they reside - Fairmount Park and Maximo.  The school with the lowest predicted pass performance was Melrose, with a fourth grade student body that is 95% African American, 82% free or reduced lunch, and 18% ESE.  Based on these disadvantaging factors, we'd expect only 11% of 4th graders to earn a 3 or above on FCAT reading.  In reality 18% did.  Lakewood also performed noticeably above expectations, while Campbell Park performed at a level pretty close to where they'd expect them to be.

One more thing to point out about the regression - it has an r^2 value of .724.  In equally confusing speak, what this means is that the model accounts for 72% of the variance in school level FCAT performance.  That is, factors not directly related to teaching or curriculum nearly three-quarters of a school's performance on the FCAT exam.  

What did we learn?

Based on this initial test of FCAT reading scores of 4th graders in Pinellas, it seems that two schools - Fairmount Park and Maximo actually perform less than would be expected of them, while two other "failure factories" actually outperform (statistically speaking).  Campbell Park is generally right where our model said it would be.

Are these five schools failing their students?  By the Times' benchmark, only two seem to be - at least when looking solely at Reading scores (I'll run the same analysis with the math scores for a later post).  Lakewood and Melrose, on the other hand, actually seem to be doing a pretty good job on FCAT reading, considering their student demographics.  Perhaps not where they'd like to be, but performing at a level that the Gods of statistics would say is commendable.

No comments:

Post a Comment