Tuesday, August 18, 2015

Kriseman has the right idea

Since posting this analysis, I've regularly heard from people that we shouldn't be "pointing fingers" or "casting blame" but rather working towards solutions on the issue of poor performing elementary schools.  I hate reading this, because as social scientist, I'm not interested in casting blame - I'm looking for the source of the problem.  By this same logic, we shouldn't waste our time trying to find where a leak is coming from, we should instead work on soaking up the water.  It would be like a physician treating symptoms without concerning him or herself with what was causing the symptoms.  Doesn't work for education either.

My analysis has suggested that, at least for three of the schools the Times calls "failure factories," the source of the problem might not be the school.  I haven't said what it might be simply because my research doesn't support a source - it only supports the notion that the schools, in some cases, might not be it.

With that in mind, I actually think Mayor Kriseman is on the right track by proposing an education director - a liaison to work with Pinellas County Schools and the communities to see what can be done from within the community to improve student performance.  I know that Bill Foster formerly had one in Lori Matway who I worked with during the four years I spent as a scholarship committee member for the Pinellas Education Foundation.  I wasn't aware that the position had been vacated, but I'm at least glad to hear that Kriseman has a plan to fill it.  This at least recognizes that the community has at least a part in the success of our students - all students, and that we should be doing more to foster growth for at risk students.  It's the reason I worked on behalf of the Pinellas Education Foundation, and the reason I've taken up this blog.

Perhaps this will convince Janelle Irwin to reconsider the box I believe she's mistakenly placed me into.

Monday, August 17, 2015

Response to Janelle Irwin

SaintPetersBlog's Janelle Irwin reviewed my blog postings today and wrote a critical, yet thought provoking response to my contentions.  There were a few things I found a bit unfair, which I will mention here shortly, but for the most part I was glad she took up her case and contributed to the dialogue on this topic.  Despite what Janelle may have taken away from my posts, I believe that she and I are probably equally concerned about the issue of poor academic achievement in St. Petersburg's urban, low-income core, even if we see it as resulting from different causal factors.

First off, a compliment - kudos to you Janelle for raising three kids and choosing to continue your education - that's no easy task no matter how you look at it.  As a an education advocate, I was happy to read this.

In response, I'll start with a minor point, yet one worth mentioning.  Your blog title begins "Dad blogger argues...".  As you mentioned in your post, I am a market research professional.  I also offered up that I am nearing completion of Ph.D. studies.  Any of those would have seemed a more appropriate descriptor.  As a feminist, I would never describe a professional female writing on a subject for which she has professional expertise as a "mom blogger."  I believe if you think about that, you'd agree.  No big deal, I think we're just all trying to improve in this capacity.

As far as the content we disagree over, let's start with this idea of "resegregation."  You appear to be of the belief, as the Times authors are, that by shifting back to the traditional neighborhood or community school model in Pinellas is responsible for lower FCAT performance at the schools listed in the article.  Well you're right.  In fact, I don't disagree with this at all.  Of course increasing the proportion of at-risk students at a school is going to reduce test performance.  Statistically, there's absolutely no reason why we would expect a school that is now comprised of a larger share of at risk students not to see score declines.

The better question is, were we just shuffling deckchairs on the Titanic by moving them around in the first place?  If you took all of baseball's best players and randomly redistributed them across every Major League team, you'd see greater parity.  Did moving a poor Black student to a better performing school have a positive  impact on their personal exam performance?  This would be a good question to ask, but it's beyond the scope of what we're talking about here since we're evaluating schools, not individual students (as we probably should be in the first place).

Second, I was disappointed that you didn't attack my specific assertions as presented by the data.  I hold two political science degrees and taught the subject for nearly ten years.  As a political science student yourself, I would have hoped you would have approached one of your faculty advisers and developed a plan to dispute my data or data assertions.  As I've mentioned before, I welcome this solely because I hope I'm wrong.  I hope the issue is the schools and not some community level factor outside of their controls - because if it is a problem with the schools, it's a lot easier to fix.

Finally, I'll suggest that my reasoning isn't as mind-boggling as you suggest.  Schools are an easy scapegoat.  My principal research areas in my Ph.D. program is community colleges, and it's a phenomenon there as well.  Politicians and community leaders regularly point to community college leaders and questioning why they haven't solved the employment gap.

Is it that large of a stretch to suggest that a child who comes from a home where the parents (but usually parent) are working multiple jobs, shares a residence with several family members or friends who come and go at all hours of the day/night, or is exposed to violence, drug use, or other social ills associated with poverty, might be at a disadvantage before they even show up for school?  A friend who formerly taught inner city once asked me in a state of despair, "how am I supposed to prepare a kid for an exam when they haven't eaten since lunch the day before?"

I've presented data that suggests that three factors account for 75% a school's FCAT performance, and that none of those three factors are things the school can control.  Is it fair to go into a school like Melrose that is seemingly comprised of the largest share of at risk students in the entire state (have to double check this, but they're at least within a few spots), and expect the faculty and leadership of that school to produce exam scores that are on par with the average performance of every other school in the state?  Absolutely not - it's entirely unfair.  Is it reasonable to expect them to grow their scores in a manner that shows statistical improvement over past performances?  Absolutely.  Just not when you're benchmarking against a period in which their students weren't nearly as "at risk".

As you mention, I consider myself a liberal leaning individual.  It's why I cringed when you wrote that I implied that I wouldn't want to send my kid to a school comprised of mostly Black students.  The truth, Janelle, is that I don't want to send my kid to any school other than the public school in their neighborhood.  That includes magnet or fundamental programs.  I don't want my kid to be bused to a magnet program because I believe in the logic and value of neighborhood schools.

What should we do?  That's a bigger question than my research is ready to address, if I'm just being honest.  I do believe that reducing poverty and improving job prospects in these communities will improve the quality of students attending these schools.  You'd essentially be eliminating some of the most important factors that make at-risk kids actually at-risk.  But I don't have the solution that problem, and I don't think you do either.  I don't think our politicians do, or social commentators.  It's truly a plague on our communities that is affecting our schools.  But if you're asking me if we should ax an entire school administration and expect things to all of a sudden turn around, well that's a data question I can answer.

Janelle, things "are that bad" at two schools the Times has written about, as I discuss in my blog.  No doubt about it - they perform way below their expectations.  Go in and make massive changes - my data seems to suggest that they should perform at a certain level given teachers who teach and administrators who lead at a level commensurate with their peers at other schools.  They're not performing at the level, which suggests significant change is necessary.

The situations that "aren't that bad" are for the three others schools the Times calls "failure factories."  Yes, only 13% of 4th graders pass the FCAT math exam at Melrose.  That's not good enough, and we should be ashamed.  But my data tells me that a school comprised of that many impoverished, special education, African-American students would only pass 5% of students if the teachers and administrators were performing "on average."  The 25% of variance we can't explain with the model is why Melrose passes 13% instead of 5%, and a very solid explanation is that the teachers and administrators there are actually responsible for this improvement over statistical expectations.

No, I do not sit back and shrug my shoulders and say "ahh, what can we do?".  Instead, I'm sitting here saying "maybe the schools aren't the problem."  You can call me wrong on this regard all day long, but please don't take me to be someone who doesn't care, or who isn't committed to seeing these students perform better.

Visual summary of model results

I've tossed a lot of numbers at you thus far, so it's only fair I summarize the findings to this point in a single, visual graphic.

For those who haven't read the posts below, I've evaluated 4th graders at each of the five "failure factories" not based on their raw FCAT performance as a percentage of students achieving a satisfactory grade of 3 or above, but rather based on their predicted performance when controlling for three factors that account for nearly 75% of FCAT performance: 1) percentage of African-American students; 2) percentage of free or reduced lunch students, and; 3) percentage of ESE (exceptional student education, minus gifted) students.

The plots below reflect deviation from where the regression model predicts each school should perform based on district averages for nearly all Pinellas elementary schools (I excluded charters and a couple of schools with exceptionally low totals of 4th grade test takers).  The values are percentage point deviations, so a school that passes 10%, though is expected to pass 15% would show as a "-5".

To summarize what we've found to this point, the Times seems to have two schools nailed.  Maximo and Fairmount definitely underperform on both FCAT reading and math when controlling for the handicapping factors detailed above - by quite a bit.

According to the model, the Times has unfairly pegged Melrose, which sees a larger proportion of its 4th grade students achieve a 3 or above on both the reading and math exams than the model would say we should expect.

Two other schools, Campbell Park and Lakewood, have mixed results.  Each beats model expectations for reading, while coming short on Math.  Lakewood makes the best case for rejecting the "failure factory" label, while clearly Campbell Park has some ground to make on its math performance.  Failure factories?  Jury is still out on these two.

FCAT Math Results - Still only two failure factories

So earlier I gave an explanation of my method for evaluating 4th grade FCAT reading scores and mentioned that I would repeat the process with the FCAT math scores.  Here are the results of this statistical model:

Again, all three variables affect the percentage of a school's students who earn a 3 or above on FCAT math, though the effects differed somewhat from reading.  Here the percentage of ESE students was actually the dominating variable.  Special education students had a more difficult time with FCAT math than they did FCAT reading.  Otherwise, the results were similar to before - poverty, as measured through free and reduced lunch percentage had a larger impact than race, but both mattered.  The model's r^2 value was .746, meaning the these three factors explain 75% of a school's FCAT math performance.

So how did the schools do compared to their handicap?  Let's take a look:

The two schools we could justifiably say were failing, Fairmount Park and Maximo, showed similar poor performances in FCAT math scores relative to their predictive performance when controlling for student demographics.

One school, Melrose, beat expectations.  Again, Melrose's fourth graders are 95% African-American, 82% free or reduced lunch, and 18% ESE - by far the highest ESE of the five schools, which is why their predicted pass range for FCAT math is onlyl 5%.  I'm sure the Times authors and the media, and perhaps even District leaders would say that 5% is a poor performance benchmark.  Maybe so, but given Melrose's student body, that's what it is.  13% of Melrose fourth graders earned a 3 or above, meaning they beat their true benchmark by 8-points.

Although Campbell Park performed on par with expectations for FCAT reading, their math score was sub-par by a small, but noteworthy amount.  We predicted that 20% of their fourth graders would earn a 3 or above when, in actuality, only 14% did.

Lakewood had beat its FCAT reading expectations by 10-points, but was pretty close to flat in FCAT math, coming two percentage points shy of its expected performance of 19% of students earning a 3 or above.  Cause for concern and attention?  Yes.  Is it a "failure factory?"  The data doesn't support that claim.

What did we learn?
The data still supports the contention that only two of the five schools are potentially "failure factories" - Fairmount and Maximo.  Campbell Park and Lakewood need to work on their FCAT math, but they did well enough on FCAT reading against expectations to believe that they are within striking distance of performing to expectations.  Melrose, the poorest, most African-American school in the lot, exceeded its statistically determined performance benchmark again in Math.

What's next?
Next up on the analysis slate is understanding whether or not Black students are truly worse off in Pinellas than other parts of the state when controlling for the demographics we've included in this model.  Will begin working on that model as soon as I get home tonight.

Sunday, August 16, 2015

FCAT Reading Scores - Only two of the five failure factories actually fail

The Times' story has focused on FCAT reading and math scores as a metric of school performance.  There's a variety of reasons why I, and many others, would argue that exam scores aren't always the best metric of student or school performance, but that's beyond the scope of this blog.  If that's the benchmark the Times' has set, then I'll run with it.

The Times has suggested that the five elementary schools in question are failing their predominantly African American student populations as measured through test scores that are below acceptable levels, and just looking at the FCAT pass performance of these schools gives you an idea why.  Math and reading scores are noticeably lower than for other elementary schools in the district, as well as others in the state.

Here's where things get tricky, though.  We know that each of these schools is characterized by two distinct trends: 1) higher than average percentage of Black students, and; 2) communities with higher than average levels of poverty.  It so happens that we also know that both of these factors are statistically related to lower test performance.  Let's not get into the reasons why right now, but rather agree that this is a universal trend across the United States, and not a problem specific to Pinellas or St. Petersburg.

We would expect a school comprised of a larger than average percentage of Black students whose families earn less than the mean U.S. household income to have FCAT scores below the average for all schools.  That's not to say we expect less of these students, or should ever feel comfortable with this, but that's the nature of the beast.  Statistics can't show racial preference, they can only show what is.

Fortunately, we have a way of statistically controlling for the effects of race and income as they relate to exam performance.  In fact, we can even predict where a school's exam scores would be expected to be based solely on the proportion of Black and impoverished students taking the exam.  We do this through what we call multivariate regression modeling.

The Times claims that the five schools in question are failing their students at a disproportionately large rate.  For this to happen, we'd need to calculate what we should expect the school's FCAT passage rate to be based on factors outside the control of the school.  Schools that perform below this prediction may in fact be contributing to decline, whereas schools performing at or above their predicted exam passage rate likely aren't doing anything to harm students - they may even be helping them overcome these outside factors.

To test this, I looked at the percentage of 4th graders earning a 3 or above on FCAT reading for 2014 for each elementary in Pinellas County, minus charter schools and those with too few students tested to include.  All told, this resulted in 74 elementary schools included in the sample.

The purpose of my regression analysis was to predict FCAT reading scores based on the percentage of students taking the exam who were Black, on a free or reduced lunch program (our proxy for income), or who were ESE excluding gifted (exceptional student education - think special education/learning disabled).  If you're going to make an apples to apples comparison across schools, you have to hold each of these three factors constant.  Otherwise you're asking some schools to start the race thirty second after everyone else, while still expecting them to finish with the pack.

Here are the results of the regression model:

Let me help you interpret this if you're not one who regularly reads regression outputs.  First, all three factors tested were statistically related to the percentage of 4th graders earning a 3 or above on the FCAT reading exam at a 95% confidence level.  That is, we're incredibly certain that the race, income and special education status of a school's 4th graders influences their scores.

The actual impact of each of these factors can be seen in the column labeled "B".  This is the beta coefficient, which tells us what happens to the percentage of students earning a 3 or above when we adjust the independent variable by 1-point.  That can be a bit tricky to wrap our heads around with percentages, so let's just move the decimal point one place to the left and say the following:

  • As the percentage of Black fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 2%
  • As the percentage of free or reduced lunch fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 8%
  • As the percentage of ESE fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 3%
What this tells us isn't a surprise - race matters less than income or special education student status, but all three still matter independently of one another.  So how do we predict based on this output?  Simple - we use the regression equation, which multiplies the actual value for each school by their respective beta coefficient, adds those up (plus the constant), and comes up with a predicted dependent variable value (% earning 3 or above).  So let's run those equations and see what we get.

% FCAT Reading 3 or Above

Based on these results, only two of the five "failure factories" performed worse than we would have otherwise expected controlling for the communities in which they reside - Fairmount Park and Maximo.  The school with the lowest predicted pass performance was Melrose, with a fourth grade student body that is 95% African American, 82% free or reduced lunch, and 18% ESE.  Based on these disadvantaging factors, we'd expect only 11% of 4th graders to earn a 3 or above on FCAT reading.  In reality 18% did.  Lakewood also performed noticeably above expectations, while Campbell Park performed at a level pretty close to where they'd expect them to be.

One more thing to point out about the regression - it has an r^2 value of .724.  In equally confusing speak, what this means is that the model accounts for 72% of the variance in school level FCAT performance.  That is, factors not directly related to teaching or curriculum nearly three-quarters of a school's performance on the FCAT exam.  

What did we learn?

Based on this initial test of FCAT reading scores of 4th graders in Pinellas, it seems that two schools - Fairmount Park and Maximo actually perform less than would be expected of them, while two other "failure factories" actually outperform (statistically speaking).  Campbell Park is generally right where our model said it would be.

Are these five schools failing their students?  By the Times' benchmark, only two seem to be - at least when looking solely at Reading scores (I'll run the same analysis with the math scores for a later post).  Lakewood and Melrose, on the other hand, actually seem to be doing a pretty good job on FCAT reading, considering their student demographics.  Perhaps not where they'd like to be, but performing at a level that the Gods of statistics would say is commendable.

Background on Failure Factories/Inquiry

In August of 2015, the Tampa Bay Times ran an investigative story titled "Failure Factories."  In it, the authors claim that Pinellas County Schools, through either malicious intent or gross negligence, instituted policy changes that led to a substantial decline in the academic performance of five specific elementary schools in St. Petersburg: 1) Campbell Park Elementary; 2) Fairmount Park Elementary; 3) Lakewood Elementary; 4) Maximo Elementary, and; 5) Melrose Elementary.  The five schools each reside in predominantly African-American parts of the city, and are likewise each comprised of a majority African-American students.  The story's narrative suggests that these are some of the worst performing elementary schools in the State of Florida.

The accusations laid by the Times' authors in the piece are substantial and damning.  In a time when perceived race relations are already at a low-point due to ongoing debate regarding allegations of police targeting of Black men, the story's timing is certainly instep with a larger narrative that elicits tremendous emotional response - particularly among those on the far left of the political spectrum.  Riding a wave of sorts, the Times story seeks to combat widely held notions regarding the Black community and Black student performance, namely the belief that comparatively poor academic performance among Black students is principally a function of familial and community level factors, rather than problems with schools or teachers themselves, or worse, systematic, institutionalized racism.

Resegregation, aka the Community School Model
At the heart of the Times' angst is the decision Pinellas County Schools made to discontinue a program of school integration that required students to forgo the schools in their residential community, and instead be bused to schools outside their community for the sake of racial and economic integration.  Without going too deeply here, the idea is, at the very least, a logical one.  Introduce a larger share of middle class, better performing students into lower income schools to improve the school's performance through an osmosis like infusion of diverse students statistically predisposed to perform better due to their family's socioeconomic status.

As a liberal leaning individual myself, I often like to joke that my liberalism has its limits - and those limits are my two kids.  Naturally, middle class families fought the program, many opting out altogether by sending their children to private schools, or moving from the District altogether.  Parents balked, and rightly so, at having their kids bused to a distant school when many of their neighborhood peers were attending the school in their community.  It should be noted that it wasn't merely white, middle class students being bused past their local school.  Many African-American students were being bused to schools in better areas, creating the problem of elementary schools in middle class areas turning down applications from local residents, while low income area schools were only two-thirds full.

Interestingly enough, the busing model was generally opposed by those of all races (source), though African-American community leaders felt strongly that a school of all Black children would receive less attention and funding than a school with a sizable portion of White students.  Given human beings' tendencies towards what sociologists refer to as "causal attribution bias," it's not a stretch to imagine that this was true - that White voters in the county would be less sympathetic to the plight of schools that were overwhelmingly Black than they would schools with a mixed student body.  It's as unfortunate to our human story as it is a strongly weaved part of the human condition.

What the Times is purposely referring to as "resegregation" could just as easily be called a return to the "community schools" model, if one were trying to be more neutral about it than the Times apparently is.  There is a literature in education research that suggests that student success is driven in large part by the community's commitment to its schools.  There are those, including myself, who further argue that spatial proximity has a large impact on this.  Easier to support a school when all of your neighbors send their children there, and you're physically invested, as well as emotionally invested.  You want the school to succeed because it's a part of your community, regardless of whether or not you have children there.  During Pinellas' period of socioeconomic integration, children were bused to schools in other communities.  It's easy to see how, under the assumptions of the community schools model, support for such institutions would consequently be weakened.

The Times' Principal Claims
It's important to summarize here  the Times' major accusations claims so that they might be held to statistical scrutiny, at least to the extent they can be.  As best I can extrapolate from the article, the story's authors make three major claims:

  1. The five elementary schools identified in the story perform considerably worse than they should.
  2. The District is responsible for the poor performance of students in the five identified schools.
  3. Low income Black students in Pinellas perform worse than low income Black students in other communities.
Iv'e dedicated this blog to exploring these three contentions using analysis of available data.  While the Times' story is visually appealing, and tells a cognitively engaging story, it was written by journalists, not statisticians.  Statistics are a lot like religion - a little bit of knowledge can lead you astray, while a deep knowledge usually brings you back.  Before I begin addressing the aforementioned claims, it's necessary that I provide some background on myself as a researcher, and how my experiences and background might influence my perceptions of the data.

About the Author
I imagine that I share something with the authors of the Times article, in that I too want to see all students have the opportunity to succeed, regardless of race, income, or any other identifying characteristic.  In criticizing this piece's claims, I should make it known that I do so merely out of a desire to see that the most accurate portrayal of what's going on in these schools be put forward.  Each of these institutions is comprised of a staff of individuals that form a team, and I would hate to see these teams drug through the mud, or worse, face job elimination, as a result of potentially false assertions.  It's my desire that we get the story right before making rash decisions.

I'm a professional storyteller, I just tell those stories with data.  I've spent over a decade getting paid to do just that, and was fortunate to receive the training and preparation of two separate Ph.D. programs - one of which I may actually finish by the end of next year.  I'm a white male in his 30s, a father, and a husband.  My wife and I are fortunate to earn a very comfortable living.  I mentioned earlier than my politics tends to lean liberal, but if you were to ask, I identify myself as an "extreme moderate."  Not extreme in the sense that I have no opinions, but rather that my opinions are subject to a tremendous amount of variance along the political spectrum - typically averaging out somewhere near the middle, or just to the left.

Above all else, I am a person interested in the truth.  As such, I hope that someone will read my analysis and question it.  Not only question it, but try to debunk it.  I am hardly driven by pride, and if I believe something to be one way when it's not, it is my extreme desire that someone show me the error of my ways.

Finally, I'm a fan of education.  It's the true vehicle of social mobilization - the pathway to opportunity for all.  I'm a former educator, primarily teaching community college in my spare time (in addition to my regular career), but I did teach public high school for a year and then some (the then some was filling in as a replacement teacher at the end of a term).  I want to see these kids succeed, which is why I wan't to find out why they aren't.