Friday, September 16, 2016

Trigaux's Gloomy Observation Made Not So Gloomy

Bob Trigaux accurately observed that in a list of the top 25 metros, Tampa Bay was dead last for median income - leading him to assert that "we are poor"

Sounds nice, except you can't make an apples to apples comparison of the markets included in this report.  I speculated that two factors work to Tampa Bay's income disadvantage - retirees and cost of living.  I did a quick stat analysis to confirm utilizing US Census data at the county level (except for St. Louis which is split - used municipal data there).  I combined Pinellas and Hillsborough Counties for the Tampa Bay measures.  

15.8% of Tampa Bay's population is 65+, compared to 11.4% for the other 24 metros. A one percentage point increase in your 65+ population reduces your median income by $1,132. Normalize Tampa Bay's population to the group average, and that's an extra $5k accounted for by our comparatively larger retired population - people with minimal incomes, but also likely with reserves/savings.

Furthermore, the average cost of living for the other 24 metros is 14% above the national index. Tampa Bay? 2% below it. A one percentage point increase in cost of living relative to the index increases median income by $530, as it takes more income to get people to live in more expensive places.  Normalized, that's an additional $8,500 hit Tampa Bay takes.   

What does this mean? If Tampa Bay's age and cost of living were controlled for, our median income would be $13.5k higher - putting it at $62.4k - or 15th out of the 25 metros.  Not in the top tier, but probably not "poor".  

Oh, and for those scoring at home, 65+ population and cost of living relative to the index accounted for 74% of the variance in the median incomes of these communities.

Tuesday, May 3, 2016

Forbes Article and an Update

Maureen Sullivan, a Forbes contributor, recently took up the charge of questioning the (now) Pulitzer Prize winning Failure Factories Story ran by the Tampa Bay Times.  In doing so, she referred to my blog postings, and the analysis that I've posted here which questions the fairness of the Times labeling all five of these schools "failure factories" (or any of them for that matter, but I'll save that for a later post).

You can read Maureen's article here

I graciously sent Maureen a note thanking her for exploring this topic further, as well as explaining why I had left my analysis where I did.

First and foremost, I was doing the stat work in the evenings on my time off, which became increasingly challenging with family responsibilities (including raising young twin sons with my wife) and my non-profit commitments (many of which are education related).

Second, I was afraid that someone might either maliciously or ignorantly misinterpret my findings and analysis as being somehow driven by racist feelings that I don't have - which happened here unfortunately.  Education research is a hobby for me - something I'm passionate about simply because I care about public education and educators.  My primarily daily responsibility, however, is to my professional career, and the manner in which it helps me to provide for my family.  An accusation of racism, particularly when the witch hunt had reached full pitch fork and torch stage a few months ago, could have potentially caused negative career implications, regardless of how unfair they might be.  Putting my ego aside, and my family first, I stepped aside.

This being said, I've given Maureen permission to use my name.  It's clear that she understands that my goal wasn't to suggest anything ridiculous about Black children or their ability to learn, but rather to determine whether the Times had given these educators and administrators a fair shake.  My concern was, as it is today, that good teachers will be moved out of these schools, and that many of these children will lose a connection to those teachers for reasons that have nothing to do with their dedication or abilities as educators.  Unfortunately, in talking to some of the families of kids at these schools, this is precisely what is happening.

I've been working in the background on issues related to school improvement, including at risk students in these schools.  I'm glad to see the interest in my data, and hope that others will take up the charge as well and look at it from angles I haven't considered.  The truth doesn't always get you a shiny trophy, but that's OK.

Tuesday, August 18, 2015

Kriseman has the right idea

Since posting this analysis, I've regularly heard from people that we shouldn't be "pointing fingers" or "casting blame" but rather working towards solutions on the issue of poor performing elementary schools.  I hate reading this, because as social scientist, I'm not interested in casting blame - I'm looking for the source of the problem.  By this same logic, we shouldn't waste our time trying to find where a leak is coming from, we should instead work on soaking up the water.  It would be like a physician treating symptoms without concerning him or herself with what was causing the symptoms.  Doesn't work for education either.

My analysis has suggested that, at least for three of the schools the Times calls "failure factories," the source of the problem might not be the school.  I haven't said what it might be simply because my research doesn't support a source - it only supports the notion that the schools, in some cases, might not be it.

With that in mind, I actually think Mayor Kriseman is on the right track by proposing an education director - a liaison to work with Pinellas County Schools and the communities to see what can be done from within the community to improve student performance.  I know that Bill Foster formerly had one in Lori Matway who I worked with during the four years I spent as a scholarship committee member for the Pinellas Education Foundation.  I wasn't aware that the position had been vacated, but I'm at least glad to hear that Kriseman has a plan to fill it.  This at least recognizes that the community has at least a part in the success of our students - all students, and that we should be doing more to foster growth for at risk students.  It's the reason I worked on behalf of the Pinellas Education Foundation, and the reason I've taken up this blog.

Perhaps this will convince Janelle Irwin to reconsider the box I believe she's mistakenly placed me into.

Monday, August 17, 2015

Response to Janelle Irwin

SaintPetersBlog's Janelle Irwin reviewed my blog postings today and wrote a critical, yet thought provoking response to my contentions.  There were a few things I found a bit unfair, which I will mention here shortly, but for the most part I was glad she took up her case and contributed to the dialogue on this topic.  Despite what Janelle may have taken away from my posts, I believe that she and I are probably equally concerned about the issue of poor academic achievement in St. Petersburg's urban, low-income core, even if we see it as resulting from different causal factors.

First off, a compliment - kudos to you Janelle for raising three kids and choosing to continue your education - that's no easy task no matter how you look at it.  As a an education advocate, I was happy to read this.

In response, I'll start with a minor point, yet one worth mentioning.  Your blog title begins "Dad blogger argues...".  As you mentioned in your post, I am a market research professional.  I also offered up that I am nearing completion of Ph.D. studies.  Any of those would have seemed a more appropriate descriptor.  As a feminist, I would never describe a professional female writing on a subject for which she has professional expertise as a "mom blogger."  I believe if you think about that, you'd agree.  No big deal, I think we're just all trying to improve in this capacity.

As far as the content we disagree over, let's start with this idea of "resegregation."  You appear to be of the belief, as the Times authors are, that by shifting back to the traditional neighborhood or community school model in Pinellas is responsible for lower FCAT performance at the schools listed in the article.  Well you're right.  In fact, I don't disagree with this at all.  Of course increasing the proportion of at-risk students at a school is going to reduce test performance.  Statistically, there's absolutely no reason why we would expect a school that is now comprised of a larger share of at risk students not to see score declines.

The better question is, were we just shuffling deckchairs on the Titanic by moving them around in the first place?  If you took all of baseball's best players and randomly redistributed them across every Major League team, you'd see greater parity.  Did moving a poor Black student to a better performing school have a positive  impact on their personal exam performance?  This would be a good question to ask, but it's beyond the scope of what we're talking about here since we're evaluating schools, not individual students (as we probably should be in the first place).

Second, I was disappointed that you didn't attack my specific assertions as presented by the data.  I hold two political science degrees and taught the subject for nearly ten years.  As a political science student yourself, I would have hoped you would have approached one of your faculty advisers and developed a plan to dispute my data or data assertions.  As I've mentioned before, I welcome this solely because I hope I'm wrong.  I hope the issue is the schools and not some community level factor outside of their controls - because if it is a problem with the schools, it's a lot easier to fix.

Finally, I'll suggest that my reasoning isn't as mind-boggling as you suggest.  Schools are an easy scapegoat.  My principal research areas in my Ph.D. program is community colleges, and it's a phenomenon there as well.  Politicians and community leaders regularly point to community college leaders and questioning why they haven't solved the employment gap.

Is it that large of a stretch to suggest that a child who comes from a home where the parents (but usually parent) are working multiple jobs, shares a residence with several family members or friends who come and go at all hours of the day/night, or is exposed to violence, drug use, or other social ills associated with poverty, might be at a disadvantage before they even show up for school?  A friend who formerly taught inner city once asked me in a state of despair, "how am I supposed to prepare a kid for an exam when they haven't eaten since lunch the day before?"

I've presented data that suggests that three factors account for 75% a school's FCAT performance, and that none of those three factors are things the school can control.  Is it fair to go into a school like Melrose that is seemingly comprised of the largest share of at risk students in the entire state (have to double check this, but they're at least within a few spots), and expect the faculty and leadership of that school to produce exam scores that are on par with the average performance of every other school in the state?  Absolutely not - it's entirely unfair.  Is it reasonable to expect them to grow their scores in a manner that shows statistical improvement over past performances?  Absolutely.  Just not when you're benchmarking against a period in which their students weren't nearly as "at risk".

As you mention, I consider myself a liberal leaning individual.  It's why I cringed when you wrote that I implied that I wouldn't want to send my kid to a school comprised of mostly Black students.  The truth, Janelle, is that I don't want to send my kid to any school other than the public school in their neighborhood.  That includes magnet or fundamental programs.  I don't want my kid to be bused to a magnet program because I believe in the logic and value of neighborhood schools.

What should we do?  That's a bigger question than my research is ready to address, if I'm just being honest.  I do believe that reducing poverty and improving job prospects in these communities will improve the quality of students attending these schools.  You'd essentially be eliminating some of the most important factors that make at-risk kids actually at-risk.  But I don't have the solution that problem, and I don't think you do either.  I don't think our politicians do, or social commentators.  It's truly a plague on our communities that is affecting our schools.  But if you're asking me if we should ax an entire school administration and expect things to all of a sudden turn around, well that's a data question I can answer.

Janelle, things "are that bad" at two schools the Times has written about, as I discuss in my blog.  No doubt about it - they perform way below their expectations.  Go in and make massive changes - my data seems to suggest that they should perform at a certain level given teachers who teach and administrators who lead at a level commensurate with their peers at other schools.  They're not performing at the level, which suggests significant change is necessary.

The situations that "aren't that bad" are for the three others schools the Times calls "failure factories."  Yes, only 13% of 4th graders pass the FCAT math exam at Melrose.  That's not good enough, and we should be ashamed.  But my data tells me that a school comprised of that many impoverished, special education, African-American students would only pass 5% of students if the teachers and administrators were performing "on average."  The 25% of variance we can't explain with the model is why Melrose passes 13% instead of 5%, and a very solid explanation is that the teachers and administrators there are actually responsible for this improvement over statistical expectations.

No, I do not sit back and shrug my shoulders and say "ahh, what can we do?".  Instead, I'm sitting here saying "maybe the schools aren't the problem."  You can call me wrong on this regard all day long, but please don't take me to be someone who doesn't care, or who isn't committed to seeing these students perform better.


Visual summary of model results

I've tossed a lot of numbers at you thus far, so it's only fair I summarize the findings to this point in a single, visual graphic.

For those who haven't read the posts below, I've evaluated 4th graders at each of the five "failure factories" not based on their raw FCAT performance as a percentage of students achieving a satisfactory grade of 3 or above, but rather based on their predicted performance when controlling for three factors that account for nearly 75% of FCAT performance: 1) percentage of African-American students; 2) percentage of free or reduced lunch students, and; 3) percentage of ESE (exceptional student education, minus gifted) students.

The plots below reflect deviation from where the regression model predicts each school should perform based on district averages for nearly all Pinellas elementary schools (I excluded charters and a couple of schools with exceptionally low totals of 4th grade test takers).  The values are percentage point deviations, so a school that passes 10%, though is expected to pass 15% would show as a "-5".


To summarize what we've found to this point, the Times seems to have two schools nailed.  Maximo and Fairmount definitely underperform on both FCAT reading and math when controlling for the handicapping factors detailed above - by quite a bit.

According to the model, the Times has unfairly pegged Melrose, which sees a larger proportion of its 4th grade students achieve a 3 or above on both the reading and math exams than the model would say we should expect.

Two other schools, Campbell Park and Lakewood, have mixed results.  Each beats model expectations for reading, while coming short on Math.  Lakewood makes the best case for rejecting the "failure factory" label, while clearly Campbell Park has some ground to make on its math performance.  Failure factories?  Jury is still out on these two.

FCAT Math Results - Still only two failure factories

So earlier I gave an explanation of my method for evaluating 4th grade FCAT reading scores and mentioned that I would repeat the process with the FCAT math scores.  Here are the results of this statistical model:


Again, all three variables affect the percentage of a school's students who earn a 3 or above on FCAT math, though the effects differed somewhat from reading.  Here the percentage of ESE students was actually the dominating variable.  Special education students had a more difficult time with FCAT math than they did FCAT reading.  Otherwise, the results were similar to before - poverty, as measured through free and reduced lunch percentage had a larger impact than race, but both mattered.  The model's r^2 value was .746, meaning the these three factors explain 75% of a school's FCAT math performance.

So how did the schools do compared to their handicap?  Let's take a look:


The two schools we could justifiably say were failing, Fairmount Park and Maximo, showed similar poor performances in FCAT math scores relative to their predictive performance when controlling for student demographics.

One school, Melrose, beat expectations.  Again, Melrose's fourth graders are 95% African-American, 82% free or reduced lunch, and 18% ESE - by far the highest ESE of the five schools, which is why their predicted pass range for FCAT math is onlyl 5%.  I'm sure the Times authors and the media, and perhaps even District leaders would say that 5% is a poor performance benchmark.  Maybe so, but given Melrose's student body, that's what it is.  13% of Melrose fourth graders earned a 3 or above, meaning they beat their true benchmark by 8-points.

Although Campbell Park performed on par with expectations for FCAT reading, their math score was sub-par by a small, but noteworthy amount.  We predicted that 20% of their fourth graders would earn a 3 or above when, in actuality, only 14% did.

Lakewood had beat its FCAT reading expectations by 10-points, but was pretty close to flat in FCAT math, coming two percentage points shy of its expected performance of 19% of students earning a 3 or above.  Cause for concern and attention?  Yes.  Is it a "failure factory?"  The data doesn't support that claim.

What did we learn?
The data still supports the contention that only two of the five schools are potentially "failure factories" - Fairmount and Maximo.  Campbell Park and Lakewood need to work on their FCAT math, but they did well enough on FCAT reading against expectations to believe that they are within striking distance of performing to expectations.  Melrose, the poorest, most African-American school in the lot, exceeded its statistically determined performance benchmark again in Math.

What's next?
Next up on the analysis slate is understanding whether or not Black students are truly worse off in Pinellas than other parts of the state when controlling for the demographics we've included in this model.  Will begin working on that model as soon as I get home tonight.


Sunday, August 16, 2015

FCAT Reading Scores - Only two of the five failure factories actually fail

The Times' story has focused on FCAT reading and math scores as a metric of school performance.  There's a variety of reasons why I, and many others, would argue that exam scores aren't always the best metric of student or school performance, but that's beyond the scope of this blog.  If that's the benchmark the Times' has set, then I'll run with it.

The Times has suggested that the five elementary schools in question are failing their predominantly African American student populations as measured through test scores that are below acceptable levels, and just looking at the FCAT pass performance of these schools gives you an idea why.  Math and reading scores are noticeably lower than for other elementary schools in the district, as well as others in the state.

Here's where things get tricky, though.  We know that each of these schools is characterized by two distinct trends: 1) higher than average percentage of Black students, and; 2) communities with higher than average levels of poverty.  It so happens that we also know that both of these factors are statistically related to lower test performance.  Let's not get into the reasons why right now, but rather agree that this is a universal trend across the United States, and not a problem specific to Pinellas or St. Petersburg.

We would expect a school comprised of a larger than average percentage of Black students whose families earn less than the mean U.S. household income to have FCAT scores below the average for all schools.  That's not to say we expect less of these students, or should ever feel comfortable with this, but that's the nature of the beast.  Statistics can't show racial preference, they can only show what is.

Fortunately, we have a way of statistically controlling for the effects of race and income as they relate to exam performance.  In fact, we can even predict where a school's exam scores would be expected to be based solely on the proportion of Black and impoverished students taking the exam.  We do this through what we call multivariate regression modeling.

The Times claims that the five schools in question are failing their students at a disproportionately large rate.  For this to happen, we'd need to calculate what we should expect the school's FCAT passage rate to be based on factors outside the control of the school.  Schools that perform below this prediction may in fact be contributing to decline, whereas schools performing at or above their predicted exam passage rate likely aren't doing anything to harm students - they may even be helping them overcome these outside factors.

To test this, I looked at the percentage of 4th graders earning a 3 or above on FCAT reading for 2014 for each elementary in Pinellas County, minus charter schools and those with too few students tested to include.  All told, this resulted in 74 elementary schools included in the sample.

The purpose of my regression analysis was to predict FCAT reading scores based on the percentage of students taking the exam who were Black, on a free or reduced lunch program (our proxy for income), or who were ESE excluding gifted (exceptional student education - think special education/learning disabled).  If you're going to make an apples to apples comparison across schools, you have to hold each of these three factors constant.  Otherwise you're asking some schools to start the race thirty second after everyone else, while still expecting them to finish with the pack.

Here are the results of the regression model:


Let me help you interpret this if you're not one who regularly reads regression outputs.  First, all three factors tested were statistically related to the percentage of 4th graders earning a 3 or above on the FCAT reading exam at a 95% confidence level.  That is, we're incredibly certain that the race, income and special education status of a school's 4th graders influences their scores.

The actual impact of each of these factors can be seen in the column labeled "B".  This is the beta coefficient, which tells us what happens to the percentage of students earning a 3 or above when we adjust the independent variable by 1-point.  That can be a bit tricky to wrap our heads around with percentages, so let's just move the decimal point one place to the left and say the following:

  • As the percentage of Black fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 2%
  • As the percentage of free or reduced lunch fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 8%
  • As the percentage of ESE fourth graders increases 10%, the percentage of students earning a 3 or above on FCAT reading declines 3%
What this tells us isn't a surprise - race matters less than income or special education student status, but all three still matter independently of one another.  So how do we predict based on this output?  Simple - we use the regression equation, which multiplies the actual value for each school by their respective beta coefficient, adds those up (plus the constant), and comes up with a predicted dependent variable value (% earning 3 or above).  So let's run those equations and see what we get.

% FCAT Reading 3 or Above

Based on these results, only two of the five "failure factories" performed worse than we would have otherwise expected controlling for the communities in which they reside - Fairmount Park and Maximo.  The school with the lowest predicted pass performance was Melrose, with a fourth grade student body that is 95% African American, 82% free or reduced lunch, and 18% ESE.  Based on these disadvantaging factors, we'd expect only 11% of 4th graders to earn a 3 or above on FCAT reading.  In reality 18% did.  Lakewood also performed noticeably above expectations, while Campbell Park performed at a level pretty close to where they'd expect them to be.

One more thing to point out about the regression - it has an r^2 value of .724.  In equally confusing speak, what this means is that the model accounts for 72% of the variance in school level FCAT performance.  That is, factors not directly related to teaching or curriculum nearly three-quarters of a school's performance on the FCAT exam.  

What did we learn?

Based on this initial test of FCAT reading scores of 4th graders in Pinellas, it seems that two schools - Fairmount Park and Maximo actually perform less than would be expected of them, while two other "failure factories" actually outperform (statistically speaking).  Campbell Park is generally right where our model said it would be.

Are these five schools failing their students?  By the Times' benchmark, only two seem to be - at least when looking solely at Reading scores (I'll run the same analysis with the math scores for a later post).  Lakewood and Melrose, on the other hand, actually seem to be doing a pretty good job on FCAT reading, considering their student demographics.  Perhaps not where they'd like to be, but performing at a level that the Gods of statistics would say is commendable.