Monday, April 13, 2009

Nobody Wants To Know The Truth


I'm an advocate of premising social policy on sound research and analysis. You try to find out what's working in other places, and build on those ideas. You monitor your own implementations of those ideas to be sure that your policies are, in fact, working. Ideally, you can build on working ideas, revise or eliminate ideas that aren't working, and make things better.

But what happens when those who implement policies, and those who are affected by those policies or by potential changes in those policies, either don't care about implementing the best public policy solutions or are afraid of what they will uncover if they shine a spotlight on existing policies or the alternatives? One of the most obvious contexts where you can see this phenomenon at work is in K-12 education. When "reforms" come, they're often ad hoc, seat-of-the-pants, "this sounds good"-type reforms, that may or may not last. But most of the time, there appears to be little to no interest in achieving actual reform or improvement. Sometimes you get the worst of both worlds - a reform agenda that doesn't appear to be concerned with quality, and which may in fact be rooted in a political agenda that is something quite apart from ensuring quality public education.

For example, advocates of charter schools are often strong advocates of standardized testing of children in public schools, but they often seek to exempt charter schools from those testing requirements. That prevents a direct, side-by-side evaluation of charter schools and public schools in the same district. Ideally you would be able to track individual student performance, year by year, through their entire progress through the grades. That allows you to compare schools and classes not only by the performance of students at the end of a year, but also to compare the quality of students going in. If schools accept public funding, why shouldn't they have at least that much accountability? For that matter, shouldn't we be taking a similarly hard view at this type of data that already should be available in public school settings?

It isn't just the supporters of charter schools who don't get behind mandating across-the-board testing. It's also many proponents of the status quo. When you don't have the data that supports a side-by-side comparison, it's easier to argue that the status quo is better than (or at least as good as) the alternative. Teaching schools don't appear interested in collecting and analyzing data that might suggest that teaching certificates aren't superior to accelerated or alternative certification programs, or even in comparison to uncertified teachers. Teacher's unions don't necessarily want to see data that suggests that contract standards for hiring, promotion, and assignment to schools and classrooms aren't optimal or may impede good classroom learning.

A related example of this phenomenon in action comes from Teach for America. TFA has been around for about two decades, so there's been ample opportunity to develop good data. To some degree that could be done internally - TFA could develop performance tests to be administered at the start and end of the school year, to help gauge student needs, performance, and improvement in performance over the course of a school year. A school district interested in comparing general teacher performance to TFA corps member performance could administer those tests in its regular classes - or, conversely, it could develop or license tests and require their use across the district. Instead, even when students are being tested and it appears that performance could easily be tracked down to an individual level, that seems to be regarded as something to avoid.

Thus, TFA can announce on its website that a study "confirms... that corps members have a positive effect on student achievement relative to other teachers, including experienced teachers, traditionally prepared teachers, and those fully certified in their field", based on weak data from a single state, North Carolina. Skeptics can point to the amount of guesswork involved, trying to connect students to classroom teachers by who proctored an exam, trying to gauge what happened in a classroom based upon a year-end test with no start-of-year data, the exceptionally small sample size of TFA teachers, the omission of any schools below the high school level, the differences between North Carolina's demographics and those of the most vexing school districts, such as DC or New Orleans....

My point here is not to criticize TFA or corps members. With the résumé of a corps member, not a certified teacher, I've substitute taught in public schools. I've seen some terrific teachers, and others who (despite "years of experience") sleepwalk through their workday. I've seen how different a classroom experience can be between schools, and the role SES, parental involvement, and administrative competence can make. I've also seen how much easier it can be to teach high school than to teach middle school. With no wish to make this an excessively broad generalization, as some schools have severe order and discipline problems that continue through high school, I found teaching grades 10-12 to be a relative cakewalk as compared to teaching middle school, with ninth grade being something of a transition.

I found it to be at times quite difficult to explain what, to me, was an elementary math concept to a seventh grader. I didn't have a similar problem communicating with kids in tenth or eleventh grade. That is to say, when I was out of the context where teaching skills were most crucial, where I had lesser concerns about maintaining classroom order, where the kids understood their lessons at a more adult level, I could be highly effective. I didn't necessarily feel ineffective at other grade levels, but there were times when it was obvious that I lacked the skill set necessary to be more effective. Ideally, had I been a professional teacher, I would have learned some of those skills during my coursework and teacher training. (I say "ideally" because I've had certified teachers tell that, at least in the programs they attended, training in classroom management came almost as an afterthought.) Oh, sure, I would likely have also developed my own set of skills "on the job", had I continued to teach in a classroom for the next few years, but I have no illusions that, for me, it would have been a particularly easy or natural process. Given my druthers, I would have stuck with grades 10-12.

So again, here, I see TFA demonstrating a strong interest in claiming credit based upon weak data and analysis, but not trying to collect strong data or facilitate compelling analysis. I see school districts employing TFA corps members doing nothing to evaluate how the corps members compare to certified teachers. I see advocates of teacher certification and tenure suggesting that reliance upon TFA results a high level of churn among entry level teachers, causing harm to students who would benefit from having experienced teachers. But I don't see anybody advocating the type of study that would sort out and answer some of the persistent questions about TFA, teacher certification, and teacher effectiveness.

It really does amaze me sometimes that, with all of the money this nation pours into education (not to mention the combined spending of the world's other nations), there's so little apparent interest in determining which curricula, classroom management techniques, and approaches to teaching particular subjects (e.g., reading, foreign language, math) are the most effective. I once crazily hoped that having teacher colleges sponsor charter schools might provide a laboratory for education innovation and reform. Oh well.

17 comments:

  1. Here's a weird one...how did we decide what to teach? I mean, why do we teach chemistry, physics, geometry, etc? Since my room is self-contained, I can focus on what I want. I have the four core subjects plus Braille, handwriting, technology and daily living skills. Obviously I don't get to everything every day! So I focus on reading, writing, math because if you can't read, it doesn't much matter whether or not you know the scientific formula of Hydrochloric acid (does that sentence even make any sense? I just took Chemistry words that I know and put them in a sentence)

    As I've said before (probably ad nauseum :)), teachers can't control many things. I have two kids this year who were born addicted to crack. One is okay...is functionally illiterate (see prior comment) but is okay. The other is at a 5 year old level and always will be. For the past 7 months, I have tried to get the latter student to add and it isn't happening. He can draw 10 circles and draw 20 circles and count them and get 30, but he can't get the addition facts. So...am I effective? I mean, I feel like I'm getting him to do something, but if the goal is "teach Junior how to add", I've failed, but through no fault of my own.

    So, as we wonder...how do we measure an effective teacher??? (No conclusion here, just throwing that out)

    ReplyDelete
  2. You highlight why having a baseline is critical to evaluating teacher performance. If you're teaching in a 9th grade classroom where the average student reads at a third grade level, you can't be judged by the same standard as somebody who is teaching a classroom where the kids function at or above grade level.

    Yet our standards of teacher performance seem to assume either that any given teacher should get the same results from either set of students (the teacher in the more challenging class should bring average performance up to grade level), or that the more challenging classroom merits a higher grade for the school if the kids advance more than one grade level (let's say, they improve from third grade to fifth grade), as compared to the other classroom where we'll assume that the kids advanced only one grade level (9th grade to 10th grade) despite the first set of kids remaining five grade levels behind where they "should" be.

    There seems to be a lot of "madness" in this type of evaluation of teachers and schools, and not a whole lot of "method".

    ReplyDelete
  3. "The truth? You can't handle the truth." - Sorry, couldn't resist

    Measuring performance is hard in any field, even an apolitial one. Measuring student performance is just about as political a field as you will ever find.

    Nobody wants you to come-up with hard data because if you do, it might contradict there deeply held belief system. In fact, it is pretty much a sure bet that, given the cross currents at work in education, you will step on lots of toes if you insist on interjecting "facts" into the equation.

    Maybe worse, the only people interested/with resources to do the research all have vested interests in a particular outcome.

    Then there is the whole issue of how much value any social research has at all. Do you really think that people will/can do a "deep" enough study to tease out whether (and I'm making this us for arguments sake) asian american children do better at school then hispanic children because of natural aptitude, self-esteem, parental expectations, parental involvement in modeling appropriate study behavior, etc, etc. etc? (It's like when they try to measure sexual activity by asking teenagers how many sexual partners they've had in the past year, and then treating the results as gospel . . . )

    Mind you, I don't disagree with your thesis, I just don't see it happening . . . Then there is hte whole social-engineering thing.

    What would a bunch of teachers, politicians, and professional instigators do with the "fact" that the number one driver of student performance was "parental involvement" in the educational process (assuming that any of the above is true)?

    Gee, we need to stop instigating for more money for our districts directly and launch programs to help parents learn to parent . . . oops, the only parents who attend are the ones who already "get it." I know, maybe we can mandate parental behavior and force compliance . . .

    Hmmm, according to the data "per pupil" expenditurs have very little correlation to success but coming from a stable two parent household does . . . I guess that means instead of whining about poverty (and yes, I'm oversimplifying) we need to take responsibility and overhaul our educational system and strengthen families/promote family involvement by . . . yeah, that's going to happen. : )

    CWD

    ReplyDelete
  4. I think there is evidence that with enough school intervention, perhaps including a seriously lengthened school day and probably also professional homework support/tutoring, you can diminish the effect of an unsupportive home. You're in essence hiring people to do what the parents will not do.

    That, of course, is costly. And it seems to me that advocates of breaking teachers unions and KIPP-style schools are hoping that the work can be done by the existing population of teachers - extend their workdays and work loads - rather than admitting that homes do make a difference and (forgive the oversimplification of the argument) that you need to do a lot more than point to teachers and say, "You're substandard because your kids haven't overcome all of the obstacles in their lives."

    ReplyDelete
  5. 1. I'm not familiar with the evidence your citing, but I would be happy to look at it if you could point me in the right direction to find it. Theoretically, what you are saying makes since, but I'd be inclined to think it would be a lot easier to achieve in a boarding school than a traditional public one.

    2. Are you advocating the implementation of a "seriously lengthened school day and probably also professional homework support/tutoring" at all US schools, only those schools that your new and improved standardized testing indicate need it, or are you saying that you would only provide this support to certain students within a school?

    CWD

    ReplyDelete
  6. The evidence, for what it's worth, is anecdotal. There's the success of the KIPP schools, which follow that model, but with the caveat that parents who aren't sufficiently supportive to get their kids into KIPP and keep their kids in compliance with the school's attendance, dress code and homework policies aren't typical.

    I'm actually not proposing a lengthened school day, school year, or anything else. We should be taking a look at those ideas and trying to gauge what effect they will have on school and student performance, but (a) I tend to think that when we go off in those directions we confuse quantity with quality, and (b) I'm not in favor of doing anything across the board, but instead favor trying to actually figure out what works and where it works, without incurring needless expense or engaging in knee jerk "this sounds like it'll work" solutions like same-sex education and school uniforms.

    ReplyDelete
  7. . . . but Aaron, school uniforms will help with my master plan to militarize the youth of America . . . oh wait, this is your serious blog.

    I think it will be interesting to see what impact the increasing "Federalization" of our educational system will have on the old "states as laboratories" model. Although I will also readily admit that there wasn't much experimentation going on before either . . . I also have to confess that although I lack stats to support it, I kind of like the "uniforms and disciplie" approach to education . . . bet you never saw that one coming. : )

    CWD

    ReplyDelete
  8. Nope. Couldn't have imagined.

    An element of federalization that I think would be helpful, particularly given how mobile our society has become, is to set a basic national curriculum - ideally based upon the best of what we know of child development, and subject to revision.

    I think I've mentioned this to you before, but due to my childhoold relocations and school changes, I missed certain elements of the curriculum due to their being offered at different grade levels at different schools. (This was in Canada.) That's why I'm unemployable and destitute today... no, wait. I guess I'm not. But seriously, it did create some difficulty when I had to learn certain things (e.g., the parts of speech) on my own or through osmosis rather than through any direct instruction, but was expected to know them from the moment I walked into my new classroom.

    ReplyDelete
  9. Yay--thanks for doing the "you can't handle the truth" quote, CWD! You never let me down :) :)

    I was all anti-uniform before I started teaching. Now, um, I'm kinda in favor of it.

    :hangs head:

    ReplyDelete
  10. I think the uniform for students idea has some merit (beyond my desire to militarize the educational process) but I'm still struck speechless by Aaron's choice for a teacher's uniform.

    I have all kinds of snide questions I'd like to ask him about what drove that decision, but I don't think "I" can handle the truth about this one. : )

    CWD

    ReplyDelete
  11. You could buy peacock outfits for me, CWD and yourself, Aaron, and we could have a lovely garden party in downtown Ann Arbor (are you in MI, CWD????) It could be the birds of a feather party! Get it??? I just thought of that. I am clever.

    This is going to sound really, really bad. I apologize in advance for saying it. But one day I popped a button on my pants (not cuz I'm fat but because I bent down in a weird angle). I just taped my pants together with tape and a paper clip b/c my kids really couldn't see it anyway. I know that's awful. I also once picked my nose when I was teaching Braille. I'm sorry, but it had to happen.

    ReplyDelete
  12. The peacock is a symbol of pride. CWD, have you no pride?

    ReplyDelete
  13. Aaron - If a lack of pride is what it takes to avoid seeing you and me in those outfits, yes.

    TeacherPatti - Unfortunately, I now live on the East Coast . . . and I never would've looked good in one of those outfits.

    CWD

    ReplyDelete
  14. "... and I never would've looked good in one of those outfits"

    So now you have fashion sense? Just for the sake of perspective, if school uniforms were modeled after your personal high school or college 'dress code', what would the kids look like? ;-)

    ReplyDelete
  15. ". . .if school uniforms were modeled after your personal high school or college 'dress code', what would the kids look like? ;-)"

    They'd look like "me", obviously. : )

    God help them . . .

    CWD

    PS - But they would save "lots" of time in the morning that they used to spend getting their hair "just right."

    ReplyDelete
  16. I used to get up at 5:45 am to "do" my hair (read: curling iron, hairspray--it was the 80s!!). That lasted until college. Now it gets brushed and it's delighted. I finally accepted the half-Jew fro that I get when it rains and the straightness when it doesn't.

    ReplyDelete