So if you quadruple the number of exams taken, the number passed will... ?

I was browsing a few old files in preparing to clean up my hard drive, and noticed this one... 

In 2006-2007, a huge push was made to get more students to take AP courses, because (I love this) students who take AP courses are more likely to graduate than those who don't. So being in AP courses obviously motivates kids to stay in school. Move along, logicians -nothin' to see here. Principals were given incentives to increase AP enrollment as much as possible.

What harm could it do? Even if a bunch of kids who didn't really belong in the course were in it, the ones who did belong would still get what they needed, right? And the kids who didn't belong would realize that and either drop out or struggle through.

But wait, there's more. If a senior is taking a core course - say, in math, English or science - that is also an AP course, they generally must pass that class to graduate. Principals are also highly incentivized to increase the graduation rate. So flunking, say, 8 or 10 seniors who don't belong in your AP course is  (a) a great way to never teach another AP course, and (b) futile, since the administration will find a way to pass them in the final 3 weeks of school anyway.

Given that, if you teach an AP course at my school, you have two options: (1) teach the course as a regular AP course, lose 1/3 to 1/2 of the class within 8 weeks, resign yourself to having those students stay lost for the next 30 weeks and thus learn absolutely nothing in your course, but give them a C anyway, or  (2) water down the content to the point that everybody learns a little something but only the most diligent have any hope of passing the exam, and then only by doing a lot of independent study.  For the past 7 years, AP Statistics was taught using option 2, and no one has passed the exam. I chose option 1. We'll find out in mid-July if I did any better.

But meanwhile, it seems clear that a lot of teachers choose option 2. (After all, who wants to teach a class every day in which half of the students have given up weeks ago, but know they're going to pass anyway?) Here's what that does to the number of students passing the AP exam:

In the first year (2007), exams passed increased by 9% while exams taken increased by 350%. After that, exams passed declined significantly, and is now at 60% of the number it was at prior to flooding the classes with unprepared students.

Addendum: I did have a student pass, which I'm told is the first time in 7 years anyone from MHS has done so. However, I only had one other student get a 2, with the other 18 in the class getting a 1. I was somewhat disappointed with this result, and would have loved to take another shot at it in 2011-12. Nonetheless, I did prove that Option 1 at least provides a few students the chance to pass the course. I'm convinced that if all AP classes were run that way, fewer unprepared students would take the course, but a much higher percentage would pass.

Another in a long, long line of excellent examples of unintended consequences. Although in this case, it should not have been (and wasn't, by some) unanticipated consequences.

Wired Loops

Wired is an awesome magazine (thanks, Abe!). This month, the cover story is about how our brains and behavior are driven by feedback loops. According to the story, in the 1960's, Albert Bandura "observed that giving individuals a clear goal and a means to evaluate their progress toward that goal (in real time) greatly increased the likelihood they would achieve it." (I added the parenthetical "in real time," because elsewhere in the article that element is shown to be critically important.)

We provide lots of feedback to students, but a lot of it is time-delayed, and very little of it is directly applicable to evaluating their progress (or the extent of their progress) toward the goal (which is often poorly defined anyway.)

So I've been thinking about (1) how to very clearly define the goal in a course, (2) what role interim goals should play, (3) how to measure progress toward the goal or interim goal, (4) how to give immediate feedback about that progress, and (5) how to publicly track my students' progress while keeping each individual's information private.

Here's where I am so far... (more as I think more about it)

Create a pre-test for each 3-week block of work.

Lay out the calendar for the next 3 weeks, showing what's going to be covered each day and which questions on the pre-test could then be answered based on what's been covered through that day.

Give exercises (via class work) on whatever got covered that day.

Have students self-grade that work, and turn in a slip of paper with their name and score so I can track and publish it (with the names stripped out).

Give homework based on their scores (higher score = less homework). Caveat: if their quiz grade is significantly lower than their self-reported class work grades, they automatically end up in the "most homework" category.

Give quizzes I can grade within 10 minutes (10 questions, answers are right or wrong, worksheet pre-configured to spit out grade after answers are typed in). Give an in-class assignment on quiz days so that the class looks like: 5 minute warm-up, 15 minute quiz, 15 minute in-class exercise (while I grade the quizzes), 10 minute quiz return and wrap-up.

Give tests I can grade during my planning period (45 minutes) so I can return them the next day.

The Incredible Shrinking Man

My principal doesn't want me back next year... OK, I can live with that. Thus, he'd rather not praise me at today's end-of-year luncheon... makes sense to me. Unfortunately for him, my kids did really well on the state-mandated exam, and he likes to use the luncheon to hand out awards to teachers for various accomplishments during the school year. So to avoid handing me a lovely inkjet-printed certificate, he made up a category for which I didn't qualify, available only to teachers of "standard" courses (since I taught only Honors). But he neglected to notice the person he gave it to instead of me also didn't teach any standard courses. Sad, really. If he'd just come to me before-hand, I'd have told him to make the category "Most gains by a teacher who isn't me," and we could have all enjoyed the award.

The Sociological Value of Understanding the Concept of a Local Maximum

To improve the quality of education in America, the graduation rate and the number of students passing the state-mandated tests must fall dramatically.

We've done an enormous disservice to our children by relaxing graduation standards, badgering teachers into passing students who shouldn't, enrolling students in classes they shouldn't be in (thus watering down the content for everyone in the course), focusing exclusively on the lowest-performing students' results on the state-mandated tests, and redirecting blame for failure from students to teachers.

The three step, no additional cost, plan for success:

1. Hold all students to much, much higher standards for passing classes in the core curriculum and for graduating from a "traditional" high school.

2. Provide alternative track educational opportunities for children who do not wish to or are not capable of completing a traditional high school education.

3. Get out of the way.

(Aside: I distinctly recall coming home from school with a bad grade on an English test and complaining that the teacher didn't explain the material. I suppose it's possible my parents called the school to complain and demand she be replaced with someone I could better relate to, but I'm pretty sure that wasn't their first response.)

Books I Want Next Year's AP Stats Class to Read

These are some books I've read that would be valuable for people who are going to deal with statistics. Please tell me about others in the comments section, or comment on the ones I've listed.

(other books and info to come as addenda to this note)
The Black Swan / Fooled by Randomness - Nassim Taleb
The Science of Fear / Risk - Dan Gardner
The Tipping Point / Blink / Outliers / What the Dog Saw - Malcolm Gladwell
Guns, Germs and Steel - Jared Diamond
Inevitable Illusions - Massimo Piattelli-Palmarini
The Rational Optimist* - Matt Ridley
On Being Certain** - Robert Burton, M.D.


*Haven't read this one, but have read others by the same author and based on the book's blurb, think it would be the best option.

**Reading this one now; may or may not come off the list

Decaf nofat no whip latte.

Our principal today announced he would be holding an assembly for all 9th and 10th graders on Friday, effectively eliminating the last opportunity I would have to teach mathematics to my 10th graders prior to the hugely important state-mandated test. The subject of the assembly: "Do well on the hugely important state-mandated test."

So, in order to ensure our students do well on this hugely important state-mandated test, we've spent 3 days doing "benchmark" testing, to see where the students were at the beginning of the year, in the middle of the year and one month later. We've already held at least one other assembly to pump up the kids for the upcoming benchmark test, which of course doesn't count at all, except that good scores on the benchmark test make the principal look good in the eyes of his district bosses.

We've also held (and funded) a middle-of-the-school day "dance" for students whose scores improved on the 2nd benchmark test, and promised a similar event for students whose scores improved on the 3rd benchmark test. The principal attempted, in the faculty meeting today, to set the 2nd dance for tomorrow. He may have been persuaded to wait... until Friday.

We've also had to take most of a class period for each of our classes in order to take our kids to the computer rooms to practice taking the test on the computer.

On the plus side, we haven't conducted a fire drill all year, so we've saved all that time in favor of classwork. Of course, monthly fire drills are mandated by our state, but hey, what's child safety when the school's hugely important state-mandated test score is on the line?

Of course, I've also lost class time to pep rallies, donut sales, and all the other detria associated with high school life -- for example, none of the senior girls will be in my class on Friday because they will be getting their hair done for the prom. This is not just hyperbole to have an excuse to skip my class... first of all, my seniors don't need an excuse to skip my class, and second, I've actually asked several of the girls who have weaves (and a significant majority do, despite (or possibly because) of the fact I'm at a Title I school) and they've all told me it's a 4- to 6-hour process, at a minimum.

The principal justified his "test-prep" rally by saying that if they don't have it now, they're not going to get it in the last day prior to the exam. So why, last week, were we required to prepare a "3 day plan" to prepare them for the test in the week and one day we had remaining (meaning 3 per class, since I only see each class every other day)?  If we only needed 2 days, why didn't we have a 2 day plan? If the kids who're being taken out of my class to go to the assembly don't need my class, can I just give the rest of my classes a free day, since they must not need it either?

Enough.

California Dreamin'

The title of this post has nothing to do with the post itself, except that I'm visiting a friend in California as I write this.

Wednesday's USA Today (3-30-11) had a front page story on the investigation of several of DC's public schools for "irregularities" in their standardized tests. Apparently, USA Today previously published a report that more than half of DC schools were cited in the past 3 years for having unusually high "erasure rates."

First, the quote from the acting chancellor of the public school system, Kaya Henderson: "The integrity of test scores is perhaps more important to us than any other district because we use the test scores in important ways - evaluation, compensation."

Hmmm... let's see if we can find anything not to like about that sentence. It will be hard, but I'll give it a shot:

1. She's claiming that, in other districts, integrity is (perhaps) less important. Go ahead, kids (and teachers) -- if you're not in the DC public school system, cheat all you want, it's not that important.

2. The reason scores matter more in DC is that they use those test scores in important ways, which pretty clearly implies that the ways they're used in other school systems aren't that important. If the way we're using test scores doesn't matter, can we PLEASE stop using them?

3. The important ways the test scores are being used are "evaluation" and "compensation." I believe it's fair to assume she's not speaking of evaluating the students, because all tests in all districts are evaluating the students, and she's distinguishing DC's use from other districts. So the evaluation and compensation she's speaking about is of teachers and administrators. Therefore, I'm shocked -- shocked, I tell you -- to discover some teachers and administrators, believing their compensation and in all likelihood, their jobs were on the line, cheated to obtain higher scores.

We now have at least two examples (Atlanta being the other one) of the following scenario:

1. New head of the district's public school system arrives and promises to dramatically increase standardized test scores in the district.

2. New head of the district's public school system makes it clear to every teacher and administrator in the district that scores on the standardized test will be used to evaluate and compensate them. Moreover, to the extent possible, job retention decisions will also be based on these scores.

3. Test scores rise dramatically, immediately.

4. Someone investigates the scores, and finds that there is very widespread evidence of activity that is statistically improbable to the point of inconceivable, unless that activity is intentional (i.e., cheating).

5. "Deeper" investigation finds conclusive evidence of cheating at a very small number of schools, exonerating the rest of them.

6. Deeper deeper investigation finds that perhaps the first deeper investigation wasn't entirely uninfluenced by the school district's desire to depict the problem as a few rogue administrators and teachers, rather than a system-wide response to the school district's leadership.

For those of you keeping score at home, here's what we have:
Validity of standardized testing in assessing student progress: very suspect (e.g., this article, along with hundreds of others)
Validity of using standardized testing to assess teacher competence: even more suspect (this paper and others)
Likelihood of cheating by school administrators if standardized test scores are the primary measure of competence: very hard to say, but a study from the 90's showed 4% - 5% of teachers cheated to improve student scores when stakes were relatively low, and cheating doubled when stakes got higher for the students. When stakes get higher for the teachers and administrators, the number is likely to be much, much higher. The New York Times recently reported on investigations in Georgia, Indiana, Massachusetts, Nevada, Virginia and Texas. There are undoubtedly many more cases that have not been uncovered.

Solution? Not "No Child Left Behind," obviously. Unfortunately, President Obama's Race to the Top program, while acknowledging the failure of the current standardized tests, essentially just asks for new ones.

Perhaps we should be looking for ways to assess teachers and administrators based on the kinds of things assessments in the private sector are based -- peer review, on-going evaluation and guidance by an immediate supervisor, multiple evaluations from multiple sources (to reduce the likelihood of individual bias in evaluations) and student learning gains, properly placed in context with all the other assessment information.

I guess I'm California dreamin' after all...