You Won't Fix Colleges With Standardized Tests

David Brooks shares some observations about college:

Colleges are supposed to produce learning. But, in their landmark study, “Academically Adrift,” Richard Arum and Josipa Roksa found that, on average, students experienced a pathetic seven percentile point gain in skills during their first two years in college and a marginal gain in the two years after that....

This research followed the Wabash Study, which found that student motivation actually declines over the first year in college. Meanwhile, according to surveys of employers, only a quarter of college graduates have the writing and thinking skills necessary to do their jobs.

But perhaps the problem is not with colleges, per se, and is more the consequence of a culture shift: the notion that everybody can and should attend college. The more lazy, unmotivated, disinterested and/or untalented students you place a college class, the more you change the classroom experience.

As much as Brooks laments the good old days, which apparently means 1961,¹ when "students spent an average of 24 hours a week studying", I'm skeptical both of self-reports of hours spent studying (particularly those cherry-picked from a half-century ago) and also of the implication that, prior to the entry of the unwashed masses courtesy of student loans and the G.I. Bill, colleges historically did a better job educating students than they do at present - at least if the student wants to learn. The "Gentleman's C" for children of the wealthy and well-connected is part of that same history.

Brooks also appears to hold the conceit that during the post-war boom employers were deeply concerned about how much their new hires had learned in college. The college graduates I know from that era describe how the subject matter of their degree didn't much matter to employers. Their bachelor's degrees were akin to a union card, something that qualified them for jobs that required bachelor's degrees. Something similar happened in the late 1980's and early 1990's, as computer technology expanded and people were welcomed into computer and IT jobs with little regard for their undergraduate major, or even if they had completed a college degree. Similarly, there was a period in the 2000's during which consulting firms could not hire enough MBA's and started to draw in graduates from law schools and other programs. If the pool of candidates with the preferred degree is too small, employers will expand the applicant pool. When the pool of degreed candidates is sufficient, the "union card" effect returns.

Meanwhile, contrary to Brooks' suggestion, students are more aware than ever of the market value of their degrees. For many years, a significant population of incoming students has been asking, "What sort of job can I get with this degree," not "How much will I learn along the way." Some of my relatives who easily obtained jobs after completing college in the 1960's and 1970's are testaments to the fact that, in a booming job market, weak critical thinking skills and majoring in partying are not significant impediments to getting a job.

Does Brooks believe that MBA graduates are in slight demand? As graduate programs go it's an easy degree, two years, no capstone or thesis. Business students are among those who study the least while in college. How does Brooks reconcile that with his notion that employers want colleges to produce hard-working students who study a lot?

In his analysis, Brooks makes a classic error: he believes that education can make you smarter. Education can help you learn to reason, can provide a structure for more efficient thinking, and can help you maximize your potential, but it cannot make you smarter.

Here's another dirty little secret: When Brooks talks about colleges that are interested in assessing how much and how well their students are learning, he speaks of "schools like Bowling Green and Portland State". Not schools like Harvard or Yale. Why? If you can be extremely selective in terms of who you admit, although you may not show a significantly higher improvement of your students' critical thinking skills over the course of their degree programs, they start and end at a higher point than their peers at less exclusive institutions. But more than that, we're back to the union card issue. Schools like Harvard and Yale have done a great job branding themselves and, by implication, their graduates. In a nationwide job market, even when there's a glut of job seeking graduates from schools like Bowling Green and Portland State, the numbers of Harvard and Yale graduates does not increase - and between their small number and the exclusivity of their brand, they have an advantage in the job market even if they're not as intellectually qualified as their state school peers. If you recruit at Harvard you're also apt to get somebody who "fits" with an organization that serves the needs of society's well-heeled. Will a graduate from Bowling Green have similar social experiences coming in the door? Why gamble, right? Let's also not forget how many jobs are obtained through connections as opposed to through job fairs and advertisement - roughly 60% - and Ivy league schools insert you into fantastic networks.

Brooks' notion that you can measure college learning in a manner that is "real and transparent to outsiders" is, to put it mildly, absurd. If employers want to know whether their applicants can perform certain tasks, it's easy enough for the employer to impose a testing requirement as part of the application process. Further, as with any test, a testing process focuses on matters that are often subjective or arbitrary, and a one-time test tells you little that is useful. How would Brooks imagine that tests could be made objective and transparent, but at the same time be objective, fair, minimize cheating, relevant to any given student's curriculum, revised annually, and updated² regularly (in many cases it would have to be annually) so as to ensure continued relevance to employers?

Brooks has the notion that a college could come up with its own assessment method and "broadcast the results to prospective parents, saying, 'We may not be prestigious or as expensive as X, but here students actually learn.'" The problems with that? Multiple. First, once the assessment method is implemented, assuming it works, it will be years before employers will see any difference in the graduating class. If a school has a weak reputation it would involve a tremendous leap of faith by a student to enroll based upon the promise that a new "assessment method" will result in their being a hot commodity in the job market by the time they graduate. And even after the new improved graduates start graduating, it will be years before employers trust that the change is real, not just hype, and even then you won't be Harvard, Yale, or anything close to it. You may have better graduates, but you won't have the brand, the network, the endowment, the exclusivity....

The best way to improve the quality of your graduating class is to improve the quality of your incoming freshmen. But colleges are tripping over each other trying to bring in students in order to maintain tuition revenue, creating the opposite effect - students who really shouldn't be attending college are able to get admitted and are often able to graduate. Those students are not going to be helped by "assessment methods" at the college level, any more than they were made college-ready through standardized testing during their K-12 years. It is a glut of college graduates, not hours spent studying or attempts to measure improvement in critical thinking, is the most profound cause of the commoditization of the college diploma. And to the extent that employers no longer trust that colleges screen out undesirable employees, that's an inevitable consequence of removing any meaningful barriers to college enrollment.
-------------------
1. Recall that Brooks appears to believe that U.S. culture peaked in 1963.

2. Here I speak of revision in terms of producing a different set of questions from year to year, as otherwise the answers get out, and of updates in terms of making sure that the subject matter of the test remains current and relevant, a significant and difficult task.