Thursday, June 07, 2018

An Example of Skepticism of Science

I'm skeptical by nature. Many often call me cynical. Sometimes I come across an article, a study or something that, while supposedly a proper example of science, sends off all sorts of red flags for me. I recently came across an article in the Guardian that highlights a recent Yale University study that sought to find the biases that may lead young black students to be suspended/expelled at higher rates than whites. (And I use that phrase intentionally: "sought to find". But we'll get to that momentarily.) The article's content concerned me enough that I decided to look at this study a little more and found some interesting things.

First and foremost, I don't like the "science" of starting with a preconceived conclusion and then crafting a study to find evidence to support that conclusion. While this can be ok in hard sciences which focus so exclusively on physical processes and quantitative outcomes, in social sciences, where qualitative or behavioral actions are what is being analyzed, it can be extremely dangerous. This is almost explicitly research confirmation bias and makes the study start off highly questionable in my view - this kind of bias can affect everything from design to interpretation. But let's move on.

From the article:
The teachers were told the following: ... Your job is to press the enter key on the external keypad every time you see a behavior that could become a potential challenge.
Since I didn't capitalize and put flashing gifs around those words, let me emphasize, they were specifically told to signal on anything that "could become" a "potential" challenge. So the instructions are essentially telling teachers to be hypersensitive for any behavior whatsoever that might possibly become a potential challenge. Not telling them to indicate when a behavioral challenge occurs. Not to indicate the moment they believe a behavioral challenge is starting. But to point out anything whatsoever that could somehow, at some point in the future, turn toward into a potential challenge.

How vague and open-ended is this?

Child picks up scissors - "could possibly become a potential challenge"
Child starts to speak loudly - "could possibly become a potential challenge"
Child starts swinging pencil around -  "could possibly become a potential challenge"
Child vehemently disagrees with fellow student - "could possibly become a potential challenge"
Child stands up -  "could possibly become a potential challenge"
Child disagrees with conclusion of a Yale study - "could possibly become a potential challenge" (and is possibly racist)

You get the idea. Obviously the question relies on an incredibly subjective measure. I've seen certain kids (#notall) where rocking in a chair is a "tell" - a precursor - to a wild moment of explosive energy - where the kids suddenly has a compulsion to get out of their chair and vent some of their energy - sometimes (though rarely) with damaging or harmful results to themselves and others.

Let's also keep in mind that it's probably fair to assume that most teachers are going to wish to err on the side of caution when it comes to such instructions. Better to appear overly cautious than to appear to disregard or miss the signs leading up to a problem.

Then after quoting what teachers were instructed to do, the Guardian moves the goalposts in the very following line:
While the teachers were asked to detect “challenging behavior”, no such behavior existed in any of the videos.
Wait, wait, wait. They were NOT told to detect challenging behavior. They were asked to indicate any "behavior that could become a potential challenging behavior." So a teacher that has seen and dealt with the rocking-in-their-chair situations in the past, and is then told to look for "potential challenging behavior" may signal on a rocking child that, in this case, never leads to challenging behavior.

To me, if you want to understand why black students are being suspended, THEN LOOK AT THE SUSPENSION CASES!!  I think going to the source of the information would be relevant here. Are the suspensions dubious? Are they credible? Isn't that pertinent here?

Look at that data and determine if the cases justified suspension or not. If they did NOT, then you can certainly make the case that perhaps there was discrimination at play. But to give people prompts which are overly vague and then deduce from their hypersensitivity that they must be racist is not impartial science.