Bryan Alexander is facilitating an online book club reading of Cathy O’Neil’s Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. I am about two weeks behind (typical), so I will focus on just a couple of his questions for Part 1.
A. “What would it take for an education algorithm to meet all of O’Neil’s criteria for not doing damage?”
The big problem with almost all educational measurement is the use of clumsy at best proxies (O’Neil uses the term) for the learning we want to measure. Since the fundamental output metric is a test with all the possibilities for manipulation that suggests, when we then try to measure what input changes improve that output we are are at least two levels of abstraction removed. Until we can measure educational outcomes some way other than by means of a crude manipulable proxy, I’m not sure we can fix this.
B. “What are the best ways to address the problem of “false positives”, of exceptionally bad results, of anomalies?”
I think the best way to solve this problem is to place some limits (preferably not determined by an algorithm) on the kinds of decisions we allow algorithms to make without human input. The potential harm of a bad book recommendation from Amazon is much lower. That probably means thoughtful review of every adverse algorithmic recommendation by at least one live human being. Of course, thus undermines the efficiency and scale that algorithms are designed to create. An important step is to acknowledge that algorithms are not neutral even if they manage not to be arbitrary. They encoded the assumptions and biases of their creators, and acknowledging those assumptions and biases is a key part of the design process.
The DC schools example draws attention to the importance of checking for flawed input data. After all, the algorithm is only as accurate as the data you feed it.
Notes: O’Neil’s three criteria for a Weapon of Math Destruction are “opacity, scale, and damage.” She uses the initialism WMD. I wish she had come up with something else, because of the namespace confusion with chemical, biological and nuclear weapons.
Opacity makes me think of Frank Pasquale’s The Black Box Society, which I haven’t read yet. The synopses of Pasquale’s book make me wonder how his and O’Neil’s work intersect.