Op-ed: a computer scientist weighs in on the downsides of AI.
Quoting the summary:
"Given that ML systems (including facial recognition systems) can produce biased output, how should society treat them? Remember that, often, the choice is not between algorithmic output and perfection but between algorithmic decisions and human ones—and humans are demonstrably biased, too. That said, there are several reasons to be wary of the "algorithmic" approach.
GIGO
One reason is that people put too much trust in computer output. Every beginning programmer is taught the acronym "GIGO:" garbage in, garbage out. To end users, though, it's often "garbage in, gospel out"—if the computer said it, it must be so. (This tendency is exacerbated by bad user interfaces that make overriding the computer's recommendation difficult or impossible.) We should thus demand less bias from computerized systems precisely to compensate for their perceived greater veracity.
The second reason for caution is that computers are capable of doing things—even bad things—at scale. There is at least the perceived risk that, say, computerized facial recognition will be used for mass surveillance. Imagine the consequences if a biased but automated system differentially misidentified African-Americans as wanted criminals. Humans are biased, too, but they can't make nearly as many errors per second.
Our test, then, should be one called disparate impact. "Algorithmic" systems should be evaluated for bias, and their deployment should be guided appropriately. Furthermore, the more serious the consequences, the higher the standard should be before use."