Aeon Magazine contributor, Steven Poole, posted a great piece on the problems with entrusting more and more of our lives to computers.
I don't know how far-fetched worries about the powers of computers and algorithms seem to people today. But I do think those fears are increasingly warranted, as technology progresses.
Poole provides many examples of computers being used or tested to drive cars, filter content on the Internet, rank search-engine results, predict recidivism (i.e. predict the likelihood that a convicted criminal will reoffend), target retail promotions, disseminate college course and professional training material, conduct business and stock trading transactions. That computers play such key roles in important areas of our private and public lives will probably be old news to most. But it's still the sort of thing that needs to be watched carefully. Because computers have demonstrably done so much to improve the quality of our lives, and since they are capable of performing a large number of tasks so much more efficiently than human beings do, it might seem to naturally follow that our lives would be better all-way-round if only computers were running everything.
And even if we haven't completely bought into these utopian visions, we still need to be careful about handing over more limited and restricted tasks to the computers. The fact that a computer performs a particular task more efficiently than a human agent is not the only factor to take into consideration. And we should be even more cautious when the 'task' is complex and multi-faceted. Poole provides a number of interesting examples of computer 'errors' that have already surfaced and that (if not corrected) could have serious and far-reaching negative consequences for the legal and judicial systems, the development of human culture and civilization, the quality of education, and the preservation of liberal democracy and personal freedom. But he doesn't try to diagnose the source of those problems.
Of course, whenever a problem does crop up in an algorithm--when a program malfunctions and begins spewing out bad data or bad results, software engineers will go in and 'diagnose' the problem. They will then develop a patch or rewrite the software or add some additional sub-routines that will correct for the identified error and ensure that the program's outputs fall within the expected parameters. When Poole or I worry about these problems that are already cropping up in the powerful computer systems that exist today, we're not worrying about the same thing that the software engineers are worried about. They will tackle each individual problem as it arises and create a fix for it. But what I want to know is whether there is something common to all of these errors. And as the stakes get higher and higher for each subsequent 'error'--as more and more of our lives become dependent on computers functioning well--I want to know whether it's possible to make any helpful predictions about the kinds of errors that might crop up in the future.
I'd like to suggest that, when thinking about the limitations of computers and algorithms, there are two key ideas to keep in mind. These ideas are simple enough that just about anyone can understand them, but I think they also capture very deep features of computers.
1) There are significant limitations on the extent to which any computer simulation (of a physical system or decision procedure, for instance) can ever match the corresponding real phenomenon.
The kind of computer that poses these sorts of worries is the kind that makes decisions--or the kind that simulates decision-making. Whether it's a program that plays chess, or that recommends movies or books to you (based on your buying and browsing history), or that drives a car, what all of these have in common is that they simulate decision-making procedures. Sometimes its uncanny how 'perceptive' and 'discerning' these programs can be. But it's important not to be taken in. The computer doesn't actually understand what it's doing. The computer is not making its 'decisions' based on any sort of knowledge or skill or understanding. It's decision-making is only a simulation.
To grasp the difference I'm trying to highlight between the 'simulation' and the 'real thing,' I like to point to the example of computer-animated films. Toy Story, Finding Nemo, The Incredibles--how are these films made? Well, most of the character animation is created by manipulating digital puppets. This is often done frame-by-frame. A character's pose and expression is set for each moment in the film. But then there's all the set-dressing in the background, the environmental factors and atmospherics, the lighting, the characters' hair and clothes--how is all of that put together? Much of it is generated by sophisticated computer programs that simulate those features of the character's environment. So, for instance, if the wind is blowing through a character's hair in one scene, or that character is running through another scene, the animator will not 'animate' the hair and clothing frame by frame. Rather, software engineers develop programs that simulate the behavior of hair and clothing. The animator will set the positions of the puppet in that scene, and then the program will take care of making the hair and clothing 'look right'.
If you watch the special features on just about any computer-animated film, there will be a section in which the hair and clothing (and other) simulations are discussed. And it turns out that it's extremely challenging to develop those simulations. Of course the main challenge (and ultimate goal) for these software designers is to get the hair and clothing to 'look right'. The film viewer knows what real hair and clothing look like, and how they appear in different environments (when it's windy, when it's humid, when it's soaking wet). The animated versions of clothing and hair need to act sufficiently like the real thing; otherwise they will be distracting to the viewer. But while a traditional 2-D animator would just have to draw the hair and clothing a certain way (and that's often hard enough), the software engineers for these 3-D films have to actually create 3-D simulated hair. Programs for animated hair have to keep track of the behavior of thousands, even millions, of individual hairs. Each one of those hairs must be able to interact with the other hairs, the character's body, and the larger environment in a way that--when all the pieces are put together--looks realistic to the viewer. Each hair needs to be able to clump and separate, wave through the air, hang, twirl, etc., etc. And guess what? It's really hard to get all those properties into the hair and clothing. Many computer generated films will come with blooper reels that show some of the amusing images that come from running these not-yet-perfected programs.
The results are funny, when we're only dealing with hair and clothing simulations. And the stakes are very low, when the programs are only trying to get the hair and clothing to behave in a way that looks realistic to a film viewer. But what if, instead of simulating hair and clothing, some computer is trying to simulate the decision-making process of a correctional psychologist. And what if at stake is whether or not a convicted criminal is granted an early release from prison for good behavior. Suddenly the stakes go way up. And if it's tough for programmers to simulate the behavior of physical systems like hair and clothing, shouldn't we expect that it would be even more difficult to simulate the decision-making procedures of people in complex and sensitive situations.
Of course there's data out there that suggests that human decision-making processes are actually badly contaminated by all sorts of emotional and non-rational factors. Some people think that handing over decision-making to computers would provide for greater objectivity, reliability, and consistency. But even granting that point, notice what that involves. In order to program a computer to make decisions for us, a programmer has to assume that she's able to enumerate all of the relevant factors. That's a bold assumption to make--especially when the stakes are so high. And the programmer also has to be able to successfully simulate sensitivity to all those factors.
Again, remember: computers don't understand what they're doing. A CGI hair simulator doesn't understand anything about hair. And a decision-making simulator doesn't understand anything about the decisions that it's making or the outputs that it's generating. And most likely, (I would suspect in all cases, but I can't say for sure) the actual computational processes that generate that hair simulation or generate that decision-output bear no resemblance and only the most tenuous connection to the features that make real hair behave as it does or to the processes that human decision-makers go through.
Consider: what is the standard for measuring the success of a CGI hair simulator? Answer: the output--what the final product looks like visually. That's the only standard. And what is the standard for measuring the success of a decision-making simulator? Answer: the output--whether or not the computer spits out the 'right' results. Whether such a simulator is 'good' depends only on the outputs and not on the processes involved. To get a sense of how this might be problematic, imagine a simulator that was designed to predict the positions the planets in the night sky, as they would appear to someone standing at a particular point on earth. If the simulator were programmed in such a way as to take the elliptical orbits of the planets into account when generating its predictions, we could expect that it would generate very accurate predictions. However, it would also be also possible to generate equally accurate predictions using a system that was premised on the planets moving in only circular orbits. Something like this was actually done by scientists working before Kepler. The system is more complicated; it requires the introduction of epicycles into the model, but it works just as well as the other system at generating the right outputs. We could imagine programmers learning, one day, that the epicycle-simulator was generating predictions that deviated from the actual facts. How would they respond. They might respond just by creating a patch, or writing an additional subroutine, that would correct for the error and get the outputs back on track.
Now the planets simulator case might be seen as offering a basis for hope that as programmers of other simulators (including decision-making simulators) progress, they will actually develop simulations that not only get the 'right' results more consistently but that also do simulate more and more accurately the actual mental processes that human decision-makers use. That's a possibility that I can't rule out in principle. However, before becoming too hopeful, we should remember that there is a big difference between simulating physical systems and simulating decision-making processes. And if our primary criteria for testing and evaluating simulations is whether it generates the right output in a large-enough percentage of test cases, then I think we're missing something important. Does that mean that we shouldn't even try to generate these sorts of programs? Not necessarily, but as the stakes increase and as the complexity of the tasks-to-be-performed increases, we need to keep in mind that any errors will come at an increasingly high cost. Raising the issue of the complexity of the task-to-be-performed will lead to our talking about the second point.
2) Computers work well for accomplishing very particular tasks that are well-defined and for which a specific and limited number of rules can be clearly articulated. Computers work less well when they are designed to accomplish a wide variety of tasks--especially when the boundaries of these various tasks are not well-defined and how the tasks interrelate cannot be captured by clearly articulated rules.
To help us think about this, I will refer to one of those sci-fi artificial intelligence apocalypse stories: the 2004, action film: i, Robot. Inspired by Isaac Asimov's short stories, i, Robot, explores precisely the sort of thing that I'm worrying about in this blog post. The envisioned world is one in which personal robots have become thoroughly integrated into society. They've become so sophisticated and are so completely trusted and accepted that they are given the responsibility of caring for children. They do drive cars and manage cities. And the idea that a robot could commit a crime or harm a person is almost impossible for the average citizen to fathom.
This great confidence is based, in large part, on the public's trust in the three laws of robotics--hardwired into every robot and completely inviolate. The first law of robotics states: "A robot may not injure a human being, or through inaction, allow a human being to come to harm." But a series of apparent and inexplicable 'malfunctions' lead the movie's main character, Del Spooner (played by Will Smith) to realize that there is a basic problem with these laws. By the end of the film, this string of malfunctions are traced back to a powerful computer, VIKI, who, in order to carry out the first (and most fundamental) law of robotics, attempts to seize control of the entire planet, to impose martial law, in order to prevent all possible injury or harm to human beings.
The question this raises is, what went wrong. The first law of robotics states, "A robot may not injure a human being, or through inaction, allow a human being to come to harm." If you were creating a line of extremely powerful robots, that seems like just the sort of rule you'd want to hardwire into them, to override any subsequent programming that could possibly lead to human injury. And yet VIKI manages to take this plausible-sounding principle and use it to justify a global take-over. The principle seems right, but it has this bad result. How can we make sense of it?
Interestingly, the film's writers suggest that VIKI's basic problem is that she is 'all head and no heart.' The world is saved, in part, by one robot whose programmer designed it to feel emotion--which ability required that it also be able to override the three laws. In effect the film says that it's better to be imperfectly rational and have a heart, than be perfectly rational and have no heart. The lack of confidence in rationality that is expressed in the film is interesting. The truth is that imperfect rationality and a heart can be just as deadly--maybe even more so--than pure rationality. I think that the film's creators actually misdiagnosed the problem with VIKI.
VIKI's problem is not that she is 'all head and no heart.' What is wrong with VIKI is that her programming has expanded and is no longer carefully circumscribed by clear rules. Computers work best for accomplishing very particular tasks that are well-defined and for which a specific and limited number of rules can be clearly articulated. Now one of the rules--in fact the most important rule--that VIKI has states, "A robot may not injure a human being, or through inaction, allow a human being to come to harm." What that law is supposed to express is a limitation or constraint on what VIKI can accomplish. One might expect that a robot, so-constrained, would be created with a particular and limited set of tasks in mind: manufacture cars, cook dinner, clean the house, take care of the lawn, filter my e-mail messages, walk the dog, escort the kids to their friends' house. The problem for VIKI, within the world of the film, is that her boundaries and tasks are not well-defined, and so she interprets the first law of robotics, not as a limitation or constraint, but as her primary mandate or objective--roughly, do not injure human beings and prevent as many human injuries as possible. And if accomplishing that task means taking over the world, curtailing freedoms, violating liberty; so be it.
Another way to understand VIKI's problem is in terms of what she understands or doesn't understand about human beings. Obviously human beings created VIKI in order to serve them--in order to make their lives easier by accomplishing certain tasks more cleanly and efficiently. But what does VIKI 'understand' about human beings and her relationship to them. The most basic thing that VIKI knows about human beings is that they are to be preserved and protected. That is the content of the first law of robotics. VIKI does not understand that human beings are autonomous and creative beings. She does not understand that for human beings a life of enslavement may be less desirable than death. She doesn't understand anything about human values. And how could she? She was never designed to understand those things. She was only designed to perform a few carefully delimited tasks. And as long as she was limited to those few tasks, this lack of understanding posed no problem. But once VIKI developed to the point where her influence was not carefully limited and constrained, that lack of understanding manifested itself with alarming results. So, again, VIKI's problem is not that she lacks emotion, but that she lacks understanding. When a computer's tasks are very limited and circumscribed, that lack of understanding is not a problem. But if a computer's tasks are unlimited and open-ended, that lack of understanding, I expect, becomes much more of a liability.
Let's see if I can illustrate my point using an example that doesn't come from science fiction. Think about a program that recommends books. It looks at your browsing and purchasing history, identifies patterns, and recommends books that have a similar contend and subject matter, or that were purchased by people with a similar browsing and purchasing history. Now if one's goal is to read more books like those that I already enjoy, a computer program like this is a great tool for accomplishing that. But there are other goals that one might have, related to reading, that wouldn't be so well served by such a program: the goal of reading books that challenge my accepted beliefs or the goal of reading books that have been influential in the history of ideas. A program that works by analyzing your browsing and purchasing history may not be helpful for that. Now someone who is reflective would not be in danger of confusion here. Someone who is reflective will know what the recommending program is for and only use it with the one particular goal in mind.
But what will hold for the person who is less reflective? Could someone make the mistake of thinking that the most culturally influential and important books are those with the most 'hits'? Could someone unwittingly abdicate control of the content of culture to the machines? Handing over this kind of culture-forming power to computers might be fine if we could program a computer that was actually sensitive to all of the relevant factors. But that doesn't look like it's going to happen any time soon. And the programs that we currently have--many of them are fine, but only for accomplishing limited and carefully defined tasks. The worry is just that our enthusiasm about the power of computers will cause us to overlook the difference.
***
That's more of a reflection than a thorough-going and rigorous analysis. But hopefully it's helpful. Like I said earlier, Poole does a nice job of illustrating the problem, but he doesn't try to diagnose the problem. Plenty of people have written stuff on the diagnostic side. If what I've written intrigues you, you should follow up by exploring that stuff more.
