The field of artificial intelligence has within it a question that is difficult to answer. What is intelligence? Defining intelligence raises some challenges, as you might expect,  and so we have somewhat squishy ways of defining artificial intelligence. The father of artificial intelligence is Alan M. Turing, and he came up with a test for when AI equals human intelligence. His test has had its share of challenges, but it still is used today.

Turing described the Turing Test as follows, “a machine has artificial intelligence when there is no discernible difference between the conversation generated by the machine and that of an intelligent person.” The Test first appeared in a paper Turing published in October 1950 titled, Computing Machinery and Intelligence. Incidentally, Turing called his test The Imitation Game, and a movie with the same name is being released November 28th. So far, no AI software has passed the Turing Test.

A Turing Test For Law

As the legal community becomes more interested in how computers can be used to augment (or, shudder, replace) lawyers in doing some tasks, lawyers frequently pose a question to test the skills of AI software designed for tasks in the legal industry. Those lawyers ask whether a computer could be programmed to function at the level of a first year associate in a law firm.

To allay the fears of any law students reading this post, the consensus among those who are programming AI software to handle legal issues is that we are not close to having a computer that could replace a first year associate. So at this time, the test posed by lawyers (like the Turing Test for human intelligence) has not been passed.

Having heard lawyers ask the first year associate question many times, I think we need to rephrase it. What a first year associate does in a law firm varies widely from firm to firm. I don’t think that standard is sufficiently consistent to use as a measure of legal AI intelligence. Instead, I propose that we take one step back and look at the moment when a law student graduates from law school. We can then rephrase the test as by focusing on legal intelligence after law school graduation.

Before going further, we should consider another question often asked. Could a computer today pass the bar exam? Unfortunately for those of us who suffered through months of study and then waited months to hear whether we passed, the answer is probably yes. No one has invested the resources to test this answer, but given how computers have performed in other settings (playing chess, Jeopardy!) it seems that passing a bar exam is not a big stretch. To clarify a bit, I’m not talking about producing written answers to open ended questions. Rather, I’m talking about passing the multi-state portion of the bar exam, which consists of multiple-choice questions. However, and with all due respect to those who produce the mult-state bar exam, passing the test has little to do with legal intelligence and a lot to do with memorizing a bundle of (somewhat arbitrary) rules. My favorite advice for answering questions: “You will encounter questions where all of the answers are wrong. Choose the least wrong answer.”

Going back to our legal intelligence question, and focusing on that moment after graduation, we could now re-phrase the Turing Test, “a machine has artificial legal intelligence when there is no discernible difference between the conversation on legal issues generated by the machine and that of an intelligent recently graduated law student.” To put some more definition around the comparison, I would further define a “recently graduated law student” as someone who graduated from an accredited US law school no more than six months prior to the test being administered. The test would use recently graduated law students mixed with those computers in one room, and the judges in another, with the groups communicating by text (no audio). The judges pose questions and try to determine whether it is the recent graduate or the computer answering (see how the University of Reading’s test was run, as described below). I’m going to call this the Jay Test, naming it after John Jay, the first Chief Justice of the US Supreme Court.

There are several significant points to consider about the Jay Test. First, it is a much narrower test than the Turing Test. In the Turing Test, the human can ask the computer about anything, whereas in the Jay Test the questions are limited to legal topics a just graduated lawyer would know. Second, the Jay Test does not assess the quality of the answers. That is, just like the Turing Test, the Jay Test does not depend on the computer getting the correct answer. The question is whether the human interrogator can tell the difference between the computer answers and a human’s answers (both of which could be wrong). Third, the Jay Test does not mean the computer could practice law. As we know, there is a gap between graduating from law school and having the skills to deliver legal services.

The Annual Jay Test Challenge

In June 2014, rumors started circulating that a computer had passed the Turing Test. In this recent challenge at the University of Reading, 30 human judges interacted with a mix of 30 computer programs and humans, with the judges attempting to distinguish between the computer programs and the humans. After more information came out, the consensus among the AI community is that the Turing Test still sits out there without a successful challenger.

I suggest in the legal community that we focus on the Jay Test. We could set up an annual challenge, perhaps at an industry event such as the Association of Corporate Counsel’s Annual Meeting, where a team of judges attempts to distinguish between just graduated law students and legal AI programs. That day when legal AI software could pass the test certainly would mark a turning point in the legal industry.

For those of you interested in this topic, I’d like to hear what you think about the Jay Test and the idea of legal AI software generally. You can reach out to me on Twitter: @leanlawstrategy.