Time for a core metrics set in lawI recently helped moderate part of a webinar hosted by Viewabill during which we discussed a broad range of issues related to using metrics. The panelists were a combination of in-house and law firm lawyers, with some legal operations folks added to the mix to ground us in the real world. The discussions were interesting, and highlighted our evolving views that the practice of law is, and should be, more metric driven.

At one point, we wandered into a discussion about what metrics and standards each participant is using as key performance indicators. It quickly became clear that the answers were all over the map. Each participant had his or her favorite metric, but we had few overlaps. I described the world of legal metrics as the “wild west” right now. We work in an environment where we really don’t have a set of core metrics we can look to and use to compare performance on an absolute (us versus us) or relative (us versus others) basis.

Lawyers have practiced for a long time without a core set of metrics. At first blush, you may think the lack of a core set is not a big deal. But, lawyers also did not focus on efficiency for a long time. In the modern legal world, we are focusing on efficiency and metrics have become not just important, but essential.

To be clear, I don’t think everyone needs to use the same metrics across the board. Each firm and each company will have things important to them and will want to use metrics to measure those things. But, it would help companies and firms alike if there were some core metrics everyone could use both to measure period over period change and to measure how each is doing compared to others.

What are those metrics? That is the open question right now. I have some ideas as do others, but let’s not focus on the specifics. There are some efforts under way to build a core set of metrics and I think we will get there as long as we keep in mind the benefits to all from having the set. The point of establishing core metrics is that we all start using a common language for measuring what is happening in legal service delivery and, more importantly, how we are improving it. Right now, without that common language, discussions about improvement are a confusing mish mash of incompatible tongues.

It really isn’t strange to want industry-wide common metrics. Certainly for corporations, law departments are probably the only department that doesn’t operate in an industry with common metrics. Some lawyers may prefer the lack of accountability, but most clients recognize those fun days are over.

As the legal industry evolves it also needs to mature. For law, that includes developing and using a cores set of metrics that helps all of us keep improving.

 

Metrics help demonstrate lawyer value to an organizationThe end of the year is almost here. It’s time for holiday parties, finishing off year-end transactions, squeezing in a few days off, and gearing up for the next push. When I was running a manufacturing and distribution facility, I discovered it also was time to work on those year-end performance reviews. Gulp, 700 performance reviews! Well, not literally. I had 700 people working in the plant and yes, they all got performance reviews. But, of course, I didn’t personally do all 700. Nevertheless, I had to review many of them, do the reviews for those who did report to me, and make sure overall that our grading worked (not everyone can get As, not everyone should get Fs).

Performance reviews can and should have subjective criteria. But, they also should have objective criteria. We had both for the 700, and having both really helped. Someone might have a great review on the subjective factors, but if they were doing half the work of others in their group and there wasn’t any discernable reason, it was a problem. In law departments, talking about objective criteria can result in a lot of eye rolling, head shaking, and deep sighs of “he doesn’t get it.”

I found, however, that when I brought objective criteria into the conversation as an employee, it helped quite a bit. I could demonstrate, in a meaningful way, what I had accomplished during the year. I included in my self-reviews that I worked on X contracts, assisted in the negotiation and documentation of Y joint ventures, led Z projects, and so on. I also tried to include improvement or quality metrics, where possible.

As a lawyer reporting to a general counsel, these metrics were helpful because they channeled discussions towards where I was doing too much, or where I could do more. As a general counsel, I found metrics to be a great help with CEOs and CFOs. Both CEOs and CFOs are very comfortable talking about metrics. They use metrics to measure the organization’s performance and many key players. Metrics form the basis of compensation discussions (e.g., earnings per share was always a key metric in the executive compensation programs where I worked). The metrics didn’t push aside the subjective criteria, it made the discussions about them more meaningful because there was a quantitative context.

With a new year around the corner, think about what metrics you could keep in 2015 to measure your performance. The beginning of a year is a great time to start tracking things, and doing say takes seconds a day. Start a list, keep it in a drawer, and write down new major projects and tally smaller items. Your New Year’s resolution may fail and you may not succeed in getting to 1,000 situps and 1,000 pushups a day just like the SEAL team members, but it would be nice to explain that by the end of February you already had dusted off 50 contracts, a joint venture, and a couple of EEOC claims.

Law department activity often resembles 10 year olds playing soccer – lots of action, but no clear plan. Compounding general management problems, politics can influence service decisions tilting them away from what needs doing toward who wants it done. As CLOs know, all clients are created equal, but some clients are more equal than others. The CLO who receives an inquiry from the CEO at the same time she receives an inquiry from a junior manager and then focuses on the junior manager’s request, does so at her peril. Ideally, the CLO would focus on service requests based on objective criteria. In this post, I’m going to argue that law departments should adopt more transparent and objective criteria for prioritizing matters and use that system to help address the “more for less” challenge.

The Current State

CLOs operate in delicately balanced ecosystems where the climate is constantly changing. Within this environment, CLOs must manage their law departments efficiently. To most employees, that means turning around legal work quickly, at high quality levels, and with minimum disruption to the business. But, CLOs tend to operate on a somewhat ad hoc basis to meet these client goals. They consider many factors when assigning and prioritizing work, including political power of the client, urgency, perceived value to the organization of the matter, perceived risk of the matter, cost, and existing client demands. The ad hoc process often doesn’t match what a risk-based process would yield.

Today, the prioritization process is mostly art and opaque to the client. When the client requests an estimated turnaround time, she may get a vague response (“I’ll shoot for next week”) or even something less (“As soon as we can get to it”). As new matters hit the law department, whatever the client received as the initial time estimate may shift, as the CLO constantly reshuffles the workload to meet new client demands.

The current system drives many problems, including inefficient workflows (starting and stopping matters), backups on high risk matters while low risk matters move forward, and inconsistent decision-making. It also creates friction between the CLO and the law department’ customers. Usually, no one is happy and the CLO hopes that overall the process prevents a mob from forming at his office door.

The Evolving Risk Climate

Corporations have been living in an evolving regulatory risk climate for many years. As the public’s concern has grown about the impact a corporation can have on the economy when it does not adequately address risks, the SEC has expanded what public corporations must disclose about their risk mitigation practices. Effective February 28, 2010, public corporations were required in their proxy statements (See Regulation S-K, Item 407(h)) to “disclose the extent of the board’s role in … risk oversight, such as how the board administers its oversight function …”

Many corporations had started down the road of implementing enterprise risk management systems by the time the SEC disclosure requirement became effective, though the requirement helped move many others along. Large private corporations have followed the lead of their public peers. The ERMs vary widely, though all ultimately tend to measure two things: (1) the severity of risks, and (2) the probability that the risks will turn into realities. Severity of risk may be measured along more than one dimension. For example, corporations may measure operational, financial and reputational risks and blend the results into an overall severity risk score. The risk scores can be plotted on a heat map with the corporation determining what level of resources to apply to the risk based on where the lines intersect on the map:

Source: Author

For example, during the financial crisis, some corporations scored the risks of one or more banks in their credit line not being able to fund loans high on both severity and probability, placing the risk in the critical box.

A Law Department Risk System

Although a law department will feed perceived organization legal risks into the corporation’s ERM systems, law departments tend to operate without internal ERM systems measuring risks at the matter level. There are many reasons for this gap, but most frequently the explanation comes down to, “because we have never used one.” Change comes hard to lawyers.

A law department can build an ERM system into an existing or concurrently developed triage system for taking in matters. The system does not have to be complex, use expensive technology, or take a lot of time to develop, implement or manage. These reasons are the “usual suspects” for why law departments don’t undertake improvement projects, yet all the reasons can be easily avoided.

Let’s build a simple system for a very small law department. Using Excel, the CLO creates a spreadsheet that lists simple intake information for each matter (e.g., client name, matter name, matter description, matter type). It pays to put some thought into the categories, so that searching or sorting the matters is relatively easy.

For each matter, the CLO also complete four fields, entering a number from 1 to 3 in each field (1 means low and 3 means high risks or likelihood):

  • Reputational Risk
  • Financial Risk
  • Operational Risk
  • Likelihood of Risk

Imagine a client submits a supply contract for materials used in manufacturing one of the company’s key products to the CLO for review. The CLO evaluates the contract risk as follows:

  • Reputational Risk: 2 (if the company has a disruption in selling the product, it will cause some, but not severe, damage to the corporation’s reputation).
  • Financial Risk: 3 (since the product is key for the company, a disruption in sales would have a significant negative financial impact)
  • Operational Risk: 3 (not getting the materials would shut down a substantial part of operations)
  • Likelihood of Risk: 1 (the company supplying materials has done so for 30 years, the CLO’s corporation is its largest customer and stopping supply because of a contract delay would devastate the supplier)

The CLO gives the contract a severity score of 3 ((2+3+3)/3) and a likelihood score of 1 which places the risk in the upper left box of the heat map – a Medium risk.

The CLO uses the same approach for each subsequent matter coming to the law department. If needed, the CLO could use a 5-point scale, giving greater differentiation in the scores. As the CLO plots the risks, it becomes clear where the department should focus its resources. Critical tasks deserve more immediate attention and resources than high, high more than medium, and medium more than low. Sharing the system and the score for a matter with a client will help the client understand why their matter isn’t getting the resource level they think it deserves. It also gives them a chance to challenge the score, if they believe the CLO has overlooked something.

Using the matter risk scores, the CLO then assigns resources to each matter based on its overall risk to the corporation. Risk, of course, does not mean simply the chance that something negative will happen. Risk also includes opportunity costs. The risk of not negotiating a joint venture could be the loss of the opportunity to enter a new territory, resulting in the corporation not realizing a new, large revenue stream.

A Future State

The law department risk system I described is not the perfect system. As the saying goes, “perfect is the enemy of good,” and to start off a law department simply needs a good system. Perfect comes with high costs. But, many lawyers don’t like (hate?) doing anything when they believe the result will be less than the best. Faced with a risk management system like the one I describe above, those lawyers will attack it, criticize it, and ultimately decry that since it isn’t the best they will hold off implementing it until they can design the best system.

Beyond the fallacy that those lawyers actually will design the best system, lies the tremendous opportunity cost of waiting. Imagine that by implementing the system I describe, the law department achieves a 1% improvement in its performance each week for the next year. Calculate the compound effect, and at the end of that year the law department will have realized a 68% improvement in its performance. Putting off improvements today in the hope of coming up with a better system tomorrow typically results in postponed benefits and those benefits are less significant.

Focus on What Matters

The days of managing law departments by gut rather than by metrics are coming to a close. There are no valid reasons for using anything less than modern management approaches to guide a corporation through the rapidly accumulating legal, regulatory, and compliance risks in the global environment. Fortunately, lawyers don’t have to blaze a path to modern management through uncharted lands. Applying sound and proven management techniques, such as risk-based matter prioritization, will help reduce the burdens on law departments while improving workflow and allowing them to do “more with less.”

This is a two-part post addressing quality in the practice of law. In the first post, I addressed the need for quality standards. In this post, I’ll address how I think we can begin developing quality standards useful across clients and firms.

In my first post, I explained the need for quality standards in the practice of law. Today, quality is left to each lawyer and each client to determine on an ad hoc basis. Rather than encouraging uniformly high quality legal services tailored to meet client needs at various price points, this practice has led, I believe, to a broad range of quality in legal services. Even more disturbing, quality does not translate to price. In some cases, clients pay very high prices for mediocre quality, in other cases they pay very low prices for very high quality, and in most cases the correlation (as I perceive it, since we don’t have standards or metrics) between quality and prices is inconsistent.

Out traditional explanation for not having consistent quality standards in the legal industry is that it is difficult to measure objectively something that is subjective. As I demonstrated in my first post, even things subjective can be measured with a relatively high level of objectivity. I don’t see any reason why we can’t bring quality standards to the practice of law.

Contracts without Standards

We can start with a favorite topic of in-house lawyers, contracts. Commercial contracts are the bread-and-butter and bane of in-house lawyers. An in-house lawyer may do a few, dozens or even hundreds of contracts each year. Those contracts may be all of one type, for example distribution agreements, or may comprise many types, including vendor agreements, non-disclosure agreements, professional services agreements, and so on.

Each lawyer brings his or her quality standard to each contract. Typically, the lawyer makes a risk assessment of the contract, compares that to the company’s risk tolerance and standards within the department or organization, and then devotes time to the contract process commensurate with that risk assessment. Generally, this process all happens within the lawyer’s head. When confronted with the next contract, the lawyer repeats the process. Over time, the lawyer hopefully develops some internal consistency in how she performs the risk evaluation. The lawyer in the next office, however, may also be consistent, but reach entirely different results.

When working on the contract, the lawyer again uses an internal quality system. She drafts contract provisions that seem clear to her and she hopes will be clear to her client. She drafts according to her personal preferences and experience. Having read a case recently about the importance of a well-written force majeure clause and the risks of a poorly drafted one, our lawyer revises the contract’s force majeure clause to match what she read in the article. She continues her review, revising the contract by inserting wording, modifying phrases and editing clauses to fit her personal style.

The completed contract is signed and filed. The client is happy, because the signing of the contract means it can move forward with the business arrangement. As time passes and no disputes arise, the client’s confidence in the lawyer’s skills grows, because the contract is serving its purpose. The contract is complex, of course, so when an issue does arise the business people consult the lawyer. Her counterpart in the other company’s law department left soon after the contract was signed, and his replacement is not familiar with the negotiating history. While he can read the contract provisions, he has no context for interpreting those provisions. Without real disputes, however, our lawyer’s client is satisfied and says that she does quality work.

Quality Standards for Contracts

Let’s imagine a different situation. Our lawyer’s first task is to evaluate the contract according to pre-defined risk standards implemented by the law department. The standards have been reviewed and approved by the department lawyers. They based the risk evaluation system on the company’s system, so clients will find it familiar when the lawyers discuss risk with them. Each lawyer who joins the department receives a policy explaining the standards and how to apply them. The policy contains examples from the contracts typically received by the department. After studying the policy, the new lawyer is given a test during which he evaluates contracts according to the standards set out in the policy. Other lawyers review the results and explain where their evaluations differed from those of the new lawyer. This process continues as the lawyer receives and evaluates actual contracts submitted for law department review. After a short period, this step ends because the new lawyer’s evaluations match those of his colleagues.

The law department prepared its risk standards based on the company’s overall risk process. By doing it this way, the law department avoids creating a risk management process separate from the company and at the same time establishes a way of discussing risk that allows lawyers and business people to share a common terminology. The risk map might look something like this:

The lawyers would measure financial, operational, and reputation risk along with the probability of the risks being realized. Placement on the map indicates the overall risk level to the company and guides the lawyer to the path to take for reviewing the contract.

Now that she knows the risk for the contract, our lawyer can compare contract provisions to the department’s playbook. The playbook indicates which contract provisions are important at specified risk levels and gives the lawyer guidance when negotiating those provisions. When the lawyer reaches the force majeure provision in the contract, she remembers the advice she read recently in an article. She prepares a new force majeure provision for her colleagues to review and sends it to them with a copy of the article and a brief explanation of why she is suggesting updating the playbook. If there is a consensus, she will use the new clause in the contract. If not, she will wait for a consensus before using the clause. This improvement process ensures that one lawyer doesn’t get ahead of the department, while at the same time encourages continuous improvement.

As she moves through the contract, our lawyer pays more attention to the critical clauses and the drafting clarity. In several instances, she understands the provision, but believes the drafting is complicated and will make it difficult for the business people to accurately implement the contract. However, before making changes she copies each clause and pastes it into a readability calculator. The calculator, which uses the widely accepted Flesch-Kincaid Grade Level scoring system, returns a 12th grade level reading score for the first clause she checks. Although that indicates someone with a reading comprehension at the 12th grade level should be able to understand the clause, the department lawyers know that is deceptive. After extensive discussion, the department agreed to use a standard of 8th grade for general commercial contracts. The department found that contracts drafted at that level resulted in very few interpretation questions coming to the law department (a nice work reduction) and even fewer misinterpretations and disputes. (A 12th grade score for many legal documents would be pretty good. A much higher score is not unusual. This blog post has a 10.2 score. The Harvard Law Review typically scores in the low 30s. The scores can go down to -3.40, though even children’s books seldom score this low. An exception is Dr. Seuss’ Green Eggs and Ham which has a -1.3 score.) After a few tries, she is able to bring the score down to 8.1 by simplifying the language, breaking up some sentences, and deleting redundant provisions.

Quality Brings Broader Benefits With It

The law department implementing these simple quality measures should see many benefits. Remember that variation is the most significant reason for waste, including poor quality. Reducing variation will reduce waste and improve quality. Our first step was to reduce variation when assessing the risk associated with a contract. By establishing a system through which contracts with similar risk profiles receive similar risk ratings, we have reduced waste. High-risk contracts will receive more attention, reducing the likelihood of disputes (and disputes that don’t go to court still waste many resources). Low-risk contracts will not be over lawyered. The playbook approach means that when contracts are revised, we will have greater consistency across contracts. Two lawyers reviewing contracts will go to the same playbook and use the same substitute provisions when making revisions. Finally, applying the Flesch-Kincaid scoring system to contract provisions will help us simplify drafting. Our clauses will be easier to read and understand, reducing the likelihood of misunderstandings, errors and disputes.

We can check whether we are receiving these benefits by using simple metrics. For example, we could track questions to the law department per contract and Flesch-Kincaid score per contract. The metrics will tell us where we are seeing improvements and where we have further work to do. The metrics also will give us a baseline against which we can measure future improvements. Over time, we will want to bring greater consistency to what we do, adjust risk levels, and continue to improve our drafting.

I recognize that contract drafting involves two parties. Sometimes, the other party will not agree to the changes you want to make. They may refuse to accept your force majeure clause. They may disagree with your attempts to simplify and clarify drafting. But, sometimes they will and in those cases you made progress. When you have difficulty getting the other side to agree, you can rely on some of the information you have gathered to support your arguments in favor of your changes. For example, having a systematic risk scoring system can help you demonstrate to the other party why the changes are important to your client. In addition, being able to show your client where the contract falls on your law department risk scale may help you get your client’s support for the change. Similarly, using the Flesch-Kincaid scale could help you demonstrate to the other party why your clause will help both parties understand the agreement and, hopefully, avoid disputes.

Don’t Let Perfect Come Before Progress

My goal was not to devise the perfect quality system for contract drafting. You may find (and I hope you do) better quality systems to use. I believe, however, that we can apply quality metrics to legal services. Doing so, will help all of us reduce waste, improve quality, and establish a more objective framework for measuring the impact of innovation on the delivery of legal services.

 

This is a two-part post addressing quality in the practice of law. In this post, I’ll address the need for quality standards. In the next post (on Friday), I’ll address how I think we can begin developing quality standards useful across clients and firms.

We talk about quality legal services all the time. Yet, we never talk about quality. Within the legal industry, we have no common definition of quality. In fact, we don’t have any definition of quality. We adhere to a “I know it when I see it” standard. In the past, when the legal industry was not as competitive as today, that standard worked. Today, however, we should look for a better standard and find one that can be used across clients and law firms. Let’s start with the American Society for Quality’s definition of quality:

A subjective term for which each person or sector has its own definition. In technical usage, quality can have two meanings: 1. the characteristics of a product or service that bear on its ability to satisfy stated or implied needs; 2. a product or service free of deficiencies.

The first sentence acknowledges that quality is subjective by person or sector. That should comfort lawyers, since it gives us permission to have a quality definition for practicing law than is different for making widgets. Many lawyers misunderstand quality programs, like six sigma, and think that they will be held to an arbitrary quality standard or a standard more suited to manufacturing than services. We can dispel that myth. Quality varies from industry to industry.

As the ASQ definition notes, quality also can vary from person to person. My wife and I recently had a large portion of our house painted. We hired a painter who had received rave reviews from many families in our neighborhood. He certainly did a good job, but not as good as our prior painter (who was not available). We would not give him a rave review. Our definition of quality obviously differs from the definitions our neighbors use.

As a general counsel, I saw quality differences among outside lawyers and in-house lawyers all the time. Every major firm I interviewed described itself as providing high quality legal services. Yet, looking across the firms I hired to represent my clients, the quality range was enormous. Some firms provided excellent quality, and some firms barely escaped malpractice levels. When I would ask general counsel at other companies for recommendations, I also found that my quality judgment frequently differed from the recommendations I received. One challenge, of course, was that all of our quality definitions were subjective. In fact, we probably weren’t using the same standards. I may have measured quality based on written work product. My peer general counsel may have measured quality based on satisfaction with the outcomes.

Doctors School Lawyers

Some lawyers accept this as just the way it is. But, I think we can do much better. Even when we want to evaluate something subjective, we can come up with effective quality measurement systems. Let’s take an example from healthcare. In 1952, Virginia Apgar, a professor of anesthesiology at Columbia School of Medicine, proposed that obstetricians use a measure she created called the Apgar score to measure the health of newborn babies. The score runs from 0 to 10 and is calculated by summing individual scores on five criteria, each measured on a three-point (0 to 2) scale. A low score indicates a baby is having problems, and a high score indicates a healthy baby. Doctors and nurses score newborn babies one minute and three minutes after birth, and at other intervals depending on the initial scores. The five criteria in the score are Appearance Pulse Grimace Activity Respiration.

Measuring a baby’s appearance for the Apgar score is not an exact science. The scale is based on skin color, where 0 means blue, pale; 1 means body pink, extremities blue; and 2 means completely pink. Obviously, one doctor’s blue could be another doctor’s pink. Yet, today the Apgar score remains the standard for measuring a newborn baby’s health. It works because doctors and nurses become skilled at scoring as they observe newborn babies. Over time, the doctors and nurses develop a high scoring consistency and repeatability. Scores across newborn babies are similar and multiple doctors and nurses give the same score to a newborn baby. The scores are subjective, yet sufficiently reliable that we use them to determine whether a newborn needs immediate intervention to survive.

Medical professionals still use the Apgar score because having an immediate evaluation of a newborn baby’s health is critical. If the doctor does not determine the health of a newborn immediately after it is born, she misses the opportunity to provide treatment that will quickly address a potentially critical health situation. In law, having a quality score for a brief or contract is not critical. A contract or brief does not have that critical immediacy. Most contracts never end up in disputes, and poorly written briefs still win cases. Clients and lawyers have lived without quality measures because they could.

Complexity Breeds Errors

Today, I think we should move to quality measures for legal services even though we can live without them. Legal services are becoming more complex. The more complexity a task or issue involves, the more chances for errors. The errors probably will not be ones that make the difference between winning and losing, but they are errors nevertheless. Errors in service delivery increase the cost of services. Errors in substance also may increase the cost (through corrections), and could impact the result. Without quality measures, we have no systematic way of identifying the root causes for the errors (since we won’t have identified the errors in the first place), and eliminating those causes. In essence, without quality measures we are simply saying we accept living with inconsistent, and at times poor, quality.

Legal service delivery models also involve more complex staffing models. We include lawyers, project managers, process excellence specialists, statisticians, technologists, para-professionals, administrative and clerical specialists, and third-party service vendors. Each of these disciplines brings its definition of quality to matters. Without common standards, we have trouble communicating a quality definition to the team. As teams flex with new members joining and other members leaving, the absence of quality standards makes it difficult for team members to share a common quality understanding. To that add the complexity of working across borders, where quality standards in one country may differ significantly from quality standards in another country. The combinations of these variables ensure that quality will vary significantly from matter to matter, provider to provider, and team to team.

Low quality saps morale, resources, and trust. When I was an associate at a large law firm, I was working on a large lawsuit. We had to issue many deposition subpoenas and it was my job to get them out timely, track our progress and make sure the process was completed on schedule. Unfortunately, I was new to the firm and my definition of quality was substantially different than the standards of those helping me. I spent significant time fixing things that were not done correctly. I kept adding steps to the process, hoping additional checkpoints would improve quality. Those checkpoints certainly added work, but they were not attacking the root causes of the problems. I was very frustrated (as was the lead partner). I learned a critical lesson about getting everyone on the same “quality standard” page. For a team to meet a quality standard everyone must share the same standard and performance metrics must be visible to all team members.

In my next post, I’ll address how I think we can begin developing quality standards useful across clients and firms.

“When Americans say it was great, I know it was good. When they say it was good, I know it was okay. When they say it was okay, I know it was bad.” Laura Klos Sokol

 

We all provide quality legal services. We know that, because we tell everyone we provide quality legal services. No one has shown us a convincing argument to the contrary, so we must be right. But how does a law department demonstrate that it really provides quality legal services? How does it show quality is improving or, conversely, that quality has not declined? General counsel are always on the watch for ways to demonstrate the value of their law departments to organizations, but demonstrating quality legal services is a tough one.

When I moved from being a partner in a law firm to my first in-house position, the general counsel who led the law department gave me his view on how our business clients evaluated the quality of our work. I have tested his theory over the years, and it still seems fairly accurate.

Measuring Quality by Proxy

He explained that our clients had great difficulty objectively measuring the quality of our work. When we wrote or reviewed a contract, the document typically went in a drawer. Our clients might refer to it from time to time, for example when rates changed, to renew the agreement, or to terminate the relationship. But, unless a dispute arose, our clients assumed the contract was good. Even if a dispute arose, they wouldn’t know if the contract was good or bad. Most disputes resulted in a negotiated resolution. Very, very infrequently a mediator, judge or jury might consider a contract and decide whether we won or lost based on a contract provision.

This same story played out many ways. We would give legal advice and our clients would (more or less) follow it. It was the unusual situation that would lead our clients to conclude the advice was not so good. Lawsuits were complicated – was it the facts or some legal work that led to a bad settlement or loss?

Given all the challenges in directly evaluating our legal work, the general counsel concluded, clients used proxies to measure the quality of our legal work. They looked at the finished product. Was the contract clear? Was it free of typos? Did it cover all the issues they wanted covered? If the law department used outside counsel, did the work come in at or under budget?

Our clients looked at other indicia. Was the contract finished when we said it would be, or was it late? What was the experience like working with the lawyers? Were the lawyers always saying “no” or did they come up with creative solutions? In his view, the clients found ways of evaluating the lawyers and used those ways as proxies for  quality legal work.

Quality from the Client’s Perspective

Initially, this approach of measuring the quality of legal work through proxies seemed a bit strange, and even unfair. Having come from a large law firm environment, which prided itself on quality legal work, I focused on the work product itself. We did the research, left no stone unturned, and polished the language. Seniors reviewed what juniors had done, and through cycles of review we generated a high quality product. However, the more I practiced in-house and thought about the issue, and the more I saw how companies evaluated quality, the more I realized quality could have a broader definition.

For the same reasons that measuring the quality of a contract or bit of legal advice was difficult, looking solely to that measure was not useful. If contracts and legal adviceinfrequently were part of disputes, then measuring their quality alone was not that valuable. Measuring the quality of the legal service experience was a more meaningful measure. A great contract delivered late meant less than an acceptable contract delivered on time. Very accurate legal advice resulting in “no you can’t do it” was not as meaningful as “there is some risk, but here is a way to do it.” Quality was more about satisfying the client’s needs within an acceptable time frame and at an acceptable risk level.

As I broadened my quality horizon, using proxies to measure quality really became using criteria relevant to my client to measure quality. I used the same approach when measuring the quality of services I received. A visit to the doctor was measured based on whether I had a long wait, whether the doctor was attentive or rushed, and whether I left feeling my medical needs had been met. I could not directly evaluate the quality of his medical work. When I took my car to the dealer for service, I measured quality on the cost, how quickly I got my car back, whether it was clean or dirty when it was returned, and the interactions I had with the service personnel. I could not directly measure the quality of the work done on my car.

Good is Still Good, and Great is Still Great

Using this broader quality definition did not mean abandoning other quality measures. I did not want our law department clients walking away satisfied only to have the company face a series of lawsuits because contract terms were poorly drafted. I used measures (such ask the total cost of all disputes divided by total legal spending) to keep track of general quality levels. I found, however, that client quality measures were more meaningful in determining whether the law department was meeting the organization’s needs.

I also found that using the same approach to measuring the quality of legal services provided by law firms was helpful. I retained some excellent law firms with very qualified lawyers to help my clients when I was a general counsel. But sometimes, my clients would argue that the organization was not receiving acceptable quality legal services from those firms. The more I dug, the more I found that the perceived lack of quality was not tied to the work product. Rather, it was tied to cost, timeliness, meeting budgets, creativeness, and other factors that formed a part of the legal services delivery. Even lawyers joining the law department directly from outside firms would start using these measures after a short period.

We spend a lot of time fretting about finding the “right” measure for things, and by doing so we waste of lot of time that could have been used to improve things. Measuring quality through various measures such as timeliness, actual cost to budget, and client satisfaction can get us quite far and provide great information for improving what we do, even though they may not be the perfect quality measures.

Experiment by measuring what is most important to your clients and how close you come to meeting their expectations, and you will find you can quickly identify areas for improvement. If you tackle those areas, your client will perceive the quality of your legal services rising.

When it comes to practicing law, metrics are not most lawyers’ strong suit. We focus on the substance, not on the journey. Of course, to some extent that is good because without substance the journey would not be worth it. But, as is becoming more apparent to all of us, focusing only on the substance and not on the journey has some unpleasant consequences. Still, even when we focus on the journey we tend not to measure our performance. Metrics just are not our strong suit.

When I was running a very large manufacturing and distribution center, I had over twenty metrics for the manufacturing side that we measured every day. However, only four of those metrics mattered enough to be used in determining my bonus. At any time, I was expected to know our facility’s performance on those metrics for the most recent day, week, month and quarter. They were productivity, quality, safety, and complete-and-on-time (percentage of shipments delivered to the customer that were complete when delivered and delivered on or before the due date). Overall, our goal was to constantly improve on all four.

Before I give you my four suggested law department metrics, here are a few things to consider. Productivity is important, but not the only thing in life. If you do something very quickly, but poorly, have you accomplished much? Budgets are important and misses are not good, both high and low. Finally, never forget we are in a service business.

My Four Suggested Law Department Measures

With those suggestions, here are my top four metrics:

1) Productivity. I believe every department should have a productivity measure. Ideally, this measure should address the basic productivity question: Are you doing more with less? Is your department accomplishing more value-added work with fewer resources? The resources part of the equation is relatively easy to measure. There is no one right way, but I would take the department’s total outside spending (law firms, vendors, etc.), add to that the total in-house labor cost, and add to that the total non-labor in-house spending (combined, the total law department resources). You could try to allocate corporate overhead, but to make life easy I would skip that piece. For the value-added work, the challenge is more difficult. For purposes of this post, I’ll skip that piece and just use total corporate revenues. So, productivity equals total corporate revenues divided by total law department resources. A $2 billion revenue company that has total law department spending of $10 million has a 200 to 1 ratio. The higher the ratio of revenue to resource use, the better. So, the goal for the next year might be 210 to 1. Revenues could increase to $2.1 billion while total law department spending stays at $10 million; revenues could stay at $2 billion while total law department spending dropped to roughly $9.5 million; or some combination of revenue and total law department spending changes.

2) Quality. This metric often is overlooked. As with value-added work, defining quality goes well beyond the scope of this post. So, for this one I’ll use a somewhat simple metric. I’ll add the total cost of litigation (transaction costs plus settlements plus judgments, minus recoveries) to the total costs of other disputes (arbitration, mediation, and disputes that don’t reach some tribunal) to the costs of regulatory investigations. I’ll divide that sum by the total law department resources. If the company spent $5 million on all of its disputes (transaction costs plus settlements and damages awards) and spent $10 million on total law department resources (which includes the transaction costs for those disputes), the ratio would be 0.5. If the company spent $7 million the next year on all of its disputes, and still spent $10 million on total law department resources, the ratio would increase to 0.7. The lower the ratio, the better. (Yes, you can argue the law department wasn’t responsible for the disputes arising. On the other hand, you could argue that effective preventive legal work would have avoided or minimized the disputes. As law departments become more adept at preventive legal work, the amounts spent on disputes should decrease.)

3) Financial. I would divide the actual spending through a given date by the budgeted spending through that given date. If the actual spending on total law department resources for the first quarter was $2 million and the budgeted total law department resources for the first quarter was $2.5 million, the ratio is 0.8. If the numbers for the second quarter were $3 million and $2.75 million, the ratio would be 1.09. The closer the result is to one, the better. Deviation from one in either direction is not what we want, since it means we were over or under budget.

4) Client Service. I would use a short survey after every matter for each client. The survey would use a five-point scale from very satisfied to very dissatisfied. The survey would cover a few points, for example: Were you satisfied with the service overall? Was the service provided on time? The higher the average score across all questions, the better.

These metrics are not perfect and like all metrics, you can find ways to criticize them. I like them because they are easy to understand, easy to calculate, can be measured frequently throughout the year, and address core aspect of law department operations.

By having a discrete set of metrics you constantly use and by setting goals for improvement, you make it much easier for your team to know what is important and where to focus their energy. Without metrics, everything is important and unimportant, and you shouldn’t be surprised to find your team spending time on things that don’t add much value.

What metrics would you choose? This question comes up frequently so it would be great to hear from you on what metrics you think are best for measuring law department performance.

Normally I would not share this information so publicly, but I am feeling especially generous today. Sometimes, all of us are surprised by how something very simple can help us through really complex challenges. Life is complicated. The problems clients face are complicated. The legal world is complicated and becoming more so as regulations and compliance requirements increase. On top of that, your own life is complicated. How is a simple lawyer supposed to wade through all of this complexity and solve problems?

Ignore the Fishbones

One way to get to the root cause of problems is to use a fishbone diagrams. If this sounds a bit weird, well it is, but it does work. You put the problem at the head of the fish and group causes along the bones. You then target causes and work your way to a possible solution. I’m not wild about fishbone diagrams, so I use a different approach, but get ready for some math!

The Simplest Math You Will Ever Do

Another way to get to the root cause of a problem is the equation Y = Function (x). Y is the problem, and on the right side of the equal sign you list a series of things that might cause the problem (the x’s). If your problem is “I can’t find my car keys,” possible causes could be:

  1. I don’t put them where I can find them again,
  2. other people borrow them,
  3. I forget where I put them, and
  4. people cover them with other things.

For round 1, let’s eliminate the things we can’t control or over which we have little control: numbers (2) and (4). Of the remaining two causes, we decide to focus on number (3).

We have a new problem: “I forget where I put my keys” and that becomes the Y in our next equation. We come up with possible causes:

  1. as I get older my memory lapses,
  2. I have too many things to remember,
  3. I get distracted when I put the keys down, and
  4. someone moves the keys after I put them down.

Again, let’s eliminate the things we can’t control or over which we have little control: numbers (1) and (4). Of the remaining two causes, we decide to focus on number (2).

We continue this cycle, drilling down to the root cause of our problem. Sometimes it only takes a few rounds and sometimes it takes many more. But, eventually, we get to a point where we have a cause we think is worth tackling: I can’t find my car keys, because I have too many things to remember in addition to where I put them.

At this point, we want to test a solution. I’ll write down where I put my keys on a note and put it in my pocket. When I go to look for the keys, all I have to do is check the note. I can test the solution, modify it, or try another solution, and then keep repeating the process until I have solved the problem.

The Power of Y

This equation is a powerful tool. Why are my kids late for school? Why am I tired each day? Why are legal fees too high on litigation matters we send out? Why does it take three weeks to get a contract reviewed? Asking “why” and capturing the possible causes is not a great invention. It simply brings discipline and structure to something, which many times we take for granted. We handle many things as part of our random walk through life. On Monday we do it one way, on Tuesday another, and on Wednesday we forget to do it.

By introducing a bit of discipline and structure, you can turn things around. You problem solve left and right. You are the guru, the new soothsayer. All of this, and you simply learned a bit of math.

 


McKinsey recently hosted a discussion among eight company executives who are all on the frontlines of their respective company’s data-analytics efforts. At first blush, it may seem there was no reason to include lawyers. But, read the topics they covered and think again:

Are data and analytics overhyped?
Do privacy issues threaten progress?
Is talent acquisition slowing strategy?
What organizational models work best?
What’s the best way to assure adoption?

The Missing Seat at the Table

Are there any topics on the list where lawyers should not be involved? This isn’t the hubris of “we are lawyers and know everything.” This is a question of the trusted advisor role and helping our clients craft business strategies at acceptable risk levels. Clearly, lawyers have some insights on the privacy issue. But each one of the issues listed above has legal implications that could affect the business strategy. Even outside the legal aspects, lawyers as senior executives in their organizations must think about the implications of various alternatives on a host of issues. Would an organization model that crosses jurisdictions raise compliance issues? Do various talent acquisition strategies raise potential complications in light of evolving legislative concerns?

Thinking in Three Dimensional Chess

The increasing balkanization of responsibility areas within organizations (CMOs have experts in social media, etc.) and the speed of change, means that traditional communication lines become more complex. Lawyers and law departments remain uniquely suited, typically sitting as a node where many of these communication lines intersect, to help evaluate and integrate diverse strategies flowing through many departments. Those communication lines flow horizontally, but also vertically and diagonally as lawyers work with businesspersons at all different levels in the organization.

Unless lawyers show leadership on the business side of these issues, they will be challenged to get invitations to the discussions that help shape strategies. How often do we note that, if we had just made it to the table earlier, we could have helped mitigate or avoid serious risk?

I encourage you to read through the McKinsey article (“Views from the front lines of the data-analytics revolution”) and think why you would not want to be involved in that discussion. Part of our dialogue about the future of law should include the future of lawyers as analytic strategists.

Did you go to law school to study statistics? Did you become a lawyer to use statistics? Or, did you become a lawyer in part because you were running in the opposite direction from the STEM (science technology engineering math) crowd? I’m going to guess most of you (IP lawyers excluded) hoped to avoid math after taking the obligatory introductory math class in college.

You Didn’t Run Fast Enough

Well, surprise, math – in the form of statistics – is back. We are in the age of big data. Companies use analytics in many aspects of their businesses. You may say “so what.” After all, the business folks may use statistics, but that doesn’t mean you need to use statistics. In my opinion, that would be a shortsighted view.

Statistics is quite relevant to the practice of law. For example, as an employee benefits lawyer, statistics can help you with executive compensation issues (the boards of directors I have worked with asked plenty of questions that required statistical analyses). If you are a litigator, understanding statistics can help you with discovery issues. If you are a corporate lawyer, statistics can help you answer questions such as “is this provision market.” The list goes on, but you get the idea.

So What’s the R?

“R is a free software programming language and software environment for statistical computing and graphics,” says Wikipedia.  For example, you could use R to analyze how much your company spends on lawsuits. You could use it to analyze settlement payments, or time to get a trademark registered, or steps to complete contracts. You could use it to analyze spending patterns in your law department budget. Whether you use R or something else, software for statistical analyses is widely available.

Lawyer, Know Thyself

 If you haven’t read Thinking, Fast and Slow by Daniel Kahneman, you should. When you do, you will learn the fascinating story of how your brain thinks. In particular, you will learn how your brain avoids statistical analyses and how that hurts your decision-making process. In fact, by not triggering statistical decision-making, you are limiting (and sometimes hurting) your judgment as a lawyer.

One short post can’t do justice to the world of statistics and how important it is to your future. But, knowing that your future should involve understanding statistics to improve your judgment calls for clients, I hope you will take the next few steps.