AI APPROACH AND TRIALS

Once it became clear that generative AI was a powerful tool that could impact the way the legal sector operated, a cross-department group was set up within Eversheds Sutherland, including different teams, both legal and non-legal. Its task was to investigate and evaluate the generative AI landscape and decide on the best approach for the firm.

The team was led by the Client Excellence Partner, part of the Legal Technology team, and included members from almost every team in Eversheds Sutherland, from lawyers to the IT team, the Governance and Compliance team, the Marketing and Comms team and most importantly the Knowledge team, with the Head of Knowledge sitting on the board of this team from its inception. The Knowledge team’s role was varied, including:

• Acting as a first point of contact for knowledge generative AI products

• Arranging and assisting in the running of trials

• Gathering feedback from trials

• Assisting in cost negotiations

• Assisting our Governance & Compliance team in negotiation of terms and conditions

• Onboarding users to new knowledge generative AI products

• Arranging and promoting training for knowledge generative AI products

The first decision that Eversheds Sutherland faced when looking at generative AI was choosing which tools should be looked at. At times over the past 12-18 months, it has felt as though a new generative AI knowledge tool has been announced every week. A decision had to be made on which tools would take priority. These are new tools offering new functionality, so this required additional time and effort to assess them, and it was important to ensure that teams had the bandwidth to do this properly.

When assessing generative AI tools, Eversheds Sutherland’s approach was to return to first principles, looking past suppliers’ marketing spiel and focussing on what the tool actually does. This was guided by the following questions:

• How will this impact the way a lawyer works and the advice they can provide?

• What can we do with the content provided by the tool?

• Is the underlying data the tool is ‘learning’ from of a sufficiently high quality?

Once a decision had been made on which tools would be assessed, it was important to ensure that the right people were involved. For legal tools the most important participants were legally trained lawyers and PSLs, but not exclusively. Non legally trained people can still provide useful feedback. For example, the Knowledge teams will be familiar with the legal databases that most generative AI tools will be part of and can both assist in the administration of trials and evaluate the quality of the tools.

It was then important to decide on what metrics would be used to assess the tools, and how we could best assess the tool’s functionality, as well as the value it would bring to our clients. Early in the process it was also key to identify any new areas that the IT and Compliance teams would have to review.

Another important consideration when making the decision on which tools to look at was cost. Before a decision is made it is essential to confirm where any AI cost will sit. At this time, generative AI is still an amorphous concept. It could be an IT or Knowledge cost, or a firm may have its own separate budget for it. When deciding on which tools to review we did not want to introduce any tools that people may like but are not financially viable to subscribe to after the trial. Generative AI tools are not cheap and represent a significant investment, around the cost of a major legal database in most cases.

Compliance and IT teams were involved at the earliest opportunity possible. Generative AI tools are still a new technology and both law firms and suppliers are feeling their way through this. There is not a great deal of precedent for the functionality of the tools and the changes in usage this presents. This meant it took additional time for agreements to be negotiated and approved. Which is why for Eversheds Sutherland the IT and Compliance teams were involved from the very beginning, confirming that they were happy for us to even trial the tools.

The approach Eversheds Sutherland took was to look at the generative AI tools from major UK legal database providers (LexisNexis, Thomson Reuters, vLex) initially, then tools for our non-UK offices, and finally the more niche subject specific tools.

The questions listed below were at the core of Eversheds Sutherland’s approach:

• Who should be involved in the trials?

• What feedback mechanism should be used and what are the important metrics?

• What value would the tool add to client work?

• Does it justify the cost?

• What areas will concern the Compliance and IT Security teams?

As generative AI tools offer new functionality it was clear from the beginning that it would require more time and effort than for traditional non-generative AI knowledge tools. Because of this it was important to ensure that the trial participants were willing to commit to this before the trials began. Internal meetings were held, initially with those who had expressed interest in being involved in the trials, to let them know exactly what was expected of them.

In order to maintain the level of commitment expected throughout the trials, a billing time code specifically for generative AI trials was created. This allowed participants to dedicate time to testing and have this reflected in their time recording. We also created a dedicated Microsoft Teams chat for the participants, allowing them to discuss the tools and any problems or solutions encountered, and to offer live feedback whilst the trial was ongoing.

Knowledge was heavily involved in the trials process:

• Pulling together lists of trials participants

• Setting up initial meetings to set expectations

• Arranging introductory supplier training and any additional training throughout the trial

• Collating feedback from the trial

• Arranging the meeting for the final decision on which generative AI tool to progress with

The Compliance team was also involved from the very start of the trials. Again, taking a first principles approach looking at:

• What the tool does and how this will impact the work that the lawyers do

• How the functionality changes the way data is used within a product

• How long should the subscription be?

• How can we futureproof the subscription to ensure we have access to future developments?

The biggest issue Compliance flagged was the change in the way data was used within the tools. There were two main issues arising from this.

Firstly, to use generative AI tools effectively a lot more data is required than for traditional knowledge tools, possibly including client data. A number of generative AI tools also allow the uploading of documents for analysis. Because use of the tools allows the use and sharing of sensitive data the team had to look at:

• Where is the data stored and how long is it retained for?

• What measures are taken to secure the data?

• What legislation covers the area where the data is stored?

This was a key issue from the beginning. Within the trials, no ‘live data’ was used. Trial participants came up with their own theoretical scenarios to test the tools as Compliance had not had the opportunity to properly assess the data security which suppliers had in place for the tools.

Secondly, certain clients had made it clear that they did not want any of their data being used with generative AI tools. This meant that processes had to be agreed to ensure that this did not happen. The majority of the tools assessed by Eversheds Sutherland were accessed via single sign on (SSO). This meant it was difficult to turn off specific features for specific users. It is also important that users have access to the tools for work related to other clients. Processes had to be put in place to protect this client data.

Client security has been at the centre of Eversheds Sutherland’s approach since our AI policy was first introduced. Even if a knowledge tool offers a free upgrade to a generative AI tool the policy is to opt out until it has been reviewed and approved by our Compliance and IT Security teams. Eversheds Sutherland will not use any generative AI product unless we can ensure it meets the expected standards for quality and data security.

This was reflected in the time it took to agree terms and conditions once a decision had been made on subscriptions – it took approximately six months from agreeing a subscription to final signature of the subscription agreement taking place.

TOOL ONBOARDING AND TRAINING

Once the trial had ended Eversheds Sutherland decided on a tool to move ahead with and agreed an initial 12-month contract. Was the hard work over at this point? Not by any means. Now the tool had to be announced and rolled out to the firm, a training approach had to be agreed and a decision on how to monitor usage and measure the value the tool offered also had to be made. This is an area where the Knowledge team took the lead.

The tool Eversheds Sutherland chose to move ahead with was Lexis+ AI. The initial announcement of this came in a firmwide email from the Managing International Partner. It was felt that this would reach more readers than an email from Knowledge. The email specified which tool had been chosen, why it fitted Eversheds Sutherland’s needs and how the tool would be rolled out. Following this, all subsequent communication came from the Knowledge team. This included an email to all fee earners with details of the training that would be available and how they could sign up for these sessions.

Training was provided by Lexis, with sessions tailored for each fee earning team. The sessions covered how to use the tool efficiently (how to use the different features, how to review the answers generated and to access underlying data). Though the sessions were provided by Lexis, Eversheds Sutherland’s AI policy informed the content. The core of this policy was that generative AI tools are an addition to, not a replacement for, traditional legal databases and research. The sessions were tailored to each team so that the lawyers were more invested in them. After these sessions a rolling programme of general sessions was arranged for people who may not have been able to attend their team specific training. Take up for all of the training was very high.

In addition to this, a Lexis+ AI training page was created on the intranet. This included video tutorials and recordings of the training sessions and quick reference guides provided by Lexis. Again, footfall to this page has been encouragingly high.

Once the tool had been launched and people were trained and had access, the hard work for the Knowledge team was far from finished. In fact, the work post launch brought new challenges. It was important to look at how usage was tracked and what represented a return on investment for Lexis+ AI. The tool is not cheap, representing a significant investment for Eversheds Sutherland. What return on investment would be acceptable and how would the team measure this?

For traditional resources the Knowledge team has relied on a mixture of both quantitative and qualitative metrics to measure value. Usage data is used to see what resources are being used and by whom, whilst the team’s relationship with contacts within legal teams is leveraged to know what value they place on the resource. A resource may have very low usage statistics, but for a small team it is essential and needs to be retained.

Would this approach work for a generative AI product? The tools are not giving us access to new content in itself. Theoretically a lawyer could create the content the tool provides themselves. It is the time that is saved using the tool that offers value. The team needed to look at how to calculate how much time the tool saves and assign a monetary value to this.

So far, the Knowledge team have adapted their current process, but with more emphasis on qualitative data.

Usage data was collected and evaluated to see how the tool is being used and what proportion of potential users have accessed it. A control group composed of trial participants was also created to provide regular feedback via a questionnaire. The questions on this are as follows:

• Which functionalities have you used in the previous quarter?

• What specific legal tasks did you accomplish?

• How accurate and reliable were the outputs generated?

• How did Lexis+ AI affect your work and productivity?

The questionnaire was circulated to the control group quarterly as well as an additional, randomly selected, 30 users. A link to the questionnaire was also made available via the Lexis+ AI intranet page for other people to contribute. The approach is that lawyers are best placed to judge what time savings and efficiencies the tool provides, whilst usage stats can be used to identify areas that are underused. This helps inform the Knowledge team’s approach to the tool. Is a feature underused because additional training is required, or does it not offer value to the users? Lawyers are best placed to evaluate how much time the tool is saving, so feedback is an essential part of the return on investment (ROI) calculations.

DIFFERENCES BETWEEN GENERATIVE AI AND TRADITIONAL KNOWLEDGE TOOLS

One of the main differences between adopting generative AI and traditional knowledge tools was the time this took. The generative AI tools are new technology and there is little precedent for their functionality and how users interact with the tools. All aspects of the subscriptions, from trials to negotiations to training, took a significantly longer time than traditional knowledge resources. IT Security and Compliance teams had to be involved earlier than normal to make a proper assessment of how sensitive data would be used within the tools and what procedures had to be put in place to protect this data.

Secondly, there was a big difference in how the value of the tools is measured. AI tools offer a new way of accessing content, not necessarily new content themselves. This required a lot more involvement from lawyers to evaluate the value, looking not at what is available but how available content is transformed by the tools.

This evaluation has continued post subscription to Lexis+ AI. The market is developing so quickly, with new tools launching and improvements being made to existing tools, that it is essential to maintain awareness of all products to ensure that the firm has access to the best possible tools.

SUPPLIER CONTENT AND GENERATIVE AI

A different challenge Eversheds Sutherland faced when evaluating generative AI was working with suppliers to integrate it into lawyers’ workflow. This focussed on negotiating permission to use proprietary material in generative AI tools. For example, ensuring that agreements would allow the use of this material in Microsoft Copilot to summarise a document, or an internally developed generative AI chatbot to access material on the intranet.

For this, Eversheds Sutherland again returned to first principles, working with suppliers to confirm what they consider AI, and whether they make a distinction between generative AI and non-generative AI. Since generative AI captured the zeitgeist, lots of products which were not previously AI, such as spell checkers and translation tools, have become ‘AI powered’. It is essential that the distinctions between these products and genuine generative AI products are agreed.

Once the definition of generative AI had been agreed with the relevant suppliers, exactly what the purpose of using the content in any generative AI was clearly set out, making it clear that any content used would be stored locally within Eversheds Sutherland’s internal servers and would never be used for training on a generative AI product. This strategy was not 100% successful, with some suppliers declining to give permission for any of their content to be used in any generative AI tools. This has led to additional messaging from the Knowledge team to ensure that users are aware of any restrictions on the content delivered by these suppliers.