THE VALUE AND DIFFICULTY OF MEASURING

There are many ways that knowledge artifacts and flows can benefit organizations (see Figure 1).

Figure 1.

The benefits of Knowledge Management.

Measuring the numbers of knowledge artifacts or activities is not usually difficult. You could probably easily find out how many articles you have in your firm's database or how many training meetings took place last month. The difficulty lies in understanding how those artifacts/activities are acting on your firm to help or hinder it, in its goals; understanding the quality of those knowledge artifacts/activities and whether they provide a good return on investment.

When considering the effect of artifacts and activities, there are usually many complex factors at play. The effects of a single knowledge project are intricately entwined with other business systems and affected by serendipitous events and events outside the organization's control.

For example, a lawyer is mulling over a complex issue and is planning to look at the firm's knowledge bank before drafting a complex document. He decides to get a coffee first. Whilst making his coffee he chats to another lawyer and discovers that she has recently dealt with the same issue, researched it intensively and found a neat solution which her client was delighted with.

This is clearly a successful knowledge activity/interaction: the first lawyer finds a solution in an efficient and effective way which will please his client and the second lawyer is not particularly inconvenienced. What, however, would be measured and what would this interaction tell us about this firm's KM systems? Nothing would be recorded in any database viewing/download figures and it may not be recalled by either lawyer during an annual knowledge survey. Is this a success for the knowledge-sharing culture, a failure of the knowledge database, neither or both?

Similarly, when lawyers are trying to understand whether it is more cost-effective to impose a particular workflow or checklist, this may have a more complicated result than a simple change in billings or profits (see Figure 2). There could be lower chargeable hours per matter and lower billing levels in the short term, but higher quality advice, higher customer satisfaction, lower professional negligence claims, lower attrition rates and more matters undertaken or higher profit fixed fees in the long term.

Figure 2.

The effects of Knowledge Management projects.

How well KM activities work in a law firm depends greatly on its culture, history and the dedication of the individuals involved, as well as factors outside the firm's control.

Although it is undoubtedly difficult to value ‘knowledge’ and knowledge activities and systems, the pragmatic truth is that those who run organizations need to understand what return on investment they will get for their spending on KM, which projects or schemes ought to be rolled out more widely and which projects or activities should be changed.

WHERE TO START WITH KM MEASUREMENT

Figure 3 shows the process for designing and planning a measurement system for a particular KM project.

Figure 3.

How to create and use measurement systems in KM.

I recommend that you start with the following:

1. 1. Why: what is the purpose in measuring

2. 2. Who: who is your audience for the measurement

3. 3. How: understand the practicalities of measuring for an accurate result.

You should also keep in mind these two truisms:

What is measured within a business tends to get noticed and improved. Things that aren't measured often languish on the ‘to do’ pile.

“You get what you measure” Footnote ²

Measurement itself can have unexpected and often negative results, with employees focusing on improving figures themselves rather than the business case behind those measurements.

‘WHY’: UNDERSTANDING YOUR PURPOSE

Your first task is to understand your purpose for the measurement:

1. 1. What is the business benefit of knowing this answer?

2. 2. What are you going to do with the results?

You need to take the appropriate level of measurements to inform and support your decision-making and action. There is no point in measurement for measurement's sake and too many measurements can be counterproductive as they can create confusion.

If you can't take action based on the measurement in relation to that KM project or activity, don't waste time and money on that aspect unless you are gathering evidence for a known future action.

If you are unsure whether your measurement is actionable, ask yourself whether you want the measurement to go up, down or stay flat. If you don't know which way you want the measurement to go, or don't know what you will do in each situation, the measurement is not actionable.

Common reasons for measuring KM activities are:

• • Improvement/quality and

• • Judgment/accountability

For example, most KMers will be measuring to find out:

• • If their actions or an investment are adding value to the organization.

• • If a particular implementation is on track, a particular approach is working and to identify gaps in their approach.

• • If they should expand a pilot KM project, tweak it or shelve it.

• • What their users/stakeholders natural modes of working are, in order to design KM activities which suit them and so are more likely to be adopted.

• • If their stakeholders are fulfilling their part of the “KM-bargain”, and if not, how the system can be adapted to encourage more participation.

• • Identify the key barriers to engagement.

Your ‘purpose’ for measuring will drive the development of your measurement systems.

Given how difficult it is to understand the effect of KM activities and projects on business outcomes (see Figure 2), I recommend that you take a balance of measurements. The complexity of your measurement system will depend upon the size and nature of the project and budget, but I recommend that you at least include qualitative and quantitative, leading and lagging measurements for even small projects. Whether you measure across all four perspectives of the Balanced Scorecard (see Box 1) depends on the project and its budget. Too many measurements are as unhelpful as too few.

For example, you run knowledge creation and acquisition projects as part of your KM work and you would like to trial a change to a training and learning programme by swapping from in-person training lunches based on lectures, to online videos and quizzes. You could wait until the programme has run its course, had a positive or negative effect on the knowledgeability of your staff and led to a measurable change in numbers of clients and profitability, or you could take leading measurements to give you early warning if the programme isn't working. Leading measurements in this case could include the numbers of people taking, finishing or passing the online courses. If the numbers of people wanting to start the course drops, word may have spread that people do not feel the courses are worth investing time in. If people are keen to start and finish, but fail the quizzes, you will get an early warning that something is not right with the content or delivery of the knowledge.

WHO: UNDERSTANDING THE AUDIENCE FOR THE MEASUREMENT

You measure your KM projects in order to take some form of action, so it doesn't really matter how sound your dataFootnote ⁶ is, if you cannot communicate the results in a way that is persuasive for those who need to take action.

Common audiences for the measurement are:

• • Those running KM activities or projects;

• • Users, stakeholders or communities your KM activities/projects serve;

• • Managers and directors at various levels in the firm and those who allocate resources.

Once you've clarified your audience, you need to ask yourself:

Concentrate on the measurements or KPIs within the four Balanced Scorecard perspectives that your audience cares particularly about. You can look at financial stewardship in several ways, but your senior managers probably have a favourite measurement they always take notice of. No matter how great you think your measurements are, if they don't persuade your audience, who will need to take action, they haven't fulfilled their purpose.

If you are uncertain about your audience's preferences, ask them. This isn't a matter of ‘sexing up’ data to mislead. It is about avoiding wasting time measuring one aspect when a different but similar one would have a more persuasive effect on that audience. Even if their preferred measurement isn't the perfect one in your opinion, still use it. It has the benefit of being established and authoritative to your audience. If you really can't use it, explain how your measurements link to it.

For example, you are measuring to demonstrate that a particular KM project is making a difference to profitability and ought to be rolled out more widely and your audience is senior partners and your Executive Board.

Ask yourself:

• • What do those senior partners care about? What keeps them up at night?

• • What data have they previously taken notice of?

• • What “language” do they listen to and speak?

This audience may worry about profitability, levels of innovation when new competitors enter their market, or key staff attrition after a merger, depending on what is happening in the firm at the time. They may mainly care about profit per equity partner or they may be focused on market position and client satisfaction surveys.

Alternatively, if you are using a measurement to help you understand whether a smaller KM project appears to be a beneficial use of your own limited time, you may focus on measurements which you personally find compelling. For example, you think a project relating to serendipitous conversation will improve trusting networks and person-to-person knowledge sharing. It isn't a complex, expensive, unpopular or onerous project, so you may want to minimize the cost of measurement and focus on a simple one-off qualitative measurement, such as a survey or focus group. Once you have satisfied yourself that the project appears to be working and is worth rolling out further, you may widen measurements across the four balanced scorecard perspectives to ensure that there are no unanticipated negative results which could steer your firm off course and which will include measurements meaningful for the new audience, who may not be persuaded by a focus group, but may be persuaded by a reduction in early notification of potential professional negligence claims or an increase in client satisfaction measurements in relation to knowledgeability of staff.

Try also, where possible, to gather stories which illustrate your data, as well as the data itself. Whilst an increase in general client satisfaction measurements may be compelling, hearing the audio file of the actual interview with a client expressing in their own words how the knowledgeability of staff persuaded them to place more orders is even more compelling.

HOW: DECIDING ON THE PRACTICALITIES OF MEASUREMENT

Once you are clear on the purpose of your measurement and its audience, you will need to decide on the practicalities. Your measurements should be reliable and valid and, for larger projects, you should spell out your methodology.

You need to be clear about:

• • What measurements you are going to take and how you are going to take them;

• • Who will collect the data;

• • How, when and how often will they collect it.

You will also need to consider how much you can afford to spend on measurement. Some measurements will be collected in any event so are low in cost (although don't forget that you'll still need to analyze the results) and some will cost money to arrange, collect and analyse. The nature of your KM project and its potential to affect your organization will determine your budget for measurement.

As you are only measuring in order to take action in relation to your KM project and activities, you need to have a trustworthy baseline from which to note changes. You must also ensure you take the same measurements in the same way before and after the change for the results to be reliable (comparing ‘apples with apples’).

A balance of measurements (using qualitative and quantitative, leading and lagging, across the four facets of the balanced scorecard) can get us closer to understanding what is happening because of our activities and overcoming the difficulties outlined above. It will also help us to even out any gamingFootnote ⁷ (see Box 2).

User surveys are extremely valuable for qualitative research, but can't be used too often because of user-fatigue. Surveys can include both qualitative and quantitative elements and can focus on one project or activity or combine questions about several. Where you would like to distil an opinion into an easily comparable figure, I recommend using a Likert Scale (where your respondents rate their response from 1 to 5, 1 being ‘strongly disagree’ and 5 being ‘strongly agree’). There is more advice on how to write a good survey at Annex 4. Informal discussions are just as important as formal soundings, so also get to know what people think by talking to them.

At this point, I recommend that you take a moment to read and understand the ten measurement design tests (Figure 4 and Annex 3) described by Professor Andrew Neely, Pro-Vice-Chancellor, Enterprise and Business Relations, University of Cambridge and expert in designing KPIs and measuring business performance.

Figure 4.

The Neely Design Tests.

Whilst not all KM projects will be sufficiently large to warrant measurements which fulfil all these design tests, these tests are a great checklist to ensure that:

• • You measure only what is necessary and relevant at the time that it is needed;

• • Your measurements are the easiest and cheapest measurements to take, and remain consistent whoever gathers the data;

• • You continue to measure only what you set out to measure.

You need to understand what is truly important to the firm to ensure that you concentrate on the right aspects. Once you have your data, remember that all measurements are not equal in value: ‘non-chargeable time saved’ within a law firm is unlikely to be more important than quality, firm reputation or new business won, unless the changes are sizeable.

ANALYSIS OF DATA

As part of this practical guide, I won't be delving deeply into statistical analysis. My own experience has been that, in most cases, only simple tools are needed. You are primarily looking for relationships and trends, i.e. when X happens, what happens to Y? You will usually just need percentages, averages, ratios and divergence from the mean/median/mode. If the picture you uncover appears to be more complex, I recommend you tap into your network for help from someone with the right data analysis skills.

REPORTING RESULTS

No matter how carefully you collate your data, you will need to communicate your results in a compelling way for it to be persuasive.

• • Engage with your audience's needs and wants.

• • Give enough context so that people can understand the results.

• • Include the human face to the figures – a story or anecdote will have impact on your audience.

• • Decide where and how to publicise the results. If you plan to release the results on to the intranet, who will see it? Do you want all the results available to everyone or do you want a detailed version for the Executive Board and a slimmed down version for the whole organisation? What state of mind will your audience be in when they receive your results? How can you maximize their understanding?

UPDATING YOUR MEASUREMENTS

As you gain experience in measuring your KM projects/activities, you will need to revise and update your measurement system. This could be for many reasons, but the most common are:

• • Some of the goals (the ‘Why’) need refining.

• • Your firm has grown, merged, adapted to different customers, changed its IT systems or similar.

You will need to identify any overlapping measurements and simplify them and you will need to replace measurements that are proving to be unhelpful, or those that have become too costly to justify. You will also learn which measurements work particularly well with which audiences.

All the time you will need to keep a delicate balance between keeping measurements the same (so you can identify trends over time, measuring ‘apples against apples’), adapting to new circumstances, and simplifying and focusing your measurements.

I recommend that you revisit your measurements and the Neely design tests at suitable intervals, perhaps annually, and adapt the measurements as appropriate.

SUMMARY

Use Figure 3 page to help you plan and implement your measurement systems.

1. 1. Define the business objectives of measurement and understand what success will look like;

2. 2. Identify those being measured and the audience for the results—each stakeholder views KM and defines success differently;

3. 3. Decide what data will be collected and how. Use Neely's measurement design tests to ensure your measurements are appropriate;

4. 4. Analyze and communicate the measures in the language of your audience;

5. 5. Take action in relation to business objectives based on the results;

6. 6. Review your combination of measures to ensure the right balance of quantitative and qualitative, leading and lagging measurements, and adjust as your goals develop;

7. 7. Keep under review unexpected consequences and ‘gaming’ the system of measurement;

8. 8. Do not measure everything—focus on what's important.