Estimating agent performance in a call routing center system

Abstract

systems and methods are disclosed for estimating and assigning agent performance characteristics in a call routing center. performance characteristics (e.g., sales rate, customer satisfaction, duration of call, etc.) may be assigned to an agent when the agent has made few calls relative to other agents or otherwise has a large error in their measure of one or more performance characteristics used for matching callers to agents (e.g., via a performance based or pattern matching routing method). A method includes identifying agents of a plurality of agents having a number of calls fewer than a predetermined number of calls (or an error in the performance characteristic exceeding a threshold), assigning a performance characteristic to the identified agents (that is different than the agent's actual performance characteristic), and routing a caller to one of the plurality of agents based on the performance characteristics of the plurality of agents.

Description

Using equation (9), and taking point 1 as the n=0 case, as point 2 as n=N, gives:
R=m×0+c (Point1)
p=m×N+c (Point2)  (10)

From which it follows that:

$\begin{array}{cc}m=\frac{p-R}{N}& \left(11\right)\\ c=R& \end{array}$

Therefore the calculation of adjusted agent performance, p_adj, for an agent with n calls, where 0<n<N, can be determined as follows:

$\begin{array}{cc}{p}_{\mathrm{adj}}=\frac{p-R}{N}n+R& \left(12\right)\end{array}$

With continued reference to FIG. 7, in one exemplary process, an average characteristic is determined for a plurality of agents at 702. For example, with the illustrated example described here, R from equation (7) is computed. Further, agents are identified having an error, e.g., a fractional error, exceeding a predetermined threshold value at 704. The agents may be identified by computing equations (1), (2), and (3), for example.

If the error exceeds the threshold, e.g., if e>t, a performance characteristic may be assigned to the identified agents at 706. For example, N may be determined from equation (6) and used to determine an adjusted agent performance, p_adj, from (12). It is noted that equation 12 provides an adjustment to an actual performance characteristic of the agent, the adjustment based on a liner interpolation between two points. In other examples, however, other interpolations may be used for adjusting/assigning a performance characteristic. Additionally, a substitution of a value unrelated to the agent's actual performance characteristic may be used, e.g., the average rate or some fraction of the average rate.

The final selection or mapping of a caller to an agent based on actual and assigned performance characteristics may then be passed to a routing engine or router for causing the caller to be routed to the agent at 708. It is noted that the described actions of the exemplary methods described do not need to occur in the order in which they are stated and some acts may be performed in parallel. Further, additional matching models for scoring and mapping callers to agents may be used in a similar fashion, and a plurality of matching algorithms may be used and weighted against each other for determining a final selection of a caller-agent pair.

Determining a threshold number of calls or error value in practice may be evaluated and selected in many different manners. For example, typically, one would like to set t such that the performance of a routing system is optimized. For example, using a small threshold may increase performance, however, if an unnecessarily small value is used, the system will needlessly reduce performance accuracy for agents who have a total number of calls <N and whose performance is far from the mean.

One manner for determining the effect of and selecting (or changing) a threshold value is with a Monte Carlo simulation. FIGS. 8A-8E illustrate graphs for an exemplary method and Monte Carlo simulation for selecting a desired threshold for estimating agent performance characteristics. In one example, an overall distribution of agent performance is chosen, which can be viewed as the true agent performances, i.e., with no error, as would be found for an infinite sample of calls for each agent. FIG. 8A illustrates an exemplary near normal distribution of 1000 agents with mean AP=0.3 and SD=0.1 in a Monte Carlo simulation (note that the handful of negative performances were set to zero).

For a given number of free agents (shown from the set {2, 5, 10, 20, 40}) and for a given value of t (shown from the set t=0, 0.1, 0.2, . . . , 1) free agents are selected randomly from the above distribution. The agent having the maximum true AP of the set of free agents is determined, and is therefore the correct agent for an exemplary performance based matching algorithm to select.

Next, each of the true agent performances is dithered (i.e., noise or error is intentionally added) to simulate the actually measured agent performances with an error due to the finite sample. For example, the dithering is applied by adding an error term to each agent's AP of N(0, 1):
p_Dithered=p_True(1+tN(0,1))  (13)
where p_True is the true agent performance, and N(μ, σ) is the normal distribution.

From this, one can check that the actually selected agent (i.e., one with maximum dithered agent performance) is the same as the one chosen in step (3), and if not, record the error in performance of the selected agent that occurred. This process can be repeated (e.g., 1,000 or more times) for each combination of number of free agents (e.g., 2, 5, 10, 20, and 40) and value of t. FIG. 8B illustrates the Monte Carlo results for this example; in particular, illustrating the percentage of calls routed to the agent with maximum true AP of the available agents versus t. As one would expect with at of zero, the correct (i.e., highest performing) agent is always chosen and the fraction of correct agents selection declines as t increases. Also, the loss of selection accuracy increases with the number of free agents available.

Another metric one can consider in determining a threshold value is the absolute value of the difference between the true agent performance of the actually selected agent (based on the noisy AP values) and the true agent performance of the agent that would have been selected (had the selection been based on true agent performances). FIG. 8C illustrates absolute value of error in performance of selected agent versus t (where the number of agents increases the mean absolute value of error for a given fraction error in agent performance, as shown).

Additionally, since the range of agent performances available varies between different mapping implementations, another metric includes expressing the error as a fraction of the standard deviation of the agent performance. FIG. 8D illustrates the error of AP of a selected agent measured in standard deviations of AP versus t. As illustrated, as the number of agents increases, the error in performance increases for a given t.

Accordingly, a contact center routing operator may analyze various different metrics in selecting a suitable or tolerable t, and which may further be varied depending on the number of agents, expected available agents, distribution of performance, and so on. Further, other estimation and simulation techniques may be used to assist an operator in setting thresholds.

FIG. 8E illustrates an exemplary impact on the increase (or boost) of a performance based matching process from the Monte Carlo simulation data; in particular, the fraction of boost lost versus t. If one assumes that boost is proportional to the average agent performances of the selected agents, one can compare the true AP's of the agents selected with the AP's of those which would have been selected had the true AP's been known, and thus calculate a fractional decrease in the boost. With this assumption it follows:

$\begin{array}{cc}\text{Decrease of Boost}\propto \frac{\begin{array}{c}\text{Mean True AP of Agents}\\ \text{Selected from Dithered Data}\end{array}}{\begin{array}{c}\text{Mean True AP of Agents}\\ \text{Selected from True Data}\end{array}}& \left(14\right)\end{array}$

It is noted that FIG. 8E, and the decrease in boost model, is deficient in that it assumes the same level of noise or error for all agents, whereas in a real call center there will be a range of imprecision depending on the accumulated number of calls for each agent. Accordingly, the model and calculations could easily be improved upon by including either an empirical or theoretical distribution of total calls per agent.

Many of the techniques described here may be implemented in hardware or software, or a combination of the two. Preferably, the techniques are implemented in computer programs executing on programmable computers that each includes a processor, a storage medium readable by the processor (including volatile and nonvolatile memory and/or storage elements), and suitable input and output devices. Program code is applied to data entered using an input device to perform the functions described and to generate output information. The output information is applied to one or more output devices. Moreover, each program is preferably implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language.

Each such computer program is preferably stored on a storage medium or device (e.g., CD-ROM, hard disk or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the procedures described. The system also may be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner.

FIG. 9 illustrates a typical computing system 900 that may be employed to implement processing functionality in embodiments of the invention. Computing systems of this type may be used in clients and servers, for example. Those skilled in the relevant art will also recognize how to implement the invention using other computer systems or architectures. Computing system 900 may represent, for example, a desktop, laptop or notebook computer, hand-held computing device (PDA, cell phone, palmtop, etc.), mainframe, server, client, or any other type of special or general purpose computing device as may be desirable or appropriate for a given application or environment. Computing system 900 can include one or more processors, such as a processor 904. Processor 904 can be implemented using a general or special purpose processing engine such as, for example, a microprocessor, microcontroller or other control logic. In this example, processor 904 is connected to a bus 902 or other communication medium.

Computing system 900 can also include a main memory 908, such as random access memory (RAM) or other dynamic memory, for storing information and instructions to be executed by processor 904. Main memory 908 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 904. Computing system 900 may likewise include a read only memory (“ROM”) or other static storage device coupled to bus 902 for storing static information and instructions for processor 904.

The computing system 900 may also include information storage system 910, which may include, for example, a media drive 912 and a removable storage interface 920. The media drive 912 may include a drive or other mechanism to support fixed or removable storage media, such as a hard disk drive, a floppy disk drive, a magnetic tape drive, an optical disk drive, a CD or DVD drive (R or RW), or other removable or fixed media drive. Storage media 918 may include, for example, a hard disk, floppy disk, magnetic tape, optical disk, CD or DVD, or other fixed or removable medium that is read by and written to by media drive 912. As these examples illustrate, the storage media 918 may include a computer-readable storage medium having stored therein particular computer software or data.

In alternative embodiments, information storage system 910 may include other similar components for allowing computer programs or other instructions or data to be loaded into computing system 900. Such components may include, for example, a removable storage unit 922 and an interface 920, such as a program cartridge and cartridge interface, a removable memory (for example, a flash memory or other removable memory module) and memory slot, and other removable storage units 922 and interfaces 920 that allow software and data to be transferred from the removable storage unit 918 to computing system 900.

Computing system 900 can also include a communications interface 924. Communications interface 924 can be used to allow software and data to be transferred between computing system 900 and external devices. Examples of communications interface 924 can include a modem, a network interface (such as an Ethernet or other NIC card), a communications port (such as for example, a USB port), a PCMCIA slot and card, etc. Software and data transferred via communications interface 924 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communications interface 924. These signals are provided to communications interface 924 via a channel 928. This channel 928 may carry signals and may be implemented using a wireless medium, wire or cable, fiber optics, or other communications medium. Some examples of a channel include a phone line, a cellular phone link, an RF link, a network interface, a local or wide area network, and other communications channels.

In this document, the terms “computer program product,” “computer-readable medium” and the like may be used generally to refer to physical, tangible media such as, for example, memory 908, storage media 918, or storage unit 922. These and other forms of computer-readable media may be involved in storing one or more instructions for use by processor 904, to cause the processor to perform specified operations. Such instructions, generally referred to as “computer program code” (which may be grouped in the form of computer programs or other groupings), when executed, enable the computing system 900 to perform features or functions of embodiments of the present invention. Note that the code may directly cause the processor to perform specified operations, be compiled to do so, and/or be combined with other software, hardware, and/or firmware elements (e.g., libraries for performing standard functions) to do so.

In an embodiment where the elements are implemented using software, the software may be stored in a computer-readable medium and loaded into computing system 900 using, for example, removable storage media 918, drive 912 or communications interface 924. The control logic (in this example, software instructions or computer program code), when executed by the processor 904, causes the processor 904 to perform the functions of the invention as described herein.

It will be appreciated that, for clarity purposes, the above description has described embodiments of the invention with reference to different functional units and processors. However, it will be apparent that any suitable distribution of functionality between different functional units, processors or domains may be used without detracting from the invention. For example, functionality illustrated to be performed by separate processors or controllers may be performed by the same processor or controller. Hence, references to specific functional units are only to be seen as references to suitable means for providing the described functionality, rather than indicative of a strict logical or physical structure or organization.

The above-described embodiments of the present invention are merely meant to be illustrative and not limiting. Various changes and modifications may be made without departing from the invention in its broader aspects. The appended claims encompass such changes and modifications within the spirit and scope of the invention.

Claims

1. A computer implemented method for estimating agent performance in a call-center routing environment, the method comprising:

identifying, by one or more computers, one agent from a plurality of agents, wherein the one agent identified has a number of calls fewer than a predetermined number;

computing and assigning, by the one or more computers, a respective assigned outcome performance characteristic to the one identified agent, wherein the assigned outcome performance is different from an actual outcome performance characteristic for the identified agent, and is based at least in part on the performance of one or more other of the agents; and

routing, by the one or more computers, a caller to an agent of the plurality of agents based on respective outcome performance characteristics of the plurality of agents, the outcome performance characteristics including the assigned outcome performance characteristic to the one identified agent.

2. The method of claim 1, wherein the assigned outcome performance characteristic is an average outcome performance characteristic of the plurality of agents.

3. The method of claim 1, wherein the assigned outcome performance characteristic is based at least in part on an average outcome performance characteristic of the plurality of agents.

4. The method of claim 1, wherein the assigned outcome performance characteristic for the one identified agent is based at least in part on agent outcome performance data of one or more agents having similar demographic data as the respective agent.

5. The method of claim 1, wherein the computing and assigning step comprises computing, by the one or more computers, an adjustment to an actual outcome performance characteristic of the one identified agent based on one or more criteria.

6. The method of claim 1, wherein the computing step comprises determining an interpolation between an actual outcome performance characteristic value of the one identified agent and an outcome performance characteristic of the plurality of agents based at least in part on the criterion of a number of calls received by the one identified agent.

7. The method of claim 1, wherein the predetermined number of calls is determined relative to an average number of calls for the plurality of agents.

8. The method of claim 1, wherein the predetermined number of calls is associated with an error threshold in the outcome performance characteristic of the agents.

9. The method of claim 1, wherein the agent outcome performance characteristic comprises a sales rate.

10. The method of claim 1, wherein routing the caller is based on an outcome performance based matching algorithm.

11. The method of claim 1, wherein routing the caller is based on a pattern matching algorithm, and the assigned agent outcome performance characteristic is input into the pattern matching algorithm.

12. A computer implemented method for estimating agent outcome performance in a call-center routing environment, the method comprising:

determining, by one or more computers, respective actual outcome performance characteristics for each of a plurality of agents;

determining, by the one or more computers, an outcome performance characteristic representative of the plurality of agents;

identifying, by the one or more computers, a respective one of the agents having an error in the respective agent's outcome performance characteristic greater than a predetermined threshold;

assigning, by the one or more computers, an outcome performance characteristic for the identified agent which assigned outcome performance is different from the actual outcome performance characteristic for the one identified agent and which is based at least in part on the representative outcome performance characteristic for the plurality of agents; and

routing, by the one or more computers, a caller to one of the plurality of agents based on the outcome performance characteristics, the outcome performance characteristics including the assigned outcome performance characteristic to the identified agent.

13. The method of claim 12, wherein the error comprises a fractional error.

14. The method of claim 12, wherein the assigned outcome performance characteristic is an average outcome performance characteristic of the plurality of agents.

15. The method of claim 12, wherein the assigned outcome performance characteristic is based at least in part on an average outcome performance characteristic of the plurality of agents.

16. The method of claim 12, wherein the assigning step comprises computing, by the one or more computers, an adjustment to an actual outcome performance characteristic of the identified agent.

17. The method of claim 16, wherein the computing step comprises determining an interpolation between an actual outcome performance characteristic value of the respective identified agent and an average outcome performance characteristic of the plurality of agents.

18. The method of claim 12, wherein the predetermined threshold is based on a fractional error of the outcome performance characteristic.

19. The method of claim 12, wherein the agent outcome performance characteristic comprises a sales rate.

20. The method of claim 12, wherein routing the caller is based on an outcome performance based matching algorithm.

21. The method of claim 12, wherein routing the caller is based on a pattern matching algorithm, and the assigned agent outcome performance characteristic is input into the pattern matching algorithm.

22. A system for routing callers to agents in a call center routing environment, the system comprising:

one or more computers configured with computer-readable program code, that when executed, will cause performance of the steps:

identifying, by the one or more computers, one agent from a plurality of agents, wherein the one agent identified has a number of calls fewer than a predetermined number;

computing and assigning, by the one or more computers, a respective outcome performance characteristic to the one identified agent, wherein the assigned outcome performance is different from an actual outcome performance characteristic for the respective identified agent, and is based at least in part on the performance of one or more other of the agents; and

routing, by the one or more computers, a caller to an agent of the plurality of agents based on respective outcome performance characteristics of the plurality of agents, the outcome performance characteristics including the assigned outcome performance characteristic to the one identified agent.

23. A system for routing callers to agents in a call center routing environment, the system comprising:

one or more computers configured with computer-readable program code, that when executed, will cause performance of the steps:

determining, by the one or more computers, respective actual outcome performance characteristics for a plurality of agents;

determining, by the one or more computers, an outcome performance characteristic representative of the plurality of agents;

identifying, by the one or more computers, a respective one of the agents having an error in the respective agent's outcome performance characteristic greater than a predetermined threshold;

assigning, by the one or more computers, an outcome performance characteristic for the identified agent which assigned outcome performance is different from the actual outcome performance characteristic for the one identified agent and which is based at least in part on the representative outcome performance characteristic for the plurality of agents; and

routing, by the one or more computers, a caller to one of the plurality of agents based on the outcome performance characteristics, the outcome performance characteristics including the assigned outcome performance characteristic to the identified agent.

24. A non-transitory computer readable storage medium comprising computer readable instructions for carrying out, when executed by one or more computers, the method:

identifying, by the one or more computers, one agent from a plurality of agents, wherein the one agent identified has a number of calls fewer than a predetermined number;

computing and assigning, by the one or more computers, a respective assigned outcome performance characteristic to the one identified agent, wherein the assigned outcome performance is different from an actual outcome performance characteristic for the identified agent, and is based at least in part on the performance of one or more other of the agents; and

routing, by the one or more computers, a caller to an agent of the plurality of agents based on respective outcome performance characteristics of the plurality of agents, the outcome performance characteristics including the assigned outcome performance characteristic to the one identified agent.

25. A non-transitory computer readable storage medium comprising computer readable instructions for carrying out, when executed by one or more computers, the method:

determining, by the one or more computers, respective actual outcome performance characteristics for a plurality of agents;

determining, by the one or more computers, an outcome performance characteristic representative of the plurality of agents;

identifying, by the one or more computers, a respective one of the agents having an error in the respective agent's outcome performance characteristic greater than a predetermined threshold;

assigning, by the one or more computers, an outcome performance characteristic for the identified agent which assigned outcome performance is different from the actual outcome performance characteristic for the one identified agent and which is based at least in part on the representative outcome performance characteristic for the plurality of agents; and

routing, by the one or more computers, a caller to one of the plurality of agents based on the outcome performance characteristics, the outcome performance characteristics including the assigned outcome performance characteristic to the identified agent.

26. The method of claim 1, wherein the predetermined number of calls is determined based at least in part on a conversion rate for the plurality of the agents.

27. The method of claim 1, further comprising:

computing, by the one or more computers, a boost of performance based at least in part on a representative number for agent outcome performances of the agents that were selected and based at least in part on the representative number for the agent outcome performances of the agents that would have been selected based on their actual agent performances.

28. The system of claim 22, wherein the predetermined number of calls is determined based at least in part on a conversion rate for the plurality of the agents.

29. The system of claim 22, where the one or more computers are further configured with computer-readable program code, that when executed, will cause performance of the step:

computing, by the one or more computers, a boost of performance based at least in part on a representative number for agent outcome performances of the agents that were selected and based at least in part on the representative number for the agent outcome performances of the agents that would have been selected based on their actual agent performances.

30. The system of claim 22, wherein the assigned outcome performance characteristic for the one identified agent is based at least in part on agent outcome performance data of one or more agents having similar demographic data as the respective agent.

31. The system of claim 22, wherein the assigned outcome performance characteristic is based at least in part on an average outcome performance characteristic of the plurality of agents.

32. The system of claim 22, wherein the computing and assigning step comprises computing, by the one or more computers, an adjustment to an actual outcome performance characteristic of the one identified agent based on one or more criteria.

33. The system of claim 32, wherein the computing step comprises determining an interpolation between an actual outcome performance characteristic value of the one identified agent and an outcome performance characteristic of the plurality of agents based at least in part on the criterion of a number of calls received by the one identified agent.

34. A method for pairing contacts and agents in a contact center system comprising:

determining, by at least one computer processor configured to perform contact/agent pairing operations in the contact center system, a measured performance rating of a first agent of a plurality of agents;

assigning, by the at least one computer processor, an adjusted performance rating to the first agent when a calculated error in the measured performance rating exceeds a threshold, wherein the adjusted performance rating is different from the measured performance rating and is based at least in part on the performance of one or more other of the plurality of agents;

pairing, by the at least one computer processor, a contact with the first agent of the plurality of agents based on the adjusted performance rating of the first agent; and

establishing, by the at least one computer processor, a connection between the paired contact and first agent in the contact center system based on the adjusted performance rating of the first agent.

35. The method of claim 34, wherein the threshold was determined from a Monte Carlo simulation.

36. The method of claim 34, wherein the threshold was determined based on a hypothetical normal distribution of performance ratings across the plurality of agents.

37. The method of claim 34, wherein an expected increase to overall contact center performance by assigning the adjusted performance rating was estimated when the threshold was determined.

38. The method of claim 34, wherein hypothetical or measured agent performance ratings were dithered when the threshold was determined.

39. The method of claim 34, wherein a formula for assigning a baseline performance rating or an interpolation between the baseline performance rating and the measured performance rating as the adjusted performance rating was generated when the threshold was determined.

40. The method of claim 34, wherein, after pairing the contact with the first agent based on the adjusted performance rating of the first agent, a performance of the contact center system is determined to be higher than a performance of the contact center system based on pairing using measured performance ratings across the plurality of agents.

41. A system for pairing contacts and agents in a contact center system comprising:

at least one routing engine configured to perform contact/agent pairing operations in the contact center system, wherein the at least one routing engine is further configured to:

determine a measured performance rating of a first agent of a plurality of agents;

assign an adjusted performance rating to the first agent when a calculated error in the measured performance rating exceeds a threshold, wherein the adjusted performance rating is different from the measured performance rating and is based at least in part on the performance of one or more other of the plurality of agents;

pair a contact with the first agent of the plurality of agents based on the adjusted performance rating of the first agent; and

establish a connection between the paired contact and first agent in the contact center system based on the adjusted performance rating of the first agent.

42. The system of claim 41, wherein the threshold was determined from a Monte Carlo simulation.

43. The system of claim 41, wherein the threshold was determined based on a hypothetical normal distribution of performance ratings across the plurality of agents.

44. The system of claim 41, wherein an expected increase to overall contact center performance by assigning the adjusted performance rating was estimated when the threshold was determined.

45. The system of claim 41, wherein hypothetical or measured agent performance ratings were dithered when the threshold was determined.

46. The system of claim 41, wherein a formula for assigning a baseline performance rating or an interpolation between the baseline performance rating and the measured performance rating as the adjusted performance rating was generated when the threshold was determined.

47. The system of claim 41, wherein, after the contact is paired with the first agent based on the adjusted performance rating of the first agent, a performance of the contact center system is determined to be higher than a performance of the contact center system based on pairing using measured performance ratings across the plurality of agents.

48. An article of manufacture for pairing contacts and agents in a contact center system comprising:

a non-transitory computer processor readable medium; and

instructions stored on the medium;

wherein the instructions are configured to be readable from the medium by at least one computer processor configured to perform contact/agent pairing operations in at least one routing engine of the contact center system and thereby cause the at least one computer processor to operate so as to:

determine a measured performance rating of a first agent of a plurality of agents;

assign an adjusted performance rating to the first agent when a calculated error in the measured performance rating exceeds a threshold, wherein the adjusted performance rating is different from the measured performance rating and is based at least in part on the performance of one or more other of the plurality of agents;

pair a contact with the first agent of the plurality of agents based on the adjusted performance rating of the first agent; and

establish a connection between the paired contact and first agent in the contact center system based on the adjusted performance rating of the first agent.

49. The article of manufacture of claim 48, wherein the threshold was determined from a Monte Carlo simulation.

50. The article of manufacture of claim 48, wherein the threshold was determined based on a hypothetical normal distribution of performance ratings across the plurality of agents.

51. The article of manufacture of claim 48, wherein an expected increase to overall contact center performance by assigning the adjusted performance rating was estimated when the threshold was determined.

52. The article of manufacture of claim 48, wherein hyothetical or measured agent performance ratings were dithered when the threshold was determined.

53. The article of manufacture of claim 48, wherein a formula for assigning a baseline performance rating or an interpolation between the baseline performance rating and the measured performance rating as the adjusted performance rating was generated when the threshold was determined.