The present invention in various implementations provides for a method and system for removing suspect duplicate data in a database having a plurality of datasets for a suspect processing transaction, using a plurality of matching engines, comparing results of matching engines in a logically predetermined comparative assessment, and thereafter providing an ordered priority of results of matching engines to identify suspect candidates.
|
1. A method for removing suspect duplicate data in a database having a plurality of datasets with predefined searchable informational characteristics for a suspect processing transaction, comprising:
selecting a plurality of matching engines, each matching engine for creating information for a suspect table of the respective matching engine and determining weights and scores of the respective matching engine, wherein the plurality of matching engines comprises a default matching engine (DME) for determining weights and scores for DME assessments and storing suspect information to a suspect table, and one or more predetermined auxiliary matching engines (AMEs) for determining weights and scores and updating the suspect information by storing records in a suspect Augment table,
gathering associated records from the datasets for analysis,
defining one or more external logics having logic criteria for:
(i) relating results of the matching engines for comparative assessment in a predetermined manner and
(ii) providing ordered priority of matching engine results to identify suspect candidates, wherein the one or more external logics comprise an external logic component (ELC), having predefined logic criteria for matching which matching engine result of one or more of the one or more AMEs will be comparatively assessed with matching engine results of the DME determined weights and scores, and an external default logic (EDC) with rules execution for providing priority of matching engine results assessed by the matching engines to identify the result,
creating select information characteristics in the respective suspect tables,
determining the respective weights and scores from the matching engines in accordance with the predetermined manner and the logic criteria, and
producing a result indicating a presence or absence of one or more suspect candidates in relation to the plurality of datasets.
13. A computer program product for identifying and ranking, in one pass, suspect duplicate data in a database having a plurality of datasets with predefined searchable informational characteristics for a suspect processing transaction, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising: a first executable portion for:
selecting a plurality of matching engines, each matching engine for creating information for a suspect table of the respective matching engine and determining weights and scores of the respective matching engine, wherein the plurality of matching engines comprises a default matching engine (DME) for determining weights and scores for DME assessments and storing suspect information to a suspect table, and one or more predetermined auxiliary matching engines (AMEs) for determining weights and scores and updating the suspect information by storing records in a suspect Augment table,
gathering associated records from the datasets for analysis,
defining one or more external logics having logic criteria for:
(i) relating results of the matching engines for comparative assessment in a predetermined manner and
(ii) providing ordered priority of matching engine results to identify suspect candidates, wherein the one or more external logics comprise an external logic component (ELC), having predefined logic criteria for matching which matching engine results of one or more of the one or more AMEs will be comparatively assessed with matching engine results of the DME determined weights and scores, and an external default logic (EDC) with rules execution for providing priority of matching engine results assessed by the matching engines to identify the result, creating select information in the respective suspect tables, determining the respective weights and scores from the matching engines in accordance with the predetermined manner and the logic criteria, and producing a result having a presence or absence of a suspect candidate in relation to the plurality of datasets.
8. An ordered system for comparatively assessing in a predefined manner, in a single pass, predefined characteristics of individual results of a plurality of matching engines in relation to a plurality of customer data in one or more databases for an invoked transactional process, comprising:
identifying a plurality of records from one or more datasets of client data having possible duplicate data therein,
selecting a first matching engine and a second matching engine, wherein each matching engine is for creating information for an information table of the respective matching engine and determining weights and scores of the respective matching engine, wherein the first matching engine comprises a default matching engine (DME) for determining weights and scores for DME assessments and storing suspect information to an information table of the DME, and the second matching engine comprises one or more predetermined auxiliary matching engines (AMEs) for determining weights and scores and updating the suspect information by storing records in one or more information tables of the AMEs,
defining a plurality of external logics having a plurality of logic criteria for:
(i) relating results of the matching engines for comparative assessment in a predetermined manner and
(ii) providing ordered priority of matching engine results to identify suspect candidates, wherein the plurality of external logics comprise an external logic component (ELC), having predefined logic criteria for matching which matching engine result of one or more of the one or more AMEs will be comparatively assessed with matching engine results of the DME determined weights and scores, and an external default logic (EDC) with rules execution for providing priority of matching engine results assessed by the matching engines to identify the result,
populating a first information table from a first matching engine with select characteristics,
creating a second information table from a second matching engine with select characteristics,
determining the respective weights and scores from the information tables of the respective matching engines in accordance with the predetermined manner and the logic criteria, and
producing a result identifying one or more records of the dataset for removal.
2. The method of
3. The method of
5. The method of
6. The method of
9. The system of
10. The system of
11. The system of
12. The system of
14. The product of
|
The present invention relates generally to transactional searching and more specifically to a method and system for comparing predefined data characteristics in a predetermined manner.
Customer records and varied information that are often a part of complex data systems and business applications present a variety of challenges in management, identification, analysis and segregation, for example. It is generally believed, based on estimates in certain industries, that the amount of unstructured data that may reside in planned and developing databases could easily exceed eighty percent. With such an amount of unstructured data, additional complexities arise, especially when the data is present on different platforms, is of varying freshness, and may be inconsistent across the platforms.
Determining which records are duplicates of one another, for instance, where within a complex data system there may exist multiple databases, each having records comprising particular data within, can be a particularly difficult issue. Quality issues also arise in which data stewardship matters become a central concern. Further, in matters concerning customer relationship management (CRM) which often necessitate data integrity to realize optimal returns on data structure investments, removing “bad customer data” and especially duplicate customer data is of key concern.
The IBM Websphere® Customer Center (WCC) is a real-time, service-oriented customer application that provides users with a single view of the customer and also with business processes which provide for maintaining customer data shared between a front and back office arrangement. The WCC effectively acts as an intelligent customer data hub to manage customer data through its Customer Master Data Management (CMDH) hub. Via the CMDH, the WCC manages business rules, event detection, data validation and duplicate suspect processing (DSP). While other vendors have attempted to merely singly match search results to transactional records as a solution to searching in response to the issues raised previously, the WCC via its DSP, enables a client to persist and process duplicate suspects of any customer (i.e., customer data) in the system.
In operation, the DSP searches for potential suspect candidates for a given customer (or party as used herein) and then provides the suspect candidate list to a matching engine which then scores values for each of the candidates residing on the list of suspect candidates. The DSP creates a suspect table (i.e., SuspectTable) as part of the process. The values determined by the DSP in this process are then assessed to determine similarity or dissimilarity as between candidate suspects.
A challenge in the process of assessment of candidate suspects, is the final decision to remove customer data that only appears to be “similar” to that of other data in the system. In particular, the deletion of one set of customer data based on the similar presence of a second but different set of other customer data also present, in relation to a determinative evaluation by a single matching engine and its respective scoring scheme, without a significant confidence of accuracy, is often disconcerting to business management. Typically then, for business management to gain additional confidence in the initial determination of results to make a final decision, further analysis is often conducted and additional searching, comparison, scoring and assessments are performed over time in multiples “passes” on data which is then both different in content and time-value to the original data analyzed.
Unfortunately, as a result, business management may often have multiple results and suspects created for data sets which are neither static nor original in content when compared to the initial set of data, thereby creating further hesitation and confusion in making a final decision. Operatively, data users typically direct findings determined from the traditional process above for further integration steps and processes, thereby creating multiple reviews and assessment passes. Certain of these additional integration activities may involve further matching engine investigation or analysis tool, undertaking data transformation steps, and using Extract, Transform and Load (ETL) tools, for instance.
It is therefore desirable to have an improved method for determining and identifying, in one pass and with heightened confidence, suspect candidates from customer data in relation to results of a plurality of predetermined matching engines via comparative assessments at a predefined time. The present invention addresses such a need.
The present invention in accordance with various implementations, provides for a method for removing suspect duplicate data in a database having a plurality of datasets with predefined searchable informational characteristics for a suspect processing transaction, comprising: selecting a plurality of matching engines, each matching engine capable of creating information for its respective suspect tables and determining its respective weights and scores; gathering associated records from said datasets for analysis; defining one or more external logics having logic criteria for: relating results of said matching engines for comparative assessment in a predetermined manner and providing ordered priority of results of matching engines to identify suspect candidates; creating said information in said respective suspect tables; determining said respective weights and scores from said matching engines in accordance with said predetermined manner and said logic criteria; and producing a result having a presence or absence of suspect candidates in relation to said plurality of datasets.
The present invention in accordance with various implementations is a computer program product for identifying and ranking in one pass suspect duplicate data in a database having a plurality of datasets with predefined searchable informational characteristics for a suspect processing transaction, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising: a first executable portion capable of selecting a plurality of matching engines, each matching engine capable of creating information for its respective suspect tables and determining its respective weights and scores; gathering associated records from said datasets for analysis; defining one or more external logics having logic criteria for: relating results of said matching engines for comparative assessment in a predetermined manner and providing ordered priority of results of matching engines to identify suspect candidates; creating said information in said respective suspect tables; determining said respective weights and scores from said matching engines in accordance with said predetermined manner and said logic criteria; and producing a result having a presence or absence of a suspect candidate in relation to said plurality of datasets.
The present invention relates generally to transactional searching and more specifically to a method and system for comparing predefined data characteristics in a predetermined manner using a plurality of predetermined matching engines.
The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiments and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.
The present invention in various implementations, provides for an improved method for determining and identifying, with heightened confidence, suspect candidates from customer data in relation to a plurality of predetermined matching engines for comparatively assessing in a predefined manner predefined characteristics of individual results of said matching engines in relation to a plurality of customer data.
In
From the implementation depicted in
Where it is determined that one or more AMEs will be used to assess the DME's determined weights and scores at 131, the selected or matched one or more AMEs will be invoked in a manner of one of either real-time or near real-time at 132. The DME will persist the suspect information at 138 to the suspect table 140 and the AMEs will update the suspect information at 139 by persisting records in the Suspect Augment table at 150. At 150, data for a Suspect Augmentation table is created for the suspect candidates of the invoked suspects of 130.
Further from the implementation depicted in
In one particular implementation the EDC is predefined to be of the following ordered arrangement:
If present, matching engine 1 matching result
where in the above logic flow, a matching engine may be a commercial matching engine such as WCC, a customized or unique solution, or similar.
In another implementation, Weight, Matching Engine and Suspect Category are added to a SuspectTable to maintain the current status of a suspect record. A new table, SuspectAugment, is also introduced to maintain the result of all match operations that have been executed on a single suspect record. For a single suspect, the SuspectAugment table can hold multiple records to maintain the Matching Engine, Weight, Suspect Category and Augmentation Reason resulting from different matching engines.
In a further implementation of the present invention, by example and not of limitation, the information on the strength of similarity and dissimilarity between the customer data and its suspect candidates (“comparative information”) are maintained with the following entities in the context of suspect: matching score, non-matching score, matching engine weight, matching weight. Other information relations are envisioned to form comparative information as well, in view of a data user's needs in particular. Further, the result of invoking one or more different matching engines in relation to the same suspect record is also maintained in the separate table, SuspectAugment.
A data user, in a further implementation of the present invention, may implement a getSuspect transaction to view the matching results from the DME and the developed or resulting augmentations of the AMEs.
Operationally, by example then, in an addParty transaction where there are three matching engines (one DME and two AMEs), a DME is first selected, a commercial matching engine is enabled via an external rule with a synchronous mode of operation, and a second commercial matching engine is enabled via an external rule with an asynchronous mode of operation. One of the commercial matching engines may be that of a WCC customized matching engine for instance, where such may a DME as well.
In the implementation of
From
The inclusion of an AME type or of a specific named matching engine as an AME is limited only to those matching engines which are acceptable and interoperable with a matching engine interface of a system of the present invention. It is envisioned that the interface would be inclusive of a designated party matcher styled interface which would accept current (e.g., Abilitec, QualtyStage, etc.) and future developed matching engines.
The present invention and its various implementations overcomes the limitations and inefficiencies in the field as the present invention, in one implementation or another: i) improves the overall integrity of individual data within one or more datasets; ii) improves the data integrity across a data system at predefined time independent of individual data platform; iii) provides the ability for a data user to obtain a plurality of results from predetermined matching engines in more than one manner; iv) provides for a deterministic or a probabilistic approach for interpretation and assessment of one or more results obtained; v) provides for the ability to invoke any predetermined matching engine in either of real-time or near real-time operational mode, and vi) provides improved confidence to a data user that a final decision based on the results of the improved method are suitable for certain customer data deletion, merger, or other modification.
For performance consideration, data users can also customize the application in order to utilize the default deterministic matching engine as the real-time operation and then issue an event to asynchronously augment this result with a probabilistic matching engine.
As used herein the term “matching engine” is intended to be broad in scope and may be an algorithm, software, process, method, rule-based set, or similar, which may also determine scoring directly or indirectly for and in relation to certain predetermined data characteristics.
Although the present invention has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and those variations would be within the spirit and scope of the present invention. Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the appended claims.
In addition to the processes and implementations of the present invention described thus far, the invention may also be used for data systems, archived data retrieval systems, database operations, real-time data collection systems, near real-time data acquisition systems, and other types of systems and devices therein containing the same or similar applications and uses.
Khatibi, Mohammad, Chouinard, Karen Dixon, Wadhwa, Sachin
Patent | Priority | Assignee | Title |
8306964, | Jul 20 2010 | Microsoft Technology Licensing, LLC | Extraction of rich search information from index servers via an alternative asynchronous data path |
8577833, | Jan 04 2012 | International Business Machines Corporation | Automated data analysis and transformation |
8768880, | Jan 04 2012 | International Business Machines Corporation | Automated data analysis and transformation |
8805777, | Apr 13 2012 | International Business Machines Corporation | Data record collapse and split functionality |
Patent | Priority | Assignee | Title |
6327597, | Aug 06 1999 | International Business Machines Corporation | Method, memory system, and computer program product for implementing lock based delayed space recycle control |
6728704, | Aug 27 2001 | VALTRUS INNOVATIONS LIMITED | Method and apparatus for merging result lists from multiple search engines |
7376697, | Nov 12 2001 | Malikie Innovations Limited | Arrangement of data synchronization in a telecommunications system |
20050091174, | |||
20060155739, | |||
20060200455, | |||
20070271228, | |||
20080201304, | |||
EP135270, | |||
EP1158421, | |||
WO171562, | |||
WO2005020106, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Aug 29 2007 | CHOUINARD, KAREN DIXON | International Business Machines Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 019774 | /0227 | |
Aug 29 2007 | KHATIBI, MOHAMMAD | International Business Machines Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 019774 | /0227 | |
Aug 29 2007 | WADHWA, SACHIN | International Business Machines Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 019774 | /0227 | |
Aug 31 2007 | International Business Machines Corporation | (assignment on the face of the patent) | / | |||
Jun 29 2012 | International Business Machines Corporation | SAP AG | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028540 | /0522 | |
Jul 07 2014 | SAP AG | SAP SE | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 033625 | /0334 |
Date | Maintenance Fee Events |
Oct 24 2013 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Nov 01 2017 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Nov 04 2021 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
May 11 2013 | 4 years fee payment window open |
Nov 11 2013 | 6 months grace period start (w surcharge) |
May 11 2014 | patent expiry (for year 4) |
May 11 2016 | 2 years to revive unintentionally abandoned end. (for year 4) |
May 11 2017 | 8 years fee payment window open |
Nov 11 2017 | 6 months grace period start (w surcharge) |
May 11 2018 | patent expiry (for year 8) |
May 11 2020 | 2 years to revive unintentionally abandoned end. (for year 8) |
May 11 2021 | 12 years fee payment window open |
Nov 11 2021 | 6 months grace period start (w surcharge) |
May 11 2022 | patent expiry (for year 12) |
May 11 2024 | 2 years to revive unintentionally abandoned end. (for year 12) |