Merge/Purge for Direct Mail – The Removal of Duplicate Records
Posted by Greg Kathol on Thu, Apr 22, 2010
The merge/purge process is defined as the removal of duplicate records from a given data set or in this case, a mailing list. There are several reasons to perform this task. Many direct mail lists, whether a house list or in the form of multiple purchased lists and/or a combination of both, can contain duplicated information. The merge/purge process will eliminate the unwanted records based on certain criteria you select.
Criteria for de-duping a mailing list may include: one record per address, one per household, one per name per address or other fields like a keycode or unique ID. One record per address means that a list will be processed to remove all duplicate addresses. Typically a duplicate address represents the entire address. For example: 1234 Maple St., Anytown, CA 90000. Any records matching that will be removed so only one record with that address remains in the list. If there is an apartment or suite number associated with the address then the duplicate record will only be removed if there is an exact match with the apartment or suite number at the address. This will ensure that apartment complexes, condos, townhomes and business parks don't get unnecessarily removed. Note: a purchased mailing list may not include apartment or suite numbers. You may find that your list contains quite a few families at one address that will be removed using this technique.
This next match type is one record per household. This process removes all records that have a last name matching. For example, a record with the last name of Smith at one address will be removed if there is an exact match with another address and name in the data. An example of this would be an individual named Joe Smith and in individual named John Smith living at the same address. One of those records would be removed by utilizing this record removal process. The assumption here is that both people are member of the same household.
The removal of one record per name per address is the process of eliminating all records that have an exact address match and an exact name match. Note: Joe Smith 1234 Maple St., Anytown, CA 90000 does not match Jane Smith 1234 maple St., Anytown, CA 90000 since the full name is different. You can however ask that the records be de-duped only by last name per address. That way only one mail piece will arrive per name. If there is more then one name (last or full) at that address then each person with a different last name will receive a mail piece. An example of this could be two families or people with different last names living at the same address. It's also important to note that there is no distinction made between a name such as Robert or Bob. This name will be treated as two different records. However, there are more sophisticated data cleansing methods for recognizing, matching and clearing common name versions.
The last category is perhaps the easiest to remove duplicates from. Removing records by keycode or unique ID such as a customer number, account number or other assigned unique identifier can simply be done by matching those fields and eliminating one of the matches. However, this will remove whatever is in that field, no matter what. The only criteria here is the unique identifier. If you don't feel the integrity of the data is high enough to do this, then run one of the other merge purge processes first, then perform the unique file de-dup.
Most of these merge/purge or duplication removals are performed by data processing experts who are familiar with direct mail and what is needed to ensure the highest level of data integrity. Many of these processes can get quit sophisticated on the back end, but with a simple working knowledge of how each process works, you will be able to talk to your mail house or data expert about your particular needs and let them do hard work for you.
Hopefully this gives you a clearer understanding of the merge/purge process and what it really means to remove duplicates. If you have any questions or comments, feel free to leave them below.
By Greg Kathol