What is Data Anonymization?

Written by
CloverDX
CloverDX

Data anonymization is a process of masking data so that individual people or records can’t be identified. It enables life-like data to be used for software testing, analytics, visualization or sharing with third parties, but with any sensitive data safely obscured. 

Unlike some other methods of creating non-identifiable data, data anonymization resembles your original dataset as closely as possible, keeping some important characteristics and relationships that can make a big difference to your analysis or testing. 

Why would you want to use data anonymization?

In an ideal world, you’d test on your production data to give you the most accurate test results. But there are many reasons you can’t do that:

  • You need to comply with data protection, data privacy and compliance regulations. If your data contains personal information, you have a duty to protect it.
  • You need to get value from your data, but your data contains corporate information that you don’t want to share.
  • You need to protect your system resources; testing on live systems or production data can have an impact on performance.
  • Licensing restrictions: some software requires a full licence while working with production data, even in a development setting.
  • You need more data than you have available. For instance, machine learning or AI tools need large amounts of life-like data to create realistic models.

There are various ways of creating fake data that avoid some of these issues, as we look at below, but all have their pros and cons. Essentially, for the most accurate testing you want to use data that’s fake, but still captures real-world situations.

webinar - data anonymization for better software testing - watch now

Different methods of data masking

Data can be made un-identifiable in various ways, some easier than others, and some more useful than others.

Randomized data

Randomizing data is relatively easy, but provides limited value. As you can see in the example below, your data may retain some formatting rules, but otherwise is pretty much gibberish. This means that if you want to test your software, you don’t have a very realistic dataset (so your tests aren’t going to catch many potential bugs). It also means that you can’t perform any meaningful analysis on your data.

  BEFORE (actual data) AFTER (randomized data)
Name Frank Smith Xxuzyg Mbdhu
Social Security Number 543-69-1573 888-88-8888
City Denver Xyzzz
Date of Birth 24 Jul 1975 1 Jan 2000

Synthetic data 

Synthetic data - generating artificial data that resembles your original dataset but contains completely fake information - is a step above randomized data.  It’s more elaborate, generates valid values (so better for testing and analysis) but still has some limitations.

  BEFORE (actual data) AFTER (synthetic data)
Name Frank Smith John Doe
Social Security Number 543-69-1573 123-45-6789
City Denver Chicago
Date of Birth 24 Jul 1975 8 Feb 2014

Limitations of synthetic data

  • It’s only as good as the underlying datasets or models used to create it. For example:
    • If you use English characters only, you won’t get any non-English names generated (which could be important if you’re trying to test software that’s used by a large number of people).
  • It doesn’t keep relationship or identification information that you might want, e.g.:
    • US Social Security Numbers (SSNs) contain numbers that identify a state. You want to make sure you’re not using “real” SSNs that can be linked to a person, but you might want to keep information on how many SSNs from each state you have. Generating completely synthetic SSNs means you lose this.
  • It won’t always give you a realistic distribution of values (one that matches your original data)
    • Synthesized data might distribute all values equally across a scale - whereas your original data might not have this same distribution.
  • It won’t contain the same errors as your original data
    • Errors are difficult to synthesize, but are something that will undoubtedly exist in your production data, so you have to make sure you test for those conditions.
Data Anonymization with CloverDX

Synthetic data vs data anonymization

Unlike synthesized data, data anonymization does preserve some attributes of your original dataset.

Anonymization can for example change 'Frank from Denver' into 'John from Denver'. No longer a real person, but your data still keeps accurate information on the number of people in Denver (although you do of course lose information on the number of Franks. It’s important to decide which information is important for you to keep in your particular case).

Rather than creating completely fake data, data anonymization masks your existing dataset. There are several different methods of anonymizing your data, including changing certain values to remove identifying information, shuffling data around or altering values slightly.

The example below shows how some characteristics of the original production data remain, but it's no longer possible to identify individual records or tie sensitive information to a particular person.

BEFORE (actual data) AFTER (anonymized data)
Name Frank Smith       Frank Smith
SSN 543-69-1573 235-41-8875   543-67-0008 235-81-9568
City Denver New York   Delaware Minneapolis
Birth 24 Jul 1975 14 Sep 1957   28 Jul 1975 17 Sep 1957


Advantages of data anonymization over synthetic data

Anonymized data:

  • Is more closely related to your original data
  • Can inject impurities into your data, making it more life-like
  • Keeps some useful information on relationships and statistical distribution
  • Can be used in end-to-end system testing, because it keeps relationships between systems
  • Can be used for AI or machine learning, because it doesn’t skew reality
  • Makes your testing better, so reduces the chance of embarrassing bugs being discovered after go-live
White Paper: Conquering the Challenges of Data Anonymization

Learn more about data anonymization

Our webinar Data Anonymization for Better Software Testing explores how data anonymization can help you get better test data and improve the quality of your software releases. We'll go into detail about data anonymization; the different methods of achieving it; and the pros and cons of each approach, as well as taking a look at how CloverDX can help manage the data anonymization process at enterprise scale.

Register for the webinar now > 

Posted on August 05, 2019
DISCOVER  CloverDX Data Integration Platform  Design, automate and operate data jobs at scale. Learn more <>

Related Content

Subscribe to our blog


Where to go next