- Notifications
You must be signed in to change notification settings - Fork55
Universal tool to anonymize database. GDPR (General Data Protection Regulation) data protection act supporting tool.
DivanteLtd/anonymizer
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Anonymizer is a universal tool to create anonymized DBs for projects.
Why is it so important to work with anonymized databases? In the development process, you should never use your production database - it is risky and against the law. Working with a production database in development is risky, because by some mistake, you can might make your clients data available to the whole world. In our world, general data is one of most valuable things and EU tries to protect citizens' general data via the General Data Protection Regulation (GDPR).
Some more quick wins for GDPR? Take a look at ourrecent blog post.
Anonymizer replaces all data in your database by anonymized random data. The most important feature of Anonymizer is data formatting. All generated data is similar to the original data used by real users. The example below shows anonymized data in a Magento 1 sales_flat_quote_address table.
- ruby >= 2.2
- mysql >= 5.6
- Magento 1.9.x / 2.x
- Pimcore 4 / 5
- Sylius 1.0.0-beta.2
Of course you can anonymize any other database - this is just an example.
Clone this repository and add a configuration file for your first project to theconfig/project/ directory.
The file name reflects the anonymized database's name. For example, if you need to anonymize a db dump namedexample.sql.gz, the configuration file should be namedexample.json.
Project configuration files always have anextended type. In the project configuration file, you have to set thetype andbasic_type which is connected with the supported framework. Only framework configuration files use thebasictype.
custommagento_1_9magento_2_0pimcore_4pimcore_5sylius
The anonymized dump is given the same name as the original database dump file. If you want to change this name, you can set the keyrandom_string in configuration file - the value of this key will be added to end of the filename. In the example below, the output file will be namedexample_ba74a64a152b84ec004d03caba15ba95.sql.gz.
{ "type": "extended", "basic_type": "magento_1_9", "random_string": "ba74a64a152b84ec004d03caba15ba95",Anonymizer can work with both locally and remotely stored database dumps. Database dumps from remote servers are downloaded by rsync.
In the example below, the database dump file is in the/path/to/database/dump/ directory.
"dump_server": { "host": "", "port": "", "user": "", "path": "/path/to/database/dump/"}In the example below, the database dump file is stored on a remote server with an IP address of1.2.3.4 and ssh port of5022. The ssh user's name isanonymizer, the directory on remote host with the database dump is/path/to/database/dump/. In this case, let's assume that we need to add--rsync-path=\"sudo rsync\" option to our rsync dump download command.
"dump_server": { "host": "10.15.4.254", "user": "anonymizer", "port": "5022", "passphrase": "", "path": "/media/drbd0/backup/sqldump/sqldump", "rsync_options": "--rsync-path=\"sudo rsync\""}Anonymizer can replace the original data by anonymized entries or truncate the data in the destination table.
- firstname
- lastname
- login
- telephone
- company
- street
- postcode
- city
- full_address
- vat_id
- ip
- quote
- website
- iban
- json
- uniq_email - unique email address, because ofbug in mysql, anonymizer can't generate uniq email address based on email and some uniqeu value form UUID mysql methiod
- uniq_login - unique login
- regon - Polish REGON number with validation
- pesel - Polish PESEL number with validation
In the example below, data in theuser_address table will be replaced by new, anonymized data. The example database contains auser_address table with the following columns -firstname,lastname,postcode,address,city,email,phone,company,vat_id. We will replace all columns' contents with some valid data, consistent with its previous type.
"tables": { "user_address": { "firstname": { "type": "firstname", "action": "update" }, "lastname": { "type": "lastname", "action": "update" }, "postcode": { "type": "postcode", "action": "update" }, "address": { "type": "street", "action": "update" }, "city": { "type": "city", "action": "update" }, "email": { "type": "email", "action": "update" }, "phone": { "type": "telephone", "action": "update" }, "company": { "type": "company", "action": "update" }, "vat_id": { "type": "vat_id", "action": "update" } }}In the example below, the data in thelog_customer table will be truncated.
"tables": { "log_customer": { "only_truncate": { "action": "truncate" } }}In the example below we will empty data in column with configuration values, keeping other columns intact.
"tables": { "some_configuration_table": { "config_value": { "action": "empty" } }}In below example valuePLN will be assigned to columnbase_currency for all users.
"tables": { "users": { "base_currency": { "action": "set_static", "value": "PLN" } }}Anonymizer can also work with Magento's EAV model. In the example below, the customer attributeabout_me in thecustomer_entity_text table will be replaced with a random phrase.
"tables": { "customer_entity_text": { "value": { "action": "eav_update", "attributes": [ { "code": "about_me", "type": "quote", "entity_type": "customer" } ] } }}Anonymizer can now update values of JSON encoded data. In below example we anonymize json stored inadditional_data column.You should familiarize with MySQL JSON path expressions.
{ "id": 123, "user": { "first_name": "John", "last_name": "Smith", "phone": "123-456-789" }, (...) }"tables":{ "subscriptions":{ "additional_data":{ "action":"json_update", "fields":[ { "path":"$.user.first_name", "type":"firstname" }, { "path":"$.user.last_name", "type":"lastname" }, { "path":"$.user.phone", "type":"telephone" } ] } }}Anonymizer can run custom, row queries before and after anonymization process. In the example below, the anonymizer runs two queries before and one after.
"tables": {},"custom_queries": { "before": [ "DELETE FROM some_column WHERE date > '2019-12-25'", "INSERT INTO some_column2 SET table = 'value'" ], "after": [ "INSERT INTO some_column SET name = 'admin', pass = '1234567890'", ]}{ "type": "extended", "basic_type": "magento_1_9", "random_string": "ba74a64a152b84ec004d03caba15ba95", "dump_server": { "host": "10.15.4.254", "user": "anonymizer", "port": "5022", "passphrase": "", "path": "/media/drbd0/backup/sqldump/sqldump", "rsync_options": "--rsync-path=\"sudo rsync\"" } "tables": { "user_address": { "firstname": { "type": "firstname", "action": "update" }, ... }, "log_customer": { "only_truncate": { "action": "truncate" } }, "customer_entity_text": { "value": { "action": "eav_update", "attributes": [ { "code": "about_me", "type": "quote", "entity_type": "customer" } ] } }, "subscriptions":{ "additional_data":{ "action":"json_update", "fields":[ { "path":"$.user.first_name", "type":"firstname" }, { "path":"$.user.last_name", "type":"lastname" }, { "path":"$.user.phone", "type":"telephone" } ] }, "comment": { "type": "quote", "action": "update" } } }Before you run anonymizer you should add configuration file. Copy sample config file fromconfig/env/sample.yml toconfig/env/<env_name>.yml
RUBY_ENV=<env_name> bundle exec rake project:anonymize[example]Run development docker environment using the command below:
docker-compose -f dev/docker/docker-compose.dev.yml upOn docker environment run the commands:
bundle installbundle exec rspec spec/If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are warmly welcome.
The code featured in this project is licensed under MIT license.
We are a Software House from Europe, existing from 2008 and employing about 150 people. Our core competencies are built around Magento, Pimcore and bespoke software projects (we love Symfony3, Node.js, Angular, React, Vue.js). We specialize in sophisticated integration projects trying to connect hardcore IT with good product design and UX.
We work for Clients like INTERSPORT, ING, Odlo, Onderdelenwinkel or CDP, the company that produced The Witcher game. We develop two projects:Open Loyalty - loyalty program in open source andVue.js Storefront.
We are part of the OEX Group which is listed on the Warsaw Stock Exchange. Our annual revenue has been growing at a minimum of about 30% year on year.
Visit our websiteDivante.co for more information.