Repository files navigation

We found ourselves repeating codefor commonly used fields on each newPhoenix project/App ...
We wanted amuch easier/faster way of building appsso we created a collection of pre-defined fieldswith built-in validation, sanitising and security.Fields makes definingEcto Schemas fasterand more precise.

An Elixir package that helps you add popular custom typesto your Phoenix/Ecto schemas so you can build apps faster!

@dwyl we are firm believers that personal data(Personally Identifiable Information (PII)) should be encrypted "at rest"i.e. all "user" data should be encryptedbefore being stored in the database.This project makes hashing, encryption anddecryption for secure data storagemuch easier for everyone.

This package was born out of our researchinto the best/easiest way to encrypt data inPhoenix:dwyl/phoenix-ecto-encryption-example

This module is for people building Elixir/Phoenix appswho want to shipsimpler and more maintainable code.

We've attempted to makeFieldsasbeginner-friendly as possible.
If you get stuck using it or anything is unclear, please ask forhelp!

Start usingFields in your Phoenix App today with these 3 easy steps:

Add thefields package to your list of dependencies in yourmix.exs file:

defdepsdo[{:fields,"~> 2.10.3"}]end

Once you have saved themix.exs file,runmix deps.get in your terminal to download.

In order to use Encryption and Hashing,you will need to have environment variablesdefined forENCRYPTION_KEYS andSECRET_KEY_BASE respectively.

export ENCRYPTION_KEYS=nMdayQpR0aoasLaq1g94FLba+A+wB44JLko47sVQXMg=export SECRET_KEY_BASE=GLH2S6EU0eZt+GSEmb5wEtonWO847hsQ9fck0APr4VgXEdp9EKfni2WO61z0DMOF

If you need to create a secureSECRET_KEY_BASE value, please see:How to create Phoenixsecret_key_base
And forENCRYPTION_KEYS, see:How to create encryption keys

In our case we use a.env fileto manage our environment variables.See:github.com/dwyl/learn-environment-variables
This allows us to securely manage our secret keys in devwithout the risk of accidentally publishing them on Github.
When wedeploy our Apps, we use our service provider'sbuilt-in key management service to securely store Environment Variables.e.g:Environment Variables on Heroku

Each field can be used in place of an Ecto type when defining your schema.

An example for defining a "user" schema usingFields:

schema"users"dofield:first_name,Fields.Name# Length validated and encryptedfield:email,Fields.EmailEncrypted# Validates email then encryptsfield:address,Fields.AddressEncrypted# Trims address string then encryptsfield:postcode,Fields.PostcodeEncrypted# Validates postcode then encryptsfield:password,Fields.Password# Hash password with argon2 industry standardtimestamps()end

Each field is defined as anEcto type,with the relevant callbacks.So when you callEcto.Changeset.cast/4in your schema's changeset function,the field will be correctly validated.For example, calling cast on the:email fieldwill ensure it is a valid format for an email addressRFC 5322.

When you load one of the fields into your database,the correspondingdump/1 callback will be called,ensuring it is inserted into the database in the correct format.In the case ofFields.EmailEncrypted,it will encrypt the email addressusing a given encryption keybefore inserting it.

Likewise, when you load a field from the database,theload/1 callback will be called,giving you the data in the format you need.Fields.EmailEncrypted will be decrypted back to plaintext.This all happens 100% transparently to the developer.It'slike magic. But the kind where you canactuallyunderstand how it works!(if you're curious, read thecode)

Each Field optionally defines aninput_type/0 function.This will return an atomrepresenting thePhoenix.HTML.Form input type to use for the Field.For example:Fields.DescriptionPlaintextUnlimited.input_typereturns:textarea which helps us render the correct field in a form.

The fieldsDescriptionPlaintextUnlimitedandHtmlBody useshtml_sanitize_exto remove scripts and help keep your project safe.HtmlBody is able to display basic html elementswhilstDescriptionPlaintextUnlimited displays text.Remember to useraw when renderingthe content of yourDescriptionPlaintextUnlimitedandHtmlBody fieldsso that symbols such as & (ampersand) and Html are rendered correctly.e.g:<p><%= raw @product.description %></p>

• Address - an address for a physical location.Validated and stored as a (plaintext)String.
• AddressEncrypted - an address for a customeror user which should be stored encrypted for data protection.
• DescriptionPlaintextUnlimited -filters any HTML/JS to avoid security issues. Perfect for blog post comments.
• Encrypted - a general purpose encrypted field.converts any type of datato_string and then encrypts it.
• EmailEncrypted - validate and strongly encryptemail address to ensure they are kept private and secure.
• EmailHash - when an email needs to be looked up fastwithout decrypting. Salted and hashed with:sha256.
• EmailPlaintext - when an email address ispublicthere's no advantage to encrypting it. e.g. a customer support email.
• Hash - a general-purpose hash field using:sha256,useful if you need to store the hash of a value. (one way)
• HtmlBody - useful for storing HTML data e.g in a CMS.
• Name - used for personal namesthat need to be kept private/secure. Max length 35 characters. AES Encrypted.
• Password - passwords hashed usingargon2.
• PhoneNumberEncrypted - a phone number that should be kept private gets validated and encrypted.
• PhoneNumber - when a phone number isnotsensitive information and can be stored in plaintext.
• Postcode - validated postcode stored asplaintext.
• PostcodeEncrypted - validated and encrypted.
• Url - validate a URL and store asplaintext(not encrypted)String
• UrlEncrypted - validate a URL and store as AESencryptedBinary
• IpAddressPlaintext - validate an ipv4 and ipv6 address and store asplaintext
• IpAddressHash - hash for ipv4 or ipv6
• IpAddressEncrypted - validate an ipv4 and ipv6 address and store as AESencryptedBinary

Detailed documentation available onHexDocs:hexdocs.pm/fields

mix t

mix c

If there is a field that you need in your appthat is not already in theFields package,please open an issue so we can add it!github.com/dwyl/fields/issues

If you want an in-depth understanding of how automatic/transparentencryption/decryption works using Ecto Types, see:github.com/dwyl/phoenix-ecto-encryption-example

If you are rusty/new on Binaries in Elixir,take a look at this post by @blackode:
https://medium.com/blackode/playing-with-elixir-binaries-strings-dd01a40039d5

If you have questions, please open an issue:github.com/dwyl/fields/issues

A recent/good example is:issues/169

EmailEncryptedandEmailHashserve very different purposes.Briefly:withencryptionthe output isalways differentis meant for safely storing sensitive datathat we want todecrypt laterwhereas withhashthe output isalways the sameit cannot be "unhashed" butcan be used tocheck a value,i.e. you can lookup ahashed value in a database.

The best way to understand how these workis to see it for yourself.Start anIExsession in your terminal:

iex -S mix

You should see output similar to the following:

Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit] [dtrace]Compiling 23 files (.ex)Generated fields appInteractive Elixir (1.12.3) - press Ctrl+C toexit (typeh() ENTERfor help)

That confirms thefields module has compiled.

Now that you've initializedIEx,issue the following commands:

iex> email ="alex@gmail.com""alex@gmail.com"iex(2)> encrypted = Fields.AES.encrypt(email)<<48, 48, 48, 49, 20, 6, 117, 239, 107, 251, 80, 156, 109, 46, 6, 75, 119, 89,  72, 163, 156, 243, 60, 6, 17, 166, 130, 239, 93, 222, 65, 186, 185, 78, 77, 2,  80, 194, 241, 31, 28, 24, 155, 172, 208, 185, 142, 64, 65, 127>>

Note: theFields.EmailEncrypteduses theAES.encrypt/1 behind the scenes,that's why we are using it here directly.You could just as easily have written:{:ok, encrypted} = Fields.EmailEncrypted.dump(email)this is just a shorthand.

That output<<48, 48, 48 ... 64, 65, 127>> is abitstringwhich is the sequence of bits in memory.The encrypted data - usually called"ciphertext" -is not human readable, that's a feature.But if you want todecrypt it back to its human-readable form,simply run:

iex(3)> decrypted = Fields.AES.decrypt(encrypted)"alex@gmail.com"

So we know that an encrypted value can be decrypted.In the case ofEmailEncrypted this is usefulwhen we want to send someone an email message.For security/privacy,we want their sensitive personal data to be storedencrypted in the Database,but when we need to decrypt it to send them a message,it's easy enough.

If you run theFields.AES.encrypt/1 functionmultiple times in your terminal,you willalways see different output:

iex(4)>Fields.AES.encrypt(email)<<48,48,48,49,168,212,210,53,233,104,27,235,199,43,87,74,3,2,211,114,187,229,157,182,37,34,209,37,66,160,30,126,238,180,146,133,227,53,245,228,119,191,117,247,37,176,130,110,...>>iex(5)>Fields.AES.encrypt(email)<<48,48,48,49,196,170,48,97,75,206,148,204,41,149,64,50,27,56,112,19,53,108,86,153,154,53,53,97,232,133,97,88,214,254,40,84,65,227,75,123,212,222,63,221,176,130,11,173,...>>iex(6)>Fields.AES.encrypt(email)<<48,48,48,49,201,239,104,101,140,232,0,216,183,168,220,130,24,236,205,220,239,112,112,168,86,235,84,115,108,116,16,234,184,72,111,144,245,1,125,207,230,68,126,111,84,83,23,90,...>>iex(7)>Fields.AES.encrypt(email)<<48,48,48,49,176,131,145,182,128,43,11,100,253,73,179,144,139,45,211,156,155,117,119,59,152,148,45,36,95,141,35,242,182,51,235,162,186,132,23,34,174,171,157,115,54,211,124,247,...>>

The first 4 bytes<<48, 48, 48, 49, are the samebecause we are using the same encryption key.But the rest isalways different.

Ahash functioncan be used to map data of arbitrary sizeto fixed-size values.i.e.any length ofplaintext willresult in thesame lengthhashvalue.Ahash function isone-way,it cannot be reversed or "un-hashed".Thehashvalue isalways the samefor a given string of plaintext.

Try it inIEx:

iex(1)>email="alex@gmail.com""alex@gmail.com"iex(2)>Fields.Helpers.hash(:sha256,email)<<95,251,251,204,181,59,239,4,218,193,35,20,223,131,219,101,30,17,97,146,103,115,3,185,230,137,218,137,209,111,48,236>>iex(3)>Fields.Helpers.hash(:sha256,email)<<95,251,251,204,181,59,239,4,218,193,35,20,223,131,219,101,30,17,97,146,103,115,3,185,230,137,218,137,209,111,48,236>>iex(4)>Fields.Helpers.hash(:sha256,email)<<95,251,251,204,181,59,239,4,218,193,35,20,223,131,219,101,30,17,97,146,103,115,3,185,230,137,218,137,209,111,48,236>>

The hashvalue is identical for the given input textin this case the email address"alex@gmail.com".

If you use theFields.EmailHash.dump/1 function,you will see the same hash value(because the same helper function is invoked):

iex(5)>Fields.EmailHash.dump(email){:ok,<<95,251,251,204,181,59,239,4,218,193,35,20,223,131,219,101,30,17,97,146,103,115,3,185,230,137,218,137,209,111,48,236>>}iex(6)>Fields.EmailHash.dump(email){:ok,<<95,251,251,204,181,59,239,4,218,193,35,20,223,131,219,101,30,17,97,146,103,115,3,185,230,137,218,137,209,111,48,236>>}

When theEmailHash is stored in a databasewe can lookup an email address by hashing itand comparing it to the list.

The best way ofvisualizing thisis to convert the hash value (bitstring)tobase64 so that it ishuman-readable:

iex(1)>email="alex@gmail.com""alex@gmail.com"iex(2)>Fields.Helpers.hash(:sha256,email)|>:base64.encode"X/v7zLU77wTawSMU34PbZR4RYZJncwO55onaidFvMOw="iex(3)>Fields.Helpers.hash(:sha256,email)|>:base64.encode"X/v7zLU77wTawSMU34PbZR4RYZJncwO55onaidFvMOw="

Imagine you have a database table calledpeople that has just 3 columns:id,email_hash andemail_encrypted

With this "database" table,we can nowlookup an email address to find out theirid:

iex(4)>Fields.Helpers.hash(:sha256,"alice@gmail.com")|>:base64.encode"+zXMhia/Z2I64nul6pqoDZTVM1q2K21Pby6GtPcm9iE="

This matches theemail_hash in the second row of our table,thereforeAlice'sid is2 in the database.