Repository files navigation

Why would you want to use Google's NoSQLCloud Datastorewith Rails?

When you want a Rails app backed by a managed, massively-scalable datastore solution. Cloud Datastoreautomatically handles sharding and replication. It is a highly available and durable database thatautomatically scales to handle your applications' load. Cloud Datastore is a schemaless databasesuited for unstructured or semi-structured application data.

• Setup
• Model Example
• Controller Example
• Retrieving Entities
• Datastore Consistency
• Datastore Indexes
• Datastore Emulator
• Example Rails App
• CarrierWave File Uploads
• Track Changes
• Nested Forms
• Datastore Gotchas

Generate your Rails app without ActiveRecord:

rails new my_app -O

You can remove the db/ directory as it won't be needed.

To install, add this line to yourGemfile and runbundle install:

gem'activemodel-datastore'

Create a Google Cloud accounthere and create a project.

Google Cloud requires the Project ID and Service Account Credentials to connect to the Datastore API.

Follow theactivation instructions to enable theGoogle Cloud Datastore API. When running on Google Cloud Platform environments the Service Accountcredentials will be discovered automatically. When running on other environments (such as AWS or Heroku)you need to create a service account with the role of editor and generate json credentials.

Set your project id in anENV variable namedGCLOUD_PROJECT.

To locate your project ID:

1. Go to the Cloud Platform Console.
2. From the projects list, select the name of your project.
3. On the left, click Dashboard. The project name and ID are displayed in the Dashboard.

If you have an external application running on a platform outside of Google Cloud you also need toprovide the Service Account credentials. They are specified in two additionalENV variables namedSERVICE_ACCOUNT_CLIENT_EMAIL andSERVICE_ACCOUNT_PRIVATE_KEY. The values for these twoENVvariables will be in the downloaded service account json credentials file.

SERVICE_ACCOUNT_PRIVATE_KEY = -----BEGIN PRIVATE KEY-----\nMIIFfb3...5dmFtABy\n-----END PRIVATE KEY-----\nSERVICE_ACCOUNT_CLIENT_EMAIL = web-app@app-name.iam.gserviceaccount.com

On Heroku theENV variables can be set under 'Settings' -> 'Config Variables'.

Active Model Datastore will then handle the authentication for you, and the datastore instance canbe accessed withCloudDatastore.dataset.

There is an example Puma config filehere.

Let's start by implementing the model:

classUserincludeActiveModel::Datastoreattr_accessor:email,:enabled,:name,:role,:statedefentity_properties%w[emailenablednamerole]endend

Data objects in Cloud Datastore are known as entities. Entities are of a kind. An entity has oneor more named properties, each of which can have one or more values. Think of them like this:

• 'Kind' (which is your table and the name of your Rails model)
• 'Entity' (which is the record from the table)
• 'Property' (which is the attribute of the record)

Theentity_properties method defines an Array of properties that belong to the entity in clouddatastore. Define the attributes of your model usingattr_accessor. With this approach, Railsdeals solely with ActiveModel objects. The objects are converted to/from entities automaticallyduring save/query operations. You can still use virtual attributes on the model (such as the:state attribute above) by simply excluding it fromentity_properties. In this example stateis available to the model but won't be persisted with the entity in datastore.

Validations work as you would expect:

classUserincludeActiveModel::Datastoreattr_accessor:email,:enabled,:name,:role,:statevalidates:email,format:{with:/\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\z/i}validates:name,presence:true,length:{maximum:30}defentity_properties%w[emailenablednamerole]endend

Callbacks work as you would expect. We have also added the ability to set default values throughdefault_property_valueand type cast the format of values throughformat_property_value:

classUserincludeActiveModel::Datastoreattr_accessor:email,:enabled,:name,:role,:statebefore_validation:set_default_valuesafter_validation:format_valuesbefore_save{puts'** something can happen before save **'}after_save{puts'** something can happen after save **'}validates:email,format:{with:/\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\z/i}validates:name,presence:true,length:{maximum:30}validates:role,presence:truedefentity_properties%w[emailenablednamerole]enddefset_default_valuesdefault_property_value:enabled,truedefault_property_value:role,1enddefformat_valuesformat_property_value:role,:integerendend

Now on to the controller! A scaffold generated controller works out of the box:

classUsersController <ApplicationControllerbefore_action:set_user,only:[:show,:edit,:update,:destroy]defindex@users=User.allenddefshowenddefnew@user=User.newenddefeditenddefcreate@user=User.new(user_params)respond_todo |format|if@user.saveformat.html{redirect_to@user,notice:'User was successfully created.'}elseformat.html{render:new}endendenddefupdaterespond_todo |format|if@user.update(user_params)format.html{redirect_to@user,notice:'User was successfully updated.'}elseformat.html{render:edit}endendenddefdestroy@user.destroyrespond_todo |format|format.html{redirect_tousers_url,notice:'User was successfully destroyed.'}endendprivatedefset_user@user=User.find(params[:id])enddefuser_paramsparams.require(:user).permit(:email,:name)endend

Each entity in Cloud Datastore has a key that uniquely identifies it. The key consists of thefollowing components:

• the kind of the entity, which is User in these examples
• an identifier for the individual entity, which can be either a a key name string or an integer numeric ID
• an optional ancestor path locating the entity within the Cloud Datastore hierarchy

Queries entities using the provided options. When a limit option is provided queries up to the limitand returns results with a cursor.

users=User.all(options={})parent_key=CloudDatastore.dataset.key('Parent',12345)users=User.all(ancestor:parent_key)users=User.all(ancestor:parent_key,where:['name','=','Bryce'])users=User.all(where:[['name','=','Ian'],['enabled','=',true]])users,cursor=User.all(limit:7)# @param [Hash] options The options to construct the query with.## @option options [Google::Cloud::Datastore::Key] :ancestor Filter for inherited results.# @option options [String] :cursor Sets the cursor to start the results at.# @option options [Integer] :limit Sets a limit to the number of results to be returned.# @option options [String] :order Sort the results by property name.# @option options [String] :desc_order Sort the results by descending property name.# @option options [Array] :select Retrieve only select properties from the matched entities.# @option options [Array] :distinct_on Group results by a list of properties.# @option options [Array] :where Adds a property filter of arrays in the format[name, operator, value].

Find entity by id - this can either be a specific id (1), a list of ids (1, 5, 6), or an array of ids ([5, 6, 10]).The parent key is optional. This method is a lookup by key and results will be strongly consistent.

user=User.find(1)parent_key=CloudDatastore.dataset.key('Parent',12345)user=User.find(1,parent:parent_key)users=User.find(1,2,3)

Queries for the first entity matching the specified condition.

user=User.find_by(name:'Joe')user=User.find_by(name:'Bryce',ancestor:parent_key)

Cloud Datastore has documentation on howDatastore Querieswork, and pay special attention to the therestrictions.

Cloud Datastore is a non-relational databases, or NoSQL database. It distributes data over manymachines and uses synchronous replication over a wide geographic area. Because of this architectureit offers a balance of strong and eventual consistency.

What is eventual consistency?

It means that an updated entity value may not be immediately visible when executing a query.Eventual consistency is a theoretical guarantee that, provided no new updates to an entity are made,all reads of the entity will eventually return the last updated value.

In the context of a Rails app, there are times that eventual consistency is not ideal. For example,let's say you create a user entity with a key that looks like this:

@key=#<Google::Cloud::Datastore::Key @kind="User", @id=1>

and then immediately redirect to the index view of users. There is a good chance that your new useris not yet visible in the list. If you perform a refresh on the index view a second or two laterthe user will appear.

"Wait a minute!" you say. "This is crap!" you say. Fear not! We can make the query of users stronglyconsistent. We just need to use entity groups and ancestor queries. An entity group is a hierarchyformed by a root entity and its children. To create an entity group, you specify an ancestor pathfor the entity which is a parent key as part of the child key.

Before using thesave method, assign theparent_key_id attribute an ID. Let's say that 12345represents the ID of the company that the users belong to. The key of the user entity will nowlook like this:

@key=#<Google::Cloud::Datastore::Key @kind="User", @id=1, @parent=#<Google::Cloud::Datastore::Key @kind="ParentUser", @id=12345>>

All of the User entities will now belong to an entity group named ParentUser and can be queried by theCompany ID. When we query for the users we will provide User.parent_key(12345) as the ancestor option.

Ancestor queries are always strongly consistent.

However, there is a small downside. Entities with the same ancestor are limited to 1 write per second.Also, the entity group relationship cannot be changed after creating the entity (as you can't modifyan entity's key after it has been saved).

The Users controller would now look like this:

classUsersController <ApplicationControllerbefore_action:set_user,only:[:show,:edit,:update,:destroy]defindex@users=User.all(ancestor:User.parent_key(12345))enddefshowenddefnew@user=User.newenddefeditenddefcreate@user=User.new(user_params)@user.parent_key_id=12345respond_todo |format|if@user.saveformat.html{redirect_to@user,notice:'User was successfully created.'}elseformat.html{render:new}endendenddefupdaterespond_todo |format|if@user.update(user_params)format.html{redirect_to@user,notice:'User was successfully updated.'}elseformat.html{render:edit}endendenddefdestroy@user.destroyrespond_todo |format|format.html{redirect_tousers_url,notice:'User was successfully destroyed.'}endendprivatedefset_user@user=User.find(params[:id],parent:User.parent_key(12345))enddefuser_paramsparams.require(:user).permit(:email,:name)endend

See here for the Cloud Datastore documentation onData Consistency.

Every cloud datastore query requires an index. Yes, you read that correctly. Every single one. Theindexes contain entity keys in a sequence specified by the index's properties and, optionally,the entity's ancestors.

There are two types of indexes,built-in andcomposite.

By default, Cloud Datastore automatically predefines an index for each property of each entity kind.These single property indexes are suitable for simple types of queries. These indexes are free anddo not count against your index limit.

Composite index multiple property values per indexed entity. Composite indexes support complexqueries and are defined in an index.yaml file.

Composite indexes are required for queries of the following form:

• queries with ancestor and inequality filters
• queries with one or more inequality filters on a property and one or more equality filters on other properties
• queries with a sort order on keys in descending order
• queries with multiple sort orders
• queries with one or more filters and one or more sort orders

NOTE: Inequality filters are LESS_THAN, LESS_THAN_OR_EQUAL, GREATER_THAN, GREATER_THAN_OR_EQUAL.

Google has excellent doc regarding datastore indexeshere.

The datastore emulator generates composite indexes in an index.yaml file automatically. The filecan be found in /tmp/local_datastore/WEB-INF/index.yaml. If your localhost Rails app exercises everypossible query the application will issue, using every combination of filter and sort order, thegenerated entries will represent your complete set of indexes.

One thing to note is that the datastore emulator caches indexes. As you add and modify applicationcode you might find that the local datastore index.yaml contains indexes that are no longer needed.In this scenario try deleting the index.yaml and restarting the emulator. Navigate through your Railsapp and the index.yaml will be built from scratch.

Install the Google Cloud SDK.

$ curl https://sdk.cloud.google.com | bash

You can check the version of the SDK and the components installed with:

$ gcloud components list

Install the Cloud Datastore Emulator, which provides local emulation of the production CloudDatastore environment and the gRPC API. However, you'll need to do a small amount of configurationbefore running the application against the emulator, seehere.

$ gcloud components install cloud-datastore-emulator

Add the following line to your ~/.bash_profile:

export PATH="~/google-cloud-sdk/platform/cloud-datastore-emulator:$PATH"

Restart your shell:

exec -l $SHELL

To create the local development datastore execute the following from the root of the project:

$ cloud_datastore_emulator create tmp/local_datastore

To create the local test datastore execute the following from the root of the project:

$ cloud_datastore_emulator create tmp/test_datastore

To start the local Cloud Datastore emulator:

$ cloud_datastore_emulator start --port=8180 tmp/local_datastore

There is an example Rails 5 app in the test directoryhere.

$ bundle$ cloud_datastore_emulator create tmp/local_datastore$ cloud_datastore_emulator create tmp/test_datastore$ ./start-local-datastore.sh$ rails s

Navigate tohttp://localhost:3000.

Active Model Datastore has built in support forCarrierWavewhich is a simple and extremely flexible way to upload files from Rails applications. You can usedifferent stores, including filesystem and cloud storage such as Google Cloud Storage or AWS.

Simply requireactive_model/datastore/carrier_wave_uploader and extend your model with theCarrierWaveUploader (after including ActiveModel::Datastore). Follow the CarrierWaveinstructions for generatingan uploader.

In this example it will be something like:

rails generate uploader ProfileImage

Define an attribute on the model for your file(s). You can then mount the uploaders usingmount_uploader (single file) ormount_uploaders (array of files). Don't forget to add the newattribute toentity_properties and whitelist the attribute in the controller if using strongparameters.

require'active_model/datastore/carrier_wave_uploader'classUserincludeActiveModel::DatastoreextendCarrierWaveUploaderattr_accessor:email,:enabled,:name,:profile_image,:rolemount_uploader:profile_image,ProfileImageUploaderdefentity_properties%w[emailenablednameprofile_imagerole]endend

You will want to add something like this to your Rails form:

<%= form.file_field :profile_image %>

TODO: document the change tracking implementation.

Adds support for nested attributes to ActiveModel. Heavily inspired byRails ActiveRecord::NestedAttributes.

Nested attributes allow you to save attributes on associated records along with the parent.It's used in conjunction with fields_for to build the nested form elements.

See RailsActionView::Helpers::FormHelper::fields_for for more info.

NOTE: Unlike ActiveRecord, the way that the relationship is modeled between the parent andchild is not enforced. With NoSQL the relationship could be defined by any attribute, or withdenormalization exist within the same entity. This library provides a way for the objects tobe associated yet saved to the datastore in any way that you choose.

You enable nested attributes by defining an:attr_accessor on the parent with the pluralizedname of the child model.

Nesting also requires that a<association_name>_attributes= writer method is defined in yourparent model. If an object with an association is instantiated with a params hash, and thathash has a key for the association, Rails will call the<association_name>_attributes=method on that object. Within the writer method callassign_nested_attributes, passing inthe association name and attributes.

Let's say we have a parent Recipe with Ingredient children.

Start by defining within the Recipe model:

• an attr_accessor of:ingredients
• a writer method namedingredients_attributes=
• thevalidates_associated method can be used to validate the nested objects

Example:

classRecipeattr_accessor:ingredientsvalidates:ingredients,presence:truevalidates_associated:ingredientsdefingredients_attributes=(attributes)assign_nested_attributes(:ingredients,attributes)endend

You may also set a:reject_if proc to silently ignore any new record hashes if they fail topass your criteria. For example:

classRecipedefingredients_attributes=(attributes)reject_proc=proc{ |attributes|attributes['name'].blank?}assign_nested_attributes(:ingredients,attributes,reject_if:reject_proc)endend

Alternatively,:reject_if also accepts a symbol for using methods:

classRecipedefingredients_attributes=(attributes)reject_proc=proc{ |attributes|attributes['name'].blank?}assign_nested_attributes(:ingredients,attributes,reject_if:reject_recipes)enddefreject_recipes(attributes)attributes['name'].blank?endend

Within the parent modelvalid? will validate the parent and associated children andnested_models will return the child objects. If the nested form submitted params containeda truthy_destroy key, the appropriate nested_models will havemarked_for_destruction setto True.

When a query does not specify a sort order, the results are returned in the order they are retrieved.As Cloud Datastore implementation evolves (or if a project's indexes change), this order may change.Therefore, if your application requires its query results in a particular order, be sure to specifythat sort order explicitly in the query.