Already have an account? Sign in

Heap (v1)

Heap integration summary

Stitch’s Heap integration replicates data from Avro files published to Amazon S3 via Heap’sConnect for Amazon S3 feature. Refer to theSchema section for a list of objects available for replication.

Heap feature snapshot

A high-level look at Stitch's Heap (v1) integration, including release status, useful links, and the features supported in Stitch.

Connecting Heap

Heap setup requirements

To set up Heap in Stitch, you need:

• Access to Heap Connect using Amazon S3. Stitch’s Heap integration currently only replicates data from Heap Amazon S3 instances.

• Permissions in AWS Identity Access Management (IAM) that allow you to create policies, create roles, and attach policies to roles. This is required to grant Stitch authorization to your S3 bucket.

Step 1: Retrieve your Amazon Web Services account ID

1. Sign into your Amazon Web Services (AWS) account.
2. Click theuser menu, located between thebell andGlobal menus in the top-right corner of the page.
3. ClickMy Account.

4. In theAccount Settings section of the page, locate theAccount Id field:

   

Keep this handy - you’ll need it to complete the setup.

Step 2: Add Heap as a Stitch data source

1. Sign into your Stitch account.

2. On the Stitch Dashboard page, click theAdd Integration button.

3. Click theHeap icon.

4. Enter a name for the integration. This is the name that will display on the Stitch Dashboard for the integration; it’ll also be used to create the schema in your destination.

   For example, the name “Stitch Heap” would create a schema calledstitch_heap in the destination.Note: Schema names cannot be changed after you save the integration.

5. In theS3 Bucket field, enter the name of the bucket. Enter only the bucket name: No URLs,https, or S3 parts. For example:heap-rs3-stitch-bucket
6. In theAWS Account ID field, paste the account ID you retrieve inStep 1.

Step 3: Define the historical replication start date

The Sync Historical Data setting defines the starting date for your Heap integration. This means that dataequal to or newer than this date will be replicated to your data warehouse.

Change this setting if you want to replicate data beyond Heap’s default setting of1 year. For a detailed look at historical replication jobs, check out theSyncing Historical SaaS Data guide.

Step 4: Create a replication schedule

In theReplication Frequency section, you’ll create the integration’sreplication schedule. An integration’s replication schedule determines how often Stitch runs a replication job, and the time that job begins.

Heap integrations support the following replication scheduling methods:

• Replication Frequency

• Anchor Scheduling

To keep your row usage low, consider setting the integration to replicate less frequently. See theUnderstanding and Reducing Your Row Usage guide for tips on reducing your usage.

Step 5: Grant access to your bucket using AWS IAM

Next, Stitch will display aConfigure Your Heap Integration page. This page contains the info you need to configure bucket access for Stitch, which is accomplished via an IAM policy and role.

Note: Saving the integration before you’ve completed the steps below will result in connection errors.

• Step 5.1: Create an IAM policy

• Step 5.2: Create an IAM role for Stitch

• Step 5.3: Check and save the connection in Stitch

Step 5.1: Create an IAM policy

An IAM policy is JSON-based access policy language to manage permissions to Heap resources.

For more info about the permissions the auto-generated policy Stitch IAM policy grants, click the link below.

To create the IAM policy:

1. In AWS,navigate to the IAM service by clicking theServices menu and typingIAM.
2. ClickIAM once it displays in the results.
3. On the IAM home page, clickPolicies in the menu on the left side of the page.
4. ClickCreate Policy.
5. In theCreate Policy page, click theJSON tab.
6. Select everything currently in the text field and delete it.
7. In the text field, paste the IAM policy from theConfigure Your Heap Integration page in Stitch.
8. ClickReview policy.
9. On theReview Policy page, give the policy a name. For example:stitch_heap
10. ClickCreate policy.

Step 5.2: Create an IAM role for Stitch

In this step, you’ll create an IAM role for Stitch and apply the IAM policy from the previous step. This will ensure that Stitch is visible in any logs and audits.

To create the role, you’ll need theAccount ID,External ID, andRole Name values provided on the StitchConfigure Your Heap Integration page.

1. In AWS, navigate to theIAM Roles page.
2. ClickCreate Role.
3. On theCreate Role page:
   1. In theSelect type of trusted entity section, click theAnother AWS account option.
   2. In theAccount ID field, paste the Account ID from Stitch.Note: This isn’t your AWS account ID from Step 1 - this is the Account ID that displays in Stitch on theConfigure Your Heap Integration page.
   3. In theOptions section, check theRequire external ID box.
   4. In theExternal ID field that displays, paste the External ID from the StitchConfigure Your Heap Integration page:
   5. ClickNext: Permissions.
4. On theAttach permissions page:
   1. Search for the policy you created in theprevious step.
   2. Once located, check the box next to it in the table.
   3. ClickNext: Tags.
5. If you want to enter any tags, do so on theAdd tags page. Otherwise, clickNext: Review.
6. On theReview page:

   1. In theRole name field, paste the Role Name from the StitchConfigure Your Heap Integration page:

      Remember: Role names are unique to the Stitch Heap integration they’re created for. Attempting to use the same role for multiple integrations will cause connection errors.

   2. Enter a description in theRole description field. For example:Stitch role for Heap integration.
   3. ClickCreate role.

Step 5.3: Check and save the connection in Stitch

After you’ve created the IAM policy and role, you can save the integration in Stitch. When finished, clickCheck and Save.

Step 6: Set objects to replicate

The last step is to select the tables and columns you want to replicate.Learn about the available tables for this integration.

Note: If a replication job is currently in progress, new selections won’t be used until the next job starts.

• Selection options
• Individual tables and columns
• All tables and columns

Initial and historical replication jobs

After you finish setting up Heap, itsSync Status may show asPending on either the Stitch Dashboard or in the Integration Details page.

For a new integration, aPending status indicates that Stitch is in the process of scheduling the initial replication job for the integration.This may take some time to complete.

Free historical data loads

The first seven days of replication, beginning when data is first replicated, are free. Rows replicated from the new integration during this time won’t count towards your quota. Stitch offers this as a way of testing new integrations, measuring usage, and ensuring historical data volumes don’t quickly consume your quota.

Heap replication

Replication in Stitch’s Heap integration depends on two factors:

1. How Heap syncs data to your Amazon S3 bucket, and
2. How Stitch identifies new data in Heap integrations

Heap data syncs to Amazon S3

Heap dumps data into Amazon S3 periodically.By default, this is on a nightly basis.

According to Heap’s documentation:

Heap will provide a periodic dump of data into S3 (nightly by default). Data will be delivered in the form of Avro-encoded files, each of which corresponds to one downstream table (though there can be multiple files per table). Dumps will be incremental, though individual table dumps can be full resyncs, depending on whether the table was recently toggled or the event definition modified.

This means that while files will only include new and updated data pertinent to that specific object (table), a full resync may be included.

Key-based Incremental Replication using file modification timestamps

To identify new and updated data for replication, Stitch will use file modification timestamps asReplication Keys and store them on a per-table basis. This means that only files dumped from a new Heap data sync will be selected for replication.

While data from Heap integrations is replicated usingKey-based Incremental Replication, the behavior for this integration differs subtly from other integrations.

The table below compares Key-based Incremental Replication andReplication Key behavior for Heap to that of other integrations.

Heap table reference

Custom attributes

Heap’s data model is dynamic, meaning it changes as custom attributes are added to object types in your account. For example: Adding new user attributes to theuser object.

This means that the Heap schema in your destination may also change over time as you add new attributes in Heap.

When a new attribute is added to an object in Heap, it will display as a selectable field in the Stitch app.Note: To include the field in replication, you’ll need to select it in Stitch. Stitch will not automatically select new fields.

The schema documentation following this section outlines the default attributes for each object type according toHeap’s documentation.

Event tables

For each event type you define in Heap, a table for that event will be available for selection in Stitch.

For example: If there’s aSign up - Click button event, there will be a table namedsign_up_click_button.

Refer to the[event_type] schema documentation for a list of default event attributes.

Note: When new event types are added in Heap, you will need to select the table and fields in Stitch to include it in replication.

event_tables.event_id = pageviews.event_id

event_tables.session_id = pageviews.session_id

event_tables.user_id = pageviews.user_id

event_tables.event_id = sessions.event_id

event_tables.session_id = sessions.session_id

event_tables.user_id = sessions.user_id

event_tables.user_id = user_migrations.to_user_id

event_tables.user_id = users.user_id

pageviews.event_id = event_tables.event_id

pageviews.session_id = event_tables.session_id

pageviews.user_id = event_tables.user_id

pageviews.event_id = sessions.event_id

pageviews.session_id = sessions.session_id

pageviews.user_id = sessions.user_id

pageviews.user_id = user_migrations.to_user_id

pageviews.user_id = users.user_id

sessions.event_id = event_tables.event_id

sessions.session_id = event_tables.session_id

sessions.user_id = event_tables.user_id

sessions.event_id = pageviews.event_id

sessions.session_id = pageviews.session_id

sessions.user_id = pageviews.user_id

sessions.user_id = user_migrations.to_user_id

sessions.user_id = users.user_id

user_migrations.to_user_id = event_tables.user_id

user_migrations.to_user_id = pageviews.user_id

user_migrations.to_user_id = sessions.user_id

user_migrations.to_user_id = users.user_id

users.user_id = event_tables.user_id

users.user_id = pageviews.user_id

users.user_id = sessions.user_id

users.user_id = user_migrations.to_user_id

Did this article help? If you have questions or feedback, feel free tosubmit a pull request with your suggestions,open an issue on GitHub, orreach out to us.