Introduction

Congrats, you’ve successfully gone through all the step-by-step instructions to build out your source plugin. You can compare what you have with thefinished example repository.

This part of the tutorial will explain individual advanced topics in a reference style, not tutorial style like the parts before. Or in other words: This living document won’t give step-by-step explanations but rather general instructions, tips, and recommendations. Integrating them all into the example repository would go beyond the scope of this guide. Pick and choose what makes sense for your own source plugin, although we highly recommend always addingtest coverage to your plugin.

While this document recommends a limited set of tools, feel free to use the tools you’re familiar and productive with.

Testing

From our experience a well tested source plugin is more reliable and easier to maintain. You have greater confidence that new features and bug fixes don’t introduce new bugs. It’ll also help you understand the code better after coming back to it after a while, especially if not you but a collaborator has written it.

There is no silver bullet in testing and it’ll depend on your source plugin what form of test coverage it needs. Generally speaking, a mix of unit and end-to-end tests gives a good foundation for every source plugin.

Unit testing

Here are some tips for unit testing source plugins:

• We’d recommend usingJest orVitest as unit testing frameworks.
  • For detailed instructions on setting up Jest with Gatsby, read theUnit Testing guide.
• The most important part of your source plugin is thesourceNodes API. You’re fetching data there and creating nodes from that data. Both parts can be unit tested.
  • Don’t run unit tests against an API that is online somewhere. Your tests will fail when the API is momentarily offline or the machine itself doesn’t have an internet connection. Use HTTP mocking packages likenock to create a placeholder API for your tests.
  • If you’re modifying the incoming data before creating nodes, be sure to test that behavior. In the previous parts of this tutorial you’ve created utility functions (nodeBuilder andcreateAssetNode) as one of their advantages is that you can then test their logic in isolation.
• Read theUnit testing an options schema guide for thepluginOptionsSchema API.

Thefinished example repository uses Jest to unit test thepluginOptionsSchema and utility functions forsourceNodes.

End-to-End testing

If you want to test your plugin inside a browser and a running Gatsby app, you should use end-to-end testing tools. You could test the Image CDN feature you’ve added inPart 6 or if certain schema customizations are working correctly. It can also give you confidence that your source plugin works duringgatsby build from start to finish.

Here are some tips for end-to-end testing source plugins:

• We’d recommend usingCypress orPlaywright as end-to-end testing frameworks.
  • For detailed instructions on setting up Cypress with Gatsby, read theEnd-to-End Testing guide.
• If you have a monorepo with your plugin and an example site (like this tutorial) you should use the example site for your end-to-end tests.
• When running your end-to-end tests in a CI environment (e.g. tests running on each PR) do test the output ofgatsby build. So before running the tests, rungatsby build and thengatsby serve.
  • Locally you can test againstgatsby develop to iterate more quickly on the tests.

Content Sync

Content Sync is a Gatsby Cloud feature for improving the preview experience for content authors. Be sure to read theContent Sync conceptual guide before continuing. To enable this feature in your source plugin you will need to make sure that your data source (or CMS) also works with Content Sync.

Source plugin

The source plugin needs to create node manifests using theunstable_createNodeManifest action.

1. Create thecustomCreateNodeManifest utility function that will be a wrapper around theunstable_createNodeManifest action. It contains the logic for when to call the action and with which arguments.

   Theattributes argument should allow you to create a unique manifest ID later on.isPreview exists because the source plugin should only create node manifests for preview content. ThecustomNodeFilteringFn function is a placeholder for your custom logic to filter out any items depending on yourgatsbyNode. You may want to filter out certain types of nodes, for example if your CMS has nodes that will never be previewed like redirect nodes or other types of non-content data.

2. Identify which nodes you’ll want to create a node manifest for. These typically are nodes that you can preview, entry nodes, top level nodes, etc. An example of this could be a blog post or an article, any node that can be the “owner” of a page. A good place to call this action is whenever you callcreateNode.

3. unstable_createNodeManifest has two required arguments:manifestId andnode.node is the Gatsby node you can reuse,manifestId needs to be created.

   Important:manifestId needs to be created with attributes coming from the CMSonly as the CMS will need to create the exact same manifest, too. Youmust not use Gatsby information for it.

   ThismanifestId must be uniquely tied to a specific revision of specific content. In the example below theid of the content and a timestamp is used. This will vary from your data source/CMS.

   unstable_createNodeManifest optionally also accepts a third argument calledupdatedAtUTC which should be the time in which the node was last updated. If you don’t have that information feel free to leave it out.

Data source / CMS

The CMS needs to send a preview webhook to Gatsby Cloud when content is changed and open the Content Sync waiting room. The details of the instructions below will differ from CMS to CMS depending on how extensions and their settings are created/stored.

You’ll need two pieces of information from Gatsby Cloud:

• Preview webhook URL (format:https://webhook.gatsbyjs.com/hooks/data_source/<id>)
• Content Sync URL (format:https://gatsbyjs.com/content-sync/<siteId>)

You can find both URLs on the Gatsby Cloud site settings page (readBuild and Preview Webhooks to learn more). Both URLs need to be user configurable in the CMS extension.

Here are the other steps to integrate Content Sync into your CMS:

1. Store both URLs inside the CMS extension so that you can reuse them later.

2. Inside the CMS extension, create amanifestId with the same attributes as shown in thesource plugin section. They must match.

3. Store acontentId inside the extension. ThecontentId should be a unique identifier for the project and stay consistent across all previews. So e.g. storing the project ID is a good option.

   contentId will enable so called “eager redirects” which causes the user to be redirected to their site frontend as soon as possible. When a user first previews their content, they’ll stay in the Content Sync loading screen until their preview is ready. On subsequent previews of the same piece of content, they’ll be redirected as soon as the page loads.

4. Create a button in your CMS that does the following:

   1. POST to the preview webhook URL of Gatsby Cloud

   2. Open the Content Sync waiting room. You can do this by redirecting to an URL in the shape of:

      As an example:

Tips

Here are some things to keep in mind and some “gotchas” depending on how the CMS acts:

• Inside the CMS, sometimes you will need to wait to make sure you have the correctupdatedAt timestamp as some CMS may take a second to update their backend and then wait for the change to propagate to the frontend. While others will immediately update the frontend and then propagate that to the backend. You will need themost up to date timestamp when opening the Content Sync UI waiting room.
• Make sure that a preview webhook is being sent to Gatsby Cloud after the content is edited, whether it’s before you press the “Open Preview” button or the “Open Preview” is the trigger that sends the webhook.
• The node manifests get written out in thepublic dir of your gatsby site, so you can check the manifests on your local disk at/public/__node-manifests/<sourcePluginName>/<manifestId>.json or you can navigate directly to that piece of contenthttps://<your-domain>/__node-manifests/<sourcePluginName>/<manifestId>.

MIME types

A media type (also known as a Multipurpose Internet Mail Extensions or MIME type) indicates the nature and format of a document, file, or assortment of bytes. Learn more about MIME types onMDN.

InPart 2 you learned how to usecreateNode. Inside theinternal object that holds thetype andcontentDigest you can optionally also pass amediaType key.

The usage for a markdown file looks something like this:

Source plugins are responsible for setting themediaType on their nodes (if necessary). By providing this key you tell transformer plugins (seeCreating a Transformer Plugin guide) that the node contains raw content that they can transform. Use eitherofficial media types or a made-up one if your data doesn’t fit into any existing bucket.

This distinction between Gatsby source plugins and transformer plugins is important as it allows both to be contained and focused on their specific job. Source plugins don’t need to worry about transforming their data, transformer plugins don’t care where the data is coming from.

For example, the node above with thetext/markdown MIME type will be transformed bygatsby-transformer-remark into furtherMarkdownRemark nodes.

Tracing

If you want to give your users the ability to see tracing of your source plugin (e.g. inOpenTelemetry format) or use the performance tracing yourself to debug bottlenecks, be sure to read thePerformance Tracing guide. The guide has all the information you’d need.

In context of source plugins, thetracing andparentSpan Node API helpers are relevant.

When callingstartSpan on the root tracer pass theparentSpan along, for example:

Summary

You’ve also learned about more advanced topics now, great!

Take a moment to think back on what you’ve learned so far. Challenge yourself to answer the following questions from memory:

• Which parts should you unit test?
• What are end-to-end tests good for?
• What is Content Sync?
• What happens when a source plugin defines amediaType for its nodes?

Key takeaways

• A good mixture of unit tests (testing the heart of the source plugin) and end-to-end tests (testing the functionality from start to finish) will give you confidence in shipping changes.
• Content Sync can improve the preview experience for content editors.
• MIME types can help Gatsby transformers discover nodes from source plugins.
• You can use tracing to inspect your source plugin more deeply.

What’s coming next?

In Part 8 you’ll learn how you can publish your source plugin to npm, making it available to everyone.