Thursday, January 31, 2019

Many frontend frameworks rely on JavaScript to show content. This can mean Google might take some time to index your content or update the indexed content.

A workaround we discussed at Google I/O this year isdynamic rendering. There are many ways to implement this. This blog post shows an example implementation of dynamic rendering usingRendertron, which is an open source solution based on headless Chromium.

Which sites should consider dynamic rendering?

Not all search engines or social media bots visiting your website can run JavaScript. Googlebot might take time to run your JavaScript and hassome limitations, for example.

Dynamic rendering is useful for content that changes often and needs JavaScript to display. Your site's user experience (especially the time tofirst meaningful paint) may benefit from considering hybrid rendering (for example,Angular Universal).

How does dynamic rendering work?

Dynamic rendering means switching between client-side rendered and pre-rendered content for specific user agents.

You will need a renderer to execute the JavaScript and produce static HTML. Rendertron is an open source project that usesheadless Chromium to render. Single Page Apps often load data in the background or defer work to render their content. Rendertron has mechanisms to determine when a website has completed rendering. It waits until all network requests have finished and there is no outstanding work.

This post covers:

1. Take a look at a sample web app
2. Set up a smallexpress.js server to serve the web app
3. Install and configure Rendertron as a middleware for dynamic rendering

The sample web app

The "kitten corner" web app uses JavaScript to load a variety of cat images from an API and displays them in a grid.

Here is the #"no" dir="ltr" is-upgraded syntax="GDScript">constapiUrl='https://api.thecatapi.com/v1/images/search?limit=50';consttpl=document.querySelector('template').content;constcontainer=document.querySelector('ul');functioninit(){fetch(apiUrl).then(response=>response.json()).then(cats=>{container.innerHTML='';cats.map(cat=>{constli=document.importNode(tpl,true);li.querySelector('img').src=cat.url;returnli;}).forEach(li=>container.appendChild(li));})}init();document.querySelector('button').addEventListener('click',init);

The web app uses modern JavaScript (ES6), whichisn't supported in Googlebot yet. We can use themobile-friendly test to check if Googlebot can see the content:

While this problem is simple to fix, it's a good exercise to learn how to setup dynamic rendering. Dynamic rendering will allow Googlebot to see the cat pictures without changes to the web app code.

Set up the server

To serve the web application, let's useexpress, anode.js library, to build web servers.

The server code looks like this (find thefull project source code here):

constexpress=require('express');constapp=express();constDIST_FOLDER=process.cwd()+'/docs';constPORT=process.env.PORT||8080;//Servestaticassets(images,css,etc.)app.get('*.*',express.static(DIST_FOLDER));//PointallotherURLstoindex.htmlforoursinglepageappapp.get('*',(req,res)=>{res.sendFile(DIST_FOLDER+'/index.html');});//StartExpressServerapp.listen(PORT,()=>{console.log(`NodeExpressserverlisteningonhttps://localhost:${PORT}from${DIST_FOLDER}`);});

You cantry the live example - you should see a bunch of cat pictures, if you are using a modern browser. To run the project from your computer, you neednode.js to run the following commands:

npm install --save express rendertron-middlewarenode server.js

Then point your browser tohttps://localhost:8080. Now it's time to set up dynamic rendering.

Deploy a Rendertron instance

Rendertron runs a server that takes a URL and returns static HTML for the URL by using headless Chromium. We'll followthe recommendation from the Rendertron project and useGoogle Cloud Platform.

Please note that you can get started withthe usage tier that's available without payment; using this setup in production may incur costs according to theGoogle Cloud Platform pricing.

1. Create a new project in theGoogle Cloud console. Take note of the "Project ID" below the input field.
2. Install theGoogle Cloud SDK as described in the documentation and log in.
3. Clone the Rendertron repository from GitHub with:

   git clone https://github.com/GoogleChrome/rendertron.gitcd rendertron

4. Run the following commands to install dependencies and build Rendertron on your computer:

   npm install && npm run build

5. Enable Rendertron's cache by creating a new file calledconfig.json in the rendertron directory with the following content:

   { "datastoreCache": true }

6. Run the following command from the rendertron directory. SubstituteYOUR_PROJECT_ID with your project ID from step 1.

   gcloud app deploy app.yaml --projectYOUR_PROJECT_ID

7. Select a region of your choice and confirm the deployment. Wait for it to finish.
8. Enter the URLYOUR_PROJECT_ID.appspot.com. You should see Rendertron's interface with an input field and a few buttons.

When you see the Rendertron web interface, you have successfully deployed your own Rendertron instance. Take note of your project's URL (YOUR_PROJECT_ID.appspot.com) as you will need it in the next part of the process.

Add Rendertron to the server

The web server is usingexpress.js and Rendertron has anexpress.js middleware. Run the following command in the directory of theserver.js file:

npm install --save rendertron-middleware

This command installs the rendertron-middleware from npm so we can add it to the server:

constexpress=require('express');constapp=express();constrendertron=require('rendertron-middleware');

Configure the bot list

Rendertron uses theuser-agent HTTP header to determine if a request comes from a bot or a user's browser. It has awell-maintained list of bot user agents to compare with. By default this list does not include Googlebot, because Googlebot can execute JavaScript. To make Rendertron render Googlebot requests as well, add Googlebot to the list of user agents:

constBOTS=rendertron.botUserAgents.concat('googlebot');constBOT_UA_PATTERN=newRegExp(BOTS.join('|'),'i');

Rendertron compares theuser-agent header against this regular expression later.

Add the middleware

To send bot requests to the Rendertron instance, we need to add the middleware to ourexpress.js server. The middleware checks the requesting user agent and forwards requests from known bots to the Rendertron instance. Add the following code to server.js and don't forget to substituteYOUR_PROJECT_ID with your Google Cloud Platform project ID:

app.use(rendertron.makeMiddleware({proxyUrl:'https://YOUR_PROJECT_ID.appspot.com/render',userAgentPattern:BOT_UA_PATTERN}));

Bots requesting the sample website receive the static HTML from Rendertron, so the bots don't need to run JavaScript to display the content.

Testing our setup

To test if the Rendertron setup was successful, run the Mobile-Friendly Test again.

Unlike the first test, the cat pictures are visible. In the HTML tab we can see all HTML the JavaScript code generated and that Rendertron has removed the need for JavaScript to display the content.

Conclusion

You created a dynamic rendering setup without making any changes to the web app. With these changes, you can serve a static HTML version of the web app to crawlers.

Posted byMartin Splitt, Open Web Unicorn