GET, POST, and safely surfacing more of the web

Tuesday, November 01, 2011

As the web evolves, Google's crawling and indexing capabilities also need to progress. Weimproved our indexing of Flash, built a more robustinfrastructure called Caffeine, and we even startedcrawling forms where it makes sense. Now, especially with the growing popularity of JavaScript and, with it, AJAX, we're finding more web pages requiringPOST requests—either for the entire content of the page or because the pages are missing information and/or look completely broken without the resources returned fromPOST. For Google Search this is less than ideal, because when we're not properly discovering and indexing content, searchers may not have access to the most comprehensive and relevant results.

We generally advise to useGET for fetching resources a page needs, and this is by far our preferred method of crawling. We've started experiments to rewritePOST requests toGET, and while this remains a valid strategy in some cases, often the contents returned by a web server forGET vs.POST are completely different. Additionally, there are legitimate reasons to usePOST (for example, you can attach more data to aPOST request than aGET). So, whileGET requests remain far more common, to surface more content on the web, Googlebot may now performPOST requests when we believe it's safe and appropriate.

We take precautions to avoid performing any task on a site that could result in executing an unintended user action. OurPOST requests are primarily for crawling resources that a page requests automatically, mimicking what a typical user would see when they open the URL in their browser. This will evolve over time as we find better heuristics, but that's our current approach.

Let's run through a fewPOST request scenarios that demonstrate how we're improving our crawling and indexing to evolve with the web.

Examples of Googlebot'sPOST requests

  • Crawling a page via a POST redirect
        <html>      <body onload="document.foo.submit();">        <form name="foo" action="request.php" method="post"          <input type="hidden" name="bar" value="234"/>        </form>      </body>    </html>
  • Crawling a resource via aPOSTXMLHttpRequest: In this step-by-step example, we improve both the indexing of a page and its Instant Preview by following the automaticXMLHttpRequest generated as the page renders.
    1. Google crawls the URL, yummy-sundae.html.
    2. Google begins indexing yummy-sundae.html and, as a part of this process, decides to attempt to render the page to better understand its content and/or generate the Instant Preview.
    3. During the render, yummy-sundae.html automatically sends an XMLHttpRequest for a resource, hot-fudge-info.html, using thePOST method.
            <html>        <head>          <title>Yummy Sundae</title>          <script src="jquery.js"></script>        </head>        <body>          This page is about a yummy sundae.          <div></div>          <script>            $(document).ready(function() {            $.post('hot-fudge-info.html', function(data)            {$('#content').html(data);});            });          </script>        </body>      </html>
    4. The URL requested throughPOST, hot-fudge-info.html, along with its data payload, is added to Googlebot's crawl queue.
    5. Googlebot performs aPOST request to crawl hot-fudge-info.html.
    6. Google now has an accurate representation of yummy-sundae.html for Instant Previews. In certain cases, we may also incorporate the contents of hot-fudge-info.html into yummy-sundae.html.
    7. Google completes the indexing of yummy-sundae.html.
    8. User searches for "hot fudge sundae".
    9. Google's algorithms can now better determine how yummy-sundae.html is relevant for this query, and we can properly display a snapshot of the page for Instant Previews.

Improving your site's crawlability and indexability

General advice for creating crawlable sites is found in ourHelp Center. For webmasters who want to help Google crawl and index their content and/or generate the Instant Preview, here are a few simple reminders:

  • PreferGET for fetching resources, unless there's a specific reason to usePOST.
  • Verify that we're allowed to crawl the resources needed to render your page. In the example above, if hot-fudge-info.html is disallowed byrobots.txt, Googlebot won't fetch it. More subtly, if the JavaScript code that issues theXMLHttpRequest is located in an external.js file disallowed by robots.txt, we won't see the connection between yummy-sundae.html and hot-fudge-info.html, so even if the latter is not disallowed itself, that may not help us much. We've seen even more complicated chains of dependencies in the wild. To help Google better understand your site it's almost always better to allow Googlebot to crawl all resources.
    You can test whether resources are blocked throughWebmaster ToolsLabs>Instant Previews.
  • Make sure to return the same content to Googlebot as is returned to users' web browsers.Cloaking (sending different content to Googlebot than to users) is a violation of ourWebmaster Guidelines because, among other things, it may cause us to provide a searcher with an irrelevant result —the content the user views in their browser may be a complete mismatch from what we crawled and indexed. We've seen numerousPOST request examples where a webmaster non-maliciously cloaked (which is still a violation), and their cloaking—on even the smallest of changes—then caused JavaScript errors that prevented accurate indexing and completely defeated their reason for cloaking in the first place. Summarizing, if you want your site to be search-friendly, cloaking is an all-around sticky situation that's best to avoid.
    To verify that you're not accidentally cloaking, you can useInstant Previews within Webmaster Tools, or try setting the User-Agent string in your browser to something like:
    Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)
     Your site shouldn't look any different after such a change. If you see a blank page, a JavaScript error, or if parts of the page are missing or different, that means that something's wrong.
  • Remember to include important content (that is, the content you'd like indexed) as text, visible directly on the page and without requiring user-action to display. Most search engines are text-based and generally work best with text-based content. We're always improving our ability to crawl and index content published in a variety of ways, but it remains a good practice to use text for important information.

Controlling your content

If you'd like to prevent content from being crawled or indexed for Google Web Search, traditionalrobots.txt rules remain the best method. To prevent the Instant Preview for your page(s), please see ourInstant Previews FAQ which describes theGoogle Web Preview User-Agent and thenosnippetmeta tag.

Moving forward

We'll continue striving to increase the comprehensiveness of our index so searchers can find more relevant information. And we expect our crawling and indexing capability to improve and evolve over time, just like the web itself. Please let us know if you have questions or concerns.

Written by, Software Engineer, Indexing Team, and, Developer Programs Tech Lead

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.