Explained by Google (SEO) crawl budget. [Exclusive Guide]

3:23 pm January 17, 201714769

Google crawl is a really important to take into account and to optimize SEO indicator? How Google matched to a crawl by site budget? Google just provide a few answers to frequently asked questions about the crawl of a site by Google and the calculation of crawl budget.

optimization of Google crawl: an interest for all sites?

according to Google , if a site for less than a few thousand pages, the webmaster should (most of the time) not having to worry about the crawl of its site by Google.

Google crawl optimization is an especially interesting practice for large sites (more than 10 000 pages indexed) and those who generate dynamically new URLs based on settings.

what did crawl budget and how is it assigned?

attributed to a site crawl budget depends on several factors:

all of these factors determine the budget of crawl assigned to a site. For Google, crawl budget is simply the amount of URLs that Google can and wish crawler on a site .

which can negatively impact the crawl of a Google site?

according to Google, the URLs of low quality which impact negatively the crawl and indexing of the interesting pages of a site are the following:

  • the URLs generated via a navigation facets or with identifiers of sessions
  • the URLs with internal duplicate content to the site
  • the URLs that meet in code 200 as they no longer exist (soft 404)
  • the hacked pages
  • pages with infinite navigation (ex (: calendars)
  • low quality pages or with the content of spam

these issues slow discovery of content and URLs of quality on a site , that impacted so their indexing and necessarily their positioning on the search engine.

frequently asked questions related to the crawl with their official responses by Google

the loading time of a site affect its crawl budget? Can a significant number of errors also adversely affect him?

more a site to load quickly, more Google’s crawler a large number of pages. The loading time of a site impact so actually crawl budget.

if a site returns many 50 x errors, Google will decrease automatically crawl budget.

the crawl is a factor in positioning?

the crawl is useful to index pages but it is not a factor of positioning in any event.

the alternative URLs count in crawl budget?

Yes. All URLs crawlées by Google count in the crawl budget. To discover these alternative URLs (hreflang, mobile site on another URL, page AMP,…), Google must crawler the page source and destination page.

the use of “crawl-delay” in the robots.txt is supported by Google?

no, no official directive is not interpreted by Google.

the nofollow for an impact on the budget of crawl?

Yes if all of the internal links pointing to a page in nofollow, Google won’t get her crawler. If only some of the links that point to this page are nofollow.

If you are looking for tips for optimize the crawl of your site , this article from Google is still relevant according to Gary Illyes: https://webmasters.googleblog.com/2009/08/optimize-your-crawling-indexing.html


your opinion on this info?