Explained by Google (SEO) crawl budget. [Exclusive Guide]
Google crawl is a really important to take into account and to optimize SEO indicator? How Google matched to a crawl by site budget? Google just provide a few answers to frequently asked questions about the crawl of a site by Google and the calculation of crawl budget.
optimization of Google crawl: an interest for all sites?
according to Google , if a site for less than a few thousand pages, the webmaster should (most of the time) not having to worry about the crawl of its site by Google.
Google crawl optimization is an especially interesting practice for large sites (more than 10 000 pages indexed) and those who generate dynamically new URLs based on settings.
what did crawl budget and how is it assigned?
attributed to a site crawl budget depends on several factors:
- site response time: more the server will be quick to respond (fast to load site) will more Googlebot crawler a volume of pages important simultaneously.
- any limits configured in Search Console : base, Google optimizes him even the rhythm to which he is crawler a site based on its response time. If the response time is very fast, the crawl rate will be higher, on the other hand, if the server is slow, Google will slow down the crawl rate. However, since the section “Site settings” in Search Console , webmasters can ask Google to reduce its speed to maximum exploration. They can also configure the acceptable maximum crawl by their server speed.
- the volume of popular pages to crawler : Google tends to crawler popular pages much more frequently than less popular pages so that they are as up-to-date as possible in the index as possible (ex popular page: home).
- the volume of new pages to crawler : a site that will be frequently added new pages will require more budget of crawll what a static website that hasn’t changed since several weeks / months.
- the redesign of a site with change of URLs / domain : the redesign of a site with impact very strongly the Google crawl URLs change because it quickly needs to update the old URLs by the news.
all of these factors determine the budget of crawl assigned to a site. For Google, crawl budget is simply the amount of URLs that Google can and wish crawler on a site .
which can negatively impact the crawl of a Google site?
according to Google, the URLs of low quality which impact negatively the crawl and indexing of the interesting pages of a site are the following:
- the URLs generated via a navigation facets or with identifiers of sessions
- the URLs with internal duplicate content to the site
- the URLs that meet in code 200 as they no longer exist (soft 404)
- the hacked pages
- pages with infinite navigation (ex (: calendars)
- low quality pages or with the content of spam
these issues slow discovery of content and URLs of quality on a site , that impacted so their indexing and necessarily their positioning on the search engine.
frequently asked questions related to the crawl with their official responses by Google
the loading time of a site affect its crawl budget? Can a significant number of errors also adversely affect him?
more a site to load quickly, more Google’s crawler a large number of pages. The loading time of a site impact so actually crawl budget.
if a site returns many 50 x errors, Google will decrease automatically crawl budget.
the crawl is a factor in positioning?
the crawl is useful to index pages but it is not a factor of positioning in any event.
the alternative URLs count in crawl budget?
Yes. All URLs crawlées by Google count in the crawl budget. To discover these alternative URLs (hreflang, mobile site on another URL, page AMP,…), Google must crawler the page source and destination page.
the use of “crawl-delay” in the robots.txt is supported by Google?
no, no official directive is not interpreted by Google.
the nofollow for an impact on the budget of crawl?
Yes if all of the internal links pointing to a page in nofollow, Google won’t get her crawler. If only some of the links that point to this page are nofollow.
If you are looking for tips for optimize the crawl of your site , this article from Google is still relevant according to Gary Illyes: https://webmasters.googleblog.com/2009/08/optimize-your-crawling-indexing.html