Crawlers? Spiders? Robots? What’s that? [Quick Guide]

9:27 pm March 3, 201711592

Learn how to get acquainted with the crawlers, spiders, robots or bots yet. These elements essential to the proper functioning of a search engine and indexing your content. Among the most well-known and what you should see on your site we can quote ‘Googlebot’ to Google and Bing’s ‘ BingBot”.

crawler-1

What’s that?

Don’t be afraid of these terms, it is not question of swimming, or Spider man. Whatever, there is question here of navigation, weave the fabric of the web.

In fact, crawlers, spiders, agents, bots, and even robots are terms that all mean the same thing. These are navigation programs which runs continuously the web pages and their content to index the content.

To do so, they follow links embedded in a page and return periodically to take into account new content – if there for been – or any change in content.

A robot can do two things:

crawler-2

understand that these robots don’t have the time to get bored… The web is so active, new links are created continuously, new content is constantly added a site or on another. In other words, these robots roam the web 24/24 in order to be most effective possible and index more quickly discovered content.

The robot is the essential element that collects information for the search engine.

How does a robot?

Here is a diagram which overall includes the functioning of the collection of information by a robot:

crawler-3

Attention, however, the operation of a search engine does not stop at one of the crawler. We will discuss the operation in more detail in a future article.