Search engine ‘spiders’ are robots that seek out webpages to display in search engines. Below we’ll discuss how they work and why they’re important.

A search engine robot is a very simple program that has some basic functionality to help it understand web pages. However, spiders only have limited functionality to interpret websites: they cannot interpret frames, Flash video, images, or JavaScript; they can’t enter password-protected areas and can’t click buttons; they can be stopped by dynamically-generated URLs and JavaScript navigation. However, within HTML code, they’re able to retrieve data by travelling through the web to find information and links.

The robot makes a list of the web pages in the system at the ‘submit a URL page, then searches for these web pages in order from the list the next time it goes on the web. Sometimes a robot will find your page whether you have submitted it or not because other site links may lead the robot to your site. Building your link popularity and getting links from other topical sites back to your site is important. The first thing a robot does when it arrives is to check for a robots.txt file. This file tells the robots which sites are off-limits. Usually these are files that should be of no concern because they are binaries or other files that are not needed by the robot.

Links are collected from every page that is visited. These links are used in following those links to other pages. The robot gets around on the World Wide Web by following links from one place to another.

Once the spider has gathered all the information it needs, and based on how the spider is set up in the search engine, it will index the site information and send it to the search engine database.

A robot ‘reads’ your site by collecting data on any visible text, on tags you may have in the coding of your page, and on any links available. These are the things that determine what the search engines ‘think’ your content is about, so these are the things you really need to pay attention to when building a site that you want to have high visibility in search results.

The search engine sorts the information that has been delivered to the databases which has become a part of the search engine and directory ranking process. This allows it to display the results. Databases are updated periodically. Robots visit you regularly to find any changes to your pages so that the latest information will be available. The way in which the search engine is set up determines how the number of visits you get is calculated. This can vary with different search engines. If your website is down or experiencing a large amount of traffic, the robot may not be able to access the page they are trying to visit. The website may not be re-indexed when this occurs. This depends on how frequently your site is visited by the robot. In the hope that your site will be accessible again, the robot will re-visit your site to see if it has become accessible.

Justin Harrison is an internationally recognised Internet Marketing Consultant who provides world class SEO Services to website owners. For more information visit: http://www.seorankings.co.za