Search Engines, Spiders and Web Crawlers

How traffic results from search engine bots

Unless you published a direct link to your website which is clicked by a Web user, it is the search engines that ultimately bring your website to the attention of visitors who might eventually become the prospective customers that every Internet business craves. Hence it helps to know how these search engines actually work and how they present information to customers initiating a search.

There are basically two types of search systems. The first is search engines that utilize robots or bots which are commonly referred to as crawlers or spiders. The second is Web directories which do not utilize crawlers or spiders, but categorize whole web sites rather than one page or a set of keywords, and human editors review site submissions for fitness. For the purpose of this post we’ll discuss search engines.

Search Engines use spiders to index websites. When you submit your website pages to a search engine by completing their required submission form, the search engine spider will index your entire site. A spider€™ is an automated program that is run by the search engine system. Spider visits a web site, read the content on the actual site, the site’s Meta tags and also follow the links that connects the site to other sites.

Search engines and traffic results vary

The spider then returns all that information to a central depository, where the data is indexed. It will visit each link you have on your website and index those sites as well. Some spiders will only index a certain number of pages on your site, so don’t create a site with 500 pages!

The spider will periodically return to the sites to check for any information that has changed. The frequency with which this happens is determined by the moderators of the search engine. A spider is almost like a book in the sense that it contains a table of contents, the actual content, as well as links and references for all the websites it finds during its search, and it may index up to a million pages a day.

Some examples of popular search engines are, Excite, Lycos, AltaVista and Google, each one employing a different methodology or search structure known as an algorithm. When you ask a search engine to locate information, it is actually searching through the index which it has created and not actually searching the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through the indices.

Search engine bots know your site well

One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing which is often referred to as spamdexing. Then each algorithm analyzes the way in which pages link to, and are linked from, other pages in the Web.

By checking how pages link to each other, a search engine can both determine what a page is about, if the keywords of the linked pages are similar to the keywords on the original page and a number of other features and performance activity of that particular website.