How Search Engine Works

Crawler-based search engines are what most of us use most of the time, with Google being the most obvious example. Search engines like Google, MSN, Yahoo!, and Ask.com send out an automated "crawler" or "spider" to scour the Internet, link by link, and gather information about the pages and sites that it visits. It then indexes the site, recording all kinds of information: content, images, inbound and outbound links, and meta data (data about data: page title, description, keywords, language, and more).

Crawlers are completely automated and run 24/7, reporting back to the search engine what it finds online. All of the collected information is written into a database, so that the next time you search for "partitioning a hard drive," for example, it already knows which pages have this information and it doesn't have to actually search the whole Web just for you. The search engines are really searching their own database, which they update fairly often. Search engines will visit your site anywhere from daily to monthly, but they all check back with your site on a regular basis to see if you've added new pages or content.