What are search engines and how do they work?
Search engines are usually "crawler based," meaning they crawl the internet searching for web pages by scanning for key words. They scan the content and information, and follow links that lead to other pages. Search engines operate using algorithms, which find information on websites, and store them in a large index, or catalogue. As sites are updated and changes occurs, the index is updated as well.
As users query the web for keywords, they provide answers and find results from what the engine has crawled. Then the index is stored, by calculating the relevancy. The engines sort through pages, and bring up the matches that are getting to the closest keywords searched.
The pages are ranked according to the greatest relevance of content. The order of ranking greatly contributes to the site's popularity and success.
How did Search Engine Development Begin?
Search engines were first introduced in 1945 when American engineer and scientist Vannevar Bush published an article in The Atlantic Monthly, emphasizing the necessity for an expansive index for all knowledge: "[Information]
has been extended far beyond our present ability to make real use of the record. A record, if it is to be useful to science, must be continuously extended, it must be stored...Our ineptitude in getting at the record is largely caused by the artificiality of the systems of indexing.
The human mind does not work this way. It operates by association."
A decades later, college students and electrical engineers attempted to make this kind of index a reality. They developed Archie (named for "archives"). The very first search engine was introduced in 1990, and was elaborated to search and store directory listings on file transfer protocol sites.
Other search engines began to follow, when Gopher (a site that indexed text files) was highlighted. It was then closely followed by Veronica and Jughead, both of which corresponded to Gopher. These early versions marked the way of the new era technologies utilized by main line search engines today.
Search Engines Development Timeline
The first tool created by Alan Emtage and L. Peter Deutsch for indexing, and is considered the first basic search engine. What began a school project at McGill University in Montreal, was an index that predated the world wide web. Gopher, released in 1991 by students from the University of Minnesota, was a protocol used to index and search for documents online as a form of anonymous FTP. Archie, Gopher and similar counterparts lost traction in the late 90's.
(1993) Lycos ( the dog )
It was created as a university project, but was the first to attain commercial search engine success. In 1999, Lycos (Go and grab the dog) was the most visited search engine in the world and was available in 40 countries. Now, it is comprised of a social network with email, web hosting and media entertainment pages.
Created by Brian Pinkerton WebCrawler was the first software that indexed complete pages online. AOL purchased WebCrawler using the technology for their network. When Excite purchased WebCrawler, AOL used Excite to run their program NetFind.
WebCrawler was one of the foundational search engines.
Steve Kirsch created Infoseek as a search engine and was bought by The Walt Disney Company in 1998. They merged it with Starwave and became go.com. Eventually, it was replaced by Yahoo.
An industry leader, was once the most popular search engines of its time. It differed from its contemporaries because of two factors: Alta Vista used a multi-threaded crawler (Scooter) that covered more webpages than people knew existed at the time.
It also had a well-organized search-running back-end advanced hardware. By 1996 AltaVista had become the sole search results provider for Yahoo. In 2003 Alta Vista was bought by Overture Services, Inc., which only months later was acquired by Yahoo
Founded originally as "Architext" by Stanford University students, Excite was launched officially having purchased two search engines (Magellan and WebCrawler), and signed exclusive agreements with Microsoft and Apple. Fellow Stanford students Larry Page and Sergey Brin once offered to sell their startup "Google" to the Excite team for $1 million. The offer was refused. Google became a $180 billion company and Excite is now used as a personal portal called My Excite.
A search engine also popular in the 90's. It was launched by Wired Magazine and is now owned by Lycos.
It's a search engine developed by Aaron Flin and shortly thereafter sold to Go2net. Today Dogpile fetches results from Google, Yahoo, and Yandex.
It started as research project by Stanford students Larry Page and Sergey Brin. They created a search engine that would rank websites based on the number of other websites that linked to that page. Prior to this, other engines have ranked sites based on the number of times the search term appeared on the webpage. This strategy developed the world's most successful search engine today.
It was the engine used by Microsoft, sourcing search results from Inktomi, and later from Looksmart. By 2006, Microsoft started performing their own image searches, and MSN became branded as Windows Live Search. Later Live Search and now Bing (2009) which was set to replace the Yahoo search engine.
(1998) MSN Search
It was the engine used by Microsoft, sourcing search results from Inktomi, and later Looksmart. By 2006, Microsoft started performing their own image searches; MSN became branded as Windows Live Search, then Live Search and finally Bing (2009) which was set to replace the Yahoo search engine.
It Began in 1994 out of FTP Search, from Norwegian University of Science and Technology. Then, it turned into Fast Search & Transfer or FAST. Alltheweb (1999) was said to have once rivaled Google, but the number of users declined when Overture bought the company in 2003.
(1999) AOL Search
AOL Search bought Web Crawler (one of the major crawler-based engines of it's time) in 1995. After a number of deals, purchases and exchanges, AOL relaunched their search engine, calling it AOL Search. Teaming with Google, the search engine relaunched in 2006 with newer features including video, search marketplace.
Meaning "expert" in Gaelic, Teoma was a search engine created by professor Apostolos Gerasoulis and Tao Yang at Rutgers University. Teoma's subject-specific technology centered on a link popularity algorithm which allowed pages to rank higher if other pages with a similar content and subject matter linked back to the page. Teoma was acquired by Ask Jeeves in 2001 and rebranded as Ask.com.
It was a crawler-based search engine that was introduced as a beta, and was owned by Looksmart. Initially the site was well-reputed with an unsullied database, and an automatic clustering of search results by using a technology called WiseGuide. Looksmart bought WiseNut in 2002 and was eventually closed in 2007.
Secure Search Engines
It is a metasearch engine that offers a proxy service for Ixquick and an email service that provides privacy protection called StartMail. It was relaunched in 2005 and included a re-engineered metasearch algorithm. Devoted to privacy, Ixquick entirely ended recording IP addresses and only used one cookie that is set to remember the user's search preferences for future searches. It is removed once a user does not return to the search engine homepage after 3 months. Ixquick was re-certified with Startpage in 2013.
(2006) DuckDuckGo ( The Duck )
It is a search engine that does not store or share any information about the user. It is unique to other search engines by providing all users the same results for a given search term, as well as providing search results from what they describe as the "best sources" rather than from the most sources. DuckDuckGo's results are sourced from places like Yahoo, Search Boss, Wikipedia, Wolfram Alpha, and their own web crawler the DuckDuckBot, etc
It is a secure search engine, meaning it pulls all the same results as Google, but uses the privacy protection of Ixquick that allows users to search with privacy.
Specialized Search Engines
(2009) Wolfram Alpha
It is a "computational knowledge engine" that answers factual queries by computing the answer from externally sourced "curated data" instead of listing relevant websites which could lead to the answer.
Major Non-US Search Engines
Originally standing for "Yet Another Indexer," Yandex is the largest search engine in Russia, and ranked as the 4th largest search engine in the world serving over 150 millions searches per day.
It is one of the main search engines in China, based on a special identification technology that classifies and groups articles. Baidu locates information, products, and services through Chinese language search terms (via phonetic Chinese), advanced searches, snapshots, spell checker, stock quotes, news, images, video, space information, weather, train and flight schedules and other local information. Baidu's greatest competitors are Google Hong Kong and Yahoo! China.
Local Search Engines
With its conventional name "Yellow Pages", Yelp began as an email service exchange recommending local business. Yelp now is connected to social networking sites and functions as a search engine where users can access reviews for companies/restaurants/businesses under a specific search/product. Yelp recently announced that it is now powering the Microsoft Bing local search engine results.
It is a location-based social networking search engine for mobile devices, utilizing a GPS hardware system where users can search for restaurants/entertainment, etc in their immediate locale and connect with others in the area.