Search Engines

A search engine, also known as a web search engine, is a system software that discovers relevant contents on the internet based on given queries. It is one of the most widely used softwares all over the world.

History

WHOis was the first domain search engine with limited capabilities invented in 1982. Alan Emtage, together with Bill Heelan and Peter Deutsch developed a content search engine named Archie that could make a directory list of files on FTP sites and produced a searchable database. The first full-fledged web search engine was W3Catalog developed at the University of Geneva in 1993.

A search engine has three steps to implement to produce a result. These three steps are detailed below:

1. Crawling: AKA spidering, is a process in which an automated script or computer program is used to crawl through the web for specific
   content. This helps in producing an index of relevant contents. A crawler is also sometimes called a Spiderbot. It can detect HTML errors and broken links.
   Robots.txt file contains data on how to crawl or if not to crawl the website.
2. Indexing: After a web crawler crawls through the web, an index of related sites is produced and displayed in paging. Keywords and metadata
   are specifically searched to index webpages. Formats that are supported by search engines to index are:
   ASCII text file
   CAB
   HTML
   PDF
   MS Word, Excel & PowerPoint
   Lotus Notes
   RAR
   TAR
   XML
   ZIP etc.
3. Ranking: Finally, relevant sitse are given ranks based on factors such as quality, uniqueness, mobile optimized , accessability, backlinks, loading speed etc. Google Search Console
   helps to find out if a particular website is properly accessible.

Next, webpages are displayed as what is called SERP (Search Engine Result Pages). This is published in paging depending on the browser’s configuration.

Custom Search Engines

A custom search engine helps to view specific results as required by the users. Currently, these are the two popularly active custom search engines:

1. Google Custom Search Engine: Developed at Google in 2006, it provides a beautiful way of customising searches, refine and view as expected by a user.
   Official Website: cse.google.com
2. Bing Custom Search: Developed by Microsoft Corporation, it has a powerful API to refine search results and show the user. It is totally free to use at any scale with no ads.
   Official Website: http://www.customsearch.ai

Popular Search Engines

As of today, there are many vendors who provide search engine services. Google’s search engine has the largest market share worldwide on all the platforms. However, Yandex leads the market in Russia whereas Baidu tops in China.

Writing a Search Engine

Since a search engine is a multi-level complex software to make, its internals can be divided into three layers:

1. Frontend: JavaScript
2. Middle Layer: General-purpose languages (ex. Java, Python, PHP)
3. Backend: C++, C, Rust, GO

Conclusion

Search engines are a crucial part of today’s technology. The more relevant and neutral contents users get, the more they benefit.