Make a Search Engine in PHP and MySQL

Why would you want to make a search engine anyway? There already is a search engine to rule them all. You can use Google to find just about anything in the Internet and I doubt you will ever have the same computing and storage capabilities as the big G.

So why then make your own search engine?

To make money of course!

… and to become famous as the creator of the next big search engine or because as a programmer or engineer you like challenges. Makeing a REAL search engine is not easy and if you’re like me you like challenges.

The third application is a customized, high speed site search for your large thousands of pages website. An indexed search engine will be a lot faster than a full text search function and if Google’s site search isn’t flexible enough for your site you can make your own search functionality.

SEARCH: THE BASICS

The basis of any BIG search engine is a word to web page index, basically a long list of words and how well they relate to different web pages.

To make a search engine you have to do four things:

• Decide what pages to fetch and fetch them
• Parse out words, phrases and links from the page
• Give a score to every keyword or key phrase indicating how well the phrase relates to that pages and store the scores in the search engine index
• Provide a way for users to query the index and get a list of matching web pages

This is not hard for a seasoned programmer. It can be done in a day if you know regular expressions and have some experience with HTML and databases.

Now you have a working search engine, just add a lot of computers and hard drives and you’ll soon index all of the Internet. If you’re not prepared to go that far a one terabyte disk will hold an index of about 50 million pages.

HOW TO SCORE PAGES

When you have implemente basic search functionality there’s still a lot to do before any real person will want to use your search engine.

An index is not enough. What’s challenging is how to score pages to give the end user the search results that’s most relevant to his idea of what hi is searching for.

You’ll need to decide how much weight to put on keywords in the tile tag, description and main web page contents. To make useful scores you will also want to boost keywords found in the URL of the page and in the anchor text of inbound links.

Keeping track of inbound links is the most useful and most challenging of the above, you’ll need to keep a separate database table with info on all links between pages you index.

WHAT TO INDEX AND NOT TO INDEX

Also you’ll find that after a while the real world content of the Internet will fill you index with junk like spam, affiliate pages, parked domains, work in progress homepages, link farms etc…

When indexing from the Internet you will have to find ways to filter out the junk content from what people are actually reading and searching for.

To start with you could limit how deep into sub directories you crawl, how many link hops from a domain index page you crawl and how many links per web page to allow.

PARSING WEBSITES

There’s a million ways, both right and wrong to write HTML and when you index from the Internet you will need to handle all of them.

When parsing pages you have to handle the complete HTML/XHTML standard (all versions of it) and also all the non-standard dialects that is used because web browsers tend to accept them.

To be able to read all pages you will also need to parse client side java script, handle frames, CSS and iframes.

This is a large part of the work on a general search engine, to be able to read all sorts of content.

WHY SO MANY URLS?

Finally you’ll need to deal with the fact that many websites have many URLS pointing to the same web page. Just look at this example:

dmoz.org
www.dmoz.org
dmoz.org/index.html
www.dmoz.org/index.html

All those URLs point to the same web page. If you don’t make special code to handle that you’ll soon have 4 results in your search engine (one for every URL) all going to the same page. Users will not like you.

There is also query strings containing a session ID. These will make almost unlimited URLs pointing to the same page.

google.com?SID=4434324325325
google.com?SID=4387483748377
google.com?SID=7654565644466

To the search engine there will be a really big number of pages all containing the same content.

The quick fix of course is to not index pages that include a query string. Or to strip the query string from pages. This works but will also remove a lot of legitimate content (think forums) from your index.

You now have all the information you need to make a site search engine. If you’re going for a general Internet search engine there’s a lot more details you need to include. Like robots.txt, site maps, redirects, proxies, recognizing content types, advanced ranking algorithms as well as handling terabytes of data.

This tutorial is based on the search engine technology powering the new search engine at SecretSearchEngineLabs.com

There will be more details in a future article, stay tuned. Good luck with your next search engine project.