Warning

This project was completed as part of COMP1405z and COMP1406z final projects, as a result code cannot be displayed publicly due to the possibility of academic integrity violations. Contact Ariz Kazani or Kishan Rajagunathas for access.

Web Crawler & Search Engine

Introduction

This project implements a web crawler and search engine in Java (with a version also in python that does not have a GUI). The crawler navigates the web starting from a seed URL, collecting data from various web pages, and storing the information necessary for search and ranking purposes. The search engine component processes user queries, providing relevant search results based on the collected data using various PageRank algorithms.

Installation

This step slightly varies between the two versions.

• Python

  1. Simply get the latest version of python

  2. Run a crawl to get searchable data.

  3. Create a method in a python file that accepts user queries and displays the results (the search engine will return top 10 results).

• Java

  1. Download Java and JavaFX.

  2. Configure JavaFX, to be in the path of the project.

  3. Run a crawl to get searchable data.

  4. CrawlerApp.java will accept user queries.

Usage & Features

1. Simple and interactive GUI (for java version).

2. Fast and accurate queries guaranteed with various mathematical models and algorithms used.

3. Efficient data storage.

License

All Rights Reserved

This project and all the associated files are the intellectual property of Ariz Kazani & Kishan Rajagunathas. Unauthorized copying, distribution, modification, or use of this work without express written permission from the owner is strictly prohibited.

Credits

Web Crawler & Search Engine components written by Ariz Kazani

GUI & Documentation written by Kishan Rajagunathas