The goal of this delivery was to implement the !crawl, !search and !wn_search commands. These commands essentially implement a small search system, where !crawl adds pages to the database, !search searches for terms within these pages, and !wn_search searches for similar words if the term used is not in the database. The problem this system aims to solve (web scraping and search) is relatively complex, making this stage of development significantly more challenging than previous ones. However, it has more interesting applications, as the implemented software can be used for database formation through web scraping and for inverted index searches and WordNet-based searches. These techniques are highly useful, especially given the important role data plays today (particularly with the advancement of AI technologies). Thus, knowing how to collect data, store it in a structured manner, and efficiently search through it is of utmost importance.
The main challenges faced during this stage were organizing the steps of creating the database, where it is necessary to first fetch the pages, then update the vectorizer, and consequently add or update the words in the database. The most frequently used sources were ChatGPT to assist with code architecture questions and specific library issues, class materials to set up the database, and the BeautifulSoup documentation for web scraping.