A domain specific query language for the scraping, modification and graphical representation of web data

The purpose of this project was to create a way to reduce the complexity and implementation time of coding custom web scraping classes from scratch. This was done by providing a layer of abstraction over Scrapy spider classes in the form of a domain specific language which followed a syntax similar to that of SQL. Data manipulation and graphical representation functionality was also implemented.

A high level overview of the program:

Sequence diagram:

Class diagram:

Backus-Naur language definition: