Design and implementation of a Scraping system for Sport News

Resumen

Nowadays, the way people read print media has changed. The sale of newspapers in their printed format has fallen by around 60% in the last decade. This has occurred as a result
of the invention of the internet and the capability of accessing content from thousands of sites through the World Wide Web.
Based on this information we have completed this project of design and implementation of a system of extracting news on the Web for further analysis. The thematic of the project
could be diverse, but it has been decided to focus on sports news, in particular news about the most popular football teams in Spain.
The project is divided into several phases related to the processing of information. For the first three phases it has been used a workflow between them using Luigi. First of those phases is the extraction of data from the Web and to implement it several web spiders have been developed using Scrapy. Subsequently, this information has been analyzed using Senpy to extract feelings and emotions in each article. The last step in the work ow is to store
obtained data in Elasticsearch.
The last phase of the project has been to create a graphical interface in order to display obtained data in the analysis and being able to compare them. To do this, Polymer Web Components has been used in conjunction with D3.js library. Using these technologies, dfferent widgets have been created in order to compare results between dfferent teams and newspapers.
This project will allow the user to perform a complete analysis of the dfferent newspapers along with the diferent football teams thanks to the knowledge of emotions and feelings that are generated in the written press over time.