Filipe Lourenco : Debug crawler locally

Required Github repositories:

• https://github.com/Monsido/crawler

How to get your testing setup prepared

Intro

In order to be able to test the application locally, at the moment it is necessary to download the actual application onto our machine and run it.

Tools necessary for install locally the application:

• Download and install https://www.jetbrains.com/idea/download/#section=mac (Community Edition)

• Install Homebrew https://brew.sh/

• Download and install Java SDK 11 https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html

• Install Maven: https://maven.apache.org/download.cgi

In case of something going wrong use: ‘Ctrl + c’: stop application when its running in the terminal

Initial setup

• Link your account on the github in the IntelliJ
  IntelliJ > Preferences > Version Control > GitHub

• Clone the repo and then add maven framework support in the project and set JDK 11 as SDK

• Execute mvn install:install-file -pl common -pl controller -pl page_parser -pl worker (jutf7 jar will be downloaded and put at common/lib, you also need access to iputils repo)

• Add github token on the .m2/settings.xml

  • <server>
<id>iputils</id>
<username>yourGitHubUserName</username>
<password>pasteTokenGeneratedFromGitHubHere</password>
</server>

• Execute mvn -U clean install -DskipTests