MyCrawler

Web Crawler

Building

This is a Maven project. To build it, run the following command:

mvn clean package

Configuring

Edit the mycrawler_aem_<xx>.yml file as needed.

Running

From the project-folder:

java -jar target/MyCrawler-0.0.1-SNAPSHOT.jar mycrawler_aem_<xx>.yml

or

./run_crawler.sh

Frameworks

The 'crawler' is built on several popular Java frameworks:

• Crawler4j
• SnakeYAML
• GSON