Skip to content

Node.js powered asynchronous scraper for reddit, scrapes imgur and gfycat images.

Notifications You must be signed in to change notification settings

mikedevita/node-reddit-scraper

Repository files navigation

node-reddit-scraper Build Status Join the chat at https://gitter.im/mikedevita/node-reddit-scraper

a node.js powered asynchronous scraper for reddit and imgur images.

Supports multiple subreddits in the config file. At the present moment only supports images from imgur.


Installation

Installation is straight forward

git clone https://github.com/mikedevita/node-reddit-scraper
cd node-reddit-scraper
cp config.example.js config.js
npm install

Imgur v3 api requires a client-ID

Go here https://imgur.com/account/settings/apps and create a new app. Paste the Client-ID and Client-Secret into the config.js.

auth: {
  clientId: 'foo',
  clientSecret: 'bar'
},

Usage

This app is designed to be run via crontab or based on some sort of schedule.

Edit the config.js file to include your subreddits, use the example as a guidelines.

To invoke, just run npm run or node index.js.

You can use this on a crontab like so..

# run every day at midnight
0	0	*	*	*	node /path/to/node-reddit-scraper/index.js

Contributing

pull requests are welcomed, just please adhere to the .eslintrc and .editorconfig's provided. Also run tests and make sure they pass fully.

To run tests just use npm test to execute the tests.

Note Due to how long reddit takes to respond the Reddit unit tests might fail due to a timeout error, if thats the case re run it or skip that single test.

About

Node.js powered asynchronous scraper for reddit, scrapes imgur and gfycat images.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published