KWoC 2018

- - -

- -

- - -

This repo contains the server code for Kharagpur Winter of Code 2018.

If you have access to the secrets repository, clone appending the --recursive option.

Used the following tutorial to set up the server - https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-16-04

Deployment

The server resides on the KWoC DO server and below are the instructions to update the website.

NOTE: NEVER merge upstream master directly on server; server has multiple files which are not on git but are very very crucial (such as users data).

cd kwoc
./update.sh

update.sh is a shell script (made by @thealphadollar) that shrinks the entire process to single event.

The same shell script can be added to cron for automatic updates at an interval of 30 minutes via adding the following lines in crontab. */30 * * * * /home/kwoc/kwoc/update.sh > home/kwoc/auto-update.log 2>&1

The cronjob is functional at the moment and logs for latest cron job can be found at /home/kwoc/auto-update.log

Development Workflow

To launch the server locally, follow the below steps:

For the first time

# Run these commands in the root folder of the project
pip install pipenv
pipenv shell --three
pipenv install --dev
gunicorn kwoc.app:app

If you want the server to update to code changes without restarting the server, use a --reload parameter.

gunicorn kwoc.app:app --reload  # live code changes

For subsequent usage

pipenv shell
gunicorn kwoc.app:app # --reload for live code changes

To generate projects for website

# download the csv file
# RUN tag_scrapper (see it's readme for more information)
cd /path/to/kwoc/gh_scraper
python3 project_gen.py

Open the projects webpage and check :)

Generating statistics for students

# keep updated gh_scraper/projects.csv and gh_login/student.csv
cd gh_scraper/stats
# change since (line 160) and until (line 161)
python3 generate_statistics.py

NOTE: The below command is already added in the update.sh script

To add cronjob :

crontab -e          # A file will open, type the following
*/60 * * * * /usr/bin/python3 /home/kwoc/kwoc/gh_scraper/stats/generate_statistics.py > /home/kwoc/last_gen_stats.log 2>&1

Workflow for updating the projects list

• Add the projects details to /path/to/kwoc/gh_scrapper/projects.csv. Make sure that you dont remove the column headings in the first line.

• Add the GITHUB_TOKEN for authenticating API to your environment.

• Go to /path/to/kwoc/tag_scrapper/ and run the command python3 main.py to generate tags

• Go to /path/to/kwoc/gh_scrapper/ and run the command python3 project_gen.py to generate the desired template

Update /mid-term form

There is an option box for choosing mentor which uses list_of_mentors.json. To populate the JSON:

$ cd /path/to/kwoc
$ python3 populate_mentor_list.py

This script also updates the secrets/mentor_unique_ids.json file which is used to map an encrypted url endpoint to each mentor so that they can fill in their mentor evaluations.

The code at the end should be mailed to them once all students have filled in their mid-term/end-term evaluations.

After mentor fills the form

Once the mentor has filled the form, firstly we take a note of it and block the particular id so that the mentor cannot fill the form again.

We store the list of students he passes and fails in secrets/pass.txt and secrets/fail.txt in the following format:

github_username PASS/FAIL

We also store his elaborate response in a google sheet.

Update /students.csv - Remove Mentors and Duplicates

Run the following commands once, while deploying in the server to remove mentors and duplicate entries from gh_login/student.csv -

$ cd /path/to/kwoc
$ python3 rm_mentors_from_students.py

Add colleges to colleges.json

There is a <datalist> in /student_registration form which uses colleges.json. Due to the request of students, colleges need to be added manually. But, it's not recommended to edit JSON files manually as a simple syntax error will make the server down. To automate this process:

1. Add colleges to /path/to/colleges_to_add.txt in each line.

$ cd /path/to/kwoc
$ python3 add_colleges.py

Sending Mass Mails

We use Sendgrid to send mass mails. A script which uses the sendgrid API is present inside and following are the steps to mails to all.

# set environment variable SENDGRID_KEY with the secret sendgrid key
cd /path/to/kwoc/

You can skip the below two steps if you have already setup the pipenv environment.

pipenv shell
pipenv install --dev

Below steps are compulsory.

cd kwoc
# Go into the file and follow the instructions about putting emailids in csv file,
# setting your content, subject and from mail.
python3 sendgrid_mail.py

NOTE:

• Contact seniors to get the sendgrid key.
• If the script gets stuck somewhere, wait for it to resume. If you decide to restart, please remove the entries till which mail has been sent, and re-run the script. It happens frequenty.
• Sample text and csv file is provided in the folder; strictly follow them.
• Content is to be written in HTML (again see sample content).

Server Folder Backup

All github and non-github data (such as student.csv) are present in /home/kwoc/kwoc on server. In a case when this folder is damaged or modified in a way that it becomes futile for use, we are keeping a backup of the same in /home/kwoc/kwoc-backup. The former is the master folder and the latter is the slave and are kepy in sync via the below cron script.

25 */2 * * * /usr/bin/rsync -av --delete /home/kwoc/kwoc /home/kwoc/kwoc-backup > /home/kwoc/last-backup.log 2>&1