update from Gluejar

.gitignore

@@ -6,14 +6,17 @@ settings/keys/*
 *.dot
 reports
 ENV
-venv
+.env
 .DS_Store
 build
 deploy/last-update
 logs/*
-cache/*
 celerybeat.pid
 celerybeat-schedule
 .gitignore~
-static/scss/**/*.css.map
-*.retry
+assets/*
+*.ipynb
+dump.rdb
+Pipfile.lock
+
+*.css.map

.python-version

@@ -0,0 +1 @@
+3.9.11

Pipfile

@@ -0,0 +1,43 @@
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[packages]
+celery = "==4.4.7"
+django = "==1.11.29"
+django-contrib-comments = "==2.0.0"
+django-extensions = "==3.1.1"
+django-registration = "==2.4.1"
+social-auth-app-django = "==2.1.0"
+django-tastypie = "==0.14.1"
+django-el-pagination = "==3.2.4"
+django-selectable = "==1.1.0"
+django-notification = {editable = true, ref = "1ad2be4adf3551a3471d923380368341452e178a", git = "git+https://github.com/eshellman/django-notification.git"}
+django-email-change = {editable = true, ref = "fb063296cbf4e4a6d8a93d34d98fe0c7739c2e0d", git = "git+https://github.com/eshellman/django-email-change.git"}
+django-ckeditor = "==5.6.1"
+django-storages = "==1.5.2"
+sorl-thumbnail = "==12.6.3"
+django-mptt = "==0.8.6"
+pyepub = "==0.5.0"
+django-sass-processor = "==0.8.2"
+mysqlclient = "==1.4.6"
+mailchimp3 = "==3.0.14"
+boto3 = "==1.17.91"
+pymarc = "==4.2.1"
+beautifulsoup4 = "==4.11.1"
+gitberg = "==0.8.7"
+risparser = "==0.4.3"
+pyoai = "==2.5.0"
+django-jsonfield = "==1.0.0"
+mechanize = "==0.4.5"
+stripe = "==2.76.0"
+selenium = "==3.141.0"
+requests-mock = "==1.8.0"
+redis = "==3.5.3"
+xhtml2pdf = ">=0.2.15"
+pillow = "==9.5.0"
+pypdf = ">=5.0.0"
+
+[requires]
+python_version = "3.9"

README.md

@@ -1,7 +1,7 @@
 regluit - "The Unglue.it web application and website"
 =======
 
-This repo - https://github.com/EbookFoundation/regluit will be the place for collaborative development for Unglue.it. Add issues and submit pull requests here. As of January 19, 2017, https://github.com/Gluejar/regluit is still being used for production builds.
+Another repo - https://github.com/EbookFoundation/regluit will eventually be the place for collaborative development for Unglue.it. Add issues and submit pull requests there. As of September 1, 2019, https://github.com/Gluejar/regluit is still being used for production builds.
 
 The first version of the unglue.it codebase was a services-oriented project named "unglu".
 We decided that "unglu" was too complicated, so we started over and named the new project "regluit".
@@ -10,49 +10,24 @@ contains four main applications: `core`, `frontend`, `api` and `payment` that ca
 and configured on as many ec2 instances that are needed to support traffic.
 The partitioning between these modules is not as clean as would be ideal. `payment` is particularly messy because we had to retool it twice because we had to switch from Paypal to Amazon Payments to Stripe.
 
-regluit was originally developed on Django 1.3 (python 2.7) and currently runs on Django 1.8.
+regluit was originally developed on Django 1.3 (python 2.7) and currently runs on Django 1.11 Python 3.8).
 
-Development (Vagrant + Virtualbox)
--------
-
-The recommended method for local development is to create a virtual machine with [Vagrant](https://www.vagrantup.com/) and [Virtualbox](https://www.virtualbox.org/wiki/Downloads).
-With this method, the only requirements on the host machine are `virtualbox` and `vagrant`.
-Vagrant will use the `ansible-local` provisioner, therefore installing python and ansible on the host machine is not necessary.
-
-__Instructions for Ubuntu 16:__
-1. Install virtualbox: `sudo apt-get install virtualbox`
-2. Install vagrant: `sudo apt-get install vagrant`
-3. Clone the `EbookFoundation/regluit` repository.
-4. Navigate to the base directory of the cloned repo (where `Vagrantfile` is located).
-5. Run `vagrant up` to create the VM, install dependencies, and start necessary services.
-  * Note: This step may take up to 15 minutes to complete.
-6. Once the VM has been created, run `vagrant ssh` to log in to the virtual machine you just created. If provisioning was successful, you should see a success message upon login.
-  * If virtualenv doesn't activate upon login, you can do it manually by running `cd /opt/regluit && source venv/bin/activate`
-7. Within the VM, run `./manage.py runserver 0.0.0.0:8000` to start the Django development server.
-8. On your host machine, open your web browser of choice and navigate to `http://127.0.0.1:8000`
-
-__Instructions for other platforms (Windows/OSX):__
-* Steps are essentially the same, except for the installation of Vagrant and Virtualbox. Refer to each package's documentation for specific installation instructions.
-
-_NOTE:_ If running Windows on your host machine, ensure you are running `vagrant up` from an elevated command prompt, e.g. right click on Command Prompt -> Run As Administrator. 
-
-
-Development (Host Machine)
+Develop
 -------
 
 Here are some instructions for setting up regluit for development on
-an Ubuntu system. If you are on OS X see notes below
-to install python-setuptools in step 1:
+an Ubuntu system. If you are on OS X see notes below.
 
-1. Ensure MySQL and Redis are installed & running on your system.
+
+- Ensure MySQL 5.7 and Redis are installed & running on your system.
 1. Create a MySQL database and user for unglueit.
 1. `sudo apt-get upgrade gcc`
-1. `sudo apt-get install python-setuptools git python-lxml build-essential libssl-dev libffi-dev python2.7-dev libxml2-dev libxslt-dev libmysqlclient-dev`
+1. `sudo apt-get install python-setuptools git python-lxml build-essential libssl-dev libffi-dev python3.8-dev libxml2-dev libxslt-dev libmysqlclient-dev`
 1. `sudo easy_install virtualenv virtualenvwrapper`
 1. `git clone [email protected]:Gluejar/regluit.git`
 1. `cd regluit`
 1. `mkvirtualenv regluit`
-1. `pip install -r requirements_versioned.pip`
+1. `pip install -r requirements.txt`
 1. `add2virtualenv ..`
 1. `cp settings/dev.py settings/me.py`
 1. `mkdir settings/keys/`
@@ -63,8 +38,9 @@ to install python-setuptools in step 1:
 1. `deactivate ; workon regluit`
 1. `django-admin.py migrate --noinput`
 1. `django-admin.py loaddata core/fixtures/initial_data.json core/fixtures/bookloader.json` populate database with test data to run properly.
-1. `django-admin.py celeryd --loglevel=INFO` start the celery daemon to perform asynchronous tasks like adding related editions, and display logging information in the foreground.
-1. `django-admin.py celerybeat -l INFO` to start the celerybeat daemon to handle scheduled tasks.
+1. `redis-server` to start the task broker
+1. `celery -A regluit worker --loglevel=INFO ` start the celery daemon to perform asynchronous tasks like adding related editions, and display logging information in the foreground. Add ` --logfile=logs/celery.log` if you want the logs to go into a log file.
+1. `celery -A regluit beat --loglevel=INFO` to start the celerybeat daemon to handle scheduled tasks. 
 1. `django-admin.py runserver 0.0.0.0:8000` (you can change the port number from the default value of 8000)
 1. make sure a [redis server](https://redis.io/topics/quickstart) is running
 1. Point your browser to http://localhost:8000/
@@ -77,121 +53,40 @@ CSS development
 Production Deployment
 ---------------------
 
-OBSOLETE
-Below are the steps for getting regluit running on EC2 with Apache and mod_wsgi, and talking to an Amazon Relational Data Store instance.
-Instructions for setting please are slightly different.
-
-1. create an ubuntu ec2 instance (e.g, go http://alestic.com/ to find various ubuntu images)
-1. `sudo aptitude update`
-1. `sudo aptitude upgrade`
-1. `sudo aptitude install git-core apache libapache2-mod-wsgi mysql-client python-virtualenv python-mysqldb redis-server python-lxml postfix python-dev libmysqlclient-dev`
-1. `sudo mkdir /opt/regluit`
-1. `sudo chown ubuntu:ubuntu /opt/regluit`
-1. `cd /opt`
-1. `git config --global user.name "Raymond Yee"`
-1. `git config --global user.email "[email protected]"`
-1. `ssh-keygen`
-1. add `~/.ssh/id\_rsa.pub` as a deploy key on github https://github.com/Gluejar/regluit/admin/keys
-1. `git clone [email protected]:Gluejar/regluit.git`
-1. `cd /opt/regluit`
-1. create an Amazon RDS instance
-1. connect to it, e.g. `mysql -u root -h gluejardb.cboagmr25pjs.us-east-1.rds.amazonaws.com -p`
-1. `CREATE DATABASE unglueit CHARSET utf8;`
-1. `GRANT ALL ON unglueit.\* TO ‘unglueit’@’ip-10-244-250-168.ec2.internal’ IDENTIFIED BY 'unglueit' REQUIRE SSL;`
-1. update settings/prod.py with database credentials
-1. `virtualenv ENV`
-1. `source ENV/bin/activate`
-1. `pip install -r requirements_versioned.pip`
-1. `echo "/opt/" > ENV/lib/python2.7/site-packages/regluit.pth`
-1. `django-admin.py syncdb --migrate --settings regluit.settings.prod`
-1. `sudo mkdir /var/www/static`
-1. `sudo chown ubuntu:ubuntu /var/www/static`
-1. `django-admin.py collectstatic --settings regluit.settings.prod`
-1. `sudo ln -s /opt/regluit/deploy/regluit.conf /etc/apache2/sites-available/regluit`
-1. `sudo a2ensite regluit`
-1. `sudo a2enmod ssl rewrite`
-1. `cd /home/ubuntu`
-1. copy SSL server key to `/etc/ssl/private/server.key`
-1. copy SSL certificate to `/etc/ssl/certs/server.crt`
-1. `sudo /etc/init.d/apache2 restart`
-1. `sudo adduser --no-create-home celery --disabled-password --disabled-login` (just enter return for all?)
-1. `sudo cp deploy/celeryd /etc/init.d/celeryd`
-1. `sudo chmod 755 /etc/init.d/celeryd`
-1. `sudo cp deploy/celeryd.conf /etc/default/celeryd`
-1. `sudo mkdir /var/log/celery`
-1. `sudo mkdir /var/run/celery`
-1. `sudo chown celery:celery /var/log/celery /var/run/celery`
-1. `sudo /etc/init.d/celeryd start`
-1. `sudo cp deploy/celerybeat /etc/init.d/celerybeat`
-1. `sudo chmod 755 /etc/init.d/celerybeat`
-1. `sudo cp deploy/celerybeat.conf /etc/default/celerybeat`
-1. `sudo mkdir /var/log/celerybeat`
-1. `sudo chown celery:celery /var/log/celerybeat`
-1. `sudo /etc/init.d/celerybeat start`
-
-## setup to enable ckeditor to work properly
-
-1. `mkdir /var/www/static/media/`
-1. `sudo chown ubuntu:www-data /var/www/static/media/`
-
-
-Updating Production
---------------------
-
-1. Study the latest changes in the master branch, especially keep in mind how
-it has [changed from what's in production](https://github.com/Gluejar/regluit/compare/production...master).
-1. Update the production branch accordingly.  If everything in `master` is ready to be moved into `production`, you can just merge `master` into `production`. Otherwise, you can grab specific parts.  (How to do so is something that should probably be described in greater detail.)
-1. Login to unglue.it and run [`/opt/regluit/deploy/update-prod`](https://github.com/Gluejar/regluit/blob/master/deploy/update-prod)
-
+See http://github.com/EbookFoundation/regluit-provisioning
 
 OS X Developer Notes
 -------------------
 
 To run regluit on OS X you should have XCode installed
 
-Install virtualenvwrapper according
-to the process at http://blog.praveengollakota.com/47430655:
+Install MySQL:
+    `brew install [email protected]`    
+    `mysql_secure_installation`
+    `mysqld_safe --user=root -p`
+
 
-1. `sudo easy\_install pip`
-1. `sudo pip install virtualenv`
-1. `pip install virtualenvwrapper`
+We use pyenv and pipenv to set up an environment.
 
 Edit or create .bashrc in ~ to enable virtualenvwrapper commands:
-1. `mkdir ~/.virtualenvs`
-1. Edit .bashrc to include the following lines:
-
-    export WORKON_HOME=$HOME/.virtualenvs
-    source your_path_to_virtualenvwrapper.sh_here
 
-In the above web site, the path to virtualenvwrapper.sh was
-/Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh
-In Snow Leopard, this may be /usr/local/bin/virtualenvwrapper.sh
+1. `pipenv install -r requirements.txt`
+1. Edit .zshrc to include the following lines:
 
-Configure Terminal to automatically notice this at startup:
-Terminal –> Preferences –> Settings –> Shell
-Click "run command"; add `source ~/.bashrc`
+    `eval "$(pyenv init -)"`
+    `export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/10/bin`
+    `export PATH=$PATH:/usr/local/opt/mysql-client/bin:$PATH`
+    `export ANSIBLE_VAULT_PASSWORD_FILE=PATH_TO_VAULT_PASSWORD`
 
-If you get 'EnvironmentError: mysql_config not found'
-edit the line ~/.virtualenvs/regluit/build/MySQL-python/setup_posix.py
-1. mysql_config.path = "mysql_config"
-to be (using a path that exists on your system)
-1. mysql_config.path = "/usr/local/mysql-5.5.20-osx10.6-x86_64/bin/mysql_config"
+If you get `EnvironmentError: mysql_config not found`
+you might need to set a path to mysqlconfig
 
 You may need to set utf8 in /etc/my.cnf
 collation-server = utf8_unicode_ci
 
     init-connect='SET NAMES utf8'
     character-set-server = utf8
 
-Selenium Install
----------------
-
-Download the selenium server:
-http://selenium.googlecode.com/files/selenium-server-standalone-2.5.0.jar
-
-Start the selenium server:
-'java -jar selenium-server-standalone-2.5.0.jar'
-
 MARC Records
 ------------
 
@@ -232,7 +127,12 @@ MARC Records
     * if you have records with both DIRECT and UNGLUE links, you'll need two MARCRecord instances
     * if you have both kinds of link, put them in _separate_ records, as marc_format can only take one value
 
+MySQL Migration
+---------------
+
+## 5.7 - 8.0 Notes
 
-# vagrant / ansible
+* Many migration blockers were removed by by dumping, then restoring the database.
+* After that, RDS was able to migrate
+* needed to create the unglueit user from the mysql client
 
-[How to build machines using Vagrant/ansible](docs/vagrant_ansible.md)

__init__.py

@@ -0,0 +1,8 @@
+from __future__ import absolute_import, unicode_literals
+
+# This will make sure the app is always imported when
+# Django starts so that shared_task will use this app.
+from .celery_module import app as celery_app
+
+__all__ = ('celery_app',)
+