First public release (v1.6)

Installation_Guide.md

@@ -0,0 +1,219 @@
+# LGR Toolset Web Application
+
+This application aims to be an easy-to-use editing and authoring tool for LGR.
+
+## Conventions
+
+In this document, commands executed as a user are prefixed with `$`, whereas
+commands executed as root are prefixed with `#`. Some commands will need to be
+executed in a virtualenv (see below), and will be prefixed with `(venv)$`.
+
+This document was written for CentOS 7, 64 bits, **with SELinux disabled**.
+To disable SELinux:
+
+	# setenforce 0
+
+and make sure that `SELINUX` is set to `disabled` in `/etc/selinux/config`.
+
+It should however work for other Linux distributions with some minor tweaks.
+
+## Installation
+
+### System configuration
+
+Install the nginx webserver used to serve as a frontend. On CentOS, you need to
+enable the EPEL repository before:
+
+	# yum install epel-release
+	# yum install nginx
+
+Install virtualenv, used to create the Python environment:
+
+	# yum install python-virtualenv
+
+Install the dependencies needed to build the lxml module:
+
+	# yum install make gcc libxml2-devel libxslt-devel
+
+Create a new user/group that will be used to run the application, as well as the
+log directory where log files will be placed:
+
+	# mkdir /var/www
+	# useradd lgr -d /var/www/lgr
+	# chmod 765 /var/www/lgr
+	# mkdir -p /var/log/lgr/
+	# chown lgr:lgr /var/log/lgr
+
+Copy the archives for all provided dependencies (`picu`, `munidata`, `lgr-core`
+and `lgr-django`) in a specific directory, and extract web-application:
+
+	# su -l lgr
+	$ mkdir /var/www/lgr/packages
+	$ cp picu-*.tar.gz munidata-*.tar.gz lgr-core-*.tar.gz lgr-django-*.tar.gz /var/www/lgr/packages
+	$ mkdir /var/www/lgr/lgr-django
+	$ tar xvf /var/www/lgr/packages/lgr-django-*.tar.gz -C /var/www/lgr/lgr-django --strip-components=1
+
+Create and configure the virtualenv:
+
+	$ virtualenv-2.7 /var/www/lgr/venv
+	$ source /var/www/lgr/venv/bin/activate
+	(venv)$ cd /var/www/lgr/lgr-django
+	(venv)$ pip install -r etc/requirements.txt -f /var/www/lgr/packages
+
+Also install the application server:
+
+	(venv)$ pip install gunicorn
+
+Note: you may see the following error during the installation:
+
+	File "/var/www/lgr/venv/lib/python2.7/site-packages/gunicorn/workers/_gaiohttp.py", line 68
+	    yield from self.wsgi.close()
+	             ^
+	SyntaxError: invalid syntax
+
+The error can be ignored, and gunicorn will still be installed sucessfully.
+See [https://github.com/benoitc/gunicorn/issues/788](https://github.com/benoitc/gunicorn/issues/788).
+
+
+## ICU installation
+
+If your distribution ships ICU, install it using the package manager.
+Otherwise, you can [download the ICU library](http://site.icu-project.org/download/)
+in various formats (source, binary packages for some distributions).
+Note: You only need to install the library files (`.so.XX`) and may install other
+files at your discretion.
+
+For example, to install ICU 52 (Unicode 6.3), as root:
+
+	# mkdir /root/icu
+	# cd /root/icu
+	# yum install wget
+	# wget http://download.icu-project.org/files/icu4c/52.1/icu4c-52_1-RHEL6-x64.tgz
+	# tar xvf icu4c-52_1-RHEL6-x64.tgz -C / --wildcards usr/local/lib/*.52*
+
+The current supported ICU versions (and their corresponding Unicode version) are:
+
+* ICU 52 (Unicode 6.3)
+* ICU 50 (Unicode 6.2)
+* ICU 49 (Unicode 6.1)
+* ICU 46 (Unicode 6.0)
+* ICU 44 (Unicode 5.2)
+
+Once you have installed the ICU library, please make sure that the location of
+the `.so` files is in the `$LD_LIBRARY_PATH` environment variable.
+If not, you can adjust the configuration system-wise (if you are using the
+binary packages from ICU website):
+
+	# echo "/usr/local/lib" > /etc/ld.so.conf.d/lgr.conf
+	# ldconfig
+
+## Django Configuration
+
+A template for production configuration is provided:
+
+	# su -l lgr
+	$ source /var/www/lgr/venv/bin/activate
+	(venv )$ cp /var/www/lgr/lgr-django/src/lgr_web/settings/deploy.py.template /var/www/lgr/lgr-django/src/lgr_web/settings/local.py
+
+Please configure the `ALLOWED_HOSTS` array with the hostname serving the LGR
+application. For example, if your application is available at
+`lgr-editor.example.com`, then the configuration should be `ALLOWED_HOSTS = ['lgr-editor.example.com']`.
+Also configure `SUPPORTED_UNICODE_VERSIONS` as well as `UNICODE_DATABASES` according to your ICU installation.
+
+The default database, used to store sessions, is `sqlite`.
+It is possible to configure the use of another server using the `DATABASES` dictionary.
+
+For some tools, computation is performed in background and results are sent by e-mail.
+Please edit the e-mail parameters if necessary, in order to be able to send e-mails correctly, mainly
+`EMAIL_HOST` that should contain the server address, `EMAIL_PORT` that should contain the server port,
+`EMAIL_HOST_USER`, `EMAIL_HOST_PASSWORD`, `EMAIL_USE_TLS` and/or `EMAIL_USE_SSL`
+if your server needs such configuration and `DEFAULT_FROM_EMAIL` in order to
+correctly set the e-mail sender address.
+
+You can also set-up the broker used to pass messages between the interface
+and the background task processor by using the `BROKER_URL` key.
+It is also possible to set the timeout for background tasks using `CELERYD_TASK_SOFT_TIME_LIMIT`.
+
+The `REPERTOIRE_STORAGE_LOCATION` setting can be adjusted to point to the location of
+the repertoire LGR files eg. MSR. Likewise the `LGR_STORAGE_LOCATION` setting is
+the location of the built-in LGR you want to provide as an example.
+Finally, the `LGR_RNG_FILE` points to the RNG schema file for XML LGR files.
+If you want to modify these settings, please make sure that the `lgr` user has
+read-access to the new locations.
+
+If the application is not served in HTTPS, then the two variables `SESSION_COOKIE_SECURE`
+and `CSRF_COOKIE_SECURE` need to be set to `False`.
+
+Before starting the django application for the first time, you need to run the
+following commands to set the environment up:
+
+	(venv)$ cd /var/www/lgr/lgr-django
+	(venv)$ ./manage.py migrate
+	(venv)$ ./manage.py collectstatic
+
+You also have to setup a periodic clean-up of the sessions and storage. Example for a daily
+run at midnight (to adapt according to your cron scheduler):
+
+	(venv)$ crontab -e
+	0 0 * * * /var/www/lgr/venv/bin/python /var/www/lgr/lgr-django/manage.py cleanstorage
+
+You can adjust the session and storage duration with `SESSION_COOKIE_AGE` and `STORAGE_DURATION`
+settings respectively. Ensure that `SESSION_COOKIE_AGE` is at least two times `STORAGE_DURATION`,
+or there may be leftover directories.
+
+If you do not want to clean storage, replace the `cleanstorage` command by `clearsessions`.
+
+## Serving the application
+
+To serve the application, the following architecture in used:
+
+* The celery application is used to launch background tasks.
+* The gunicorn application server is used to run the python code.
+* The nginx webserver will act as a frontend proxy as well as serving static content.
+* Requests related to the application are relayed to the gunicorn server.
+
+### Celery configuration
+
+The celery process will be launched by the systemd service manager.
+
+Copy the provided service file to systemd's configuration directory:
+
+	# cp /var/www/lgr/lgr-django/etc/systemd/lgr-celery.service /etc/systemd/system/
+
+The process will be automatically started with the gunicorn service so there is nothing more to do.
+
+### Gunicorn configuration
+
+The gunicorn process will be launched by the systemd service manager.
+
+Copy the provided service file to systemd's configuration directory:
+
+	# cp /var/www/lgr/lgr-django/etc/systemd/lgr-django.service /etc/systemd/system/
+
+Also copy the temporary files configuration and create them:
+
+	# cp /var/www/lgr/lgr-django/etc/systemd/lgr-django.conf /etc/tmpfiles.d/
+	# systemd-tmpfiles --create
+
+Reload systemd daemon and enable and start the gunicorn process:
+
+	# systemctl daemon-reload
+	# systemctl enable lgr-django
+	# systemctl start lgr-django
+
+### nginx configuration
+
+Use the provided configuration file (in `etc/nginx/lgr-django.conf`) as a template to
+configure the served host:
+
+	# cp /var/www/lgr/lgr-django/etc/nginx/lgr-django.conf /etc/nginx/conf.d/nginx-django.conf
+
+Edit the file and set the `server_name` value to the name of the host.
+
+Enable nginx daemon and start it:
+
+	# systemctl enable nginx
+	# systemctl start nginx
+
+Do not forget to configure your firewall to allow access on port 80 (or 443 if
+you configure HTTPS) and output on port 25 (or any port configured for e-mail server).

LICENSE

@@ -0,0 +1,28 @@
+Copyright (c) 2015-2016 Internet Corporation for Assigned Names and
+Numbers (“ICANN”). All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+
+    * Neither the name of the ICANN nor the names of its contributors
+      may be used to endorse or promote products derived from this
+      software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY ICANN AND CONTRIBUTORS ``AS IS'' AND ANY
+EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ICANN OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+THE POSSIBILITY OF SUCH DAMAGE.

MANIFEST.in

@@ -0,0 +1,22 @@
+include manage.py README.md runtests.sh Makefile LICENSE
+include Installation_Guide.md
+include bin/bootstrap bin/gen_secret_key
+include etc/requirements.txt
+include etc/nginx/lgr-django.conf etc/systemd/lgr-django.service etc/systemd/lgr-django.conf etc/systemd/lgr-celery.service
+include doc doc/conf.py doc/index.rst doc/Makefile
+recursive-include doc/_static *
+include docker/Dockerfile docker/start.sh
+include src/lgr_web/resources/lgr.rng
+include src/lgr_web/resources/lgr/README.txt
+recursive-include src/lgr_editor/locale *
+recursive-include src/lgr_editor/templates *
+recursive-include src/lgr_editor/static *
+recursive-include src/lgr_web/locale *
+recursive-include src/lgr_web/templates *
+recursive-include src/lgr_web/assets *
+recursive-include src/lgr_web/settings *.template
+recursive-include src/lgr_web/resources *.xml
+recursive-include src/lgr_validator/templates *
+recursive-include src/lgr_validator/locale *
+recursive-include src/lgr_tools/templates *
+recursive-include src/lgr_tools/locale *

Makefile

@@ -0,0 +1,71 @@
+#MKMSG_OPTIONS=--no-wrap
+MKMSG_OPTIONS=
+MAKEMESSAGES=PYTHONPATH=../.. ../../manage.py makemessages 
+COMPILEMESSAGES=PYTHONPATH=../.. ../../manage.py compilemessages
+
+LANGUAGES= \
+	   -l ar \
+	   -l en \
+	   -l fr \
+
+
+MO_FILES= \
+	  src/lgr_web/locale/ar/LC_MESSAGES/django.mo \
+	  src/lgr_web/locale/en/LC_MESSAGES/django.mo \
+	  src/lgr_web/locale/fr/LC_MESSAGES/django.mo \
+	  src/lgr_editor/locale/ar/LC_MESSAGES/django.mo \
+	  src/lgr_editor/locale/en/LC_MESSAGES/django.mo \
+	  src/lgr_editor/locale/fr/LC_MESSAGES/django.mo \
+	  src/lgr_validator/locale/ar/LC_MESSAGES/django.mo \
+	  src/lgr_validator/locale/en/LC_MESSAGES/django.mo \
+	  src/lgr_validator/locale/fr/LC_MESSAGES/django.mo \
+	  src/lgr_tools/locale/ar/LC_MESSAGES/django.mo \
+	  src/lgr_tools/locale/en/LC_MESSAGES/django.mo \
+	  src/lgr_tools/locale/fr/LC_MESSAGES/django.mo \
+
+PO_FILES= \
+	  src/lgr_web/locale/ar/LC_MESSAGES/django.po \
+	  src/lgr_web/locale/en/LC_MESSAGES/django.po \
+	  src/lgr_web/locale/fr/LC_MESSAGES/django.po \
+	  src/lgr_editor/locale/ar/LC_MESSAGES/django.po \
+	  src/lgr_editor/locale/en/LC_MESSAGES/django.po \
+	  src/lgr_editor/locale/fr/LC_MESSAGES/django.po \
+	  src/lgr_validator/locale/ar/LC_MESSAGES/django.po \
+	  src/lgr_validator/locale/en/LC_MESSAGES/django.po \
+	  src/lgr_validator/locale/fr/LC_MESSAGES/django.po \
+	  src/lgr_tools/locale/ar/LC_MESSAGES/django.po \
+	  src/lgr_tools/locale/en/LC_MESSAGES/django.po \
+	  src/lgr_tools/locale/fr/LC_MESSAGES/django.po \
+
+all: messages
+
+messages: $(MO_FILES)
+
+update: update-lgrweb update-lgreditor update-lgrvalidator update-lgrtools
+
+update-lgrweb:
+	cd src/lgr_web && $(MAKEMESSAGES) $(LANGUAGES) $(MKMSG_OPTIONS)
+
+update-lgreditor:
+	cd src/lgr_editor && $(MAKEMESSAGES) $(LANGUAGES) $(MKMSG_OPTIONS)
+
+update-lgrvalidator:
+	cd src/lgr_validator && $(MAKEMESSAGES) $(LANGUAGES) $(MKMSG_OPTIONS)
+
+update-lgrtools:
+	cd src/lgr_tools && $(MAKEMESSAGES) $(LANGUAGES) $(MKMSG_OPTIONS)
+
+src/lgr_web/locale/%/LC_MESSAGES/django.po:
+	cd `echo "$@" | sed -e 's#\/locale/.*##'` && django-admin.py makemessages $(LANGUAGES) $(MKMSG_OPTIONS)
+
+src/lgr_editor/locale/%/LC_MESSAGES/django.po:
+	cd `echo "$@" | sed -e 's#\/locale/.*##'` && django-admin.py makemessages $(LANGUAGES) $(MKMSG_OPTIONS)
+
+src/lgr_validator/locale/%/LC_MESSAGES/django.po:
+	cd `echo "$@" | sed -e 's#\/locale/.*##'` && django-admin.py makemessages $(LANGUAGES) $(MKMSG_OPTIONS)
+
+src/lgr_tools/locale/%/LC_MESSAGES/django.po:
+	cd `echo "$@" | sed -e 's#\/locale/.*##'` && django-admin.py makemessages $(LANGUAGES) $(MKMSG_OPTIONS)
+
+$(MO_FILES): $(PO_FILES)
+	cd `echo "$@" | sed -e 's#\/locale/.*##'` && $(COMPILEMESSAGES)