Merge pull request #7 from Viagenie/release-2.0.0

Release 2.0.0
icann · Sep 17, 2018 · 42faf6c · 42faf6c
2 parents 8a33ab2 + dc97a99
commit 42faf6c
Show file tree

Hide file tree

Showing 103 changed files with 27,410 additions and 1,676 deletions.
diff --git a/Release_Notes.md → Changelog.md b/Release_Notes.md → Changelog.md
@@ -1,4 +1,67 @@
-# LGR-Toolset Release notes
+# Changelog for lgr-django
+
+## 2.0.0 (2018-09-06)
+### New features
+- Support of Python 3. Compatibility with python2 is preserved for this release.
+- Add MSR-3 validating repertoire.
+- Create a "Tag" management page.
+- Create a function to assign tags/WLE to code points.
+- Allow choosing reference id when adding reference in editor.
+- Add tags in code point list view.
+### Improvements
+- Improve display of LGR validation when rebuild is valid.
+- Catch exceptions generated when input file is not a UTF-8 encoded file.
+- Add required file encoding on label fields in forms.
+- Update harmonization tool.
+- When adding an out-of-repertoire variant, automatically add it to the repertoire + add the mapping to the code point.
+- Improve wording of landing page.
+- Improve handling of output for comparison tools for large LGRs.
+- Update variant type to blocked for sample French LGR.
+- Add notice that import function is permissive.
+- Remove length limitation on label validation. Process will be run asynchronously if label can generate too many variants.
+- Export testing of variant symmetry and transitivity in AJAX view.
+- Add combined form of sequences in HTML output.
+- Sort resource/repertoire file lists.
+- Remove script parameter for harmonization tools.
+- Display rule name(s) when a code point does not comply with a contextual rule.
+- Better notification for invalid language tags.
+- Better consistency on forms' labels.
+### Fixes
+- Fix redirection when adding a code point fails.
+- Fix link to label in "Validate label" view.
+- Remove references from repertoire on deletion.
+- Replace number of variants by number of variant mappings in LGR validation statistics.
+- Add proper error message when adding a tag to a sequence.
+- Add support for missing language exception.
+- Fix script retrievals from repertoires.
+- Do not display useless error message when editing existing references.
+- Fix creation of references from code point/variant edition screen.
+- Fix deletion of action.
+- Display code points in hexadecimal in HTML output.
+- Add identity mapping for newly added variants not present in repertoire.
+- Fix header hiding page content.
+- Catch error in HTML output on ill-formed LGR's WLE.
+
+## 1.9 (2017-03-09)
+New features:
+- New interface to view all forms of a label: display the U-label, A-label and code point sequence.
+- Display the combined form of sequences.
+- New tool to check that multiple LGRs are harmonized.
+- New function to populate the missing symmetric and transitive variants.
+- New function to add code point from a script.
+- When adding a variant from another script, automatically add it to the repertoire if not present.
+Fixes:
+- Improve handling of large LGR.
+- Fix variant type that could be be changed from "blocked".
+- Fix label validation with ranges.
+- Always display failed rule when validating a label.
+- Improve display for summary (now renamed to LGR validation).
+- Fix tag count in HTML output.
+- Fix variant member count in HTML output.
+- Use hyperlinks for the table of references in HTML output.
+- Reduce number of rows in the variant set table in HTML output.
+- Add number of mappings on variant set table in HTML output.
+- Add link to save HTML output.
 
 ## 1.8.1 (2017-11-15)
 Fixes:

diff --git a/Installation_Guide.md b/Installation_Guide.md
@@ -25,15 +25,11 @@ Install the nginx webserver used to serve as a frontend. On CentOS, you need to
 enable the EPEL repository before:
 
 	# yum install epel-release
-	# yum install nginx
+	# yum install nginx redis
 
 Install virtualenv, used to create the Python environment:
 
-	# yum install python-virtualenv
-
-Install the dependencies needed to build the lxml module:
-
-	# yum install make gcc libxml2-devel libxslt-devel
+	# yum install python34-virtualenv
 
 Create a new user/group that will be used to run the application, as well as the
 log directory where log files will be placed:
@@ -55,7 +51,7 @@ and `lgr-django`) in a specific directory, and extract web-application:
 
 Create and configure the virtualenv:
 
-	$ virtualenv-2.7 /var/www/lgr/venv
+	$ virtualenv-3 /var/www/lgr/venv
 	$ source /var/www/lgr/venv/bin/activate
 	(venv)$ cd /var/www/lgr/lgr-django
 	(venv)$ pip install -r etc/requirements.txt -f /var/www/lgr/packages
@@ -64,17 +60,6 @@ Also install the application server:
 
 	(venv)$ pip install gunicorn
 
-Note: you may see the following error during the installation:
-
-	File "/var/www/lgr/venv/lib/python2.7/site-packages/gunicorn/workers/_gaiohttp.py", line 68
-	    yield from self.wsgi.close()
-	             ^
-	SyntaxError: invalid syntax
-
-The error can be ignored, and gunicorn will still be installed sucessfully.
-See [https://github.com/benoitc/gunicorn/issues/788](https://github.com/benoitc/gunicorn/issues/788).
-
-
 ## ICU installation
 
 If your distribution ships ICU, install it using the package manager.
@@ -93,6 +78,10 @@ For example, to install ICU 52 (Unicode 6.3), as root:
 
 The current supported ICU versions (and their corresponding Unicode version) are:
 
+* ICU 60 (Unicode 10.0)
+* ICU 58 (Unicode 9.0)
+* ICU 56 (Unicode 8.0)
+* ICU 54 (Unicode 7.0)
 * ICU 52 (Unicode 6.3)
 * ICU 50 (Unicode 6.2)
 * ICU 49 (Unicode 6.1)
@@ -107,6 +96,21 @@ binary packages from ICU website):
 	# echo "/usr/local/lib" > /etc/ld.so.conf.d/lgr.conf
 	# ldconfig
 
+### Latest ICU versions
+
+Starting from ICU 58 (Unicode 9.0.0), binary distributions of ICU require a newer C++ runtime that what is available on Centos7.
+You will need to recompile the library in order to support the newer versions.
+
+	# yum install gcc gcc-c++
+	# mkdir icu-60 ; cd icu-60
+	# wget http://download.icu-project.org/files/icu4c/60.2/icu4c-60_2-src.tgz
+	# tar xvf icu4c-60_2-src.tgz
+	# mkdir build ; cd build
+	# # You should switch to an unpriviledged user to compile
+	# ../icu/source/runConfigureICU Linux
+	# make check
+	# make install
+
 ## Django Configuration
 
 A template for production configuration is provided:
@@ -169,6 +173,7 @@ To serve the application, the following architecture in used:
 
 * The celery application is used to launch background tasks.
 * The gunicorn application server is used to run the python code.
+* A redis server is used to store messages between the webapp interface and the celery application.
 * The nginx webserver will act as a frontend proxy as well as serving static content.
 * Requests related to the application are relayed to the gunicorn server.
 
@@ -195,7 +200,7 @@ Also copy the temporary files configuration and create them:
 	# cp /var/www/lgr/lgr-django/etc/systemd/lgr-django.conf /etc/tmpfiles.d/
 	# systemd-tmpfiles --create
 
-Reload systemd daemon and enable and start the gunicorn process:
+Reload systemd daemon and enable and enable/start the following processes:
 
 	# systemctl daemon-reload
 	# systemctl enable lgr-django

diff --git a/README.md b/README.md
@@ -9,8 +9,8 @@ The following modules are reusable Django apps that can be included in other Dja
 
 * `lgr_editor` contains the code related to the LGR web-editor.
 * `lgr_validator` contains the code related the label validation module.
-* `lgr_tools` contains the code related to all utilities: testing LGR and
-  label sets, LGR comparisons, etc.
+* `lgr_tools` contains the code related to all utilities: testing LGR and label sets, LGR comparisons, etc.
+* `lgr_renderer` contains the code and templates used to generate the static exports of the LGR (HTML only for now).
 
 ## Acknowledgment
 
@@ -53,7 +53,7 @@ THE POSSIBILITY OF SUCH DAMAGE.
 ### Pre-requisites
 
 * Operating system: Tested on Linux and Mac OS X 
-* Python 2.7
+* Python 2.7, >= 3.4
 * [LibXML2](http://www.xmlsoft.org/) [MIT License] used by the lxml Python bindings
 * [ICU4C](http://site.icu-project.org/) [ICU License]
 * Python modules (listed in `etc/requirements.txt`, or dependencies thereof)
@@ -65,7 +65,8 @@ THE POSSIBILITY OF SUCH DAMAGE.
   * [picu](https://pypi.python.org/pypi/picu) [MIT/X License]
   * [munidata](https://github.com/icann/munidata) [BSD License]
   * [natsort](https://pypi.python.org/pypi/natsort) [MIT License]
-* A [redis](https://redis.io/) server for asynchronous computations
+  * [django-redis-cache](https://github.com/sebleier/django-redis-cache) [BSD License]
+* A [redis](https://redis.io/) server for cache and asynchronous computations
 
  For documentation generation:
 
@@ -203,47 +204,6 @@ of each Unicode version that we support.
 Keys are the Unicode version, like '6.3.0' Values are a dict of the kwargs to pass to the `munidata.manager.register`
 function along with the version.
 
-## Deploying application on a Docker image
-
-### Pre-requisites
-
-* Install Docker (version 1.9).
-* Copy the tarball archives of the project (lgr-core, lgr-django, munidata, picu) in the docker folder.
-
-### Create the Docker image
-
-The Docker image name and tag are set with the -t option name:tag.
-Tag is optional (default: latest).
-
-Build arguments are specific options for our Docker image.
-At the moment you can configure the allowed hosts for Django with the hosts argument and some e-mail parameters (server address and sender).
-
-    # cd docker
-    # docker build -t lgr-toolset --build-arg hosts="lgr.example" --build-arg email_srv="email-srv.lgr.example" --build-arg email_from="[email protected]" .
-
-### Run the Docker image
-
-The Docker image name and tag should be the same as in build command to run the
-relevant container.
-
-The -h option should contain an allowed host specified in the hosts argument
-when building the container.
-
-    # docker run -h lgr.example -d -p 80:80 lgr-toolset
-
-### Debug the Docker image
-
-The following command allows to get a bash shell in the Docker image in order to perform debug operations:
-
-    # docker run -h lgr.example -p 80:80 -i -t lgr-toolset bash
-
-Then to launch the server in the Docker image run:
-
-    # ./start.sh &
-
-You can now debug the Docker image.
-
-
 ## Deploying application to production
 
 Please see Installation\_Guide

diff --git a/bin/bootstrap b/bin/bootstrap
@@ -5,7 +5,7 @@ DSITE="src/lgr_web"
 
 if [ ! -e venv ]; then
 	echo "==> Creating virtualenv"
-	virtualenv2 venv || exit 3
+	virtualenv venv || exit 3
 fi
 
 # activate virtualenv

diff --git a/bin/gen_secret_key b/bin/gen_secret_key
@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+from __future__ import print_function
 
 from random import choice
 
@@ -31,4 +32,4 @@ def write_secret_key(path):
 
 
 if __name__ == '__main__':
-    print gen_secret_key()
+    print(gen_secret_key())
diff --git a/docker/Dockerfile b/docker/Dockerfile
diff --git a/docker/start.sh b/docker/start.sh
diff --git a/etc/requirements.txt b/etc/requirements.txt
@@ -3,11 +3,12 @@ Django==1.8.4
 django-widget-tweaks==1.4.1
 django-multiupload==0.5.2
 celery[redis]==3.1.23
+django-redis-cache==1.7.1
 
 # LGR/Unicode modules
-picu==0.8
-munidata==1.2
-lgr-core==1.8.1
+picu==1.0
+munidata==2.0.0
+lgr-core==2.0.0
 
 # Natural sorting implementation
 natsort==5.0.3
diff --git a/etc/systemd/lgr-django.service b/etc/systemd/lgr-django.service
@@ -1,7 +1,8 @@
 [Unit]
 Description=LGR Django application
 After=network.target
-Wants=lgr-celery.service
+Wants=lgr-celery.service redis.service
+After=redis.service
 
 [Service]
 WorkingDirectory=/var/www/lgr/lgr-django