Add remaining time estimate for file uploads #163
Milestone
Comments
pli888
added a commit
to pli888/gigadb-website
that referenced
this issue
Nov 25, 2019
…ebsite into rija-file-upload-wizard This commit merges pull request File upload wizard $346: 1. [gigascience#146: User stories, GitHub issues, Acceptance tests and dependencies] (rija#146) 1. [gigascience#147: Prototype of client/server file upload and software architecture blueprint] (rija#147) 1. [gigascience#148: Setup upload server and ftp servers in Docker Compose and for CI/CD] (rija#148) 1. [gigascience#151: Build admin function to create restricted access: filesystem directories & security token ] (rija#151) 1. [gigascience#152: Build secure and reliable upload server based on validated prototype] (rija#152) 1. [gigascience#153: Add remaining time estimate for file uploads] (rija#163) 1. [gigascience#165: Add FTP server interface to the Drop-Off area] (rija#165) This pull request has the following functionalities: File Upload Wizard FileDrop administration API and a prototype using it: * Create or Delete a FileDrop account and filesystem for a specific dataset through a REST API. * JSon Web Token based authentication for accessing the REST API * The FileDrop account's filesystem can be interacted with through a TUSd file upload server and an FTP server * A watcher (based on Linux's inotify) service moves files around the filesystem from the drop-off area to a download area (where ftp links used in mockup page are built from) * The initial prototype has been updated to use the API and to be optionally deployable from within the project * the File Upload Wizard is a standalone, full-stack web application built out of the [Yii 2 Advanced template](https://www.yiiframework.com/extension/yiisoft/yii2-app-advanced/doc/guide/2.0/en/start-installation) but still accessed through GigaDB website's URL space and sharing the same Database server and code repository * Nginx setup was updated to proxy multiple web apps and other services (Tusd). See ``ops/configuration/nginx-conf/sites`` * Nginx setup was updated to make it easy to disable/enable specific webapps (used for feature flagging the prototype). See ``ops/configuration/nginx-conf/enable_sites`` * Significant changes to the CI/CD scripts for deploying multiple webapps and for improved operational safety * Added [TUSd](https://tusd.io) service as container * Added [Pure-FTPD](https://pureftpd.org) service as container * Added a [docker-inotify-command](https://github.com/coppit/docker-inotify-command) service as container This PR is meant to lay down infrastructure changes needed to implement File Upload Wizard, so no visible business features are delivered here but the prototype was updated to use the API and to be deployable at ``/proto/``. Both the [GoogleDocs spreadsheet for tasks](https://docs.google.com/spreadsheets/d/1bogVGdvchB790TpMQDN3LL_BvolXWoCni76k1Qi7OiA/edit?pli=1#gid=0) and my [Git project board](https://github.com/users/rija/projects/1) should be up-to-date. No change to **gigadb** database, but a new database **fuwdb** to support File Upload Wizard web application's functions. **fuwdb**'s schema is almost entirely built out of Yii2 migrations in ``fuw/app/console/migrations``. Only the **user** table is created by the Yii2 Advanced Template's [Composer script](https://www.yiiframework.com/extension/yiisoft/yii2-app-advanced/doc/guide/2.0/en/start-installation#installing-using-composer). No change to GigaDb model classes. New model layer in the File Upload Wizard webapps divided in 3 categories: * Backend models (``fuw/app/backend/models/FiledropAccount.php`` and ``fuw/app/backend/models/DockerManager.php``): admin functions, used for admin management of Filedrop accounts * Frontend models: user facing for future integration between GigaDB UX and FUW's REST API, currently just scaffolded code from Yii2 Advanced template * Common models: model and config used by backend and frontend. ``fuw/app/common/models/User.php`` is used for the JSon web token (JWT) based authentication to the REST API. No change to GigaDB controller classes. There is a new REST controller in File Upload Wizard for creating and deleting filedrop accounts ``fuw/app/backend/controllers/FiledropAccountController.php``. That controller that corresponds to the ``FiledropAccount.php`` model was automatically scaffoled using `gii`. However, I've added a custom ``fuw/app/backend/actions/FiledropAccountController/DeleteAction.php`` action so that DELETE command marks the model with a special status instead of deleting it. N/A N/A The test setup for GigaDB was updated to fix inconsistent and fragile approach to loading and restoring database data and to work in context of multi-webapp testing. ([relevant commit](rija@f84d739#diff-ddbd905c83513d6eac66f30e9a493068)) Also Behat acceptance tests now use profiles instead of tags for configuration. See ``behat.yml`` So acceptance are run with a profile as argument. e.g: ``` $ docker-compose run --rm test bin/behat --profile local -v --stop-on-failure ``` The 'cli' profile is for Gitlab CI environment. The File Upload Wizard has unit tests and functional tests. There's no business functionality completed yet so no acceptance tests has been added yet. The tests are located in: * ``fuw/app/backend/tests/`` * ``fuw/app/common/tests/`` * ``fuw/app/frontend/tests/`` [Codeception](https://github.com/Codeception/Codeception) is setup in File Upload Wizard directory as a test runner and test framework for unit, functional and acceptance tests (and it can support Behat features). It is integrated, extensible, up-to-date and maintained and work with the most recent and future version of underlying technology (PHP, PHPunit, Yii2, Selenium). The config files relevant to Codeception are: * ``fuw/app/backend/codeception.yml`` * ``fuw/app/common/codeception.yml`` * ``fuw/app/frontend/codeception.yml`` * ``fuw/app/codeception.yml`` More info on testing is included in ``docs/fuw/developer_guide.md``. The overall flow and technologies haven't changed. The following changes have been made to accomodate CI and CD of a multi-webapps project or to fix structural weakness in the CI/CD process: * Build manifests (Dockerfile and Production-Dockerfile) should be part of each webapp directory space, but service composition (docker-compose.\*.yml), appliance provisioning (terraform) and system configuraiton (ansible) should stay centralized * It is dangerous to only have one Terraform state for all environments as a provisioning mistake can take down the wrong environment (e.g: production), so now there is a distinct Terraform state for each environment represented by a separate environment directory in ``ops/infrastructure/envs`` * Previously, provisioning a new environment required duplicating Terraform and Ansible code. Now they are modularized with the bulk of the code (Ansible roles and Terraform modules) stored outside the environment directory because they are environment-agnostic. * The Ansible playbook and variables and Terraform main script and variables are in the environment-specific directory in ``ops/infrastructure/envs`` Furthermore, the provisioning for GigaDB has been upgraded to Debian 9 Stretch and the database server upgraded to Postgresql 9.6. Finally, there are updates to Gitlab CI config ``.gitlab-ci.yml``: * distinct stages to deploy on staging for GigaDB only (**deploy_gigadb**), GigaDB and File Upload Wizard API (**deploy_apps**), or GigaDB with File Upload Wizard API and the prototype (**deploy_proto**) * It is no longer necessary to have a separate stage for initial deployment of Let's Encrypt certificate. * The CI/CD docs have been updated in light of changes in provisioning * There's a new ``docs/fuw`` directory with documentation relating to File Upload Wizard (with the quickstart guide in ``docs/fuw/index.md``) * I now use [mkdocs](https://www.mkdocs.org) to create a doc server and to build a doc web site easily (configured with ``mkdocs.yml`` at the root of the project) * There's a ``docs/index.md`` with instructions for using mkdocs and to serve as index page for doc server/website. * updated ``README.md`` and ``docs/INSTALL.md`` to reflect the change in starting and testing the GigaDB webapp Bind-mounting of Docker Daemon unix socket inside container is a security vulnerability, so to allow inter-container communication, File Upload Wizard use the ``Docker-PHP`` php library that uses the Docker Daemon API instead. See ``DockerManager.php`` for the current implementation. (I haven't changed yet the GigaDB acceptance tests that still use the unix socket approach). However on the Mac, the current version of Docker For Desktop doesn't expose the API on a TCP port, so when working on File Upload Wizard, we need to manually expose the API on a TCP port during development. See ``docs/fuw/index.md``
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: