File upload wizard #346
Closed
File upload wizard #346
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Nov 7, 2019
|
When trying to get #346 to work in my
Running
Seems to be caused by a problem in |
|
When running unit tests, I got an error message: The problem seems to be caused by |
|
Thanks for the feedback, I’ll have a look |
|
On running the acceptance tests, the This |
|
When running the CI/CD pipeline, the |
|
Hi @pli888 , Regarding the composer error, I just saw that you made a corresponding fix to the develop branch 3 weeks ago. I'll rebase the PR branch for that and to ensure I haven't missed any other changes. Regarding Postgresql 9.6 version, I mentionned it in the PR, but I should have explicitely warn to change the version in the .env file and I should have updated the env-sample file as well. I'll do all the above now, tweak the readme, and then will look at the acceptance tests issues you are seeing (I didn't have those problems so my install must have some assumptions). |
added 22 commits
November 27, 2019 16:00
before restoring the original database after running acceptance tests, we need to drop the database because some of test database dump loaded during test run have out-dated schema that cause the post-tests pg_restore process to fail. Due to the need for all clients to be disconnected before dropping database, the drop and restore has to be done as a AfterSuite hooks in Behat when we still got a chance to kill connections and restart php container. It's also more elegant than doing it in the bash script.
not really needed as it was there for debug and it has been removed from Debian 9 causing that step to fail
Gigadb web site can run more than one PHP webapp behind the Nginx web site. Only one Docker Compose file is used to select the webapps to start and their dependencies. There is also integrated test runners across all web apps for unit tests and functional tests There is one Postgresql server holding all the necessary database schema used by the PHP webapps. Each new and future webapp has their own directory that should match the Yii2 advanced template and can be made of separate sub-webapps as well. A Dockerfile and test directories for each webapp/sub-webapps should be kept in each webapp directory, but the provisioning, CI/CD setup, and integrated test runners are centralized. functional_custom_bootstrap.php: functional tests need the main database to have certain test data, but at the same time it needs to backup the current database before running and restoring it after running, hence the need of a special bootstrap file "functional_custom_bootstrap.php" for Gigadb's webapp functional tests. Before it wasn't needed there because these hooks were done in the overall test runner that ran all the suites for the only webapp, but now we need to be flexible with each suite and run each suite for all webapps. Since webapps may have a distinct database schema, such hooks should be within a webapp's test directories. enable_sites: "enable_sites" is a shell script (use Alpine's ash shell) that symlink nginx server configs from site-availables to sites-enabled directories. Until they appear in the later, nginx won't be able to serve them. It allow Docker compose to start nginx for a selected list of webapps. That and the integrated test runner make it easier to configure new webapps.
added 28 commits
November 27, 2019 18:01
there is now three stages to choose from for deploying to staging: deploy_gigadb: deploy regular gigadb website with nothing related to File Upload Wizard deploy_apps: deploy GigaDB website and the File Upload Wizard API deploy_proto: deploy GigaDB website and the File Upload Wizard API and the prototype missed a step for deploying GigaDB: docker-compose run --rm less missed a step for deploying File Upload Wizard: docker-compose run --rm fuw-config
…nto file-upload-wizard
to avoid otherwise successful tests to return non-zero exit status when File Upload Wizard container are not running, the test runner only conditionally run FUW tests if the console container is running and the success status of the GigaDB test run is preserved and used as exit status at the end
By versioning the host data directory used by postgresql container, it will enable data migration when upgrading between versions of postgresql with breaking data format changes. Then a tool like this can be used: https://github.com/tianon/docker-postgres-upgrade Fixed the gigadb_testdata.pgdmp to preserve the rebased from develop while being in 9.6 postgresql format (and remove the step in acceptance scenarios that's a fix no longer necessary)
This looks like and old bug rather than a regression. It may happen on production too. There was a framework level validation limiting max characters to 50 only. The acceptance tests didn't fail and pick that up because the url used in the scenario was unfortunately exactly 50 characters. Fixed the test scenario and the validation to match the database column max character.
This looks like and old bug rather than a regression. It may happen on production too. It is connected to the bug of the redirect not working with long urls, as in the test for this didn't fail because of the same root cause of the redirect issue (see previous commit), but this time the dodgy characters entered in the test scenario was longer than the artificially limited field length so the dodgy keyword was indeed rejected, but for the wrong reason. Added proper validation and unit tests. The acceptance scenario for this feature doesn't need fixing.
|
Hi @pli888, Feel free to try running this PR again, I made a lot of fixes (see list below). All tests suites are passing locally on my machine and in the CI: I've managed to reproduce the issue you've had with the redirect test that initiated my commits c674b9c and 9fd113b below which are not regressions but old bugs that may also happen on live. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull request for issue:
This is a pull request for the following functionalities:
File Upload Wizard FileDrop administration API and a prototype using it:
ops/configuration/nginx-conf/sitesops/configuration/nginx-conf/enable_sitesThis PR is meant to lay down infrastructure changes needed to implement File Upload Wizard, so no visible business features are delivered here but the prototype was updated to use the API and to be deployable at
/proto/.Both the GoogleDocs spreadsheet for tasks and my Git project board should be up-to-date.
Changes to the database schema
No change to gigadb database, but a new database fuwdb to support File Upload Wizard web application's functions.
fuwdb's schema is almost entirely built out of Yii2 migrations in
fuw/app/console/migrations.Only the user table is created by the Yii2 Advanced Template's Composer script.
Changes to the model classes
No change to GigaDb model classes.
New model layer in the File Upload Wizard webapps divided in 3 categories:
fuw/app/backend/models/FiledropAccount.phpandfuw/app/backend/models/DockerManager.php): admin functions, used for admin management of Filedrop accountsfuw/app/common/models/User.phpis used for the JSon web token (JWT) based authentication to the REST API.Changes to the controller classes
No change to GigaDB controller classes.
There is a new REST controller in File Upload Wizard for creating and deleting filedrop accounts
fuw/app/backend/controllers/FiledropAccountController.php.That controller that corresponds to the
FiledropAccount.phpmodel was automatically scaffoled usinggii.However, I've added a custom
fuw/app/backend/actions/FiledropAccountController/DeleteAction.phpaction so that DELETE command marks the model with a special status instead of deleting it.Changes to the view template files
N/A
Changes to the style
N/A
Changes to the tests
In GigaDB:
The test setup for GigaDB was updated to fix inconsistent and fragile approach to loading and restoring database data and to work in context of multi-webapp testing. (relevant commit)
Also Behat acceptance tests now use profiles instead of tags for configuration. See
behat.ymlSo acceptance are run with a profile as argument. e.g:
The 'cli' profile is for Gitlab CI environment.
In File Upload Wizard
The File Upload Wizard has unit tests and functional tests. There's no business functionality completed yet so no acceptance tests has been added yet.
The tests are located in:
fuw/app/backend/tests/fuw/app/common/tests/fuw/app/frontend/tests/Codeception is setup in File Upload Wizard directory as a test runner and test framework for unit, functional and acceptance tests (and it can support Behat features). It is integrated, extensible, up-to-date and maintained and work with the most recent and future version of underlying technology (PHP, PHPunit, Yii2, Selenium).
The config files relevant to Codeception are:
fuw/app/backend/codeception.ymlfuw/app/common/codeception.ymlfuw/app/frontend/codeception.ymlfuw/app/codeception.ymlMore info on testing is included in
docs/fuw/developer_guide.md.Changes to the provisioning
The overall flow and technologies haven't changed. The following changes have been made to accomodate CI and CD of a multi-webapps project or to fix structural weakness in the CI/CD process:
ops/infrastructure/envsops/infrastructure/envsFurthermore, the provisioning for GigaDB has been upgraded to Debian 9 Stretch and the database server upgraded to Postgresql 9.6.
Finally, there are updates to Gitlab CI config
.gitlab-ci.yml:Changes to the documentation
docs/fuwdirectory with documentation relating to File Upload Wizard (with the quickstart guide indocs/fuw/index.md)mkdocs.ymlat the root of the project)docs/index.mdwith instructions for using mkdocs and to serve as index page for doc server/website.README.mdanddocs/INSTALL.mdto reflect the change in starting and testing the GigaDB webappSecurity related changes
Bind-mounting of Docker Daemon unix socket inside container is a security vulnerability, so to allow inter-container communication, File Upload Wizard use the
Docker-PHPphp library that uses the Docker Daemon API instead. SeeDockerManager.phpfor the current implementation.(I haven't changed yet the GigaDB acceptance tests that still use the unix socket approach).
However on the Mac, the current version of Docker For Desktop doesn't expose the API on a TCP port, so when working on File Upload Wizard, we need to manually expose the API on a TCP port during development. See
docs/fuw/index.md