Refresh the ES index after each index/bulk call, so results are available for search right after

[jblom]

After posting new documents to DANE and subsequently calling DANE to add new Tasks, the index is not always quick enough to refresh, so the Tasks cannot be added (because the docs are not available yet)

To avoid this race condition. The bulk index call needs to be extended with a refresh=True.

[jblom]

Update: so in order for the fix to be picked up, the DANE-server needs to use the new DANE code (with the extra refresh param). Since the DANE-server was never properly upgraded (ES7 client, latest DANE version, linting, code formatting etc) this is a nice opportunity to do so.

This PR shows the current progress. The basic upgrade and bugfix are ready to be tested in the ASR pipeline. CLARIAH/DANE-server#4

[jblom]

Update: the fix to DANE seems to work in the DANE-server API

I've tested it, using the improved main.py of the dane-workflows library

[jblom]

@KleinRana @mwigham or @wmelder the new DANE-server is deployed in the cluster and seems to work fine for me. If you could test with a fresh workflow? When it properly works, please close this issue.

In any case: please report your results in this issue.