wis2box 1.0b6 released

[tomkralidis]

This is the sixth 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes. This release implements the required updates to the WIS2 Topic Hierarchy regarding dataset topics. Users are strongly encouraged to update to this latest release for WIS2 compliance.

Enhancements/updates:

• Implementation of the WIS2 Topic Hierarchy update to merge country and centre-id (see the section below)
• UI: add option/toggle to display station points as clusters wis2box-ui: turn clustered markers on/off via ENV variable or map toggle  #588
• duplicate-data-check: wis2box will not publish the same data twice and instead print an error wis2box should not publish the same data twice #600
• updated-data-check: wis2box will detect updates and add a link to indicate data is updated handling corrected data in the automated data workflow #565
• wis2box-webapp inspect button to use internal URL wis2box-api:1.0b5 bufr2geojson.py needs to use internal URL #582
• WNM: provide data inline in notification messages when data size less than 4096 bytes
• Storage guidelines are now 24GB
• Station list: barometer height is no longer calculated if not defined (left null/blank)
• updating of BUFR tools to their latest versions
• collect NGINX logs in loki

Bug fixes:

• BUFR handling:
  • "Passed array is too small" fix (bufr transformation error: Passed array is too small #559)
  • fix TSI matching (bufr-to-bufr pipeline does not match on tsi #581)
• WCMP2
  • fix update date calculation
  • fix contact information
  • fix metadata message publication
• WTH
  • remove pywcmp in lieu of pywis-topics for topic handling

Choosing a valid centre-id for the pre-operational phase

The WIS2 Topic Hierarchy merges country and centre-id in a new centre-id that uniquely defines a WIS2-node.

If you plan to use wis2box-1.0b6 to run a WIS2-node in the pre-operational phase, please ensure to use your new centre-id starting with your TLD during the configuration process.

If you are unsure how to define the centre-id for your WIS2-node, please contact WMO.

New installation

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int

Migrating existing installations

The following steps are required when upgrading from wis2box 1.0b5 to 1.0b6:

# stop current wis2box
cd ~/wis2box-1.0b5 && python3 wis2box-ctl.py stop

# download new release in home-directory
cd ~/ && wget https://github.com/wmo-im/wis2box/releases/download/1.0b6/wis2box-setup-1.0b6.zip

# unzip and go to new directory
unzip wis2box-setup-1.0b6.zip && cd wis2box-1.0b6

IMPORTANT: before starting the new wis2box-release your configuration files need to be updated to use the new WTH. You can use the migration-script to update your configuration files

# run migration script to update configuration
python3 migration/update_configuration.py

The migration-script will ask you to answer a series of questions to replace country-code and old centre-id with your new centre-id, as per the following example:

Enter your 3-letter country-code: ita
3-letter country-code: ita ? (y/n) y
Enter your old centre-id: maaike-test
Old centre-id: maaike-test ? (y/n) y
Enter your new centre-id: it-maaike-test
New centre-id: it-maaike-test ? (y/n) y
Updated file 'metadata-synop.yml'
Updated file 'data-mappings.yml'

Check and verify that your data-mappings.yml file contains the new topic hierarchy. An example can be found below:

data:
    it-maaike-test.data.core.weather.surface-based-observations.synop:
        plugins:
            txt:
                - plugin: wis2box.data.synop2bufr.ObservationDataSYNOP2BUFR
                  notify: true

Continue with the update:

# start wis2box
python3 wis2box-ctl.py start  # wait until the new images are downloaded

# login to wis2box and execute commands
python3 wis2box-ctl.py login

# create new data collection, replace with your filename
wis2box data add-collection /data/wis2box/metadata/discovery/metadata-synop.yml

# publish new discovery metadata, replace with your filename
wis2box metadata discovery publish /data/wis2box/metadata/discovery/metadata-synop.yml

Verify the wis2box UI to ensure that there are 2 entries per dataset (one for old and one for new topic hierarchy):

Continue with the update:

# reindex your data collection items, replace with your old and new discovery metadata ID
wis2box data reindex-collection-items urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop urn:x-wmo:md:it-maaike-test:surface-based-observations.synop

# unpublish your old dataset using the old discovery metadata ID
wis2box metadata discovery unpublish urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop

# associate stations to the new topic hierarchy
wis2box metadata station add-topic origin/a/wis2/it-maaike-test/data/core/weather/surface-based-observations/synop

# exit the wis2box-management container
exit

Finally you can clean the unused Docker images to preserve diskspace:

docker image prune -a

(Or docker image prune without -a depending on your docker version)

As a final step, you will need to update any data ingestion processes to copy to new corresponding folder in the wis2box-incoming bucket

(Old)

(New)

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b6

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int