- Add support for Python 3.13 and deprecate Python 3.9
- DWD Road: Use correct 15 minute resolution
- Bump polars to >=1.0.0
- Change DWDMosmixValues and DWDDmoValues to follow the core _collect_station_parameter method
- Allow only single issue retrieving with DWDMosmixRequest and DWDDmoRequest
- Fix state column in station list creation for DWD Observation
- Make fastexcel non-optional
- Remove upper dependency bounds
- DWD Road: Add new station groups, log warning if no data is available, especially if the station group is one of the temporarily unavailable ones
- Explorer: Fix DWD Mosmix request kwargs setup
- Fix multiple Geosphere parameter and unit enums
- Explorer: Fix wrap (parameter, dataset) in iterator
- Adjust parameter typing of apis
- Rename parameters - units in parameter names are now directly following the number - temperature parameters now use meter instead of cm and also have a unit - e.g. TEMPERATURE_AIR_MEAN_2M, CLOUD_COVER_BETWEEN_2KM_TO_7KM, PROBABILITY_PRECIPITATION_HEIGHT_GT_0_0MM_LAST_6H
- Bump pyarrow version to <18
- Fix EaHydrology station list parsing
- Rename EaHydrology to EAHydrology
- Fix propagation of settings through EAHydrology values
- Fix DWD Road api
- Bump environs to <12
- Explorer: Fix json export
- EaHydrology: Fix date parsing
- Hubeau: Use correct frequency unit
- Fix group by unpack
- Allow passing --listen when running the explorer to specify the host and port
- Rename warming stripes to climate stripes
- Add precipitation version
- Replace custom Settings class with pydantic model
- Interpolation/Summary: Require start and end date
- Fix multiple issues with interpolation and summarization
- Enable interpolation and summarization for all services
- Fix dropna argument for DWD Mosmix and DMO
- Adjust DWD Mosmix and DMO kml reader to parse all parameters
- Fix to_target(duckdb) for stations
- Fix init of DwdDmoRequest
- Fix DWD Obs station list parsing again
- Allow wide shape with multiple datasets
- Adjust column specs for DWD Observation station listing
- Maintain order during deduplication
- Change threshold in filter_by_name to 0.0...1.0
- Warming stripes: Add option to enable/disable showing only active stations
- Migrate explorer to streamlit
- Explorer: Disable higher than daily resolutions for hosted version
- UI: Add warming stripes
- Fix parsing of DWD Observation stations where name contains a comma
- Docker: Install more extras
- Cli/Restapi: Return empty values if no data is available
- Fix setting NOAA GHCN-h date to UTC
- Refactor index caching -> Remove monkeypatch for fsspec
- NOAA GHCN Hourly: Fix date parsing
- Add NOAA GHCN Hourly API (also known as ISD)
- Remove join outer workaround for polars and use outer_coalesce instead
- Allow duckdb for Python 3.12 again
- Update REST API index layout
- Bump polars to 0.20.10
- Docker: Bump to Python 3.12
- Docker: Reduce image size
- Restapi: Add health check endpoint
- Set upper version bound for Python to 4.0
- Add temporary workaround for bugged line in IMGW Hydrology station list
- Fix parsing of dates in NOAA GHCN api
- Make pandas optional
- Allow for passing kwargs to the to_csv method
- Fix issue when using force_ndarray_like=True with pint UnitRegistry
- Fix issue with DWD DMO api
- CI: Add support for Python 3.12
- Docker: Enable interpolation in wetterdienst standard image
- IMGW Meteorology: Drop workaround for mixed up station list to fix issue
- WSV Hydrology: Fix issue with station list characteristic values
- DWD Observation: Remove redundant replace empty string in parser
- NWS Observation: Read json data from bytes
- EA Hydrology: Read json data from bytes
- Replace partial with lambda in most places
- IMGW: Use ttl of 5 minutes for caching
- Restapi: Unify station parameter and add alias
- Interpolation: Make maximum station distance per parameter configurable via settings
- Result: Convert date to string only if dataframe is not empty
- Restapi: Move restapi from /restapi to /api
- Add example for comparing Mosmix forecast and Observation data
- Fix parsing of DWD Observation 1 minute precipitation data
- [Breaking] Use start_date and end_date instead of from_date and to_date
- Use artificial station id for interpolation and summarization
- Rename taken station ids columns for interpolation and summarization
- Add workaround for issue with DWD Observation station lists
- Fix DWD DMO access again
- Add lead time argument - one of short, long - for DWD DMO to address two versions of icon
- Rework dict-like export formats and tests with extensive support for typing
- Improve radar access
- Style restapi landing page
- Replace timezonefinder by tzfpy
- Cleanup error handling
- Make cli work with DwdDmoRequest API
- Cleanup cli docs
- Fix DWD Observation API for 5 minute data
- Remove direct tzdata dependency
- Replace pandas read_fwf calls by polars substitutes
- Export: Add support for InfluxDB 3.x
- [Streamlit] Add sideboard with settings
- [Streamlit] Add station information json
- [Streamlit] Add units to DataFrame view and plots
- [Streamlit] Add JSON download
- Return data correctly sorted
- Fix multiple issues with DwdObservationRequest API
- Raise minimum version of polars to 0.19.6 due to breaking changes
- Make parameters TEMPERATURE_AIR_MAX_200 and TEMPERATURE_AIR_MIN_200 summarizable/interpolatable
- Add streamlit app for DWD climate stations
- Add sql query function to streamlit app
- Fix imgw meteorology station list parsing
- Improve streamlit app plotting capabilities
- Fix DWD DMO api
- Add implementation for DWD DMO
- Fix DWD solar date string correction
- Fix documentation and unit conversion for Geosphere 10minute radiation data
- Fix Geosphere parameter names
- Revise type hints for parameter and station_id
- Fix Geosphere Observation parsing of dates in values -> thanks to @mhuber89 who discovered the bug and delivered a fix
- Fix bug with Geosphere parameter case
- Add retry to functions
- Add IMGW Hydrology API
- Add IMGW Meteorology API
- Rename FLOW to DISCHARGE and WATER_LEVEL to STAGE everywhere
- Fix pyarrow dependency
- Backend: Migrate from pandas to polars
- Sources: Add DWD Road Weather data
Attention!
Switching to Polars may cause breaking changes for certain user-space code
heavily using pandas idioms, because Wetterdienst now returns a Polars DataFrame.
If you absolutely must use a pandas DataFrame, you can cast the Polars DataFrame
to pandas by using the .to_pandas() method.
- Fix Unit definition for RADIATION_GLOBAL
- Fix JOULE_PER_SQUARE_METER definition from kilojoule/m2 to joule/m2
- Update docker images
- Fix now and now_local attributes on core class
- Fix precipitation index interpolation
- Fix setting empty values in DWD observation data
- Fix DWD Radar composite path
- Explorer: Fix function calls
- Drop Python 3.8 support
- Fix DWD Observations 1 minute fileindex
- CLI: Fix cli arguments with multiple items separated by comma (,)
- Fix fileindex/metaindex for DWD Observation
- SCALAR: Improve handling skipping of empty stations, especially within .filter_by_rank function
- DOCS: Fix precipitation height unit
- DOCS: Fix examples with "recent" period
- Make all parameter levels equal for all weather services to reduce complexity in code
- Change
tidyoption toshape, whereshape="long"equalstidy=Trueandshape="wide"equalstidy=False - Naming things: All things "Scalar" are now called "Timeseries", with settings prefix
ts_ - Drop some unnecessary enums
- Rename Environment Agency to ea in subspace
- SCALAR: Change tidy option to be set to True if multiple different entire datasets are queried This change is in accordance with exporting results to json where multiple DataFrames are concatenated.
- CLI: Add command line options
wetterdienst --versionandwetterdienst -vto display version number - Further cleanups
- Change Settings to be provided via initialization instead of having a singleton
- Add Geosphere Observation implementation for Austrian meteorological data
- RADAR: Clean up code and merge access module into api
- DWD MOSMIX: Fix parsing station list
- DWD MOSMIX: Fix converting degrees minutes to decimal degrees within the stations list. The previous method did not produce correct results on negative lat/lon values.
- Update wetterdienst explorer with clickable stations and slighly changed layout
- Improve radar tests and certain dict comparisons
- Fix problem with numeric column names in method gain_of_value_pairs
- Interpolation/Summary: Now the queried point can be an existing station laying on the border of the polygon that it's being checked against
- Geo: Change function signatures to use latlon tuple instead of latitude and longitude
- Geo: Enable querying station id instead of latlon within interpolate and summarize
- Geo: Allow using values of nearby stations instead of interpolated values
- Fix timezone related problems when creating full date range
- UI: Add interpolate/summarize methods as subspaces
- Fix bug where duplicates of acquired data would be dropped regarding only the date but not the parameter
- Add NOAA NWS Observation API
- Add Eaufrance Hubeau API for French river data (flow, stage)
- Fix NOAA GHCN access issues with timezones and empty data
- Fix DWD Observation urban_pressure dataset access (again)
- Add example to dump DWD climate summary observations in zarr with help of xarray
- Fix DWD Observation urban_pressure dataset access
- Add support for reading DWD Mosmix-L all stations files
- Add summary of multiple weather stations for a given lat/lon point (currently only works for DWDObservationRequest)
- Make DwdMosmixRequest return data according to start and end date
- Fix passing an empty DataFrame through unit conversion and ensure set of columns
- Add interpolation of multiple weather stations for a given lat/lon point (currently only works for DWDObservationRequest)
- Fix access of DWD Observation climate_urban datasets
- Slightly adapt the conversion function to satisfy linter
- Fix parameter names:
- we now use consistently INDEX instead of INDICATOR
- index and form got mixed up with certain parameters, where actually index was measured/given but not the form
- global radiation was mistakenly named radiation_short_wave_direct at certain points, now it is named correctly
- Adjust Docker images to fix build problems, now use python 3.10 as base
- Adjust NOAA sources to AWS as NCEI sources currently are not available
- Make explorer work again for all services setting up Period enum classes instead of single instances of Period for period base
- Use lxml.iterparse to reduce memory consumption when parsing DWD Mosmix files
- Fix Settings object instantiation
- Change logging level for Settings.cache_disable to INFO
- Add DWD Observation climate_urban datasets
- Fix DWD Mosmix station locations
- Move cache settings to core wetterdienst Settings object
- Fix two parameter names
- Fix correct mapping of periods for solar daily data which should also have Period.HISTORICAL besides Period.RECENT
- Fix passing through of empty dataframe when trying to convert units
- Update dependencies
- Update dependencies
- Add DWD Observation 5 minute precipitation dataset
- Add test to compare actually provided DWD observation datasets with the ones we made available with wetterdienst
- Fix one particular dataset which was not correctly included in our DWD observations resolution-dataset-mapping
- Fix EA hydrology access
- Update ECCC observation methods to acquire station listing
- Fix using shared FSSPEC_CLIENT_KWARGS everywhere
- Add option to skip empty stations (option tidy must be set)
- Add option to drop empty rows (value is NaN) (option tidy must be set)
- Add UKs Environment Agency hydrology API
- Fix acquisition of DWD weather phenomena data
- Set default encoding when reading data from DWD with pandas to 'latin1'
- Fix typo in EcccObservationResolution
- Fix acquisition of historical DWD radolan data that comes in archives
- Fix creation of empty DataFrame for missing station ids
- Fix creation of empty DataFrame for annual data
- Revert ssl option
- Circumvent DWD server ssl certificate problem by temporary removing ssl verification
- Add implementation of WSV Pegelonline service
- Clean up code at several places
- Fix ECCC observations access
- Change integer dtypes in untidy format to float to prevent loosing information when converting units
- Improve integrity of dataset, parameter and unit enumerations with further tests
- Change source of hourly sunshine duration to dataset sun
- Change source of hourly total cloud cover (+indicator) to dataset cloudiness
- Fix naming of sun dataset
- Fix DWD Observation monthly test
- Fix monthly/annual data of DWD observations
- Simplify parameters using only one enumeration for flattened and detailed parameters
- Rename dataset SUNSHINE_DURATION to SUN to avoid complications with similar named parameter and dataset
- Rename parameter VISIBILITY to VISIBILITY_RANGE
- Add datasets EXTREME_WIND (subdaily) and MORE_WEATHER_PHENOMENA (daily)
- Add support for Python 3.10 and drop Python 3.7
- Extend explorer to use all implemented APIs
- Fix cli/restapi: return json and use NULL instead of NaN
- Fix missing station ids within values result
- Add details about time interval for NOAA GHCN stations
- Fix falsely calculated station distances
- Add support for Python 3.10, drop support for Python 3.7
- Add Wetterdienst.Settings to manage general settings like tidy, humanize,...
- Rename DWD forecast to mosmix
- Instead of "kind" use "network" attribute to differ between different data products of a provider
- Change data source of NOAA GHCN after problems with timeouts when reaching the server
- Fix problem with timezone conversion when having dates that are already timezone aware
- Fix cli error with upgraded click ^8.0 where default False would be converted to "False"
- Fix access to ECCC stations listing using Google Drive storage
- Remove/replace caching entirely by fsspec (+monkeypatch)
- Fix bug with DWD intervals
- Add NOAA GHCN API
- Fix radar index by filtering out bz2 files
- [FIX] Add missing positional dataset argument for _create_empty_station_parameter_df
- [FIX] Timestamps of 1 minute / 10 minutes DWD data now have a gap hour at the end of year 1999 due to timezone shifts
- [BREAKING] Introduce core Parameter enum with fixed set of parameter names. Several parameters may have been renamed!
- Add FSSPEC_CLIENT_KWARGS variable at wetterdienst.util.cache for passing extra settings to fsspec request client
- Start migrating from
dogpile.cachetofilesystem_spec
- Enable selecting a parameter precisely from a dataset by passing a tuple like [("precipitation_height", "kl")] or [("precipitation_height", "precipitation_more")], or for cli/restapi use "precipitation_height/kl"
- Rename
wetterdienst showtowetterdienst info, make version accessible via CLI withwetterdienst version
- Bug when querying an entire DWD dataset for 10_minutes/1_minute resolution without providing start_date/end_date, which results in the interval of the request being None
- Test of restapi with recent period
- Get rid of pandas performance warning from DWD Mosmix data
- Bugfix acquisition of DWD radar data
- Adjust DWD radar composite parameters to new index
- Bugfix tidy method for DWD observation data
- Update readme on sandbox developer installation
- Bugfix show method
- Change cli base to click
- Add support for wetterdienst core API in cli and restapi
- Export: Use InfluxDBClient instead of DataFrameClient and improve connection handling with InfluxDB 1.x
- Export: Add support for InfluxDB 2.x
- Fix InfluxDB export by skipping empty fields
- Add show() method with basic information on the wetterdienst instance
- Make tidy method a abstract core method of Values class
- Fix DWD Mosmix generator to return all contained dataframes
- Add origin and si unit mappings to services
- Use argument "si_units" in request classes to convert origin units to si, set to default
- Improve caching behaviour by introducing optional
WD_CACHE_DIRandWD_CACHE_DISABLEenvironment variables. Thanks, @meteoDaniel! - Add baseline test for ECCC observations
- Add DWD Observation hourly moisture to catalogue
- Add capability to export data to Zarr format
- Add Wetterdienst Explorer UI. Thanks, @meteoDaniel!
- Add MAC ARM64 supoort with dependency restrictions
- Radar: Verify HDF5 responses instead of returning invalid data
- Add support for stations filtering via bbox and name
- Add support for units in distance filtering
- Rename station_name to name
- Rename filter methods to .filter_by_station_id and .filter_by_name, use same convention for bbox, filter_by_rank (previously nearby_number), filter_by_distance (nearby_distance)
- Mosmix: Use cached stations to improve performance
- Make .discover return lowercase parameters and datasets
- Use direct mapping to get a parameter set for a parameter
- Rename DwdObservationParameterSet to DwdObservationDataset as well as corresponding columns
- Merge metadata access into Request
- Repair CLI and I/O subsystem
- Add capability to export to Feather- and Parquet-files to I/O subsystem
- Deprecate support for Python 3.6
- Add
--reloadparameter towetterdienst restapifor supporting development - Improve spreadsheet export
- Increase I/O subsystem test coverage
- Make all DWD observation field names lowercase
- Make all DWD forecast (mosmix) field names lowercase
- Add Environment and Climate Change Canada API
- Rename humanize_parameters to humanize and tidy_data to tidy
- Radar: Use OPERA as data source for improved list of radar sites
- Add StationsResult and ValuesResult to allow for new workflow and connect stations and values request
- Add accessor .values to Stations class to get straight to values for a request
- Rename Stations to Request and use upper camel case e.g. DwdObservationRequest
- Add top-level API
- Fix issue with Mosmix station location
- Fix date filtering of DWD observations, where accidentally an empty dataframe was returned
- DWD: Add missing radar site "Emden" (EMD, wmo=10204)
- Mosmix stations: fix longitudes/latitudes to be decimal degrees (before they were degrees and minutes)
- Change key STATION_HEIGHT to HEIGHT, LAT to LATITUDE, LON to LONGITUDE
- Rename "Data" classes to "Values"
- Make arguments singular
- Create general Resolution and Period enumerations that can be used anywhere
- Create a full dataframe even if no values exist at requested time
- Add further attributes to the class structure
- Make dates timezone aware
- Restrict dates to isoformat
- Fix 10minutes file index interval range by adding timezone information
- Move more functionality into core classes
- Add more attributes to the core e.g. source and timezone
- Make dates of internal data timezone aware, set start date and end date to UTC
- Add issue date to Mosmix class that actually refers to the Mosmix run instead of start date and end date
- Use Result object for every data related return
- In accordance with typical naming conventions, DWDObservationSites is renamed to DWDObservationStations, the same is applied to DWDMosmixSites
- The name ELEMENT is removed and replaced by parameter while the acutal parameter set e.g. CLIMATE_SUMMARY is now found under PARAMETER_SET
- Remove StorageAdapter and its dependencies
- Methods self.collect_data() and self.collect_safe() are replaced by self.query() and self.all() and will deprecate at some point
- Bump
h5pyto version 3.1.0 in order to satisfy installation on Python 3.9
- InfluxDB export: Fix export in non-tidy format (#230). Thanks, @wetterfrosch!
- InfluxDB export: Use "quality" column as tag (#234). Thanks, @wetterfrosch!
- InfluxDB export: Use a batch size of 50000 to handle larger amounts of data (#235). Thanks, @wetterfrosch!
- Update radar examples to use
wradlib>=1.9.0. Thanks, @kmuehlbauer! - Change wherever possible column type to category
- Increase efficiency by downloading only historical files with overlapping dates if start_date and end_date are given
- Use periods dynamically depending on start and end date
- Fix inconsistency within 1 minute precipitation data where historical files have more columns
- Improve DWD PDF parser to extract quality information and select language.
Also, add an example at
example/dwd_describe_fields.pyas well as respective documentation.
- Upgrade to dateparser-1.0.0. Thanks, @steffen746, @noviluni and @Gallaecio!
This fixes a problem with timezones on Windows. The reason is that
Windows has no zoneinfo database and
tzlocalswitched frompytztotzinfo. #222
- CLI: Obtain "--tidy" argument from command line
- Extend MOSMIX support to equal the API of observations
- DWDObservationSites now filters for those stations which have a file on the server
- DWDObservationData now also takes an individual parameter independent of the pre-configured DWD datasets by using DWDObservationParameter or similar names e.g. "precipitation_height"
- Newly introduced coexistence of DWDObservationParameter and DWDObservationParameterSet to address parameter sets as well as individual parameters
- Imports are changed to submodule thus now one has to import everything from wetterdienst.dwd
- Renaming of time_resolution to resolution, period_type to period, several other relabels
- Large refactoring
- Make period type in DWDObservationData and cli optional
- Activate SQL querying again by using DuckDB 0.2.2.dev254. Thanks, @Mytherin!
- Fix coercion of integers with nans
- Fix problem with storing IntegerArrays in HDF
- Rename
DWDStationRequesttoDWDObservationData - Add
DWDObservationSitesAPI wrapper to acquire station information - Move
discover_climate_observationstoDWDObservationMetadata.discover_parameters - Add PDF-based
DWDObservationMetadata.describe_fields() - Upgrade Docker images to Python 3.8.6
- Move intermediate storage of HDF out of data collection
- Fix bug with date filtering for empty/no station data for a given parameter
- Radar data: Add non-RADOLAN data acquisition
- Add TTL-based persistent caching using dogpile.cache
- Add
example/radolan.pyand adjust documentation - Export dataframe to different data sinks like SQLite, DuckDB, InfluxDB and CrateDB
- Query results with SQL, based on in-memory DuckDB
- Split get_nearby_stations into two functions, get_nearby_stations_by_number and get_nearby_stations_by_distance
- Add MOSMIX client and parser. Thanks, @jlewis91!
- Add basic HTTP API
- Add test for Jupyter notebook
- Add function to discover available climate observations (time resolution, parameter, period type)
- Make the CLI work again and add software tests to prevent future havocs
- Use Sphinx Material theme for documentation
- Fix typo in enumeration for TimeResolution.MINUTES_10
- Add test for Jupyter notebook
- Add function to discover available climate observations (time resolution, parameter, period type)
- enhance usage of get_nearby_stations to check for availability
- output of get_nearby_stations is now a slice of meta_data DataFrame output
- add RADOLAN support
- change module and function naming in accordance with RADOLAN
- extend DWDObservationData to take multiple parameters as request
- add documentation at readthedocs.io
- [cli] Adjust methods to work with multiple parameters
- establish code style black
- setup nox session that can be used to run black via nox -s black for one of the supported Python versions
- add option for data collection to tidy the DataFrame (properly reshape) with the "tidy_data" keyword and set it to be used as default
- fix integer type casting for cases with nans in the column/series
- fix humanizing of column names for tidy data
- [cli] Add geospatial filtering by distance.
- [cli] Filter stations by station identifiers.
- [cli] Add GeoJSON output format for station data.
- improvements to parsing high resolution data by setting specific datetime formats and changing to concurrent.futures
- fix na value detection for cases where cells have leading and trailing whitespace
- change column name mapping to more explicit one with columns being individually addressable
- add full column names for every individual parameter
- more specific type casting for integer fields and string fields
- [cli] Add geospatial filtering by number of nearby stations.
- Simplify release pipeline
- small updates to readme
- change updating "parallel" argument to be done after parameter parsing to prevent mistakenly not found parameter
- remove find_all_match_strings function and extract functionality to individual operations
- parameter, time resolution and period type can now also be passed as strings of the enumerations e.g. "climate_summary" or "CLIMATE_SUMMARY" for Parameter.CLIMATE_SUMMARY
- enable selecting nearby stations by distance rather then by number of stations
- initial release
- update README.md
- update example notebook
- add Gh Action for release
- rename library