Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Get OSM objects by node/way id numbers #249

Open
mtennekes opened this issue Mar 8, 2022 · 23 comments
Open

[FEATURE] Get OSM objects by node/way id numbers #249

mtennekes opened this issue Mar 8, 2022 · 23 comments

Comments

@mtennekes
Copy link

Is your feature request related to a problem? Please describe.
I would like to know if there is an osmextract variant for osmdata::opq_osm_id

Describe the solution you'd like
I have a large number of OSM way ids for which I would like to obtain the spatial objects. I already downloaded the OSM data for the region of interest.

Describe alternatives you've considered
osmdata::opq_osm_id gives me a time-out error

Additional context
None

@mtennekes mtennekes changed the title [FEATURE] [FEATURE] Get OSM objects by node/way id numbers Mar 8, 2022
@agila5
Copy link
Collaborator

agila5 commented Mar 8, 2022

Hi @mtennekes! If you know that those ways lay in a given OSM extract, I think that you can run something like this:

# packages
library(osmextract)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright.
#> Check the package website, https://docs.ropensci.org/osmextract/, for more details.

# define 5 osm_ids for 5 ways in the Isle of Wight
my_osm_id <- c("123588694", "533585723", "61240667", "570495917", "566733078")

# read + subset those ways
iow_subset <- oe_get(
  place = "Isle of Wight", 
  query = paste0("SELECT * FROM lines WHERE osm_id IN (", paste0(my_osm_id, collapse = ","), ")")
)
#> The input place was matched with: Isle of Wight
#> The chosen file was already detected in the download directory. Skip downloading.
#> The corresponding gpkg file was already detected. Skip vectortranslate operations.
#> Reading query `SELECT * FROM lines WHERE osm_id IN (123588694,533585723,61240667,570495917,566733078)' from data source `C:\Users\andrea.gilardi\Documents\osm-data\geofabrik_isle-of-wight-latest.gpkg' 
#>   using driver `GPKG'
#> Simple feature collection with 5 features and 9 fields
#> Geometry type: LINESTRING
#> Dimension:     XY
#> Bounding box:  xmin: -1.495138 ymin: 50.65387 xmax: -1.143836 ymax: 50.74988
#> Geodetic CRS:  WGS 84

Created on 2022-03-08 by the reprex package (v2.0.1)

If you have already downloaded the OSM data using any other method, check oe_read(). Hope that helps.

@mtennekes
Copy link
Author

Thanks! Not successful yet.

This is what I got with osmdata (to compare with):

ids = c(6429507L, 6439305L, 6439218L, 6439342L, 6437077L, 6437113L, 6437085L, 6437055L, 6437019L, 6437007L) 

library(osmdata)
dat1 <- opq_osm_id (type = "way", id = ids) %>%
	opq_string () %>%
	osmdata_sf ()
dat1

which returns:

Object of class 'osmdata' with:
                 $bbox : 
        $overpass_call : The call submitted to the overpass API
                 $meta : metadata including timestamp and version numbers
           $osm_points : 'sf' Simple Features Collection with 57 points
            $osm_lines : 'sf' Simple Features Collection with 10 linestrings
         $osm_polygons : 'sf' Simple Features Collection with 0 polygons
       $osm_multilines : NULL
    $osm_multipolygons : NULL

So all 10 linestrings are there.

With your method:

iow_subset <- oe_get(
	place = "Netherlands",
	query = paste0("SELECT * FROM lines WHERE osm_id IN (", paste0(ids, collapse = ","), ")")
)
iow_subset

with output

The input place was matched with: Netherlands
The chosen file was already detected in the download directory. Skip downloading.
The corresponding gpkg file was already detected. Skip vectortranslate operations.
Reading query `SELECT * FROM lines WHERE osm_id IN (6429507,6439305,6439218,6439342,6437077,6437113,6437085,6437055,6437019,6437007)' from data source `/home/mtes/local/data/geofabrik_netherlands-latest.gpkg' using driver `GPKG'
Simple feature collection with 0 features and 9 fields
Bounding box:  xmin: NA ymin: NA xmax: NA ymax: NA
Geodetic CRS:  WGS 84

and

Simple feature collection with 0 features and 9 fields
Bounding box:  xmin: NA ymin: NA xmax: NA ymax: NA
Geodetic CRS:  WGS 84
 [1] osm_id     name       highway    waterway   aerialway  barrier    man_made   z_order    other_tags geometry  
<0 rows> (or 0-length row.names)

I downloaded oe_match("Netherlands").

Could you take a look @agila5 ? Thanks.

@agila5
Copy link
Collaborator

agila5 commented Mar 9, 2022

Hi @mtennekes. I'm sorry (and maybe I miss something obvious) but I cannot run your osmdata example:

library(osmdata)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright
ids = c(6429507L, 6439305L, 6439218L, 6439342L, 6437077L, 6437113L, 6437085L, 6437055L, 6437019L, 6437007L) 
dat1 <- opq_osm_id (type = "way", id = ids) %>%
  opq_string () %>%
  osmdata_sf ()
#> Error in opq_osm_id(type = "way", id = ids): id must be character or numeric.

Created on 2022-03-09 by the reprex package (v2.0.1)

Session info
sessioninfo::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Windows 10 x64 (build 19042)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  English_United Kingdom.1252
#>  ctype    English_United Kingdom.1252
#>  tz       Europe/Berlin
#>  date     2022-03-09
#>  pandoc   2.16.2 @ C:/PROGRA~1/Pandoc/ (via rmarkdown)
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version   date (UTC) lib source
#>  backports     1.4.1     2021-12-13 [1] CRAN (R 4.1.2)
#>  cli           3.2.0     2022-02-14 [1] CRAN (R 4.1.2)
#>  crayon        1.5.0     2022-02-14 [1] CRAN (R 4.1.2)
#>  curl          4.3.2     2021-06-23 [1] CRAN (R 4.1.1)
#>  digest        0.6.29    2021-12-01 [1] CRAN (R 4.1.2)
#>  ellipsis      0.3.2     2021-04-29 [1] CRAN (R 4.1.1)
#>  evaluate      0.14      2019-05-28 [1] CRAN (R 4.1.1)
#>  fansi         1.0.2     2022-01-14 [1] CRAN (R 4.1.2)
#>  fastmap       1.1.0     2021-01-25 [1] CRAN (R 4.1.1)
#>  fs            1.5.2     2021-12-08 [1] CRAN (R 4.1.2)
#>  generics      0.1.2     2022-01-31 [1] CRAN (R 4.1.2)
#>  glue          1.6.2     2022-02-24 [1] CRAN (R 4.1.2)
#>  highr         0.9       2021-04-16 [1] CRAN (R 4.1.1)
#>  htmltools     0.5.2     2021-08-25 [1] CRAN (R 4.1.1)
#>  httr          1.4.2     2020-07-20 [1] CRAN (R 4.1.1)
#>  jsonlite      1.8.0     2022-02-22 [1] CRAN (R 4.1.2)
#>  knitr         1.37      2021-12-16 [1] CRAN (R 4.1.2)
#>  lattice       0.20-45   2021-09-22 [2] CRAN (R 4.1.2)
#>  lifecycle     1.0.1     2021-09-24 [1] CRAN (R 4.1.1)
#>  lubridate     1.8.0     2021-10-07 [1] CRAN (R 4.1.1)
#>  magrittr      2.0.2     2022-01-26 [1] CRAN (R 4.1.2)
#>  osmdata     * 0.1.9.001 2022-03-09 [1] Github (ropensci/osmdata@bf0d639)
#>  pillar        1.7.0     2022-02-01 [1] CRAN (R 4.1.2)
#>  pkgconfig     2.0.3     2019-09-22 [1] CRAN (R 4.1.1)
#>  purrr         0.3.4     2020-04-17 [1] CRAN (R 4.1.1)
#>  R.cache       0.15.0    2021-04-30 [1] CRAN (R 4.1.1)
#>  R.methodsS3   1.8.1     2020-08-26 [1] CRAN (R 4.1.1)
#>  R.oo          1.24.0    2020-08-26 [1] CRAN (R 4.1.1)
#>  R.utils       2.11.0    2021-09-26 [1] CRAN (R 4.1.1)
#>  R6            2.5.1     2021-08-19 [1] CRAN (R 4.1.1)
#>  Rcpp          1.0.8     2022-01-13 [1] CRAN (R 4.1.2)
#>  reprex        2.0.1     2021-08-05 [1] CRAN (R 4.1.1)
#>  rlang         1.0.2     2022-03-04 [1] CRAN (R 4.1.2)
#>  rmarkdown     2.11      2021-09-14 [1] CRAN (R 4.1.1)
#>  rstudioapi    0.13      2020-11-12 [1] CRAN (R 4.1.1)
#>  rvest         1.0.2     2021-10-16 [1] CRAN (R 4.1.1)
#>  sessioninfo   1.2.2     2021-12-06 [1] CRAN (R 4.1.2)
#>  sp            1.4-6     2021-11-14 [1] CRAN (R 4.1.2)
#>  stringi       1.7.6     2021-11-29 [1] CRAN (R 4.1.2)
#>  stringr       1.4.0     2019-02-10 [1] CRAN (R 4.1.1)
#>  styler        1.6.2     2021-09-23 [1] CRAN (R 4.1.1)
#>  tibble        3.1.6     2021-11-07 [1] CRAN (R 4.1.2)
#>  utf8          1.2.2     2021-07-24 [1] CRAN (R 4.1.1)
#>  vctrs         0.3.8     2021-04-29 [1] CRAN (R 4.1.1)
#>  withr         2.4.3     2021-11-30 [1] CRAN (R 4.1.2)
#>  xfun          0.29      2021-12-14 [1] CRAN (R 4.1.2)
#>  xml2          1.3.3     2021-11-30 [1] CRAN (R 4.1.2)
#>  yaml          2.2.2     2022-01-25 [1] CRAN (R 4.1.2)
#> 
#>  [1] C:/Users/andre/Documents/R/win-library/4.1
#>  [2] C:/Program Files/R/R-4.1.2/library
#> 
#> ------------------------------------------------------------------------------

If I convert the input to numeric (i.e. non integer) or character I get a different error:

library(osmdata)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright
ids = c(6429507L, 6439305L, 6439218L, 6439342L, 6437077L, 6437113L, 6437085L, 6437055L, 6437019L, 6437007L) 
dat1 <- opq_osm_id (type = "way", id = as.numeric(ids)) %>%
  opq_string () %>%
  osmdata_sf ()
#> Error in opq_osm_id(type = "way", id = as.numeric(ids)): Only a single id may be entered.

dat1 <- opq_osm_id (type = "way", id = as.character(ids)) %>%
  opq_string () %>%
  osmdata_sf ()
#> Error in opq_osm_id(type = "way", id = as.character(ids)): Only a single id may be entered.

Created on 2022-03-09 by the reprex package (v2.0.1)

On the other hand, if I consider just a unique osm_id then everything works fine (vectortranslate_options is just a faster version of query argument in this case) :

library(osmdata)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright
library(osmextract)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright.
#> Check the package website, https://docs.ropensci.org/osmextract/, for more details.

ids = 6429507
dat1 <- opq_osm_id(id = ids, type = "way") %>% 
  opq_string () %>%
  osmdata_sf ()
dat1
#> Object of class 'osmdata' with:
#>                  $bbox : 
#>         $overpass_call : The call submitted to the overpass API
#>                  $meta : metadata including timestamp and version numbers
#>            $osm_points : 'sf' Simple Features Collection with 33 points
#>             $osm_lines : 'sf' Simple Features Collection with 1 linestrings
#>          $osm_polygons : 'sf' Simple Features Collection with 0 polygons
#>        $osm_multilines : NULL
#>     $osm_multipolygons : NULL

dat2 <- oe_get(
  place = "Neatherlands", 
  vectortranslate_options = c("-where", "osm_id = '6429507'")
)
#> The input place was matched with: Netherlands
#> The chosen file was already detected in the download directory. Skip downloading.
#> Start with the vectortranslate operations on the input file!
#> 0...10...20...30...40...50...60...70...80...90...100 - done.
#> Finished the vectortranslate operations on the input file!
#> Reading layer `lines' from data source 
#>   `C:\Users\andre\Documents\osm-data\geofabrik_netherlands-latest.gpkg' 
#>   using driver `GPKG'
#> Simple feature collection with 1 feature and 9 fields
#> Geometry type: LINESTRING
#> Dimension:     XY
#> Bounding box:  xmin: 6.421445 ymin: 52.62376 xmax: 6.434561 ymax: 52.64008
#> Geodetic CRS:  WGS 84

# Quick plot
par(mfrow = c(1, 2))
plot(dat1$osm_lines$geometry)
plot(dat2$geometry)

Created on 2022-03-09 by the reprex package (v2.0.1)

Do you have any idea what's going on? Can you paste the version of osmextract and osmdata packages?

@mtennekes
Copy link
Author

Very strange. Also your dat2 example resulted in a 0-row sf object. Also strange that osmdata didn't work for you.

 osmdata     * 0.1.8   2021-10-17 [1] CRAN (R 4.1.2)
 osmextract  * 0.4.0   2021-10-27 [1] CRAN (R 4.1.2)

Which versions did you use?
Any other things we can test?

@agila5
Copy link
Collaborator

agila5 commented Mar 9, 2022

You're right. I don't get an error with version 0.1.8

remotes::install_version("osmdata", "0.1.8", quiet = TRUE)
library(osmdata)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright
ids = c(6429507L, 6439305L, 6439218L, 6439342L, 6437077L, 6437113L, 6437085L, 6437055L, 6437019L, 6437007L) 
dat1 <- opq_osm_id (type = "way", id = ids) %>%
  opq_string () %>%
  osmdata_sf ()
dat1
#> Object of class 'osmdata' with:
#>                  $bbox : 
#>         $overpass_call : The call submitted to the overpass API
#>                  $meta : metadata including timestamp and version numbers
#>            $osm_points : 'sf' Simple Features Collection with 57 points
#>             $osm_lines : 'sf' Simple Features Collection with 10 linestrings
#>          $osm_polygons : 'sf' Simple Features Collection with 0 polygons
#>        $osm_multilines : NULL
#>     $osm_multipolygons : NULL

Created on 2022-03-09 by the reprex package (v2.0.1)

but I get an error with the current CRAN (i.e. 0.1.9)

remotes::install_cran("osmdata", quiet = TRUE)
#> Installing 1 packages: osmdata
#> package 'osmdata' successfully unpacked and MD5 sums checked
library(osmdata)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright
ids = c(6429507L, 6439305L, 6439218L, 6439342L, 6437077L, 6437113L, 6437085L, 6437055L, 6437019L, 6437007L) 
dat1 <- opq_osm_id (type = "way", id = ids) %>%
  opq_string () %>%
  osmdata_sf ()
#> Error in opq_osm_id(type = "way", id = ids): id must be character or numeric.
dat1
#> Error in eval(expr, envir, enclos): object 'dat1' not found

Created on 2022-03-09 by the reprex package (v2.0.1)

and github version (i.e. 0.1.9.001):

remotes::install_github("ropensci/osmdata", quiet = TRUE)
library(osmdata)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright
ids = c(6429507L, 6439305L, 6439218L, 6439342L, 6437077L, 6437113L, 6437085L, 6437055L, 6437019L, 6437007L) 
dat1 <- opq_osm_id (type = "way", id = ids) %>%
  opq_string () %>%
  osmdata_sf ()
#> Error in opq_osm_id(type = "way", id = ids): id must be character or numeric.
dat1
#> Error in eval(expr, envir, enclos): object 'dat1' not found

Created on 2022-03-09 by the reprex package (v2.0.1)

Tomorrow I will create a new issue in the osmdata repo. [Done, see: https://github.com/ropensci/osmdata/issues/268]

I still don't understand what's the problem with osmextract but I think it might be related to #245. Could you try installing the github version of osmextract, clean all files in the download directory and try again?

@mtennekes
Copy link
Author

I've tried the GH version, but without success. Without knowledge of the file types, I noticed that gpkg is some kind of processed file. When I first start the query it returns the message "Start with the vectortranslate operations on the input file!" and takes a bit of time. When I rerun the query, it returns the (in my case empty) output immediately.

What I notices is that it gives warnings about being unable to open database:

> iow_subset <- oe_get(
+ 	place = "Netherlands",
+ 	query = paste0("SELECT * FROM lines WHERE osm_id IN (6429507)")
+ )
The input place was matched with: Netherlands
The chosen file was already detected in the download directory. Skip downloading.
Start with the vectortranslate operations on the input file!
0...10...20...30...40...50...60...70...80...90...100 - done.
Finished the vectortranslate operations on the input file!
Reading query `SELECT * FROM lines WHERE osm_id IN (6429507)' from data source `/home/mtes/local/data/geofabrik_netherlands-latest.gpkg' using driver `GPKG'
Simple feature collection with 0 features and 9 fields
Bounding box:  xmin: NA ymin: NA xmax: NA ymax: NA
Geodetic CRS:  WGS 84
There were 50 or more warnings (use warnings() to see the first 50)
> warnings()
Warning messages:
1: In CPL_gdalvectortranslate(source, destination, options,  ... :
  GDAL Error 4: sqlite3_open(/home/mtes/local/data/geofabrik_netherlands-latest.gpkg) failed: unable to open database file
2: In CPL_gdalvectortranslate(source, destination, options,  ... :
  GDAL Error 4: sqlite3_open(/home/mtes/local/data/geofabrik_netherlands-latest.gpkg) failed: unable to open database file
3: In CPL_gdalvectortranslate(source, destination, options,  ... :
  GDAL Error 1: Cannot write in temporary node file ./osm_tmp_nodes_6185_3 : No such file or directory
4: In CPL_gdalvectortranslate(source, destination, options,  ... :

@agila5
Copy link
Collaborator

agila5 commented Mar 14, 2022

And what happens when you run

osmextract::oe_get(
  place = "Netherlands", 
  query = paste0("SELECT * FROM lines WHERE osm_id IN (6429507)"), 
  download_directory = tempdir()
)

in a fresh R session?

@mtennekes
Copy link
Author

Doesn't work unfortunately:

> warnings()
Warning messages:
1: In CPL_gdalvectortranslate(source, destination, options,  ... :
  GDAL Error 1: Cannot write in temporary node file ./osm_tmp_nodes_21125_1 : No such file or directory
2: In CPL_gdalvectortranslate(source, destination, options,  ... :
  GDAL Error 1: Cannot write in temporary node file ./osm_tmp_nodes_21125_1 : No such file or directory
3: In CPL_gdalvectortranslate(source, destination, options,  ... :
  GDAL Error 1: Cannot write in temporary node file ./osm_tmp_nodes_21125_1 : No such file or directory
4: In CPL_gdalvectortranslate(source, destination, options,  ... :
  GDAL Error 1: Cannot write in temporary node file ./osm_tmp_nodes_21125_1 : No such file or directory
5: In CPL_gdalvectortranslate(source, destination, options,  ... :

Any other tests I can run?

@Robinlovelace
Copy link
Member

What version of GDAL are you both running? I'm on 3.4.0

library(sf)
#> Linking to GEOS 3.10.1, GDAL 3.4.0, PROJ 8.2.0; sf_use_s2() is TRUE

Created on 2022-03-14 by the reprex package (v2.0.1)

@Robinlovelace
Copy link
Member

Update, good news, it works for me 👍

osmextract::oe_get(
+     place = "Netherlands", 
+     query = paste0("SELECT * FROM lines WHERE osm_id IN (6429507)"), 
+     download_directory = tempdir()
+ )
The input place was matched with: Netherlands
You are trying to download a file from https://download.geofabrik.de/europe/netherlands-latest.osm.pbf
This is a large file (1074 MB)!
Are you sure that you want to download it? 

1: Yes
2: No

Selection: 1
  |========================================================================================================================| 100%
File downloaded!
Start with the vectortranslate operations on the input file!
0...10...20...30...40...50...60...70...80...90...100 - done.
Finished the vectortranslate operations on the input file!
Reading query `SELECT * FROM lines WHERE osm_id IN (6429507)' from data source `/tmp/RtmpFWe8gC/geofabrik_netherlands-latest.gpkg' using driver `GPKG'
Simple feature collection with 1 feature and 9 fields
Geometry type: LINESTRING
Dimension:     XY
Bounding box:  xmin: 6.421445 ymin: 52.62376 xmax: 6.434561 ymax: 52.64008
Geodetic CRS:  WGS 84

@agila5
Copy link
Collaborator

agila5 commented Mar 15, 2022

What version of GDAL are you both running? I'm on 3.4.0

I run 3.2.1

Any other tests I can run?

Can you run the examples in sf::gdal_utils() and report back? For example, what is the output of the following?

in_file <- system.file("shape/storms_xyz.shp", package="sf")
out_file <- paste0(tempfile(), ".gpkg")
gdal_utils(
  util = "vectortranslate",
  source = in_file,
  destination = out_file, # output format must be specified for GDAL < 2.3
  options = c("-f", "GPKG")
)
gdal_utils(
  util = "vectortranslate",
  source = in_file,
  destination = out_file,
  options = c(
  "-f", "GPKG", # output file format for GDAL < 2.3
  "-s_srs", "EPSG:4326", # input file SRS
  "-t_srs", "EPSG:2264", # output file SRS
  "-overwrite"
  )
)
st_read(out_file)
st_read(in_file)

@mtennekes
Copy link
Author

I'm using:

Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 6.3.1; sf_use_s2() is TRUE

Now rebooting to Windows to test...

@mtennekes
Copy link
Author

Yes, Windows works!

With:

Linking to GEOS 3.9.1, GDAL 3.2.1, PROJ 7.2.1; sf_use_s2() is TRUE

@agila5
Copy link
Collaborator

agila5 commented Mar 15, 2022

And that example fails on Ubuntu with GDAL 3.0.4?

@Robinlovelace
Copy link
Member

Looks like a minimum GDAL version to me.

@agila5
Copy link
Collaborator

agila5 commented Mar 15, 2022

I don't think so since the following

osmextract::oe_get(
  place = "Isle of Wight", 
  query = "SELECT * FROM lines WHERE osm_id = '413'", 
  download_directory = tempdir()
)
#> Warning: st_crs<- : replacing crs does not reproject data; use st_transform for
#> that
#> The input place was matched with: Isle of Wight
#>   |                                                                              |                                                                      |   0%  |                                                                              |                                                                      |   1%  |                                                                              |=                                                                     |   1%  |                                                                              |=                                                                     |   2%  |                                                                              |==                                                                    |   2%  |                                                                              |==                                                                    |   3%  |                                                                              |==                                                                    |   4%  |                                                                              |===                                                                   |   4%  |                                                                              |===                                                                   |   5%  |                                                                              |====                                                                  |   5%  |                                                                              |====                                                                  |   6%  |                                                                              |=====                                                                 |   7%  |                                                                              |=====                                                                 |   8%  |                                                                              |======                                                                |   8%  |                                                                              |======                                                                |   9%  |                                                                              |=======                                                               |   9%  |                                                                              |=======                                                               |  10%  |                                                                              |=======                                                               |  11%  |                                                                              |========                                                              |  11%  |                                                                              |========                                                              |  12%  |                                                                              |=========                                                             |  12%  |                                                                              |=========                                                             |  13%  |                                                                              |==========                                                            |  14%  |                                                                              |==========                                                            |  15%  |                                                                              |===========                                                           |  15%  |                                                                              |===========                                                           |  16%  |                                                                              |============                                                          |  17%  |                                                                              |============                                                          |  18%  |                                                                              |=============                                                         |  18%  |                                                                              |=============                                                         |  19%  |                                                                              |==============                                                        |  19%  |                                                                              |==============                                                        |  20%  |                                                                              |==============                                                        |  21%  |                                                                              |===============                                                       |  21%  |                                                                              |===============                                                       |  22%  |                                                                              |================                                                      |  22%  |                                                                              |================                                                      |  23%  |                                                                              |=================                                                     |  24%  |                                                                              |=================                                                     |  25%  |                                                                              |==================                                                    |  25%  |                                                                              |==================                                                    |  26%  |                                                                              |===================                                                   |  26%  |                                                                              |===================                                                   |  27%  |                                                                              |===================                                                   |  28%  |                                                                              |====================                                                  |  28%  |                                                                              |====================                                                  |  29%  |                                                                              |=====================                                                 |  29%  |                                                                              |=====================                                                 |  30%  |                                                                              |=====================                                                 |  31%  |                                                                              |======================                                                |  31%  |                                                                              |======================                                                |  32%  |                                                                              |=======================                                               |  32%  |                                                                              |=======================                                               |  33%  |                                                                              |=======================                                               |  34%  |                                                                              |========================                                              |  34%  |                                                                              |========================                                              |  35%  |                                                                              |=========================                                             |  35%  |                                                                              |=========================                                             |  36%  |                                                                              |==========================                                            |  37%  |                                                                              |==========================                                            |  38%  |                                                                              |===========================                                           |  38%  |                                                                              |===========================                                           |  39%  |                                                                              |============================                                          |  39%  |                                                                              |============================                                          |  40%  |                                                                              |============================                                          |  41%  |                                                                              |=============================                                         |  41%  |                                                                              |=============================                                         |  42%  |                                                                              |==============================                                        |  42%  |                                                                              |==============================                                        |  43%  |                                                                              |===============================                                       |  44%  |                                                                              |===============================                                       |  45%  |                                                                              |================================                                      |  45%  |                                                                              |================================                                      |  46%  |                                                                              |=================================                                     |  46%  |                                                                              |=================================                                     |  47%  |                                                                              |=================================                                     |  48%  |                                                                              |==================================                                    |  48%  |                                                                              |==================================                                    |  49%  |                                                                              |===================================                                   |  49%  |                                                                              |===================================                                   |  50%  |                                                                              |===================================                                   |  51%  |                                                                              |====================================                                  |  51%  |                                                                              |====================================                                  |  52%  |                                                                              |=====================================                                 |  52%  |                                                                              |=====================================                                 |  53%  |                                                                              |=====================================                                 |  54%  |                                                                              |======================================                                |  54%  |                                                                              |======================================                                |  55%  |                                                                              |=======================================                               |  55%  |                                                                              |=======================================                               |  56%  |                                                                              |========================================                              |  57%  |                                                                              |========================================                              |  58%  |                                                                              |=========================================                             |  58%  |                                                                              |=========================================                             |  59%  |                                                                              |==========================================                            |  59%  |                                                                              |==========================================                            |  60%  |                                                                              |==========================================                            |  61%  |                                                                              |===========================================                           |  61%  |                                                                              |===========================================                           |  62%  |                                                                              |============================================                          |  62%  |                                                                              |============================================                          |  63%  |                                                                              |=============================================                         |  64%  |                                                                              |=============================================                         |  65%  |                                                                              |==============================================                        |  65%  |                                                                              |==============================================                        |  66%  |                                                                              |===============================================                       |  67%  |                                                                              |===============================================                       |  68%  |                                                                              |================================================                      |  68%  |                                                                              |================================================                      |  69%  |                                                                              |=================================================                     |  69%  |                                                                              |=================================================                     |  70%  |                                                                              |=================================================                     |  71%  |                                                                              |==================================================                    |  71%  |                                                                              |==================================================                    |  72%  |                                                                              |===================================================                   |  72%  |                                                                              |===================================================                   |  73%  |                                                                              |====================================================                  |  74%  |                                                                              |====================================================                  |  75%  |                                                                              |=====================================================                 |  75%  |                                                                              |=====================================================                 |  76%  |                                                                              |======================================================                |  76%  |                                                                              |======================================================                |  77%  |                                                                              |======================================================                |  78%  |                                                                              |=======================================================               |  78%  |                                                                              |=======================================================               |  79%  |                                                                              |========================================================              |  79%  |                                                                              |========================================================              |  80%  |                                                                              |========================================================              |  81%  |                                                                              |=========================================================             |  81%  |                                                                              |=========================================================             |  82%  |                                                                              |==========================================================            |  82%  |                                                                              |==========================================================            |  83%  |                                                                              |==========================================================            |  84%  |                                                                              |===========================================================           |  84%  |                                                                              |===========================================================           |  85%  |                                                                              |============================================================          |  85%  |                                                                              |============================================================          |  86%  |                                                                              |=============================================================         |  87%  |                                                                              |=============================================================         |  88%  |                                                                              |==============================================================        |  88%  |                                                                              |==============================================================        |  89%  |                                                                              |===============================================================       |  89%  |                                                                              |===============================================================       |  90%  |                                                                              |===============================================================       |  91%  |                                                                              |================================================================      |  91%  |                                                                              |================================================================      |  92%  |                                                                              |=================================================================     |  92%  |                                                                              |=================================================================     |  93%  |                                                                              |==================================================================    |  94%  |                                                                              |==================================================================    |  95%  |                                                                              |===================================================================   |  95%  |                                                                              |===================================================================   |  96%  |                                                                              |====================================================================  |  97%  |                                                                              |====================================================================  |  98%  |                                                                              |===================================================================== |  98%  |                                                                              |===================================================================== |  99%  |                                                                              |======================================================================|  99%  |                                                                              |======================================================================| 100%
#> File downloaded!
#> Start with the vectortranslate operations on the input file!
#> 0...10...20...30...40...50...60...70...80...90...100 - done.
#> Finished the vectortranslate operations on the input file!
#> Reading query `SELECT * FROM lines WHERE osm_id = '413'' from data source `/tmp/RtmpQWh3o3/geofabrik_isle-of-wight-latest.gpkg' using driver `GPKG'
#> Simple feature collection with 1 feature and 9 fields
#> Geometry type: LINESTRING
#> Dimension:     XY
#> Bounding box:  xmin: -1.083348 ymin: 50.68616 xmax: -1.082644 ymax: 50.68625
#> CRS:           4326

Created on 2022-03-15 by the reprex package (v2.0.1)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 3.6.3 (2020-02-29)
#>  os       Ubuntu 18.04.6 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language en_GB:en
#>  collate  en_GB.UTF-8
#>  ctype    en_GB.UTF-8
#>  tz       Europe/Rome
#>  date     2022-03-15
#>  pandoc   2.17.1.1 @ /home/ag21052/.local/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date (UTC) lib source
#>  assertthat    0.2.1      2019-03-21 [3] CRAN (R 3.5.3)
#>  backports     1.4.1      2021-12-13 [3] CRAN (R 3.6.3)
#>  class         7.3-20     2022-01-13 [4] CRAN (R 3.6.3)
#>  classInt      0.4-3      2020-04-07 [3] CRAN (R 3.6.3)
#>  cli           3.2.0      2022-02-14 [1] CRAN (R 3.6.3)
#>  crayon        1.5.0      2022-02-14 [1] CRAN (R 3.6.3)
#>  curl          4.3.2      2021-06-23 [3] CRAN (R 3.6.3)
#>  DBI           1.1.2      2021-12-20 [3] CRAN (R 3.6.3)
#>  digest        0.6.29     2021-12-01 [1] CRAN (R 3.6.3)
#>  dplyr         1.0.8      2022-02-08 [1] CRAN (R 3.6.3)
#>  e1071         1.7-9      2021-09-16 [1] CRAN (R 3.6.3)
#>  ellipsis      0.3.2      2021-04-29 [3] CRAN (R 3.6.3)
#>  evaluate      0.15       2022-02-18 [1] CRAN (R 3.6.3)
#>  fansi         1.0.2      2022-01-14 [3] CRAN (R 3.6.3)
#>  fastmap       1.1.0      2021-01-25 [1] CRAN (R 3.6.3)
#>  fs            1.5.2      2021-12-08 [3] CRAN (R 3.6.3)
#>  generics      0.1.2      2022-01-31 [1] CRAN (R 3.6.3)
#>  glue          1.6.1      2022-01-22 [1] CRAN (R 3.6.3)
#>  highr         0.9        2021-04-16 [3] CRAN (R 3.6.3)
#>  htmltools     0.5.2      2021-08-25 [1] CRAN (R 3.6.3)
#>  httr          1.4.2      2020-07-20 [3] CRAN (R 3.6.3)
#>  KernSmooth    2.23-20    2021-05-03 [4] CRAN (R 3.6.3)
#>  knitr         1.37       2021-12-16 [1] CRAN (R 3.6.3)
#>  lifecycle     1.0.1      2021-09-24 [3] CRAN (R 3.6.3)
#>  magrittr      2.0.2      2022-01-26 [1] CRAN (R 3.6.3)
#>  osmextract    0.4.0.9000 2022-03-15 [1] local
#>  pillar        1.7.0      2022-02-01 [1] CRAN (R 3.6.3)
#>  pkgconfig     2.0.3      2019-09-22 [3] CRAN (R 3.6.1)
#>  proxy         0.4-26     2021-06-07 [1] CRAN (R 3.6.3)
#>  purrr         0.3.4      2020-04-17 [3] CRAN (R 3.6.3)
#>  R.cache       0.15.0     2021-04-30 [1] CRAN (R 3.6.3)
#>  R.methodsS3   1.8.1      2020-08-26 [3] CRAN (R 3.6.3)
#>  R.oo          1.24.0     2020-08-26 [3] CRAN (R 3.6.3)
#>  R.utils       2.11.0     2021-09-26 [3] CRAN (R 3.6.3)
#>  R6            2.5.1      2021-08-19 [1] CRAN (R 3.6.3)
#>  Rcpp          1.0.8      2022-01-13 [3] CRAN (R 3.6.3)
#>  reprex        2.0.1      2021-08-05 [3] CRAN (R 3.6.3)
#>  rlang         1.0.1      2022-02-03 [1] CRAN (R 3.6.3)
#>  rmarkdown     2.11       2021-09-14 [1] CRAN (R 3.6.3)
#>  rstudioapi    0.13       2020-11-12 [3] CRAN (R 3.6.3)
#>  sessioninfo   1.2.2      2021-12-06 [3] CRAN (R 3.6.3)
#>  sf            1.0-5      2021-12-17 [3] CRAN (R 3.6.3)
#>  stringi       1.7.6      2021-11-29 [3] CRAN (R 3.6.3)
#>  stringr       1.4.0      2019-02-10 [3] CRAN (R 3.5.2)
#>  styler        1.6.2      2021-09-23 [3] CRAN (R 3.6.3)
#>  tibble        3.1.6      2021-11-07 [1] CRAN (R 3.6.3)
#>  tidyselect    1.1.2      2022-02-21 [3] CRAN (R 3.6.3)
#>  units         0.8-0      2022-02-05 [1] CRAN (R 3.6.3)
#>  utf8          1.2.2      2021-07-24 [3] CRAN (R 3.6.3)
#>  vctrs         0.3.8      2021-04-29 [3] CRAN (R 3.6.3)
#>  withr         2.4.3      2021-11-30 [3] CRAN (R 3.6.3)
#>  xfun          0.29       2021-12-14 [3] CRAN (R 3.6.3)
#>  yaml          2.3.5      2022-02-21 [1] CRAN (R 3.6.3)
#> 
#>  [1] /home/ag21052/R/x86_64-pc-linux-gnu-library/3.6
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

works on my VM running

sf::sf_extSoftVersion()
#>           GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
#>        "3.7.1"        "2.4.2"        "5.2.0"         "true"        "false" 
#>           PROJ 
#>        "5.2.0"

Created on 2022-03-15 by the reprex package (v2.0.1)

@mtennekes, if you want we could set a (skype/zoom/meet/whatever) call to check the problem together. I don't have any further suggestion unfortunately on why that doesn't work on your pc.

@mtennekes
Copy link
Author

Thanks @agila5 I got the job done via Windows so there is no urgency anymore, but it would be nice to find out the issue. So a zoom would be nice. Then we can also share some research ideas. Also interested @Robinlovelace ?

@Robinlovelace
Copy link
Member

Sure! My n. 1 priority would be upstream improvements to providers: it makes no sense that I can download West Yorkshire but not a region in India. Something @bdon can help with? Apologies for the out-of-the-blue tagging but Dustin Carlino suggested you may be able to help with something like this. I'm well up for a call in any case 👍

@bdon
Copy link

bdon commented Mar 16, 2022

@Robinlovelace you're welcome to link users to download PBF files via Web UI at http://protomaps.com/downloads - this is a free service that lets you extract any rectangular or polygonal region in the world.

The Web UI service however is rate-limited, and I provide programmatic access only for users in private and not to the general public. The reason for this is that the Protomaps Extracts service is generated 100% on demand, snapshotting a minute-level transaction, so can take a long time just to perform the indexing before streaming the PBF bytes out.

The other upstream providers have pre-specified areas that may be updated once a day or week, but since those are static files they're much less intensive to serve over the web.

My suggestion is: add docs and a first-class way for users to download a PBF "by hand", place it in the persistent directory and then reference it in the code. This would add some slight friction to the user experience but would let you effectively support any PBF source out there.

@Robinlovelace
Copy link
Member

My suggestion is: add docs and a first-class way for users to download a PBF "by hand", place it in the persistent directory and then reference it in the code. This would add some slight friction to the user experience but would let you effectively support any PBF source out there.

Many thanks for the suggestion, this sounds good to me. Thoughts @agila5 ? I've not tried the workflow but it sounds promising.

@agila5
Copy link
Collaborator

agila5 commented May 9, 2022

Hi @bdon and thanks for your message.

Thoughts @agila5 ? I've not tried the workflow but it sounds promising.

I think that's a reasonable idea but I have one question. Am I right saying that, in general, it's not possible to download those PBF extracts directly from R? So the workflow should be something like 1) download the .osm.pbf file from the browser UI for a given rectangle/polygon; 2) read in the file using one of our functions; 3) manipulate/plot/whatever?

btw: @mtennekes, if you are still interested in a quick chat to chat about the aforementioned problem, I'm free on Tuesday/Wednesday/Thursday from 9AM to 6PM.

@mtennekes
Copy link
Author

btw: @mtennekes, if you are still interested in a quick chat to chat about the aforementioned problem, I'm free on Tuesday/Wednesday/Thursday from 9AM to 6PM.

I'm still looking forward to meeting you @agila5, but have to postpone it to mid June (or later), when I will catch up with that project. Will let you know! Btw, I also recently switched from laptops (in my case from Ubuntu/Windows dual boot to Mac). The problem I had was only on Ubuntu, so cannot reproduce it anymore.

@agila5
Copy link
Collaborator

agila5 commented May 10, 2022

I'm still looking forward to meeting you @agila5, but have to postpone it to mid June (or later), when I will catch up with that project.

Ok, no problem! Send me a message when you start working again with OSM data.

so cannot reproduce it anymore.

Ops ahahaha No worries then, we will just wait if there is any new issue from Ubuntu users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants