Skip to content
Snippets Groups Projects

Updating the planet extracting data

Last edited by ltog

Updating datasets

List of commands

Update the complete planet file
osmupdate planet-latest.osm.pbf planet-latest-updated.osm.pbf [[--hour] --day]
Generate a .o5c change file between old and current planet
osmupdate [--keep-tempfiles] [[--hour] --day] planet-latest.osm.pbf planet-update.o5c

Workflow variants

  1. Always make new extracts from the updated planet file using osmconvert
  2. Update the planet and keep temp files (osmupdate --keep-tempfiles). Apply osmupdate [-b | -B] --keep-tempfiles on extract files. This reuses the downloaded diffs and therefore saves time (for downloading) and bandwidth.
  3. Use osmupdate planet.pbf diff.o5c to generate a combined diff (potentially spanning multiple daily updates) between old and current planet. Apply the .o5c file to the extracts (is this possible?) and to the planet to update these files. The .o5c file should also be usable to update databases with osmosis/osm2pgsql according to http://wiki.openstreetmap.org/wiki/Osmupdate#Assembling_an_OSM_Change_file .

Note concerning variant 3: According to Stefan Keller one possible workflow might be the following: Update the extract using the (unclipped) .o5c file and then use osmconvert/osmosis to again clip the boundary.

Extracting data

osmosis

Installation (Ubuntu)
sudo apt-get install osmosis
bbox
export JAVACMD_OPTIONS=-server;
export JAVACMD_OPTIONS=-Xmx2G;
osmosis --read-pbf file=$pbfinfile --bounding-box top=$maxlat left=$minlon bottom=$minlat right=$maxlon --write-pbf file=$pbfoutfile
poly
export JAVACMD_OPTIONS=-server;
export JAVACMD_OPTIONS=-Xmx2G;
osmosis --read-pbf file="$pbfinfile" --bounding-polygon file="$polyfile" --write-pbf file="$pbfoutfile"
Tuning

http://wiki.openstreetmap.org/wiki/Osmosis/Tuning

osmconvert

Installation (Ubuntu)
sudo apt-get install osmctools
bbox
osmconvert $pbfinfile -b=$minlon,$minlat,$maxlon,$maxlat --out-pbf [--complete-ways | --complex-ways] -o=$pbfoutfile --hash-memory=2000
poly
osmconvert $pbfinfile -B=$polyfile --out-pbf [--complete-ways | --complex-ways] -o=$pbfoutfile --hash-memory=2000
Tuning
osmconvert --help

osmchange

Keine Unterstützung für .pbf / .o5m Dateien.

Installation (Ubuntu)

???

bbox
<$osminfile ./osmchange -b=$minlon,$minlat,$maxlon,$maxlat >$osmoutfile

poly

<$osminfile ./osmchange -B=$polyfile >$osmoutfile

Benchmarks

osmosis bbox
time osmosis --read-pbf file=switzerland-latest.osm.pbf --bounding-box top=47 left=7 bottom=46 right=8 --write-pbf file=osmosis-out.pbf

0m25

osmosis poly
time osmosis --read-pbf file="switzerland-latest.osm.pbf" --bounding-polygon file="polygon.poly" --write-pbf file="osmosis-out.pbf"

1m05

osmconvert bbox
time osmconvert switzerland-latest.osm.pbf -b=7,46,8,47 --out-pbf -o=osmconvert-out.pbf

0m10

time osmconvert planet-150119.osm.pbf -b=7,46,8,47 --out-pbf -o=osmconvert-out.pbf --hash-memory=2000

14m22 (Stefans Laptop) 9m29 (Fat Server, 20GB) 9m37 (Fat Server, 2GB)

osmconvert poly
time osmconvert switzerland-latest.osm.pbf -B=polygon.poly --out-pbf -o=osmconvert-out.pbf

0m18

Cutting out 1° tiles

Runtime lsplitter.sh

1st pass: 2.5h
2nd pass: 223m24s (245m47s if instances work on the same temporary files in parallel)

Voraussetzungen:

  • Multipolygon, aber nicht selbst-überschneidend.

Zu klären

  • Sich selbst überschneidende Polygonlinie (ok, oder muss es abgefangen werden)
  • bbox über Datumsgrenze -> vertagen
  • Grenzüberlappende Linien, Relationen (osmconvert: Flächen) abschneiden oder behalten? -> Behalten.
  • Werden folgende Daten benötigt?
    • Autor -> Verwerfen
    • Erstellungsdatum -> Verwerfen
    • Versionsnummer -> Verwerfen
    • ... -> Verwerfen
  • Dokumentationssprache? -> Englisch