Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Governmental data Import and OpenStreetMap. Cas...

Satoshi IIDA
September 10, 2017

Governmental data Import and OpenStreetMap. Case report of Japan

Talk at State of the Map Taiwan 2017.

Satoshi IIDA

September 10, 2017
Tweet

More Decks by Satoshi IIDA

Other Decks in Research

Transcript

  1. About me • OpenStreetMap: Mapper since 2010 • OpenStreetMap Foundation

    Japan: Vice President since 2014 • Main activity: Community support • OpenStreetMap Foundation: member since 2013 • Daily Work on JIPDEC • OpenData & Geospatial consultant
  2. Today’s Agenda 1. General information about Import. 2. Past Imports

    & Current relationship on Japan region. 3. Import methodology (Based on my experience).
  3. Abstract • Importing (also known as Bulk Importing) is the

    process of uploading external data to OSM. • [snip] • Unlike editing the map or uploading GPX traces, importing combines an existing dataset with the OSM dataset, and thus typically includes a complex merging process. ——Quote from OSM wiki - import
  4. 3 perspectives about import 5FDIOJDBM 0QFSBUJPOBM -FHBM 'JMFGPSNBUDPOWFSU %FEJDBUFEBDDPVOU -JDFOTFDPNQBUJCJMJUZ

    5BHHJOHTDIFNF 0SHBOJ[FBUBTLUFBN 4QFDJBMQFSNJTTJPO :PVIBWFUPNFFUUIPTFSFRVJSFNFOUT
  5. Note: “Community” mentioned above include both local community & international

    community. So it is mandatory to be reviewed from international members on “Imports" Mailing List Procedure of import • Step 1 - Prerequisites • Step 2 - Community Buy-in • Step 3 - License approval • Step 4 - Documentation • Step 5 - Import Review • Step 6 - Uploading From “OSM wiki import/Guidelines”
  6. 2008 2010 2012 2014 2017 MLIT: Kokudo Suuchi Joho (国⼟土数値情報,

    KSJ) data import, 2008-2011 Yahoo Japan!/ALPS: Road data import, 2011-2014 MLIT: KSJ Municipality boundary re-import, 2014-2015 Municipality : City planning OpenData, 2013-2016 • Urayasu City building footprint • Sabae City building footprint • Sizuoka City building footprint • Tondabayashi City road and building footprint Japan National OpenData Strategy start Import Guideline is discussed and improved
  7. Current relationship from OSM community • OSM community has major

    relationship with • Open Data advocates • Researchers, Civic tech activists, Developers • Civic, NPO activities • Town walk activity • Government Officers • Geospatial Information Distribution (Japanese geo data platform) • Local government officers • Central government officers (especially MLIT, GSI)
  8. Major Data from GSI • Traceable license of their GSI

    MAP (地理理院地図) tiles. • Base map, disaster response • GSI map has special advantage for accurate positioning (due to it is created under Survey law quality assurance). • And most important, it is distributed XYZ tile format!!
  9. GSI data which we could not use • OSM could

    NOT import Raw data from GSI. • Survey Data (測量量成果) like Fundamental Geospatial Data (基盤地図情報) is protected under Survey law. • This data is super useful, but very hard to tackle to…
  10. Current relationship with Local governments • Some local governments are

    viewing OpenStreetMap as… • use case of their Open Data. • Free (sometimes “as beer”, “less cost”) map data platform • Touchpoint for Civic activity (esp. through Mapping Party) • Each municipality has different view on OpenStreetMap. • It should be acceptable. • At first stage, OpenStreetMap have to make our data useful to civic, includes those governments.
  11. File format convert • Projection convert: GDAL (ogr2ogr) • File

    Format & encoding convert: ogr2osm • Especially shapefile -> osm xml • Shapefile attribution to OSM tag convert • I recommend Paul Norman’s version • Tell your FOSS4G friends for more details :) Technical
  12. Tag Scheme • Investigate original data column to determine which

    column to be used. • Need discussion on talk-* ML and imports ML. • Most of original column is USELESS. Technical
  13. Dedicated account • Make a import dedicated account to separate

    your work from your own account. • [sample] • Urayasu_bld_import • helicoysfh_tondabayashi_import • KSJ2_adm_bnd_imprt • It is very easy to revert if some trouble happens. Operational
  14. Organize your Team • Collaborative import is becoming popular. •

    Tasking Manager is a great tool to manage collaboration of team members. • e.g. imports by HOT team, Mapbox team, local communities and so on… • This could be reduce tasks for 1 person. • And it helps training other members. Operational
  15. Or you have to be a Lonely Hero… CC BY,

    Kevin Dooley “batman” on flikr
  16. TIPS at work • Merging with existing data is the

    most hardest work. • Divide tasks for small amounts. • Especially if it is about building or address import. • OSM US team has published some tools regarding address conflation during NYC and SF import. • Validation is mandatory. Operational
  17. Legal perspective • CCBY is not compatible with ODbL. •

    Taiwan Open Government Licence is not compatible as well. (Japanese one is also) • So you have to ask government if we try to import the data into OpenStreetMap. Legal
  18. Special Permission • OpenStreetMap Foundation has published a template for

    asking permission. • Based on CC BY 4.0 publication (also 3.0, 2.0). • 2 items of incompatibility • Attribution on “Contributor” Page (Attribution is not signed to OSM data itself) • Technical restriction on Downstream (e.g. DRM. This is obviously different) Legal
  19. NON! • Think different :) • Application to officers =

    Making a touchpoint. • A chance to introduce, discuss and include them. • A chance to walk together!
  20. Import is hard. But it has very effective to improve

    data coverage. And also effective to improve collaboration.