1. Where 2.0 2010: Maps from Scratch I: Building
    1. Introduction
    2. Data I: Shapefiles
    3. Data II: OSM and PostGIS
    4. Presentation
    5. Links

Where 2.0 2010: Maps from Scratch I: Building

Introduction

(slides)

Explain:

Data I: Shapefiles

Things we'll cover:

We start by going to the appropriate directory, and grabbing the data in a zipfile. The original file comes from http://www.naturalearthdata.com but we've made some small changes. We unzip the file into the current directory. 

cd ~/building_01_shapefiles
curl -OL http://mapsfromscratch-resources.s3.amazonaws.com/50m_admin_0_map_subunits-clipped.zip
unzip 50m_admin_0_map_subunits-clipped.zip

The shapefile format actually consists of groups of files with similar names. You will see four new files from the archive: .shp, .dbf, .shx, and .prj:

It's worth noting that these are just plain files sitting in a directory. Shapefile is a file format originally developed by the ESRI corporation, but now used as a lowest-common-denominator interchange file format by jsut about every geographic software package that works with vector data.

With the shapefiles in place, we can create an image based on the stylesheet already sitting in the directory. 

nik2img.py -v -d 900 600 -e -20037508 -7500000 20037508 18500000 style.xml out.png

The call to nik2img.py has several parts:

  1. -v just means "be verbose", so we can see what's happening as it happens.
  2. -d 900 600 creates an output image whose dimensions are 900x600 pixels.
  3. -e -20037508 -7500000 20037508 18500000 is the projected extent. The spherical mercator projection we use here is expressed in meters on the ground. At the equator the earth is approximately 40,075km around, so our horizontal extent goes from -20,038km to +20,038km. The other two values are vertical extent, and span the area between the northernmost islands of Canada to just north of Antarctica.
  4. style.xml is an XML stylesheet that Mapnik uses to store rendering instructions. These consist of data layers and style rules. This stylesheet is very simple with just one of each - a layer for the shapefile data, and a style rule to determine what color to fill it with.
  5. out.png is the image file we want to create.

We can do a second render using geographic coordinates to demonstrate how to zoom into particular places in the world.

This is Europe: 

nik2img.py -v -d 900 600 -d -12 35 38 60 style.xml out.png

This is the UK: 

nik2img.py -v -d 900 600 -d -6 50 2 53 style.xml out.png

You can see in that last one we lose a lot of detail with the worldwide data.

Data II: OSM and PostGIS

Things we'll cover:

We start again in the appropriate directory, and grab two files this time. The first contains a shapefile, similar to the worldwide data we used previously. This one is an OSM-derived coastline from http://hypercube.telascience.org/~kleptog/. The second is an OSM extract, based on a recent download from http://downloads.cloudmade.com/north_america/united_states/california. 

cd ~/building_02_postgis
curl -OL http://mapsfromscratch-resources.s3.amazonaws.com/processed_p-clipped.zip
curl -OL http://mapsfromscratch-resources.s3.amazonaws.com/bayarea-2010-03.osm.bz2

First we pull the coastline shapefile into PostGIS. For that, we'll use a tool called shp2pgsql. 

unzip processed_p-clipped.zip
shp2pgsql -dID -s 900913 processed_p-clipped.shp coastlines | psql -U postgres building_02_geodata
rm processed_p-clipped.*

As in the previous section, this zipfile contains a simple collection of files. We'd like to take advantage of the indexing and other features offered by a spatial database, so we'll convert the file to a table. The spatial database we use is called PostGIS, an extension to the open source PostgreSQL database that offers advanced vector geometry features.

The call to shp2pgsql has a few interesting parts:

  1. -dID is a set of three options: drop the table if one already exists, Index the spatial geometry, and use the more compact Dump format.
  2. -s 900913 assigns a projection, the Google-style spherical mercator. We know to use this because the source of the coastline files, http://hypercube.telascience.org/~kleptog/, tells us to.
  3. processed_p-clipped.shp coastlines are the names of the source file and the destination database table.
  4. | psql -U postgres building_02_geodata pipes the output of shp2pgsql directly to PostgreSQL.

Note that when we are done we can simply delete the coastline files. Try rendering the map now. 

nik2img.py -v -d 900 600 -c -122.355 37.799  -z 11 style.xml out.png

We're using nik2img.py slightly differently than before, giving it a center point and a zoom level: just south of Treasure Island in the middle of the Bay, and 11 - more on this later. Look at the image and notice that we now have a highly detailed coastline.

Next, we move on to the OSM extract. The .osm file is a slightly different beast. Use bzcat to look inside: 

bzcat bayarea-2010-03.osm.bz2 | head

You'll see that it's just simple XML. There is a list of nodes, and it you were to look further you'd see a list of ways as well. Each has a latitude and longitude assigned, along with the username of the person who created that node. Some nodes have associated tags as well, note the one near the top.

We pull the OSM extract into PostGIS with a tool called osm2pgsql. This part should take about 2-3 minutes to run: 

osm2pgsql --create -d building_02_geodata -U postgres -p bayarea_osm -m -u -S /usr/share/osm2pgsql/default.style bayarea-2010-03.osm.bz2

osm2pgsql performs a similar action to shp2pgsql:

  1. --create specifies that we're creating a new set of database tables.
  2. -d building_02_geodata -U postgres -p bayarea_osm are the database, user account, and table name prefix for PostgreSQL.
  3. -m is shorthand for mercator, and specifies the same 900913 projection as in shp2pgsql above.
  4. -u means UTF8-sanitize, to fix possible text problems.
  5. -S /usr/share/osm2pgsql/default.style selects a so-called "stylesheet", which specifies how the free-form tags of OSM map to specific database columns.

With the database populated, we can create an image of streets and highways near the Bay Bridge. 

nik2img.py -v -d 900 600 -c -122.355 37.799  -z 11 style.xml out.png

This call to nik2img.py is slightly different:

  1. -c -122.355 37.799 centers the map at a point just south of Treasure Island, http://maps.google.com/?q=37.799,-122.355&z=11.
  2. -z 11 zooms the map to level 11. Zoom levels in Google-style slippy maps go from 0 for the whole world, to about 18 for most detailed streets you might need.

Presentation

Things we'll cover:

There isn't a lot of code in this section, mostly explanations.

In a web browser, go to the building_03_presentation directory of your server. Note the files there:

Click slippy-map.html.