We continue to develop resources related to the COVID-19 pandemic. See COVID-19 initiatives on Appropedia for more information.
OSSTIP/WP1- Initial data gathering and familiarisation
- 1 WP Essentials
- 2 Requirements Summary
- 3 Results
- 3.1 Relevant Project blog entries
- 3.2 OpenStreetMap data
- 3.3 Transit Network schedules GTFS
- 3.4 GIS Shapefiles of the transit network
- 4 Extensions
This is a Work Package as part of the OSSTIP project.
- Publically-available GIS data of transport networks and related shapes,
- sample GTFS files,
- some samples of modified network GIS files.
- Short report write-ups on this Wiki page [DONE - see #Results section below.]
- Saving relevant data in Dropbox [DONE - see paths below]
- Dropbox/OSSTIP_Common/GIS-DataScripts - contains scripts necessary to do data conversions, e.g. Metlink format to GTFS
- Dropbox/OSSTIP_Common/GIS-Network-Data-Melb - contains relevant key data such as GTFS files for Melbourne, and an OpenStreetMap file of Melb metro region.
- N.B. :- currently am using train and tram network shapefiles from BZE work, see Dropbox/OSSTIP_BZE/Melbourne_GIS_NetworkDataWork
Estimated Time: Small-medium
Status: Complete (as of 23-10-2013) (however several relevant #Extensions)
The goal during this WP would be to simultaneously assemble and better understand the key data currently available for Melbourne’s transport network, compared to requirements of the GIS-T tools of interest (being evaluated in WP2 (p5)), and initial plans for how to bridge the available needs.
Key formats and data sources are likely to comprise:
- GIS vector Shapefiles of public transport networks and routes (and how to best organise them);
- Timetables, likely in the form of the GTFS (General Transit Feed Specification)1;
- Street networks, like OpenStreetMap, that will support including walking (and possibly cycling) into the journey planner module;
- Supplementary data that might support later stages of the project: e.g. capacity and loading of current transport networks, VISTA travel activity data, etc.
Best sources for the data should be documented :- and if a good public source doesn’t yet exist, then ways of generating the required data from intermediate forms should be recorded.
The WP would include assembling as complete as possible a GTFS timetable for Melbourne as it currently stands based on available data, and at the same time documenting and sharing expertise within the team of how to effectively work with the format. It could also include experimenting with tools to extract useful information from GTFS files, such as route maps, and current average speeds and frequencies. This kind of initial experimentation would form the basis for OSSTIP/WP5s workflow for converting between the tools.
With the kind of optimal workflow for converting between the formats as specified in OSSTIP/WP5 in mind, at this stage we should also start preliminary work on how to add/attach required ‘metadata’ to the routes that will support their conversion into a full timetable – such as desired frequency, and average speed.
The summary of key data is:
- General street network GIS data available from OpenStreetMap, particularly the Metro Extracts site.
- For transit schedules: we are using the GTFS format, after collecting transit schedules in another format and re-formatting using scripts.
- For transit networks, BZE already had a source for train, tram and bus networks. If using the same process in future in other cities, these would hopefully be available from govt agencies, e.g. the regional transit authority.
If we also wish to do analysis comparing public transit to car-based transport in the region, this will require some data collected by the regional transport agencies - see the Extension section below.
Relevant Project blog entries
Accessing OSM data for Melbourne
Procuring OpenStreetMap data for Melbourne proved relatively straightforward - though it does need a fairly high-speed internet connection and quota, as even zipped up, these files run into the 50MB_+ range. This is only to be expected when downloading a street network for an entire city!
The most helpful source I found for getting OSM data for the Melbourne Metro region to support trip planning calculations was from the excellent "Metro Extracts" website:- http://metro.teczno.com/#melbourne
This website updates the OSM data monthly from the latest global records, so is a good source.
Other possibilities that gave helpful info, and could be useful in future:
- http://wiki.openstreetmap.org/wiki/Downloading_data#Choose_your_region has useful tips, including how to use web service APIs to ask for sub-sections of large datasets, but I didn't need to use this in the end.
- http://download.bbbike.org/osm/bbbike/Melbourne/ - OSM Extracts for Melbourne, from a variety of providers, with a cycling focus.
Using The OSM Data as a QGIS Layer
A small tip is that to help with visualisation and analysis of network data in QGIS, we've found the OpenStreetMap plugin useful.
Instructions for enabling the plugin are at http://wiki.openstreetmap.org/wiki/QGIS_OSM_Plugin - basically this can be done within QGIS itself.
Then under either the "Web" menu (Windows) or "Plugins" menu (Mac OS X) there will be an OpenStreetMap sub-menu, where you can do things such as load the OSM data as a vector layer.
GTFS schedules are the data format preferred by many Open Source tools.
GTFS data availability
For the sake of this project, we were able to collect GTFS data by downloading timetable data in a different format from a publicly available source, and then using several Python scripts to re-format the data into GTFS format. The Python scripts rely on the Google Transit Data Feed library discussed below.
As of 23-10-2013 :- it would be good to further process the converted data, as there are some issues with the bus network sections still. This will improve the results generated in OSSTIP/WP4.
We doubt these scripts would be useful to others, but if interested we may be able to make them available.
The longer-term solution is to advocate for GTFS to be regularly published by the local transit agency - see the Extensions section below.
Tools for initial testing and evaluation of GTFS data
The friend who helped me set up GTFS data for Melbourne also recommended the useful Google Transit Data Feed (software library). In addition to a range of scripts and libraries for transforming and checking the data, Transit Feed includes a very helpful small web-application, ScheduleViewer, that can allow interactive viewing and checking of GTFS datasets for a region. See image at right.
If you have Python already installed on your systems, and the 'easy_install' Python package installer tool setup, then you can install the transit data feed library with just the followign at the terminal:
sudo easy_install simplejson sudo easy_install transitfeed
GIS Shapefiles of the transit network
For the project, we also need to work with the PT networks directly in a GIS tool.
One possibility is extracting the routes from GTFS files - see GTFS#Converting GTFS to other formats, such as GIS shapefiles.
- The JEQL option I looked at but it seems JEQL is only available in Windows, so am leaving for now ...
- Trying gtfsdb also as another option to load the GTFS data into a GIS-ready database, then do further work from there. See OSSTIP/Loading GTFS to a GIS database for notes so far.
Fortunately, BZE already has train and tram networks as KML files, and have developed shapefiles for their own bus network.
I could explore other options for obtaining the data, such as from http://data.vic.gov.au - probably bus networks (as well as train, tran), will be the challenge.
Update 2013-10-23 :- apparently the OpenStreetMap database in the Melbourne region contains up-to-date information of bus stop locations, thanks to a volunteer carefully checking them all. This could be useful to download for cross-checking against other sources, e.g. the GTFS files.
(2013-10) Car-travel time data for Melbourne to aid accessibility comparisons
Due to BZE's requirements in OSSTIP/WPBZE2, we would also like access to car-based travel time data for Melbourne between Travel Analysis Zones (TAZs) in order to help develop both:
- Accessibility comparison metrics such as can be calculated and visualised in the Open Source Accessibility Toolkit tool;
- This is also necessary for BZEs model of potential mode-shift based on a 4-step-like travel demand estimation, assignment approach.
Current approach is to submit a request to AURIN to see if we can get access to use this data for this project.
(2013-10) Advocating to Melbourne's Transit agencies (e.g. PTV) for a planned proper release of well-formed GTFS feed
Beyond this particular trial project of OSSTIP, an important aspect is a more sustainable source of good public transport schedule data.
See the General_Transit_Feed_Specification#Availability_of_GTFS_feeds_around_the_world page section for some commentary and links to relevant discussion papers about the benefits of GTFS publishing from an agency perspective, and ways to overcome perceived concerns.
Update:- In March 2014, Public Transport Victoria (PTV) did release an API for access to public transport information, see https://www.data.vic.gov.au/raw_data/ptv-timetable-api/6056 . However, unfortunately this did not include a GTFS feed. Various interested parties are in discussion with PTV about the possibility of a GTFS feed being made available.