Making Routes and Visualizing in QGIS Using SQL

PgRouting analyzes graphs to find the shortest path between two nodes through edges. In a street network, you can think of nodes as intersections and edges as the streets themselves. I will be using the Chicago street network, which can be found here. I have also uploaded it to the server, named chicagostreets.

1. In order to develop data capable of using PgRouting a source and target geometry must be set up at the start and end of each road. These will serve as the start and end destination points. To perform this follow the “Creating a routable road network” section of this tutorial (blog also has many helpful QGIS tutorials). Another thing to get familiar with in SQL is the WITH Clause. This allows you to reference other SQLs within one query.

2. Once your network is made you can start routing. Follow this tutorial to get multiple options for routing. pgr_dijkstra and pgr_kdijkstraPath are two that work well for visualizing one path or one startpoint to many paths.

pgr_kdijkstraPath3. By joining the geometries in your network to the results of your routing you can import this into QGIS and see your paths.

To reference my SQL query from the tutorial in class, click link.

Uploading and Visualizing Census Data in QGIS

Census data is accessible in two forms on the US Census site: Tiger/Line Geographic Shapefiles and as Tabulated Data (CSV) from the census proper, including a range of demographic, housing, and economic data. Follow these steps to join these two data types to generate geolocated maps of this data in QGIS (for a slightly outdated, but very detailed step-by-step tutorial, go here):

  1. find and download the census regions Shapefiles you are interested in from the Tiger/Line site. Choose appropriate region and granularity – census blocks are the smallest geographical unit used by the US census. Make sure to download the tiger regions from the year that corresponds to the tabular data you are getting (i.e. 2010, 2000, etc.)
  2. find and download the Tabulated datasets you are interested in from the US census factfinder2 site. (Click on the “Advanced Search” tab; Click the “Geography” tab; Click the “Name” tab; Enter the Name of the Region you are interested in – “Wayne County, Michigan” for instance; In the “Geography Filters” Column pick the appropriate geographical unit – e.g. “block”. After all of this, it should give you a list of regions that meet the search criteria. Select the region(s) you want and press “Add”. Close the Geographies pop-up, and you should see all of the tabulated datasets for this region and geographical unit. Select the dataset you want to download and press “View.” Verify the data is what you want and Press “Download.” In the download dialogue box, check the “Data and Annotations in Separate Files” box and press “OK” to start the download. This will bring up one more dialogue window with a progress bar. Once the progress is complete, press “download” to finally download the Zip file.
  3. Screen Shot 2014-01-21 at 8.05.31 PM
  4. Follow this tutorial to prep and import the two files to QGIS and join them based on the appropriate geometryID column (e.g. the block ID #) – You may want to do some cleaning of the csv file to remove any unwanted columns and to ensure the geoID column data matches with the shapefile. Use the find/replace command in Excel (CTRL-F) to facilitate this work.

Responsive City TED

“We think we can develop a vehicle that operates on bike lanes, that’s accessible to elderly, disabled, women in skirts, business people, and address the issues of energy, congestion, mobility, aging, and obesity simultaneously.”
For the first time in history more people live in cities than in rural areas and the trend is accelerating worldwide. In this TEDPrize City 2.0 talk Kent Larson, director of the MIT Media Lab Changing Places’ group, tackles the problems of increased congestion and pollution with new models for urban dwellings, shared-use vehicles, and responsive environments.

Play the City

Play the City proposes using city gaming in real planning and city design procedures. Our Re-Play Noord! game was part of La Fabrique de la Cite’s Building the Shared City conference August 2012 in Amsterdam.
The legal plan of Overhoeks is slowly but decisively progressing. We ask what would be the most desirable form of an organic development around the Shell Tower and Grootlab? Do we really need 70.000 sqm of extra offices in Overhoeks as the legal plan suggests? How to organize the remaining 200.000sqm of housing job? Can small entrepreneurs be the authors of this task replacing a giant investor?

More Info:http://www.playthecity.eu/

Reinvent Payphones

Reinvent Payphones Winner: NYC I/O: The Responsive City – Best in Community Impact

Control Group and Titan partnered to create NYC I/O— the transformation of the corner payphone into a digital node that will usher in a new era of The Responsive City. By updating the payphone with a modern array of sensors and displays to create a foundational input/output system for an open, urban-scale computing platform, we can allow New York City to respond to and serve the people. Through open access to real time data and a distribution platform for community, civic, arts and comercial apps and messaging, we can create a safer, more efficient, and more enjoyable city.

NYC manages a telecommunications network of 11,412 public pay payphones throughout the five boroughs. Payphone use has decreased with mobile device adoption, but payphones still serve the communications needs of thousands of New Yorkers a day, especially in times of emergency. The City asked the tech and design community to reinvent New York City payphones to make our city more accessible, safer, healthier, and better informed.
GOALS

  • Create a full data network by using existing payphone infrastructure.
  • Make each payphone a primary input/output tool for interactive communication between the city and citizens
  • Make the software and hardware open and flexible to evolve over time to accomodate changes in technologies over time

Microsoft Power Map Preview for Excel Getting Started

1     What is Power Map?

Microsoft Power Map for Excel is a three-dimensional (3-D) data visualization tool for Microsoft Excel 2013 which provides a powerful method for people to look at information in new ways. It enables the data discoveries that might not be seen in traditional two-dimensional (2-D) tables and charts. With Power Map you can plot geographic and temporal data visually, analyze that data on a 3-D globe and over time, and create visual tours to share with others.

NOTE: Power Map and Project Codename GeoFlow” are the same. “GeoFlow” was the codename we used for Power Map in the current preview release that’s available in the Microsoft Download Center.

With Power Map you can:

  1. Map Data: plot more than a million rows of data in 3-D, visually on Bing maps from an Excel worksheet table or the Data Model in Excel.
  2. Discover Insights: gain new understandings by seeing your data in geographic space and seeing time-stamped data change over time.
  3. Share Stories: capture screenshots and build cinematic, guided video tours that can be shared broadly, engaging audiences like never before. Tours can be exported to video and shared that way as well.

This preview of Power Map supports Arabic, Japanese, Spanish and English versions of Office Pro Plus.

Download the whole tutorial online:

http://office.microsoft.com/en-us/excel-help/power-map-preview-for-excel-HA104102903.aspx

 

Creating Convex Hulls with SQL to QGIS: Part 2

We created clusters (convex hulls) of venues making distinctions between the venues by categories. We started off by creating a mathematical algorithm that combines multiple equations:

  • We needed to determine a method of comparison for all the venues. Our method used the average distance between all the venues. We used the average distance, which we calculated by comparing the distance from one venue to all other venues and doing this for all the venues and then dividing the number of venues. In essence, it is a permutation with repetition calculation.
  • After we calculated the average distance we set a threshold divider of the average distance to determine the size of the clusters that we wanted to use.
  • Then we had to match the venue points of comparison to cluster only by category whereby venue points would only cluster when one venue category was equal to another venue category.
  • The next step was to create the shape of the cluster creating a geometry polygon to import as a PostGIS shapefile.

Step 1:

CREATE TABLE avgdistances2 AS

SELECT * FROM  (SELECT f1.gid as v1id, f1.fsid, f2.gid as v2id, st_distance(f1.geom,f2.geom) AS dist, f1.upperlevelcat as f1ul, f2.upperlevelcat as f2ul

FROM fsvenuewithupperlevelcat2 as f1

CROSS JOIN fsvenuewithupperlevelcat2 as f2  WHERE f1.checkinct > 200 and f1.gid<>f2.gid) AS q1

WHERE q1.dist < ( SELECT SUM(avgdist)/count(hack) as avgdisttotal    

FROM     (SELECT f1.gid, 1 as hack, SUM(st_distance(f1.geom,f2.geom))/COUNT(f1.gid) AS avgdist

FROM fsvenue as f1

CROSS JOIN fsvenue as f2

WHERE  f1.gid <> f2.gid     GROUP BY f1.gid) as subq    GROUP BY subq.hack)/4 AND f1ul=f2ul  ORDER BY q1.v1id;

Step 2:

CREATE TABLE fsvenue_cathulls(   gid serial NOT NULL,   fsid character varying(256),   CONSTRAINT fsvenue_hulls_pkey PRIMARY KEY (gid) ) WITH (   OIDS=FALSE );

ALTER TABLE fsvenue_hulls  OWNER TO poweruser; GRANT ALL ON TABLE hullmp3 TO poweruser;

SELECT AddGeometryColumn(‘fsvenue_hulls’,’geom’,3435,’MULTIPOINT‘,2);

INSERT INTO fsvenue_hulls (fsid,geom) SELECT ad.fsid, st_collect(v.geom)  FROM avgdistances2 AS ad Left JOIN fsvenue AS v ON ad.v2id = v.gid WHERE v1id <> 42 GROUP BY ad.fsid ORDER BY ad.fsid;

CREATE VIEW fsvenue_cathulls AS SELECT v.gid, v.fsid, v.upperlevelcat, st_convexhull(h.geom)::geometry(‘Polygon’, 3435) as geom FROM fsvenue_hulls as h LEFT JOIN fsvenuewithupperlevelcat2 as v ON h.fsid = v.fsid WHERE st_npoints(h.geom) > 3;

SELECT * FROM fsvenue_cathulls;

Step 3:

Import PostGIS file to QGIS

Result from category 1: Arts & Entertainment

Bucktown & Wicker Park

arts

 

Creating Convex Hulls with SQL to QGIS: Part 1

In order to create clusters based on foursquare categories, we first made a table in PostgreSQL called upperlevelcat and assigned a gid for each of the nine categories.

Foursquare Categories:

upperlevel cat table

To link these categories to venues, we did a double join : fscategory table, fsvenue, and fsvenue_fscategory tables.

fscategory table

fscategory2

fsvenue table

fsvenue table

fsvenue_fscategory

fsvenue_fscategory

SQL statement for double join:

SELECT v.herenow, v.checkinct, v.userct, join1.upperlevelcat, join1.name as catname, v.name as venuename,v.lat,v.lon,v.geom

FROM (SELECT * from fsvenue_fscategory

LEFT JOIN fscategory as fsc ON fsc.fsid=fsvenue_fscategory.cid) AS join1

LEFT JOIN fsvenue as v ON  v.fsid=join1.vid

Result

result

SQL and PgRouting Continued

In order to get pg routing to work, first a transportation network must be formed with a source and target column created to specify which start and end points are used. The SQL below creates a network  in a buffer of a 1 mile radius and examines the Damen CTA Stop. To alter the name of the table created, change the buffer distance, or to change the CTA Stop, change the red values in the SQL below.

CREATE TABLE damennetwork AS
WITH ctastop AS
(SELECT * FROM cta_railstations
WHERE gid = 32),
transpo AS
(SELECT
row_number() OVER (ORDER BY transportation.gid)::integer AS gid,
transportation.gid AS id, 
transportation.street_nam AS name,
transportation.length AS cost,
transportation.geom,
st_startpoint(ST_LineMerge(transportation.geom)) as source,
st_endpoint(ST_LineMerge(transportation.geom)) as target
FROM ctastop
LEFT JOIN transportation
ON st_within(transportation.geom, 
st_setsrid(st_buffer(ctastop.geom, 5280), 3435))),
nodes AS
(SELECT DISTINCT transpo.source AS gid 
FROM transpo
UNION
SELECT DISTINCT transpo.target AS gid 
FROM transpo)
SELECT transpo.gid, transpo.id, transpo.name, transpo.cost, transpo.geom, source.id as source, target.id as target
FROM transpo
JOIN 
(SELECT row_number() OVER (ORDER BY nodes.gid)::integer AS id, 
nodes.gid AS geom
FROM nodes
GROUP BY nodes.gid)
AS source ON transpo.source = source.geom
JOIN 
(SELECT row_number() OVER (ORDER BY nodes.gid)::integer AS id, 
nodes.gid AS geom
FROM nodes
GROUP BY nodes.gid) 
AS target ON transpo.target = target.geom

To be able to find a route with this SQL use the below SQL. Input the name of your table used in the previous SQL where there are the “damennetwork” in red. To change to start and end points, change to “502” and “341” respectfully.

SELECT seq,id1 as node, id2 as edge, route.cost, damennetwork.name as streetname, damennetwork.geom 
FROM pgr_dijkstra('
 SELECT gid AS id,
 source::integer,
 target::integer,
 cost::double precision AS cost
 FROM damennetwork', 502, 341, false, false ) 
AS route
LEFT JOIN damennetwork
ON route.id2 = damennetwork.gid;

What we have been working on is using the CTA stop as a start point and all the venues as the end points. The first thing I did was take all the venues in the same buffer as the network and corresponded them to their closest edge (the target). Different buffers and train stops can be chosen by the the red numbers.

WITH ctastop AS
(SELECT * FROM cta_railstations
WHERE gid = 32),
p AS
(SELECT
venues.gid AS gid, 
venues."legal name" AS company,
venues."doing busi" AS name,
venues.address AS address,
venues.latitude AS lat,
venues.longitude AS lon,
venues.geom AS geom
FROM 
ctastop
LEFT JOIN venues
ON st_within(venues.geom, 
st_buffer(ctastop.geom, 5280)))
SELECT rcost.*, p_to_l.targetID, p_to_l.geom FROM
(SELECT p.gid as restaurantID,
min(st_distance(p.geom,l.geom)) as distance
FROM p, damennetwork AS l
GROUP BY restaurantid) 
AS rcost
LEFT JOIN 
(SELECT p.gid as restaurantID, l.target as targetID, l.geom,
st_distance(p.geom,l.geom) as distance
FROM p, damennetwork AS l) 
AS p_to_l
ON rcost.distance = p_to_l.distance
AND rcost.restaurantID = p_to_l.restaurantID

The same must be done to find the start point (source) which is the CTA stop. Change to red value to select a different trainstop.

SELECT * FROM (WITH ctastop AS
(SELECT * FROM cta_railstations
WHERE gid = 32)
SELECT row_number() OVER (ORDER BY sourceid.distance)::integer AS gid,
sourceid.* FROM (
SELECT p.gid as restaurantID, l.source as sourceID, l.geom,
st_distance(p.geom,l.geom) as distance
FROM ctastop AS p, damennetwork AS l) as sourceid) as source
WHERE source.gid = 1

This gave me a result of a 292 start point, which I will be plugging into the next SQL (in red). The other numbers are the results from the venue target table formed two SQLs ago.

CREATE TABLE routetestwm AS
SELECT seq,id1 as path, id2 as node, id3 as edge, route.cost, damennetwork.name, damennetwork.geom
FROM pgr_kdijkstraPath('
 SELECT gid AS id,
 source::integer,
 target::integer,
 cost::double precision AS cost
 FROM damennetwork', 292, array[336,457,227,179,179,291,351,515,424,244,427,250,349,457,138,291,291,263,230,117,115,208,325,594,284,301,368,116,91,471,543,398,251,251,251,230,230,365,138,463,404,301,356,212,342,203,203,342,478,286,292,519,41,242,242,242,242,242,242,356,57,57,289,310,286,452,277,251,212,427,251,182,242,471,445,251,251,93,291,286,305,262,46,262,46,262,288,288,371,117,398,105,105,286], 
 false, false ) 
AS route
LEFT JOIN damennetwork
ON route.id3 = damennetwork.gid;

This table can be brought into qgis as the network of paths from a CTA stop to any venue.

All Routes

Galvanic Skin Response Sensor

Galvanic skin response readings are simply the measurement of electrical resistance through the body. Two leads are attached to two fingertips. One lead sends current while the other measures the difference. This setup measures GSR every 50 milliseconds. Each reading is graphed, while peaks are highlighted and an average is calculated to smooth out the values. A baseline reading is taken for 10 seconds if the readings go flat (fingers removed from leads).

Arduino Code:

void setup(){
  Serial.begin(9600);
}

void loop(){
  int a=analogRead(0);
  if (Serial.available() > 0) {

    byte inbyte=Serial.read();
    if(inbyte=='a'){
      Serial.print(a,BYTE);

    }
  }
}

Processing Code:

import processing.serial.*;
Serial myPort;  

int hPosition = 1;     // the horizontal position on the graph
float currentReading;
float lastReading;
int count=0;
int zeroLinePos=0;

float gsrAverage,prevGsrAverage;
float baseLine=0;
long lastFlatLine=0;
color graphColor=color(255,255,255);
int baselineTimer=10000;
int gsrValue;
int gsrZeroCount=0;
float gsrRange=0;
int downhillCount=0;
int uphillCount=0;
boolean downhill;
boolean peaked=false;
float peak, valley;

void setup () {
  size(900, 450);
  // List all the available serial ports
  //println(Serial.list());

  myPort = new Serial(this, Serial.list()[0], 9600);
  currentReading=0;
  lastReading=0;
  gsrAverage=0;
  background(0);

  smooth();
}

void draw () {
  //best delay setting for gsr readings
  delay(50);
  //image(myMovie, 0, 0);

  if (gsrValue<15 &&gsrValue>-15){
    if( gsrZeroCount>10){
      currentReading=0;//flatline
      gsrAverage=0;
      baseLine=0;
      lastFlatLine=millis();
      gsrZeroCount=0;
      // println("reset");

    }
    gsrZeroCount++;
  }
  else{
    currentReading=gsrValue-baseLine;
    gsrZeroCount=0;
  }

  if(millis()-lastFlatLine>baselineTimer){
    baseLine=gsrAverage;
  }

  //graph colors
  if(gsrAverage>0 && gsrAverage<height/2.0*.25) graphColor=color(255,255,255);
  else if(gsrAverage>height/2.0*.25 && gsrAverage<height/2.0*.5) graphColor=color(255,250,100);
  else if(gsrAverage>height/2.0*.5 && gsrAverage<height/2.0*.75) graphColor=color(255,250,0);
  else if(gsrAverage>height/2.0*.75) graphColor=color(255,100,0);

  gsrRange=peak-valley;

  // at the edge of the screen, go back to the beginning:
  if (hPosition >= width) {
    hPosition = 0;

    //cover last drawing
    fill(0,200);
    noStroke();
    rect(0,0,width,height);
  }
  else {
    hPosition+=1;
  }

  gsrAverage=smooth(currentReading,.97,gsrAverage);

  //draw stuff

  //spike
  noStroke();
  if(gsrRange>200){
    fill(255);
    ellipse(10,10,20,20);
  }
  else{
    fill(0);
    ellipse(10,10,20,20);
  }

  //graph
  strokeWeight(.5);
  stroke(graphColor);
  line(hPosition-1, height/2.0-lastReading, hPosition, height/2.0-currentReading);
  stroke(255,0,100);
  line(hPosition-1,height/2.0-prevGsrAverage,hPosition,height/2.0-gsrAverage);

  //draw peaks
  int thres=7;

  noFill();
  stroke(255,0,0);
  strokeWeight(2);

  if (currentReading-thres>lastReading&& peaked==true){
    downhill=false;
    //println(downhillCount);
    uphillCount++;
    downhillCount=0;
    point(hPosition-1, height/2.0-lastReading);
    valley=lastReading;
    peaked=false;

  }
  if(currentReading+thres<lastReading && peaked==false){
    //println(uphillCount);
    downhill=true;
    uphillCount=0;
    downhillCount++;
    point(hPosition-1, height/2.0-lastReading);
    peak=lastReading;
    peaked=true;
  }

  prevGsrAverage=gsrAverage;
  lastReading=currentReading;
  //send 'a' for more bytes
  myPort.write('a');
}

void serialEvent (Serial myPort) {
  int inByte=myPort.read();
  //0-255
  gsrValue=inByte;
}

void keyPressed(){
  if (keyCode==DOWN)zeroLinePos+=3;
  if (keyCode==UP)zeroLinePos-=3;

  strokeWeight(1);
  stroke(255,0,0);
  line(0,zeroLinePos,2,zeroLinePos);
}

int smooth(float data, float filterVal, float smoothedVal){
  if (filterVal > 1){      // check to make sure param's are within range
    filterVal = .99;
  }
  else if (filterVal <= 0){
    filterVal = 0;
  }
  smoothedVal = (data * (1 - filterVal)) + (smoothedVal  *  filterVal);
  return (int)smoothedVal;
}



Tips: Arduino code is not supported by new version, so you have to alter the code.
      The resistor have to be around 300K, because human body's resistance is about 300K , if it is too low,
      the reading would be very low(almost zero), if it too high, the reading woud be 1023.
      The basic principle is that the analogRead is actually reading the voltage of the resistor, not the body, 
      if body resistance is much much bigger than the resistor, even if the body resistance changed , the voltage
      of the resistor will not change much(I am sure we have learned this in senior high school)