Neighborhood boundaries based on social media activity

Researchers at the School of Computer Science at Carnegie Mellon University investigate the structure of cities in Livehoods, using foursquare check-ins.

The hypothesis underlying our work is that the character of an urban area is defined not just by the the types of places found there, but also by the people who make the area part of their daily routine. To explore this hypothesis, given data from over 18 million foursquare check-ins, we introduce a model that groups nearby venues into areas based on patterns in the set of people who check-in to them. By examining patterns in these check-ins, we can learn about the different areas that comprise the city, allowing us to study the social dynamics, structure, and character of cities on a large scale.

It’s most interesting when you click on location dots. A Livehood is highlighted and a panel on the top right tells you what the neighborhood is like, related neighborhoods, and provides stats like hourly and daily pulse and a breakdown location categories (for example, food and nightlife). Does foursquare have anything like this tied into their system? They should if they don’t.

There’s only maps for San Francisco, New York City, and Pittsburgh right now, but I’m sure there are more to come.

Want more on the clustering behind the maps? Here’s the paper [pdf].livehoods_icwsm12livehoods

Twitter for Developers: New API V1.1 & Creating an Application

Twitter introduced its new version of the API, V1.1, this changes the whole authentication process. According to a Wired article, Twitter’s New API Rules Likely Spell the End of Third-Party Clients, developers saw this as a threat to data usage. Anyone using Twitter now has to be authenticated and has a rate limit. There is no more typing the search into the URL bar.

How to create an application in the twitter developer page:

1) https://dev.twitter.com/apps

2) Click create new application

3)  Fill out the application details page

Create an application

4) Under the OAuth tool tab you can access your consumer key, consumer secret, access token, and access token secret for authentication.

 

 

Esri enriches maps with Tweets and the Streaming API

New Idea

On March 11, 2011, a 9.0 magnitude earthquake struck near the east coast of Honshu, Japan. As the news broke, many turned to Twitter to get the latest updates from people on the ground who experienced the disaster first hand. Within hours of learning about the devastating event, the @Esri team used Twitter to launch an interactive map that combined trusted sensor data with Tweets and other social feeds like Flickr and YouTube. The team layered Tweets over an information-rich map that showed earthquake location, a shakemap, and aftershocks from USGS. The resulting product helped the world understand the impact of the earthquake and resulting tsunami.

Adding Tweets to mapping technology gives insights into what people are saying and where they are saying it. It can highlight spatial trends in the conversation. Combining a Twitter conversation with authoritative data sources, like 911 calls, insurance claims, demographics, weather reports, and earthquake feeds, can provide a human perspective on the situation. By adding social intelligence to its mapping and analyzing the Twitter conversations, Esri visualizes the most engaging Tweets over space and time to get a better understanding of how a crisis event spreads and where resources are needed. Esri maps can even help predict the weather, just take a look below.

Implementation

@Esri‘s proof of concept uses public Tweets to tell the story of what is happening on the ground in real-time. This has been hugely successful and has allowed the team to generate more projects that use Twitter in decision making platforms for retail, public safety, and insurance customers.

Esri’s Twitter mapping work began with the Twitter Search API. Although the API provided a low barrier of entry for accessing geo-tweets filtered by topic, Esri quickly realized that the quantity of information available through the Search API could not facilitate decision making. Esri turned to Twitter’s data partner Gnip to provide a source of data that they combined with their spatial analysis engine. Esri has prototyped and will switch to using the Streaming API to find Tweets for all pubic facing applications on Esri.com, like this Severe Weather Map.

Impact

The Japan earthquake map was picked up by news organizations including CNN, ABC, Al Jazeera, and Wired resulting in over 500,000 page views to the application on Esri.com in the days following the event. Typical page views for Esri.com disaster response pages are around 5,200 per week. Esri customers and partners have also had great success implementing Twitter in their maps. Esri’s Spain’s map of the 2011 elections, combining demographics, polling locations, and social conversation, received 4 million requests per hour at its peak and was linked from the home page of Spain’s largest media organization.

As a side note, Esri built the Pubic Information Map application featured in the maps in this case, and encourages readers to go to ArcGIS.com to download the code and begin exploring social media mapping for workflows.

Using the Twitter Search API

Limitations

  • The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.
  • You cannot use the Search API to find Tweets older than about a week.
  • Queries can be limited due to complexity. If this happens the Search API will respond with the error:{"error":"Sorry, your query is too complex. Please reduce complexity and try again."}
  • Search is focused in relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using the Streaming API instead.
  • The near operator cannot be used by the Search API. Instead you should use the geocode parameter.
  • Queries are limited to 1,000 characters in length, including any operators.
  • When performing geo-based searches with a radius, only 1,000 distinct subregions will be considered when evaluating the query.
  • In API v1.1, the Search API requires some form of authentication — either OAuth 1.0A or app-only auth
  • Recent Enhancements

    • API v1.1’s Search methods return tweets in the same format as other REST API methods.
    • Classic pagination is not offered in API v1.1. You must use since_id and max_id to naviagte through results.
    • The user IDs returned in the Search API now match the user IDs utilized in the Twitter REST & Streaming APIs. You no longer need to maintain a mapping of “search IDs” and “real IDs.”
    • In v1, use include_entities=true to have Tweet Entities included for mentions, links, media, and hashtags.
    • in_reply_to_status_id and in_reply_to_status_id_str are now included with @replies, allowing you to know the replied-to status ID, which can be looked up using GET statuses/show/:id.

    Rate Limits

    Rate Limiting on API v1.1’s search/tweets

    The GET search/tweets is part of the Twitter REST API 1.1 and is rate limited similarly to other v1.1 methods. See REST API Rate Limiting in v1.1 for information on that model. At this time, users represented by access tokens can make 180 requests/queries per 15 minutes. Using application-only auth, an application can make 450 queries/requests per 15 minutes on its own behalf without a user context.

    Rate Limiting on deprecated search.twitter.com

    The Rate Limits for the legacy Search API are not the same as for the REST API. When using the Search API you are not restricted by a certain number of API requests per hour, but instead by the complexity and frequency.

    As requests to the Search API are anonymous, the rate limit is measured against the requesting client IP.

    To prevent abuse the rate limit for Search is not published. If you are rate limited, the Search API will respond with an HTTP 420 Error. {"error":"You have been rate limited. Enhance your calm."}.

    PUTTING TWEETS ON THE MAP: LESSONS LEARNED AND THE WAY FORWARD

    On July 20, 2012 by Nate Ricklin

    Twitter Big Data + Geolocation = Massive Insight

    It’s a simple idea: Twitter + Geo.  What are people saying and where are they saying it?  These are basic questions, but getting the answers is surprisingly difficult.  In this blog post I’ll talk about some of the shortcomings with the Geo layer in Twitter’s API offerings, and what we built to get the functionality that we needed.

    At first glance using Twitter’s built-in “geo” functionality seems pretty straightforward, but dive into it and you’ll soon realize that there’s a lot to be desired.  In fact, Talking with my good buddy Charles at GNIP, catching a stream of tweets coming from a geographic area is notoriously difficult, and there’s no clear good way to do it yet.

    Twitter’s Built-in Geo-tags

    The first thing you might try when trying to put tweets on the map is looking at the “geo” field in tweets returned from the Twitter Search API.  Go ahead, try it out:  http://search.twitter.com/search.json?q=blue%20angels .  Look at the “geo” fields in the returned results, and you’ll see that exactly zero tweets have embedded geotags.  In our own internal testing, we typically see that only about a quarter to a half a percent of tweets actually have embedded geotags.  I’ve never seen it above 1%.  That’s a lot of data flying around without a home.

    Search API Location Information

    Twitter’s search API allows you specify a location (in lat/lng) and a radius that those tweets should originate within, and it goes by both actual embedded geotags in tweets as well as the “location” field that people fill out in their Twitter profiles in free-form fashion.  Problem solved, right?

    Well, not quite.  There are still a couple of problems with this:

    If you’re monitoring many keywords, the number of searches you need to perform (number of keywords you care about times the number of locations you care about) starts to really add up and you quickly run into API limits trying to track it all.

    The second problem is that Twitter’s Search API location search has been, and remains buggy and inconsistent.  Here are a few links to some of the issues that I’ve been following on this front:

    https://dev.twitter.com/discussions/3360 – General Location Bugginess

    https://dev.twitter.com/discussions/4004 – Geocode search results fall out of specified radius

    https://dev.twitter.com/issues/98 – Issue #98: Geocode search results fall out of specified radius (allegedly fixed as of 2012-05-17)

    https://dev.twitter.com/discussions/4003 – Geocode search volume lower than expected

    https://dev.twitter.com/issues/141 – Issue #141: Geocode search volume lower than expected (not resolved)

    But check this out: Perform this geo-search out in the Nevada desert:

    Streaming API Location Information

    Well maybe the Twitter Streaming API offers a solution? The main problem with the Streaming API is that you can filter by keywords OR filter by location, but not both at the same time.  Here’s the official word from the Twitter documentation:

    http://tomnod.com/2012/07/20/putting-tweets-on-the-map-lessons-learned-and-the-way-forward/

Apigee: Making API’s easier

Apigee API Providers

https://apigee.com/console/others

After doing research on authentication for API’s, I came across Apigee. This is a tremendous tool for developers. With many API Providers, including Twitter and Foursquare this API management platform provides the opportunity for mashups among several social media API’s. I signed up for the service and tested authentication through twitter. I did a query on trends in Chicago using the WOIED. The authentication was successful and my results show Apigee snapshot of trends in Chicago. The results returned included trends like Mayweather, Bears, and Vikings. This is only a simple example, but was used to test authentication.

content-type:
application/json;charset=utf-8
x-frame-options:
SAMEORIGIN
x-rate-limit-remaining:
14
last-modified:
Sun, 15 Sep 2013 21:14:53 GMT
status:
200 OK
date:
Sun, 15 Sep 2013 21:14:53 GMT
x-transaction:
5d9ed74c8d02f3eb
pragma:
no-cache
cache-control:
no-cache, no-store, must-revalidate, pre-check=0, post-check=0
x-rate-limit-limit:
15
expires:
Tue, 31 Mar 1981 05:00:00 GMT
set-cookie:
lang=en
set-cookie:
guest_id=v1%3A137927969384488540; Domain=.twitter.com; Path=/; Expires=Tue, 15-Sep-2015 21:14:53 UTC
content-length:
1526
x-rate-limit-reset:
1379280593
server:
tfe
strict-transport-security:
max-age=631138519
x-access-level:
read-write-directmessages
[
  {
    "trends": [
      {
        "name": "#teamlevi",
        "url": "http://twitter.com/search?q=%23teamlevi",
        "promoted_content": null,
        "query": "%23teamlevi",
        "events": null
      },
      {
        "name": "#MINvsCHI",
        "url": "http://twitter.com/search?q=%23MINvsCHI",
        "promoted_content": null,
        "query": "%23MINvsCHI",
        "events": null
      },
      {
        "name": "#Bears",
        "url": "http://twitter.com/search?q=%23Bears",
        "promoted_content": null,
        "query": "%23Bears",
        "events": null
      },
      {
        "name": "#Vikings",
        "url": "http://twitter.com/search?q=%23Vikings",
        "promoted_content": null,
        "query": "%23Vikings",
        "events": null
      },
      {
        "name": "EJ Manuel",
        "url": "http://twitter.com/search?q=%22EJ+Manuel%22",
        "promoted_content": null,
        "query": "%22EJ+Manuel%22",
        "events": null
      },
      {
        "name": "#Mayweather",
        "url": "http://twitter.com/search?q=%23Mayweather",
        "promoted_content": null,
        "query": "%23Mayweather",
        "events": null
      },
      {
        "name": "Christian Ponder",
        "url": "http://twitter.com/search?q=%22Christian+Ponder%22",
        "promoted_content": null,
        "query": "%22Christian+Ponder%22",
        "events": null
      },
      {
        "name": "Devin Hester",
        "url": "http://twitter.com/search?q=%22Devin+Hester%22",
        "promoted_content": null,
        "query": "%22Devin+Hester%22",
        "events": null
      },
      {
        "name": "Jay Cutler",
        "url": "http://twitter.com/search?q=%22Jay+Cutler%22",
        "promoted_content": null,
        "query": "%22Jay+Cutler%22",
        "events": null
      },
      {
        "name": "Jordy Nelson",
        "url": "http://twitter.com/search?q=%22Jordy+Nelson%22",
        "promoted_content": null,
        "query": "%22Jordy+Nelson%22",
        "events": null
      }
    ],
    "as_of": "2013-09-15T21:14:53Z",
    "created_at": "2013-09-15T21:04:18Z",
    "locations": [
      {
        "name": "Chicago",
        "woeid": 2379574
      }
    ]
  }
]

What combinations of API’s would provide an interesting data tool for the city of Chicago?

Apigee Dashboard

Apigee can also be used for development of mobile apps through their dashboard feature.

Sources:

http://techcrunch.com/2009/08/25/apigees-ambition-is-to-be-the-google-analytics-for-apis/

Social Media as a Tool for Urban Design


This video discusses how we can use social media as a design tool for the city. Speakers talk about harvesting the twitter, foursquare, and sonar data through APIs and how this relates to an urban environment. Peoples movement patterns are studied through check-ins on four square. What tools can we use to connect people through the city? What services are people going outside their neighborhoods for? Is there an opportunity for small businesses to move in and do we have zoning or business licenses in place to support this?  For instance, if people are going far to go to the coffee shop etc. As the video demonstrates these types of technologies are already being used by Starbucks.

By looking at these movement patterns and trends on twitter give we can begin to identify a place. By using geo locations one can begin to map these qualities of the city. What type of rezoning of neighborhoods would support small business and create jobs in communities in Chicago?

Sarah Williams discusses a research project, Industry in Motion, which studies the movement patters of fashion designers in the New York Garment District. This area is under contention for rezoning. Designers check in throughout the day in Foursquare. What she explains is that this manufacturing is important to the vitality not only locally but regionally. The manufacturing sector has declined greatly in the U.S., for example in Detroit with the loss of the auto manufacturing industry, and we need these type of manufacturing sectors they support our work force.

What do these moving patterns tell us in Chicago?

New York Garment District: Designer Movements (Manufacturing Industry) through Foursquare