Making more of Foursquare check-ins (part 1)

I read Jamie Rubin’s Going paperless blog. Jamie is a science fiction writer and Evernote ambassador.

In a recent post, he describes how he integrates Evernote, Foursquare (via IFTTT) , and Sunrise calendar. (Other services and products may be available, these are what I use). You can read his post to get some ideas, but one key thing he does is to create a separate Evernote note for each check-in. This is easy using an IFTTT recipe.

I often try some of his ideas out, although there are two main problems I encounter:

  • He lives in the US – some of the services are not available in the UK (although there may be alternatives).
  • He uses Mac/iOS, and I use Windows/Android. Again, some of the apps he uses are not available for Android.

Having said that, I take some of the ideas seriously enough to think about coding some solutions.

Evernote, IFTTT, and Foursquare

I’ve been using IFTTT for quite some time – particularly since getting a more powerful Android phone – and I’m looking for ways of making it easier to record things such as check-ins.

IFTTT is a go-between for different applications – the name comes from IF That Then This. For example IF Foursquare checkin Then add Evernote entry.  These triggered events are called recipes, and so the things that you can add (from the That to the This) are called ingredients.

My Foursquare to Evenote IFTTT recipe appends records to a single note for each Foursquare check-in. This note is in my Journal notebook. (I use similar recipes for recording other things into my journal).

There are some advantages – I have a single document for check-ins, for example. But there are disadvantages:

  • the document can become very long (and new entries are appended – inserted at the end of the document) and need lots of scrolling;
  • periodically, the document needs to be split and that can create conflicting changes;
  • it is not necessarily easy to find a particular check-in;
  • check-in dates are in “March 31, 2013 at 08:02PM” format which makes it hard to sort.

So I thought that I’d change the recipe to generate a separate note for each check-in. This has the advantage that the create date reflects the check-in (well, more accurately the date the note was created by the IFTTT recipe) making it easy to sort and filter using the powerful search tools that Evenote provides.

So far so good.

However, I thought that it would be good to create notes for the older check-ins too – how hard could it be?

How hard could it be?

Well, for a start I have more than 1400 check-ins, 1200 them using the IFTTT recipe. Some of the them are like

Others (older ones) are like

That’s going to be a problem to parse. Not impossible though.

(Also, the check-ins journal has weather information pushed in at 0600 each day).

In practice, I can copy the notes into my favourite programming editor, run a few regular expression-based search and replace operations, and bingo! I get a standardised format:

20140414 16:55|cr2uIG|Some place|For some reason

That has the date/time in a sensible order, the | is a separator (you couldn’t see the tab characters), the second field is part of a short link to the venue, the other two fields are the venue and a description.

The more geeky readers may realise that I could have got the original data by using the Evenote API, or (as I tried) exporting notes from Evenote to create an enex (XML) document that could be parsed. Let’s just say that there are problems with either approach, mainly due to understanding the markup and formatting of the different entries in my journal documents.

This is the point at which I discovered that for some reason (probably temporary insanity) that I didn’t have links to all the check-ins. In fact, about half of them were missing link information. Also, the new recipe creates a note that contains both a link to the venue, and a link to a (static) Google map image. (Different recipe, slightly different ingredients).

I also discovered that for some chains (like supermarkets) the venue name is the same for every location, but the venue link is different, so a pain when I know that I check into some supermarkets in multiple places.

So, what to do.

A possible approach

Maybe I can get the information from Foursquare. In fact, maybe I can get information about check-ins from Foursquare that predate my IFTTT recipe?

There is some good news, and some bad news.

The good news is that Foursquare has a page where you can play with (explore) the API, and there is quite good documentation available. The other good news is that you can get a list of all the checkins if you go to the Foursquare feeds page. (By the way, you need to be logged into Foursquare for this). The interesting feed is the iCal (ICS) option which creates an .ics file containing all the checkins.

The bad news?

  1. The iCal file has a URL to the check-in, not the venue. (My IFTTT has a link to the venue).
  2. To get the venue short link, you need to first get the check-in, then find the venue id, and then retrieve the full venue record.
  3. To get the map picture link you need to get the longitude and latitude from the venue, then construct a Google maps query, then get Google to shorten that link. (I expect other services are also available).

This is what you do

(Apologies to Jimmy Young and the JY prog).

I have a strategy to do this as a Java (or maybe Perl) program:

  1. Download the iCal file from Foursquare (manually).
  2. Parse the iCal file to get a list of check-in Ids. (In fact, I will create a class to hold information about the check-in, including the date/time and description, and venue).
  3. For each check-in, request the check-in object from Foursquare. This will be a programatically constructed request that will require the OAuth token, and will return a JSON object.
    1. Grab any additional information from the object that I care about.
    2. Get the venue ID, and add it to a list of venue objects. I only need to retrieve the venue once even if I have checked in multiple times.
  4. For each venue, request the venue object from Foursquare. Again need the OAuth token.
    1. Grab the short URL for the venue.
    2. Grab the latitude/longitude information for the venue.
    3. Grab any other information about the venue (including the name/description)
  5. For each venue, construct a Google maps request URL. I don’t think I need to actually go there …
  6. Using the Google maps request URL, use the Google link shortener.
  7. For each check-in, using the check-in and venue information (including all the links), create an Evernote note (*).
    1. Title should be like “Checked in to <venue>: <checkin_description>
    2. Note should be in the Journal notebook.
    3. Tags should include foursquare.
    4. Create date should be the date/time of the check-in.
    5. The body should have a link to the venue and include the map picture.

There is a gotcha – Foursquare limits the number of API requests, so I can grab at most 500 items (check-ins or venues) per hour. There may also be a limit for note creation by Evernote.

(*) That’s another world of pain. There are several ways of doing this. The Evernote API is possible. It works with the web service rather than the local notebook, and requires handling authentication. Alternatively, can create notes using the Evenote script application (ENScript.exe). Or, create an .enex import file. For some reason, I favour the latter.

Comments are closed