Digging to the Bottom of an Obscure Twitter API Thing

It’s a very satisfying thing to succeed at digging.

While I am way way behind Tom Woodward in API wrangling, I’ve put them to use in projects- using the FLickr API for pechaflickr and 5 Card Stories.

I also put the twitter API to use in the Daily _____ sites, e,g, the DS106 Daily Create.

The way these sites work is that for their tweets to be associated with a daily, people need to include in their tweets a specific account (e.g. @ds106dc) and the hashtag for that day (e.g. #tdc2025). Under the hood, my code checks twitter ever hour for mentions to the specific account, and for each one of the finds, it adds as a response if it has not been added before and if it includes a hashtag in the series.

I’ve been bothered in more recent sites, like the Daily Digital Alchemy and the new Daily Extend that I see fairly often tweets that should be picked up (they have the necessary elements) but the site misses them.

For example in today’s Daily Extend #oext8 Tunnel to your antipodes- and then swim? (this one is quite clever, you should try it!) I saw in my tweetdeck column a response from Sally Wilson (which is extra cool as she is not directly part of the project)

@ontarioextend Antipodean time tunnelling: Toronto – Indian Ocean, 1776 to rendezvous with Capt James Cook on the HMS Resolution #oext8 pic.twitter.com/7RPvvSxUQQ

— Sally Wilson (@pomathorn) August 2, 2017

Sally did everything correct, including @ontarioextend and the hashtag #oext8 … but her tweet was not being picked up by the site.

Bummer.

Some days I jump in the dashboard and manually add them (they are saved as a custom post type), muttering, “one day I will fix my buggy code.”

That day was today.

To back up, to do this kind of thing, my site has to authenticate with the Twitter API and make it’s request. When I build the site I put to use the Storm Class for WordPress plugin I found when I built the site. It is designed to manage the oAuth authentication and to pull data for a user’s timeline as a support for other plugins.

I made my own version of the plugin, because it’s call to the API:

$result = $connection->get('statuses/user_timeline', $options);

1	$result = $connection->get('statuses/user_timeline', $options);

will get a users timeline, whereas what I need to get mentions is

 $result = $connection->get('statuses/mentions_timeline', $options);

1	$result = $connection->get('statuses/mentions_timeline', $options);

As a diagnosis, I needed to look at the data I was getting back from the Twitter API. So I used the Apigee Twitter API Console — logging in via the twitter credentials for @ontarioextend (third menu), and issuing a call to https://api.twitter.com/1.1/statuses/mentions_timeline.json

I was pretty stumped as all I could see was the header responses, where the heck is the data? It’s way way at the bottom right, a single line of JSON

Running a twitter API call in Apigee… where the heck is the data?

I could not figure out anyway to get at the JSON results; so brute forced it by doing a command A – Command C, literally copying everything in both of those result panes. I pasted it into BBEdit, but till that JSON was one big blob. So I copied just the JSON part, and pasted THAT into JSON Editor Online.

Now I could inspect the JSON results (I am sure there is a more direct way, but this is how I dig at code, with a big shovel).

I searched to find the data for Sally’s tweet and saw a problem:

Just part of the JSON data for Sally’s tweet

You can see the text of her tweet, but the array for the hashtags is empty. How is that? Her tweet had the #oext8 hashtag in it.

My first thought there was a bug with the Storm code I had used and I might have to recode everything with the more robust TwitterOAuth PHP library. I searched in the Storm code, maybe it was not parsing the hashtags right. Long shot.

But the code does not include anything with parsing twitter data… doh, that’s not even what it does. All the code does is set up communication with twitter (authentication), sends an API request, and returns it as a PHP array.

Now I am on to something. Maybe it’s twitter’s fault, not me?

I poke around the results and find a tweet who’s hashtags are picked up in the API call (it happens to be my tweet)

You can see here that the #oext5 hashtag is returned in the hashtags array, so it is picked up by my code.

It might be obvious now in the screenshots what the difference is… and it jumped at me. For Sally’s tweet the parameter truncated was returned as true whereas in my tweet, truncated is false. And you can see that the text of Sally’s tweet in the data is truncated, so that the hashtag she includes is at the end, thus chopped.

I’m on the trail now! Dig! I google on twitter api truncated and find a question in the Twitter developers forum Truncated text and media entities not returned. At the bottom is the chunk of gold:

Add tweet_mode=extended to the url

This is another parameter I can test first in the API console, by changing the API call to

https://api.twitter.com/1.1/statuses/mentions_timeline.json?count=50&tweet_mode=extended

1	https://api.twitter.com/1.1/statuses/mentions_timeline.json?count=50&tweet_mode=extended

Running the results through by same Rube Goldberg method, I can now verify that Sally’s tweet’s text is now full in the data, and the hashtag array correctly contains the one she used:

And that is the fix I need to add to my modified version of the Storm Twitter plugin, in oauthGetTweets function changing

 $options = array_merge($options, 
    array('screen_name' => $screenname, 
          'count' => 20));

$options = array_merge($options,

array('screen_name' => $screenname,

'count' => 20));

to be

 $options = array_merge($options, 
    array('screen_name' => $screenname, 
          'count' => 20,
          'tweet_mode' => 'extended'));

$options = array_merge($options,

array('screen_name' => $screenname,

'count' => 20,

'tweet_mode' => 'extended'));

And I have been testing it on the 3 Daily _____ sites I manage and it seems to be Getting All The Tweets it should.

I have updated the GitHub repo for the Daily Blank but if you are running one of these sites, the only file you need to update is StormTwitterClass.php in the oauth-twitter-feed-for-developers plugin.

I am sure there is a reason that the twitter API returns truncated tweets by default but it seems a bit quirky that it would base what it returns for hashtags on the truncated tweet and not the original.

But what do I really know about APIs?

Not a whole lot. Enough to dig around and sometimes, find the right bone.

It’s damn satisfying.