Smokin’ Yahoo Pipes

cc licensed flickr photo by Wade from Oklahoma

… in which the technology blog author bares his ignorance by discovering a tool that is several years old….

With a new sense of urgency, I implore that some big web company buy and save Yahoo– if not just to preserve the most valuable important vital web 2.0 app in the universe but…. Yahoo pipes must be kept alive.

I’ve not of the concept of Pipes for a long time, but for some reason, never ventured inside (although I have the a personalized use of Social Media Firehouse in my reader).

I follow with amazement how deftly Tony Hirst pipe work weaves together feeds from disparate data sources, performs complex transformations, tosses them out to some Google spreadsheet, and produces some geo-data mashup with photos on a map.

It seems magical.

But its not.

Yahoo pipes really allows you to perform programming like actions on RSS and data content, from a visual interface. It is one of the most amazing and elegant tools I have seen. Please keep it alive. I just got started.

I am getting high on smoking Yahoo pipes (if this sentence does not bring me spam, I have no idea what will… wait, I do, but my Mom reads my blog).

My first pipe is not that complex, but gives me the smallest taste as to what else I might do.

The Problem

Since 2007 I’ve had our NMC Horizon Project use delicious for tagging resources relevant to the project, as a moire dynamic way of easily sharing emerging technology resources. We have tags for each year hz07, hz08, hz09 plus newer ones this year for our Australia-New Zealand version (hzau08]) and the newest on for K12.

I used the feeds in many place- sidebars and pages on the wikis e.g. http://horizon.nmc.org/wiki/Tagging, but I also bring in the current feed as a block on our drupal site (see sidebar of http://horizon.nmc.org).

The problem I was just noodling is that now the current Horizon report is out, but we still want and see people tagging things for that year (hz09) plus we are asking them to tag things for the next round (hz10). So which do I put in my drupal block? Do I end up with a pile of drupal blocks for each year?

So I started thinking I would try some kind of RSS mixing tool, one of those where you can combine several RSS feeds into one. There are a lot of them. And down at the bottom list was Yahoo Pipes– and it seemed like a good idea to finally jump in and play around.

In the Pipes

Yahoo pipes is a free tool, and you just need a Yahoo account to use it (if you use flickr, you are already set up). There’s a fair amount of documentation on the site, including a list of tutorials (that are fed from a tag in delicious, see how recursive this is?).

I’m not going to explain how it works (that’s what all those tutorials are for), but essentially, you start with a blank grid, and you drag little “modules” form the left- each module provides a function, or a thing you can do with RSS feeds. You connect them with “pipes” that connect them into a flow, and at the end you get content.

The powerful thing about this is that you can perform complex transformations on data that comes in from RSS that would only be possible otherwise by writing custom programming code. But what’s better is that in creating it, you think in terms of the high level design of the flow of information, where in programming you’d be in this non linear space of text code abstraction and, if you are like me, getting twisted up in syntax (my kingdom for a missing “;”).

My First Pipe

The other fabulous part about pipes is that when you see one, you can inspect it, clone, it, and essentially could remix pipes into new pipes. Mine is at http://pipes.yahoo.com/cogdog/Sjfy0nsr3hGENOF2rLQIDg. I’ll outline the parts, which took me less than 10 minutes to create (mostly because I kept finding new pieces to try).

So what my pipe does is:

1. Take as inputs 3 horizon delicious feeds 2010 Horizon Project tags, 2009 Horizon Project tags, and 2008 Horizon Project tags. I did find that Pipes could not autodiscover the feeds from the tag page, so I entered the URLs directly. The default delcious feeds give you 15 items, in a URl like http://feeds.delicious.com/v2/rss/tag/hz09?count=15, and you can get more simply by changing that last number. These are “piped” into the Union module, that mixes the feeds together. If I stopped here, I’d get as putput what those RSS Mixers do. But I can do more…
2. There are some things in the tagged feeds I don’t want. We ask that people tag web sites that are about emerging technology. but a lot of people just tag the Horizon project or the report. These are removed with the Filter module where I filter out any tagged items with “Horizon Report” in the title or with “nmc.org” in the link URL.
3. Often many people tag the same site, or tag it in different years, so I apply the Unique module to the flow, which makes sure I only get one item for a specific link url.
4. Next, the Sort module allows me to order the output. I played with a few options, the typical would be reverse ordering by PubDate (date the link was tagged). I questioned this as fewer people tag things with the older tags (you’d have to know the topics to push them into older tag streams, and I may be one of 3 people who remember these). I did try an alpha order by title, and my revert to that. See Pipes are dynamic- you can change all this on the fly.
5. Out comes the new feed!

And that’s it. As you are constructing the pipes, you can tap the orange title on each module, and the Debugger pane at the very bottom displays a preview of the output. This is really essential to test as you build.

Once published, you get all kinds of ways to grab the feed- as simple as the RSS URL, but also adding to MyYahoo, they even have a button to add as a Google widget, but also JSON, PHP, a service to have results emailed…

Pipe Dreams

I have just scraped one molecule deep into the surface of what is possible, What Yahoo Pipes offers is a way to do more with the feed then what they give you. Filter out words, links. Change the order. Connect images from flickr with keywords from another feed source. It is the uber remix machine.

I am hooked on Pipes.

Now someone, please save Yahoo.

 Share this barking on social media

If this kind of stuff has value, please support me by tossing a one time PayPal kibble or monthly on Patreon