Feed WordPress 101: Feeding The Machine

This is part 3 of 5 in a series of posts for Building Connected Courses: Feed WordPress 101

• Basic Concepts of Syndication – and what to think about even before you touch that WordPress thing
• Installing and Setting up Feed WordPress – Minimal settings, and planning the way content is sliced, diced, and recombined
• »» Feeding the Machine «« – How to get RSS feeds into the aggregator without losing a finger
• Some Feed Magic – Optional ways to improve feeds from sites such as flickr, twitter, etc, creating a twitter archive, RSS Feed TLC
• A Few More Tricks – leveraging categories, adding attribution, setting featured images

At the end of our last section on setting up Feed WordPress, we reviewed the basic process for adding a site to our aggregator. In this post, I explore this farther, because, while one lump of coal at time gets them into the machine, but often we want a shovel full. And not all coal is the same.

I will share some approaches for adding new sites to Feed WordPress, and share a few of the gotchas you might face with feeds from different sites. For the most part, this is a manual process, though as we see below, you can as them in bulk. If you clean them off first.

What Are You Putting in the Machine? How Much?

If you are just syndicating a few feeds into your own site, or maybe for a single class of students, it’s not much work to manually add the feeds. Collecting them can be as simple as asking students to email you the address for their blog, or collecting them via something like a Google Form. Many of us dream of a single one button solution, but my experience in creating at least half a dozen of these sites are is… feeds can be messy.

A range of examples:

• Add Them One By One. On my blog here, I have maybe three feeds from other sites that I may blog on where I want their posts to be republished here. For example, on my photo site I use a special category for just posts I want to be re-published here.
• Custom Code a [Semi] Automatic Process Martha Burtis did much of the work to create a signup form for ds106 (wher we subscribe to more that 750 blogs in Feed WordPress). This requires a paid plugin (Gravity Forms) and custom code to process the form input. I used a similar approach on the Thought Vectors site (123 blogs subscribed); working with Tom Woodward, we created a quasi “Choose an Adventure” approach to guide people to enter the correct information. While the code worked well, we found we still had to manually fix more than half the submissions. You might be surprised that many people cannot cut and paste the URL for their own site (I had one student who only found it by googling her blog name, sigh)
• For Project Community, ETMOOC, and Connected Courses— and my best recommendation; I created a Google Form to collect the blog information. With some knowledge about the feed structures, it does not take too much longer to process them (see below for more). I actually checked more than 500 blog links in ETMOOC, which sounds crazy. But as a subtle added benefit, you can right away scan the “audience” of your course or at least how they represent themselves in a blog.

Why is Not Easy?

There are cases where it is much easy– when a blog is entirely devoted to your course content, e.g. when you are subscribing to all blog posts, it is dead easy. You can let Feed WordPress “autodiscover” the entire blog feed URL, that information is embedded into its HTML code.

Or you can try some code I have used on other sites, the “Magic Box” that should automatically find the main RSS feed for an entire blog.

It gets trickier when you let people use feeds for a category, tag, label in a blog. WordPress blogs behave the best; if you enter the URL for say the category on this site — http://cogdogblog.com/doodoo, Feed WordPress will report that it finds like seven possible feeds. Check each one carefully- two are the feeds for the entire blog; one is for comments, and one is for the category content. The content preview on the right should help determine if it is finding the correct content.

But it will not work for Blogger; where the organizational scheme is called “labels”. The URL for a label looks something like http://cogdoghouse.blogspot.com/search/label/coolstuff — this will display all posts I have applied that label to. But Feed WordPress can never locate the proper URL for it’s RSS feed, in fact it does not exist anywhere in the code. Blogger actually hides it (why?) — the URL for a blogger label RSS feed actually looks like http://cogdoghouse.blogspot.com/feeds/posts/default/-/coolstuff. The same goes trying to find feeds for tumblr tagged posts.

And so we circle back to the strategy you choose for your course. You can insist/require that people use/create a blog that is 100% devoted to the content of your course. Or you can live with knowing that when I add my blog feed to the course, you may receive a lot of non relevant blog posts in your site if I write about other things.

If you are teaching a very wide open course, you likely want to allow the most flexible range of options. This can mean you have to amplify up your Feed Detection Skills (see below).

In summary, Feed WordPress is most effective when you add the direct URL for the appropriate RSS feed, rather than asking it to figure the feed out.

Forming It Up

We are using a Google Form to collect blog addresses for Connected Courses (embedding the form in a WordPress page). I am not going to review how to make forms in google docs. But the main advantage is that it tracks all your info in a spreadsheet, and you can set up notifications so when someone submits a form, you can act on it right away.

What you ask for us up to you. I typically ask for:

• A name (leave an option to create a pseudonym)
• email (just if you need to contact them if the URL does not work)
• twitter name (often easier to contact people, also, you could use to create a twitter list)
• any other info you may want to collect to help you categorize feeds (For connected Courses, we ask a role)
• The URL for their content
• An option for them to leave comments

The URL of course is the critical part, and no matter how well I try to explain it, many people still do not quite get it. Some people will enter a URL for a single post; or a page/static content. Or something that is not even a URL.

When I review the submissions, I go to the step of verifying that the URL works. In some cases (see below for a guide), I will have to edit to make sure it is a URL Feed WordPress can handle.

Here is a peek at the responses form for the Connected Courses form (it’s early so we only have a few)

My method for keeping track is to color a row green once I have added it to Feed WordPress, so at a glance I can see how many new ones there are to process. If there is a problem with a feed, I might color it pink until I can. A Convention like this is critical if multiple people might be monitoring the form.

The entry on line two from Mike Wesch looks ok, he has created a tag on his blog, but hmmm, the URL http://mediatedcultures.net/tagged/cc does not work. It might be because Mike has not yet posted anything with that tag, but I am guessing too that the URL (since it is a wordpress blog) should be http://mediatedcultures.net/tag/cc (which does not produce anything).

I am not picking on ya, Mike– it’s a great example. If I enter the URL in Feed WordPress, it will generate an error. So in this case, I will have to contact him for followup.

Do you have to add them one at a time in Feed WordPress? Heck no. Once you have a series of feeds, if you can copy them from the spreadsheet or a text file, one URL at a time, say… these are my three feed URLs

Copy them. Go to your WordPress Dashboard, and find the link under Syndication for Syndicated Sites. On the right side where last time we added a single source, click the link labeled Add Multiple. Paste in the URLs.

Note the message at the top. You need to be sure these URLs are either the URL for the home page of a blog (since the first one it finds will be the one you want) or is the exact URL for a blog feed. You will not get a chance to pick a feed.

Feed WordPress will check each URL, if it cannot find any feeds you will get an error. But look at that last one, it found a feed for Mike’s blog, great!

Not so great. Mike uses Feedburner on his site- the only feed it can ever provide is a feed for his entire blog (he would have to create a new Feedburner URL for a tag on his site- see, nuances). I can uncheck his feed to skip it.

So we can add batches of feeds. I have done as many as 20 at a time.

The way we set up the automatic categories in the last post all new feeds submitted will be also set to add the Syndicated and Blogs categories to all posts. If you are doing any extra categorization per feed (role, section, group), you will have to add that information.

Here is how we edit single feeds. From the list of feeds in the syndicated sites view, hover over a site you want to edit, and click Categories

When you get to the Categories & Tags Settings: CogDogBlog Lab screen, scroll down to Categories. It indicates that it is using already the global default categories Syndicated and Blogs. We can then add any other categories for this blog we may want to add, such as maybe the Participants role.

Once you make these changes, all new posts will be put in the categories strong>Syndicated, Blogs, and Participants.

And if you have a feed that is not a blog, you can override the default settings by checking the option for No. Only use the categories I set up on the left. Do not use the global defaults for posts from this feed and then click the boxes on the left for the ones that are appropriate.

You can use this process for blog URLs you collect in any way. The general steps are:

• Check the URLs- make sure they even work, and also that they point exactly to the main blog URL or a proper URL for the feed from a tag, category, etc.
• Make the list of URLs to add, one URL per line
• Add one or more URLs to Feed WordPress
• Edit each Feed to add any categories if appropriate

Becoming a Feed Detective

Sure you can let Feed WordPress do all the work for you, and for most Connected Courses you will be fine, especially where you, as teacher, may suggest which blog platform students should use. In wide open courses, you may get everything and anything under the sun. in either case, your savviness is better off understanding more about where these RSS Feed things live.

View Source- Get into the Code

You cannot get hurt looking beneath the hood of a web page. Go ahead, use the command in your browser to View Source (or something similar). OMG! It’s gibberish:

Fear not. Command-F is your friend (it is the command to search)- search on “RSS”. You are looking for parts of the code that look like:

or:

RSS feeds come in 2 or 3 or 92 varieties, but Atom and RSS are equivalent as far as what Feed WordPress needs. This is from a blogger site; WordPress blogs have have a few more. The first one it finds with the label application/atom+xml or application/rss+xml is what Feed WordPress looks for, and the URL is what we seek.

If you cannot find any of these, the site may not have an RSS feed at all. If you examine the source of a Google Label URL, e.g. http://cogdoghouse.blogspot.com/feeds/posts/default/-/coolstuff, you may find a feed, but it will be for the overall blog.

But this is usually a last resort. The three major blog platforms we typically deal with have RSS feeds you can deduce from patterns.

WordPress

WordPress, whether self hosted or on WordPress.com, Eduglogs.org. etc makes it really easy to find a feed for almost anything- the over all site, a category archive, a tag archive, event comments– just put /feed on the end of any URL* (see below for the exception).

If the main blog URL is http://iamsoconnected.org/ — the feed for the entire blog is http://iamsoconnected.org/feed

For a category URL like http://iamsoconnected.org/category/recipes — the feed for all posts in this category is http://iamsoconnected.org/category/recipes/feed

For a tag URL like http://iamsoconnected.org/tag/yummyyummy — the feed for all posts with this tag is
ttp://iamsoconnected.org/tag/yummyyummy/feed

You can even get an RSS feed for all comments on a WordPress blog – http://iamsoconnected.org/comments/feed

Now here is the exception. A few WordPress blogs are not set up to use what are called “pretty URLs”; a category URL might look like: http://iamsoconnected.org/?cat=23 if that is the case, what we tack on to its feeds is &feed=rss2 — so that a category feed for this blog is http://iamsoconnected.org/?cat=23&feed=rss2

A tag feed that looks like http://iamsoconnected.org/?tag=224 — the feed is http://iamsoconnected.org/?tag=224&feed=rss2

And the feed for the entire blog? Bueller? Bueller? yes http://iamsoconnected.org/?feed=rss2

Blogger

Feed WordPress will be able to find the correct feed for an entire site, so a blog at http://iamsoconnected.blogspot.com will end up being found as http://iamsoconnected.blogspot.com/feeds/posts/default

But if this blog is using a label such as pinkpony the feed will have to be manually constructed as http://iamsoconnected.blogspot.com/feeds/posts/default/-/pinkpony

It’s not pretty, but that’s what works.

tumblr

Like WordPress, tumblr has a simple formula for finding a feed- just add a /rss to the end of any tumblr URL.

Feed WordPress can find the feed for an entire tumblr- for a site at http://iamsoconnected.tumblr.com it’s feed is found at http://iamsoconnected.tumblr.com/rss

If a participant is using a tag to denote their content for syndication, it’s URL will look like http://iamsoconnected.tumblr.com/tagged/recipes and (drumroll) it’s RSS feed is http://iamsoconnected.tumblr.com/tagged/recipes/rss

One thing to watch out for on tumblr is that people typically will copy it’s address from their dashboard; which turns out to be http://www.tumblr.com/blog/iamsoconnected For anyone except the owner, this URL will automatically resolve to http://iamsoconnected.tumblr.com BUT you cannot use http://www.tumblr.com/blog/iamsoconnected/rss as an RSS feed.

Squarespace

While not as common as other platforms, Squarespace offers a lot of flexibility in terms of design, and to create sites where the blog part is not necessarily at the front. That said, it does have full featured blog tools with comments, tags, and categories.

The URLs for tags and categories are visible wherever you see a link or menu, but there is no indication of an RSS feed. But you can easily generate the feed by taking on &format=RSS to any category/tag URL e.g. the links to a category might be

http://connectaplooza.squarespace.com/?category=connectedcourses

so the RSS feed for this is

http://connectaplooza.squarespace.com/?category=connectedcourses&format=RSS

Equally, users of squarespace might use tags to identify their posts

http://connectaplooza.squarespace.com/?tag=ccourses

and you should be able to guess the feed url

http://connectaplooza.squarespace.com/?tag=ccourses&format=RSS

medium

Medium presents a compelling and popular platform for writing. It provides no tags or categories to organize writings, but each author does have an RSS feed that corresponds to their content.

So a link to a writer’s work in medium looks like (this is mine)

https://medium.com/@cogdog

Yes, an “@” sign in the URL looks weird, but is actually acceptable as a proper URL. The RSS URL that medium provides is not indicated anywhere on a published page or author archive, but if you sniff the source of any medium article, you should be able to find that the form of a feed is:

https://medium.com/feed/@cogdog

But using this as a source in Feed WordPress will generate a HTTP Request Error (it’s a long story but medium blocks that method Feed WordPress uses to fetch feeds).

It’s a mess. I even wrote an article on there @medium your rss feeds are @ mess. I got a very detailed, helpful (in terms of explaining) response from medium engineer Nick Santos.

All is not loss, if you do not mind using a Yahoo tool that may or may not be around a long time. But thanks to a suggestion from Martin Hawksey I was able to make a Yahoo Pipe that converts the medium feed into one that Feed WordPress can handle.

It does work, and while you may feel like the whole thing is held together with rubber bands and duct tape… well you are right. Welcome to the internet.

Other Platforms

You might find GoogleSite, drupal, and other assorted URLs come into your site. You might have to look around the site for a Syndication link, or poke around the source. Not all sites have RSS feeds. If you run into one you cannot puzzle out, leave me a comment below- I am working on my PhD in Feed Detection.

Next Step

We have dealt with just blogs here. In the next section, we will look into dealing with feeds from sites like flickr, instagram, Youtube, diigo, twitter.

Building Connected Courses: Feed WordPress 101

• Basic Concepts of Syndication – and what to think about even before you touch that WordPress thing
• Installing and Setting up Feed WordPress – Minimal settings, and planning the way content is sliced, diced, and recombined
• »» Feeding the Machine «« – How to get RSS feeds into the aggregator without losing a finger
• Some Feed Magic – Optional ways to improve feeds from sites such as flickr, twitter, etc, creating a twitter archive, RSS Feed TLC
• A Few More Tricks – leveraging categories, adding attribution, setting featured images

---

Featured Image: shovelingcoal flickr photo by Mr Michael shared under a Creative Commons (BY-NC-SA) license