I’ve been experimenting for years with how to create and structure community news feeds. Readers want to follow breaking news and relevant information as it becomes available — and not just on Facebook and Twitter. All this time, I’ve been looking for an easy and reliable way to merge a bunch of related RSS feeds into a single feed that would make this easier for my readers.
What seems like a simple problem turns out to be difficult to do well. The problem is that it’s an easy enough problem that everyone thinks they can solve it, so no one is willing to invest what it takes to create a quality implementation. I’ve tried a lot of methods short of writing my own aggregator, which have no interest in doing.
Speed is an issue for all aggregators, because they must recognize that a feed has been updated, read the feed, add it to the merged feed, and deliver the new feed to its subscribers. This can slow things down, because neither the aggregator nor the end user can poll too often. Also, many of these services do not scale very well and slow to a crawl as soon as they become popular.
A lot of sites aim to create merged feeds for you. About half those you’ll find on most lists via Google no longer exist. So how can you count on any of the others to be there in the long run, or to be fast and reliable enough to use in a production system?
Yahoo Pipes is ineffably cool, and a pretty good solution, but it’s slow, kind of obtuse, and unreliable. It doesn’t appear to be getting a lot of attention from Yahoo, which is a good sign you shouldn’t make it part of your infrastructure.
The Simplepie RSS parser has a nice-looking website and lots of links to interesting examples, but the project seems dead in the water. I was able to find a good example of an RSS aggregator that did most of what I was looking for, modified it to make it my own, and added it to my site. However, maintenance became a problem. First, the script slowed down my host. Shortly after that, it started to get aggressive in its reading of my friends’ sites. It was becoming clear that unless I wanted to go from being a publisher to a programmer, that rolling my own aggregator from Simplepie simply took too much work.
I finally realized I could solve this problem using WordPress, of all things. In a couple of hours, I was able to create a news aggregator using WordPress and a plugin called FeedWordPress.
My new aggregator can pull in headlines and summaries from multiple sites and deliver a combined feed. The aggregator itself isn’t intended to be a destination — right now I’m the only person who even knows where it is. It’s just a bit of infrastructure that works reliably and fast. I’m able to read the merged site’s feed with the magpie reader built into Expression Engine and deliver the results as a local news digest.
FeedWordPress has a terrible name and an ugly website, but it works really well and has plenty of options for managing the way you gather and display feeds. And it’s shareware. I sent in my donation.