Aggregated blog and news feeds


David Kane over at Eph Blog posed a question on how to create a feed that aggregates news sources on a particular topic. As he illustrates, a single news search engine alone limited insofar as it may not cover enough news sources. The comments to his post discuss a few possible ways to solve the problem. Here is one idea I came up with:

First, having the ability to "opt in" to individual news/blog stories will work better than blindly re-syndicated everything in a search feed. It is even harder to try retroactively trying to filter the bad or duplicate entries out after the fact. By the time you get around to editing the feed, other blog aggregators have usually already polled the feed and have remembered the entry you wanted to remove. (Lesson: Don't let bad data get into feeds, as bad data is difficult to recall.)

With that in mind, here is one approach that could work rather effectively. While it may seem complicated at first, I imagine that this is just about the easiest way to give yourself such a high degree of control over the end result. While the technique is biased toward Google there may also be a way to do the same thing with Microsoft or Yahoo or others.

The advantages to this approach are that it:



There are a number of steps to follow, but none are particularly difficult or require any knowledge of programming or the underlying technology itself.

Start by setting up your account.1) Make sure you have a Google account. If you're not already a Gmail user, you can create a new account here.

Next, select the feeds or searches you want to aggregate and share.2) Head over to Google News and enter in a search that brings up relevant results for the topic. For Dave's example, try "Williams College".

3) Notice the Atom link on the left side of the screen. If you are using Firefox 2, click on it and it will prompt you to select your preferred blog aggregator. Choose Google Reader and subscribe to the feed. If you are not using Firefox 2, cut-and-paste the URL into the Google Reader "Add Subscription" box or use the "Subscribe" bookmarklet available on the Google Reader "Goodies" page.

Repeat steps 2 and 3 for every news search you would like to track.4) Go to Google Blog Search and enter another search. Again, this example uses the search keywords "Williams College".

5) Subscribe to each relevant blog search using Google Reader.

Repeat steps 4 and 5 for every news search you would like to track.6) Subscribe to as many other RSS or Atom feeds that you can find that return interesting results for your topic. This could include other news or blog searches or individual blogs on the topic. I recommend getting a diverse selection if you really want to cover everything. For the best results, use feeds that link directly to the relevant content and not an interstitial page.

Repeat step 6 until you have a broad representation of all possible sources on the topic.7) Go back to Google Reader. Click on the "Manage Subscriptions" link to bring up the controls for the list of feeds.

8) Find the feeds that you know will product only high-quality results every time. (Depending on the topic or on the degree of accuracy you require, there may not be any feeds of that high quality.) Select those feeds and add a common label to all of them. For Dave's example, we'd label them "williams".

Next we will add the ability to augment the trusted feeds with hand-picked entries from other feeds. This part gets a little complicated, but the technique works and it is powerful and flexible. But if all you want to do is share trusted feeds, you can skip steps 9-11. You have several choices here, but they all amount to the same thing: create one or more feeds of hand-picked items and use Google Reader to automatically merge them in to the publicly aggregated feed.9a) Use the Google Reader (share) button to flag relevant items. These entries will automatically be shared publicly. (Upside, easiest to flag items. Downside, you only get to "share" things for this one purpose. Requires additional steps.)

9b) Use the Google Reader (star) button to flag relevant items. You will also need to go to the "Settings" screen and click on the "Tags" menu. Use the (share) button to share all of "your starred items". (Upside, easy to flag items. Downside, you only get to use your "star" for this purpose. Also, all of your "starred" entries will be public. Requires additional steps.)

9c) Use the Google Reader (edit tags) button to flag relevant items by manually adding your shared label (e.g., "williams") to each. (Upside, no additional steps required. Downside, takes some typing to flag an entry.)

9d) Use a different feed reader or tool to flag and publicly re-syndicate posts. (Upsides, the other feed reader might be better than Google Reader. Downsides, it probably isn't. Requires additional steps.)

9e) Use an army of friends/interns/robots to produce more shared feeds that you want to subscribe to. (Upsides, more people picking out high-quality content. Downsides, need to recruit army of friends/interns/robots. Requires additional steps.)

Repeat step 9 to flag interesting entries whenever you see them. The more frequently the better.10) Subscribe to the feeds containing the extra items you want to include in your aggregated feed. In the case of 9a or 9b you can find the location of the shared feed by going to the "Settings" page and the "Tags" menu. Look for the "view shared page" link. Follow that link and subscribe to the feed found there. (If you have problems subscribing to the feed in Google Reader, you may need to view the source of the public page, search for the rel="alternate" link URL, and use cut-and-paste into the "Add Subscription" box by hand. Don't worry, you only need to do this when setting up the feed the first time.)

11) Label all of the feeds containing hand-picked items with the same tag you used before. In this example, "williams".

Now we're ready to create and share an aggregated feed.12) Go to the Google Reader "Settings" screen and select the "Tags" menu. You should see a list containing "Your Starred Items", "Your Shared Items", and the new label you created for the aggregated feeds. Click on the icon that looks like and toggle it to look like . Click on "view public page" to find the aggregated feed page for that label.

The public aggregated feed URL should look something like:

http://www.google.com/reader/shared/user/00000000000000000000/label/tag


You can also use the corresponding Atom feed, which will have a URL like:

http://www.google.com/reader/public/atom/user/00000000000000000000/label/tag
13) For bonus points, cleaner URLs, and feed tracking abilities, use FeedBurner to "wrap" that feed before giving it out. Thus your final feed could be something like:

http://feeds.feedburner.com/username/label


And there you have it.

Hope this helps!