“…the only time you should let search engines index the output of an aggregator is when that’s the only place the feeds appear as HTML. Otherwise, it’s duplicate content, and that’s going to wind up hurting someone, whether it’s you or the source.”
I think a debate is due about the Irish Blog aggregator‘s policy of indexing our feeds. I know a tiny amount about aggregating blogs, but of course not as much as Roger Galligan, for whom I voted in the Irish Blog Awards for Best Contribution to the Irish Bloggersphere, which he deservedly won. But I wonder if the site is over-reaching itself.
It is my belief that the responsible setting for search engine robots in feed aggregators is “noindex, follow”. Gregarius, the open source aggregator, decided to set this as the default option, persuaded by the arguments in the essay quoted above.
I like the Irish Blogs aggregator, I really do. But I’ve noticed recently that posts I have made on various subjects turn up in Google search, but the accompanying link is to the irishblogs.ie archive. They are my words, but people clicking on them are taken to the aggregator first, because search engines don’t like to duplicate posts, and irishblogs.ie has a higher pagerank score than my humble blog – no doubt helped by all of us willing to sign up and add our feed and make it a popular site. This makes increasing my blog’s pageranking more difficult, because my posts do not show up as much in Google. Irishblogs.ie, however, is very likely to become increasingly more authoritative, at my expense, in terms of pagerank – and that means, as well as losing visitors to my blog, adsense income. (Which is peanuts for me at the moment, like most of us I imagine – but I can’t see it improving, now.)
Since Irish Blogs has begun to move from a voluntary effort with a lot of goodwill, to a commercial enterprise with sponsors, I think that some accountability is called for, some clear terms of service that I can take or leave. Planet Journals has a robots setting of “noindex”. Why can’t Irish Blogs?
An aggregator should only, in my opinion, syndicate the feeds of its contributors so that they can all be read in one place: a browser or newsreader. It should not, I believe, muscle in on the indexing of its contributory blog posts in search engines. I did not give irishblogs.ie my explicit permission for that when I signed up, and I do not believe it is reasonable or standard practice.
Roger, over to you…