RSS Last Mile
Is RSS only for geeks? Should users be required to understand what ‘XML’ or ‘RSS’ mean, in order to take advantage of subscription and aggregation? Are subscription and aggregation useful for a broad range of users, or only for ?power? users?
To me, the answers are obvious. I believe that subscription and aggregation are features that appeal to the mainstream, and the number of users who use RSS without having any clue about the underlying technologies could easily dwarf the number of ?power? users. There are certainly people who feel differently — people who think that aggregator usage is low because most users don’t want or need the functionality. But I’m pretty sure that uptake is low because of poor user experience at this point.
To be honest, the non-technical end-to-end user experience for discovering and using subscription/aggregation is pretty crappy today. The orange RSS icon has been spreading like wildfire throughout the Microsoft sites, and plenty of people are subscribing; but I have a feeling that even more people are confused and never realize that the orange RSS icon is actually offering them a valuable service. Even if you ran a usability lab asking normal I.T. pros to ?browse the MSDN subscriber home page, and then configure your system so that you get notified when new downloads are offered?, I bet the results would be horrifying — so even for people with some degree of tech savvy, who are primed by being told exactly what functionality they can enable, the user experience sucks.
So, in the spirit of making improvements informed by objective measurements, a bunch of people inside Microsoft have started analyzing the end-to-end user experience for non-technical users encountering subscription and aggregation for the first time. There has been lots of analysis, both inside and outside the ivory tower (for example, Roy Osherove’s commentary). And this analysis has suggested all sorts of good ideas that are worth trying out. The trick at this point is to prototype and test the various ideas under some well-controlled situations to figure out which works best. I think that Julien is going in a very promising direction, and it will be interesting to see what kind of lessons he learns in experimenting with this.
The other technique I find really interesting right now is the ?informative RSS skin? that Ciam has recently deployed. The idea is to put a skin on the feed which encouraged the user to get an aggregator and gives some simple instructions for subscribing. We’re pretty sure that this is a good idea, and expect usability labs to show this. However,we had some debates internallyabout whether the skin should try to render the feed in a useful manner, or whether it should make it very clear that the feed was not intended for human consumption. For example, you’ll note that Ben commented on Ciam’s blog saying that the ?pretty? format was actually more confusing for him, a seasoned RSS user who was expecting an XML blob. I’m really intrigued by this observation, and feel that it could even go beyond seasoned users — for example, a complete newbie may look at the file and think, ?why would I need an extra piece of software to view this feed, I seem to be viewing it just fine right now?? Or, suppose you ran the usability lab I described above, where you ask power users to ?sign up for notifications of site changes?. Once they reach the ?pretty? RSS feed, they might just assume that they have succeded in signing up for subscriptions. Of course, this is all wild speculation, and I really believe that the current iteration of pretty skin is a big improvement over the status quo, but I want to see more data.
So how do we objectively measure the effectiveness of one technique or another? There are numerous ways we could test this. Here are some I can think of (and I really would appreciate comments or ideas about others, or whether these are measuring the right things or not):
- Invite twenty users to go through a classic usability lab, where we assign them a simple task. ?Browse to MSDN downloads page and sign up to get notified when new downloads are offered.? Give a third of them a page with traditional ?raw XML? RSS, a third of them a page with pretty RSS and informative text, and a third of them a page with the ?not for human consumption? feed. Get a sample of users that is about half ?power? users and half normal information workers. Observe carefully how they do. This is how we normally do usability labs, but it can be expensive to do it correctly, filter for the right users, and so on.
- Configure two versions of the MSDN Download page — one with the pretty RSS, and one with the ?not for human consumption? RSS. Have each page point to a list of aggregators via a redirect URL that you could monitor for clickthroughs. Randomly serve one page to half of your users, the other to the rest. Measure how many clickthroughs you get over a one week period. This would not measure effectiveness versus ?raw XML?, and would not measure how successful the users were at obtaining an aggregator. It would also tell you nothing about the confusion factor for experienced users of RSS (who presumably would never click on the list of aggregators anyway). But it would tell you which page was more effective at encouraging newbies to seek out additional education about aggregators.
- Run a survey. Surveys work well for some things, and are potentially a lot less expensive, but I’m not sure there would be much to offer here.