Suppose that you are browsing a web page that uses the word "reify", and you want to know what that means. You also happen to know that you can look up any word by using a URL like http://www.dictionary.com/cgi-bin/dict.pl?term=someword (Replace someword with the word you want to look up). If the author of the page had added a hyperlink like this to the word "reify", you would be able to click on it to look it up. But why depend on the author? Why doesn't your web browser let you highlight a word, and then allow you to select a command to define the word, which just replaces someword in the URL above with the text you have highlighted? Then you would automatically be taken to the definition of the word "reify". After looking at the definition, you might still be confused by what that word really means, but it would be a neat feature, right? Suppose that the web browser could go a step further, and recognize that the phrase Grateful Dead refers to a music group. It could then link you to RollingStone.com, even if the original author of the page hadn't bothered. In fact, such tools are available today. Though it is not the first or the only example, Comet Systems Smart Cursors is one such rudimentary tool.

Now consider the case of browsing a page like www.news.com. What if the browser were able to detect that you were at a news.com site, and provide you with a menu option "About the owner of this site" that could pull up company information about CNET (the owner of news.com). This is similar to the above, but in this case the browser is being smart about the whole site or domain instead of words within a page.

Next, pretend that your browser has two buttons in the toolbar, a "thumbs up" and a "thumbs down". Now any time that you are browsing the web, if you find a page to be particularly terrible, you hit the "thumbs down" button. On the other hand, if you find a page to be so inspiring that you can't restrain yourself, just hit the "thumbs up" button. Each time you hit the button, your disposition is recorded somewhere. Now when you browse to any page, your browser can also give you helpful hints in the status bar, like "80% of your friends gave this page a thumbs-up", or "90% of your friends say that this page stinks". In fact, search engines like Google could use this information to rank search results. Note that you could also use rankings (e.g. one to five) and calculate average ratings, etc.

Imagine you are browsing the web and you see a page that gives an excellent description of dolphins. You want to remember it for possible later use, so you go to the "My Encyclopedia" menu of your web browser and select "add page". You're presented with a dialog box asking you to select categories for this encyclopedia article, and you enter "Dolphins;Sea Mammals". Your metadata is stored on a central server somewhere. Now any time that you or anyone else goes to the "My Encyclopedia->Browse Articles" menu and browses to the "Sea Mammals" or "Dolphins" categories, this article will appear. This technique could be used for directories or any other categorization scheme as well.

Ok, so what if it's not enough to share ratings with people? Imagine that you are browsing the home page of a fellow Asheron's Call gamer, where he claims to be the best player on the 'net. You know that is totally bogus, so just use your browser to add a comment to the page (actually, you probably don't have permissions to edit that guy's page, so your annotation would be sent to an annotation server somewhere). Now when your friends browse to the same page, they'll be able to see your rebuttal as well (because their web browsers will automatically contact the annotation server to see if there are any comments), explaining why you are a much better player.

Another sort of annotations are related links. Imagine if your web browser allowed you to recommend related web pages for every page that you view. Any time that you view a web page, your browser could show you the top five related pages recommended by other people. Obviously there are many variations on this concept.

Finally, one potentially controversial tool would be something that allowed you to change the content of other people's web pages. Your changes would probably not be applied directly to the page, but saved in a separate and independent annotation server. Then, when anyone browsed to that page, the browser would download the changes and apply them on-screen. Clearly, you would want to limit who was allowed to modify pages that you viewed and perhaps to be able to distinguish modifications from the original text.

Consider annotations, for example. Even if two different annotation tools keep track of slightly different metadata (perhaps one records the date of the annotation and another does not), it could be wise for the metadata formats to be as interoperable as possible, because then each tool would have the future option of enticing users away from the other tool. Allowing metadata to be shared across tools encourages competition among tools and allows users to choose the tool (annotation viewer, for example) which has the features they want. It also provides a stable platform to create tools that combine features of other tools.

Suppose that you have one tool which keeps track of company names and and mailing addresses, while you have a completely different tool which keeps track of all of your friends' email addresses and company names. A tool that was smart enough could look at a friend's company name and match that with the information about company mailing addresses and infer a mailing address for your friend. Widespread inferencing is something that is probably not an immediate opportunity for tool creators, but as long as the metadata created by tools is interoperable, it should be easier for future tools to use that data to deduce new metadata that wasn't ever recorded. In the example above, nowhere was it ever recorded what your friend's mailing address at work was -- this information was attainable because the two different sets of metadata were interoperable and because the tool reading the data knew how to infer that information.

Think of the bazaars that were once common in cosmopolitan cities. Why is it better to go to the bazaar to barter your goods than to stand on your street corner trying to barter? The answer is common sense; it is better to go to the bazaar because that is where everyone else goes. One would imagine that humans have instinctively understood this for thousands of years, but today we attach some pseudoscientific mathematical formula and call it "The Network Effect". How tool vendors can apply this common sense is to realize that their tools will be even more compelling to users if all of the users of different tools have all of their metadata collected into a shared space. Just like people from different cities and different cultures meet at the bazaar to barter, different tools can choose to save metadata into a shared city of interoperability. If users of tools such as third voice can view annotations made by users of other tools, and users of other tools can view annotations made by third voice, users will see much more value in using annotation tools. Note that this so-called network effect is applicable to certain situations, particularly in the collaborative areas to which I limit my discussions of "the semantic web". For example, it would be blindly stupid for Google to export all of their metadata in a format that could be used by MSN search for free. Not only would Google not benefit from such a move, users would not benefit either. Although it makes sense to barter goods in a place where everyone else is bartering, there are no "network effect" benefits likely to be gained if everyone were to collect in one spot to read a book or watch television, for example.

In fact, metadata is intended to be indexed, searched, and processed by tools, while web pages are designed for humans to view one page at a time. The best way to index something to allow fast searching and processing is to centralize it. If you consider Napster, you can see that the metadata is centralized for fast querying, and the data is decentralized for fast retrieval. This design rule, "decentralized data, centralized metadata", can be seen on the web today. At the same time as HTML pages are being decentralized further and further from the core through services like akamai and inktomi edge caching, searching functions on the Internet are being centralized into ever more powerful engines like Google. Note that it is possible to have actual metadata stored in a decentralized way, but the indexing and querying of that metadata will be most efficient if centralized.

The final scalability challenge threatening to cloud metadata's bright future is the probability that the amount of metadata being generated will be far greater than the data being tagged, and the number of queries made against metadata could exceed the number of queries made to request the related web pages. You can easily imagine the combined size of annotations and comments about the www.microsoft.com home page being much larger that the size of the web page itself. Not that this would be a problem, but remember -- we are talking about centralizing this stuff at least a little. You might take hope from the fact that Google actually keeps complete local copies of every page that it indexes, so in a sense Google has already decentralized the entire web onto its storage arrays. But there are two things that suggest this is not the most instructive example. First, storing a page is not the same as storing dynamically changing metadata about that page (which may exceed the size of the page) without having that metadata go stale. Second, buying enough permanent storage to cache the entire web is the sort of expense that would guarantee that only a few centralized players would be in the market, and this overcentralization would tend to defeat the idea of "the semantic web".

Places like epinions.com, yahoo groups, and MSN communities allow people to post their opinions about all sorts of things, from products to politicians. Once you post your opinion, it usually shows up right away. Other people need to go to the site and specifically request your opinion to see it, though. If you have your own web server, you can always write your opinions into an HTML file and save it to the server as well.

Some message boards allow you optionally to have the messages sent to you as e-mail. Mailing lists allow you to e-mail your opinion to a group of people who are interested in (or at least tolerant of) your opinions. Some mailing lists also archive e-mails to the web or allow you to post to the web, but there is an important difference between the web-based message boards and e-mail lists. With an e-mail list, you do not get to see the other person's opinion the instant they send it, but you don't have to go poll a central web site to get the opinion, either. Using e-mail pushes the information closer to you and adds a little bit of latency.

When you post your opinion to a usenet newsgroup, people on the same server as you will see the message pretty quickly, but someone on the other side of the globe may have to wait much longer as the NNTP servers pass the opinion along. You get higher latency, but you don't have to have copies of all the opinions in your e-mail inbox. The other important thing about NNTP is that the system is designed to allow multiple distributed write masters to create data, and then the data is replicated around the system. A majordomo-style mailing list or an HTTP-based message board both involve central write masters. The multi-master approach of NNTP makes it a good choice for sharing opinions when there are many people participating and huge volumes of opinion traffic.

With chat, everyone sees your opinion pretty much instantly. You need to keep the number of people in a chat room small, and messages terse, though, because things would get overwhelming if you were passing along the quantity of information that goes over a large usenet newsgroup. To get extremely low latency, you give up scalability as well. Large IRC networks will add multiple write servers and synchronize between, which adds a small amount of latency.

There has been some hype in recent years about "push" technologies. To me, the difference between "push" and "pull" is this: when a system delivers a message to a system that you own, then it is called "push". When a system keeps the message on a machine that it controls, we say that we have to "pull" it down when we want it on our own machine. "Pushing" data to systems that want it (as is the case in the SMTP and IRC examples above) is pretty expensive, so most systems that push data like to delete the data once they are done pushing. For example, how easy is it to go back to an IRC server and see what messages were sent yesterday? On the other hand, web-based message boards and NNTP servers tend to keep copies of the messages around for longer periods of time; that is, they have more liberal retention policies. If we define latency as being "the amount of time between the moment that someone publishes an opinion until the time that the opinion is sitting on a machine that you own", the choice of technique obviously has an impact, although it doesn't seem dependent on push or retention. You can technically keep hitting the "reload" button on your browser for a web-based discussion board and get posts the instant they arrived. You would probably irritate the board owners, and that is certainly not the most scalable use for a web-based board, but it's possible.

Going back to the scaling problems of metadata, you'll notice that most of the above communication techniques involve some amount of centralization. It is possible to do multi-party chat in a more decentralized way than IRC (MSN IM, AIM, etc.), and it is possible to share more permanent messages and documents collaboratively in a more decentralized way than NNTP (Groove), but these techniques aren't designed to scale into large numbers of users. This is a basic law of scalability; the more individual nodes that allow updates, the more expensive it will be to merge those updates together later. Systems like NNTP decentralize in the directions of the user as much as possible, to allow faster retrieval. This is similar to the way that akamai edge caching moves the data closer to the users who demand it the most. At the same time, however, the number of NNTP servers per user remains low (like 0.0001), and updates are batched to "flow" throughout the network of servers. You could put an NNTP server on everybody's personal machine, but then everyone would have to wait a year or two before seeing someone else's post. In general, we call it syndication whenever one source of information publishes that information to other sources, and we call it aggregation when one site pulls information from multiple sources. Syndication could be seen as a decentralizing flow, while aggregation would be a centralizing flow. But typically syndication sources publish to other aggregators, and the users are eventually required to go to an aggregator and retrieve the information, rather than have it syndicated directly to their desktops. This is the case with NNTP, where one NNTP server may serve as a source for many other servers, but users generally poll the server for new articles rather than having an entire newsfeed delivered to their desktop. The distinction is sort of fuzzy, but it's mostly the same issue described with "Push, Retention, and Latency" discussed earlier in this document. Often sources that syndicate data will syndicate an entire article. On the other hand, aggregators will sometimes aggregate metadata (like headlines and descriptions) and simply point to the actual location of the article. You can think of NNTP this way -- when you post an article to the newsgroups, the entire article is copied to thousands of aggregators world-wide. If you take this analogy a step further, when you use a newsreader to connect to a news server, you usually only collect the titles of the articles and pointers to some sort of ID that you can use to grab the entire article from the server if you want. In other words, NNTP servers past mostly data amongst themselves, and newsreaders extract metadata from the news servers to start. (Now, if the distinctions aren't fuzzy enough for you yet, remember that we are talking about sharing opinions, and we are pretending that all newsgroup postings are really opinions about some web page or another. So the articles themselves are metadata, and the list of topics being downloaded by the newsreader is technically metadata about metadata. This should make you realize that the distinction between metadata and data is fairly arbitrary, and that there is nothing wrong with taking meta to multiple levels -- in fact, in this example it is a good architectural decision. If that makes sense, then you are a very smart person or I am a good writer, or both.)

Aggregation can also be used in the sense that databases aggregate data by summarizing, averaging, or other similar functions across groupings of the data. For example, an NNTP server could collect all of the opinions posted to a newsgroup that give IBM a rating between one and five, and then combine all of those messages into a single message with just the average rating before sending the information to another NNTP server.

The other important thing about all of the communication techniques listed earlier is that the user is never forced to cope with all of the data going over the network. For example, on IRC you only join the channels that you are interested in. You only go to web-based message boards that interest you, you only read newsgroups that you find interesting, and so on. This process of selectively subscribing (or joining a channel) to get at interesting opinions is basically a method of filtering. In fact, the filtering can take place between servers on the network. For example, a company running a corporate NNTP server may choose to filter out all of the alt.* newsgroups, but keep the rest.

Ok, so the title of this topic is a bit stupid. I wanted to keep continuity with the "Decentralize Data, Centralize Metadata" theme earlier, but convey the idea that what will really be happening is a series of flows involving centralization, aggregation, and subsequent decentralization. In the examples we have given above, the process of deciding how decentralized the data gets (to push it closer to users for fast retrieval) and deciding how to syndicate and aggregate the data is quite manual. That is, humans at various points decide to establish an NNTP or IRC server, decide which other servers to connect with, establish filtering rules, retention policies and so on. The current trend toward edge-caching of content through techniques like inktomi or akamai is actually very similar to the aggregation/filtering cycle, though. In a sense, the edge caches act as aggregators of web content, and they filter the content they aggregate based on popularity (which is a bit of semantic metadata that they have to determine dynamically). Edge caches available today are not nearly as sophisticated in their range of options as are the manual syndication and aggregation systems, but they are able to operate largely without human configuration. As edge-cache intelligence continues to increase, and as aggregation and syndication systems add more intelligence, it is possible to imagine these two techniques converging.

You could filter metadata based on the e-mail address of the submitter. You could possibly use membership in a group to determine trust (for example, you could trust members of the yahoo group "p2p-discuss").

Digital signatures would be necessary in cases where you cannot tolerate the risk of your metadata being poisoned by spoofed entries. Note that there is nothing saying that digital signatures are an all-or-nothing idea. You could perhaps do something like, give all people with verified digital certificates a trust level of 6, all people with hotmail accounts a trust level of 3, and all others a trust level of 0.

Suppose that you browse to a page that clearly sucks, and your browser tells you that 12% of people visiting it thought the page was excellent. You could perhaps tell your browser to assign a trust rating of zero to all of the people who thought it was a good page. You could go even further and post your ratings of these people as metadata, and then all of your friends would know to not trust these bozos, too. Some examples of this technique would be the user ratings on ebay, the "web of trust" on epinions, or the trust hierarchy on advogato.

If your main goal is to filter out all metadata except that which is most likely to match your interests and opinions, there are data mining techniques that can cluster data based on patterns of similarity. This is the same technique used by amazon to let you know that "other people who liked this book also liked these other books." The metadata layer could automatically detect clusters of similarity and group users based on those patterns.