Fixed link

Collective Evaluation Project An application of the "new information theory"

Yi-Cheng Zhang

posted on 14 June 2005

As if it were yesterday, Econophysics Forum web site actually is seven years old. Around the summer of 1998 a handful of my colleagues who later to be labelled as ’econophysicists’ were wondering how to tap the then new WWW phenomenon to foster the fledging community. The dream was to have a kind of ‘virtual institute’, enabling scientists working on the margins of the traditional disciplines and institutions to mingle and work together. Seven years is a long period, especially in the Information Age. The original vision didn’t bear out completely, but it was an interesting experiment leveraging then primitive web technologies and to have gathered a consistent crowd among the physicists and other scholars in the interdisciplinary domain. Typing ’Econophysics’ into any major search engine, the top item is probably this site.

In the meantime the web has made light years of strides in both concepts and technical capabilities. My own group has shifted our research focus from financial market modelling gradually to the so-called ‘new information theory’. Since a couple years we have been trying to hammer out a theoretical framework of the web-based information filtering mechanism. Though much sophisticated theoretical work still lies ahead, we come now to a point that we feel confident enough to apply our theories to real web service. The first steps are already taken: we just bought a brand new server and we are going to completely overhaul the current web site.
The vision is to apply our theories eventually on the broad WWW, beyond sciences, but you will see we’ll first take our own community as ’guinea pigs’ to test water.

What’s big deal? To cut the story short currently we face a unprecedented information explosion. Human agents in work or in daily life can’t reliably determine what’s relevant and what of better quality. Take our beloved preprint archives Cond-Mat for instance, I remember ten years ago we routinely read all the titles and abstracts. Paul Ginsparg, the founder of the Los Alamos archives (now moved with him to Cornell since 2001), has estimated that the total papers on beyond 200’000. This is just a tiny speckle in the WWW, which is thought to have over 10 billion pages. Human attention span, like our researcher’s will remain the same in the face of exponential growth of accessible information. Powerful search engines like Google and Yahoo are invaluable tools to find the proverbial needle in the haystack. However, suppose we find the relevant paper, we’d not be sure of its quality and even a referee can’t always verify the results and validate the claims of a paper. The traditional journals can act as a filter to select good from mediocre and shaky papers. But this filtering mechanism is outdated and expensive, our university just cancelled a whole bundle of subscriptions from Elsevier, we just can’t afford the 30% hike in annual online access fees. University of Fribourg isn’t one of the poorest institutions around. There is something fundamentally amiss with scientific publishing. Yes, a reputable journal lends credibility to a published paper, but we know the filtering mechanisms are far from perfect. Typically a paper is sent to a couple of referees, who may have many other things to mend than study it carefully, and moreover, we all have biases and agendas. The fate for the paper is decided whimsically and sometimes randomly.

Here comes what the Internet can make big change. Recently we have studied models in which there are objects and raters (think of papers and referees/readers). The objects have intrinsic qualities only God knows for sure; the raters are limited in judging capabilities and they have occasional wilful biases and agendas. The evaluation by these less than perfect raters can appear very noisy and contradictory, for the scholars we need to co-determine both raters’ judging capabilities as well as the objects’ quality. Out of the noisy source it’s only wishful thinking to decode the God-given values, however we aim not at exact values but the best approximation possible. It turns out that a rater’s rating history can betray his rating capability, based on what others have rated on the same set of objects. The idea is to reduce to the minimum the inconsistency in the data. What more heartening, the larger is the system, the performance is better.

Here is how the theoretical results can be applied. Suppose we launch a new web service, the users (members as we call them) can comment (if you have both insight and time) on a given paper and vote on it (if you’re in a hurry). But you want to be careful with your voting and comments, as your credibility is at sake and our computers will be constantly recalculating your weight in the community. The final evaluation is not a simple average of how many people voted, but a sophisticated algorithm akin to Google’s PageRank attributes different weights for the voting matrix (ith reader reviewing jth paper e.g.). In our simulations the system is surprisingly reliable and resistant to wilful abuses and noisy voters.

Who will be the person initially proposing the paper to be evaluated in the first place? Up until now for our Econophysics Forum it was Paolo, Giancarlo, and Lionel who spend half a day or so to manually select what you get in the « weekly selection ». However for the new service the self-organized model is much more appealing: in principle everybody can propose a paper from any subject, communities for any topic will slowly grow. There is incentive to be the initiator: if she proposed a paper which later judged favourably by her fellow researchers, than her credibility grows. This rule acts also as an antidote against spamming by dubious papers, the initiators will have their credibility reduced. A paper in traditional journals is for posterity , eternally so. But the web based service enables interactions even years after a paper’s publication. An insightful reader may spot hard-to-find mistakes or may suggest a new way of looking at the same thing; the author may want update his paper with aid of the collective wisdom. There many creatures on the web are already doing all these things, notably the phenomenal Wikipedia. There are many nice features we can learn from the newer animals like, citeUlike, etc. What we do is still useful, instead of tinkering with the rules we’d be aided with rigorous scientific methods. Before any real application massive simulations are necessary to minimize
costly structural errors.

We believe that a new paradigm for scientific publishing is around corner, and we are eager to put the theoretical models into practice. To make the self-organized model work we need your support in various ways. What can be done for scientific publishing can be easily generalized for issues of much larger scope, any product and service of daily life needs badly reliable evaluation. Google is more relevant than Altavista because it implicitly uses webmaster’s inadvertent voting-pointing to a given site-to achieve a total difference user experience than Altavista did. What if all users can give their feedbacks, and these feedbacks are properly interpreted, the end result can indeed be dazzling. If the collective evaluative filtering mechanism can indeed be realized as in theoretical studies, it will let the cream to pop up and garbage down, collectively we can achieve much higher quality in whatever we do.


PS: We are looking for IT talents to join the efforts of our development team, self-educated programmers are welcome also!

PS1: Your thoughts? Blog this!