Comparing the performance of CBIR (Content-Based Image Retrieval) algorithms is difficult. Private data sets are used, so it is controversial to compare CBIR algorithms developed by different researchers. Also, the performance of CBIR algorithms is usually measured on an isolated, well-tuned PC or workstation.

In a real-world environment, however, the CBIR algorithms would only constitute a minor component among the many interacting components needed to facilitate a useful CBIR application e.g., Web-based applications on the Internet. The Internet, being a shared medium, dramatically changes many of the usual assumptions about measuring CBIR performance. Any CBIR benchmark should be designed from a networked systems standpoint. Networked system benchmarks have been developed for other applications e.g., text retrieval, and relational database management. These benchmarks typically introduce communication overhead because the real systems they model are distributed applications e.g., an airline reservation system. The most common type of distributed computing architecture uses a client/server model.

To participate in the Benchathlon and assess the performance of your algorithm,

• you have to break up your CBIR engine in a client and a server part
• you have to implement a program interface between the client and the server; to make your system universal, you need to use a standard protocol between the server and the client
• the performance analysis harness is plugged in between your server and your client and uses the same protocol
• at this time the recommended protocol is MRML

Nota bene:

• the CBIR server can run your server or you can install it on benchathlon.net (HP-UX platform) after requesting an account from Stéphane or Giordano
• the image database in on benchathlon.net and new images will be added shortly before the event
• the client will run on a laptop computer (Windows 98 platform) provided at EI; if possible it should be a Java applet that can run in a browser such as Internet Explorer or Netscape Communicator
• the metadata will not be accessible to the participants; it is used by the performance analysts to generate the ground truth

At this time we are not yet ready staging the image collection and ground truth. However, to allow interested participants to prepare papers to be submitted at the Internet Imaging III conference at EI 2002 we are making available our development code. Please try it out and give us feedback on how we can improve it to make your life easier; the contact person is Wolfgang Muller.

• Retrieve the example MRML server from ftp://benchathlon.sourceforge.net/pub/benchathlon/
  • This Perl code does not do any retrieval -- it is intended to be a useful base for connecting your CBIR code to MRML
  • There are no CBIR algorithms in the server. The user of the example server is supposed to add them into a place marked within the source code
  • There are man pages for the classes used by the server, but there is no man page yet for the server itself which refers you to the class man pages
• The server as is, is a dummy. It gives a fixed selection of images if you click the "random" button, and it gives back the query images when it receives query images. So there is no CBIRS in there
• To all other MRML signals (like open-session, for example) the example server gives the answers expected by a client, in the most simple MRML-compliant way (i.e. there are constants sent, there is no real session management etc.)
• To make the server a real fully functional server, you add a Perl function which talks to your CBIRS system. This can be a simple system call using Perl, this can be opening a socket for talking with an existing server, this can be using CORBA etc.
• All the Server wants from you is some Perl function that takes an XML parse tree (as documented in the man page if I'm not mistaken) and gives back one such parse tree
• What you need to add is simpler than a CGI, as you can choose the input/output format of your CBIRS to your taste, and testing is easier.
  • However, if you have an existing CGI you want to interface with the MRML example server, this is easy, too
  • There are Perl modules for issuing HTTP requests, which could thus talk to a CGI
• There is no installation to be done, if the needed Perl modules (XML::Parser, XML::Writer) are installed on the system. I think the demo.phtml points on the starter page to the viper tutorial. The interface to the example server works exactly the same way
• Try the functionality of the server at http://pc7143.unige.ch/demo.phtml
• Subscribe to the distribution list at Topica; this will allow you to track our development progress and learn when the Benchathlon server is ready for use