Looking for a specific image

• Looking if the exact same image is in the database
  Goal: How fast can a system find this out?
  Measures:
  • Response time for a correct answer
  • Accuracy of the reply
  • Positions of the relevant images
  
  
• Looking if the query image is part of an image in the database
  Goal: How quickly does a system find part of an image and with which accuracy where accuracy might be more important than time
  Measures: same as above
  
  
• Looking if a geometrically altered image is part of an image in the database
  Goal: How quickly does a system find part of an image and with which accuracy where accuracy might be more important than time
  Measures: same as above
  
  
• Looking if a compressed version of an image is in the database (ie. strong JPEG compression)
  Goal: How quickly does a system find a compressed image and with which accuracy
  Measures: Response time for a correct answer, Accuracy of the reply, positions of the relevant images

Looking for a number of similar images

• Query by example with known groundtruth
  Goal: Find images that are relevant for a certain query image
  Measures:
  • normalized average rank (see BIRDS-I) as a leading indicator
  • Precision/recall graph
  • Precision and recall at certain important cutoff points
  • Rank of the first relevant image other than the query image
  • Average rank
  • Primary recall
  
  
• Evaluation of positive and/or negative feedback
  Goal: How well can the results be improved with feedback, how many steps of feedback
  Measures: Possibly the measure of secondary recall etc, proposed by C. Leung can the same measures be used as for the first query step to have a comparison between the two?
  
  
• How well can a system adapt its output for the same starting image but with different ground truth sets
  Goal: How well can the system adapt the output to the need of different users?
  Measures: the same measures as before but with different relevance sets
  • Can we get different relevance sets from the ground truth?
  • Can we use the same measures as stated before and average them over the different relevance sets?
  
  

Looking for a sketch of an image uUsing incomplete information)

• How well can a sketch of an image be found?
  Goal: speed and accuracy of the reply
  Measures: time and accuracy

Target search

• How quickly can an image be found while browsing
  Goal: Find an image as quickly as possible
  Measures: Number of images that have to be viewed before the correct one is found

• Inserting an image into the database
  Goal: time it takes to insert an image
  Measures: time
  
  
• Inserting an image into the database and find a known image similar to this one
  Measures: time it takes to insert an image and how accurate the response is
  time and accuracy

• Tests where two systems are explicitly compared
  see the article of A. Dimai at Visual 99
  
  
• Tests for special application areas such as trademarks or medical imaging are the measures really different or can the same measure be used as before just applied to a different set of groundtruth and images
  
  
• Measure the scalability of a CBIR system with respect to a large collection size (10,000;100,000;1,000,000 images)
  Goal: Measure the time it takes with collections of different sizes to be able to interpolate the response time for even larger collection sizes
  Measures: time change with respect to the collection size
  
  
• Evaluation of CBIR interfaces
  Goal: Find the most efficient user interface for a certain task