Transaction monitoring and online questionnaires will provide objective and quantitative data on the use of the Cheshire system, as well as statistics on users' opinions about interface and retrieval features. User queries and relevance judgements collected in the transaction logs will provide the basis for evaluating the retrieval effectiveness of the system using recall and precision measures. Estimation of these figures will use methods developed in evaluation of the prototype Cheshire system. While quantitative measures like recall and precision are useful, it is also important to find out directly how satisfied the users are with the results of their searches. The questionnaire will provide a means for gathering this kind of data as well. A system based on probabilistic retrieval like Cheshire has not really been made available before for real use in a public setting. How the system is used will be evaluated in a number of ways.
The online questionnaire and transaction logs will allow for analysis of the different types of searches conducted on the system. One goal is to determine if trends in searching shift from type to type (e.g. title to subject or topic) with the use of a probabilistic catalog. It will also be of particular interest to compare search results and user satisfaction based on search type.
The users of the system will come from a variety of different social groups. Another important area of evaluation is how these new tools affect the information-seeking behavior of these groups. In addition, it will be interesting to see if there are disparities in satisfaction.
This system will be available to users at workstations with a graphical interface as well as to network users via the Z39.50 protocol. System performance and user satisfaction will be compared to determine the influence of the interface on searching. The effect of the interface on user performance (e.g. typographical errors) will also be studied.
Finally, the graphical interface itself will be evaluated in terms of ease of learning, effectiveness, and pleasantness of use.
[NOTE: The preliminary data collection and analysis will be complete by the time of the conference and results will be given during the presentation of the paper.]