Experimental components for the evaluation of interactive information retrieval systems

Journal of Documentation - Tập 56 Số 1 - Trang 71-90 - 2000
Pia Borlund1
1(Department of Information Studies, Royal School of Library and Information Science, Birketinget 6, DK 2300 Copenhagen S, Denmark)

Tóm tắt

This paper presents a set of basic components which constitutes the experimental setting intended for the evaluation of interactive information retrieval (IIR) systems, the aim of which is to facilitate evaluation of IIR systems in a way which is as close as possible to realistic IR processes. The experimental setting consists of three components: (1) the involvement of potential users as test persons; (2) the application of dynamic and individual information needs; and (3) the use of multidimensional and dynamic relevance judgements. Hidden under the information need component is the essential central sub‐component, the simulated work task situation, the tool that triggers the (simulated) dynamic information needs. This paper also reports on the empirical findings of the metaevaluation of the application of this sub‐component, the purpose of which is to discover whether the application of simulated work task situations to future evaluation of IIR systems can be recommended. Investigations are carried out to determine whether any search behavioural differences exist between test persons‘ treatment of their own real information needs versus simulated information needs. The hypothesis is that if no difference exists one can correctly substitute real information needs with simulated information needs through the application of simulated work task situations. The empirical results of the meta‐evaluation provide positive evidence for the application of simulated work task situations to the evaluation of IIR systems. The results also indicate that tailoring work task situations to the group of test persons is important in motivating them. Furthermore, the results of the evaluation show that different versions of semantic openness of the simulated situations make no difference to the test persons’ search treatment.

Từ khóa


Tài liệu tham khảo

10.1016/0306-4573(92)90004-J

10.1016/0020-0271(73)90096-X

10.1016/0306-4573(90)90050-C

10.1108/eb026436

10.1002/asi.4630260604

10.1002/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J

Schamber L, 1994, Annual Review of Information Science and Technology, 29, 3

10.1002/(SICI)1097-4571(199404)45:3<142::AID-ASI4>3.0.CO;2-6

10.1016/S0306-4573(98)00025-9

10.1108/eb026722

10.1145/160688.160760

10.1108/eb026843

Kuhlthau C.C., 1993, Seeking meaning: a process approach to library and information science

10.1108/EUM0000000007198

Cleverdon C.W., 1966, Design., 1

Lancaster W.F., 1997, Readings in information retrieval.

10.1016/0306-4573(92)90002-H

10.1145/160688.160692

10.1016/0306-4573(94)00047-7

10.6028/NIST.SP.500-225

Harman, D.K. Overview of the Fourth Text REtrieval Conference (TREC4). In: Harman, D.K., ed. Proceedings of TREC-4. Gaithersburg, MD: NIST, 1996, 1-23. (NIST Special Publication 500-236.)

10.6028/NIST.SP.500-238

Harman, D.K. The TREC conferences. In: Sparck Jones, K. and Willett, P., eds. Readings in information retrieval. San Francisco: Morgan Kaufmann, 1997, 247-256.

10.1145/290941.291019

10.1016/0306-4573(94)00051-4

10.1002/(SICI)1097-4571(199601)47:1<85::AID-ASI8>3.0.CO;2-Z

Mackay D.M., 1960, What makes the question? The Listener, 62

10.5860/crl_29_03_178

10.1108/eb026960

Wersig G., 1971, München-Pullach: Verlag Dokumentation Saur KG

Dervin B., 1992, Qualitative research in information management.

10.1016/0143-6236(84)90070-X

10.1108/eb026727

Saracevic T., 1990, ASIS Proceedings, 42, 47

10.1002/(SICI)1097-4571(199404)45:3<207::AID-ASI10>3.0.CO;2-1

Ingwersen P., 1996, Information science: integration in perspective.

Campbell I., 1990, RdSystem technical notes. Department of Computing Science

Borlund P., 1999, Mira '99: Evaluating interactive information retrieval

Morita M., 1994, Proceedings of the 17th ACM Sigir Conference on Research and Development of Information Retrieval, 1994, 272

Thông tin
Thông tin xuất bản

Journal of Documentation

Tập 56 Số 1

71-90

Thông tin tác giả