Evaluation of digital libraries

Fuhr, N., Hansen, P., Mabe, M., Micsik, A., Solvberg, I.: Digital Libraries: A Generic Classification and Evaluation Scheme. In: Lecture Notes In Computer Science, vol. 2163, pp. 187–199. Springer, Berlin (2001) Research and Advanced Technology for Digital Libraries: 5th European Conference, ECDL 2001, Darmstadt, Germany, September 4–9, 2001. Proceedings

DELOS: DELOS Workshop on Evaluation of Digital Libraries: Testbeds, Measurements, and Metrics. Technical report (2002) http://www.sztaki.hu/conferences/deval

Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, http://dlib.ionio.gr/wp7/workshop2004_program.html (2004).

Cleverdon, C.: The Cranfield tests on index language devices. In: Sparck-Jones, K., Willett, P. (eds.) Readings in Information Retrieval, pp. 47–59 Morgan Kaufmann, San Francisco (1997)

Turpin, A., Hersh, W.: Why batch and user evaluations do not give the same results. In: Proceedings of the SIGIR01, pp. 225–231. ACM, New York (2001)

CCSDS: A reference model for an open archival information system. Document Number: ISO 14721:2003 (2003) http://ssdoo.gsfc.nasa.gov/nost/wwwclassic/documents/pdf/ CCSDS-650.0-B- 1.pdf

IFLANET: Functional requirements for bibliographic records (1998) http://www.ifla.org/VII/s13/frbr/frbr.pdf

Levy D. and Marshall C. (1995). Going digital: a look at assumptions underlying digital libraries.. Commun. ACM 38: 77–84

Borgman, C.: What are digital libraries? Competing visions. Inf. Process. Manage. 35, 277–243 (1999)

DELOS: Reference models for digital libraries: actors and roles - final report. Technical report (2003) http://www.dli2. nsf.gov/internationalprojects/working_group_reports/ actors_final_report.html

Agosti, M., Candela, L., Castelli, D., Ferro, N., Ioannidis Y., Koutrika, G., Meghini, C., Pagano, P., Ross, S., Schek, H.J., Schuldt, H.: A reference model for DLMSs interim report. In: Candela, L., Castelli, D. (eds.) Deliverable D1.4.2 – Reference Model for Digital Library Management Systems [Draft 1], DELOS, A Network of Excellence on Digital Libraries – IST-2002-2.3.1.12, Technology-enhanced Learning and Access to Cultural Heritage – http://146.48.87.122:8003/OLP/Repository/1.0/Disseminate/ delos/2006_WP1_D142/content/pdf?version=1 [last visited 2006, October 2] (2006)

Goncalves M.A., Fox E., Kipp N. and Watson L. (2004). Streams, structures, spaces, scenarios, societies (5S): a formal model for digital libraries. ACM Trans. Inf. Syst. 22: 270–312

Baker, S., Lancaster, F.: The measurement and evaluation of library services. Information Resources Press, Arlington (1991)

Nicholson S. (2004). A conceptual framework for the holistic measurement and cumulative evaluation of library services. J. Doc. 60: 164–182

Kyrillidou, M., Lorie, M., Hobbs, B., Choudhury, G., Webster, D., Flores, N., Heath, F.: Emerging tools for evaluating DL services: conceptual adaptations of LIBQUAL+ and CAPM. J. Digit. Inf. 4, (2003) http://jodi.ecs.soton.ac.uk/ Articles/v04/i02/Heath

Kyrillidou, M., Giersch, S.: Developing the digiqual protocol for digital library evaluation. In: JCDL ’05: Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries, New York, NY, USA, pp. 172–173. ACM, New York (2005)

Tsakonas, G., Kapidakis, S., Papatheodorou, C.: Evaluation of user interaction in digital libraries. In: Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004) http://dlib. ionio.gr/wp7/workshop2004_program.html

Tsakonas, G., Papatheodorou, C.: Analyzing and evaluating usefulness and usability in electronic information services. J. Inf. Sci. 32(5), 400–419 (2006)

Marchionini, G.: Evaluating digital libraries: a longitudinal and multifaceted view. Library Trends 49, (2000)

DELOS: A Network of Excellence on Digital Libraries (2004) Technical Annex 1, pag. 6 - DELOS Network of Excellence on Digital Libraries, as part of the Information Society Technologies (IST) Program of the European Commission (Contract G038-507618)

ISO: ISO 9241-11. Ergonomic Requirements for Office Work With Visual Display Terminals (VDT). Part 11: Guidance in Usability. International Standards Organization, London (1997)

DeGroot, S.P., Knapp, A.E.: Applying the user-centered design (ucd) process to the development of a large bibliographic navigation tool: a partnership between librarian, researcher and developer. Scopus White Paper Series (2004) http://www.elsevier.com/librarians

Blandford A., Buchanan G. and Jones M. (2004). Usability of digital libraries. Int. J. Digit. Libr. 4: 69–70

Dickstein, R., Mills, V.: Usability testing at the University of Arizona Library: how to let the users in on the design. Inf. Technol. Libr. 19, (2000)

Hilbert D.M. and Redmiles D.F. (2000). Extracting usability information from user interface events. ACM Comput. Surv. 32: 384–421

Ivory M. and Hearst M. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33: 470–516

Ke H.R., Kwakkelaar R., Tai Y.M. and Chen L.C. (2002). Exploring behavior of E-journal users in science and technology: transaction log analysis of Elsevier’s ScienceDirect OnSite in Taiwan. Libr. Inf. Sci. Res. 24: 265–291

Sfakakis, M., Kapidakis, S.: User behavior tendencies on data collections in a digital library. In: Lecture Notes in Computer Science, vol. 2458, pp. 550—559. Springer, Berlin (2002) Research and Advanced Technology for Digital Libraries: 6th European Conference, ECDL 2002, Rome, Italy, September 16–18, 2002. Proceedings

Jones S., Cunningham S.J., McNab R. and Boddie S. (2000). A transaction log analysis of a digital library. Int. J. Digit. Libr. 3: 152–169

Sutcliffe A.G., Ennis M. and Hu J. (2000). Evaluating the effectiveness of visual user interfaces for information retrieval. Int. J. Human-Comput. Stud. 53: 741–763

Kengeri R., Fox E.A., Reddy H.P., Harley H.D. and Seals C.D. (1999). Usability study of digital libraries: ACM, IEEE-CS NCSTRL, NDLTD. Int. J. Digit. Libr. 2: 157–169

Clark J.A. (2004). A usability study of the Belgian American Research Collection: measuring the functionality of a digital library. OCLC Syst. Serv. Int. Digit. Libr. Perspect. 20: 115–127

Rousseau G.K., Rogers W., Mead S.E., Sit R.A. and Rousseau J.B.A. (1998). Assessing the usability of on-line library systems. Behav. Inf. Technol. 17: 274–281

Thong J.Y.L., Hong W. and Tam K. (2002). Understanding user acceptance of digital libraries: what are the roles of interface characteristics, organizational context, and individual differences? Int. J. Human-Comput. Stud. 57: 215–242

Hartson R.H., Shivakumar P. and Perez-Quinones M.A. (2004). Usability inspection of digital libraries: a case study. Int. J. Digit. Libr. 4: 108–123

Glosiene, A., Manzuch, Z.: Usability of ICT-based Systems: State-of-the-art Review. Technical Report CALIMERA Deliverable 9 (2004)

Dicks, R.S.: Mis-usability: on the uses and misuses of usability testing. In: Proceedings of the 20th Annual International Conference on Computer Documentation, New York, pp. 26–30 ACM, New York (2002)

Doubleday, A., Springett, M.V., Sutcliffe, A.G., Ryan, M.: A comparison of usability techniques for evaluating design. In: Proceedings of the Conference on “Designing Interactive Systems: Processes, Practices, Methods and Techniques”, New York, pp. 101–110 ACM, New York (1997)

Hill L.L., Carver L., Smith T.R., Frew J., Larsgaard M., Dolin R. and Rae M.A. (2000). Alexandria digital library: user evaluation studies and system design. J. Am. Soc. Inf. Sci. 51: 246–259

Bishop A.P., Merkel C., Neumann L.J., Star S.L., Sandusky R.J. and Ignacio E. (2002). Digital libraries: situating use in changing information infrastructure. J. Am. Soc. Inf. Sci. 51: 394–413

Fox E.A., Heath L.S., Rao D., Brueni D.J., Nowell L.T., Wake W.C. and Hix D. (1993). Users, user interfaces, and objects: envision, a digital library. J. Am. Soc. Inf. Sci. 44: 480–491

Buttenfield, B.: Digital libraries: philosophies, technical design considerations, and example scenarios. In: Usability Evaluation of Digital Libraries, pp. 39–59. Haworth Press, New York (1999)

Blandford, A., Stelmaszewska, H., Bryann-Kinns, N.: Use of multiple digital libraries: a case study. In: Proceedings of the first ACM/IEEE-CS joint conference on digital libraries, pp. 179—188. ACM Press (2001)

Castillo, J., Hartson, H.R., Hix, D.: Remote usability evaluation: can users report their own critical incidents. In: CHI 1998 Conference Summary on Human Factors in Computing Systems, New York, pp. 253–254 ACM, New York (1998)

Bates M. (1990). Where should the person stop and the information search interface start? Inf. Process. Manage. 26: 575–591

Cousins, S.B.: A task-oriented interface to a DL. In: Tauber, M.J. (ed.) Conference Companion on Human Factors in Computing Systems: Common Ground, New York, pp. 103–104. ACM, New York (1996)

Pejtersen, A., Fidel, R.: A framework for work centered evaluation and design: a case study of IR on the web. Technical report (1998)

Rasmussen, J., Pejtersen, A., Goodstein, L.: Cognitive Systems Engineering. Whiley, New York (1994)

Borgman, C.: From Gutenberg to the global information infrastructure: Access to information in the networked World. MIT, Cambridge (2000)

Tenopir, C.: Use and users of electronic library resources: an overview and analysis of recent research studies. Council on Library and Information Resources, Washington (2003)

Banwell L., Ray K., Coulson G., Urquhart C., Lonsdale R., Armstrong C., Thomas R., Spink S., Yeoman A., Fenton R. and Rowley J. (2004). The JISC user behaviour monitoring and evaluation framework. J. Document. 60: 302–320

Payette S.D. and Rieger O.Y. (1998). Supporting scholarly inquiry: incorporating users in the design of the digital library. J. Acad. Libr. 24: 121–129

Yu L. and Apps A. (2000). Studying E-journal user behavior using log files: the experience of superjournal. Libr. Inf. Sci. Res. 22: 311–338

Wilson T.D. (1997). Information behaviour: an interdisciplinary perspective. Inf. Process. Manage. 33: 551–572

Kuhlthau C.C. (1991). Inside the search process: information seeking from the user’s perspective. J. Am. Soc. Inf. Sci. 42: 361–371

Ellis D. and Haugan M. (1997). Modeling the information seeking patterns of engineers and research scientists in an industrial environment. J. Document. 53: 384–403

Foster A. and Ford N. (2003). Serendipity and information seeking: an empirical study. J. Document. 59: 321–340

Belkin N.J., Oddy R.N. and Brooks H.M. (1982). ASK for information retrieval: part 1 Background and theory. J. Document. 38: 61–71

Wilson, T.D.: Exploring models of information behaviour: the uncertainty project. Inf. Process. Manage. 893–849 (1999)

Wood R. (1996). The Conspectus: a collection analysis and development success. Libr. Acquis. Pract. Theory 20: 429–453

Saracevic, T.: Evaluation of digital libraries: an overview. In: Agosti, M., Fuhr, N. (eds.) Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries, Padua, Italy (2004) http://dlib.ionio.gr/wp7/workshop2004_program.html

Agosti, M., Ferro, N., Frommholz, I., Thiel, U.: Annotations in digital libraries and collaboratories: facets, models and usage. In: Lecture Notes In Computer Science, vol. 3232, pp. 244—255 Springer, Berlin (2004) Research and Advanced Technology for Digital Libraries: 8th European Conference, ECDL 2004. Proceedings

Agosti, M., Albrechtsen, H., Ferro, N., Frommholz, I., Hansen, P., Orio, N., Panizzi, E., Pejtersen, A.M., Thiel, U.: Dilas: a digital library annotation service. In: International Workshop on Annotation for Collaboration. Methods, Tools, and Practices, pp. 91–101 (2005)

Hansen, P., Jarvelin, K.: The information seeking and retrieval process at the swedish patent and registration office: moving from lab-based to real life work task environment. In: Agosti, M., Fuhr, N. (eds.) Proceedings of the ACM-SIGIR 2000 Workshop on Patent Retrieval, Athens, Greece (2000) 43–53 http://www.sics.se/∼preben/papers/ SIGIR20000-WS.pdf

Covey, D.: Usage and usability assessment: library practices and concerns, vol. 105. CLIR (2002) http://www.clir.org/pubs/reports/pub105/pub105.pdf

Borgman, C.L.: Social Aspects of Digital Libraries-Final Report to the NSF. Technical report (1996) http://is.gseis.ucla.edu/research/dl/UCLA_DL_Report.html

Kovács, L., Micsik, A.: The evaluation computer: a model for structuring evaluation activities. In: Agosti, M., Fuhr, N. (eds.) DELOS Workshop on the Evaluation of Digital Libraries, Padova, Italy (2004) http://dlib.ionio.gr/wp7/ workshop2004_program.html

Banwell, L., Coulson, G.: Users and user study methodology: the JUBILEE project. Inf. Res. 9, 47–56 (2004) http:// informationr.net/ir/9-2/paper167.html

Jeng J. (2005). What is usability in the context of the digital library and how can it be measured? Inf. Technol. Libr. 24: 47–56

Xie H. (2006). Evaluation of digital libraries: criteria and problems from users perspectives. Libr. Inf. Res. 28: 433–452

Klas, C., Albrechtsen, H.N.F., Hansen, P., Kapidakis, S., Kovacs, L., Kriewel, S., Micsik, A., Papatheodorou, C., Tsakonas, G., Jacob, E.: A logging scheme for comparative digital library evaluation. In: Lecture Notes In Computer Science, vol. 4172, pp. 267—278 Springer, Berlin (2006) Research and Advanced Technology for Digital Libraries: 10th European Conference, ECDL 2006, Alicante, Spain, September 17—22, 2006. Proceedings