Large-scale study of web accessibility metrics
Universal Access in the Information Society - Trang 1-24 - 2022
Tóm tắt
Evaluating the accessibility of web resources is usually done by checking the conformance of the resource against a standard or set of guidelines (e.g., the WCAG 2.1). The result of the evaluation will indicate what guidelines are respected (or not) by the resource. While it might hint at the accessibility level of web resources, often it will be complicated to compare the level of accessibility of different resources or of different versions of the same resource from evaluation reports. Web accessibility metrics synthesize the accessibility level of a web resource into a quantifiable value. The fact that there is a wide number of accessibility metrics, makes it challenging to choose which ones to use. In this paper, we explore the relationship between web accessibility metrics. For that purpose, we investigated eleven web accessibility metrics. The metrics were computed from automated accessibility evaluations obtained using QualWeb. A set of around three million web pages were evaluated. By computing the metrics over this sample of nearly three million web pages, it was possible to identify groups of metrics that offer similar results. Our analysis shows that there are metrics that behave similarly, which, when deciding what metrics to use, assists in picking the metric that is less resource intensive or for which it might be easier to collect the inputs.
Tài liệu tham khảo
W3C: Introduction to Web Accessibility. (2005). Accessed in 28 of December of 2021. https://www.w3.org/WAI/fundamentals/accessibility-intro/
Henry, S.L.: Web Content Accessibility Guidelines (WCAG) Overview. (2021). Accessed in 28 of December of 2021. https://www.w3.org/WAI/standards-guidelines/wcag/
Vigo, M., Brajnik, G., Connor, J.O.: Research report on web accessibility metrics. In: Vigo, M., Brajnik, G., eds., J.O.C. (eds.) W3C WAI Symposium on Website Accessibility Metrics, First public working draft edn. W3C WAI Research and Development Working Group (RDWG) Notes. W3C Web Accessibility Initiative (WAI), ??? (2012). http://www.w3.org/TR/accessibility-metrics-report
Vigo, M., Brajnik, G.: Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers 23(2), 137–155 (2011) https://arxiv.org/abs/https://academic.oup.com/iwc/article-pdf/23/2/137/2238626/iwc23-0137.pdf. https://doi.org/10.1016/j.intcom.2011.01.001
Lopes, R., Gomes, D., Carriço, L.: Web not for all: A large scale study of web accessibility. In: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A). W4A ’10. Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1805986.1806001
Kimmons, R.: Open to all? nationwide evaluation of high-priority web accessibility considerations among higher education websites. J. Comput. Higher Educ. (2017). https://doi.org/10.1007/s12528-017-9151-3
Acosta-Vargas, G., Acosta-Vargas, P., Jadán-Guerrero, J., Salvador-Ullauri, L., Gonzalez, M.: Improvement of accessibility in medical and healthcare websites. In: Nunes, I.L. (ed.) Advances in Human Factors and System Interactions, pp. 266–273. Springer, Cham (2021)
Snaprud, M., Sawicka, A.: Large scale web accessibility evaluation - a european perspective. In: Stephanidis, C. (ed.) Universal Access in Human-Computer Interaction. Applications and Services, pp. 150–159. Springer, Berlin, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73283-9_18
Costa, D., Fernandes, N., Neves, S., Duarte, C., Hijón-Neira, R., Carriço, L.: Web accessibility in africa: A study of three african domains. In: Kotzé, P., Marsden, G., Lindgaard, G., Wesson, J., Winckler, M. (eds.) Human-Computer Interaction - INTERACT 2013, pp. 331–338. Springer, Berlin, Heidelberg (2013)
Sirithumgul, P., Suchato, A., Punyabukkana, P.: Quantitative evaluation for web accessibility with respect to disabled groups. In: Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibililty (W4A). W4A ’09, pp. 136–141. Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1535654.1535687
Freire, A.P., Bittar, T.J., Fortes, R.P.M.: An approach based on metrics for monitoring web accessibility in brazilian municipalities web sites. In: Proceedings of the 2008 ACM Symposium on Applied Computing. SAC ’08, pp. 2421–2425. Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1363686.1364259
Song, S., Bu, J., Shen, C., Artmeier, A., Yu, Z., Zhou, Q.: Reliability aware web accessibility experience metric. In: Proceedings of the 15th International Web for All Conference. W4A ’18. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3192714.3192836
Freire, A.P., Fortes, R.P.M., Turine, M.A.S., Paiva, D.M.B.: An evaluation of web accessibility metrics based on their attributes. In: Proceedings of the 26th Annual ACM International Conference on Design of Communication. SIGDOC ’08, pp. 73–80. Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1456536.1456551
Parmanto, B., Zeng, X.: Metric for web accessibility evaluation. J. Am. Soc. Inform. Sci. Technol. 56(13), 1394–1404 (2005)
Abuaddous, H.Y., Jali, M.Z., Basir, N.: Quantitative metric for ranking web accessibility barriers based on their severity. J. Inform. Commun. Technol. 16(1), 81–102 (2017)
Sullivan, T., Matson, R.: Barriers to use: Usability and content accessibility on the web’s most popular sites. In: Proceedings on the 2000 Conference on Universal Usability. CUU ’00, pp. 139–144. Association for Computing Machinery, New York, NY, USA (2000). https://doi.org/10.1145/355460.355549
Vigo, M., Brajnik, G., Arrue, M., Abascal, J.: Tool independence for the web accessibility quantitative metric. Disabil. Rehabil. Assist. Technol. 4(4), 248–263 (2009). https://doi.org/10.1080/17483100902903291
Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J.: Quantitative metrics for measuring web accessibility. In: Proceedings of the 2007 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 99–107. Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/10.1145/1243441.1243465
Velleman, E., Strobbe, C., Koch, J., Velasco, C.A., Snaprud, M.: A unified web evaluation methodology using wcag. In: Stephanidis, C. (ed.) Universal Access in Human-Computer Interaction. Applications and Services, pp. 177–184. Springer, Berlin, Heidelberg (2007)
Freire, A.P., Power, C., Petrie, H., Tanaka, E.H., Rocha, H.V., Fortes, R.P.: Web accessibility metrics: Effects of different computational approaches. In: International Conference on Universal Access in Human-Computer Interaction, pp. 664–673 (2009). https://doi.org/10.1007/978-3-642-02713-0_70. Springer
Martínez, A.B., Juan, A.A., Álvarez, D., del Carmen Suárez, M.: Wab*: A quantitative metric based on wab. In: Gaedke, M., Grossniklaus, M., Díaz, O. (eds.) Web Engineering, pp. 485–488. Springer, Berlin, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02818-2_44
Bühler, C., Heck, H., Perlick, O., Nietzio, A., Ulltveit-Moe, N.: Interpreting results from large scale automatic evaluation of web accessibility. In: Miesenberger, K., Klaus, J., Zagler, W.L., Karshmer, A.I. (eds.) Computers Helping People with Special Needs, pp. 184–191. Springer, Berlin, Heidelberg (2006). https://doi.org/10.1007/11788713_28
Hackett, S., Parmanto, B., Zeng, X.: Accessibility of internet websites through time. In: Proceedings of the 6th International ACM SIGACCESS Conference on Computers and Accessibility. Assets ’04, pp. 32–39. Association for Computing Machinery, New York, NY, USA (2003). https://doi.org/10.1145/1028630.1028638
Brajnik, G., Vigo, M.: Automatic web accessibility metrics: Where we were and where we went. In: Web Accessibility, pp. 505–521 (2019). https://doi.org/10.1007/978-1-4471-7440-0_27
Bailey, J., Burd, E.: Tree-map visualisation for web accessibility. In: 29th Annual International Computer Software and Applications Conference (COMPSAC’05), vol. 1, pp. 275–2802 (2005). https://doi.org/10.1109/COMPSAC.2005.161
Bailey, J., Burd, E.: Towards more mature web maintenance practices for accessibility. In: 2007 9th IEEE International Workshop on Web Site Evolution, pp. 81–87 (2007). https://doi.org/10.1109/WSE.2007.4380248
Brajnik, G., Lomuscio, R.: Samba: A semi-automatic method for measuring barriers of accessibility. In: Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility. Assets ’07, pp. 43–50. Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/10.1145/1296843.1296853
Brajnik, G.: Web accessibility testing: When the method is the culprit. In: Miesenberger, K., Klaus, J., Zagler, W.L., Karshmer, A.I. (eds.) Computers Helping People with Special Needs, pp. 156–163. Springer, Berlin, Heidelberg (2006)
Song, S., Wang, C., Li, L., Yu, Z., Lin, X., Bu, J.: Waem: A web accessibility evaluation metric based on partial user experience order. In: Proceedings of the 14th Web for All Conference on The Future of Accessible Work. W4A ’17. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3058555.3058576
Battistelli, M., Mirri, S., Muratori, L.A., Salomoni, P.: Measuring Accessibility Barriers on Large Scale Sets of Pages. (2011). Accessed in 28 of December of 2021. https://www.w3.org/WAI/RD/2011/metrics/paper2/
Vigo, M., Abascal, J., Aizpurua, A., Arrue, M.: Attaining Metric Validity and Reliability with the Web Accessibility Quantitative Metric. (2011). Accessed in 28 of December of 2021. https://www.w3.org/WAI/RD/2011/metrics/paper6/
Fukuda, K., Saito, S., Takagi, H., Asakawa, C.: Proposing new metrics to evaluate web usability for the blind. In: CHI ’05 Extended Abstracts on Human Factors in Computing Systems, pp. 1387–1390. Association for Computing Machinery, New York, NY, USA (2005). https://doi.org/10.1145/1056808.1056923
Lopes, R., Carriço, L.: The impact of accessibility assessment in macro scale universal usability studies of the web. In: Proceedings of the 2008 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 5–14. Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1368044.1368048
Benavidez, C.: Libro Blanco de eXaminator, (2012)
Mirri, S., Muratori, L.A., Salomoni, P.: Monitoring accessibility: Large scale evaluations at a geo political level. In: The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility. ASSETS ’11, pp. 163–170. Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/2049536.2049566
Lazar, J., Goldstein, D., Taylor, A.: Ensuring Digital Accessibility Through Process and Policy. Morgan kaufmann, ??? (2015)
Fernandes, N., Costa, D., Neves, S., Duarte, C., Carriço, L.: Evaluating the accessibility of rich internet applications. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility. W4A ’12. Association for Computing Machinery, New York, NY, USA (2012). https://doi.org/10.1145/2207016.2207019
Statstutor: Spearman’s Correlation. (2021). Accessed in 28 of December of 2021. https://www.statstutor.ac.uk/resources/uploaded/spearmans.pdf
Wikipedia: Hierarchical Clustering. (2021). Accessed in 28 of December of 2021. https://en.wikipedia.org/wiki/Hierarchical_clustering
Hackett, S., Parmanto, B.: Homepage not enough when evaluating web site accessibility. Internet Research (2009)
Abascal, J., Arrue, M., Valencia, X.: Tools for web accessibility evaluation. In: Yesilada, Y., Harper, S. (eds.) Web Accessibility: A Foundation for Research, pp. 479–503. Springer, London (2019). https://doi.org/10.1007/978-1-4471-7440-0_26