EICS 2023 is the 15th international ACM SIGCHI conference, dedicated to engineering interactive computing systems and their user interfaces. The conference has gained recognition for its commitment to advancing research at the intersection of user interface design, software engineering, and computational interaction. EICS provides a platform for contributions that elevate the standards in the engineering of interactive systems. In this blog post, I delve into our research paper titled "Enabling Customization of Discussion Forums for Blind Users," published in the June 2023 issue of Proceedings of the ACM on Human-Computer Interaction, Volume 7, under the EICS category.
In our research, we aimed to unravel the myriad usability challenges encountered by blind participants in online discussion forums. To achieve this, we conducted an interview study involving 12 blind participants with prior experience in online discussion forums. Drawing insights from their usability challenges, we devised a system that empowers blind users to customize the content of forum threads dynamically. In a subsequent user study with 14 blind participants, evaluating the system revealed noteworthy improvements compared to the existing status quo. PView, our system, significantly enhanced usability, reduced workload, and increased participant satisfaction as they interacted with the forums.
Introduction
Online discussion forums have become increasingly vital in recent years, serving as global platforms for sharing and seeking information on various subjects. These forums cover diverse topics and are integrated into multiple websites across news, entertainment, online education, video streaming, and social media. Users discuss politics, sports, science, and numerous other areas. With the growing significance of these forums, ensuring their accessibility and usability for everyone, including individuals with severe visual impairments like blindness, is paramount.
Recognizing a gap in understanding how blind users interact with these forums, this paper details a study involving 12 blind individuals who regularly use screen readers to access such forums. Based on insights from this study, the researchers developed PView, an intelligent browser extension tailored for these online forums. PView empowers blind users to customize their browsing experience by filtering out posts they prefer not to see. For instance, if a user decides to hide a post, PView, powered by a BERT-based model trained on a custom dataset, can automatically hide similar posts in real-time. This feature is handy in navigating through and avoiding irrelevant or problematic content, such as trolls or hate speech, prevalent in some forums (Figure 1). The paper further evaluates PView, demonstrating its effectiveness in enhancing usability and reducing the interaction workload for blind users, marking a significant improvement over existing browsing experiences in such forums.
 |
| Figure 1 Mohan Sunkara et al.: Depiction of the PView interface concept, where users have the option to deliberately conceal a specific post within a thread. Subsequently, PView is designed to autonomously identify and conceal any other posts that are significantly alike to the one that was hidden. |
Interview study
We recruited 12 visually impaired individuals, seven women and five men, for the interview study. The average age of the participants was approximately 47.75 years (Median = 51.5, SD = 10.64, Range = 24-62). The demographic details of the participants are summarized in Table 1.
 |
| Table 1 Mohan Sunkara et al.: Demographic details of visually-impaired participants in the interview study, based on self-reported information. |
The interviews conducted were semi-structured, focusing on two main areas:
- General inquiries regarding habits related to browsing discussion forums: Questions included topics like preferred websites for interacting with accessibility forums, methods for navigating through conversation threads, and key information in the threads that receive more attention during navigation.
- Exploration of usability challenges and adaptive strategies used when navigating through discussion forum threads: This included questions about frustrations experienced while navigating forums and the techniques employed to overcome these challenges while perusing conversation threads.
The interview data gathered was qualitatively analyzed using the open coding technique. This process involved an iterative review of user responses, during which key insights frequently emerged in the data were identified and categorized. The findings from the interview study highlighted various challenges faced by blind screen-reader users while engaging with online discussion forums. The observations from the study strongly suggested the necessity for a non-visual interface. This interface would ideally allow users to customize the content in forum threads, enabling them to efficiently and comfortably browse through posts in conversations. In response to this need, we developed the PView prototype interface.
System Design
 |
Figure 2 Mohan Sunkara et al.: The architectural schematic of PView depicts its structure as a browser extension designed to enhance the web forum experience for visually impaired users |
Figure 2 depicts an architectural schematic that outlines the workflow of the PView browser extension. This extension, upon detecting a webpage with a discussion forum, automatically extracts threads using the existing CommentsMiner algorithm. The extracted data is then stored in Firebase following a specific schema. When a user is listening to a post and presses ‘ENTER,’ PView displays two buttons, ‘Hide’ and ‘Next Comment,’ below the post and shifts the focus of the screen reader to the ‘Hide’ button. If the user selects ‘Hide,’ PView identifies and eliminates all similar posts, including the current one, from the forum and redirects the focus to the next post that hasn't been filtered out. Alternatively, selecting the ‘Next Comment’ button causes PView to advance the screen reader focus to the start of the subsequent thread, bypassing remaining posts in the current thread. PView, with its ability to provide swift access to posts and threads, significantly aids blind screen-reader users in saving time and effort by preventing the need to navigate through superfluous or irrelevant threads and posts.
To enhance the detection of similar posts, we fine-tuned BERT language model rather than relying on its pre-trained version. This was due to the unique nature of discussion forum texts, which often feature casual language, deliberate grammatical errors, and unconventional sentence structures, deviating significantly from standard texts used in training models like BERT. We created a custom dataset of forum posts and utilized TensorFlow and the Keras API for fine-tuning BERT with this dataset. Additionally, the back-end, built entirely in Python, was integrated with the JavaScript front-end through the Flask API, facilitating seamless communication and operation of the PView browser extension.
Note (1): To develop PView as a working Chrome browser extension(Not available in the public domain due to large-scale development challenges), we adhered to the instructions available on Google's official website. We employed Chrome's built-in API to handle tasks related to the DOM of webpages. Specifically, we used JavaScript functions provided by the API to capture the entire webpage DOM and transmit it to our backend server for additional processing.
User Study Evaluation
We enlisted 14 visually impaired participants (comprising 6 women and 8 men) through email lists and snowball sampling methods. These individuals had an average age of 48.29 years (Median =
49.5, SD = 9.14, Range = 28-62). A key criterion for participation was proficiency in using the JAWS screen reader, as our study was exclusively conducted on the Windows OS platform. The demographic details of the participants are outlined in Table 2.
 |
| Table 2 Mohan Sunkara et al.: Demographic information of visually-impaired participants in the PView evaluation study, based on self-reported data. |
In our within-subject experiment, participants were required to interact with discussion forums under three different conditions:
- Screen Reader: This was the baseline condition where participants used their preferred screen readers to complete the study task, representing the standard method of interaction.
- PView Hide: In this condition, participants used the 'hide' feature of PView in conjunction with their preferred screen readers while interacting with forum threads for the assigned task. This condition tested the effectiveness of PView's 'hide' functionality.
- PView Full: Here, participants had access to all the features of PView ('Hide' and 'Next Comment') while still using their preferred screen readers to interact with forum threads for the task. This condition evaluated the comprehensive functionality of PView.
 |
Figure 3 Mohan Sunkara et al.: Perceived usability, measured using the System Usability Scale (SUS), and task workload, assessed with the NASA Task Load Index (NASA-TLX), for each of the three study conditions. |
Usability was assessed using the System Usability Scale (SUS) questionnaire. The SUS consists of ten questions, each with a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The responses are then aggregated into a total score between 0 and 100. Higher SUS scores signify better usability. The SUS score statistics for each of the three study conditions are displayed in Figure 3a. Further analysis indicated that responses to the SUS Likert items were predominantly positive for the PView Full condition, generally positive for the PView Hide condition, and mostly negative for the Screen Reader condition.
The interaction workload was evaluated using the NASA-TLX, which was divided into two parts. Firstly, participants rated six subjective subscales—Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration. Secondly, they determined the importance of each subscale through pairwise comparisons. The final TLX score, ranging from 0 to 100, is derived from both parts, with lower scores indicating better performance and less workload. Analysis of the TLX responses showed that the Mental Demand, Effort, and Frustration subscales were vital in differentiating the study conditions. The Screen Reader condition scored highest (indicating more workload) in these areas, while scores were notably lower for both PView conditions. Notably, the Effort subscale was a significant differentiator between PView Full and PView Hide, with participants reporting higher effort in the PView Hide condition than the PView Full condition (Figure 3b).
Through qualitative analysis of the experimenter's notes, distinct interaction patterns emerged among participants. In the Screen Reader condition, nearly all (12 out of 14) began by fully listening to the initial posts before quickly navigating subsequent posts with shortcuts. This behavior was almost absent in the PView Full condition. In the PView Hide condition, similar behavior was occasionally observed, particularly when transitioning to the following thread. A notable difference was that many participants (8 out of 14) revisited posts in the Screen Reader condition, unlike the linear navigation in both PView conditions. Additionally, in the PView Full and Hide conditions, participants frequently used the 'Hide' feature (at least 5 and 3 times, respectively) to omit unwanted posts. In the PView Full condition, all used the 'Next Comment' feature at least twice to skip to subsequent threads.
Discussion
The user study revealed that participants initially experienced boredom with the repetitive nature of information in the screen reader default condition. However, as they navigated through multiple posts and threads using PView Full and PView Hide, their engagement increased, appreciating the diversity of opinions on discussion forum platforms. This shift highlighted PView's effectiveness in enhancing the experience of blind screen reader users in online forums. Despite its success, the study also uncovered several limitations and offered valuable insights for future research in this field, which are discussed next.
Our study's limitations include its focus on JAWS screen reader users, excluding other popular screen readers like NVDA and VoiceOver. It was also confined to English language forums, not considering multilingual discussions. PView's current support is limited to desktops, not smartphones Additionally, the evaluation was specific to news discussion forums, which might not reflect the dynamics of forums with different conversation styles.
In their subjective feedback, several participants expressed a desire for PView to remember their preferences, specifically the types of posts they opted to hide. This feature would enable automatic application of these preferences in future interactions with forums, not only on the same website but across different websites as well. Additionally, during the user study, many participants recommended incorporating two new features into PView: thread summarization and sentiment-driven filtering. Fortunately, the current landscape of natural language processing offers numerous models capable of dialog summarization and sentiment analysis, which can be seamlessly integrated into the PView framework to enhance its functionality.
Conclusion
- This paper introduced PView, a pioneering browser extension designed to help blind users customize their experience on discussion forums in real-time. This tool aims to enhance usability, reduce task load, and improve the overall user experience for visually impaired individuals.
- The development of PView was informed by insights from an interview study involving 12 blind participants who were regular online forum users utilizing screen readers.
- In a user study with 14 blind screen reader users, PView demonstrated a significant improvement in usability and a reduction in workload compared to traditional screen readers. The findings from this study not only validate PView's effectiveness but also provide valuable insights and suggestions for ongoing research, contributing to narrowing the web usability gap between sighted and blind users.
References
Sunkara, M., Prakash, Y., Lee, H.N., Jayarathna, S. and Ashok, V., 2023. Enabling Customization of Discussion Forums for Blind Users. Proceedings of the ACM on Human-Computer Interaction, 7(EICS), pp.1-20.
- YASH PRAKASH (@LunaticBugbear)