J Eng Tech Movie Media > Volume 22(4); 2021 > Article
Irvin: Student Insights Related to the Use of Simultaneous Speech Translation for Video Lectures in a University English Course*

### Abstract

For native English-speaking teachers, the ability to overcome communication issues caused by not having the same first language as their pupils is a challenge, especially with low-level students. The increased use of video lectures due to COVD-19 has made this even more difficult. This study was conducted to investigate whether the use of Artificial Intelligence-powered interlingual Simultaneous Speech Translation subtitled video lectures could be a practical solution to overcome this challenge. To that end, 14 participants from a first-semester prerequisite General English course took part in this study. A semi-structured interview was combined with surveys and descriptive statistics, and data was analyzed through qualitative means of thematic, descriptive, and inductive procedures that relied on simultaneous analysis and category construction. Key findings were as follows: First, respondents found the subtitled videos to be highly satisfactory and fairly accurate. Second, respondents reported greater content understanding as the main advance and less emphasis on improving listening ability as the primary disadvantage. Third, the use of English instead of Korean subtitles or subtitling only certain sections of the video in Korean were the main suggestions for the future. Specific responses from the student interviews and future implications are discussed.

### I. INTRODUCTION

Over the past few decades, the role of the Native English-Speaking Teacher (NEST) has been evolving in South Korea. In the early to mid-1990s, many NESTs were considered to be “The Golden Standard” (Walkinshaw & Oanh, 2014, p. 1) of English instruction to English as a Foreign language (EFL) students. However, the efforts of the Korean government to educate and prepare Non-Native English-Speaking Teachers (NNESTs) has led to less need for NESTs. More recently, scholars have investigated the strengths and weaknesses of NESTs and NNESTs (Chun, 2014, 2015; Lasagabaster & Sierra, 2002; Walkinshaw & Oanh, 2014). For example, Park and Irvin (2019) note that student level plays an important role in learners’ expectation and satisfaction toward their NESTs and NNESTs. Specifically, they found that low-level students preferred NNESTs because they were able to communicate through both the first language (L1) and second language (L2). Moreover, NESTs inability to communicate with low-level learners in their L1 has led to difficulties inside the classroom (Park & Irvin, 2019).
Making the interactions between NESTs and low-level students even tougher has been the onset of the COVID-19 pandemic. Specifically, the change from face-to-face to virtual or remote learning settings led to many sudden changes, including the use of video lectures (henceforth VLs). Following this, a rush of research was produced to identify the main challenges and impactful personal experiences of both educators and learners who dealt with emergency online learning (Basri, Husain, & Modayama, 2021; Famularsih, 2020; Mahyoob, 2020). With regards to the Korean context, Irvin and Park (2021) noted that many first-semester freshman students felt very isolated due to their teachers’ decisions to use asynchronous VLs. In particular, it was found that learners craved more interactions with their professors to overcome hardships understanding difficult sections of their VLs. Furthermore, based on the findings of Bailey and Lee (2020), the quick shift from face-to-face to virtual classes led to complications for NESTs, especially novices, in transferring their class materials online. Therefore, finding a way to utilize advanced technologies to improve VLs could be very beneficial for both students and NESTs themselves.
Currently, in South Korea, the evolution in the quality and availability of technologies inside the classroom has coincided with the changing roles of NESTs. For example, hardware such as computers, projectors, smartboards, etc. have become commonplace in Korean schools and universities. However, there has also been great progress over the past two decades in virtual lectures. In fact, even before the COVID-19 pandemic, there was a growing number of online programs, including Massive Open Online Courses (MOOCs), being developed (Lange & Costley, 2020). Research has shown that one of the biggest obstacles to the success of these MOOCs is the communication barriers that can be created due to language (Che, Luo, Yang, & Meinel, 2017). As noted earlier by Park and Irvin (2019), a similar problem can be found in a low-level EFL classroom. Therefore, instructors of online courses needed to find a way to overcome this challenge, as it was found that student satisfaction was related to the perceived impact of the video (Scagnoli, Choo, & Tian, 2019). One of the first techniques used was the use of captioning or subtitling a VL. However, the time and effort it took to make manual captions or subtitles was determined to be too costly (Chan, Kruger, & Doherty, 2019; Che et al., 2017). To overcome this deficiency, Automated Speech Recognition (ASR) programs were adopted to take Speech to Text (STT) inputs to create subtitles. While the results were not always accurate (Chan et al., 2019; Che et al., 2017), they did reduce the time and effort costs. Additionally, it created a baseline in which others could develop even better technological tools to create even better results.
One of the cutting-edge technology tools used for captioning and subtitling is that of Simultaneous Speech Translation (SST). This system built from the standard STT and ASR programs to incorporate Artificial Intelligence (AI) and Machine Translation (MT) to create real-time subtitles for VLs (Fantinuoli & Prandi, 2021). Additionally, the SST program has the ability to produce both interlingual or intralingual subtitles. For instance, the teacher could select a setting that will make their spoken words create English subtitles or a different option that translates the lecture into the learners’ L1. For NESTs, using SST in their VLs could be an excellent way to overcome any communication difficulties with their low-level students. However, prior research suggestions caution is needed by language instructors who are thinking about using subtitles (Chan et al., 2019). One of the big reasons for this is the Cognitive Load Theory (CLT). In short, this suggests that giving unclear or excessive information or stimuli can lead learners to not be able to transfer information from their working to long-term memory (Cierniak, Scheiter, & Gerjets, 2009; Costley, Fanguy, Lange, & Baldwin, 2021). Thus, the teacher must consider if the subtitles are adding to student understanding or hindering it. Moreover, for low-level students who could struggle more with concepts or directions, the Limited Capacity Theory is also relevant. That is to suggest that not only the amount of information but also the pace at which it is delivered is crucial (Lange & Costley, 2020; Mayer & Moreno, 2003).
Thus, this study has been conducted to examine the use of SST in VLs in a low-level General English university course during remote learning. Specifically, it attempts to gain insight based on student opinions of whether or not SST could be a viable option for NESTs to help bridge not only the language gap but also the asynchronous gap that has been created due to COVID-19. To that end, Microsoft PowerPoint 365 was used to create SST for four separate VLs. Moreover, interlingual simultaneous subtitles were selected for this project. This means that while the audio for each video was in English, the AI and MT were set to produce Korean subtitles. The rationale for this choice was to give the low-level learners the opportunity to utilize the L1 subtitles when they did not understand a certain portion of the VL. As research regarding interlingual simultaneous subtitling in education is currently limited, this study will ascertain student perceptions towards the use of interlingual SST in VLs through analysis of interviews and surveys. In order to guide this research, four questions were made.
• 1. What is the perceived impact of SST subtitles on satisfaction for low-level university students in a Korean context?

• 2. What specific advantages do low-level university students identify with regards to using SST subtitles during an online course?

• 3. What specific disadvantages do low-level university students identify with regards to the common and repetitive errors when using SST subtitles during an online course?

• 4. What specific suggestions do low-level university students have with regards to the future use of SST subtitles during an online course?

### 1. Native English-Speaking Teachers (NESTs) in South Korea

Starting in the early 1990s, scholars such as Medgyes (1992) and Phillipson (1992) started the discourse related to NESTs and NNESTs. However, by the mid-1990s, the need for NESTs in South Korea was about to explode. This was caused by the adoption of the 6th National Education Curriculum by the Korean government. It was this document that saw a shift from the traditional Grammar-Translation teaching style to a communicative approach (Chung & Choi, 2016; Kwon, 2000). In order to achieve their goals, the Korean government implemented several recruitment initiatives to attract NESTs and developed programs to help train NNESTs (Chung & Choi, 2016; Kwon, 2000). In simple terms, the foreign NESTs were seen as valuable resources to help enact the new communicative approach quickly. However, the long-term solution was to develop the essential skills of native Korean instructors for the new teaching style.
As the new millennium started, the trend that had begun in the 1990s continued. Specifically, a large number of foreigners were coming to South Korea annually to teach as NESTs. Yet, as the number of NESTs was growing, so too was the number of NNESTs (Chung & Choi, 2016). Scholars examining the Korean context confirmed that each of the teaching groups has its own unique advantages and disadvantages (Chun, 2014, 2015; Walkinshaw & Oanh, 2014). For example, Walkinshaw and Oanh (2014) noted that NESTs have advantages based on their pronunciation, knowledge of the language, and ability to share specific details from their own culture. While on the other hand, Chun (2015) found that a lack of knowledge related to the NESTs host country’s language, people, and culture combined with a lack of enthusiasm and responsibility could be problematic. Another notable finding from Chun’s research was that the notion that students preferred one group over another was simply not true. Instead, their choice tended to be based on the kind of teacher they were or what specific skill the student was trying to learn (Chun, 2014). More recently, Park and Irvin (2019) added to the discourse by highlighting that low-level students tend to have a higher preference for NNESTs due to their shared learning experience and ability to communicate in either the student’s L1 or L2. However, at universities across South Korea, NESTs are still teaching all levels of learners.
Another major challenge that no one could have predicted was the COVID-19 pandemic of 2020 and 2021. On March 11, 2020, the World Health Organization (WHO) labeled the coronavirus as a pandemic (Octaberlina & Musilimin, 2020), leading to a transition from face-to-face to remote lectures. From the very start, there were many issues for teachers, students, and administrators to adjust quickly due to the short notice (Mishra, Gupta, & Shree 2020). Specifically, in South Korea, even though there was a large technological infrastructure in place, many NESTs struggled to adapt. Bailey and Lee (2020) found a primary cause for these struggles was related to NEST’s lack of experiences with online teaching technologies before COVID-19. As a result of their limited time and lack of skills, many instructors chose to overcome these challenges through the use of VLs (Irvin & Park, 2021). However, by making this choice, NESTs were unknowingly creating a bigger divide between them and their low-level students. Specifically, the use of VL utilizing only the learners’ L2 would make understanding more challenging. Moreover, the lack of interactions between students and instructors would not allow for students’ questions to get immediate responses (Irvin & Park, 2021).

### 2. AI Subtitles as a Possible Solution

As noted by Bailey and Lee (2020), many novice NESTs had trouble adjusting to virtual learning due to their lack of experience before the COVID-19 pandemic. However, different approaches such as electronic learning (e-learning), mobile learning (m-learning), and digital learning (d-learning) have been developed and used by schools and universities for decades (Basak, Wotto, & Bélanger, 2016; Nartiningrum & Nugroho, 2020). One prominent recent example is the development of MOOCs. These MOOCs have become widespread as an option to learning that allows students to enroll and watch VLs whenever and wherever they like (Scagnoli et al., 2019). The increased enrollment in these virtual classrooms has allowed for data to be collected related to student satisfaction with their VL-based courses. One relevant study was that of Scagnoli et al.’s (2019), which investigated students’ insights towards the use of VLs. Through the use of surveys and focus groups, they found that student satisfaction with VLs had a strong relationship with the overall learning experience and the perceived student impact of the video on their learning. However, they found that good VLs are the product of proper planning and integration by the instructor (Scagnoli et al., 2019). These findings highlight the fact that while it is possible to create substantive VLs, it also takes great time and effort to produce online materials that connect to each individual learner.
Another key factor to student understanding and satisfaction with VLs is the quality of the video itself. Lange and Costley (2020) put this in more context by identifying 5 common media delivery problems of pace, intelligibility, quality, media, and congruence. Additionally, these authors considered how the students cognitively process information delivered to them through media. In regards to the pacing of a VL, a key factor to consider for low-level students is the Limited Capacity Theory (LCT) (Lange & Costley, 2020). Mayer and Moreno (2003) suggest that LCT is related to when a student feels the information is being delivered too quickly, limiting their comprehension (Lange & Costley, 2020). Moreover, according to Zhu (2021), that aspects such as a student’s working memory capacity, proficiency and previous experience or exposure to the L2, and ability to understand the main aspects of the target language all can affect cognitive abilities. For many low-level students, their ability to overcome these deficiencies is crucial. Yet, the use of VLs by NESTs has led to a desire for more interactions for their students (Irvin & Park, 2021). This lack of interactions has made it even more difficult for low-level learners to use strategies such as requests to repeat the materials, to present more slowly, or to ask follow-up questions to overcome their shortcomings.
Along with the LCT, the Cognitive Load Theory (CLT) can also affect student understanding and satisfaction. Sweller (2020) explains CLT as “an instructional design theory based on our rapidly expanding knowledge of human cognition” (p. 1). Moreover, that the use of the CLT will result in the identification of cognitive load effects and appropriate instructional procedures, one of which is the use of technology-assisted learning (Sweller, 2020). Thus, a recent project relevant to this study is Costley et al. (2021). Their quantitative study used survey responses from 2,012 South Korean university students related to the effects of VL viewing strategies to combat cognitive overload. First, their findings matched that of Scagnoli et al. (2019). Specifically, there is a strong need for clear and effective instruction from NESTs (Costley et al., 2021). Second, Costley’s team identified the result of poorly delivered VLs leads to cognitive load problems. In particular, they noted the work of Cierniak et al. (2009), who asserted that “high levels of cognitive load can inhibit learner attempts to transfer unclear or excessive information from working to long-term memory” (Costley et al., 2021 p. 20). Furthermore, unstructured VLs can cause students to have a high extraneous load or the processing of information that is not essential to the learning process (Cierniak et al., 2009; Costley et al., 2021). Therefore, it is crucial that NESTs using VLs consider not only the content that will be included but also what strategies can be employed to overcome student difficulties brought about by language barriers.

### 1. Participants

Table 1 shows the demographic information for all participants for this study collected during the initial survey. A total of 14 students currently enrolled in a compulsory general English course at a private university in South Korea participated in this study. The participants consisted of both male and female students (i.e., 6 males and 8 females) aged 20 to 24 years old, enrolled in a freshmen level 4-skills English language class. The students were from three different colleges at the institution (i.e., College of Health and Welfare, College of Public Health Science, and College of Natural Sciences), with students picking their class based on available time and level. Individuals enrolled at the university are required to take 6 prerequisite course hours of English language (i.e., College English 1 (CE1) and College English 2 (CE2)). Each course is separated into three specific levels (i.e., low, intermediate, and high), and an induvial student’s level is determined based on an English entrance exam taken the month before their freshmen year. Typically, students complete the required English courses during the first two semesters of their freshmen year, but some students choose to retake the course. In this research project, one participant was a senior student retaking the class to meet graduation requirements and/or increase their GPA. However, as they were enrolled in the class and completed all aspects of the project, their opinions were viewed as important to this research.
This study took place during the Spring semester, so it was a CE1 course. Thus, participants took part in a class that utilized a 4-skills textbook, with the biggest emphasis usually being placed on speaking and listening. However, due to COVID-19, classes were conducted remotely with students watching weekly asynchronous VLs. For this research project, the procedures can be found in Table 2. Data for this qualitative study was collected over six weeks. At the start of the six weeks, students were asked to fill out a pre-survey to collect demographic information and determine prior experiences using subtitles in and out of the classroom. Over the next three weeks, students watched four separate VLs presented in English with Korean SST subtitles ranging from 35 to 44 minutes long (i.e., class 1 = 35 minutes, class 2 = 38 minutes, class 4 = 39 minutes, and class 5 = 44 minutes). Typically, 4 classes would encompass two weeks of a given semester. However, due to national holidays, it was spread to three weeks. For each video, students were asked to fill out a 26-question survey on their experiences, satisfaction, challenges, and opinions related to that specific video. At the end of the first four weeks, one-on-one interviews were scheduled between the researcher and each participant. These interviews were conducted during weeks 5 and 6, with each interview lasting between 15 to 25 minutes. Finally, with the completion of all the interviews, it was determined that a follow-up survey was needed. This 20-question survey was an opportunity for participants to supplement their interview responses.

### 2. Procedures

This study took place during the Spring semester, so it was a CE1 course. Thus, participants took part in a class that utilized a 4-skills textbook, with the biggest emphasis usually being placed on speaking and listening. However, due to COVID-19, classes were conducted remotely with students watching weekly asynchronous VLs. For this research project, the procedures can be found in Table 2. Data for this qualitative study was collected over six weeks. At the start of the six weeks, students were asked to fill out a pre-survey to collect demographic information and determine prior experiences using subtitles in and out of the classroom. Over the next three weeks, students watched four separate VLs presented in English with Korean SST subtitles ranging from 35 to 44 minutes long (i.e., class 1 = 35 minutes, class 2 = 38 minutes, class 4 = 39 minutes, and class 5 = 44 minutes). Typically, 4 classes would encompass two weeks of a given semester. However, due to national holidays, it was spread to three weeks. For each video, students were asked to fill out a 26-question survey on their experiences, satisfaction, challenges, and opinions related to that specific video. At the end of the first four weeks, one-on-one interviews were scheduled between the researcher and each participant. These interviews were conducted during weeks 5 and 6, with each interview lasting between 15 to 25 minutes. Finally, with the completion of all the interviews, it was determined that a follow-up survey was needed. This 20-question survey was an opportunity for participants to supplement their interview responses.

### 3. Data Collection and Analysis

For the purposes of this qualitative study, the primary data collection tool was one semi-structured interview (Merriam, 1998). A list of 10 structured questions (see Appendix 1) was developed for the interview, with follow-up questions being asked when needed. To ensure validity and reliability, two outside experts were consulted, and the questions were revised accordingly. Furthermore, each interview was conducted using Google Meet, with the recording function being used. Each participant was told that the video call was being recorded, and every person agreed to continue to take part with this knowledge. Each interview was video recorded with participant consent for later transcription. Moreover, due to the researcher’s limited Korean ability, the interview was conducted in English. This could be one of the reasons that participant responses were short. To overcome this challenge, the researcher encouraged students to speak in Korean when they felt they had more to say. These responses were also transcribed after being translated by the two experts who are proficient in both English and Korean. After transcription of the interviews, thematic, descriptive, and inductive procedures with non-statistical analysis were used to develop main themes. As the data continued to be collected and analyzed, inductive and comparative processes were used to create new categories, combine or subdivide categories, and delete redundant or unnecessary categories (Johnson & Christensen, 2004; Merriam, 1998).
In addition to the primary tool of an interview, the use of surveys containing semi-structured and open-ended questions (Maykut & Morehouse, 1994) supplemented this research. A total of 6 surveys were used throughout the six-week project. For each survey, questions were made using Google Document, with a combination of Likert-scale and opened-ended questions. A 7-point Likert Scale was used (i.e., 1 being the extreme negative to 7 being the extreme positive opinion), while open-ended responses were created to gain more perspective on students’ authentic feelings.
All surveys were developed in English by the research, and like the interview questions, each survey was shared with 3 outside experts (i.e., 2 NNESTs and 1 NEST). However, the extra step of translating the questions into Korean was also taken. Additionally, participants were given the option to write their responses in English or Korean. Therefore, once all the data was collected, the 2 NNESTs translated the responses from Korean into English. As this study utilized simultaneous analysis and category construction, participant responses to surveys 1 through 5 were analyzed and used to identify meaningful segments of information, which were then used to find initial themes and development of the interview questions (Merriam, 1998). Furthermore, based on the short responses during the interview, survey 6 was created to ask follow-up questions to verify themes and findings. Finally, through the use of Excel 2019, descriptive analysis of the surveys (see Tables 3,-5) was implemented to calculate the mean and standard deviation (Kaur, Stoltzfus, & Yellapu, 2018).

### IV. RESULTS AND DISCUSSIONS

Through the analysis of the interviews and supplemental surveys of this study, the following four main categories have emerged; 1) low-level university students’ satisfaction towards Korean SST subtitles in English VLs; 2) low-level university students’ perceived advantages to using Korean SST subtitles in English VLs; 3) low-level university students’ perceived disadvantages to using Korean SST subtitles in English VLs; 4) low-level university students’ suggestions towards the future use of Korean SST subtitles in English VLs.

### 1. Low-Level University Students’ Satisfaction Towards Korean SST Subtitles in English VLs

With regards to students’ evaluations of how satisfied they were with the Korean SST subtitles during the VLs for College English 1 (CE1), every student (i.e., 14 out of 14) chose 5 or higher on a scale of 1 to 7 with 1 as very dissatisfied and 7 as very satisfied. This shows that students’ self-reported satisfaction level is much higher than the average score (i.e., 4 out of 7). To be more specific, 50% chose 6, 29% chose 7, and 21% chose 5. The finding of a high level of satisfaction with the Korean SST subtitles was confirmed through the interviews. In particular, several learners pointed out that the subtitles helped them understand more, and as a result, they felt more comfortable. However, it was also discovered that there were three students who tried not using the subtitles. Based on the interviews, their rationale is they felt it was not for them and hindered their language goals. Yet, all three mentioned that the subtitles were good a reference point for them to check their English listening comprehension. These results indicate that students are satisfied by either their increased comfort level or ability to check for understanding that SST subtitles provide. However, they also highlight that even within a low-level section, there can still be a wide range of knowledge and skills. Therefore, additional research to consider how SST subtitles could be employed to reach each individual student’s need is warranted.
Subtitles are very necessary… It is comfortable with subtitles. - S13
For every student it is positive, but I don’t think I care about subtitle. - S2
I use them to check for understanding or find understanding after effort. - S4
In regards to the information in Table 3, for each VL containing Korean SST subtitles, students were asked to rate their satisfaction with the video and subtitles. Each student picked their own score based on a 1 to 7 scale, with 1 as very dissatisfied and 7 being very satisfied. The mean and standard deviation of each score for the individual class was calculated. Then, the mean and standard deviation of all four classes was calculated. The results show three important trends related to student satisfaction with the subtitles. First, the mean score of subtitle satisfaction was always equal to or lower than the overall video satisfaction. Second, the mean score of subtitle satisfaction was considerably lower for the first VL compared to the other three. Third, each time the learners watched the videos with Korean SST subtitles, their satisfaction grew. These findings point to learners not only being satisfied with the videos but also being adaptable to the new technology. However, patience is needed to determine why the first video was much lower and if spending time to develop specific viewing strategies (Costley et al., 2021) could be beneficial to low-level students.

### 2. Low-Level University Students’ Perceived Advantages to Using Korean SST Subtitles in English VLs

With regards to students’ evaluations of the advantages of using Korean SST subtitles during the VLs for College English 1 (CE1), understanding and help were by far the most common response. During the 14 interviews, many participants stated that the Korean SST subtitles made the videos easy to understand, while allowing them to more easily comprehend the specific tasks that were needed to be completed. More specifically, students mentioned the biggest advantages of the subtitles were related to understanding the content, tasks, and meanings of unfamiliar words. Furthermore, it was also noted that using subtitles saved time for studying. For example, student S3 said that their listening improved, and they could save time by using the subtitles instead of searching on Naver or Google. Overall, the findings related to the understandability and helpfulness of this study are in line with those of Ferdiansyah and Nakagawa (2013). Therefore, NESTs can be encouraged by the notion that the use of Korean SST subtitles in VLs for low-level students could result in fewer communication issues between themselves and low-level learners.
I don’t speak English, subtitles is I understand. - S10
The first was seeing the video, I’m don’t understand. But time to time, again and again, I see, I watch video, I understand these tasks. - S1
Definition - I don’t know a word, I can find subtitle. - S8
Professor’s word, some parts I don’t understand. I searching Naver Google but after subtitles I don’t have to search this… - S3
Next, Table 4 shows information related to what parts of the video students understood and in which sections the subtitles helped them the most. To build off of previous research, this study aimed to determine if there were specific sections that subtitles could help them understand better. To accomplish this, each student started by rating their understanding of each section on a 1 to 7 scale, with 1 as no understanding and 7 as full understanding. Then, they were asked to evaluate whether the subtitles helped their understanding on a 1 to 7 scale, with 1 being no effect and 7 being large effect. These numbers from all four classes were collected, and the total mean and standard deviation for each section of the video was calculated. Finally, the chart was organized from largest to smallest based on the differences in mean scores.
A closer look at Table 4 shows that the Introduction, Assignments, and In-Class Activities had the highest difference in mean scores. As mentioned earlier, understanding the content was said to be the greatest advantage during the interview, but that term could be too broad. Thus, by separating the VL into 10 specific segments, attempts to pinpoint the specific content that is most important can be made. These initial survey results provide some noticeable differences, but interestingly, the interview findings are not the same. The interview responses found that while Assignment was the most frequent response, no differentiation was made between the Assignment Contents or Directions. Moreover, the Introduction was only mentioned once during the interviews. Therefore, based on both the interviews and surveys, emphasis on using subtitles for the Assignments could be appropriate. However, additional research considering the ‘most challenging’ section is needed. Specifically, this study used a list of 10 parts, but developing a standardized list of terms could be very beneficial. Furthermore, the process of dividing the VL into sections could greatly benefit from the input of learners too.
Finally, in regards to Table 5, students were asked to evaluate the accuracy of the Korean SST subtitles. In order to do this, learners were asked to rate the accuracy of the subtitles in each VL containing subtitles based on a 100% scale, with 0% meaning no accuracy and 100% equaling perfect accuracy. The totals for all four classes were collected, and the total mean and standard deviation for the subtitle accuracy for each video was calculated. In addition, the four-class mean and standard deviation were also calculated.
A closer examination of Table 5 shows that the collective mean of perceived SST subtitle accuracy for the four classes was nearly 80%. However, the mean score for the perceived accuracy of SST subtitles for the Class 1 video (M = 65.714) was considerably lower than the other three classes. It should be noted that the mean score related to Class 1’s satisfaction level was also the lowest. Yet, after Class 1, the perceived SST subtitle accuracy level increased in Class 2 (M = 80.643) and Class 3 (M = 87.714) before slightly lowering in Class 4 (M = 83.857). These findings could be the result of students having difficulties because it was their first time using this kind of technology, or it could be that Class 1 is a truer representation of PowerPoint 365’s AI capabilities. Regardless, the increase does indicate that the participants in this study determined that the videos were accurate for their needs. Now, this could be a sign that Korean SST subtitles could fulfill the desired needs of NESTs, but caution is needed. This is because while students may perceive understanding as accuracy, their lack of linguistic training and ability means that more research into the actual Word Error Rate of Microsoft PowerPoint 365’s SST, from spoken English to Korean subtitles, is needed.

### 3. Low-Level University Students’ Perceived Disadvantage to Using Korean SST Subtitles in English VLs

With regards to students’ evaluations of the disadvantage of using Korean SST subtitles during the VLs for College English 1 (CE1), the most common response in the interviews was related to a decrease in listening skills. The majority opinion was that while the subtitles did help them with understanding the context of the class, it meant that they were less likely to listen to what the instructor was saying, with S14 mentioning how they felt dependent on the subtitles. Additionally, one student mentioned how they were disappointed that Koran was being used in an English class. Specifically, student S4 said, “Why I am choose this because I want 100% English class. Reading, speaking, writing, so it is a little bad for me.” These results bring to light one of the biggest challenges for any teacher who is thinking of using SST subtitles. Mainly, while there will be a group of students in a low-level class who can strongly benefit from the subtitles to help overcome their limited capacity (Lange & Costley, 2020), there is a second group of students who have higher capabilities. In this case, the desire to not be distracted or have a feeling of cognitive overload (Lange & Costley, 2020) created by the subtitles, but instead, be able to build their listening skills. Therefore, it could be prudent to reexamine how classes are leveled. Perhaps, more weight should be placed on listening ability, or further dividing of low-level classes could be necessary. Furthermore, additional research centered on finding a connection between having a better understanding of the content with achievement is recommended.
It’s help to understand contents, but English listening skill is down. - S5
Listening skills is not improved, can’t improve because of the subtitles. - S7
And subtitle is exist, then I understand the contents very well, but I saw the subtitle. I can’t teacher saying very well. - S9
It is hard to understand when there are no subtitles. (pause) Yeah, I don’t like. (another pause) I become too dependent. - S14
Next, in regards to students’ experiences with reading the subtitles, several problems with the ASR and MT to form the Korean SST subtitles were noted. According to the survey results, the biggest negative for the learners was the simultaneous nature of the subtitles. The interviews expanded on this as many students mentioned how the randomness of the cuts and changes made the SST subtitles hard to follow at times. Moreover, several students also noted that another problem they faced was incorrect translations. Finally, one student suggested that the placement of the subtitles was uncomfortable. During this study, subtitles were placed at the top of the screen due to the way that the videos were recorded. However, S4 explained that based on their prior experience of watching TV shows or movies with the subtitles on the bottom, it was difficult to adjust. These results fall in line with previous research but also suggest that how long the SST subtitles stay on the screen is a key factor. Moreover, one student noted that when they attempted to use 2x speed for the VL, the STT subtitles did not speed up with the video. Therefore, cooperation between educators and designers making AI programs could greatly benefit all involved by improving the way the SST subtitles are displayed on the screen.
There are errors. Unfinished words. - S6
If teachers say many times the same word, the subtitles is wrong. - S7
Please, just one sentence. Just word, word, word is uncomfortable. - S12
It’s not comfort. I’m not seeing my subtitles up, subtitles up there. But I only see down subtitles, so not seeing the subtitles. - S4

### 4. Low-Level University Students’ Suggestions Towards the Future Use of Korean SST Subtitles in English VLs

With regards to students’ preference towards the use of SST subtitles in future classes, interview responses showed that 11 were in favor of using them, with the remaining 3 suggesting they did not care. Based on these responses, the question, “What language would you prefer to see the AI subtitles displayed in,” was asked in the post-interview follow-up survey. The results to this question were that 9 students preferred English subtitles, while the remaining 5 selected Korean as their preference. These findings, combined with the satisfaction results in section 4.1, show that SST subtitles could be a practical option for NESTs instructing low-level students. However, additional research into whether English or Korean subtitles would be better is needed. Furthermore, attempting to quantify the connection between learner understanding based on the SST subtitles to their academic achievement is advisable.
Additionally, the rationale for why students preferred the use of one language over the other when using SST subtitles in VLs validated many of the previous points in this study. Mainly, the group that preferred the Korean subtitles felt that they benefited greatly from an increase in their understanding of the VLs content and assignments. With one student explaining that they could understand the details faster and that they understood correctly without having to watch repeatedly. While another student requested the SST subtitles be brought back because since they had stopped, they had comprehended less of the materials. On the other hand, students who wanted English subtitles expressed their tendencies to ignore the verbal English and just focus on the Korean text. In addition, another student expressed direct concern because the slow and ever-changing nature of the AI technology made them feel like they may have missed important details. Furthermore, one student went as far as to suggest that reading the Korean test made them feel like they were in a Korean rather than an English course. Finally, the biggest motivation for the VLs to use English subtitles was related to learners’ desires to improve their listening skills. These results again show the drastic difference in levels and learning expectations. While these differences need to be considered by any NEST, they also bring to light an important aspect related to potential cognitive and extraneous load problems (Costley et al., 2021). This study’s results highlight participants’ ability to overcome potential pitfalls by simply ignoring one of the two stimuli. For example, more capable students would more often avoid looking at the subtitles to focus on improving their listening skills. While less capable students would ignore the professor’s voice, allowing them to concentrate more on the Korean text to increase their understanding. Therefore, additional research that uses English SST subtitles should determine if more students struggle with cognitive and extraneous load challenges.
I understand is correctly, and fast understand. - S13
A.I. is slow, so English subtitle not understand. - S10
11 week, probably there was subtitle, but recently there was not subtitle. So, I am a little not understand. - S11
I don’t listen your voice, I see only subtitles. - S5
Students who listening to this CE1 class, don’t effort learning English, but learning Korean. - S3
I can understand English sentence, if Korean subtitle students can’t learn listening. But if English subtitles can learn good or well. - S8
Finally, in regards to students’ suggestions towards improving the VLs with SST subtitles, the large majority of participants (i.e., 10 out of 14) said that they were pleased with the current system. However, the remaining students proposed three ways that they would like to improve the viewing experience. First, two students suggested that they would like to have an option to turn the subtitles on and off. Second, student S12 recommended, it would be better if the subtitles were generated sentence by sentence rather than word by word. Finally, one student asserted that perhaps the subtitles could be limited to certain sections of the video. Based on this final response, the question, “If the change was made to only include subtitles in certain parts of the video, could you select which sections of the video should include AI subtitles?” was asked on the post-interview follow-up survey. Participants were able to pick more than one response, and their answers were: Assignment Contents = 13, In-class Activity Contents = 8, Assignment Directions = 7, In-class Activity Directions = 4, Greetings = 1, Main Contents = 1, and Wrap Up = 1, and Final Remark = 1. Comparing these results to the data for Table 4, the four most common responses here were all in the top 5 results there. Therefore, based on the categorization of the VLs and the students’ responses, NESTs who are thinking about using Korean SST subtitles for only certain sections of their videos should prioritize activity and assignment contents and directions as the areas that will have the greatest benefit.

### 5. Discussions and Future Implications

The results of this study show similar findings to Ferdiansyah and Nakagawa (2013) that captions or subtitles in VLs can be helpful and increase understandability. Moreover, it has taken the suggestions of Fantinuoli and Prandi (2021) to investigate if SST is possible within the sphere of education. While also pushing forward the recent discussions in academia surrounding the use of interlingual subtitles made through ASR and MT. In particular, this project examined the satisfaction level, perceived advantages, perceived disadvantages, and future suggestions from low-level Korean university students who watched VLs containing Korean SST subtitles.
The first implication of this study is that interlingual SST subtitles could be a viable option for NESTs to incorporate into their VLs for low-level classes. In general, the findings of this research based on survey responses found that participants had high levels of satisfaction and believed the Korean SST subtitles to be fairly accurate. Added to this, interview responses highlighted that a valuable understanding of content could be acquired. In particular, it was found that students felt more aware of what was happening in class and what they needed to do for assignments, with one student requesting that the Korean SST subtitles be brought back. These results demonstrate that it is possible to overcome potential existing language barriers between NESTs and their low-level students due to a lack of a shared L1 (Park & Irvin, 2019). Moreover, the perceived high satisfaction rate of the participants in this study help to confirm the findings of Scagnoli et al. (2019) that a positive learning experience can come from a well-made VL. That is to suggest, the addition of the Korean SST subtitles allowed the learners to overcome learning challenges or avoid misunderstandings, resulting in a high satisfaction rating. However, not all learners had the same experience. Some felt the subtitles were distractions or not needed, but the suggestion of limiting the Korean SST subtitles to certain sections of the VL could be a perfect compromise. This could allow the lowest level students to comprehend the most challenging portions of the video while allowing more competent students the opportunity to focus on improving their listening ability. However, as Bailey and Lee (2020) noted, a major obstacle for NESTs is to overcome their lack of technology experiences. Therefore, efforts should be made to inform instructors that AI programs that use ASR, MT, and SST are available. After this, schools, universities, or teachers themselves should invest money into purchasing these programs, like PowerPoint 365. Finally, efforts should be made to educate NESTs on how to implement these technological advances into their online courses.
The final implication for this study is related to what the cost or need of this sort of program could be. To begin, the findings of this project confirm that ASR and MT can save time and money for educators (Chan et al., 2019; Che et al., 2017). The cost of PowerPoint 365 typically is $150 for a one-time installation or between$70 and \$100 every year for a shared subscription service. However, after that, setting up intralingual or interlingual SST subtitles is only six mouse clicks away. Therefore, NESTs who have low-level students have incentivization to invest into this program. Yet, the combination of the increased capabilities of NNESTs combined with low-level students’ preference towards Korean instructors who have a shared learning experience and L1 (Park & Irvin, 2019) raises questions about NESTs who teach low-level classes with little to no L1 ability. Thus, at schools and universities where administrations have the capability to assign NNESTs to low-level classes, this could be the best practical solution. However, this is not always going to be possible. Moreover, the results of this study show that there is a certain group of low-level students who are more capable and feel that learning from a NEST gives them the best chance of improving their skills, especially their listening ability. Therefore, schools and universities could provide funding for NESTs to purchase these tools or invest in creating workshops that show how to implement SST subtitles into VLs. Finally, NESTs with little ability in their students’ L1 could also invest time and money building their own skills.

### V. CONCLUSION

In the end, this study has reaffirmed some of the key findings from prior research related to the use of captions and subtitles in the field of Education. Yet, it has built upon the exhibiting data by examining how the use of cutting-edge AI-powered SST subtitles affected low-level learners taking a NESTs class. In particular, it focused on if interlingual SST subtitles would be a viable option for NESTs based on low-level learners’ satisfaction, identification of positives and negatives, and suggestions for future use. The results showed that participants of this study perceived the Korean SST subtitles to be highly satisfactory and accurate, and they suggested that limiting the subtitles to certain sections or considering the use of English SST subtitles could be beneficial. Yet, this research project was not without its limitations. First, this study only collected data for one class due to its qualitative nature, and the interviews were conducted in English. Therefore, future studies that incorporate more low-level classes with quantitative methods could be beneficial. In particular, quantitative research that can qualify student satisfaction with academic achievement could be particularly helpful. Moreover, conducting the interviews in Korean, or having an outside expert capable of acting as a translator, could elicit even richer student perspectives. Second, as mentioned during the student interviews, investigations into the use of English rather than Korean SST subtitles or only utilizing subtitles during the most challenging sections should be considered. Finally, soon the COVID-19 pandemic is likely to end, and teachers and students will return to the classroom for face-to-face classes. At that time, longstanding communication problems between NESTs and low-level students will still be present. Yet, with many schools and university campuses increasing the amount of hardware and software found inside the classroom. Future research that targets the possibility of taking Korean SST subtitles into a face-to-face class could be interesting to see if it improves understandability, satisfaction, and achievement.
##### TABLE 1
Demographic Information for Participants
Participant Code Gender Age School Year College of Study
S1 Male 20 Freshmen College of Health and Welfare
S2 Female 20 Freshmen College of Public Health Science
S3 Male 20 Freshmen College of Public Health Science
S4 Female 22 Freshmen College of Health and Welfare
S5 Female 20 Freshmen College of Public Health Science
S6 Male 20 Freshmen College of Public Health Science
S7 Male 20 Freshmen College of Health and Welfare
S8 Female 24 Senior College of Natural Sciences
S9 Female 20 Freshmen College of Public Health Science
S10 Female 20 Freshmen College of Public Health Science
S11 Female 20 Freshmen College of Health and Welfare
S12 Male 20 Freshmen College of Health and Welfare
S13 Female 20 Freshmen College of Health and Welfare
S14 Male 20 Freshmen College of Public Health Science
##### TABLE 2
Procedural Overview for This Study
Week Number Summary of Weekly Procedure
Week 1 Pre-Survey (to Collect Demographic Information & Prior Experiences)
Week 2 Lecture Video 1 (35 minutes with Video Watching Survey)
Week 3 Lecture Videos 2 & 3 (Video 2 = 38 minutes & Video 3 = 39 minutes with Video Watching Survey)
Week 4 Lecture Video 4 (44 minutes with Video Watching Survey)
Week 5 Interview Week 1 (Primary Week; 15 to 25 minutes long)
Week 6 Interview Week 2 (Make-up Week; 15 to 25 minutes long) & Follow-up Survey
##### TABLE 3
Class by Class Student Satisfaction With the Overall Video Experience and Subtitles
Class Video Video Satisfaction M SD Subtitle Satisfaction M SD
Class 1 5.643 1.336 4.429 1.95
Class 2 6 0.961 5.5 1.225
Class 3 6.214 0.975 6.071 0.73
Class 4 6.286 0.825 6.286 0.726
4 Class Total 6.036 0.289 5.5715 0.831
##### TABLE 4
Analysis of Perceived Understanding and Benefits of Subtitle for Each VL Section
Section of Video Student Perceived Understanding M SD How Much the Subtitles Helped M SD
Introduction 5.525 0.271 5.75 0.743
Assignment Contents 5.429 0.21 5.625 0.572
In-Class Activity Directions 5.464 0.289 5.625 0.661
In-Class Activity Contents 5.464 0.206 5.609 0.615
Assignment Directions 5.536 0.222 5.679 0.651
Wrap Up 5.643 0.303 5.643 0.692
Review 5.536 0.446 5.484 0.861
Greeting 5.768 0.179 5.589 0.846
Final Remarks 5.964 0.277 5.768 0.757
##### TABLE 5
Mean and Standard Deviation of Student Perceived Subtitle Accuracy
Class Video Percentage of Subtitle Accuracy in the Video M SD
Class 1 65.714 0.238
Class 2 80.643 0.137
Class 3 87.714 0.115
Class 4 83.857 0.147
4 Class Total 79.482 9.623

### REFERENCES

Bailey, DR, & Lee, AR (2020). Learning from experience in the midst of COVID-19: Benefits, challenges, and strategies in online teaching. Computer-Assisted Language Learning Electronic Journal. 21(2), 178-198. https://www.researchgate.net/publication/343219357_Learning_from_experience_in_the_midst_of_COVID19_Benefits_challenges_and_strategies_in_online_teaching.
Basak, SK, Wotto, M, & Bélanger, P (2016). A framework on the critical success factors of e-learning implementation in higher education: A review of the literature. Int J Educ Pedagog Sci, 10(7), 2409-2414. https://doi.org/10.5281/zenodo.1125677.

Basri, M, Husain, B, & Modayama, W (2021). University students’ perceptions in implementing asynchronous learning during COVID-19 era. Metathesis. Journal of English Language, Literature, and Teaching, 4(3), 263-276. https://doi.org/10.31002/metathesis.v4i3.2734.

Chan, WS, Kruger, JL, & Doherty, S (2019). Comparing the impact of automatically generated and corrected subtitles on cognitive load and learning in a first-and second-language educational context. Linguistica Antverpiensia, New Series - Themes in Translation Studies, 18https://doi.org/10.52034/lanstts.v18i0.506.

Che, X, Luo, S, Yang, H, & Meinel, C (2017). July;Automatic lecture subtitle generation and how it helps. In: Paper presented at the 2017 IEEE 17th International Conference on Advanced Learning Technologies; Timisoara, Romania. https://doi.org/10.1109/ICALT.2017.11.

Chun, SY (2014). EFL learners’ beliefs about native and non-native English-speaking teachers: Perceived strengths, weaknesses, and preferences. Journal of Multilingual and Multicultural Development, 35(6), 563-579. https://doi.org/10.1080/01434632.2014.889141.

Chun, SY (2015). Native English-speaking teachers and Korean English teachers: A qualitative analysis of Korean university students’ beliefs. Studies in Foreign Language Education, 29(3), 181-203. https://doi.org/10.16933/sfle.2015.29.3.181.

Chung, J, & Choi, T (2016). English education policies in South Korea: Planned and enacted. In R Kirkpatrick (Ed.), English language education policy in Asia. Language policy. (11), pp 286-291. Cham, Switzerland:Springer: http://doi.org/10.1007/978-3-319-22464-0_13.

Cierniak, G, Scheiter, K, & Gerjets, P (2009). Explaining the split-attention effect: Is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? Computers in Human Behavior, 25, 315-324. https://doi.org/10.1016/j.chb.2008.12.020.

Costley, J, Fanguy, M, Lange, C, & Baldwin, M (2021). The effects of video lecture viewing strategies on cognitive load. Journal of Computing in Higher Education, 33(1), 19-38. https://doi.org/10.1007/s12528-020-09254-y.

Famularsih, S (2020). Students’ experiences in using online learning applications due to COVID-19 in English classroom. Studies in Learning and Teaching, 1(2), 112-121. https://doi.org/10.46627/silet.v1i2.40.

Ferdiansyah, V, & Nakagawa, S (2013). Effect of captioning lecture videos for learning in foreign language. IPSJ SIG Technical Report, 2013-SLP-97. (13), 1-7. Retrieved from http://id.nii.ac.jp/1001/00094535/.
Fantinuoli, C, & Prandi, B (2021). Towards the evaluation of automatic simultaneous speech translation from a communicative perspective. In In : M Federico & A Waibel & MR Costa-jussà & J Niehues & S Stuker & E Salesky (Eds.), In: Proceedings of the 18th International Conference on Spoken Language Translatio; pp 245-254. Bangkok, Thailand:Association for Computational Linguistics: https://doi.org/10.18653/v1/2021.iwslt-1.29.

Irvin, C, & Park, SM (2021). An investigation of Korean college students’ perceptions of interactions during remote courses caused by COVID-19. English Language & Literature Teaching, 27(1), 67-93. https://doi.org/10.35828/etak.2021.27.1.67.

Johnson, B, & Christensen, LB (2004). Educational research: Quantitative, qualitative, and mixed approaches. (2nd ed). New York, NY:Pearson Education: Retrieved fromhttps://dspace.agu.edu.vn:8080/handle/AGU_Library/9383.

Kwon, O (2000). Korea’s English education policy changes in the 1990s. English Teaching. 55(1), 47-91. Retrieved from http://www.dbpia.co.kr/journal/articleDetail?nodeId=NODE07096239.
Lange, C, & Costley, J (2020). Improving online video lectures: Learning challenges created by media. International Journal of Educational Technology in Higher Education, 17, 1-18. https://doi.org/10.1186/s41239-020-00190-6.

Lasagabaster, D, & Sierra, JM (2002). University Students’ perceptions of native and non-native speaker teachers of English. Language Awareness, 11(2), 132-142. https://doi.org/10.1080/09658410208667051.

Mahyoob, M (2020). Challenges of e-Learning during the COVID-19 pandemic experienced by EFL learners. Arab World English Journal (AWEJ), 11(4), https://doi.org/10.24093/awej/vol11no4.23.

Mayer, RE, & Moreno, R (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43-52. https://doi.org/10.1207/S15326985EP3801_6.

Maykut, P, & Morehouse, R (1994). Beginning qualitative research: A philosophic and practical guide. New York, NY:The Falmer Press: https://doi.org/10.4324/9780203485781.

Medgyes, P (1992). Native or non-native: Who’s worth more? ELT Journal, 46(4), 340-349. https://doi.org/10.1093/elt/46.4.340.

Merriam, SB (1998). Qualitative research and case study applications in education. San Francisco, CA:Jossey-Bass.
Mishra, L, Gupta, T, & Shree, A (2020). Online teaching-learning in higher education during lockdown period of COVID-19 pandemic. International Journal of Educational Research Open, 1, 100012https://doi.org/10.1016/j.ijedro.2020.100012.

Nartiningrum, N, & Nugroho, A (2020). Online learning amidst global pandemic: EFL students’ challenges, suggestions, and needed materials. English Franca: Academic Journal of English Language and Education, 4(2), 115-140. https://doi.org/10.29240/ef.v4i2.1494.

Obach, M, Aruti, A, & Lehr, M (2007). Automatic speech recognition for live TV subtitling for hearing-impaired people. In G Eizmendi & JM Azkoitia & G Craddock (Eds.), Challenges for Assistive Technology, AAATE 07. pp 286-291. Amsterdam, The Netherlands:IOS Press.
Octaberlina, LR, & Muslimin, AI (2020). EFL students’ perspective towards online learning barriers and alternatives using Moodle/Google classroom during COVID-19 pandemic. International Journal of Higher Education, 9(6), 1-9. https://doi.org/10.5430/ijhe.v9n6p1.

Ortega, A, Garcia, JE, Miguel, A, & Lleida, E (2009). Real-time live broadcast news subtitling system for Spanish. In: Proc Interspeech; 2009; pp 2095-2098. https://doi.org/10.21437/Interspeech.2009-600.

Park, SM, & Irvin, C (2019). University students’ perceptions of native and non-native English-speaking teachers in general English education. Studies in Linguistics, 52, 253-283. https://doi.org/10.17002/sil..52.201907.253.

Phillipson, R (1992). ELT: The native speaker’s burden. ELT Journal, 46(1), 12-18. https://doi.org/10.1093/elt/46.1.12.

Scagnoli, NI, Choo, J, & Tian, J (2019). Students’ insights on the use of video lectures in online classes. British Journal of Educational Technology, 50(1), 399-414. https://doi.org/10.1111/bjet.12572.

Spring, R, & Tabuchi, R (2021). Assessing the practicality of using an automatic speech recognition tool to teach English pronunciation online. Journal of English Teaching through Movies and Media, 22(2), 93-104. https://doi.org/10.16875/stem.2021.22.2.93.

Sweller, J (2020). Cognitive load theory and educational technology. Educational Technology Research and Development, 68(1), 1-16. https://doi.org/10.1007/s11423-019-09701-3.

Walkinshaw, I, & Oanh, DH (2014). Native and non-native English language teachers: Student perceptions in Vietnam and Japan. Sage Journals, 1-9. https://doi.org/10.1177/2158244014534451.

Zhu, Z (2021). A discussion of foreign language listening problems and their causes among intermediate EFL learners in Chinese universities. Journal of Contemporary Educational Research, 5(8), 47-59. https://doi.org/10.26689/jcer.v5i8.2421.

### List of Questions for the Interview

• 2. Overall, how would you say that your first semester at university has been?

• 3. Could you please tell me what it has been like to have online classes this semester?

• 5. As you know, for classes 17 through 20, Korean AI subtitles were used. Could you describe your first reaction when you saw that the videos had Korean subtitles?

• 6. Could you share some of your specific thoughts about those class videos that used the Korean AI subtitles compared to the ones without subtitles?

• 7. I am curious, did the way that you watch the video change after the Korean AI subtitles were added?

• 8. Could you explain your thoughts about how you believe subtitles can affect English learning?

• 9. Can you tell me your thinking about how the Korean AI subtitles can be improved?

• 10. Finally, overall, can you describe your overall learning experiences during classes 17 through 20?

TOOLS
METRICS
• 0 Crossref
•  0 Scopus
• 169 View