Feedback is crucial for encouraging learning (Anderson, 1982) and its significance has been recognized in L2 writing. Among different types of feedback, error correction is the most frequently given feedback in the L2 writing process. According to Leki (1991), students believe that grammar feedback provided to them by teachers helps them improve their writing quality. While acknowledging the importance of feedback in effective L2 writing (Hyland & Hyland, 2006), research has spelt out that feedback on writing is a time-consuming job and that it can be problematic for teachers to provide feedback to students on a regular basis (Grimes & Warschauer, 2010). For L2 writing teachers, providing comprehensive corrective feedback, which is usually complex and time consuming, is inevitable. With the advent of technology, AI writing tools have provided a possible solution to alleviate teachers’ workload and time constraints and facilitate students’ writing. Current learners, characterized as digital natives, tend to prefer instant corrective feedback (O’Neill & Russell, 2022); therefore, teachers have difficulty allocating time effectively and prioritizing feedback.

The use of AI writing software has received extensive attention in L2 research, particularly when it comes to the grammatical and lexical accuracy of L2 writing. Research has provided empirical evidence on the advantages of using AI tools for L2 writing. Automated writing evaluation (AWE) has been studied for years as a way to provide timely evaluation of student writing and diminish the burden on educators to evaluate writing (Wilson & Roscoe, 2020). These systems typically use natural language processing and artificial intelligence to evaluate writing. Some studies have shown such systems can produce positive effects (Roscoe et al., 2017; Stevenson & Phakiti, 2014; Wilson & MacArthur, 2024; Zhai & Ma, 2022).

Studies have reported that using AI writing tools leads to more accurate foreign language composition compared to instances where they are not used, primarily through the reduction of grammatical errors and lexical errors. Wang et al. (2013) looked at the use of AWE software with college EFL students and examined its effect on writing accuracy. The research results reveal a significant difference between the experimental group and the control group in terms of writing accuracy following the adoption of AWE. The experimental group who used the software significantly reduce the number of grammatical errors in their writing from the pre- to the post-test, and they also outperformed the control group when it came to grammatical accuracy post-treatment. Regarding the overall effect and the exploration of students' perceptions toward their usage of the AWE software, it shows that students who used AWE display obvious writing enhancement in terms of writing accuracy and learner autonomy awareness.

Li et al. (2015) looked at the use of AWE among ESL students at an American university and found that use of the system led to a significant decrease in the total number of errors (grammar, usage, mechanics, and style) from the rough to the final draft. The participants had a favorable opinion of using criterion in the writing process. In addition, they believed that its use improved the linguistic accuracy of their compositions.

In a similar study, Ranalli et al. (2017) focused on the accuracy of the feedback given by a popular AWE system as well as the ability of American college ESL students to make use of this feedback to correct errors. The researchers found that while there were issues related to the accuracy of the feedback given, the students were able to correct their mistakes according to the feedback by the program 55-65% of the time.

In a 2017 study by Cheng, the paper examined the use of automated feedback among 138 college EFL students in Hong Kong. The participants were divided into two groups: experimental (n = 82) and control (n = 56). The experimental group could access a web-based classification system to generate online automated feedback on their second and third reflective journals while the control group could not. Results of the study indicate that the online automated feedback group significantly outperformed the control group, thus demonstrating the positive impact that the software can have on overall writing quality. While data collected from an online questionnaire survey and focus group interviews generally support the value of online automated feedback for writing. In another study, Liao (2016) examined how AWE could improve grammatical performance among 63 EFL writers. The study investigated participants’ grammatical performance in revised and subsequent new essays. The results of the study revealed that the software did help improve grammatical accuracy. More specifically, even though improvements were seen in each revision, significant improvements to new text were not found until the third draft of the essay. In recent study, Rahman et al. (2022) examined the role of an AI-assisted language learning tool in identifying and addressing grammatical errors, leading to the development of writing skills among EFL learners. The results indicated significant improvement in the EFL learners’ writing skills, and the learners themselves expressed positive perceptions regarding the effects of AI-assisted language learning on their writing abilities. Gayed et al. (2022) developed an AI-assisted language learning program to address cognitive barriers and improve the writing performance of EFL students. The findings indicated that the AI-powered language learning tool effectively enhanced students’ writing performance and reduced cognitive barriers encountered during writing tasks.

While some AWE systems are as efficient and reliable as human raters when assigning scores to writing, they are typically less accurate, more generic and verbose, and sometimes confusing to feedback recipients (Grimes & Warschauer, 2010; Wilson & MacArthur, 2024). Additionally, preparing such tools for educational settings takes substantial time (Moore & MacArthur, 2016). Research has also identified challenges associated with accuracy of the program. Bai and Hu (2017) investigated the accuracy and precision of AWE software among 30 undergraduate learners in EFL writing classrooms. They revealed that learners were not blindly accepting AWE feedback, but making more cautious decisions of feedback uptake. They claimed that automated feedback could be a supplement of peer and teacher feedback in the EFL writing classrooms instead of a replacement, and called for future studies to examine how learners made decisions of incorporating or rejecting a suggestion. In a comparative study between online peer feedback and AWE software, Shang (2019) found that the former had a greater positive effect on the grammatical accuracy of Taiwanese L2 English writers. This finding demonstrates that human feedback, whether from a teacher or a peer, can still prove to be useful or perhaps more beneficial than automated feedback.

While ChatGPT can be useful as an L2 writing assistant, adopting it as a supplemental tool for writing essays is highly advisable rather than relying on it as a content creator. A helpful approach is to encourage students to write their original outputs first and then refine them using ChatGPT. This approach helps students develop their writing skills while using ChatGPT to improve their written outputs further (Barrot, 2023). Research related to ChatGPT has been investigated recently (Mizumoto & Eguchi, 2023; Rudolph et al., 2023; Wang & Guo, 2023). Mahapatra (2024) investigated the impact of ChatGPT as a formative feedback tool on the writing skills of undergraduate ESL students. Data were collected from tertiary level ESL students and the findings indicate a significant positive impact of ChatGPT on students’ academic writing skills, and students’ perceptions of the impact were also overwhelmingly positive. The paper suggest that ChatGPT can be a good feedback tool in large-size writing classes with proper student training. In another study by Dai et al. (2023), the study investigated the feasibility of using ChatGPT to provide students with feedback to help them learn better. The results show that ChatGPT is capable of providing feedback on the process of students completing the task, which benefits students developing learning skills. In addition, the study found that ChatGPT generated more detailed feedback that fluently and coherently summarizes learners’ performance than human instructors.

With the proliferation of AI-driven tools has become easier for students to obtain feedback on their writing. They have advanced automated writing evaluation and feedback in writing (Gayed et al., 2022). While the literature on AI writing tools as a feedback tool in the writing classroom is well-established and its positive impact on writing has been investigated empirically. The use of ChatGPT in writing classes is a relatively new area and requires further investigation (Barrot, 2023). Since artificial intelligence-driven automated writing evaluation tools positively impact students’ writing, ChatGPT, a generative artificial intelligence tool, can be expected to have a more substantial positive impact. However, very little empirical evidence regarding the impact of ChatGPT on writing is available. In addition, there seems to be a necessity to investigate the shifts in learner perceptions with recent considerable improvement in AI learning tool. The purpose of the present study is to find how writing with ChatGPT improves students’ writing skills and affective aspects by analyzing the effects of using the tool on writing products.

The participants in this study were 43 Korean college freshmen attending a university in Seoul, Korea. All participants were taking a mandated general English course focused on listening and speaking. Their English proficiency was classified as at the high intermediate level through the placement test. The test was TOEIC mock test and their scores were between 550 and 800. Their ages spanned from 19 to 21 years old, with all participants having studied English for a minimum of 7 years at the time of the study. They have known ChatGPT but never used ChatGPT to edit their English writing prior to participating in this study. One class, comprising 23 participants, utilized ChatGPT software to revise scripts and constituted the experimental group (EG), while the other class, also consisting of 20 participants, received instructions without ChatGPT and constituted the control group (CG). Out of 23 participants from experimental group, three scripts including inappropriate formats were excluded.

To ensure the initial equivalence of the two groups’ writing abilities, the researchers conducted an independent t-test to examine potential differences in pretest mean scores between them. The results revealed comparable mean scores between the control group (CG: M = 58.25, SD = 6.80) and the experimental group (EG: M = 57.20, SD = 6.78), with no statistically significant difference observed (F = 0.148, p = .70). This finding indicates that participants in both groups exhibited similar writing abilities at the pretest.

ChatGPT was introduced by OpenAI as a conversational artificial intelligence in November 2022. By May 2023, when experiments were conducted, it was essential to assess the extent to which learners were aware of and utilizing ChatGPT. Particularly, there was a need to explore its application in English writing and inquire about the existence of any AI tools used for conventional English composition.

A survey was conducted, and all students responded that they have used online writing tools for English writing. To the question of asking any AI writing tools they use frequently, the total of 40 participants primarily use Naver dictionary (41%), the translation tool Papago (41%) followed by Google Translate (4.5%), ChatGPT (4.5%), and others (9%; see Figure 1).

Areas of assistance includes checking English expressions (35%), followed by checking vocabulary (26%), grammar (22%), and other writing errors (17%; see Figure 2). In Larson-Guenette’s (2013) study, most students responded that they use online writing tools daily because of time efficiency, word search, confirmation, and reference. And participants are using AI tools for writing despite being aware of its shortcomings (Valijärvi & Tarsoly, 2019).

The instruments consisted of students’ pre-test/post-test presentation script writing and open-ended questions listed in a survey. For the pre-test, students write one paragraph writing on the topic learned in the first week. For the posttest, students write a presentation script on the given topic to prepare their oral presentation on week 7. The presentation script is composed of a three-paragraph writing. The first paragraph is a self-introduction, and two paragraphs are related to the given topics dealt in the textbook. The topics are “mother nature” and “on the move.” For the first topic, the students prepared their views on extinction of a species and what they can do to save endangered species. The second topic “on the move” is related to the immigration, and the participants were asked to pick one city for the future internship program and discuss the push and pull factors that influenced their choice.

The survey questions were utilized in exploring students’ learning experiences and acceptance of new technology towards ChatGPT. A questionnaire comprises 11 questions which included 10 items on a 5-point Likert scale and one open-ended question asking opinions of the advantages or disadvantages of using ChatGPT or individual planning for future usage.

Both experimental and control groups received instructions by the same instructor, utilized identical course materials and syllabus, and underwent the same examinations. The students were mandatory to take one speaking test throughout the semester. The participants were allocated 20 minutes in class to write a script expressing their perspectives on the given topic during both the pre- and post-tests. Both groups were instructed to post the presentation scripts on a school learning management system (LMS).

Participants in the experiment group were asked to proofread and revise their drafts individually using ChatGPT. They were asked to upload their second drafts followed by checking their drafts on the ChatGPT site (https://chat.openai.com) for the free version, GPT-3.5 (Generative Pretrained Transformer-3.5). Examples or prompts were shown during class time and the instructor illustrated their usage with a sample paragraph. The prompts include “revise my writing,” “polish,” or “correct my grammatical, lexical errors.” The participants in the EG submitted their second drafts after receiving feedback from the platform and revising their initial drafts. Conversely, those in the CG wrote their drafts and provided peer feedback during class. Peer feedback include both global (content and organization) and local (grammar and mechanics) aspects of writing. Finally, both groups submitted presentation scripts to the instructor for evaluation and grading. After they submitted their second draft, students in EG groups’ perceptions and attitudes toward using the tool were collected through LMS.

An independent t-test in SPSS version 25.0 was used to analyze the data collected from the pre- and post-test writing scores of the groups. Two experienced EFL raters, including the researcher, independently graded participants’ drafts to ensure objective scoring. The holistic scores, including content and organization on students’ presentation drafts were employed to evaluate the overall writing quality (Chang et al., 2021). The raters both possess doctoral degrees and have accumulated over a decade of teaching experience at the university level. To assess the consistency between the raters, an inter-rater reliability test was conducted on the post-test script scores. The inter-rater Cronbach’s alpha coefficient was calculated at 0.82, significant at p < .05, indicating a high level of agreement between the two raters. A total of 43 presentation scripts were collected from experimental group; however, three scripts with inappropriate formats were excluded from the analysis.

For form-oriented feedback, Grammarly (2022) was employed to analyze the participants’ script writing since the program makes it possible to analyze the drafts numerically and mechanically (Dizon & Gayed, 2021). This program identifies mistakes such as spelling, punctuation, and grammar errors in students’ drafts. Additionally, it furnishes details such as the overall word count, sentence count, and approximate time required for reading and speaking in English. Moreover, the tool calculates a readability score by considering factors like sentence and word length in students’ drafts, employing the Flesch reading-ease test (Flesch, 1948). The EG students’ learning experiences towards ChatGPT in survey responses were first analyzed and then categorized.

To answer the first research question, the participants’ writing scores were compared. Paired sample t-tests were conducted to investigate whether there were significant differences between the pre- and post-test scores.

The results of the descriptive statistics and paired sample t-tests are presented in Table 2. The results showed that there was no statistically significant difference in mean scores between the two tests for the control group (p = .15).

As for the EG group, on the other hand, the findings revealed statistically significant differences between the two tests indicating that the participants in the group improved their writing. Table 3 shows the results of paired t-tests of EG group. To be specific, mean scores in the pre-test showed 62.20 (SD = 11.52) while the scores in the post-test showed 78.25 (SD = 8.94). The result of the differences between the pre- and post-tests within each group suggests that using ChatGPT in revision can be effective for their writing.

The difference in post-test scores was verified by an independent sample t-test to analyze whether there was a significant difference in the post-test mean score between the two groups. The results revealed that a significant difference was found (t = -3.72, p = .00 < .05) between the EG (M = 78.75, SD = 8.97) and the CG (M = 65.00, SD = 13.86) group, indicating that students’ writing performance in the EG had greater gains in the posttest writing than the students in the CG did. Such a result suggests that revising the script using ChatGPT can assist participant in writing significantly than revising the draft from peer feedback. The quantitative results are presented in Table 4.

An independent t-test between the writing tasks was performed, comparing their scores in two different conditions in terms of word count, vocabulary and grammatical and lexical errors. Table 5 presents the result of descriptive statistics and independent t-tests. The mean scores of the number of words and sentences in each group are shown in the table below. In CG group, it turned out the participants’ revised drafts consisted of 385.55 words and 32.24 sentences on average whereas the participants’ revised drafts consisted of 358.65 words and 26.15 sentences on average in EG group. The number of words and sentences in EG group turned out to be decreased compared to the CG group. A significant mean difference was found in the number of sentences between the two versions (t = 6.57, p < .01). The p-value for word count was not significant. The presentation script using ChatGPT does not help learners generate more text through the revision process in this study. This assumption could be related to the nature of the text. Participants reviewed their writings before memorizing lines for presentation, which may lead them to refrain from generating additional text or incorporating unique or uncommon words.

The number of vocabulary “unique” and “rare” words was calculated using Grammarly. According to the definitions of Grammarly (2022), unique words are connected with how much the writer uses words in an unusual or different way, and those unique words do not fall into the category of the 200 most frequently used words. Rare words are indicator of more advanced language use. It was found that 51.95 unique words and 21.75 rare words were used in CG group, while 55.30 unique words and 22.40 rare words were found in the EG group. The number of unique and rare words is increased in the group using ChatGPT; however, a significant difference was not found between the two versions of the script.

The number of grammatical and lexical errors presented in their revised output is shown in Table 6. The mean number of errors in CG group is 14.60 for grammar and 3.70 for the lexical part. In EG group, on the other hand, the mean scores are 6.05 and 0.35 in each category respectively. Lexical errors encompass various types such as improper word choice, unclear sentences, wordiness, inconsistencies in the text, and the use of inappropriate colloquial expressions. On the other hand, grammatical errors are associated with punctuation usage, construction of compound or complex sentences, incorrect utilization of passive voice, semicolons, and quotation marks. The findings revealed statistically significant differences between the two drafts. It indicates that the group using ChatGPT for their writing revision made less errors grammatically and lexically (p < .01).

The results of this study are consistent with the results of previous studies in that the scores using AI tools for writing have significantly increased. Previous studies have found that using AI tools resulted in higher writing scores (O’Neill, 2016) and advantages in lexical choice (Chen et al., 2015). It was similar to previous findings that feedback improved the accuracy of written communication. However, learners using ChatGPT do not get helped to generate longer text or more unique and rare vocabulary through the revision process in the study.

To investigate the learners’ affective aspects of using ChatGPT for revision, the questionnaire responses to writing using ChatGPT was analyzed. The questionnaire consisted of 10 closed-items divided into three sections: usefulness, improvements in L2 writing skills, and affective variables. A total of 23 responses were collected from experimental group. Participants were asked to indicate their level of agreement with each statement on a scale ranging from strongly disagree (SD) to strongly agree (SA).

As Table 7 shows above, over 95 % of participants in EG showed positive sentiment to ChatGPT. More than 90% of respondents found ChatGPT easy to use and a significant majority (87%) found it very convenient. None of the respondents showed negative opinion on usefulness of using ChatGPT as an AI tool in general. This suggests that ChatGPT is perceived as a user-friendly and convenient tool by the majority of users.

Table 8 depicts the results of survey questions asking the effects of using ChatGPT on writing skill. Regarding the questions asking whether it is helpful in their English writing, the majority (95.7%) of the respondents found their experience with ChatGPT either helpful or very helpful in English writing. One out of 23 respondents responded that it is not helpful. Over 90% found ChatGPT helpful or very helpful for grammar learning. None of the participants disagree to this question. In the aspects of vocabulary, majority of the respondents (78.3%) acknowledged the tool is helpful to find appropriate or better vocabulary for their writing, four participants took a neutral attitude and one disagreed. A significant majority (91.4%) acknowledged the tool’s effectiveness in organizing sentence structure and content. This suggests that users find ChatGPT beneficial for refining the structure and coherence of their written content. Users perceive ChatGPT as a valuable tool for language learning, particularly in improving grammar skills. The result can be in line with the quantitative result that writing revision for the EG made less errors in grammar and lexical. It can be suggested that this tool can help students to enhance their writing and raise confidence in writing in their affective aspect.

Table 9 presents the distribution of responses for three affective variables. Most of the EG students thought that ChatGPT was a beneficial learning tool for enhancing English writing performance. To the question asking future interest in using the tool for English writing, over 78% expressed a high future interest in English writing, four students expressed a moderate future interest and one strongly disagreed. More than 80% responded that ChatGPT lowered their writing anxiety in English writing and felt that their fear in English writing would be eliminated. More than 10% of respondents took neutral, and one respondent disagreed. These responses suggest a positive outlook for continued engagement in English writing with reduced apprehension. The overall sentiment towards ChatGPT is highly positive, with users perceiving it as a valuable and effective tool for English writing and language learning. The tool’s ease of use, convenience, and positive impact on grammar, vocabulary, and content organization contribute to its positive reception.

Table 10 shows the result of an open-ended question, the responses from participants were grouped based on their resemblance to a single item. If participants provided three distinct opinions, each was considered and tallied individually. In an open-ended survey question, many respondents mentioned that the tool was very helpful and informative and convenient to use. In addition, many participants responses the benefits highlighting the advantages of using ChatGPT as a learning means of enhancing their writing abilities especially in grammar. They wrote similar responses such as “I appreciate the aspect of quickly correcting my sentences for grammatical accuracy,” “It is useful for me to know my grammatical mistakes (such as the singular and plural of words, changing conjunctions and adjectives),” “I felt the responses were enhanced with more advanced vocabulary.” Almost 20% of responses showed that they considered the tool very helpful in connecting and structuring sentences and this is consistent to the quantitative result that the number of sentences in revised version of EG turned out to be decreased compared to the CG. The students also stated that it provides features that give users advanced alternative word choices, and expressions. One respondent reported that the tool helps partially since it suggested expressions and vocabulary in academic way that she should change it into her own word. Due to the necessity for participants to ready themselves for the oral examination and commit the script to memory, they refrained from promptly implementing the tool’s recommendations. As the participants were working on their presentation scripts, they sought simpler words rather than opting for unique or uncommon vocabulary, even though the suggested words appeared to be more advanced in language. Additionally, some mention the tool helps their alleviation of writing fear or uncertainty with English writing.

However, students pointed out the disadvantages of using ChatGPT in their presentation script writing (see Table 11). Students listed several negative comments about it. Some participants reported that ChatGPT sometimes gave wrong answers to their writing drafts. They disapproved of the comments given by ChatGPT. For instance, “There were occasional cases where it completely changed the meaning of the text, so there was a need for modification,” “It was very helpful, but the corrected sentences ended up being longer than the original ones, making them sound a bit unnatural.” There are also concerns about reliability and over-reliance of ChatGPT. One participant mentioned that relying on ChatGPT can be problematic since it never states impossibilities. Another participant reported that “There is some doubt about its accuracy. It seems a promising program with increased confidence and reliability.” Another significant issue they considered is about their effectiveness of future language learning. For instance, “It’s advisable to use it with a sound knowledge of grammar and vocabulary” and “If misused, it may not contribute significantly to the actual improvement of English proficiency.” The result is consistent with previous studies that foreign language learners tend to distrust the results of the translation tool (Briggs, 2018), perceiving it as not beneficial for foreign language learning (Yoon, 2019).