The Effectiveness of AI Chatbot-Based English Learning Methods in Improving Students’ Speaking Skills

Abstract

This mixed-methods study examined the effectiveness of AI chatbot-based English learning methods, specifically using TalkPal AI, in improving eleventh-grade students’ speaking skills at SMAN 11 Sidrap, Indonesia. A quasi-experimental nonequivalent control group design was employed for the quantitative phase, while semi-structured interviews provided qualitative insights into students’ perceptions. The experimental group (n = 18 for post-test) engaged in interactive speaking practice with TalkPal AI, while the control group (n = 14) received conventional teacher-centered instruction. Data were collected through pre- and post-speaking tests assessed using Brown’s (2004) rubric across five components fluency, accuracy, pronunciation, vocabulary, and coherence and interviews with ten purposively selected students from the experimental group. Quantitative analysis revealed that the experimental group achieved a significantly higher post-test mean score (M = 23.17) compared to the control group (M = 12.57). The Mann-Whitney U test confirmed a statistically significant difference between groups (p < 0.001), indicating the effectiveness of the AI chatbot intervention. Qualitative findings showed overwhelmingly positive perceptions: students reported increased confidence, motivation, fluency, pronunciation, and reduced speaking anxiety, attributing these gains to the chatbot’s interactive, non-judgmental, and flexible learning environment. Minor challenges included unstable internet connections and initial familiarity with the application. The results demonstrate that AI chatbot-based methods, such as TalkPal AI, constitute an effective and innovative approach for enhancing EFL learners’ speaking proficiency in secondary education contexts. Pedagogical implications for technology-integrated language teaching and recommendations for future research are discussed.