Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing

This study introduces a layered LLM-correction pipeline that isolates structural errors from unnaturalness by generating literal error corrections and idiomatic revisions for 3,830 English writing samples from 120 Japanese junior high school students. Applying the regex-based CEFR-J grammar extractor, we quantify two diagnostic measures: accuracy gaps (structures attempted but incorrectly produced) and idiomatic gaps (grammatically correct structures underused or overused relative to native norms). Results reveal distinct patterns: definite articles, third-person singular-s, and modals (would, could) exhibit significant accuracy difficulties, while-ing forms and hypothetical modals (would) show the largest idiomatic underuse, with simple present verbs, subject-verb-object patterns, and modal can conversely exhibiting the most pronounced overuse. A two-dimensional instructional typology maps error rates against idiomatic gaps, distinguishing accurate but overused grammar items from error-prone or avoided complex forms requiring targeted production practice. This framework advances pedagogical feedback by enabling teachers to diagnose whether learner difficulties arise from inaccurate execution, structural avoidance, or L1-mapped overreliance, supporting evidence-based interventions tailored to the specific needs of each learner.

Improvements to AI Systems:

1. Dual-Diagnostic Error Correction: Enhance LLM-based writing tutors to separate corrections into two distinct outputs—literal error fixes (grammar accuracy) and idiomatic rewrites (native-like naturalness)—instead of merging them. This allows the AI to explicitly flag whether a learner’s issue is structural (e.g., missing third-person-s) or stylistic (e.g., overusing simple present vs. -ing forms).

2. Personalized Gap Profiling: Implement a diagnostic module that computes per-learner accuracy gaps and idiomatic gaps (underuse/overuse) using the paper’s two-dimensional typology. The AI can then generate a profile for each student, categorizing grammar items into four quadrants: accurate+overused, accurate+underused, error-prone+overused, and error-prone+underused. This enables targeted practice recommendations (e.g., production drills for avoided hypothetical modals, reduction exercises for overused SVO patterns).

3. L1-Aware Feedback Generation: Use the paper’s finding that overuse often stems from L1 mapping (e.g., Japanese learners overusing “can” due to “dekiru”) to make the AI’s feedback culturally and linguistically contextualized. The system can detect when a learner’s error pattern aligns with known L1 interference and provide contrastive explanations (e.g., “In Japanese, ‘would’ is rarely used hypothetically; try using it in conditional sentences”).

4. Avoidance Detection and Scaffolding: Train the AI to recognize when a learner avoids complex structures (e.g., hypothetical modals, -ing forms) by analyzing their output for low frequency of these forms relative to native norms. The system can then proactively introduce these structures in incremental prompts, rather than waiting for errors to occur.

5. Regex-Based Grammar Extraction for Real-Time Analytics: Integrate a lightweight, regex-based grammar extractor (like the CEFR-J tool) into the AI’s backend to quantify error rates and idiomatic gaps on the fly. This allows the AI to generate real-time, evidence-based reports for teachers, showing which structures need classroom-wide intervention versus individual practice.

What the Improved AI System Can Do:

• Provide a split-screen correction view: “Accuracy Fixes” and “Idiomatic Revisions,” with explanations for each.

• Generate a visual 2x2 matrix for each learner, highlighting their unique strengths and weaknesses (e.g., “You use ‘can’ too often, but you avoid ‘would’—let’s practice hypotheticals”).

• Automatically adapt writing prompts to force production of underused structures (e.g., “Describe a future plan using ‘would’ and ‘-ing’ forms”).

• Offer L1-specific tips based on the learner’s native language, derived from the paper’s overuse/underuse patterns.

• Produce weekly analytics for teachers, summarizing class-level accuracy gaps and idiomatic gaps, with suggested lesson plans targeting the most problematic items (e.g., definite articles, third-person-s, hypothetical modals).