Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models
cs.SE
Submitted: 2023-12-09
Updated: 2026-10-28
Comments: ACCEPT IN TSE
Code: https://github.com/facebookresearch/codellama
Project page: https://csyuzhou.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) have demonstrated remarkable potential in code generation.
Terminology
Abstract
Large Language Models (LLMs) have demonstrated remarkable potential in code generation. The integration of Chain of Thought (CoT) reasoning can further boost their performance. However, current CoT methods often require manual writing or LLMs with over 100 billion parameters to generate, impeding their applicability in resource-constrained scenarios. In this study, we investigate lightweight Language Models (lLMs), which are defined to have fewer than 10 billion parameters. Empirically, we find that most lLMs cannot generate high-quality CoTs when prompted by the few-shot method, but can take advantage of high-quality CoTs generated elsewhere to improve their performance in code generation. Based on these findings, we design a novel approach COTTON which can leverage lLMs to automatically generate CoTs for code generation. We synthesize new datasets and conduct extensive experiments on various benchmarks. The results show that the CoTs generated by COTTON outperform the baselines in terms of automated and human evaluation metrics. In particular, the CoTs generated by COTTON boost various lLMs to achieve higher performance gains than those generated by LLMs such as ChatGLM (130B), and are competitive with those generated by gpt-3.5-turbo (175B). Our study also showcases the potential of lLMs in software engineering applications.
Sources
- AI-assisted coding: Experiments with GPT-4
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
- Challenges and Applications of Large Language Models
- Specializing Smaller Language Models towards Multi-Step Reasoning
- Improving ChatGPT Prompt for Code Generation
- A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair
- ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design
- Self-planning Code Generation with Large Language Models
- ICE-Score: Instructing Large Language Models to Evaluate Code
- Towards Reasoning in Large Language Models: A Survey
- Reasoning with Language Model Prompting: A Survey
- The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Evaluating Large Language Models Trained on Code
- Code Llama: Open Foundation Models for Code
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Large Language Models for Software Engineering: A Systematic Literature Review
- Textbooks Are All You Need
- A Survey on Large Language Model based Autonomous Agents
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters