Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'
Researchers created REPOCOD, a new benchmark featuring complex, real-world coding tasks requiring repository-level context from 11 popular projects. Testing ten LLMs showed none achieved more than 30% pass@1 on REPOCOD. This suggests current language models are insufficient for replacing programmers in real-world software development.