NovelAPIBench: Diagnosing How A Code LLM Learns to Use Novel APIs

Name: NovelAPIBench: Diagnosing How A Code LLM Learns to Use Novel APIs
Uploaded: 2026-10-04T04:28:59.540664+00:00
Duration: 806 s
Description: NOVELAPIBENCH is a dynamic benchmark that automatically finds new APIs and breaks them down into components like signatures and usage examples. It tests how different learning methods, such as RAG or SFT, handle these novel APIs. The findings show that specific knowledge parts matter most, suggesting retrieval for volatile facts and tuning for procedural skills are complementary.

NOVELAPIBENCH is a dynamic benchmark that automatically finds new APIs and breaks them down into components like signatures and usage examples. It tests how different learning methods, such as RAG or SFT, handle these novel APIs. The findings show that specific knowledge parts matter most, suggesting retrieval for volatile facts and tuning for procedural skills are complementary.

Open the full episode