Researchers from Microsoft and Hong Kong Baptist University have made a significant advancement in Large Language Models (LLMs) tailored for coding. They’ve introduced “WizardCoder”, an evolved version of the open-source Code LLM, StarCoder, leveraging a unique code-specific instruction approach.
OpenAI’s ChatGPT and its ilk have previously demonstrated the transformative potential of LLMs across various tasks. Notably, Code LLMs, trained extensively on vast amounts of code data, have addressed challenges unique to coding activities. However, a gap existed in the domain-specific instruction tailoring of Code LLMs, which this recent effort aims to address.
Drawing inspiration from the Evol-Instruct methodology, the researchers’ focus was on enhancing StarCoder’s capabilities. Their approach incorporated modifications in the evolutionary prompt process, specifically tailored for coding activities. This included refining evolutionary prompts and instructions, as well as integrating code debugging and time-space complexity constraints.
After developing an enriched instruction-following dataset, the team fine-tuned StarCoder to unveil their crown jewel: WizardCoder. Experimental results are compelling. WizardCoder not only surpasses all open-source Code LLMs in four coding benchmarks but also outshines leading closed-source LLMs, including Anthropic’s Claude and Google’s Bard, in performance metrics on HumanEval and HumanEval+. This feat is even more impressive given WizardCoder’s smaller size.
