Open development of Large Language Models for code with BigCode and StarCoder2

Abstract: In the rapidly evolving landscape of software development, Large Language Models (LLMs) for code have emerged as groundbreaking tools for code completion, synthesis, and analysis. BigCode is an open scientific collaboration for the responsible development of code LLMs. In this talk, we will cover some of the foundational elements of BigCode, including open large-scale code datasets such as The Stack, data governance, and transparency standards, as well as our approach for training the competitive StarCoder and StarCoder2 models.

Slides: The keynote slides can be downloaded here.

Bio: Loubna Ben Allal is a Machine Learning Engineer in the Science team at Hugging Face working on Large Language Models for code & Synthetic data generation. She is part of the core team behind the BigCode Project and has co-authored The Stack dataset and StarCoder models for code generation. Loubna holds Mathematics & Deep Learning Master's Degrees from Ecole des Mines de Nancy and ENS Paris Saclay.

Code Llama: Open Foundation Models for Code

Abstract: Code Llama is a collection of base and instruct fine-tuned models with 7B, 13B, 34B and 70B parameters. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use. This talk will present the different types of Code Llama models, and show how they can be used in practice for research and applications.

Slides: The keynote slides can be downloaded here.

Bio: Baptiste is a research scientist at Meta AI in Paris working in the code generation team. He works on large language models, with a special interest in applications to code. Baptiste contributed to Llama and started Code Llama. Before that, he worked on model pre-training and machine translation for programming languages.