StarCoder: join the resistance!

A language model for programming, with open weights and a responsible-use licence

StarCoder

StarCoder is a large language model (LLM) developed by the BigCode community and released in May 2023. Designed to work with code, it offers an open-weight option alongside services such as GitHub Copilot, or ChatGPT itself.

The goal is to improve developer efficiency and productivity by automatically generating code and fixing syntax errors, among other tasks. But before continuing, let's see who are the people behind this new model, what Code LLM means and what StarCoder brings us.

BigCode: democratizing access to advanced programming tools.

BigCode is an open scientific collaboration developing language models for code. It aims to make programming more accessible and efficient. It is jointly led by HuggingFace and ServiceNow, with participation from the research community.

StarCoder is the result of this collaboration. Another fruit is The Stack, a large code dataset with permissive licences. BigCode also investigates responsible model development and ways to reduce the risks of using these systems.

Features of “The Stack”

Code LLM: AI technology for programming code generation

Large language models (LLMs) learn patterns from large volumes of text and can generate new sequences that fit a context. When trained specifically on code, they are often called code language models, or Code LLMs. The ability to produce a plausible response does not guarantee correctness.

Code models use training procedures similar to those of other LLMs, but incorporate large quantities of programs, documentation and related text. This allows them to learn syntax and structural patterns across programming languages.

As a result, they are able to understand the programming language and learn to write code from the patterns and code structure in the training.

This ability to generate code can help complete snippets, suggest syntax fixes and propose solutions within a development environment. By reducing some repetitive work, it can leave more time for complex and creative tasks. Generated code still needs review and testing.

StarCoder: the open-source promise of code LLMs

BigCode introduced two related models: StarCoderBase and StarCoder. Both can complete code and support batch inference; StarCoder adds Python-specific training.

How were they trained? StarCoderBase was trained on one trillion tokens from a selected subset of The Stack. Training then continued on 35 billion Python tokens to produce StarCoder, aiming to improve performance in that language.

StarCoder Conversational Chat Interface

The 2023 release evaluations placed StarCoder among the leading openly available code models. These results applied to specific benchmarks and did not establish general superiority over all closed services, such as OpenAI GPT3.5 , or GitHub Copilot.

The team introduced measures to reduce privacy risks and improve data traceability. These do not eliminate the possibility of generating incorrect, insecure or offensive content.

BigCode released the weights under the BigCode OpenRAIL-M licence, which allows commercial use subject to its terms and restrictions.

The StarCoder license allows developers and companies to use these models in their own tools and applications, opening up a world of possibilities for AI-assisted code generation, a potential “cambrian explosion” of applications based on LLMs similar to that produced with Stable Diffusion a few months ago.

For more details on how StarCoder was trained and tuned, as well as an in-depth discussion of its performance results, see the paper “StarCoder: may the source be with you!”.

Can StarCoder’s open alliance challenge the empire of closed models?

StarCoder’s release evaluations compared models using two programming benchmarks: HumanEval and MBPP.

HumanEval contains programming problems written by people, but assesses generated code through automated tests. It checks whether functions produce the expected results; it is not a human panel scoring the readability of each answer.

MBPP stands for Mostly Basic Programming Problems. It contains Python programming tasks described in natural language, with tests for checking solutions. It is not a pairwise preference comparison between models.

LLM Ranking

In the table, StarCoder scores 33.6 on HumanEval and 52.7 on MBPP, while StarCoder-Prompted scores 40.8 and 49.5 respectively. The prompt change improves one result and lowers the other, so each task should be compared separately.

The table also includes CodeGen, CodeGeeX, LLaMA, PaLM, LaMDA and a version of Codex. Their positions vary by benchmark: CodeGen-16B-Multi and CodeGeeX do not rank third and fourth. Comparing models requires considering the task, version and evaluation conditions as well as the score.

StarCoder-Prompted is not a separate model: it is StarCoder with a task-specific prompt. In this table, it improves HumanEval from 33.6 to 40.8, but lowers MBPP from 52.7 to 49.5. This illustrates how much prompting can matter and why it is unwise to declare a universal winner.

So, Can StarCoder challenge the hegemony of OpenAI and its best language models? In the 2023 context, the answer was NO, BUT the results showed progress: on several benchmarks, StarCoder outperformed code-cushman-001, an older version of Codex, the OpenAI code model evaluated at that time.

Join the rebel alliance of LLMs with StarCoder

Beyond Programming: Natural Language

Although StarCoder was designed primarily for code, the research also explored natural language tasks such as reasoning and text comprehension. This is an interesting direction: code training can support capabilities beyond programming, without automatically making a model the best choice for every task.

Reasoning evaluations compared approaches such as PAL (Program-aided Language Models), which combines program generation with execution, and CoT (Chain-of-Thought), which uses intermediate steps. Knowledge and comprehension tasks were also studied with MMLU (Massive Multitask Language Understanding) and CoQA (Conversational Question Answering).

The results of these evaluations show that StarCoder is able to perform effectively on these natural language tasks, although its performance may be slightly lower than other larger specific models.

StarCoder has demonstrated its versatility beyond code generation, it performs well as a language model for general tasks

Ethics and Safety: A Firm Commitment

BigCode documented measures including personal-information filtering, a process for opting code out of the dataset and attribution tools. These are steps towards responsible development, not a certification that the model is safe or free of bias. LLMs can generate false information, reproduce prejudices or suggest vulnerable code, making it important to study their limitations and review their outputs.

Practical Applications: Improving the Lives of Developers and Users

StarCoder is not just a theoretical artificial intelligence tool, but a practical application that can have a significant impact on the daily lives of software developers. Its ability to understand, generate and correct code can simplify and automate a number of tasks, improving efficiency and productivity. Among them:

Automating code writing. For example, a developer can provide a high-level description of the functionality they want, and StarCoder can automatically generate useful code to implement that functionality. This can save a lot of time and effort, especially on repetitive or standard tasks, as well as avoid “blank sheet syndrome”.
Bug fixes and QA. To identify and correct syntax and semantic errors in the code. One of the most important, but sometimes tedious and thankless, tasks is code correction and QA, with StarCoder this becomes much more dynamic, helping to analyze existing code, suggesting corrections and improvements, and helping developers avoid common errors and improve the quality of their code.
Facilitate the learning of new programming languages. For developers learning a new programming language, StarCoder can be a very useful learning tool. By providing examples of what the code looks like in that language, you can help developers more quickly become familiar with the syntax and conventions of the new language.
Refactoring and improvement of existing code. By analyzing the code and suggesting improvements, it helps keep the code clean, efficient and easy to understand, and can even generate documentation and comments for the code.

In short, StarCoder has the potential to be a valuable tool for developers, automating and simplifying a series of tasks and helping to improve efficiency and code quality.

Example of how to integrate “StarCoder” into your applications through HF Transformers

StarCoder’s strengths and challenges

Strengths

It is a code language model with available weights, trained on more than 80 programming languages.
It is capable of processing inputs with a context length of more than 8,000 tokens (the number of words depends on the language and content and may be less in Spanish). This window supports longer snippets, but does not by itself determine the model’s capabilities.
The weights and training code are available under their respective licences, allowing the model to be studied and adapted.
The project incorporates responsible-development measures, although privacy, bias and inappropriate-content risks remain.
The researchers behind StarCoder have also provided attribution tools to help users identify code generated by the model and ensure compliance with applicable licensing requirements.

Challenges

As with any natural language model, StarCoder can generate inaccurate results or unreliable, especially if used in areas where programming is very specialized or technical.
StarCoder may generate offensive, discriminatory or misleading content. Measures have been taken to prevent this, but there is always a risk that the model could produce unethical results.
Although StarCoder’s weights are available, its licence imposes use restrictions that should be reviewed before integration into an application.
The technology behind StarCoder is still in a early development phase, meaning there are still many technical challenges that need to be addressed before it becomes a widely used tool.
Results from English-language tests do not guarantee equivalent performance in other languages. Our initial impression in Spanish was positive, but that is no substitute for systematic evaluation.

Conclusion

Artificial intelligence and machine learning continue to change software development. StarCoder, released by BigCode in 2023, was a significant step towards wider access to models that work with code.

As a large-scale programming language model (LLM), StarCoder opens up a world of possibilities. Its ability to automatically generate code, correct errors, and facilitate the learning of new languages can greatly improve productivity and efficiency.

However, like any emerging technology, StarCoder is not without its challenges. The generation of inaccurate, offensive or discriminatory content is an inherent risk in any language model, and although measures have been taken to mitigate these risks, it is crucial to continue working on the continuous improvement of these aspects.

In 2023, StarCoder illustrated the potential of models with available weights for AI-assisted programming. Its community, documentation and tools make it possible to study and adapt the system. The challenge remains to improve its usefulness, evaluate its limitations and review the code it generates.

What are you waiting for to try it?

Article written by Jesús Sánchez and Miguel Guerrero.

Guns don't win battles. Your mind and a good LLM, a powerful combination it is.

Annex

Summary of StarCoder Features:

15.5-billion-parameter LLM pretrained on one trillion tokens
8,192-token context window
BigCode OpenRAIL-M licence: commercial use subject to conditions
Assistance with programming and some general tasks
Know more than 80 programming languages
Open source and data
Has Plugin in VSCode

Links to know more:

Project https://hf.co/bigcode
Blog https://huggingface.co/blog/starcoder
Generate code https://huggingface.co/spaces/bigcode/bigcode-editor
Play with StarCoder: https://huggingface.co/spaces/bigcode/bigcode-playground
Try the reasoning: https://huggingface.co/spaces/bigcode/Reasoning-with-StarCoder
Explore the data set: https://huggingface.co/spaces/bigcode/search

Basic quantities to develop LLMs:

Magnitudes to take into account when developing LLMs