StarCoder: join the resistance!

An open, permissive and ethical language model for programming

StarCoder

StarCoder is a community-developed large language model (LLM). BigCode, which was launched in May 2023. This model has been designed to work with programming code and is presented as an open source alternative to tools such as GitHub Copilot, or ChatGPT itself.

The goal is to improve developer efficiency and productivity by automatically generating code and fixing syntax errors, among other tasks. But before continuing, let's see who are the people behind this new model, what Code LLM means and what StarCoder brings us.

BigCode: democratizing access to advanced programming tools.

BigCode is a collaboration of engineers and scientists that seeks to improve the way code is written and understood, making programming more accessible and efficient for everyone, by developing high-quality programming language models like StarCoder. Does it have the support of? HuggingFace and ServiceNow, although it is an open and totally community project.

StarCoder is the result of this collaboration. Another fruit is The Stack, the largest training data set available for programming. In addition, BigCode also cares about responsibility in the creation of these models, and works to ensure that they are ethical and fair in their use.

Features of “The Stack”

Code LLM: AI technology for programming code generation

Large language models (Large Language Models or LLMs) are artificial intelligence technologies that have revolutionized the way machines understand and generate text. By analyzing and learning patterns in large volumes of text, LLMs can generate new pieces of text that are surprisingly coherent and contextual. When these models are built to work specifically with programming code, it gives rise to what is known as Code LLMs.

The Code LLMs They are trained in the same way as normal LLMs, but instead of learning patterns in normal or natural text, that is, the language that people use, they learn those patterns using programming languages as a base. This means they analyze and learn from billions of lines of code written in different programming languages.

As a result, they are able to understand the programming language and learn to write code from the patterns and code structure in the training.

This ability to “learn” programr makes them powerful tools for multiple tasks, such as automatic code generation, spelling and syntax checking, and code suggestion in software development environments. This has the potential to significantly increase productivity and the efficiency of developers, since they can delegate repetitive or tedious tasks to these models and focus on more complex and creative tasks.

StarCoder: the open-source promise of code LLMs

The BigCode community has presented the Code LLM “StarCoder”, which is in turn divided into the StarCoder and StarCoderBase models. These models are not only capable of completing code snippets, but they can also perform fast inferences with large amounts of data.

How was it done? To train these models, BigCode used a curated set of one billion tokens sourced from GitHub. After this initial training, StarCoderBase was tuned specifically for Python. This fine-tuning process (fine-tuning) involved training the model on a specific Python data set, allowing the model to specialize in this language and improve its ability to generate and work with Python code.

StarCoder Conversational Chat Interface

The performance of these models has been extensively evaluated and found to be among the best open source language models. In particular, StarCoder and StarCoderBase demonstrated their superiority in terms of inference accuracy and speed, making them excellent alternatives to closed LLMs such as OpenAI GPT3.5 , or CoPilot from Microsoft.

In addition to being powerful, these models have taken ethics and security into account, taking measures to ensure that the training of the models does not violate data privacy, as well as avoiding the generation of offensive or inappropriate content, without being infallible.

BigCode has made these models available to the public under a license that allows their use in commercial applications.

The StarCoder license allows developers and companies to use these models in their own tools and applications, opening up a world of possibilities for AI-assisted code generation, a potential “cambrian explosion” of applications based on LLMs similar to that produced with Stable Diffusion a few months ago.

For more details on how StarCoder was trained and tuned, as well as an in-depth discussion of its performance results, see the paper “StarCoder: may the source be with you!”.

Can StarCoder and the rebel alliance (open-source) challenge the Empire (closed models)?

In the StarCoder evaluation, several code generation models are compared using two main metrics: HumanEval and MBPP.

HumanEval is a metric that evaluates the quality of code generated by models through human testing. In this test, human testers are presented with a set of programming problems and asked to evaluate the quality of the code generated by the models in terms of its accuracy, efficiency, and readability.

On the other hand, MBPP (Mean Binary Pairwise Preference) is a metric that uses a more statistical approach to evaluate the quality of the generated code. Instead of relying on human evaluation, MBPP compares the code generated by two different models for the same problem and calculates the probability that one is preferred over the other in terms of quality.

LLM Ranking

According to this table, StarCoder and StarCoder-Prompted are the models with the best overall performance, obtaining the best scores in HumanEval and also in MBPP. Specifically, StarCoder scored 33.6 on HumanEval and 52.7 on MBPP, while StarCoder-Prompted scored 40.8 and 49.5 on HumanEval and MBPP, respectively.

The other models also have significant success rates, with CodeGen-16B-Multi and CodeGeeX in third and fourth place in terms of performance. Although some models such as LLaMA-7B have lower hit rates, they still perform well considering they are not specifically designed for code generation.

Overall, StarCoder is presented as one of the best code generation models in terms of accuracy, surpassed only by StarCoder-Prompted, which uses additional user input information to further improve performance.

So, Can StarCoder challenge the hegemony of OpenAI and its best language models? The answer right now is NO, BUT the gap is being shortened significantly, and in many tasks, StarCoder performs better than Codex, the model for OpenAI code.

Join the rebel alliance of LLMs with StarCoder

Beyond Programming: Natural Language

A pleasant surprise has been to find that StarCoder, although designed primarily to work with programming languages, has also proven to be competent in a variety of natural language tasks, including reasoning, reading comprehension and text generation, In fact, the fact that models trained for code are better than those trained for text (natural language) is an emerging feature that is raising more than one eyebrow.

In the task of reasoning, the PAL methodology has been used (Premise-Answer-Likelihood) and CoT (Choice of Task), to compare its performance with that of other models. With regards to the reading comprehension, Its performance has been evaluated using the MMLU benchmarks (MultiModal Language Understanding) and CoQA (Conversational Question Answering).

The results of these evaluations show that StarCoder is able to perform effectively on these natural language tasks, although its performance may be slightly lower than other larger specific models. 

StarCoder has demonstrated its versatility beyond code generation, it performs well as a language model for general tasks

Ethics and Safety: A Firm Commitment

The BigCode team has put a real, and demonstrable, effort into making StarCoder as safe and ethical a model as possible. It is known that one of the most common problems with LLMs is the generation of inaccurate information or the amplification of existing prejudices. This can lead to the generation of offensive, misleading, or discriminatory content. To counteract these problems, alignment techniques have been used to ensure ethical behavior of the model and that harmful prejudices and stereotypes are not perpetuated or amplified, which is one of the main challenges with Generative AI. In section 7.3 of the paper, the strategies and measures taken to mitigate risks are detailed.

Practical Applications: Improving the Lives of Developers and Users

StarCoder is not just a theoretical artificial intelligence tool, but a practical application that can have a significant impact on the daily lives of software developers. Its ability to understand, generate and correct code can simplify and automate a number of tasks, improving efficiency and productivity. Among them:

• Automating code writing. For example, a developer can provide a high-level description of the functionality they want, and StarCoder can automatically generate useful code to implement that functionality. This can save a lot of time and effort, especially on repetitive or standard tasks, as well as avoid “blank sheet syndrome”.
• Bug fixes and QA. To identify and correct syntax and semantic errors in the code. One of the most important, but sometimes tedious and thankless, tasks is code correction and QA, with StarCoder this becomes much more dynamic, helping to analyze existing code, suggesting corrections and improvements, and helping developers avoid common errors and improve the quality of their code.
• Facilitate the learning of new programming languages. For developers learning a new programming language, StarCoder can be a very useful learning tool. By providing examples of what the code looks like in that language, you can help developers more quickly become familiar with the syntax and conventions of the new language.
• Refactoring and improvement of existing code. By analyzing the code and suggesting improvements, it helps keep the code clean, efficient and easy to understand, and can even generate documentation and comments for the code.

In short, StarCoder has the potential to be a valuable tool for developers, automating and simplifying a series of tasks and helping to improve efficiency and code quality.

Example of how to integrate “StarCoder” into your applications through HF Transformers

Summary of Strengths and Challenges of ?StarCoder

Strengths

• It is a programming language model large-scale open source, trained in more than 80 programming languages.
• It is capable of processing inputs with a context length of more than 8,000 tokens (about 6,000 words in English, less in Spanish), making it one of the largest and most powerful models available.
• It is a tool of open source, trained with permissive licenses, allowing the community of programmers to contribute to its development and improvement.
• StarCoder has been ethically trained, meaning that steps have been taken to prevent discrimination and other inappropriate behavior in code generation.
• The researchers behind StarCoder have also provided attribution tools to help users identify code generated by the model and ensure compliance with applicable licensing requirements.

Challenges

• As with any natural language model, StarCoder can generate inaccurate results or unreliable, especially if used in areas where programming is very specialized or technical.
• StarCoder may generate offensive, discriminatory or misleading content. Measures have been taken to prevent this, but there is always a risk that the model could produce unethical results.
• Although StarCoder is an open source tool, it still has usage restrictions that may limit its accessibility to some developers.
• The technology behind StarCoder is still in a early development phase, meaning there are still many technical challenges that need to be addressed before it becomes a widely used tool.
• Although the model has been evaluated in English, it is still unclear how well it will work in other languages and how the results will translate in different cultural contexts, in Spanish it seems to be working quite well.

Conclusion

Artificial intelligence and machine learning (machine learning) continue to transform various aspects of our lives, and software development is no exception. StarCoder, presented by BigCode, is a representation of this progress and a huge step forward towards the democratization of software development.

As a large-scale programming language model (LLM), StarCoder opens up a world of possibilities. Its ability to automatically generate code, correct errors, and facilitate the learning of new languages ​​can greatly improve productivity and efficiency.

However, like any emerging technology, StarCoder is not without its challenges. The generation of inaccurate, offensive or discriminatory content is an inherent risk in any language model, and although measures have been taken to mitigate these risks, it is crucial to continue working on the continuous improvement of these aspects.

Overall, StarCoder represents a significant leap forward a new era of AI-assisted programming, and automation with natural language. While there is still work to be done, the promise of this technology is undoubted. With its commitment to ethics, security and open development, StarCoder It is positioned as one of the best alternatives to the closed and black box models currently available, also counting on the support of a community committed to the continuous improvement of the models and their positive impact on our society.

What are you waiting for to try it?

Article written by Jesus Sanchez and Miguel Guerrero.

Guns don't win battles. Your mind and a good LLM, a powerful combination it is.

Annex

Summary of StarCoder Features:

• LLM of 15Bn parameters and 1Tn tokens
• 8k context tokens (one chapter long of game of thrones)
• Permissive license allowing commercial use
• Scheduling and general task assistant
• Know more than 80 programming languages
• Open source and data
• Has Plugin in VSCode

Links to know more:

• Project https://hf.co/bigcode
• blog https://huggingface.co/blog/starcoder
• Generate code https://huggingface.co/spaces/bigcode/bigcode-editor
• Play with StarCoder: https://huggingface.co/spaces/bigcode/bigcode-playground
• Try the reasoning: https://huggingface.co/spaces/bigcode/Reasoning-with-StarCoder
• Explore the data set: https://huggingface.co/spaces/bigcode/search

Basic quantities to develop LLMs:

Magnitudes to take into account when developing LLMs