DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

1. Introduction

2. Model Downloads

3. Evaluation Results

3.1 Code Generation

3.2 Code Completion

3.3 Code Fixing

3.4 Mathematical Reasoning

3.5 General Natural Language

3.6 Context Window

4. Chat Website

5. API Platform

6. How to run locally

Inference with Huggingface's Transformers

Code Completion

fromtransformersimportAutoTokenizer,AutoModelForCausalLMimporttorchtokenizer=AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base",trust_remote_code=True)model=AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base",trust_remote_code=True,torch_dtype=torch.bfloat16).cuda()input_text="#write a quick sort algorithm"inputs=tokenizer(input_text,return_tensors="pt").to(model.device)outputs=model.generate(**inputs,max_length=128)print(tokenizer.decode(outputs[0],skip_special_tokens=True))

Code Insertion

fromtransformersimportAutoTokenizer,AutoModelForCausalLMimporttorchtokenizer=AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base",trust_remote_code=True)model=AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Base",trust_remote_code=True,torch_dtype=torch.bfloat16).cuda()input_text="""<｜fim▁begin｜>def quick_sort(arr):    if len(arr) <= 1:        return arr    pivot = arr[0]    left = []    right = []<｜fim▁hole｜>        if arr[i] < pivot:            left.append(arr[i])        else:            right.append(arr[i])    return quick_sort(left) + [pivot] + quick_sort(right)<｜fim▁end｜>"""inputs=tokenizer(input_text,return_tensors="pt").to(model.device)outputs=model.generate(**inputs,max_length=128)print(tokenizer.decode(outputs[0],skip_special_tokens=True)[len(input_text):])

Chat Completion

fromtransformersimportAutoTokenizer,AutoModelForCausalLMimporttorchtokenizer=AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct",trust_remote_code=True)model=AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct",trust_remote_code=True,torch_dtype=torch.bfloat16).cuda()messages=[    {'role':'user','content':"write a quick sort algorithm in python."}]inputs=tokenizer.apply_chat_template(messages,add_generation_prompt=True,return_tensors="pt").to(model.device)# tokenizer.eos_token_id is the id of <｜end▁of▁sentence｜> tokenoutputs=model.generate(inputs,max_new_tokens=512,do_sample=False,top_k=50,top_p=0.95,num_return_sequences=1,eos_token_id=tokenizer.eos_token_id)print(tokenizer.decode(outputs[0][len(inputs[0]):],skip_special_tokens=True))

<｜begin▁of▁sentence｜>User: {user_message_1}Assistant: {assistant_message_1}<｜end▁of▁sentence｜>User: {user_message_2}Assistant:

<｜begin▁of▁sentence｜>{system_message}User: {user_message_1}Assistant: {assistant_message_1}<｜end▁of▁sentence｜>User: {user_message_2}Assistant:

Inference with SGLang (recommended)

# BF16, tensor parallelism = 8python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-Coder-V2-Instruct --tp 8 --trust-remote-code# BF16, w/ torch.compile (The compilation can take several minutes)python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct --trust-remote-code --enable-torch-compile# FP8, tensor parallelism = 8, FP8 KV cachepython3 -m sglang.launch_server --model neuralmagic/DeepSeek-Coder-V2-Instruct-FP8 --tp 8 --trust-remote-code --kv-cache-dtype fp8_e5m2

import openaiclient = openai.Client(    base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")# Chat completionresponse = client.chat.completions.create(    model="default",    messages=[        {"role": "system", "content": "You are a helpful AI assistant"},        {"role": "user", "content": "List 3 countries and their capitals."},    ],    temperature=0,    max_tokens=64,)print(response)

Inference with vLLM (recommended)

fromtransformersimportAutoTokenizerfromvllmimportLLM,SamplingParamsmax_model_len,tp_size=8192,1model_name="deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct"tokenizer=AutoTokenizer.from_pretrained(model_name)llm=LLM(model=model_name,tensor_parallel_size=tp_size,max_model_len=max_model_len,trust_remote_code=True,enforce_eager=True)sampling_params=SamplingParams(temperature=0.3,max_tokens=256,stop_token_ids=[tokenizer.eos_token_id])messages_list= [    [{"role":"user","content":"Who are you?"}],    [{"role":"user","content":"write a quick sort algorithm in python."}],    [{"role":"user","content":"Write a piece of quicksort code in C++."}],]prompt_token_ids= [tokenizer.apply_chat_template(messages,add_generation_prompt=True)formessagesinmessages_list]outputs=llm.generate(prompt_token_ids=prompt_token_ids,sampling_params=sampling_params)generated_text= [output.outputs[0].textforoutputinoutputs]print(generated_text)

7. License

8. Citation

@article{zhu2024deepseek,  title={DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence},  author={Zhu, Qihao and Guo, Daya and Shao, Zhihong and Yang, Dejian and Wang, Peiyi and Xu, Runxin and Wu, Y and Li, Yukun and Gao, Huazuo and Ma, Shirong and others},  journal={arXiv preprint arXiv:2406.11931},  year={2024}}

9. Contact