CoT prompting is usually applied as a few-shot immediate, the place the mannequin receives a job description and examples of input-output pairs. These examples embody reasoning steps that systematically result in the right reply, demonstrating the way to course of the knowledge. Thus, to carry out CoT prompting successfully, customers want high-quality demonstration examples. Nonetheless, this may be difficult for duties requiring specialised area experience. As an illustration, utilizing an LLM for medical prognosis based mostly on a affected person’s historical past would necessitate the help of area consultants, similar to docs or physicians, to articulate the right reasoning steps. Furthermore, CoT is especially efficient in fashions with a sufficiently massive parameter scale. Based on the paper [6], CoT is handiest for the 137B parameter LaMBDA [7], the 175B parameter GPT-3 [3], and the 540B parameter PaLM [8] fashions. This limitation can prohibit its applicability for smaller-scale fashions.

One other side of CoT prompting that units it aside from customary prompting is that the mannequin must generate considerably extra tokens earlier than arriving on the closing reply. Whereas not essentially a downside, it is a issue to contemplate in case you are compute-bound at inference time.

If you’d like a deeper overview, I like to recommend OpenAI’s prompting assets, out there at https://platform.openai.com/docs/guides/prompt-engineering/strategy-write-clear-instructions.

All code and assets associated to this text are made out there at this Github repository, beneath the introduction_to_prompting folder. Be happy to drag the repository and run the notebooks on to run these experiments. Please let me know if in case you have any suggestions or observations or for those who discover any errors!

We are able to discover these strategies on a pattern dataset to make understanding simpler. To this finish, we’ll work with the MedQA dataset [9], which incorporates questions testing medical and scientific data. We’ll particularly make the most of the USMLE questions from this dataset. This job is right for analyzing numerous prompting strategies, as answering the questions requires data and reasoning. We’ll take a look at the capabilities of Llama-2 7B [10] and GPT-3.5 [11] on this dataset.

Let’s first obtain the dataset. The MedQA dataset will be downloaded from this hyperlink. After downloading the dataset, we will parse and start processing the questions. The take a look at set incorporates a complete of 1,273 questions. We randomly pattern 300 questions from the take a look at set to judge the fashions and choose 3 random examples from the coaching set as our few-shot demonstrations for the mannequin.

import json
import random
random.seed(42)
def read_jsonl_file(file_path):
"""
Parses a JSONL (JSON Strains) file and returns an inventory of dictionaries.
Args:
file_path (str): The trail to the JSONL file to be learn.
Returns:
checklist of dict: A listing the place every aspect is a dictionary representing
a JSON object from the file.
"""
jsonl_lines = []
with open(file_path, 'r', encoding="utf-8") as file:
for line in file:
json_object = json.masses(line)
jsonl_lines.append(json_object)
return jsonl_lines
def write_jsonl_file(dict_list, file_path):
"""
Write an inventory of dictionaries to a JSON Strains file.
Args:
- dict_list (checklist): A listing of dictionaries to put in writing to the file.
- file_path (str): The trail to the file the place the info shall be written.
"""
with open(file_path, 'w') as file:
for dictionary in dict_list:
# Convert the dictionary to a JSON string and write it to the file.
json_line = json.dumps(dictionary)
file.write(json_line + 'n')
# learn the contents of the practice and take a look at set
train_set = read_jsonl_file("data_clean/questions/US/4_options/phrases_no_exclude_train.jsonl")
test_set = read_jsonl_file("data_clean/questions/US/4_options/phrases_no_exclude_test.jsonl")
# subsample take a look at set samples and few-shot samples
test_set_subsampled = random.pattern(test_set, 300)
few_shot_examples = random.pattern(test_set, 3)
# dump the sampled questions and few-shot samples as jsonl recordsdata
write_jsonl_file(test_set_subsampled, "USMLE_test_samples_300.jsonl")
write_jsonl_file(few_shot_examples, "USMLE_few_shot_samples.jsonl")

Prompting Llama 2 7B-Chat with a Zero-Shot Immediate

The Llama collection of fashions have been launched by Meta. They’re a decoder-only household of LLMs spanning parameter counts from 7B to 70B. The Llama-2 collection of fashions is available in two variants: the bottom model and the chat/instruction-tuned variant. For this train, we’ll work with the chat-version of the Llama 2-7B mannequin.

Let’s see how nicely we will immediate the Llama mannequin to reply these medical questions. We load the mannequin into reminiscence:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from tqdm import tqdm
questions = read_jsonl_file("USMLE_test_samples_300.jsonl")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
mannequin = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf",  torch_dtype=torch.bfloat16).cuda()
mannequin.eval()

When you’re working with Nvidia Ampere GPUs, you’ll be able to load the mannequin utilizing torch.bfloat16. It affords speedups to inference and makes use of lesser GPU reminiscence than regular FP16/FP32.

First, let’s now craft a primary immediate for our job:

PROMPT = """You can be supplied with a medical or scientific query, together with a number of doable reply decisions. Decide the fitting reply from the alternatives. 
Your response must be within the format "The reply is <correct_choice>". Don't add some other pointless content material in your response"""

Our immediate is simple. It contains details about the character of the duty and gives directions on the format for the output. We’ll see how successfully this immediate works in apply.

The Llama-2 chat fashions have a selected chat template to be adopted for prompting them.

<s>[INST] <<SYS>>
You can be supplied with a medical or scientific query, together with a number of doable reply decisions. Decide the fitting reply from the alternatives. 
Your response must be within the format "The reply is <correct_choice>". Don't add some other pointless content material in your response
<</SYS>>
A 21-year-old male presents to his main care supplier for fatigue. He reviews that he graduated from school final month and returned 3 days in the past from a 2 week trip to Vietnam and Cambodia. For the previous 2 days, he has developed a worsening headache, malaise, and ache in his arms and wrists. The affected person has a previous medical historical past of bronchial asthma managed with albuterol as wanted. He's sexually energetic with each women and men, and he makes use of condoms “more often than not.” On bodily examination, the affected person’s temperature is 102.5°F (39.2°C), blood strain is 112/66 mmHg, pulse is 105/min, respirations are 12/min, and oxygen saturation is 98% on room air. He has tenderness to palpation over his bilateral metacarpophalangeal joints and a maculopapular rash on his trunk and higher thighs. Tourniquet take a look at is unfavorable. Laboratory outcomes are as follows:
Hemoglobin: 14 g/dL
Hematocrit: 44%
Leukocyte rely: 3,200/mm^3
Platelet rely: 112,000/mm^3
Serum:
Na+: 142 mEq/L
Cl-: 104 mEq/L
Okay+: 4.6 mEq/L
HCO3-: 24 mEq/L
BUN: 18 mg/dL
Glucose: 87 mg/dL
Creatinine: 0.9 mg/dL
AST: 106 U/L
ALT: 112 U/L
Bilirubin (complete): 0.8 mg/dL
Bilirubin (conjugated): 0.3 mg/dL
Which of the next is the probably prognosis on this affected person?
Choices:
A. Chikungunya
B. Dengue fever
C. Epstein-Barr virus
D. Hepatitis A [/INST]

The duty description must be offered between the <<SYS>> tokens, adopted by the precise query the mannequin must reply. The immediate is concluded with a [/INST] token to point the tip of the enter textual content.

The position will be one in every of “person”, “system”, or “assistant”. The “system” position gives the mannequin with the duty description, and the “person” position incorporates the enter to which the mannequin wants to reply. This is similar conference we’ll make the most of in a while when interacting with GPT-3.5. It’s equal to making a fictional multi-turn dialog historical past offered to Llama-2, the place every flip corresponds to an instance demonstration and a perfect output from the mannequin.

Sounds sophisticated? Fortunately, the Huggingface Transformers library helps changing prompts to the chat template. We’ll make the most of this performance to make our lives simpler. Let’s begin with helper functionalities to course of the dataset and create prompts.

def create_query(merchandise):
"""
Creates the enter for the mannequin utilizing the query and the a number of selection choices.
Args:
merchandise (dict): A dictionary containing the query and choices.
Anticipated keys are "query" and "choices", the place "choices" is one other
dictionary with keys "A", "B", "C", and "D".
Returns:
str: A formatted question combining the query and choices, prepared to be used.
"""
question = merchandise["question"] + "nOptions:n" + 
"A. " + merchandise["options"]["A"] + "n" + 
"B. " + merchandise["options"]["B"] + "n" + 
"C. " + merchandise["options"]["C"] + "n" + 
"D. " + merchandise["options"]["D"]
return question
def build_zero_shot_prompt(system_prompt, query):
"""
Builds the zero-shot immediate.
Args:
system_prompt (str): Process Instruction
content material (dict): The content material for which to create a question, formatted as
required by `create_query`.
Returns:
checklist of dict: A listing of messages, together with a system message defining
the duty and a person message with the enter query.
"""
messages = [{"role": "system", "content": system_prompt},
{"role": "user", "content": create_query(question)}]
return messages

This perform constructs the question to supply to the LLM. The MedQA dataset shops every query as a JSON aspect, with the questions and choices offered as keys. We parse the JSON and assemble the query together with the alternatives.

Let’s begin acquiring outputs from the mannequin. The present job entails answering the offered medical query by deciding on the right reply from numerous choices. In contrast to artistic duties similar to content material writing or summarization, which can require the mannequin to be imaginative and inventive in its output, it is a knowledge-based job designed to check the mannequin’s means to reply questions based mostly on data encoded in its parameters. Due to this fact, we’ll use grasping decoding whereas producing the reply. Let’s outline a helper perform for parsing the mannequin responses and calculating accuracy.

sample = re.compile(r"([A-Z]).s*(.*)")
def parse_answer(response):
"""
Extracts the reply choice from the expected string.
Args:
- response (str): The string to seek for the sample.
Returns:
- str: The matched reply choice if discovered or an empty string in any other case.
"""
match = re.search(sample, response)
if match:
letter = match.group(1)
else:
letter = ""
return letter
def calculate_accuracy(ground_truth, predictions):
"""
Calculates the accuracy of predictions in comparison with floor reality labels.
Args:
- ground_truth (checklist): A listing of true labels.
- predictions (checklist): A listing of predicted labels.
Returns:
- float: The accuracy of predictions as a fraction of appropriate predictions over complete predictions.
"""
return sum([1 if x==y else 0 for x,y in zip(ground_truth, predictions)]) / len(ground_truth)

ground_truth = []
for merchandise in questions:
ans_options = merchandise["options"]
correct_ans_option = ""
for key,worth in ans_options.objects():
if worth == merchandise["answer"]:
correct_ans_option = key
break
ground_truth.append(correct_ans_option)

zero_shot_llama_answers = []
for merchandise in tqdm(questions):
zero_shot_prompt_messages = build_zero_shot_prompt(PROMPT, merchandise)
immediate = tokenizer.apply_chat_template(zero_shot_prompt_messages, tokenize=False)
input_ids = tokenizer(immediate, return_tensors="pt", truncation=True).input_ids.cuda()
outputs = mannequin.generate(input_ids=input_ids, max_new_tokens=10, do_sample=False)
# https://github.com/huggingface/transformers/points/17117#issuecomment-1124497554
gen_text = tokenizer.batch_decode(outputs.detach().cpu().numpy()[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
zero_shot_llama_answers.append(gen_text.strip())
zero_shot_llama_predictions = [parse_answer(x) for x in zero_shot_llama_answers]
print(calculate_accuracy(ground_truth, zero_shot_llama_predictions))

We get a efficiency of 36% within the zero-shot setting. Not a nasty begin, however let’s see if we will push this efficiency additional.

Prompting Llama 2 7B-Chat with a Few-Shot Immediate

Let’s now present job demonstrations to the mannequin. We use the three randomly sampled questions from the coaching set and append them to the mannequin as job demonstrations. Thankfully, we will proceed utilizing the chat-template assist offered by the Transformers library and the tokenizer to append our few-shot examples with minimal code modifications.

def build_few_shot_prompt(system_prompt, content material, few_shot_examples):
"""
Builds the few-shot immediate utilizing offered examples.
Args:
system_prompt (str): Process description for the LLM
content material (dict): The content material for which to create a question, just like the
construction required by `create_query`.
few_shot_examples (checklist of dict): Examples to simulate a hypothetical
dialog. Every dict should have "choices" and an "reply".
Returns:
checklist of dict: A listing of messages, simulating a dialog with
few-shot examples, adopted by the present person question.
"""
messages = [{"role": "system", "content": system_prompt}]
for merchandise in few_shot_examples:
ans_options = merchandise["options"]
correct_ans_option = ""
for key, worth in ans_options.objects():
if worth == merchandise["answer"]:
correct_ans_option = key
break
messages.append({"position": "person", "content material": create_query(merchandise)})
messages.append({"position": "assistant", "content material": "The reply is " + correct_ans_option + "."})
messages.append({"position": "person", "content material": create_query(content material)})
return messages
few_shot_prompts = read_jsonl_file("USMLE_few_shot_samples.jsonl")

Let’s visualize what our few-shot immediate appears to be like like.

<s>[INST] <<SYS>>
You can be supplied with a medical or scientific query, together with a number of doable reply decisions. Decide the fitting reply from the alternatives. 
Your response must be within the format "The reply is <correct_choice>". Don't add some other pointless content material in your response
<</SYS>>
A 30-year-old lady presents to the clinic due to fever, joint ache, and a rash on her decrease extremities. She admits to intravenous drug use. Bodily examination reveals palpable petechiae and purpura on her decrease extremities. Laboratory outcomes reveal a unfavorable antinuclear antibody, constructive rheumatoid issue, and constructive serum cryoglobulins. Which of the next underlying situations on this affected person is answerable for these findings?
Choices:
A. Hepatitis B an infection
B. Hepatitis C an infection
C. HIV an infection
D. Systemic lupus erythematosus (SLE) [/INST] The reply is B. </s><s>[INST] A ten-year-old little one presents to your workplace with a persistent cough. His mom states that he has had a cough for the previous two weeks that's non-productive together with low fevers of 100.5 F as measured by an oral thermometer. The mom denies some other medical historical past and states that he has been round one different buddy who additionally has had this cough for a lot of weeks. The affected person's vitals are inside regular limits except for his temperature of 100.7 F. His chest radiograph demonstrated diffuse interstitial infiltrates. Which organism is probably inflicting his pneumonia?
Choices:
A. Mycoplasma pneumoniae
B. Staphylococcus aureus
C. Streptococcus pneumoniae
D. Streptococcus agalactiae [/INST] The reply is A. </s><s>[INST] A 44-year-old with a previous medical historical past vital for human immunodeficiency virus an infection presents to the emergency division after he was discovered to be experiencing worsening confusion. The affected person was famous to be disoriented by residents and workers on the homeless shelter the place he resides. On presentation he reviews headache and muscle aches however is unable to supply extra info. His temperature is 102.2°F (39°C), blood strain is 112/71 mmHg, pulse is 115/min, and respirations are 24/min. Knee extension with hips flexed produces vital resistance and ache. A lumbar puncture is carried out with the next outcomes:
Opening strain: Regular
Fluid coloration: Clear
Cell rely: Elevated lymphocytes
Protein: Barely elevated
Which of the next is the probably reason behind this affected person's signs?
Choices:
A. Cryptococcus
B. Group B streptococcus
C. Herpes simplex virus
D. Neisseria meningitidis [/INST] The reply is C. </s><s>[INST] A 21-year-old male presents to his main care supplier for fatigue. He reviews that he graduated from school final month and returned 3 days in the past from a 2 week trip to Vietnam and Cambodia. For the previous 2 days, he has developed a worsening headache, malaise, and ache in his arms and wrists. The affected person has a previous medical historical past of bronchial asthma managed with albuterol as wanted. He's sexually energetic with each women and men, and he makes use of condoms “more often than not.” On bodily examination, the affected person’s temperature is 102.5°F (39.2°C), blood strain is 112/66 mmHg, pulse is 105/min, respirations are 12/min, and oxygen saturation is 98% on room air. He has tenderness to palpation over his bilateral metacarpophalangeal joints and a maculopapular rash on his trunk and higher thighs. Tourniquet take a look at is unfavorable. Laboratory outcomes are as follows:
Hemoglobin: 14 g/dL
Hematocrit: 44%
Leukocyte rely: 3,200/mm^3
Platelet rely: 112,000/mm^3
Serum:
Na+: 142 mEq/L
Cl-: 104 mEq/L
Okay+: 4.6 mEq/L
HCO3-: 24 mEq/L
BUN: 18 mg/dL
Glucose: 87 mg/dL
Creatinine: 0.9 mg/dL
AST: 106 U/L
ALT: 112 U/L
Bilirubin (complete): 0.8 mg/dL
Bilirubin (conjugated): 0.3 mg/dL
Which of the next is the probably prognosis on this affected person?
Choices:
A. Chikungunya
B. Dengue fever
C. Epstein-Barr virus
D. Hepatitis A [/INST]

The immediate is kind of lengthy, on condition that we append three demonstrations. Let’s now run Llama-2 with the few-shot immediate and get the outcomes:

few_shot_llama_answers = []
for merchandise in tqdm(questions):
few_shot_prompt_messages = build_few_shot_prompt(PROMPT, merchandise, few_shot_prompts)
immediate = tokenizer.apply_chat_template(few_shot_prompt_messages, tokenize=False)
input_ids = tokenizer(immediate, return_tensors="pt", truncation=True).input_ids.cuda()
outputs = mannequin.generate(input_ids=input_ids, max_new_tokens=10, do_sample=False)
gen_text = tokenizer.batch_decode(outputs.detach().cpu().numpy()[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
few_shot_llama_answers.append(gen_text.strip())
few_shot_llama_predictions = [parse_answer(x) for x in few_shot_llama_answers]
print(calculate_accuracy(ground_truth, few_shot_llama_predictions))

We now get an total accuracy of 41.67%. Not unhealthy, almost 6% enchancment over zero-shot prompting with the identical mannequin!

What occurs if we don’t adhere to the chat template?

Earlier, I noticed that it’s advisable to construction our immediate in response to the immediate template that was used to fine-tune an LLM initially. Let’s confirm if not adhering to the chat template hurts our efficiency. We create a perform that builds a few-shot immediate utilizing the identical examples with out adhering to the chat format.

def build_few_shot_prompt_wo_chat_template(system_prompt, content material, few_shot_examples):
"""
Builds the few-shot immediate utilizing offered examples, bypassing the chat-template
for Llama-2.
Args:
system_prompt (str): Process description for the LLM
content material (dict): The content material for which to create a question, just like the
construction required by `create_query`.
few_shot_examples (checklist of dict): Examples to simulate a hypothetical
dialog. Every dict should have "choices" and an "reply".
Returns:
str: few-shot immediate in non-chat format
"""
few_shot_prompt = ""
few_shot_prompt += "Process: " + system_prompt + "n"
for merchandise in few_shot_examples:
ans_options = merchandise["options"]
correct_ans_option = ""
for key, worth in ans_options.objects():
if worth == merchandise["answer"]:
correct_ans_option = key
break
few_shot_prompt += create_query(merchandise) + "n" + "The reply is " + correct_ans_option + "." + "n"
few_shot_prompt += create_query(content material) + "n"
return few_shot_prompt

Our prompts now seem like this:

Process: You can be supplied with a medical or scientific query, together with a number of doable reply decisions. Decide the fitting reply from the alternatives. 
Your response must be within the format "The reply is <correct_choice>". Don't add some other pointless content material in your response
A 30-year-old lady presents to the clinic due to fever, joint ache, and a rash on her decrease extremities. She admits to intravenous drug use. Bodily examination reveals palpable petechiae and purpura on her decrease extremities. Laboratory outcomes reveal a unfavorable antinuclear antibody, constructive rheumatoid issue, and constructive serum cryoglobulins. Which of the next underlying situations on this affected person is answerable for these findings?
Choices:
A. Hepatitis B an infection
B. Hepatitis C an infection
C. HIV an infection
D. Systemic lupus erythematosus (SLE)
The reply is B.
A ten-year-old little one presents to your workplace with a persistent cough. His mom states that he has had a cough for the previous two weeks that's non-productive together with low fevers of 100.5 F as measured by an oral thermometer. The mom denies some other medical historical past and states that he has been round one different buddy who additionally has had this cough for a lot of weeks. The affected person's vitals are inside regular limits except for his temperature of 100.7 F. His chest radiograph demonstrated diffuse interstitial infiltrates. Which organism is probably inflicting his pneumonia?
Choices:
A. Mycoplasma pneumoniae
B. Staphylococcus aureus
C. Streptococcus pneumoniae
D. Streptococcus agalactiae
The reply is A.
A 44-year-old with a previous medical historical past vital for human immunodeficiency virus an infection presents to the emergency division after he was discovered to be experiencing worsening confusion. The affected person was famous to be disoriented by residents and workers on the homeless shelter the place he resides. On presentation he reviews headache and muscle aches however is unable to supply extra info. His temperature is 102.2°F (39°C), blood strain is 112/71 mmHg, pulse is 115/min, and respirations are 24/min. Knee extension with hips flexed produces vital resistance and ache. A lumbar puncture is carried out with the next outcomes:
Opening strain: Regular
Fluid coloration: Clear
Cell rely: Elevated lymphocytes
Protein: Barely elevated
Which of the next is the probably reason behind this affected person's signs?
Choices:
A. Cryptococcus
B. Group B streptococcus
C. Herpes simplex virus
D. Neisseria meningitidis
The reply is C.
A 21-year-old male presents to his main care supplier for fatigue. He reviews that he graduated from school final month and returned 3 days in the past from a 2 week trip to Vietnam and Cambodia. For the previous 2 days, he has developed a worsening headache, malaise, and ache in his arms and wrists. The affected person has a previous medical historical past of bronchial asthma managed with albuterol as wanted. He's sexually energetic with each women and men, and he makes use of condoms “more often than not.” On bodily examination, the affected person’s temperature is 102.5°F (39.2°C), blood strain is 112/66 mmHg, pulse is 105/min, respirations are 12/min, and oxygen saturation is 98% on room air. He has tenderness to palpation over his bilateral metacarpophalangeal joints and a maculopapular rash on his trunk and higher thighs. Tourniquet take a look at is unfavorable. Laboratory outcomes are as follows:
Hemoglobin: 14 g/dL
Hematocrit: 44%
Leukocyte rely: 3,200/mm^3
Platelet rely: 112,000/mm^3
Serum:
Na+: 142 mEq/L
Cl-: 104 mEq/L
Okay+: 4.6 mEq/L
HCO3-: 24 mEq/L
BUN: 18 mg/dL
Glucose: 87 mg/dL
Creatinine: 0.9 mg/dL
AST: 106 U/L
ALT: 112 U/L
Bilirubin (complete): 0.8 mg/dL
Bilirubin (conjugated): 0.3 mg/dL
Which of the next is the probably prognosis on this affected person?
Choices:
A. Chikungunya
B. Dengue fever
C. Epstein-Barr virus
D. Hepatitis A

Let’s now consider Llama 2 with these prompts and observe the way it performs:

few_shot_llama_answers_wo_chat_template = []
for merchandise in tqdm(questions):
immediate = build_few_shot_prompt_wo_chat_template(PROMPT, merchandise, few_shot_prompts)
input_ids = tokenizer(immediate, return_tensors="pt", truncation=True).input_ids.cuda()
outputs = mannequin.generate(input_ids=input_ids, max_new_tokens=10, do_sample=False)
gen_text = tokenizer.batch_decode(outputs.detach().cpu().numpy()[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
few_shot_llama_answers_wo_chat_template.append(gen_text.strip())
few_shot_llama_predictions_wo_chat_template = [parse_answer(x) for x in few_shot_llama_answers_wo_chat_template]
print(calculate_accuracy(ground_truth, few_shot_llama_predictions_wo_chat_template))

We obtain an accuracy of 36%. That is almost 6% decrease than our earlier few-shot rating. This reinforces our earlier argument that it’s essential to construction our prompts in response to the template used to fine-tune the LLM we intend to work with. Immediate templates matter!

Prompting Llama 2 7B-Chat with CoT Prompting

Let’s conclude by evaluating CoT prompting. Keep in mind, our dataset contains questions designed to check medical data by means of the USMLE examination. Such questions typically require each factual recall and conceptual reasoning to reply. This makes it an ideal job for testing how nicely CoT works.

First, we should present an instance CoT immediate to the mannequin to display the way to purpose a couple of query. For this function, we’ll use one of many prompts from Google’s MedPALM paper [12].

We use this five-shot immediate for evaluating the fashions. Since this immediate fashion differs barely from our earlier prompts, let’s create some helper features once more to course of them and procure the outputs. Whereas using CoT prompting, we generate the output with a bigger output token rely to allow the mannequin to “assume” and “purpose” earlier than answering the query.

def create_query_cot(merchandise):
"""
Creates the enter for the mannequin utilizing the query and the a number of selection choices within the CoT format.
Args:
merchandise (dict): A dictionary containing the query and choices.
Anticipated keys are "query" and "choices", the place "choices" is one other
dictionary with keys "A", "B", "C", and "D".
Returns:
str: A formatted question combining the query and choices, prepared to be used.
"""
question = "Query: " + merchandise["question"] + "n" + 
"(A) " + merchandise["options"]["A"] + " " +  
"(B) " + merchandise["options"]["B"] + " " +  
"(C) " + merchandise["options"]["C"] + " " +  
"(D) " + merchandise["options"]["D"]
return question
def build_cot_prompt(instruction, input_question, cot_examples):
"""
Builds the few-shot immediate for the GPT API utilizing offered examples.
Args:
content material (dict): The content material for which to create a question, just like the
construction required by `create_query`.
few_shot_examples (checklist of dict): Examples to simulate a hypothetical
dialog. Every dict should have "query" and an "rationalization".
Returns:
checklist of dict: A listing of messages, simulating a dialog with
few-shot examples, adopted by the present person question.
"""
messages = [{"role": "system", "content": instruction}]
for merchandise in cot_examples:
messages.append({"position": "person", "content material": merchandise["question"]})
messages.append({"position": "assistant", "content material": merchandise["explanation"]})
messages.append({"position": "person", "content material": create_query_cot(input_question)})
return messages
def parse_answer_cot(textual content):
"""
Extracts the selection from a string that follows the sample "Reply: (Alternative) Textual content".
Args:
- textual content (str): The enter string from which to extract the selection.
Returns:
- str: The extracted selection or a message indicating no match was discovered.
"""
# Regex sample to match the reply half
sample = r"Reply: (.*)"
# Seek for the sample within the textual content and extract the matching group
match = re.search(sample, textual content)
if match:
if len(match.group(1)) > 1:
return match.group(1)[1]
else:
return ""
else:
return ""

cot_llama_answers = []
for merchandise in tqdm(questions):
cot_prompt = build_cot_prompt(COT_INSTRUCTION, merchandise, COT_EXAMPLES)
immediate = tokenizer.apply_chat_template(cot_prompt, tokenize=False)
input_ids = tokenizer(immediate, return_tensors="pt", truncation=True).input_ids.cuda()
outputs = mannequin.generate(input_ids=input_ids, max_new_tokens=100, do_sample=False)
gen_text = tokenizer.batch_decode(outputs.detach().cpu().numpy()[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
cot_llama_answers.append(gen_text.strip())
cot_llama_predictions = [parse_answer_cot(x) for x in cot_llama_answers]
print(calculate_accuracy(ground_truth, cot_llama_predictions))

Our efficiency dips to twenty% utilizing CoT prompting for Llama 2–7B. That is typically consistent with the findings of the CoT paper [6], the place the authors point out that CoT is an emergent property for LLMs that improves with the size of the mannequin. That being stated, let’s analyze why the efficiency dipped drastically.

Failure Modes in CoT for Llama 2

We pattern a couple of of the responses offered by Llama 2 on a number of the take a look at set questions to investigate error instances:

Whereas CoT prompting permits the mannequin to “assume” earlier than arriving on the closing reply, usually, the mannequin both doesn’t arrive at a conclusive reply or mentions the reply in a format inconsistent with our instance demonstrations. A failure mode I haven’t analyzed right here, however probably price exploring, is to test instances within the take a look at set the place the mannequin “causes” incorrectly and, subsequently, arrives on the unsuitable reply. That is past the scope of the present article and my medical data, however it’s definitely one thing I intend to revisit later.

Prompting GPT-3.5 with a Zero-Shot Immediate

Let’s start defining some helper features that assist us course of these inputs for using the GPT API. You would wish to generate an API key to make use of the GPT-3.5 API. You possibly can set the API key in Home windows utilizing:

setx OPENAI_API_KEY "your-api-key-here"

or in Linux utilizing:

export OPENAI_API_KEY "your-api-key-here"

within the present session you’re utilizing.

from openai import OpenAI
import re
from tqdm import tqdm
# assuming you might have already set the key key utilizing env variable
# if not, you too can instantiate the OpenAI shopper by offering the 
# secret key immediately like so:
# I extremely suggest not doing this, as it's a finest apply to not retailer
# the api key in your code immediately or in any plain-text file for safety 
# causes.
# shopper = OpenAI(api_key = "")
shopper = OpenAI() 

def get_response(messages, model_name, temperature = 0.0, max_tokens = 10):
"""
Obtains the responses/solutions of the mannequin by means of the chat-completions API.
Args:
messages (checklist of dict): The constructed messages offered to the API.
model_name (str): Identify of the mannequin to entry by means of the API
temperature (float): A price between 0 and 1 that controls the randomness of the output.
A temperature worth of 0 ideally makes the mannequin choose the probably token, making the outputs (principally) deterministic.
max_tokens (int): Most variety of tokens that the mannequin ought to generate
Returns:
str: The response message content material from the mannequin.
"""
response = shopper.chat.completions.create(
mannequin=model_name,
messages=messages,
temperature=temperature,
max_tokens=max_tokens
)
return response.decisions[0].message.content material

This perform now constructs the immediate within the format for the GPT-3.5 API. We are able to work together with the GPT-3.5 mannequin by means of the chat-completions API offered by the library. The API requires messages to be structured as an inventory of dictionaries for sending to the API. Every message should specify the position and the content material. The conventions adopted relating to the “system”, “person”, and “assistant” roles are the identical as these described earlier for the Llama-7B Chat Mannequin.

Let’s now use the GPT-3.5 API to course of the take a look at set and procure the responses. After receiving all of the responses, we extract the choices from the mannequin’s responses and calculate the accuracy.

zero_shot_gpt_answers = []
for merchandise in tqdm(questions):
zero_shot_prompt_messages = build_zero_shot_prompt(PROMPT, merchandise)
reply = get_response(zero_shot_prompt_messages, model_name = "gpt-3.5-turbo", temperature = 0.0, max_tokens = 10)
zero_shot_gpt_answers.append(reply)
zero_shot_gpt_predictions = [parse_answer(x) for x in zero_shot_gpt_answers]
print(calculate_accuracy(ground_truth, zero_shot_gpt_predictions))

Our efficiency now stands at 63%. It is a vital enchancment from the efficiency of Llama 2–7B. This isn’t shocking, on condition that GPT-3.5 is probably going a lot bigger and skilled on extra knowledge than Llama 2–7B, together with different proprietary optimizations that OpenAI could have included to the mannequin. Let’s see how nicely few-shot prompting works now.

Prompting GPT-3.5 with a Few-Shot Immediate

To supply few-shot examples to the LLM, we reuse the three examples we sampled from the coaching set and append them to the immediate. For GPT-3.5, we create an inventory of messages with examples, just like our earlier processing for Llama 2. The inputs are appended utilizing the “person” position, and the corresponding choice is offered within the “assistant” position. We reuse the sooner perform for constructing few-shot prompts.

That is once more equal to making a fictional multi-turn dialog historical past offered to GPT-3.5, the place every flip corresponds to an instance demonstration.

Let’s now get hold of the outputs utilizing GPT-3.5.

few_shot_gpt_answers = []
for merchandise in tqdm(questions):
few_shot_prompt_messages = build_few_shot_prompt(PROMPT, merchandise, few_shot_prompts)
reply = get_response(few_shot_prompt_messages, model_name= "gpt-3.5-turbo", temperature = 0.0, max_tokens = 10)
few_shot_gpt_answers.append(reply)
few_shot_gpt_predictions = [parse_answer(x) for x in few_shot_gpt_answers]
print(calculate_accuracy(ground_truth, few_shot_gpt_predictions))

We’ve managed to push the efficiency from 63% to 67% utilizing few-shot prompting! It is a vital enchancment, highlighting the worth of offering job demonstrations to the mannequin.

Prompting GPT-3.5 with CoT Prompting

Let’s now consider GPT-3.5 with CoT prompting. We re-use the identical CoT immediate and get the outputs:

cot_gpt_answers = []
for merchandise in tqdm(questions):
cot_prompt = build_cot_prompt(COT_INSTRUCTION, merchandise, COT_EXAMPLES)
reply = get_response(cot_prompt, model_name= "gpt-3.5-turbo", temperature = 0.0, max_tokens = 100)
cot_gpt_answers.append(reply)
cot_gpt_predictions = [parse_answer_cot(x) for x in cot_gpt_answers]
print(calculate_accuracy(ground_truth, cot_gpt_predictions))

Utilizing CoT prompting with GPT-3.5 ends in an accuracy of 71%! This represents an additional 4% enchancment over few-shot prompting. It seems that enabling the mannequin to “assume” out loud earlier than answering the query is useful for this job. That is additionally according to the findings of the paper [6] that CoT unlocked efficiency enhancements for bigger parameter fashions.

Prompting is an important ability for working with Giant Language Fashions (LLMs), and understanding that there are numerous instruments within the prompting toolkit that may assist extract higher efficiency from LLMs in your duties relying on the context. I hope this text serves as a broad and (hopefully!) accessible introduction to this topic. Nonetheless, it doesn’t intention to supply a complete overview of all prompting methods. Prompting stays a extremely energetic discipline of analysis, with quite a few strategies being launched similar to ReAct [13], Tree-of-Thought prompting [14] and many others. I like to recommend exploring these strategies to higher perceive them and improve your prompting toolkit.

On this article, I’ve aimed to make all experiments as deterministic and reproducible as doable. We use grasping decoding to acquire our outputs for zero-shot, few-shot, and CoT prompting with Llama-2. Whereas these scores ought to technically be reproducible, in uncommon instances, Cuda/GPU-related or library points might result in barely completely different outcomes.

Equally, when acquiring responses from the GPT-3.5 API, we use a temperature of 0 to get outcomes and select solely the subsequent probably token with out sampling for all immediate settings. This makes the outcomes “principally deterministic”, so it’s doable that sending the identical prompts to GPT-3.5 once more could lead to barely completely different outcomes.

I’ve offered the outputs of the fashions beneath all immediate settings, together with the sub-sampled take a look at set, few-shot immediate examples, and CoT immediate (from the MedPALM paper) for reproducing the scores reported on this article.

All papers referred to on this weblog submit are listed right here. Please let me know if I might need missed out any references, and I’ll add them!

[1] Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., … & Hu, X. (2023). Harnessing the facility of llms in apply: A survey on chatgpt and past. arXiv preprint arXiv:2304.13712.

[2] Radford, A., Wu, J., Baby, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language fashions are unsupervised multitask learners. OpenAI weblog, 1(8), 9.

[3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., … & Amodei, D. (2020). Language fashions are few-shot learners. Advances in neural info processing techniques, 33, 1877–1901.

[4] Wei, J., Bosma, M., Zhao, V. Y., Guu, Okay., Yu, A. W., Lester, B., … & Le, Q. V. (2021). Finetuned language fashions are zero-shot learners. arXiv preprint arXiv:2109.01652.

[5] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., … & Lowe, R. (2022). Coaching language fashions to comply with directions with human suggestions. Advances in Neural Info Processing Techniques, 35, 27730–27744.

[6] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., … & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in massive language fashions. Advances in Neural Info Processing Techniques, 35, 24824–24837.

[7] Thoppilan, R., De Freitas, D., Corridor, J., Shazeer, N., Kulshreshtha, A., Cheng, H. T., … & Le, Q. (2022). Lamda: Language fashions for dialog purposes. arXiv preprint arXiv:2201.08239.

[8] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., … & Fiedel, N. (2023). Palm: Scaling language modeling with pathways. Journal of Machine Studying Analysis, 24(240), 1–113.

[9] Jin, D., Pan, E., Oufattole, N., Weng, W. H., Fang, H., & Szolovits, P. (2021). What illness does this affected person have? a large-scale open area query answering dataset from medical exams. Utilized Sciences, 11(14), 6421.

[10] Touvron, H., Martin, L., Stone, Okay., Albert, P., Almahairi, A., Babaei, Y., … & Scialom, T. (2023). Llama 2: Open basis and fine-tuned chat fashions. arXiv preprint arXiv:2307.09288.

[11] https://platform.openai.com/docs/fashions/gpt-3-5-turbo

[12] Singhal, Okay., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., … & Natarajan, V. (2023). Giant language fashions encode scientific data. Nature, 620(7972), 172–180.

[13] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, Okay. R., & Cao, Y. (2022, September). ReAct: Synergizing Reasoning and Appearing in Language Fashions. In The Eleventh Worldwide Convention on Studying Representations.

[14] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., & Narasimhan, Okay. (2024). Tree of ideas: Deliberate drawback fixing with massive language fashions. Advances in Neural Info Processing Techniques, 36.