A tutorial on why LLMs wrestle with math, and methods to resolve these limitations utilizing LangChain Brokers, OpenAI and Chainlit

On this tutorial, I’ll reveal methods to use LangChain brokers to create a customized Math utility utilising OpenAI’s GPT3.5 mannequin. For the applying frontend, I shall be utilizing Chainlit, an easy-to-use open-source Python framework. This generative math utility, let’s name it “Math Wiz”, is designed to assist customers with their math or reasoning/logic questions.

Giant Language Fashions (LLMs) are identified to be fairly dangerous at Math, in addition to reasoning duties, and this can be a frequent trait amongst many language fashions. There are a number of totally different causes for this:

• Lack of coaching knowledge: One causes is the restrictions of their coaching knowledge. Language fashions, skilled on huge textual content datasets, could lack ample mathematical issues and options. This may result in misinterpretations of numbers, forgetting necessary calculation steps, and a scarcity of quantitative reasoning expertise.
• Lack of numeric representations: One more reason is that LLMs are designed to know and generate textual content, working on tokens as an alternative of numeric values. Most text-based duties can have a number of cheap solutions. Nonetheless, math issues sometimes have just one appropriate answer.
• Generative nature: As a result of generative nature of those language fashions, producing persistently correct and exact solutions to math questions will be difficult for LLMs.

This makes the “Math drawback” the proper candidate for utilising LangChain brokers. Brokers are programs that use a language mannequin to work together with different instruments to interrupt down a fancy drawback (extra on this later). The code for this tutorial is accessible on my GitHub.

We are able to begin off by creating a brand new conda setting with python=3.11.

conda create -n math_assistant python=3.11

Activate the setting:

conda activate math_assistant

Subsequent, let’s set up all crucial libraries:

pip set up -r necessities.txt

Enroll at OpenAI and acquire your personal key to start out making calls to the gpt mannequin. After getting the important thing, create a .env file in your repository and retailer the OpenAI key:

OPENAI_API_KEY="your_openai_api_key"

The applying move for Math Wiz is printed within the flowchart beneath. The agent in our pipeline may have a set of instruments at its disposal that it will possibly use to reply a consumer question. The Giant Language Mannequin (LLM) serves because the “mind” of the agent, guiding its choices. When a consumer submits a query, the agent makes use of the LLM to pick out probably the most acceptable device or a mixture of instruments to supply a solution. If the agent determines it wants a number of instruments, it would additionally specify the order wherein the instruments are used.

The agent for our Math Wiz app shall be utilizing the next instruments:

1. Wikipedia Software: this device shall be liable for fetching the most recent info from Wikipedia utilizing the Wikipedia API. Whereas there are paid instruments and APIs out there that may be built-in inside LangChain, I’d be utilizing Wikipedia because the app’s on-line supply of data.
2. Calculator Software: this device can be liable for fixing a consumer’s math queries. This consists of something involving numerical calculations. For instance, if a consumer asks what the sq. root of 4 is, this device can be acceptable.
3. Reasoning Software: the ultimate device in our utility setup can be a reasoning device, liable for tackling logical/reasoning-based consumer queries. Any mathematical phrase issues must also be dealt with with this device.

Now that we’ve a tough utility design, we are able to started fascinated by constructing this utility.

LangChain brokers are designed to reinforce interplay with language fashions by offering an interface for extra advanced and interactive duties. We are able to consider an agent as an middleman between customers and a big language mannequin. Brokers search to interrupt down a seemingly advanced consumer question, that our LLM may not have the ability to deal with by itself, into simpler, actionable steps.

In our utility move, we outlined a number of totally different instruments that we want to use for our math utility. Primarily based on the consumer enter, the agent ought to resolve which of those instruments to make use of. If a device will not be required, it shouldn’t be used. LangChain brokers can simplify this for us. These brokers use a language mannequin to decide on a sequence of actions to take. Primarily, the LLM acts because the “mind” of the agent, guiding it on which device to make use of for a specific question, and wherein order. That is totally different from LangChain chains the place the sequence of actions are hardcoded in code. LangChain affords a large set of instruments that may be built-in with an agent. These instruments embody, and are usually not restricted to, on-line search instruments, API-based instruments, chain-based instruments and so on. For extra info on LangChain brokers and their sorts, see this.

Step 1

Create a chatbot.py script and import the required dependencies:

from langchain_openai import OpenAI
from langchain.chains import LLMMathChain, LLMChain
from langchain.prompts import PromptTemplate
from langchain_community.utilities import WikipediaAPIWrapper
from langchain.brokers.agent_types import AgentType
from langchain.brokers import Software, initialize_agent
from dotenv import load_dotenv
load_dotenv()

Step 2

Subsequent, we’ll outline our OpenAI-based Language Mannequin. LangChain affords the langchain-openai package deal which can be utilized to outline an occasion of the OpenAI mannequin. We shall be utilizing the gpt-3.5-turbo-instruct mannequin from OpenAI. The dotenv package deal would already be dealing with the API key so you don’t want to explicitly outline it right here:

llm = OpenAI(mannequin='gpt-3.5-turbo-instruct',
temperature=0)

We’d be utilizing this LLM each inside our math and reasoning chains and because the resolution maker for our agent.

Step 3

When developing your personal agent, you have to to supply it with a listing of instruments that it will possibly use. Apart from the precise perform that is known as, the Software consists of some different parameters:

• identify (str), is required and have to be distinctive inside a set of instruments offered to an agent
• description (str), is optionally available however really helpful, as it’s utilized by an agent to find out device use

We’ll now create our three instruments. The primary one would be the on-line device utilizing the Wikipedia API wrapper:

wikipedia = WikipediaAPIWrapper()
wikipedia_tool = Software(identify="Wikipedia",
func=wikipedia.run,
description="A useful gizmo for looking the Web 
to    discover info on world occasions, points, dates, years, and so on. Price 
utilizing for basic subjects. Use exact questions.")

Within the code above, we’ve outlined an occasion of the Wikipedia API wrapper. Afterwards, we’ve wrapped it inside a LangChain Software, with the identify, perform and outline.

Subsequent, let’s outline the device that we are going to be utilizing for calculating any numerical expressions. LangChain affords the LLMMathChain which makes use of the numexpr Python library to calculate mathematical expressions. It’s also necessary that we clearly outline what this device can be used for. The outline will be useful for the agent in deciding which device to make use of from a set of instruments for a specific consumer question. For our chain-based instruments, we shall be utilizing the Software.from_function() methodology.

problem_chain = LLMMathChain.from_llm(llm=llm)
math_tool = Software.from_function(identify="Calculator",
func=problem_chain.run,
description="Helpful for when you must reply questions 
about math. This device is just for math questions and nothing else. Solely enter
math expressions.")

Lastly, we shall be defining the device for logic/reasoning-based queries. We’ll first create a immediate to instruct the mannequin with executing the precise activity. Then we’ll create a easy LLMChain for this device, passing it the LLM and the immediate.

word_problem_template = """You're a reasoning agent tasked with fixing 
the consumer's logic-based questions. Logically arrive on the answer, and be 
factual. In your solutions, clearly element the steps concerned and provides the 
ultimate reply. Present the response in bullet factors. 
Query  {query} Reply"""
math_assistant_prompt = PromptTemplate(input_variables=["question"],
template=word_problem_template
)
word_problem_chain = LLMChain(llm=llm,
immediate=math_assistant_prompt)
word_problem_tool = Software.from_function(identify="Reasoning Software",
func=word_problem_chain.run,
description="Helpful for if you want 
to reply logic-based/reasoning questions.",
)

Step 4

We’ll now initialize our agent with the instruments we’ve created above. We may also specify the LLM to assist it select which instruments to make use of and in what order:

agent = initialize_agent(
instruments=[wikipedia_tool, math_tool, word_problem_tool],
llm=llm,
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
verbose=False,
handle_parsing_errors=True
)
print(agent.invoke(
{"enter": "I've 3 apples and 4 oranges. I give half of my oranges 
away and purchase two dozen new ones, alongwith three packs of 
strawberries. Every pack of strawberry has 30 strawberries. 
How  many whole items of fruit do I've on the finish?"}))

We shall be utilizing Chainlit, an open-source Python framework, to construct our utility. With Chainlit, you may construct conversational AI purposes with a number of easy traces of code. To get a deeper understanding of Chainlit functionalities and the way the app is about up, you may check out my article right here:

We shall be utilizing two decorator capabilities for our utility. These would be the @cl.on_chat_start and the @cl.on_message decorator capabilities. @cl.on_chat_start can be liable for wrapping all code that must be executed when a consumer session is initiated. @cl.on_message may have the code bits that we want to execute when a consumer sends in a question.

Let’s restructure our chatbot.py script with these two decorator capabilities. Let’s start by importing the chainlit package deal to our chatbot.py script:

import chainlit as cl

Subsequent, let’s write the wrapper perform round @cl.on_chat_start decorator perform. We shall be including our LLM, instruments and agent initialization code to this perform. We’ll retailer our agent in a variable contained in the consumer session, to be retrieved when a consumer sends in a message.

@cl.on_chat_start
def math_chatbot():
llm = OpenAI(mannequin='gpt-3.5-turbo-instruct',
temperature=0)
# immediate for reasoning based mostly device
word_problem_template = """You're a reasoning agent tasked with fixing t he consumer's logic-based questions. Logically arrive on the answer, and be factual. In your solutions, clearly element the steps concerned and provides the ultimate reply. Present the response in bullet factors. Query  {query} Reply"""
math_assistant_prompt = PromptTemplate(
input_variables=["question"],
template=word_problem_template
)
# chain for reasoning based mostly device
word_problem_chain = LLMChain(llm=llm,
immediate=math_assistant_prompt)
# reasoning based mostly device                              
word_problem_tool = Software.from_function(identify="Reasoning Software",
func=word_problem_chain.run,
description="Helpful for when you must reply logic-based/reasoning questions."
)
# calculator device for arithmetics
problem_chain = LLMMathChain.from_llm(llm=llm)
math_tool = Software.from_function(identify="Calculator",
func=problem_chain.run,
description="Helpful for when you must reply numeric questions. This device is just for math questions and nothing else. Solely enter math expressions, with out textual content",
)
# Wikipedia Software
wikipedia = WikipediaAPIWrapper()
wikipedia_tool = Software(
identify="Wikipedia",
func=wikipedia.run,
description="A useful gizmo for looking the Web to search out info on world occasions, points, dates, "
"years, and so on. Price utilizing for basic subjects. Use exact questions.",
)
# agent
agent = initialize_agent(
instruments=[wikipedia_tool, math_tool, word_problem_tool],
llm=llm,
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
verbose=False,
handle_parsing_errors=True
)
cl.user_session.set("agent", agent)

Subsequent, let’s outline the wrapper perform for the @cl.on_message decorator. This may include the code for when a consumer sends a question to our app. We’ll first be retrieving the agent we set at first of the session, after which name it with our consumer question asynchronously.

@cl.on_message
async def process_user_query(message: cl.Message):
agent = cl.user_session.get("agent")
response = await agent.acall(message.content material,
callbacks=[cl.AsyncLangchainCallbackHandler()])
await cl.Message(response["output"]).ship()

We are able to run our utility utilizing:

chainlit run chatbot.py

The applying must be out there at https://localhost:8000.

Let’s additionally edit the chainlit.md file in our repo. That is created routinely if you run the above command.

# Welcome to Math Wiz! 🤖
Hello there! 👋 I'm a reasoning device that can assist you along with your math or logic-based reasoning questions. How can I
assist at present?

Refresh the browser tab for the adjustments to take impact.

A demo for the app will be considered right here:

Let’s now validate the efficiency of our bot. Now we have not built-in any reminiscence into our bot, so every question must be its personal perform name. Let’s ask our app a number of math questions. For comparability, I’m attaching screenshots of every response for a similar question from each Chat GPT 3.5, and our Math Wiz app.

Arithmetic Questions

Query 1

What's the dice root of 625?
# appropriate reply = 8.5498

Our Math Wiz app was capable of reply appropriately. Nonetheless, ChatGPT’s response is inaccurate.

Query 2

what's dice root of 81? Multiply with 13.27, and subtract 5.
# appropriate reply = 52.4195

Our Math Wiz app was capable of appropriately reply this query too. Nonetheless, as soon as once more, ChatGPT’s response isn’t appropriate. Often, ChatGPT can reply math questions appropriately, however that is topic to immediate engineering and a number of inputs.

Reasoning Questions

Query 1

Let’s ask our app a number of reasoning/logic questions. A few of these questions have an arithmetic part to them. I’d count on the agent to resolve which device to make use of in every case.

I've 3 apples and 4 oranges. I give half of my oranges away and purchase two 
dozen new ones, alongwith three packs of strawberries. Every pack of 
strawberry has 30 strawberries. How  many whole items of fruit do I've at 
the tip?
# appropriate reply = 3 + 2 + 24 + 90 = 119

Our Math Wiz app was capable of appropriately reply this query. Nonetheless, ChatGPT’s response is inaccurate. It not solely sophisticated the reasoning steps unnecessarily but additionally failed to achieve the proper reply. Nonetheless, on a separate event, ChatGPT was capable of reply this query appropriately. That is in fact unreliable.

Query 2

Steve's sister is 10 years older than him. Steve was born when the chilly battle 
ended. When was Steve's sister born?
# appropriate reply = 1991 - 10 = 1981

Our Math Wiz app was capable of appropriately reply this query. ChatGPT’s response is as soon as once more incorrect. Despite the fact that it was capable of appropriately work out the 12 months the Chilly Battle ended, it tousled the calculation within the mathematical portion. Being 10 years older, the calculation for the sister’s age ought to have concerned subtracting from the 12 months that Steve was born. ChatGPT carried out an addition, signifying a scarcity of reasoning potential.

Query 3

give me the 12 months when Tom Cruise's High Gun launched raised to the ability 2
# appropriate reply = 1987**2 = 3944196

Our Math Wiz app was capable of appropriately reply this query. ChatGPT’s response is as soon as once more incorrect. Despite the fact that it was capable of appropriately work out the discharge date of the movie, the ultimate calculation was incorrect.

On this tutorial, we used LangChain brokers and instruments to create a math solver that might additionally deal with a consumer’s reasoning/logic questions. We noticed that our Math Wiz app appropriately answered all questions, nevertheless, most solutions given by ChatGPT have been incorrect. This can be a nice first step in constructing the device. The LLMMathChain can nevertheless fail based mostly on the enter we’re offering, if it incorporates string-based textual content. This may be tackled in a number of other ways, comparable to by creating error dealing with utilities to your code, including post-processing logic for the LLMMathChain, in addition to utilizing customized prompts. You could possibly additionally enhance the efficacy of the device by together with a search device for extra subtle and correct outcomes, since Wikipedia may not have up to date info generally. You’ll find the code from this tutorial on my GitHub.

In case you discovered this tutorial useful, think about supporting by giving it fifty claps. You’ll be able to comply with alongside as I share working demos, explanations and funky facet initiatives on issues within the AI house. Come say hello on LinkedIn and X! I share guides, code snippets and different helpful content material there. 👋