How to Batch LangChain Requests in Python

📘 Introduction

Many AI apps need to send more than one request to a model. You might classify several support tickets, summarize a list of notes, rewrite product descriptions, or generate short explanations for many topics.

You could call invoke() again and again in a loop, but LangChain also gives you batching methods for running multiple inputs through the same model or chain. In this tutorial, you will learn how to use batch() and abatch() with ChatOpenAI in Python.

💡 Why batching matters

Batching helps when each input can be processed independently. Instead of manually writing a loop and collecting each response yourself, you can pass a list of inputs to the same runnable and receive a list of outputs back.

This is useful for AI engineering workflows such as labeling examples, preparing synthetic data, evaluating prompts, summarizing multiple text snippets, or running the same chain over many user questions.

✅ Prerequisites

Before getting started, make sure you have:

☑️ Python installed
☑️ Basic Python knowledge
☑️ An OpenAI API key
☑️ A terminal or command prompt

⚙️1️⃣ Create a project folder

Create a new local project folder:

mkdir langchain-batch-requests
cd langchain-batch-requests

🐍2️⃣ Create a virtual environment

Create and activate a virtual environment:

python -m venv .venv
source .venv/bin/activate

On Windows, activate it with:

.venv\Scripts\activate

📦3️⃣ Install libraries

Install the packages for LangChain and OpenAI:

pip install langchain-core langchain-openai

🔐4️⃣ Set your API key

Set your OpenAI API key as an environment variable. Replace the placeholder with your own key and never put real secrets into code files.

export OPENAI_API_KEY="your_api_key_here"

On Windows PowerShell, use:

$env:OPENAI_API_KEY="your_api_key_here"

🧪5️⃣ Start with repeated invoke calls

Before using batching, let us start with a normal loop. This shows the problem we are improving.

Create a file named loop_invoke.py and add this code:

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4.1-mini", temperature=0)

topics = [
    "LangChain batching",
    "AI prompt evaluation",
    "model latency",
]

for topic in topics:
    response = model.invoke(f"Explain {topic} in one short sentence.")
    print(f"{topic}: {response.content}")

Run the script:

python loop_invoke.py

This works, but you are manually managing the loop. For three inputs that is fine. For dozens or hundreds of independent inputs, LangChain's batch methods make the workflow cleaner.