Last modified: Nov 10, 2026
Python Chunk a List into Batches: Easy Guide
Working with massive datasets in Python can quickly overwhelm your computer memory. When you try to process millions of items at once, your program might slow down or crash completely.
The best way to solve this problem is to split your data into smaller pieces. In programming, this process is called chunking a list into batches.
Batching allows you to process data piece by piece. It is essential for sending API requests, inserting records into databases, and training machine learning models.
In this guide, you will learn several clean and reliable ways to split a Python list into equal-sized chunks. We will cover beginner-friendly loops, fast list comprehensions, memory-saving generators, and built-in tools.
What Does Chunking a List Mean?
Chunking means taking one large list and dividing it into several smaller lists of a fixed size. Each smaller list is called a chunk or a batch.
For example, imagine you have a list of ten numbers. If you chunk that list into batches of three, you will get three lists of three numbers and one final list with the remaining single number.
Notice that the final batch might be smaller than your target chunk size. Python handles this gracefully without throwing an index error when you use the right techniques.
Method 1: Using List Comprehension and Slicing
The most popular way to split a list in Python is by combining list comprehension with list slicing. This approach is compact, readable, and very fast for small to medium lists.
To make this work, we use the range() function to step through the list indices by our desired batch size. Before doing this, we often check the total size of our data, which relies on the same logic used when you get the length of a list in Python.
# Define a sample list of items
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Set the desired batch size
batch_size = 3
# Use list comprehension to create chunks
chunks = [data[i:i + batch_size] for i in range(0, len(data), batch_size)]
# Print the resulting batches
print(chunks)[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]In this example, the loop starts at index zero and jumps forward by three on every step. At each step, it slices a sub-list from the current index up to the current index plus three.
If you want to understand how the colon operator extracts these sub-lists safely without going out of bounds, you can learn more about how to slice a list in Python.
Method 2: Using a Standard For Loop
If list comprehension feels hard to read, you can write the exact same logic using a traditional loop. This method is great for beginners because every step happens on its own line.
We start by creating an empty list to hold our final batches. Then, we add each slice to our result using the append() method.
# Create a list of user names
users = ["Alice", "Bob", "Charlie", "David", "Eve", "Frank", "Grace"]
chunk_size = 2
# Initialize an empty list for the batches
batched_users = []
# Loop through the list in steps of chunk_size
for i in range(0, len(users), chunk_size):
# Extract the current batch
batch = users[i:i + chunk_size]
# Add the batch to the main list
batched_users.append(batch)
print(batched_users)[['Alice', 'Bob'], ['Charlie', 'David'], ['Eve', 'Frank'], ['Grace']]This code produces a nested list structure. If you ever need to reverse this operation and turn those batches back into a single list, you can easily flatten a nested list in Python.
Method 3: Using a Generator Function with Yield
Both list comprehension and standard loops create all batches in memory at the exact same time. If your original list has ten million items, storing all those chunks at once can consume too much RAM.
To save memory, you should use a generator function. Instead of returning a complete list of batches, a generator uses the yield keyword to produce one batch at a time.
# Define a generator function to yield chunks
def chunk_generator(data_list, size):
for i in range(0, len(data_list), size):
yield data_list[i:i + size]
# Sample dataset
numbers = [10, 20, 30, 40, 50, 60, 70]
# Process one batch at a time in a loop
for batch in chunk_generator(numbers, 3):
print("Processing batch:", batch)Processing batch: [10, 20, 30]
Processing batch: [40, 50, 60]
Processing batch: [70]Generators use lazy evaluation. This means Python calculates the next batch only when your loop asks for it, keeping your memory footprint tiny.
Method 4: Using itertools.batched in Python 3.12+
If you are using Python 3.12 or newer, you do not need to write custom slicing logic anymore. Python introduced a built-in tool inside the standard library specifically for this task.
The batched() function from the itertools module splits any iterable into tuples of a specified length. It is written in C under the hood, making it extremely fast and memory efficient.
from itertools import batched
# Define a list of tasks
tasks = ["task1", "task2", "task3", "task4", "task5"]
# Split tasks into batches of 2
task_batches = list(batched(tasks, 2))
print(task_batches)[('task1', 'task2'), ('task3', 'task4'), ('task5',)]Notice two important details about this built-in tool. First, it returns an iterator, so we wrap it in list() if we want to print all items immediately.
Second, each individual chunk is returned as a tuple rather than a list. If you need to modify the items inside each batch, you will need to convert those tuples back into lists.
Method 5: Using NumPy array_split for Data Science
If you are working in data science or machine learning, you probably already have NumPy installed. NumPy provides a powerful function called array_split() that divides data into a specific number of chunks.
Be careful with how this method works. Instead of specifying the size of each chunk, you specify how many total chunks you want to create.
import numpy as np
# Create a list of numbers
scores = [85, 90, 78, 92, 88, 76, 95]
# Split the list into 3 roughly equal arrays
split_scores = np.array_split(scores, 3)
# Convert NumPy arrays back to standard Python lists
result = [chunk.tolist() for chunk in split_scores]
print(result)[[85, 90, 78], [92, 88], [76, 95]]NumPy automatically balances the sizes so the chunks are as equal as possible. However, you should avoid installing NumPy just for basic list chunking because it is a heavy external dependency.
Which Chunking Method Should You Choose?
Choosing the right approach depends on your Python version, your dataset size, and your project requirements. Here is a quick breakdown to help you decide.
- Use itertools.batched: Choose this if you are running Python 3.12 or higher. It is the cleanest, fastest, and most modern way to handle batching.
- Use a Generator Function: Choose this when working with massive datasets or streaming data. It prevents memory spikes by yielding one chunk at a time.
- Use List Comprehension: Choose this for older Python versions when working with small or medium lists that easily fit into memory.
- Use NumPy: Choose this only when you are already performing numerical computations and need a fixed number of balanced arrays.
Common Mistakes When Chunking Lists
Even though chunking is simple, beginners often run into a few common bugs. Keep these tips in mind to write error-free code.
First, never set your batch size to zero or a negative number. Passing zero as the step value in a range loop will raise a ValueError and crash your script.
Second, remember that slicing creates shallow copies of your list elements. If your list contains complex objects like dictionaries, modifying an item inside a batch will also change that item in the original list.
Finally, always check what happens when your list is empty. All the methods shown in this guide safely return an empty list when given an empty input, so you do not need extra checks for empty data.