Blog Post #148: The Best Way to Read Large Files: Iterating Line-by-Line

by

In Post #147, we explored the .read(), .readline(), and .readlines() methods. We learned that .read() and .readlines() are risky for large files because they load everything into memory, and that using .readline() can be a bit clumsy.

So, what is the best practice? How do we read a file line-by-line in a way that is both memory-efficient and easy to read?

The answer lies in a simple, elegant, and Pythonic pattern that treats the file object itself as an iterable.

The Clumsy Way: A while Loop with .readline()

Before we see the best way, let’s look at how we might construct a line-by-line reader using the tools we already have. We could use a while True loop and break when .readline() returns an empty string (which signals the end of the file).

# This works, but it's not the best way
with open("poem.txt", "r") as file:
    while True:
        line = file.readline()
        if not line: # An empty string is Falsy, so this condition is met at the end
            break
        print(line.strip())

This code works, but it’s not very elegant. It requires an infinite loop, a manual read, an if check, and a break. There’s a much cleaner way.

The Pythonic Solution: for line in file

The best practice for reading a file line-by-line in Python is to iterate directly over the file object in a for loop.

This works because a file object in Python is an iterable. When placed in a for loop, it behaves like a lazy generator, yielding one line at a time (including the trailing newline character) until the end of the file is reached.

Let’s use the same poem.txt file from our last post.

# The Pythonic, readable, and memory-efficient way
with open("poem.txt", "r") as file:
    for line in file:
        # The 'line' variable contains one line from the file on each iteration
        print(line.strip())

This code produces the exact same result as the while loop version, but with several key advantages:

  • Readable: The code for line in file: is intuitive and perfectly describes the action you are performing.
  • Memory-Efficient: Just like .readline(), this method only loads one line into memory at a time. It is safe for files of any size, from a few bytes to many gigabytes.
  • Concise: It’s the cleanest and shortest way to write this logic.

Putting It to Use

This pattern is the foundation for most text file processing tasks. Let’s say we want to find all the lines in our poem that contain the word “are”. We can easily combine our loop with an if statement.

search_term = "are"

# We can even use enumerate() (from Post #56) to get line numbers!
with open("poem.txt", "r") as file:
    for line_number, line in enumerate(file, start=1):
        if search_term in line:
            print(f"Found '{search_term}' on line {line_number}: {line.strip()}")

The output will be:

Found 'are' on line 1: Roses are red,
Found 'are' on line 2: Violets are blue,
Found 'are' on line 4: And so are you.

This shows how beautifully Python’s idiomatic tools compose with each other.

What’s Next?

You’ve now learned the most important pattern for reading files in Python. By treating the file object as an iterable in a for loop, you can process files of any size in a way that is memory-efficient, safe, and highly readable. This should be your default approach for reading text files line-by-line.

We’ve now mastered the art of reading files. In our next post, we’ll turn our attention to the other side of the coin: writing new content. In Post #149, we will explore writing and appending to text files.

Author

Debjeet Bhowmik

Experienced Cloud & DevOps Engineer with hands-on experience in AWS, GCP, Terraform, Ansible, ELK, Docker, Git, GitLab, Python, PowerShell, Shell, and theoretical knowledge on Azure, Kubernetes & Jenkins. In my free time, I write blogs on ckdbtech.com

Leave a Comment