File Handling in Python: Modes, Buffering and the with Statement

File Handling in Python: Reading and Writing Data

Every UPI app on your phone — the one your family uses to pay a vegetable vendor or split a restaurant bill — depends on a boring, unglamorous piece of engineering underneath the QR code and the OTP: a durable, append-only ledger of every transaction attempt, written to a file, that survives even if the server handling your payment crashes one second after debiting your account. Get the file-handling logic wrong — write instead of append, forget to flush, assume a write is durable the instant the line executes — and you either lose a customer's ₹12,000 transfer record or double-count it during reconciliation. This chapter builds that logic from first principles: what a file actually is to the operating system, how Python's open() sits on top of it, and why the gap between "I called write()" and "the data is safely on disk" is the single most consequential thing to understand about file I/O.

What a file actually is, and what open() gives you

A file is not a Python concept. It is a service the operating system provides: a named region of bytes on persistent storage (an SSD or HDD), tracked by the filesystem through a structure usually called an inode, and addressed while a program is using it through a small integer called a file descriptor. When you call open("upi_ledger.txt", "a"), Python asks the OS to open that file and hands back a file descriptor; Python then wraps that descriptor in a TextIOWrapper object (what you casually call "the file object") that adds two things the raw OS handle doesn't give you directly: text encoding/decoding, and an internal memory buffer so your program isn't making an expensive system call to the kernel on every single write() or read().

That buffer is the part students consistently misjudge, and it is the mechanism the diagram below depicts. When you call f.write(line), the string is encoded to bytes and copied into a small buffer that lives in your program's own memory — not on disk, not even necessarily visible to the operating system yet. Only when that buffer fills up, or you explicitly call f.flush(), or the file is closed, does Python hand the bytes to the OS, which places them in its own page cache (kernel-managed RAM). The OS then decides, on its own schedule, when to actually push those cached pages to the physical platters or NAND cells — unless a program forces the issue with os.fsync(). Three layers of buffering sit between "I wrote a line" and "that line survives a power cut."

Opening and closing files: modes and the with statement

open(filename, mode, encoding=...) is the entry point for everything in this chapter. The mode string tells Python both the access pattern and whether it's dealing with text or bytes:

• "r" — read only; raises FileNotFoundError if the file doesn't exist.
• "w" — write only; creates the file if absent, and truncates it to zero bytes first if it already exists.
• "a" — append; creates the file if absent, and writes are always added after existing content, never overwriting it.
• "x" — exclusive creation; raises FileExistsError if the file is already there, useful when accidentally overwriting a file would be a serious bug.
• "r+" — read and write, without truncating.
• Append "b" to any of the above ("rb", "wb") for binary mode, where you read and write raw bytes objects instead of decoded str text — covered later in this chapter.

A file handle is a limited OS resource, and forgetting to close it leaks descriptors and — worse — can leave the buffer described above never flushed. Python's fix is the with statement, which turns open() into a context manager:

with open("upi_ledger.txt", "a", encoding="utf-8") as f:
    f.write("TXN10234|Rohan Iyer|1500.00|SUCCESS\n")

The guarantee with gives you is not stylistic. Python calls f.close() automatically when the block ends — including when it ends because an exception was raised partway through. Without with, an exception between open() and a manual f.close() skips the close entirely, and any buffered-but-unflushed data is at risk of being lost. Every example in this chapter uses with for exactly this reason.

Reading text: read(), readline(), readlines(), and iteration

Python gives you four ways to pull text out of a file object, and they are not interchangeable at scale:

with open("upi_ledger.txt", "r", encoding="utf-8") as f:
    whole_thing = f.read()          # one big string, entire file
    # OR
    one_line = f.readline()         # one string, up to and including "\n"
    # OR
    all_lines = f.readlines()       # a list of strings, one per line
    # OR
    for line in f:                  # iterate lazily, one line at a time
        process(line)

f.read() and f.readlines() both force Python to hold the entire file's content in memory at once — as one long string, or as a list of per-line strings respectively — so their memory cost is O(n) in the size of the file. For a small config file that's irrelevant. For a 50-million-line booking log, readlines() can exhaust available RAM before your loop even starts. Iterating directly over the file object (for line in f:) instead pulls one line at a time through the object's internal buffered reader, so memory use stays O(1) — a few kilobytes of buffer regardless of whether the file is 10 lines or 10 gigabytes. This is the pattern used in the worked example below, and it's the pattern to default to whenever you don't specifically need random access to every line at once.

Writing and appending: write() and writelines()

f.write(string) writes exactly the string you give it — it does not add a newline automatically, which is why every example in this chapter ends its lines with an explicit "\n". f.writelines(list_of_strings) writes each string in the list back-to-back, again with no automatic newlines inserted between them — the newlines have to already be part of each string, or the output runs together on one line. The choice between "w" and "a" mode, covered above, is what actually determines whether a write destroys prior content or extends it — a distinction Active Recall question 3 below turns into a concrete production bug.

Worked example: a UPI transaction ledger, fully traced

Consider a simplified payment-gateway logger. Each hour, a batch of transaction attempts needs to be appended to a plain-text ledger, and a separate function needs to read that ledger back and compute settlement totals — the exact shape of a reconciliation job a bank or payment aggregator runs.

transactions = [
    ("TXN10234", "Rohan Iyer", 1500.00, "SUCCESS"),
    ("TXN10235", "Ayesha Khan", 2999.50, "SUCCESS"),
    ("TXN10236", "Devendra Rao", 450.00, "FAILED"),
    ("TXN10237", "Priya Nair", 12000.00, "SUCCESS"),
]

def log_transactions(transactions, filename):
    with open(filename, "a", encoding="utf-8") as f:
        for txn_id, name, amount, status in transactions:
            f.write(f"{txn_id}|{name}|{amount:.2f}|{status}\n")

log_transactions(transactions, "upi_ledger.txt")

After this call, upi_ledger.txt holds exactly four lines, each pipe-separated, e.g. TXN10234|Rohan Iyer|1500.00|SUCCESS. Now the reconciliation side:

def summarize_ledger(filename):
    total_success = 0.0
    count_success = 0
    count_failed = 0
    with open(filename, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            txn_id, name, amount, status = line.split("|")
            amount = float(amount)
            if status == "SUCCESS":
                total_success += amount
                count_success += 1
            elif status == "FAILED":
                count_failed += 1
    return total_success, count_success, count_failed

total, ok, failed = summarize_ledger("upi_ledger.txt")
print(f"Successful: {ok}, Failed: {failed}, Total settled: ₹{total:.2f}")

Trace it line by line: the loop reads four lines. Line 1 splits into ("TXN10234", "Rohan Iyer", "1500.00", "SUCCESS"); status is "SUCCESS", so total_success becomes 1500.00 and count_success becomes 1. Line 2 adds 2999.50, giving total_success = 4499.50, count_success = 2. Line 3 has status "FAILED", so only count_failed increments, to 1; total_success is untouched. Line 4 adds 12000.00, giving total_success = 16499.50, count_success = 3. The function returns (16499.50, 3, 1), and the print statement outputs exactly:

Successful: 3, Failed: 1, Total settled: ₹16499.50

Two design choices here are worth naming explicitly. First, split("|") only works safely because the delimiter can never appear inside a name or amount — a fragile assumption for a real logging format, which is precisely the problem the csv module (below) solves properly. Second, the reading function guards with if not line: continue as a defensive habit for files that may contain genuinely blank lines (e.g. hand-edited or concatenated logs) — note that a well-formed trailing \n at end-of-file does not itself produce an extra empty string when iterating with for line in f: (Python's line iterator stops cleanly at EOF); the guard matters if a stray blank line is ever present in the data, or if the file is later read a different way (e.g. f.read().split("\n"), which does produce a trailing empty entry).

Random access: seek() and tell()

Reading line by line is sequential — you can't jump to the middle of a file without walking through everything before it, unless you use f.tell() (report the current byte offset) and f.seek(offset) (jump to a byte offset) together. Extending the ledger example:

with open("upi_ledger.txt", "r", encoding="utf-8") as f:
    line1 = f.readline()
    checkpoint = f.tell()          # byte offset right after line 1
    line2 = f.readline()
    line3 = f.readline()
    f.seek(checkpoint)             # jump back to the start of line 2
    reread_line2 = f.readline()
    print(reread_line2 == line2)

This prints True: seeking back to checkpoint and reading again reproduces line2 exactly, because checkpoint is the exact byte position where line 2 begins. Note that in text mode, tell() returns a byte offset (accounting for UTF-8 encoding and, on Windows, newline translation), not a character count — treating it as "characters read so far" is a subtle, common bug. In binary mode ("rb"), offsets are unambiguous raw byte positions with no such translation layer, which is why systems that need precise, fixed-length random access to records — a simplified version of how database index files locate rows without scanning the whole table — typically use binary mode with seek().

Structured data: the csv module

Pipe-splitting works until a field itself contains the delimiter. An attendance register that stores names as "Surname, First Name" is a good example — a literal comma inside a comma-separated value. Hand-rolled string joining breaks on this immediately; Python's csv module handles quoting correctly:

import csv

attendance = [
    {"roll_no": 1, "name": "Rao, Ananya", "present": "Y"},
    {"roll_no": 2, "name": "Iyer, Karthik", "present": "N"},
    {"roll_no": 3, "name": "Fernandes, Priya", "present": "Y"},
]

with open("attendance.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["roll_no", "name", "present"])
    writer.writeheader()
    writer.writerows(attendance)

with open("attendance.csv", "r", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    present_count = sum(1 for row in reader if row["present"] == "Y")

print(present_count)

csv.DictWriter automatically wraps "Rao, Ananya" in quotes in the output file ("Rao, Ananya"), so csv.DictReader reads it back as one field, not two. Tracing the read: row 1 has present == "Y" (counts), row 2 has "N" (doesn't count), row 3 has "Y" (counts) — present_count is 2, which is exactly what gets printed. The newline="" argument in both open() calls is not decorative — it disables Python's own newline translation so the csv module can manage line endings itself, which the official documentation specifically requires to avoid stray blank rows.

Binary files: when text mode is the wrong tool

Not all data is naturally text. Serializing a Python object — a dictionary, a list of objects — for later reuse calls for binary mode and the pickle module:

import pickle

student_marks = {"Ananya Rao": [92, 88, 95], "Karthik Iyer": [76, 81, 79]}

with open("marks.pkl", "wb") as f:
    pickle.dump(student_marks, f)

with open("marks.pkl", "rb") as f:
    loaded_marks = pickle.load(f)

print(loaded_marks["Ananya Rao"])

This prints [92, 88, 95] — the dictionary round-trips through the file exactly, including nested lists, which plain text formats can't do without writing your own parser. The real system-design tradeoff: pickle files are Python-specific and not human-readable (use CSV or JSON when another program, or a human, needs to read the data), and — a genuine security concern, not a textbook footnote — pickle.load() on a file from an untrusted source can execute arbitrary code during deserialization, so it should never be used to parse a file your program didn't itself create.

The common misconception: "write() means it's saved"

The single most common misconception at this level is treating f.write(line) as equivalent to "the line is now durably on disk." It is not. As the diagram shows, write() only guarantees the data has been copied into your program's own in-memory buffer. If the process is killed — a crash, a power failure, a container getting OOM-killed — before that buffer is flushed to the OS and the OS in turn writes it to storage, the data is gone, even though the line of code that called write() already returned successfully with no error. The correct mental model: a write is provisional until flush() or close() runs (which is exactly what exiting a with block guarantees), and even then the OS's own page cache may hold it briefly before an actual disk write — which is why systems with strict durability requirements call os.fsync() explicitly rather than trusting close() alone.

How Python's file buffering actually moves your data

Active recall

Attempt these before reading the answers.

1. You call open("upi_ledger.txt", "r") before log_transactions has ever run, so the file doesn't exist yet. What happens, and how would you handle it safely?
2. An IRCTC-scale booking log has 50 million lines. Why does summarize_ledger's for line in f: pattern survive reading that file, while replacing it with lines = f.readlines() followed by a loop over lines would likely crash on a normal laptop?
3. log_transactions opens the file with mode "a" so each hourly batch appends. A bug changes "a" to "w", while everything else — the transaction list, the call site, summarize_ledger — stays untouched, and the script still runs once per hour. After the 09:00, 10:00, and 11:00 runs, what does upi_ledger.txt contain, and what does summarize_ledger("upi_ledger.txt") report at 11:05?
4. Using the checkpoint pattern shown for upi_ledger.txt, would f.tell() and f.seek() still behave the same way if the file had been opened as "rb" instead of "r"?
5. If the attendance file were written with f.write(f"{roll},{name},{present}\n") instead of csv.DictWriter, what breaks for the student "Rao, Ananya", and how does the csv module prevent it?
6. io.DEFAULT_BUFFER_SIZE is 8192 bytes on most CPython installs. If each ledger line averages 46 bytes, roughly how many lines can sit in the user-space buffer before Python is forced to flush it purely because the buffer filled up?

Answers.

1. Python raises FileNotFoundError — "r" mode never creates a file. Handle it with try/except FileNotFoundError around the open, or guarantee the file exists first (e.g. open it once in "a" mode at startup, since "a" creates the file if it's missing without touching any existing content).

2. readlines() builds the entire list of 50 million line-strings in memory before your loop runs even one iteration — memory cost scales with total file size (O(n)), easily reaching multiple gigabytes and triggering a MemoryError or heavy swapping. for line in f: pulls one line at a time through the file object's internal buffered reader, so it only ever holds one line plus a few kilobytes of buffer in memory (O(1)), regardless of whether the file has 10 lines or 50 million.

3. "w" truncates the file to zero bytes the instant it's opened, before any writing happens. So the 10:00 run's open(..., "w") erases the entire 09:00 batch before writing 10:00's four lines; the 11:00 run does the same to 10:00's data. At 11:05, upi_ledger.txt holds only the 11:00 batch — four lines — and summarize_ledger reports exactly (16499.50, 3, 1), identical to the single-batch trace worked through earlier in this chapter. The 09:00 and 10:00 transactions are permanently gone, and nothing in the code raises an error to signal it — this is precisely why the choice between "a" and "w" is a data-loss bug surface, not a style preference.

4. The mechanics work the same — seek() still jumps to the saved offset and re-reads correctly — but the meaning of the offset differs. In text mode, tell() returns a byte offset that already accounts for UTF-8 encoding (and newline translation on Windows), so it should never be treated as a character count. In binary mode, offsets are raw, unambiguous byte positions with no encoding layer involved, which is why fixed-length binary record files use "rb"/"wb" with seek() for precise, predictable direct access.

5. Manual comma-joining produces the line 2,Rao, Ananya,Y. A naive comma-split reader sees four fields — 2, Rao, " Ananya", Y — instead of three, shifting every column after the name. csv.writer (used internally by DictWriter) detects the embedded delimiter and wraps the field in quotes, writing 2,"Rao, Ananya",Y; csv.reader/DictReader correctly interprets the quoted segment as one field on read.

6. 8192 ÷ 46 ≈ 178 lines. A script logging fewer than roughly 178 transactions between explicit flushes may never trigger an automatic flush from the buffer filling up on its own — the data only becomes visible to the OS when the with block exits and calls close(), or when a long-enough run finally fills the buffer. This is the concrete, numeric version of the misconception this chapter opened with: "the loop finished" is not the same guarantee as "the file was closed."

Think About It

Think about this: How would you explain file handling in python: modes, buffering and the with statement to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where file handling in python: modes, buffering and the with statement is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting file handling in python: modes, buffering and the with statement to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind file handling in python: modes, buffering and the with statement, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.