341 Intermediate

341: Buffered Stream — BufReader and BufWriter

Functional Programming

Tutorial

The Problem

Reading or writing one byte at a time with unbuffered I/O makes a system call for each operation — catastrophically slow for large files. BufReader and BufWriter add an in-memory buffer: reads fill the buffer in bulk (e.g., 8KB), and subsequent reads serve from the buffer without syscalls. Writers accumulate data in the buffer and flush in bulk. This optimization, crucial for text file processing and log writing, reduces system call overhead by orders of magnitude.

🎯 Learning Outcomes

• Use BufReader::new(reader) to wrap any reader with a 8KB internal buffer

• Use BufWriter::new(writer) to buffer writes and flush in bulk

• Process files line-by-line using BufRead::lines() — lazy, buffered

• Understand that flushing on BufWriter drop may silently discard errors — call flush() explicitly

Code Example

//! 341: Buffered Stream
//!
//! Demonstrates buffered I/O and string building in Rust using `BufReader`,
//! `BufWriter`, and `String` as analogs to OCaml's buffered channels and
//! `Buffer` module.

use std::io::{self, BufRead, BufReader, BufWriter, Read, Write};

// ---------------------------------------------------------------------------
// Approach 1: Buffered input — read all lines from a reader
// ---------------------------------------------------------------------------

/// Reads every line from `input` through a `BufReader`, returning them
/// in order. Mirrors OCaml's `read_lines_buffered`.
pub fn read_lines_buffered<R: Read>(input: R) -> io::Result<Vec<String>> {
    BufReader::new(input).lines().collect()
}

// ---------------------------------------------------------------------------
// Approach 2: Buffered output — write lines through a BufWriter
// ---------------------------------------------------------------------------

/// Writes each line followed by `'\n'` into `output` via a `BufWriter`,
/// flushing at the end. Mirrors OCaml's `write_lines_buffered`.
pub fn write_lines_buffered<W: Write>(output: W, lines: &[&str]) -> io::Result<()> {
    let mut writer = BufWriter::new(output);
    for line in lines {
        writeln!(writer, "{}", line)?;
    }
    writer.flush()
}

// ---------------------------------------------------------------------------
// Approach 3: String buffer — build a single string from many parts
// ---------------------------------------------------------------------------

/// Builds a newline-terminated string from `parts`, preallocating capacity
/// to avoid repeated allocations. Mirrors OCaml's `Buffer`-backed
/// `build_string`.
pub fn build_string(parts: &[&str]) -> String {
    let capacity = parts.iter().map(|s| s.len() + 1).sum();
    let mut buf = String::with_capacity(capacity);
    for part in parts {
        buf.push_str(part);
        buf.push('\n');
    }
    buf
}

/// Concatenates `parts` end-to-end without separators. Matches the second
/// OCaml test's `Buffer.add_string` behavior.
pub fn concat_parts(parts: &[&str]) -> String {
    let capacity = parts.iter().map(|s| s.len()).sum();
    let mut buf = String::with_capacity(capacity);
    for part in parts {
        buf.push_str(part);
    }
    buf
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn build_string_joins_with_newlines() {
        let result = build_string(&["hello", "world", "test"]);
        assert_eq!(result, "hello\nworld\ntest\n");
    }

    #[test]
    fn build_string_empty() {
        assert_eq!(build_string(&[]), "");
    }

    #[test]
    fn build_string_single() {
        assert_eq!(build_string(&["only"]), "only\n");
    }

    #[test]
    fn concat_parts_matches_buffer_add_string() {
        assert_eq!(concat_parts(&["abc", "def"]), "abcdef");
    }

    #[test]
    fn concat_parts_empty() {
        assert_eq!(concat_parts(&[]), "");
    }

    #[test]
    fn read_lines_buffered_roundtrip() {
        let input = b"alpha\nbeta\ngamma\n" as &[u8];
        let lines = read_lines_buffered(input).unwrap();
        assert_eq!(lines, vec!["alpha", "beta", "gamma"]);
    }

    #[test]
    fn read_lines_buffered_empty() {
        let input: &[u8] = b"";
        let lines = read_lines_buffered(input).unwrap();
        assert!(lines.is_empty());
    }

    #[test]
    fn read_lines_buffered_no_trailing_newline() {
        let input = b"one\ntwo" as &[u8];
        let lines = read_lines_buffered(input).unwrap();
        assert_eq!(lines, vec!["one", "two"]);
    }

    #[test]
    fn write_lines_buffered_produces_newline_terminated_output() {
        let mut out: Vec<u8> = Vec::new();
        write_lines_buffered(&mut out, &["hello", "world"]).unwrap();
        assert_eq!(String::from_utf8(out).unwrap(), "hello\nworld\n");
    }

    #[test]
    fn write_lines_buffered_empty_is_empty() {
        let mut out: Vec<u8> = Vec::new();
        write_lines_buffered(&mut out, &[]).unwrap();
        assert!(out.is_empty());
    }

    #[test]
    fn write_then_read_roundtrip() {
        let lines = ["first", "second", "third"];
        let mut buf: Vec<u8> = Vec::new();
        write_lines_buffered(&mut buf, &lines).unwrap();
        let read_back = read_lines_buffered(buf.as_slice()).unwrap();
        assert_eq!(read_back, lines);
    }
}

(* 341: Buffered Stream *)
(* OCaml channels are already buffered by default *)

(* Approach 1: Buffered input *)
let read_lines_buffered ic =
  let rec aux acc =
    match input_line ic with
    | line -> aux (line :: acc)
    | exception End_of_file -> List.rev acc
  in
  aux []

(* Approach 2: Buffered output *)
let write_lines_buffered oc lines =
  List.iter (fun line ->
    output_string oc line;
    output_char oc '\n'
  ) lines;
  flush oc

(* Approach 3: Buffer module for string building *)
let build_string parts =
  let buf = Buffer.create 256 in
  List.iter (fun s ->
    Buffer.add_string buf s;
    Buffer.add_char buf '\n'
  ) parts;
  Buffer.contents buf

(* Tests *)
let () =
  let result = build_string ["hello"; "world"; "test"] in
  assert (result = "hello\nworld\ntest\n");
  let buf = Buffer.create 16 in
  Buffer.add_string buf "abc";
  Buffer.add_string buf "def";
  assert (Buffer.contents buf = "abcdef");
  Printf.printf "✓ All tests passed\n"

Key Differences

Default buffering: OCaml's In_channel / Out_channel are buffered by default; Rust's File is unbuffered — BufReader/BufWriter must be added explicitly.

Drop flush: BufWriter's Drop implementation calls flush(), but ignores errors — always call flush() explicitly when error handling matters.

Buffer size: Default buffer is 8KB; use BufReader::with_capacity(size) for tuned buffer sizes.

Async buffered I/O: tokio::io::BufReader / BufWriter are the async-aware equivalents for use with Tokio async I/O traits.

OCaml Approach

OCaml's In_channel uses buffered I/O by default. In_channel.input_line is the standard line-by-line reader:

let count_lines path =
  let ic = In_channel.open_text path in
  let count = ref 0 in
  (try while true do ignore (In_channel.input_line_exn ic); incr count done
   with End_of_file -> ());
  In_channel.close ic; !count

Full Source

//! 341: Buffered Stream
//!
//! Demonstrates buffered I/O and string building in Rust using `BufReader`,
//! `BufWriter`, and `String` as analogs to OCaml's buffered channels and
//! `Buffer` module.

use std::io::{self, BufRead, BufReader, BufWriter, Read, Write};

// ---------------------------------------------------------------------------
// Approach 1: Buffered input — read all lines from a reader
// ---------------------------------------------------------------------------

/// Reads every line from `input` through a `BufReader`, returning them
/// in order. Mirrors OCaml's `read_lines_buffered`.
pub fn read_lines_buffered<R: Read>(input: R) -> io::Result<Vec<String>> {
    BufReader::new(input).lines().collect()
}

// ---------------------------------------------------------------------------
// Approach 2: Buffered output — write lines through a BufWriter
// ---------------------------------------------------------------------------

/// Writes each line followed by `'\n'` into `output` via a `BufWriter`,
/// flushing at the end. Mirrors OCaml's `write_lines_buffered`.
pub fn write_lines_buffered<W: Write>(output: W, lines: &[&str]) -> io::Result<()> {
    let mut writer = BufWriter::new(output);
    for line in lines {
        writeln!(writer, "{}", line)?;
    }
    writer.flush()
}

// ---------------------------------------------------------------------------
// Approach 3: String buffer — build a single string from many parts
// ---------------------------------------------------------------------------

/// Builds a newline-terminated string from `parts`, preallocating capacity
/// to avoid repeated allocations. Mirrors OCaml's `Buffer`-backed
/// `build_string`.
pub fn build_string(parts: &[&str]) -> String {
    let capacity = parts.iter().map(|s| s.len() + 1).sum();
    let mut buf = String::with_capacity(capacity);
    for part in parts {
        buf.push_str(part);
        buf.push('\n');
    }
    buf
}

/// Concatenates `parts` end-to-end without separators. Matches the second
/// OCaml test's `Buffer.add_string` behavior.
pub fn concat_parts(parts: &[&str]) -> String {
    let capacity = parts.iter().map(|s| s.len()).sum();
    let mut buf = String::with_capacity(capacity);
    for part in parts {
        buf.push_str(part);
    }
    buf
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn build_string_joins_with_newlines() {
        let result = build_string(&["hello", "world", "test"]);
        assert_eq!(result, "hello\nworld\ntest\n");
    }

    #[test]
    fn build_string_empty() {
        assert_eq!(build_string(&[]), "");
    }

    #[test]
    fn build_string_single() {
        assert_eq!(build_string(&["only"]), "only\n");
    }

    #[test]
    fn concat_parts_matches_buffer_add_string() {
        assert_eq!(concat_parts(&["abc", "def"]), "abcdef");
    }

    #[test]
    fn concat_parts_empty() {
        assert_eq!(concat_parts(&[]), "");
    }

    #[test]
    fn read_lines_buffered_roundtrip() {
        let input = b"alpha\nbeta\ngamma\n" as &[u8];
        let lines = read_lines_buffered(input).unwrap();
        assert_eq!(lines, vec!["alpha", "beta", "gamma"]);
    }

    #[test]
    fn read_lines_buffered_empty() {
        let input: &[u8] = b"";
        let lines = read_lines_buffered(input).unwrap();
        assert!(lines.is_empty());
    }

    #[test]
    fn read_lines_buffered_no_trailing_newline() {
        let input = b"one\ntwo" as &[u8];
        let lines = read_lines_buffered(input).unwrap();
        assert_eq!(lines, vec!["one", "two"]);
    }

    #[test]
    fn write_lines_buffered_produces_newline_terminated_output() {
        let mut out: Vec<u8> = Vec::new();
        write_lines_buffered(&mut out, &["hello", "world"]).unwrap();
        assert_eq!(String::from_utf8(out).unwrap(), "hello\nworld\n");
    }

    #[test]
    fn write_lines_buffered_empty_is_empty() {
        let mut out: Vec<u8> = Vec::new();
        write_lines_buffered(&mut out, &[]).unwrap();
        assert!(out.is_empty());
    }

    #[test]
    fn write_then_read_roundtrip() {
        let lines = ["first", "second", "third"];
        let mut buf: Vec<u8> = Vec::new();
        write_lines_buffered(&mut buf, &lines).unwrap();
        let read_back = read_lines_buffered(buf.as_slice()).unwrap();
        assert_eq!(read_back, lines);
    }
}

(* 341: Buffered Stream *)
(* OCaml channels are already buffered by default *)

(* Approach 1: Buffered input *)
let read_lines_buffered ic =
  let rec aux acc =
    match input_line ic with
    | line -> aux (line :: acc)
    | exception End_of_file -> List.rev acc
  in
  aux []

(* Approach 2: Buffered output *)
let write_lines_buffered oc lines =
  List.iter (fun line ->
    output_string oc line;
    output_char oc '\n'
  ) lines;
  flush oc

(* Approach 3: Buffer module for string building *)
let build_string parts =
  let buf = Buffer.create 256 in
  List.iter (fun s ->
    Buffer.add_string buf s;
    Buffer.add_char buf '\n'
  ) parts;
  Buffer.contents buf

(* Tests *)
let () =
  let result = build_string ["hello"; "world"; "test"] in
  assert (result = "hello\nworld\ntest\n");
  let buf = Buffer.create 16 in
  Buffer.add_string buf "abc";
  Buffer.add_string buf "def";
  assert (Buffer.contents buf = "abcdef");
  Printf.printf "✓ All tests passed\n"

✓ Tests Rust test suite

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn build_string_joins_with_newlines() {
        let result = build_string(&["hello", "world", "test"]);
        assert_eq!(result, "hello\nworld\ntest\n");
    }

    #[test]
    fn build_string_empty() {
        assert_eq!(build_string(&[]), "");
    }

    #[test]
    fn build_string_single() {
        assert_eq!(build_string(&["only"]), "only\n");
    }

    #[test]
    fn concat_parts_matches_buffer_add_string() {
        assert_eq!(concat_parts(&["abc", "def"]), "abcdef");
    }

    #[test]
    fn concat_parts_empty() {
        assert_eq!(concat_parts(&[]), "");
    }

    #[test]
    fn read_lines_buffered_roundtrip() {
        let input = b"alpha\nbeta\ngamma\n" as &[u8];
        let lines = read_lines_buffered(input).unwrap();
        assert_eq!(lines, vec!["alpha", "beta", "gamma"]);
    }

    #[test]
    fn read_lines_buffered_empty() {
        let input: &[u8] = b"";
        let lines = read_lines_buffered(input).unwrap();
        assert!(lines.is_empty());
    }

    #[test]
    fn read_lines_buffered_no_trailing_newline() {
        let input = b"one\ntwo" as &[u8];
        let lines = read_lines_buffered(input).unwrap();
        assert_eq!(lines, vec!["one", "two"]);
    }

    #[test]
    fn write_lines_buffered_produces_newline_terminated_output() {
        let mut out: Vec<u8> = Vec::new();
        write_lines_buffered(&mut out, &["hello", "world"]).unwrap();
        assert_eq!(String::from_utf8(out).unwrap(), "hello\nworld\n");
    }

    #[test]
    fn write_lines_buffered_empty_is_empty() {
        let mut out: Vec<u8> = Vec::new();
        write_lines_buffered(&mut out, &[]).unwrap();
        assert!(out.is_empty());
    }

    #[test]
    fn write_then_read_roundtrip() {
        let lines = ["first", "second", "third"];
        let mut buf: Vec<u8> = Vec::new();
        write_lines_buffered(&mut buf, &lines).unwrap();
        let read_back = read_lines_buffered(buf.as_slice()).unwrap();
        assert_eq!(read_back, lines);
    }
}

Deep Comparison

Core Insight

Buffering reduces system calls — wrapping raw I/O in buffers is essential for performance

OCaml Approach

• See example.ml for implementation

Rust Approach

• See example.rs for implementation

Comparison Table

Feature	OCaml	Rust
See	example.ml	example.rs

Exercises

Benchmark reading a large file byte-by-byte vs line-by-line with BufReader — measure time and system call count.

Implement a log file writer that uses BufWriter with periodic explicit flushes every 1000 lines.

Use BufReader::lines() to process a CSV file lazily, parsing each line into a Vec<String> of fields.

Open Source Repos

functional-rust

View the source for this example on GitHub — OCaml and Rust side by side in the repo.

Rust