761: Custom Serialization for Complex Types

Difficulty: 3 Level: Intermediate Implement `Serialize` manually for enums with payloads, `Option<T>`, and `Vec<T>` — shapes that derive macros handle awkwardly.

The Problem This Solves

`#[derive(Serialize)]` works perfectly for simple structs. It struggles with enums that have complex payloads you want serialized in a specific wire format, newtypes wrapping collections, and cases where you need length-prefixed encoding or custom delimiters. When you need to control exactly how bytes land on the wire — for network protocols, compact binary formats, or compatibility with legacy systems — you implement `Serialize` manually. This is the pattern behind custom `serde::Serialize` impls in production: Tokio's codec types, custom binary protocol handlers, and any library that needs a specific wire format rather than `serde_json`'s default choices. Once you understand how to hand-roll a serializer for an enum, the `serde` docs on custom serializers become clear. The key challenge with enums is that each variant has a different shape. You need a tag (to identify the variant on deserialization) and then variant-specific payload serialization. Length-prefixed strings (`"5:hello"`) are a common technique for variable-length fields in binary protocols.

The Intuition

Serialization is a function `value → bytes/string`. For enums, precede each variant's data with a tag character (`'C'` for Circle, `'R'` for Rectangle, `'P'` for Point). For strings, prefix with the byte count so the deserializer knows where the string ends. For `Option<T>`, use `'N'` for None and `'S' + payload` for Some. This self-describing approach lets the deserializer parse without knowing the type in advance.

How It Works in Rust

pub trait Serialize {
 fn serialize(&self, out: &mut String);
}

// Enum with tagged payloads
impl Serialize for Shape {
 fn serialize(&self, out: &mut String) {
     match self {
         Shape::Circle(r) => {
             out.push_str("C|");    // tag: Circle
             r.serialize(out);      // delegate to f64's Serialize
         }
         Shape::Rectangle { width, height } => {
             out.push_str("R|");
             width.serialize(out);
             out.push('|');
             height.serialize(out);
         }
         Shape::Point => out.push('P'),   // no payload needed
     }
 }
}

// Length-prefixed string encoding: "5:hello"
impl Serialize for String {
 fn serialize(&self, out: &mut String) {
     write!(out, "{}:{}", self.len(), self).unwrap();  // length:data
 }
}

// Option<T>: N for None, S + payload for Some
impl<T: Serialize> Serialize for Option<T> {
 fn serialize(&self, out: &mut String) {
     match self {
         None    => out.push('N'),
         Some(v) => { out.push('S'); v.serialize(out); }
     }
 }
}

// Vec<Shape> with count header for framing
fn serialize_shapes(shapes: &[Shape]) -> String {
 let mut out = String::new();
 write!(out, "{}\n", shapes.len()).unwrap();  // count header
 for s in shapes { s.serialize(&mut out); out.push('\n'); }
 out
}

Passing `&mut String` is idiomatic for push-based serialization — avoids allocation per field, builds the output incrementally. The `Deserialize` counterpart uses a cursor (`&str` that gets consumed as tokens are parsed) to mirror the format.

What This Unlocks

Manual `serde::Serialize` impl — once you understand tagged-variant encoding, implementing `serde::Serializer` calls for enum variants is straightforward; the pattern maps directly.
Custom wire formats — length-prefixed encoding, delimited formats, and tagged-union protocols are all expressible with this pattern; you control every byte.
Generic `Option<T>` and `Vec<T>` impls — the trait system lets you write one impl that works for any serializable type, paralleling how `serde` derives work transitively through generic types.

Key Differences

Concept	OCaml	Rust
Enum serialization	Pattern match + write tag	Same — `match` on variant, push tag + payload
Length-prefixed strings	`Printf.sprintf "%d:%s"`	`write!(out, "{}:{}", len, s)` — same idea
Generic impls	Functors or polymorphic variants	`impl<T: Serialize> Serialize for Option<T>`
Output buffer	`Buffer.t`	`&mut String` — zero-copy append via `push_str`

// 761. Custom Serialization for Complex Types
// Enums with payloads, Option<T>, Vec<T> — all hand-rolled

use std::fmt::Write as FmtWrite;

// ── Error ──────────────────────────────────────────────────────────────────────

#[derive(Debug)]
pub enum SerError {
    Eof,
    BadTag(String),
    ParseError(String),
}

// ── Traits ─────────────────────────────────────────────────────────────────────

pub trait Serialize {
    fn serialize(&self, out: &mut String);
}

pub trait Deserialize: Sized {
    fn deserialize(s: &str) -> Result<(Self, &str), SerError>;
}

// ── Primitive impls ────────────────────────────────────────────────────────────

impl Serialize for f64 {
    fn serialize(&self, out: &mut String) { write!(out, "{}", self).unwrap(); }
}
impl Serialize for u32 {
    fn serialize(&self, out: &mut String) { write!(out, "{}", self).unwrap(); }
}
impl Serialize for String {
    fn serialize(&self, out: &mut String) {
        write!(out, "{}:{}", self.len(), self).unwrap();
    }
}

impl Deserialize for f64 {
    fn deserialize(s: &str) -> Result<(f64, &str), SerError> {
        let end = s.find('|').or_else(|| s.find('\n')).unwrap_or(s.len());
        let v = s[..end].parse::<f64>().map_err(|e| SerError::ParseError(e.to_string()))?;
        Ok((v, &s[end..]))
    }
}
impl Deserialize for String {
    fn deserialize(s: &str) -> Result<(String, &str), SerError> {
        let colon = s.find(':').ok_or(SerError::Eof)?;
        let len: usize = s[..colon].parse().map_err(|e| SerError::ParseError(format!("{e}")))?;
        let rest = &s[colon + 1..];
        if rest.len() < len { return Err(SerError::Eof); }
        Ok((rest[..len].to_string(), &rest[len..]))
    }
}

// ── Domain enum ────────────────────────────────────────────────────────────────

#[derive(Debug, PartialEq)]
pub enum Shape {
    Circle(f64),
    Rectangle { width: f64, height: f64 },
    Point,
}

impl Serialize for Shape {
    fn serialize(&self, out: &mut String) {
        match self {
            Shape::Circle(r) => {
                out.push_str("C|");
                r.serialize(out);
            }
            Shape::Rectangle { width, height } => {
                out.push_str("R|");
                width.serialize(out);
                out.push('|');
                height.serialize(out);
            }
            Shape::Point => out.push('P'),
        }
    }
}

impl Deserialize for Shape {
    fn deserialize(s: &str) -> Result<(Shape, &str), SerError> {
        match s.chars().next().ok_or(SerError::Eof)? {
            'C' => {
                let rest = &s[2..]; // skip "C|"
                let (r, rest) = f64::deserialize(rest)?;
                Ok((Shape::Circle(r), rest))
            }
            'R' => {
                let rest = &s[2..];
                let (w, rest) = f64::deserialize(rest)?;
                let rest = rest.trim_start_matches('|');
                let (h, rest) = f64::deserialize(rest)?;
                Ok((Shape::Rectangle { width: w, height: h }, rest))
            }
            'P' => Ok((Shape::Point, &s[1..])),
            c => Err(SerError::BadTag(c.to_string())),
        }
    }
}

// ── Option<T> ─────────────────────────────────────────────────────────────────

impl<T: Serialize> Serialize for Option<T> {
    fn serialize(&self, out: &mut String) {
        match self {
            None => out.push('N'),
            Some(v) => { out.push('S'); v.serialize(out); }
        }
    }
}

// ── Vec<Shape> round-trip ──────────────────────────────────────────────────────

fn serialize_shapes(shapes: &[Shape]) -> String {
    let mut out = String::new();
    write!(out, "{}\n", shapes.len()).unwrap();
    for s in shapes {
        s.serialize(&mut out);
        out.push('\n');
    }
    out
}

fn deserialize_shapes(s: &str) -> Result<Vec<Shape>, SerError> {
    let mut lines = s.lines();
    let count: usize = lines.next().ok_or(SerError::Eof)?
        .parse().map_err(|e| SerError::ParseError(format!("{e}")))?;
    let mut shapes = Vec::with_capacity(count);
    for line in lines.take(count) {
        let (shape, _) = Shape::deserialize(line)?;
        shapes.push(shape);
    }
    Ok(shapes)
}

fn main() {
    let shapes = vec![
        Shape::Circle(3.14),
        Shape::Rectangle { width: 2.0, height: 5.0 },
        Shape::Point,
    ];
    let encoded = serialize_shapes(&shapes);
    println!("Encoded:\n{encoded}");
    let decoded = deserialize_shapes(&encoded).expect("decode failed");
    println!("Decoded: {decoded:?}");

    // Option demo
    let maybe: Option<u32> = Some(42);
    let mut buf = String::new();
    maybe.serialize(&mut buf);
    println!("Option<42> = {buf}");
    let none: Option<u32> = None;
    let mut buf2 = String::new();
    none.serialize(&mut buf2);
    println!("Option<None> = {buf2}");
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn circle_round_trip() {
        let s = Shape::Circle(2.718);
        let mut buf = String::new();
        s.serialize(&mut buf);
        let (decoded, _) = Shape::deserialize(&buf).unwrap();
        assert_eq!(decoded, Shape::Circle(2.718));
    }

    #[test]
    fn rect_round_trip() {
        let s = Shape::Rectangle { width: 3.0, height: 4.0 };
        let mut buf = String::new();
        s.serialize(&mut buf);
        let (decoded, _) = Shape::deserialize(&buf).unwrap();
        assert_eq!(decoded, s);
    }

    #[test]
    fn point_round_trip() {
        let s = Shape::Point;
        let mut buf = String::new();
        s.serialize(&mut buf);
        let (decoded, _) = Shape::deserialize(&buf).unwrap();
        assert_eq!(decoded, Shape::Point);
    }

    #[test]
    fn vec_shapes_round_trip() {
        let shapes = vec![Shape::Circle(1.0), Shape::Point];
        let enc = serialize_shapes(&shapes);
        let dec = deserialize_shapes(&enc).unwrap();
        assert_eq!(shapes, dec);
    }
}

(* Custom serialization for complex types in OCaml
   Handling sum types (variants), options, and lists *)

type shape =
  | Circle of float
  | Rectangle of float * float
  | Point

(* Serialize a shape to a string representation *)
let serialize_shape = function
  | Circle r -> Printf.sprintf "circle|r=%.6g" r
  | Rectangle (w, h) -> Printf.sprintf "rect|w=%.6g|h=%.6g" w h
  | Point -> "point"

let deserialize_shape s =
  match String.split_on_char '|' s with
  | ["circle"; rv] ->
    (match String.split_on_char '=' rv with
     | ["r"; v] -> (try Some (Circle (float_of_string v)) with _ -> None)
     | _ -> None)
  | ["rect"; wv; hv] ->
    (match String.split_on_char '=' wv, String.split_on_char '=' hv with
     | ["w"; w], ["h"; h] ->
       (try Some (Rectangle (float_of_string w, float_of_string h)) with _ -> None)
     | _ -> None)
  | ["point"] -> Some Point
  | _ -> None

(* Serialize a list of shapes *)
let serialize_shapes shapes =
  let parts = List.map serialize_shape shapes in
  (* length-prefix the list *)
  Printf.sprintf "%d\n%s" (List.length parts) (String.concat "\n" parts)

let () =
  let shapes = [Circle 3.14; Rectangle (2.0, 5.0); Point; Circle 1.0] in
  let s = serialize_shapes shapes in
  Printf.printf "Serialized:\n%s\n\n" s;
  (* deserialize each line after the count *)
  let lines = String.split_on_char '\n' s in
  match lines with
  | [] -> ()
  | _ :: rest ->
    List.iteri (fun i line ->
      if line <> "" then
        match deserialize_shape line with
        | Some sh -> Printf.printf "Shape %d: %s\n" i (serialize_shape sh)
        | None -> Printf.printf "Shape %d: FAILED\n" i
    ) rest