dorf/streaming plan.md

Good time to think about this — the unloading infrastructure you just built is exactly the right foundation. Here's how I'd think about the road to disk streaming:

The three problems to solve, in order

Serialization, then streaming, then diffs. They build on each other and trying to do them out of order creates rework.

Phase 1: Serialization

Before anything touches disk, you need to be able to round-trip a chunk through serialization and get back identical game state. This means deciding on a format for FloorTileData, FixtureTileData, ChunkData, and eventually entity state. serde is already in your Cargo.toml which is a good start. The key decision here is binary vs human-readable — for chunks you want binary. bincode or bitcode are the obvious choices, with bitcode being significantly more compact which matters a lot at 128 z-levels. Don't use JSON or TOML for chunk data.

The most important thing to get right in this phase is that your serialized format is versioned from day one. A version byte at the top of every chunk file costs nothing and saves you from a world-wiping format migration later.

Phase 2: Save on unload, load from disk

You already drain pending_unload in handle_chunk_unloading. The save hook goes right there — serialize the chunk data, write to disk at a deterministic path like world/chunks/{x}_{y}.bin, then proceed with the existing cleanup. On the load side, handle_chunk_events currently always generates terrain from scratch. It needs a check first: does a save file exist for this chunk? If yes, deserialize and populate TileMap directly, skipping generation. If no, generate as normal.

The path structure matters more than it seems. A flat directory of chunk files breaks badly on filesystems at large world sizes. A two-level structure like world/{x>>4}/{x}_{y}.bin keeps directory sizes manageable.

Async IO is important here — disk reads on the main thread will cause hitches. The chunk generation system is already async (spawn_terrain_tasks), so the disk load path should slot into the same task system, producing the same TerrainBlob output whether it came from generation or deserialization.

Phase 3: Diffs

This is the Dwarf Fortress part — the world as the player left it, not as it was generated. The insight here is that you don't want to serialize the entire chunk state every save, you want to store the delta from procedural generation. A chunk that hasn't been touched has a zero-byte diff. A chunk where a wall was dug out has a small diff representing just those changes.

The data model for diffs is straightforward: a list of (IVec3, TileChange) where TileChange is an enum covering tile replacement, removal, and fixture changes. On load, generate the chunk procedurally, then apply the diff on top. On save, compute the diff by comparing current state against what generation would have produced — or more practically, accumulate the diff incrementally as the player makes changes, rather than computing it at save time.

The incremental approach requires a ChunkDirtyTracker resource that records tile mutations as they happen. Every system that modifies tile data writes to the tracker. This is a small maintenance surface — there aren't many tile mutation sites right now — and it means save time is O(number of player changes) rather than O(chunk size).

What to build before any of this

Two things will make everything above much easier:

First, a world identity — a seed and world name that identifies a save slot. Right now SEED is a compile-time constant. It needs to become a runtime value loaded from or saved to a world manifest file.

Second, entity serialization strategy. Your current entities (dorfs, pigs, etc.) have no persistent state and get despawned on chunk unload. Before disk streaming is useful you need to decide: are entities saved per-chunk (everything in chunk bounds gets serialized with the chunk) or globally (all entities serialized in a separate entities file)? Per-chunk is simpler but breaks for entities that cross chunk boundaries. Global entity save with chunk-association metadata is more robust. This decision cascades into a lot of the entity architecture so it's worth settling early even if implementation is later.

The practical order

1. Add serde derives to tile data structs, pick bitcode, write a round-trip test
2. Add world identity / runtime seed
3. Save on unload (fire and forget, async write)
4. Load from disk on chunk load (slot into existing async task system)
5. Add ChunkDirtyTracker, instrument tile mutation sites
6. Switch saves from full chunk state to diff-only
7. Entity persistence (separate effort, tackle after tile persistence is stable)

The unloading work you've done means phases 3 and 4 are mostly plumbing rather than architecture. The hardest part of the whole thing is actually phase 6 — getting the diff computation right without either missing changes or over-counting them — but by the time you get there you'll have learned a lot from phases 3-5 about where tile mutations actually originate.