47 lines
5.0 KiB
Markdown
47 lines
5.0 KiB
Markdown
Good time to think about this — the unloading infrastructure you just built is exactly the right foundation. Here's how I'd think about the road to disk streaming:
|
|
|
|
**The three problems to solve, in order**
|
|
|
|
Serialization, then streaming, then diffs. They build on each other and trying to do them out of order creates rework.
|
|
|
|
**Phase 1: Serialization**
|
|
|
|
Before anything touches disk, you need to be able to round-trip a chunk through serialization and get back identical game state. This means deciding on a format for `FloorTileData`, `FixtureTileData`, `ChunkData`, and eventually entity state. `serde` is already in your `Cargo.toml` which is a good start. The key decision here is binary vs human-readable — for chunks you want binary. `bincode` or `bitcode` are the obvious choices, with `bitcode` being significantly more compact which matters a lot at 128 z-levels. Don't use JSON or TOML for chunk data.
|
|
|
|
The most important thing to get right in this phase is that your serialized format is **versioned from day one**. A version byte at the top of every chunk file costs nothing and saves you from a world-wiping format migration later.
|
|
|
|
**Phase 2: Save on unload, load from disk**
|
|
|
|
You already drain `pending_unload` in `handle_chunk_unloading`. The save hook goes right there — serialize the chunk data, write to disk at a deterministic path like `world/chunks/{x}_{y}.bin`, then proceed with the existing cleanup. On the load side, `handle_chunk_events` currently always generates terrain from scratch. It needs a check first: does a save file exist for this chunk? If yes, deserialize and populate `TileMap` directly, skipping generation. If no, generate as normal.
|
|
|
|
The path structure matters more than it seems. A flat directory of chunk files breaks badly on filesystems at large world sizes. A two-level structure like `world/{x>>4}/{x}_{y}.bin` keeps directory sizes manageable.
|
|
|
|
Async IO is important here — disk reads on the main thread will cause hitches. The chunk generation system is already async (`spawn_terrain_tasks`), so the disk load path should slot into the same task system, producing the same `TerrainBlob` output whether it came from generation or deserialization.
|
|
|
|
**Phase 3: Diffs**
|
|
|
|
This is the Dwarf Fortress part — the world as the player left it, not as it was generated. The insight here is that you don't want to serialize the entire chunk state every save, you want to store the **delta from procedural generation**. A chunk that hasn't been touched has a zero-byte diff. A chunk where a wall was dug out has a small diff representing just those changes.
|
|
|
|
The data model for diffs is straightforward: a list of `(IVec3, TileChange)` where `TileChange` is an enum covering tile replacement, removal, and fixture changes. On load, generate the chunk procedurally, then apply the diff on top. On save, compute the diff by comparing current state against what generation would have produced — or more practically, accumulate the diff incrementally as the player makes changes, rather than computing it at save time.
|
|
|
|
The incremental approach requires a `ChunkDirtyTracker` resource that records tile mutations as they happen. Every system that modifies tile data writes to the tracker. This is a small maintenance surface — there aren't many tile mutation sites right now — and it means save time is O(number of player changes) rather than O(chunk size).
|
|
|
|
**What to build before any of this**
|
|
|
|
Two things will make everything above much easier:
|
|
|
|
First, a **world identity** — a seed and world name that identifies a save slot. Right now `SEED` is a compile-time constant. It needs to become a runtime value loaded from or saved to a world manifest file.
|
|
|
|
Second, **entity serialization strategy**. Your current entities (dorfs, pigs, etc.) have no persistent state and get despawned on chunk unload. Before disk streaming is useful you need to decide: are entities saved per-chunk (everything in chunk bounds gets serialized with the chunk) or globally (all entities serialized in a separate entities file)? Per-chunk is simpler but breaks for entities that cross chunk boundaries. Global entity save with chunk-association metadata is more robust. This decision cascades into a lot of the entity architecture so it's worth settling early even if implementation is later.
|
|
|
|
**The practical order**
|
|
|
|
1. Add `serde` derives to tile data structs, pick `bitcode`, write a round-trip test
|
|
2. Add world identity / runtime seed
|
|
3. Save on unload (fire and forget, async write)
|
|
4. Load from disk on chunk load (slot into existing async task system)
|
|
5. Add `ChunkDirtyTracker`, instrument tile mutation sites
|
|
6. Switch saves from full chunk state to diff-only
|
|
7. Entity persistence (separate effort, tackle after tile persistence is stable)
|
|
|
|
The unloading work you've done means phases 3 and 4 are mostly plumbing rather than architecture. The hardest part of the whole thing is actually phase 6 — getting the diff computation right without either missing changes or over-counting them — but by the time you get there you'll have learned a lot from phases 3-5 about where tile mutations actually originate. |