dorf/INVESTIGATION_HANDOFF.md

Investigation Handoff: Dorf Job Queue System

Current State

System Overview

This is a dwarf (dorf) simulation with a job queue system, think dwarf fortress in a chunked world a la minecraft. We are early in develpment and are currently developing the job queue system so it in not quite ready. The expected DEMO flow is:

1. ONE FellTree job should always exist (when none is in progress)
2. When a tree falls, X logs spawn, each creating a HaulCargo job
3. All available dorfs (not busy felling) should take HaulCargo jobs based on the jobs priorities
4. No dorfs should be idle while haul jobs exist to assign

Known Outputs

• output12.log: One tree felled, one haul completed, no second tree chopped
• output13.log: No trees ever chopped, all dorfs idle
• output14.log: One tree felled, one haul completed, one dorf FROZE in place

Fixes Already Applied

1. Z-level spawn fix: Dorfs now spawn at Z_ABOVE * TILE_SIZE (z=240) instead of z=560 (This will allow the randomly spawned dorfs to fall to surface level. This is demo behaviour only)
2. Multiple job assignment bug fix: Added assigned_this_frame HashSet to prevent same dorf getting multiple jobs in one frame (see job_pathfinding.rs lines47-48 and 87)

Remaining Issues

1. Dorfs go idle when haul jobs still exist
2. Second/third trees not being felled or assigned
3. Only 1 of X logs get hauled
4. Dorfs freeze in place (pathfinding issue? job assignment issue?)

Key Files to Investigate

Job Systems

src/entities/tasks/job_pathfinding.rs  - Job assignment, candidate filtering
src/entities/tasks/job_queue.rs         - Job queue state management
src/entities/tasks/job_assignment.rs   - Idle taskassignment
src/entities/tasks/executor.rs         - Task execution (ChopTree, HaulCargo)
src/entities/tasks/demo.rs              - FellTree job creation logic

Pathfinding & Movement

src/entities/shared_systems/pathfinding.rs  - Entity movement, is_standable checks
src/world/tiles/tilemap.rs                   - is_standable(), find_nearest_free_cargo_tile()
src/world/generation/terrain.rs              - Terrain generation, surface positions
src/entities/sentient/dorf.rs                - Dorf spawning

Key Concepts

Z-Level System

• World uses Z coordinates for height
• Z_ABOVE = 15.0, Z_BELOW = 5.0 (in tile units)
• TILE_SIZE = 16.0 pixels
• Dorfs spawn at z=240 pixels (15 * 16), fall ~4s to surface
• is_standable(pos) checks if a tile has floor for standing
• Think dwarf fortress here. Multiple 2d grids of floor tiles, each making up one 'z' level that we can traverse, move up down on etc. Each Z level is one TILE_SIZE appart on the Z axis for consistency. Look into the tree spawn system for an example of this, as well as the log spawning, and pathfinding.rs generally.

Pathfinding Coordinate System

• Entity position: transform.translation (Vec3 in pixels)
• Tile position: pos.as_ivec3() (IVec3 in pixels)
• Entity is ABOVE tile: entity z = tile z + 1.0 (to prevent zfighting in rendering)
• Movement handles z-level traversal via ALLOWED_MOVES (24 directions including diagonals)

Job Queue States

JobState::Unclaimed    - Job available, no dorf assigned
JobState::Calculating  - Pathfinding in progress, dorfs locked
JobState::Claimed      - Job assigned to a dorf
JobState::Suspended    - Too many retries, job disabled
JobState::Complete     - Job finished

Task System

TaskQueue - VecDeque of tasks per dorf
TaskState - Pending | Active | Completed | Failed
Task::Idle - Default task when queue is empty
Task::ChopTree - FellTree job execution
Task::HaulCargo - HaulCargo job execution

Debug Logs Already Added

// dorf.rs - Spawn position
info!("[SPAWN] Dorf spawning at ({}, {}, {}) z_level={}", grid_x, grid_y, grid_z, grid_z / TILE_SIZE);

// job_pathfinding.rs - Why dorfs filtered out
info!("[PATHFIND] Dorf {:?} NOT STANDABLE at pos={:?} z_level={}", entity, pos, pos.z / ITILE_SIZE);

// pathfinding.rs - Falling/snapping behavior
info!("[MOVEMENT] Entity {:?} above world at z={}, snapping to z={}", entity, z, Z_ABOVE * TILE_SIZE);
info!("[MOVEMENT] Entity {:?} falling at z={}, z_level={}", entity, z, z / TILE_SIZE);

Investigation Prompts

For output12/output14 (one haul, no second tree):

1. Check if FellTree job exists after tree falls:

   grep -E "Processing.*unclaimed|FellTree|has_fell_tree" output12.log
   

2. Check why dorfs go idle when haul jobs exist:

   grep -E "idle=|QUEUE.*fell.*haul|candidates=|SUCCESS.*Assigned" output12.log
   

3. Check if dorfs are freezing:

   grep -E "frozen|stuck|NOT STANDABLE|NO PATH" output14.log
   

For output13 (all idle, no trees):

1. Check if FellTree job was created:

   grep -E "Processing.*unclaimed|FellTree" output13.log | head -20
   

2. Check candidate counts:

   grep -E "candidates=[0-9]" output13.log | head -20
   

Suggested Debug Logs to Add

In job_queue.rs - Track job lifecycle:

info!("[QUEUE] Job {:?} created at {:?}", job_id, kind);
info!("[QUEUE] Job {:?} state changed from {:?} to {:?}", job_id, old_state, new_state);
info!("[QUEUE] Job {:?} claimed by dorf {:?}", job_id, dorf_entity);
info!("[QUEUE] Job {:?} completed", job_id);

In executor.rs - Track task completion:

info!("[EXECUTOR] Task {:?} for dorf {:?} transitioning from {:?} to {:?}", task.name(), entity, old_step, new_step);
info!("[EXECUTOR] Task {:?} completed, queue now has {} items", task.name(), queue.len());

In demo.rs - Track FellTree job creation:

info!("[DEMO] has_fell_tree={}, any_chopping={}, creating_job={}", has_fell_tree, any_chopping, trunk_pos.is_some());

In job_pathfinding.rs - Track why jobs aren't assigned:

info!("[PATHFIND] Job {:?} scope={} retry={}, best_result={:?}", job_id, scope, retry_count, best_result);
info!("[PATHFIND] Dorfs: locked={}, not_standable={}, busy={}, idle_but_working={}", locked, not_standable, busy, idle_but_working);

In executor.rs - Track HaulCargo state machine:

info!("[HAUL] {:?} step={:?} approach={:?} target={:?} cargo={:?}", entity, step, approach, ambulatory.target, cargo_entity);

Known Gotchas

1. Dorfs spawn at z=240 - Takes ~4 seconds to fall to surface. During this time is_standable() returns false, preventing job assignment. The falling logs should help identify if dorfs are stuck mid-air.

2. Coordinate mismatch in pathfinding - The HaulCargo approach finding in executor.rs BFS searches at cargo's z-level. Verify cargo z is correct.

3. Target oscillation - Task executor and job_pathfinding both try to set ambulatory.target. The fix in executor.rs (lines 502-516) sets target when approach.is_some() but target.is_none().

4. is_busy flag - A dorf is busy if TaskState::Active AND task is NOT Idle. Idle dorfs can have their tasks replaced.

5. queue.clear() on assignment - When job_pathfinding assigns a job, it CLEARS the queue. The fix in this session adds assigned_this_frame to prevent multiple assignments, but verify this works correctly.

How to Continue

1. Analyze output12.log - Why does only one haul complete? Why no second tree?

2. Analyze output13.log - Why no FellTree job? Check demo_system.

3. Analyze output14.log - Why did a dorf freeze? Check pathfinding logs around that dorf.

4. Add new debug logs - Especially around task state transitions and job queue lifecycle.

5. Run new tests - Capture output with new debug logs to see more detail.

Commands to Run

# Check job creation
grep -E "Processing.*unclaimed|Creating|FellTree|HaulCargo" output12.log | head -50

# Check job assignment distribution (should show different dorfs)
grep "SUCCESS.*Assigned" output12.log

# Check if dorfs are stuck
grep -E "frozen|stuck|[^]]0 candidates=|NO PATH" output14.log

# Check task completion
grep -E "Chop tick: 0|picked up|COMPLETE" output12.log

# Check queue state over time
grep -E "=== QUEUE ===" output12.log

Expected Behavior After Fixes

1. Each dorf should get at most ONE job per frame (fixed)
2. Jobs should be distributed across different dorfs (fixed)
3. After tree falls, multiple HaulCargo jobs should be processed and assigned on the creation of the logs
4. All dorfs should be assigned jobs until queue is empty or no eligible dorfs remain to assign to this tick
5. New FellTree job should be created immediately after the current tree fell

---

This is a VERY WIP implementation and we are in the middle of an implementation so the state of the whole thing may be a bit messy and unclear. If you have ANY questions or comments (and you will!), please pause immediately and ask for clarification. On commit 79e0efa409 (or maybe the one before?) we had the demo system 'working'. We had X dorfs and 1 was always chopping, not always the same 1 though. The rest would haul if hauling needed done and the rest would be idle waiting for a job, either a haul or a chop. BUT this implementation was 'fake' and just misused the calculate_provisional_path function to check if a path was possible. We do NOT want to do that, we should mostly leave pathfinding.rs alone as it WORKS. We are to implement/fix src/entities/tasks/job_pathfinding.rs so for our distance calculations. We want accurate distances based on WALKABLE paths, not just the usual mathematical distances so we should use this to call the pathfinding system to get the best path lengths to the target, but limited per go as to not throttle the frametime. Use asyncs, threading etc to keep it runnable as a game. This has been worked on but currently faces the issues described above. investigate output12.log, output13.log, and output14.log to understand the current state of the demo system. All three are different runs of the same build with the only differences being the dorf spawn locations. Each run has different issues. I have been working on this for a while and I am not sure what the next steps are. Please investigate, then plan our next steps. you can also check .sisyphus/plans/job_pathfinding_system.md for more context on where we started, but its likely outdated.

---

Try not to read full files, code or logs. Although early days, these files are still LARGE. Grep in the codebase to find where we are logging and expand to see what each log means. Then greps the logs that matter for these bugs and see the orders things are happening in etc. Be smart with token usage in the main session. Delagate to subagents where possible for investigation to conserve tokens and keep the main seesion clear for planning and implementation