215 lines
11 KiB
Markdown
215 lines
11 KiB
Markdown
# Investigation Handoff: Dorf Job Queue System
|
|
|
|
## Current State
|
|
|
|
### System Overview
|
|
This is a dwarf (dorf) simulation with a job queue system, think dwarf fortress in a chunked world a la minecraft. We are early in develpment and are currently developing the job queue system so it in not quite ready. The expected DEMO flow is:
|
|
1. ONE FellTree job should always exist (when none is in progress)
|
|
2. When a tree falls, X logs spawn, each creating a HaulCargo job
|
|
3. All available dorfs (not busy felling) should take HaulCargo jobs based on the jobs priorities
|
|
4. No dorfs should be idle while haul jobs exist to assign
|
|
|
|
### Known Outputs
|
|
- **output12.log**: One tree felled, one haul completed, no second tree chopped
|
|
- **output13.log**: No trees ever chopped, all dorfs idle
|
|
- **output14.log**: One tree felled, one haul completed, one dorf FROZE in place
|
|
|
|
### Fixes Already Applied
|
|
1. **Z-level spawn fix**: Dorfs now spawn at `Z_ABOVE * TILE_SIZE` (z=240) instead of z=560 (This will allow the randomly spawned dorfs to fall to surface level. This is demo behaviour only)
|
|
2. **Multiple job assignment bug fix**: Added `assigned_this_frame` HashSet to prevent same dorf getting multiple jobs in one frame (see job_pathfinding.rs lines47-48 and 87)
|
|
|
|
### Remaining Issues
|
|
1. Dorfs go idle when haul jobs still exist
|
|
2. Second/third trees not being felled or assigned
|
|
3. Only 1 of X logs get hauled
|
|
4. Dorfs freeze in place (pathfinding issue? job assignment issue?)
|
|
|
|
## Key Files to Investigate
|
|
|
|
### Job Systems
|
|
```
|
|
src/entities/tasks/job_pathfinding.rs - Job assignment, candidate filtering
|
|
src/entities/tasks/job_queue.rs - Job queue state management
|
|
src/entities/tasks/job_assignment.rs - Idle taskassignment
|
|
src/entities/tasks/executor.rs - Task execution (ChopTree, HaulCargo)
|
|
src/entities/tasks/demo.rs - FellTree job creation logic
|
|
```
|
|
|
|
### Pathfinding & Movement
|
|
```
|
|
src/entities/shared_systems/pathfinding.rs - Entity movement, is_standable checks
|
|
src/world/tiles/tilemap.rs - is_standable(), find_nearest_free_cargo_tile()
|
|
src/world/generation/terrain.rs - Terrain generation, surface positions
|
|
src/entities/sentient/dorf.rs - Dorf spawning
|
|
```
|
|
|
|
## Key Concepts
|
|
|
|
### Z-Level System
|
|
- World uses Z coordinates for height
|
|
- `Z_ABOVE = 15.0`, `Z_BELOW = 5.0` (in tile units)
|
|
- `TILE_SIZE = 16.0` pixels
|
|
- Dorfs spawn at z=240 pixels (15 * 16), fall ~4s to surface
|
|
- `is_standable(pos)` checks if a tile has floor for standing
|
|
- Think dwarf fortress here. Multiple 2d grids of floor tiles, each making up one 'z' level that we can traverse, move up down on etc. Each Z level is one TILE_SIZE appart on the Z axis for consistency. Look into the tree spawn system for an example of this, as well as the log spawning, and pathfinding.rs generally.
|
|
|
|
### Pathfinding Coordinate System
|
|
- Entity position: `transform.translation` (Vec3 in pixels)
|
|
- Tile position: `pos.as_ivec3()` (IVec3 in pixels)
|
|
- Entity is ABOVE tile: entity z = tile z + 1.0 (to prevent zfighting in rendering)
|
|
- Movement handles z-level traversal via `ALLOWED_MOVES` (24 directions including diagonals)
|
|
|
|
### Job Queue States
|
|
```
|
|
JobState::Unclaimed - Job available, no dorf assigned
|
|
JobState::Calculating - Pathfinding in progress, dorfs locked
|
|
JobState::Claimed - Job assigned to a dorf
|
|
JobState::Suspended - Too many retries, job disabled
|
|
JobState::Complete - Job finished
|
|
```
|
|
|
|
### Task System
|
|
```
|
|
TaskQueue - VecDeque of tasks per dorf
|
|
TaskState - Pending | Active | Completed | Failed
|
|
Task::Idle - Default task when queue is empty
|
|
Task::ChopTree - FellTree job execution
|
|
Task::HaulCargo - HaulCargo job execution
|
|
```
|
|
|
|
## Debug Logs Already Added
|
|
|
|
```rust
|
|
// dorf.rs - Spawn position
|
|
info!("[SPAWN] Dorf spawning at ({}, {}, {}) z_level={}", grid_x, grid_y, grid_z, grid_z / TILE_SIZE);
|
|
|
|
// job_pathfinding.rs - Why dorfs filtered out
|
|
info!("[PATHFIND] Dorf {:?} NOT STANDABLE at pos={:?} z_level={}", entity, pos, pos.z / ITILE_SIZE);
|
|
|
|
// pathfinding.rs - Falling/snapping behavior
|
|
info!("[MOVEMENT] Entity {:?} above world at z={}, snapping to z={}", entity, z, Z_ABOVE * TILE_SIZE);
|
|
info!("[MOVEMENT] Entity {:?} falling at z={}, z_level={}", entity, z, z / TILE_SIZE);
|
|
```
|
|
|
|
## Investigation Prompts
|
|
|
|
### For output12/output14 (one haul, no second tree):
|
|
1. Check if FellTree job exists after tree falls:
|
|
```bash
|
|
grep -E "Processing.*unclaimed|FellTree|has_fell_tree" output12.log
|
|
```
|
|
|
|
2. Check why dorfs go idle when haul jobs exist:
|
|
```bash
|
|
grep -E "idle=|QUEUE.*fell.*haul|candidates=|SUCCESS.*Assigned" output12.log
|
|
```
|
|
|
|
3. Check if dorfs are freezing:
|
|
```bash
|
|
grep -E "frozen|stuck|NOT STANDABLE|NO PATH" output14.log
|
|
```
|
|
|
|
### For output13 (all idle, no trees):
|
|
1. Check if FellTree job was created:
|
|
```bash
|
|
grep -E "Processing.*unclaimed|FellTree" output13.log | head -20
|
|
```
|
|
|
|
2. Check candidate counts:
|
|
```bash
|
|
grep -E "candidates=[0-9]" output13.log | head -20
|
|
```
|
|
|
|
## Suggested Debug Logs to Add
|
|
|
|
### In job_queue.rs - Track job lifecycle:
|
|
```rust
|
|
info!("[QUEUE] Job {:?} created at {:?}", job_id, kind);
|
|
info!("[QUEUE] Job {:?} state changed from {:?} to {:?}", job_id, old_state, new_state);
|
|
info!("[QUEUE] Job {:?} claimed by dorf {:?}", job_id, dorf_entity);
|
|
info!("[QUEUE] Job {:?} completed", job_id);
|
|
```
|
|
|
|
### In executor.rs - Track task completion:
|
|
```rust
|
|
info!("[EXECUTOR] Task {:?} for dorf {:?} transitioning from {:?} to {:?}", task.name(), entity, old_step, new_step);
|
|
info!("[EXECUTOR] Task {:?} completed, queue now has {} items", task.name(), queue.len());
|
|
```
|
|
|
|
### In demo.rs - Track FellTree job creation:
|
|
```rust
|
|
info!("[DEMO] has_fell_tree={}, any_chopping={}, creating_job={}", has_fell_tree, any_chopping, trunk_pos.is_some());
|
|
```
|
|
|
|
### In job_pathfinding.rs - Track why jobs aren't assigned:
|
|
```rust
|
|
info!("[PATHFIND] Job {:?} scope={} retry={}, best_result={:?}", job_id, scope, retry_count, best_result);
|
|
info!("[PATHFIND] Dorfs: locked={}, not_standable={}, busy={}, idle_but_working={}", locked, not_standable, busy, idle_but_working);
|
|
```
|
|
|
|
### In executor.rs - Track HaulCargo state machine:
|
|
```rust
|
|
info!("[HAUL] {:?} step={:?} approach={:?} target={:?} cargo={:?}", entity, step, approach, ambulatory.target, cargo_entity);
|
|
```
|
|
|
|
## Known Gotchas
|
|
|
|
1. **Dorfs spawn at z=240** - Takes ~4 seconds to fall to surface. During this time `is_standable()` returns false, preventing job assignment. The falling logs should help identify if dorfs are stuck mid-air.
|
|
|
|
2. **Coordinate mismatch in pathfinding** - The HaulCargo approach finding in executor.rs BFS searches at cargo's z-level. Verify cargo z is correct.
|
|
|
|
3. **Target oscillation** - Task executor and job_pathfinding both try to set `ambulatory.target`. The fix in executor.rs (lines 502-516) sets target when `approach.is_some()` but `target.is_none()`.
|
|
|
|
4. **`is_busy` flag** - A dorf is busy if `TaskState::Active` AND task is NOT Idle. Idle dorfs can have their tasks replaced.
|
|
|
|
5. **`queue.clear()` on assignment** - When job_pathfinding assigns a job, it CLEARS the queue. The fix in this session adds `assigned_this_frame` to prevent multiple assignments, but verify this works correctly.
|
|
|
|
## How to Continue
|
|
|
|
1. **Analyze output12.log** - Why does only one haul complete? Why no second tree?
|
|
|
|
2. **Analyze output13.log** - Why no FellTree job? Check `demo_system`.
|
|
|
|
3. **Analyze output14.log** - Why did a dorf freeze? Check pathfinding logs around that dorf.
|
|
|
|
4. **Add new debug logs** - Especially around task state transitions and job queue lifecycle.
|
|
|
|
5. **Run new tests** - Capture output with new debug logs to see more detail.
|
|
|
|
## Commands to Run
|
|
|
|
```bash
|
|
# Check job creation
|
|
grep -E "Processing.*unclaimed|Creating|FellTree|HaulCargo" output12.log | head -50
|
|
|
|
# Check job assignment distribution (should show different dorfs)
|
|
grep "SUCCESS.*Assigned" output12.log
|
|
|
|
# Check if dorfs are stuck
|
|
grep -E "frozen|stuck|[^]]0 candidates=|NO PATH" output14.log
|
|
|
|
# Check task completion
|
|
grep -E "Chop tick: 0|picked up|COMPLETE" output12.log
|
|
|
|
# Check queue state over time
|
|
grep -E "=== QUEUE ===" output12.log
|
|
```
|
|
|
|
## Expected Behavior After Fixes
|
|
|
|
1. Each dorf should get at most ONE job per frame (fixed)
|
|
2. Jobs should be distributed across different dorfs (fixed)
|
|
3. After tree falls, multiple HaulCargo jobs should be processed and assigned on the creation of the logs
|
|
4. All dorfs should be assigned jobs until queue is empty or no eligible dorfs remain to assign to this tick
|
|
5. New FellTree job should be created immediately after the current tree fell
|
|
|
|
|
|
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
This is a VERY WIP implementation and we are in the middle of an implementation so the state of the whole thing may be a bit messy and unclear. If you have ANY questions or comments (and you will!), please pause immediately and ask for clarification. On commit 79e0efa4099ede9972897d887ad9c9a173c89aaf (or maybe the one before?) we had the demo system 'working'. We had X dorfs and 1 was always chopping, not always the same 1 though. The rest would haul if hauling needed done and the rest would be idle waiting for a job, either a haul or a chop. BUT this implementation was 'fake' and just misused the calculate_provisional_path function to check if a path was possible. We do NOT want to do that, we should mostly leave pathfinding.rs alone as it WORKS. We are to implement/fix `src/entities/tasks/job_pathfinding.rs` so for our distance calculations. We want accurate distances based on WALKABLE paths, not just the usual mathematical distances so we should use this to call the pathfinding system to get the best path lengths to the target, but limited per go as to not throttle the frametime. Use asyncs, threading etc to keep it runnable as a game. This has been worked on but currently faces the issues described above. investigate output12.log, output13.log, and output14.log to understand the current state of the demo system. All three are different runs of the same build with the only differences being the dorf spawn locations. Each run has different issues. I have been working on this for a while and I am not sure what the next steps are. Please investigate, then plan our next steps. you can also check `.sisyphus/plans/job_pathfinding_system.md` for more context on where we started, but its likely outdated.
|
|
|
|
-------
|
|
|
|
Try not to read full files, code or logs. Although early days, these files are still LARGE. Grep in the codebase to find where we are logging and expand to see what each log means. Then greps the logs that matter for these bugs and see the orders things are happening in etc. Be smart with token usage in the main session. Delagate to subagents where possible for investigation to conserve tokens and keep the main seesion clear for planning and implementation |