Skip to content

Workflow status is tracked in GitHub: https://github.com/emulebb/emulebb/issues/128. This local document is retained as an engineering spec/evidence record.

REF-054 - Stabilize and instrument download queue behavior

Summary

Add a narrow diagnostics and hardening lane for download queue behavior. The first slice should explain why downloads stall or remain idle. Later slices should stabilize ownership edges that can leave stale request or source state, while preserving stock-compatible wire semantics and current queue policy.

This item complements REF-048, which already tracks queue scan-cost reduction such as local server source-request batching.

Current Evidence

  • Upload-side diagnostics exists behind EMULEBB_ENABLE_UPLOAD_SLOT_DIAGNOSTICS, but there is no matching download queue diagnostics flag.
  • CUpDownClient::CreateBlockRequests() receives reservations from CPartFile::GetNextRequestedBlock() after the part file has linked them into requestedblocks_list; if client-side pending-block allocation or list insertion fails, a block can remain reserved without a client pending owner.
  • CDownloadQueue::ProcessLocalRequests() repeatedly scans the local source request queue to choose up to 15 files; this is covered by REF-048 and should not be duplicated here.
  • UDP reask failure tracking is called out in source as total-failure based rather than recent or consecutive-failure based.
  • A4AF cleanup has stale pointer recovery paths, but some swap/debug paths still need careful validation before dereferencing candidate part-file pointers.
  • Runtime logs from the June 3 review did not show a current download queue stall, so behavior changes should be preceded by targeted diagnostics.

Intended Shape

  1. Add compile-time download queue diagnostics.
  2. Harden block reservation ownership so requested blocks cannot be orphaned by allocation or list-link failures.
  3. Keep local source-request batching work in REF-048 unless this item only adds counters for it.
  4. Change UDP reask failure tracking from total historical failures toward recent or consecutive failures.
  5. Guard A4AF stale/debug paths before dereferencing part-file pointers and drop stale entries through existing mirror cleanup paths.
  6. Defer duplicate-source indexes and priority-sort rewrites until diagnostics proves a hotspot.

Diagnostics Scope

The download diagnostics flag should report aggregate state at a bounded cadence, not per-packet chatter. Useful fields include:

  • active, paused, stopped, completing, and error part-file counts
  • per-file source state counts: downloading, on queue, none-needed-parts, remote queue full, low-to-low, connecting, too-many-connections, banned
  • local TCP source-request queue length and oldest request age
  • Kad search active count and next eligible search age
  • pending block counts by active source
  • requested block count by part file
  • async write backlog and buffered byte pressure
  • download timeout age since last block received
  • stale source or A4AF mirror repairs
  • block reservation rollback or orphan-prevention events

Scope Constraints

  • Do not change eD2K/Kad wire semantics, packet shapes, opcodes, source admission, source request cadence, queue caps, or default scheduling policy.
  • Do not introduce secondary client/source indexes until all key mutation points are known and debug-verified.
  • Do not merge this work with broad async socket replacement or queue policy rewrites.
  • Keep the diagnostics compile-time gated and cheap when disabled.
  • Keep behavior slices granular: diagnostics, block ownership hardening, UDP reask tracking, and A4AF stale-pointer cleanup should land separately.

Candidate Implementation Notes

  • Mirror the upload diagnostics build flag pattern with a download-specific preprocessor definition and orchestration environment variable.
  • Add seam tests for reservation publication rollback before changing runtime block ownership.
  • Model UDP reask failure policy with a pure helper so recent/consecutive behavior can be tested without live peers.
  • Treat A4AF stale-pointer work as a defensive cleanup, not a swap scoring change.
  • Add counters before considering duplicate-source indexes or a vector-backed priority sort.

Acceptance Criteria

  • [ ] A compile-time download diagnostics flag exists and is wired through workspace build orchestration.
  • [ ] Download diagnostics explains source states, request queues, pending blocks, requested blocks, async write backlog, and timeout age.
  • [ ] Block reservation publication cannot leave requested blocks without a client pending owner after allocation or list-link failure.
  • [ ] UDP reask failure tracking is based on recent or consecutive failures rather than unbounded historical failures.
  • [ ] A4AF stale/debug paths validate part-file ownership before dereference.
  • [ ] Duplicate-source indexes and priority-sort rewrites remain deferred until diagnostics evidence justifies them.

Validation

  • focused native tests for block reservation rollback
  • focused native tests for UDP reask recent/consecutive failure policy
  • focused native tests for A4AF stale-pointer cleanup if a seam is introduced
  • diagnostics smoke with the compiler flag enabled
  • python -m emule_workspace validate
  • x64 Debug and Release app builds before implementation commit