spec: finalize wealth-ledger SDD migration

openspec/changes/archive/2026-10-09-reconcile-hourly-investment-sync/design.md

@@ -0,0 +1,358 @@
+# Engineering Design
+
+## Context and Constraints
+
+The deployed backend is a single NestJS process with SQLite/Prisma. Its durable
+hourly walk, connection revision publication fence, saved investment resources,
+and frontend 30-minute rereads already provide most of the approved model.
+The remaining mismatch is at the resource boundary: backend
+`InvestmentResourceMetaDto` and `investmentResourceMeta` expose legacy
+`missing|partial|complete` availability and `idle|running|failed` work state,
+while the frontend projects that legacy metadata through `useResourceQuery` and
+uses `sync=observe` as a transport mode. The generated backend `openapi.json`
+and frontend snapshot agree with each other, but they describe this AS-IS
+contract rather than the approved target.
+
+The backend-generated OpenAPI remains the executable contract. The change-local
+`system-analysis/contracts/investment-resource-envelope.schema.json` is a
+semantic target and must not become a second OpenAPI source, generated client,
+or runtime validator. Route identities, authentication/access boundaries,
+Problem Details codes, and resource payload DTOs remain established unless an
+approved target requirement explicitly changes them.
+
+No normal personal resource GET may wait for or call T-Bank. The only
+acquisition exceptions remain the first unconfirmed ledger interval and first
+unconfirmed candle range. The design preserves the single-process boundary; it
+does not add distributed scheduling, a job service, polling endpoint, manual
+sync command, public value-history read, or a new UI package dependency.
+
+## Technical Goals / Non-Goals
+
+**Goals**
+
+- Publish one generated HTTP envelope for Accounts, Portfolio, operations, and
+  events: `{ data, meta }`, with Accounts-only top-level `sync`.
+- Make resource-result state and hourly-attempt state independently truthful:
+  confirmed data stays available after a failed/partial attempt, while an
+  initial result remains `null` until complete.
+- Preserve local-read, connection-revision, and first-load invariants while
+  migrating Swagger DTOs, mapper inputs, runtime responses, and frontend
+  generated types together.
+- Keep the existing successful TanStack Query snapshot during read retries and
+  retain the existing 30-minute open-screen reread cadence as a local HTTP
+  reread, not an import trigger.
+
+**Non-Goals**
+
+- Rebuild source owners, hourly scheduling, portfolio-value storage, or the
+  legacy status endpoint beyond removing their use as a source-work command
+  from these screens.
+- Introduce a compatibility envelope, client-side DTO normalizer, duplicate
+  cache, new route/version, manual refresh/import action, or a copy of OpenAPI
+  in specs.
+- Plan documentation edits, implementation tasks, or product behavior beyond
+  the approved proposal, delta specs, and System Analysis.
+
+## Implementation Impact Map
+
+### frontend
+
+`affected` - generated transport types, `shared/api/resource-query.ts`, the
+existing brokerage-account/portfolio/operations/events query owners, and their
+page VMs/views consume the target metadata. `useScreenSync` remains the common
+30-minute open-screen reread owner, but its source-error derivation must read
+`meta.lastAttempt` rather than legacy `meta.sync`; retry stays HTTP-only.
+
+### backend
+
+`affected` - investment resource DTOs, mapper, endpoint response DTOs,
+resource readers, connection/hourly-attempt projections, Swagger annotations,
+and generated `openapi.json` implement the semantic envelope. Existing durable
+attempt, revision, membership, snapshot, forecast, coverage, and history
+owners remain the data owners.
+
+### specs
+
+`affected` - this Design makes the approved deltas and System Analysis
+implementable without duplicating the executable contract.
+
+### docs
+
+`affected` - durable AS-IS references are limited to ADR 019, ADR 017,
+`architecture/cross-cutting/data-freshness.md`,
+`domains/portfolio/flows/portfolio-loading.md`, and
+`domains/income/upcoming-events.md`. Their impact is recorded here only;
+documentation sequencing is not designed in this artifact.
+
+### qa
+
+`N/A` - no separate QA/E2E repository exists.
+
+## Decisions
+
+### Backend: one resource-meta projection and one connection-walk projection
+
+Replace the legacy shared metadata vocabulary at the existing
+`InvestmentResourceMetaDto`/`investmentResourceMeta` seam. Every affected
+resource envelope has required `meta.freshness`, `meta.completeness`,
+`meta.lastSuccessfulAt`, `meta.lastAttempt`, and `meta.availability`.
+`lastAttempt` is a resource-scoped projection with state
+`none|running|succeeded|partial|failed`, nullable start/end times, and only
+safe failed section/account/error-code entries. `none` is a concrete status,
+not an omitted field or success.
+
+The mapper derives freshness and successful time only from the returned
+confirmed resource result. A later running, partial, or failed walk neither
+changes that time nor makes data fresh. `data: null` pairs with `unknown`/
+incomplete and either `initial-loading`, `temporarily-unavailable`, or
+`connection-unavailable`; a confirmed empty collection pairs with available
+and complete. The ledger's permitted first-load rows retain incomplete
+coverage and are never represented as a confirmed complete result.
+
+Accounts additionally projects the connection-wide walk as `sync: null` when
+no attempt exists, otherwise the same complete safe attempt shape. Other
+resource envelopes do not gain top-level `sync`. This preserves the single
+shared metadata owner while avoiding the current ambiguity between per-resource
+`meta.sync` and Accounts-level walk status.
+
+The existing `resourceKey`, `dataRevision`, `coverage`, and dependency details
+remain internal projection inputs or endpoint-specific metadata only where
+their approved behavior still requires publication; they must not substitute
+for any of the five required target meanings. No client must infer target state
+from their presence.
+
+### Backend: retain storage owners and publication fences
+
+The durable hourly attempt remains the sole normal publisher. Resource readers
+read the saved current connection revision and project its persisted result;
+they do not reserve work or contact upstream. Each successful section can
+persist independently, preserving the previous confirmed resource result and
+its success time when a later section fails. Forecast publication remains
+complete-source-only, and exact per-currency value-history writes remain in the
+same successful Portfolio publication transaction.
+
+Connection PUT establishes a new revision before its first walk; DELETE and
+replacement prevent old personal snapshots and old attempt status from being
+read or published. Invalid-token knowledge blocks disclosure rather than being
+masked by saved data. Data and storage changes must be additive when a new
+persisted distinction is necessary; existing snapshots, coverage, forecasts,
+attempts, revisions, and retained value history are not reset or reclassified
+by a destructive migration.
+
+### Contract: Swagger is produced at the DTO boundary
+
+Swagger DTO annotations and endpoint response classes are the only place the
+wire schema is authored. The backend regenerates `openapi.json` from the
+implemented DTOs, then the frontend receives that published snapshot and runs
+its existing `openapi-typescript` generation. Hand-editing frontend
+`schema.d.ts`, adding handwritten parallel interfaces, or retaining a
+compatibility mapper would hide drift and is excluded.
+
+The resulting generated contract must express nullable `data`, all required
+`meta` fields, nullable attempt timestamps, nullable problem fields, Accounts
+top-level nullable `sync`, and absence of top-level `sync` on the other
+resource responses. It must retain `sync=observe` only as a read-observation
+compatibility parameter while it exists in the generated route contract; it
+cannot cause source work and is not used to represent the new hourly state.
+The generated contract reconciles field spelling with the target schema before
+release, not through a second contract artifact.
+
+### Frontend: one query cache and metadata-aware screen state
+
+`useResourceQuery` remains the sole adapter that projects resource envelopes
+into `data`, `meta`, and Accounts `sync`, preserving its existing query key,
+authenticated transport, access-group invalidation, and Client Cached Snapshot
+behavior. It must project the generated target types directly. Resource query
+owners remain at their existing entity slices; page VMs, not presentation
+views, choose initial loading, unavailable/error, empty, and loaded states.
+
+`useScreenSync` remains the single 30-minute timer for open investment
+screens. It calls ordinary local `refetch()` and its retry path continues to
+use the existing observation read without changing cache identity. It derives
+server attempt warning state from `meta.lastAttempt`, keeps transport failures
+separate, and never starts personal synchronization. Existing Accounts
+30-minute `staleTime` remains aligned with this cadence; Portfolio and other
+screen resources keep their current short cache freshness where already
+established while the open-screen timer supplies the approved reread rhythm.
+
+All affected account screens render the five metadata meanings in text or an
+element with an accessible name, not color alone. A complete empty resource is
+normal empty content. `data: null` renders initial-loading or unavailable
+states without invented values. A failed/partial hourly attempt is an
+independent nonblocking warning beside confirmed data and its original time.
+`Portfolio.isStale` remains snapshot-specific and is shown separately from
+hourly-attempt and client HTTP-error warnings. A client snapshot stays visible
+during and after a failed background HTTP reread; connection replacement or
+deletion still invalidates personal queries.
+
+## Cross-cutting Implementation
+
+The canonical state split is:
+
+| Concern | Owner | Observable representation |
+| --- | --- | --- |
+| Returned confirmed resource result | Saved resource owner | `data`, freshness, completeness, `lastSuccessfulAt`, availability |
+| Resource-relevant hourly outcome | Durable walk and resource projection | `meta.lastAttempt` |
+| Connection-wide hourly outcome | Durable walk | Accounts `sync` only |
+| In-flight/failed browser reread | TanStack Query | Existing query lifecycle and local retry warning |
+| Snapshot-specific Portfolio source age | Portfolio owner | `Portfolio.isStale` |
+
+This split prevents a source failure from becoming a synchronous GET failure,
+prevents the UI from treating its cache as server fallback, and preserves safe
+access invalidation. Safe attempt problems contain only the approved section,
+optional account ID, and safe error code: no token, upstream payload, cursor,
+connection revision, or raw exception text is serialized or logged as status
+metadata.
+
+Events retains its existing account and `all` scopes and its saved forecast
+window/filter semantics. Its metadata independently reports the latest events
+attempt, so a saved forecast can remain available when a later refresh fails.
+Operations retains its coverage semantics; only its explicitly approved missing
+historical first-load acquisition may work outside the hourly walk.
+
+## Data / Storage Changes
+
+No new storage architecture is needed. Existing durable tables/records for the
+current connection revision, hourly attempts, snapshots, operation coverage,
+forecast source windows, and retained portfolio-value points are authoritative.
+If implementation needs persisted terminal-state or problem-detail values not
+already durable, it extends the existing attempt record/schema additively and
+backfills conservatively: absent historical attempt information is projected as
+`none`, never as `succeeded`.
+
+Migrations must preserve all current user, connection, personal-resource,
+coverage, forecast, and history rows. They must not reset SQLite, rewrite an
+old partial result as complete, expose retained history across a new connection
+boundary, or delete old data merely to fit the new vocabulary. Compatibility is
+achieved by deploying the backend DTO/runtime and its generated OpenAPI as one
+server contract, followed by the frontend snapshot/types and rendering change;
+there is no claim that an old frontend understands the target envelope.
+
+## Contract Implementation
+
+The implementation contract is the generated result of the following mapping:
+
+| Target concept | Backend source | HTTP location |
+| --- | --- | --- |
+| Last confirmed result or no complete result | Existing local resource reader | `data` |
+| Result freshness and completeness | Resource mapper from confirmed result/coverage | `meta.freshness`, `meta.completeness` |
+| Original confirmed-result time | Saved source timestamp | `meta.lastSuccessfulAt` |
+| Relevant hourly result | Durable attempt plus resource section projection | `meta.lastAttempt` |
+| Whether complete data can be used | Connection/access/read result | `meta.availability` |
+| Complete current-connection walk | Durable connection-wide attempt | Accounts `sync` |
+
+The target schema's `problemSection` shape is used for both scopes. For a
+connection-wide Accounts walk, its list may include all affected sections; for
+a resource-level attempt, it is restricted to failures relevant to that
+resource. A running attempt has `finishedAt: null`; terminal attempts have a
+completion time; `none` has null timestamps and no problems. These invariants
+are validated at runtime tests and in the generated OpenAPI, rather than by
+shipping the JSON Schema to application consumers.
+
+## Developer Verification Strategy
+
+Backend verification proves the contract at DTO, mapper, controller, and
+SQLite boundaries: successful empty data; first load with `data: null`; stale
+fallback after a failed walk; partial walk with preserved data; resource versus
+Accounts attempt scoping; events forecast-window filtering; invalid-token and
+connection-revision disclosure fences; no upstream personal call on ordinary
+GET; and both first-load exceptions staying narrow. It regenerates OpenAPI and
+runs the existing OpenAPI contract, type, lint, and relevant E2E suites.
+
+Frontend verification updates the supplied snapshot, runs `openapi:check`, and
+uses MSW/page-VM tests to prove generated metadata projection, no automatic
+retry, HTTP-only retry, 30-minute open-screen reread, cache retention after a
+background error, connection invalidation, and distinct server stale/hourly
+attempt/client transport states. UI checks cover keyboard access, accessible
+status text, loading/error/empty/loaded states, and narrow viewport wrapping.
+
+Before release, verify the backend-generated OpenAPI and frontend snapshot are
+the same published artifact, then run each repository's configured verification
+commands. In this specs store, run both v2 schema validation and strict change
+validation; structural success is supplemented by a semantic comparison against
+the approved deltas and System Analysis.
+
+## UI/UX Package
+
+**Applicability and scope:** `new/changed`. This is an inline package for the
+existing Accounts, account Portfolio, operations, and events screens. It adds
+the approved status vocabulary and preserves 30-minute local rereads; it does
+not introduce new composition, graphics, navigation, or controls.
+
+**Package revision:** `1` (this Design). No visual files, external references,
+
+**Screens, states and interactions:**
+
+| Screen/state | Presentation | Interaction |
+| --- | --- | --- |
+| No complete result, initial attempt running | Existing structural loading state, labelled as loading | No manual sync action |
+| No complete result, attempt failed/unavailable | Safe unavailable/error state with target status text | Retry rereads HTTP only when recovery allows |
+| Complete empty result | Normal domain empty state and target status text | Existing navigation/actions only |
+| Complete data with running/succeeded/partial/failed attempt | Content and original success time remain visible; attempt appears as separate accessible status | No source-work action |
+| Client Cached Snapshot during/after HTTP failure | Existing content remains visible; transport warning remains distinct from server state | Existing retry rereads HTTP only |
+
+**Responsive behavior:** reuse existing Mantine/theme components and page
+layouts. Status text wraps without hiding account content or money values on
+narrow screens; color supplements an accessible label. Existing skeletons
+remain noninteractive and unfocusable. Existing cards, buttons, and icons are
+used through their public package exports; no wrapper, SVG asset, token, or
+component package revision is introduced.
+
+**Reused source and differences:** `N/A` for approved visual reuse. This is a
+new inline status package revision `1`, not an immutable reuse of a prior
+approved visual package. Existing cards, layouts, hooks, and UI packages are
+implementation inputs only; they are not the approved visual source. The
+package adds semantic presentation of the five target metadata fields and
+separate hourly-attempt/client-transport warnings. No local visual export is
+required.
+
+## Risks / Trade-offs
+
+- Envelope evolution is a coordinated backend/frontend release risk. Generated
+  artifacts prevent hidden type drift but cannot make an older client interpret
+  new enum meanings.
+- Mapping resource relevance from connection-wide attempts can over-report or
+  under-report a failure if it is not constrained by resource/section scope;
+  mapper tests must cover each affected resource.
+- A durable `none` projection for old rows is intentionally less informative
+  than inventing a historic success; it preserves truthfulness.
+- The 30-minute client timer can issue local reads while an hourly walk runs;
+  this is acceptable because reads do not contend for upstream work and retain
+  the last confirmed result.
+- Single-process scheduling remains unsuitable for multi-process deployment;
+  adding leases, a worker, or distributed coordination requires a separate
+  decision.
+- Documentation currently contains legacy request-driven/manual-refresh
+  descriptions. Until the identified durable references are updated, they must
+  not be used as implementation authority over the approved behavior.
+
+## Migration / Rollout / Rollback
+
+Rollout is backend-first at the contract source: apply only additive persistence
+changes if required, implement and test runtime projection/fences, regenerate
+and publish backend OpenAPI, then update the frontend snapshot/generated types
+and renderers. Do not enable the new projection alongside a second personal
+write owner. A frontend built from the old snapshot is not the compatibility
+target; rollout uses the matched generated artifact pair.
+
+| Field | Decision |
+| --- | --- |
+| Status | `required` |
+| Trigger | Personal-data disclosure across connection revisions, false complete/fresh state, malformed generated contract, failure to preserve confirmed fallback, or a frontend unable to read the published contract. |
+| Actions | Stop the affected rollout, preserve a SQLite copy, return to the last compatible backend/frontend pair, and fix forward. Do not restore screen-triggered/manual synchronization while the hourly owner remains enabled. |
+| Data safety | Never reset production SQLite, delete additive attempt metadata, delete snapshots/coverage/forecasts/value history, or use a destructive downgrade. Old rows without target details safely project `none`. |
+| Verification | On a representative copy, verify migration integrity, revision/access fences, ordinary local GET behavior, preserved fallback timestamps, generated OpenAPI/snapshot identity, and frontend rendering before traffic resumes. |
+
+## Solution Assessment
+
+| Option | Assessment |
+| --- | --- |
+| Evolve the existing shared DTO/mapper and generated OpenAPI | Chosen. It is the narrowest path: one resource-meta projection already owns all affected envelope metadata, Swagger already produces the contract, and frontend types already derive from the supplied snapshot. |
+| Add a target adapter/parallel endpoint or versioned envelope | Rejected. It duplicates mapping and cache paths, delays removal of ambiguous states, and creates an unnecessary second compatibility surface. |
+| Handwrite frontend target types or runtime validation | Rejected. Generated `paths`/schemas already own transport typing; manual copies would drift and a validator adds no trust boundary. |
+| Add a scheduler/queue/cache service to express status | Rejected. Durable single-process attempts and SQLite records already express the required state. A new service cannot improve this contract without changing the explicitly retained deployment model. |
+| Add a new design-system package/component family | Rejected. Existing Mantine and `@wealthledger/ui` states cover the textual, accessible status presentation; a new package would be abstraction without a new visual primitive. |
+
+The chosen solution is intentionally a contract-and-projection reconciliation,
+not a new synchronization system. It minimizes code and migration surface while
+making the approved observable model expressible through the existing owners.