specs/openspec/changes/archive/2026-10-09-reconcile-hourly-investment-sync/design.md

# Engineering Design

## Context and Constraints

The deployed backend is a single NestJS process with SQLite/Prisma. Its durable
hourly walk, connection revision publication fence, saved investment resources,
and frontend 30-minute rereads already provide most of the approved model.
The remaining mismatch is at the resource boundary: backend
`InvestmentResourceMetaDto` and `investmentResourceMeta` expose legacy
`missing|partial|complete` availability and `idle|running|failed` work state,
while the frontend projects that legacy metadata through `useResourceQuery` and
uses `sync=observe` as a transport mode. The generated backend `openapi.json`
and frontend snapshot agree with each other, but they describe this AS-IS
contract rather than the approved target.

The backend-generated OpenAPI remains the executable contract. The change-local
`system-analysis/contracts/investment-resource-envelope.schema.json` is a
semantic target and must not become a second OpenAPI source, generated client,
or runtime validator. Route identities, authentication/access boundaries,
Problem Details codes, and resource payload DTOs remain established unless an
approved target requirement explicitly changes them.

No normal personal resource GET may wait for or call T-Bank. The only
acquisition exceptions remain the first unconfirmed ledger interval and first
unconfirmed candle range. The design preserves the single-process boundary; it
does not add distributed scheduling, a job service, polling endpoint, manual
sync command, public value-history read, or a new UI package dependency.

## Technical Goals / Non-Goals

**Goals**

- Publish one generated HTTP envelope for Accounts, Portfolio, operations, and
  events: `{ data, meta }`, with Accounts-only top-level `sync`.
- Make resource-result state and hourly-attempt state independently truthful:
  confirmed data stays available after a failed/partial attempt, while an
  initial result remains `null` until complete.
- Preserve local-read, connection-revision, and first-load invariants while
  migrating Swagger DTOs, mapper inputs, runtime responses, and frontend
  generated types together.
- Keep the existing successful TanStack Query snapshot during read retries and
  retain the existing 30-minute open-screen reread cadence as a local HTTP
  reread, not an import trigger.

**Non-Goals**

- Rebuild source owners, hourly scheduling, portfolio-value storage, or the
  legacy status endpoint beyond removing their use as a source-work command
  from these screens.
- Introduce a compatibility envelope, client-side DTO normalizer, duplicate
  cache, new route/version, manual refresh/import action, or a copy of OpenAPI
  in specs.
- Plan documentation edits, implementation tasks, or product behavior beyond
  the approved proposal, delta specs, and System Analysis.

## Implementation Impact Map

### frontend

`affected` - generated transport types, `shared/api/resource-query.ts`, the
existing brokerage-account/portfolio/operations/events query owners, and their
page VMs/views consume the target metadata. `useScreenSync` remains the common
30-minute open-screen reread owner, but its source-error derivation must read
`meta.lastAttempt` rather than legacy `meta.sync`; retry stays HTTP-only.

### backend

`affected` - investment resource DTOs, mapper, endpoint response DTOs,
resource readers, connection/hourly-attempt projections, Swagger annotations,
and generated `openapi.json` implement the semantic envelope. Existing durable
attempt, revision, membership, snapshot, forecast, coverage, and history
owners remain the data owners.

### specs

`affected` - this Design makes the approved deltas and System Analysis
implementable without duplicating the executable contract.

### docs

`affected` - durable AS-IS references are limited to ADR 019, ADR 017,
`architecture/cross-cutting/data-freshness.md`,
`domains/portfolio/flows/portfolio-loading.md`, and
`domains/income/upcoming-events.md`. Their impact is recorded here only;
documentation sequencing is not designed in this artifact.

### qa

`N/A` - no separate QA/E2E repository exists.

## Decisions

### Backend: one resource-meta projection and one connection-walk projection

Replace the legacy shared metadata vocabulary at the existing
`InvestmentResourceMetaDto`/`investmentResourceMeta` seam. Every affected
resource envelope has required `meta.freshness`, `meta.completeness`,
`meta.lastSuccessfulAt`, `meta.lastAttempt`, and `meta.availability`.
`lastAttempt` is a resource-scoped projection with state
`none|running|succeeded|partial|failed`, nullable start/end times, and only
safe failed section/account/error-code entries. `none` is a concrete status,
not an omitted field or success.

The mapper derives freshness and successful time only from the returned
confirmed resource result. A later running, partial, or failed walk neither
changes that time nor makes data fresh. `data: null` pairs with `unknown`/
incomplete and either `initial-loading`, `temporarily-unavailable`, or
`connection-unavailable`; a confirmed empty collection pairs with available
and complete. The ledger's permitted first-load rows retain incomplete
coverage and are never represented as a confirmed complete result.

Accounts additionally projects the connection-wide walk as `sync: null` when
no attempt exists, otherwise the same complete safe attempt shape. Other
resource envelopes do not gain top-level `sync`. This preserves the single
shared metadata owner while avoiding the current ambiguity between per-resource
`meta.sync` and Accounts-level walk status.

The existing `resourceKey`, `dataRevision`, `coverage`, and dependency details
remain internal projection inputs or endpoint-specific metadata only where
their approved behavior still requires publication; they must not substitute
for any of the five required target meanings. No client must infer target state
from their presence.

### Backend: retain storage owners and publication fences

The durable hourly attempt remains the sole normal publisher. Resource readers
read the saved current connection revision and project its persisted result;
they do not reserve work or contact upstream. Each successful section can
persist independently, preserving the previous confirmed resource result and
its success time when a later section fails. Forecast publication remains
complete-source-only, and exact per-currency value-history writes remain in the
same successful Portfolio publication transaction.

Connection PUT establishes a new revision before its first walk; DELETE and
replacement prevent old personal snapshots and old attempt status from being
read or published. Invalid-token knowledge blocks disclosure rather than being
masked by saved data. Data and storage changes must be additive when a new
persisted distinction is necessary; existing snapshots, coverage, forecasts,
attempts, revisions, and retained value history are not reset or reclassified
by a destructive migration.

### Contract: Swagger is produced at the DTO boundary

Swagger DTO annotations and endpoint response classes are the only place the
wire schema is authored. The backend regenerates `openapi.json` from the
implemented DTOs, then the frontend receives that published snapshot and runs
its existing `openapi-typescript` generation. Hand-editing frontend
`schema.d.ts`, adding handwritten parallel interfaces, or retaining a
compatibility mapper would hide drift and is excluded.

The resulting generated contract must express nullable `data`, all required
`meta` fields, nullable attempt timestamps, nullable problem fields, Accounts
top-level nullable `sync`, and absence of top-level `sync` on the other
resource responses. It must retain `sync=observe` only as a read-observation
compatibility parameter while it exists in the generated route contract; it
cannot cause source work and is not used to represent the new hourly state.
The generated contract reconciles field spelling with the target schema before
release, not through a second contract artifact.

### Frontend: one query cache and metadata-aware screen state

`useResourceQuery` remains the sole adapter that projects resource envelopes
into `data`, `meta`, and Accounts `sync`, preserving its existing query key,
authenticated transport, access-group invalidation, and Client Cached Snapshot
behavior. It must project the generated target types directly. Resource query
owners remain at their existing entity slices; page VMs, not presentation
views, choose initial loading, unavailable/error, empty, and loaded states.

`useScreenSync` remains the single 30-minute timer for open investment
screens. It calls ordinary local `refetch()` and its retry path continues to
use the existing observation read without changing cache identity. It derives
server attempt warning state from `meta.lastAttempt`, keeps transport failures
separate, and never starts personal synchronization. Existing Accounts
30-minute `staleTime` remains aligned with this cadence; Portfolio and other
screen resources keep their current short cache freshness where already
established while the open-screen timer supplies the approved reread rhythm.

All affected account screens render the five metadata meanings in text or an
element with an accessible name, not color alone. A complete empty resource is
normal empty content. `data: null` renders initial-loading or unavailable
states without invented values. A failed/partial hourly attempt is an
independent nonblocking warning beside confirmed data and its original time.
`Portfolio.isStale` remains snapshot-specific and is shown separately from
hourly-attempt and client HTTP-error warnings. A client snapshot stays visible
during and after a failed background HTTP reread; connection replacement or
deletion still invalidates personal queries.

## Cross-cutting Implementation

The canonical state split is:

| Concern | Owner | Observable representation |
| --- | --- | --- |
| Returned confirmed resource result | Saved resource owner | `data`, freshness, completeness, `lastSuccessfulAt`, availability |
| Resource-relevant hourly outcome | Durable walk and resource projection | `meta.lastAttempt` |
| Connection-wide hourly outcome | Durable walk | Accounts `sync` only |
| In-flight/failed browser reread | TanStack Query | Existing query lifecycle and local retry warning |
| Snapshot-specific Portfolio source age | Portfolio owner | `Portfolio.isStale` |

This split prevents a source failure from becoming a synchronous GET failure,
prevents the UI from treating its cache as server fallback, and preserves safe
access invalidation. Safe attempt problems contain only the approved section,
optional account ID, and safe error code: no token, upstream payload, cursor,
connection revision, or raw exception text is serialized or logged as status
metadata.

Events retains its existing account and `all` scopes and its saved forecast
window/filter semantics. Its metadata independently reports the latest events
attempt, so a saved forecast can remain available when a later refresh fails.
Operations retains its coverage semantics; only its explicitly approved missing
historical first-load acquisition may work outside the hourly walk.

## Data / Storage Changes

No new storage architecture is needed. Existing durable tables/records for the
current connection revision, hourly attempts, snapshots, operation coverage,
forecast source windows, and retained portfolio-value points are authoritative.
If implementation needs persisted terminal-state or problem-detail values not
already durable, it extends the existing attempt record/schema additively and
backfills conservatively: absent historical attempt information is projected as
`none`, never as `succeeded`.

Migrations must preserve all current user, connection, personal-resource,
coverage, forecast, and history rows. They must not reset SQLite, rewrite an
old partial result as complete, expose retained history across a new connection
boundary, or delete old data merely to fit the new vocabulary. Compatibility is
achieved by deploying the backend DTO/runtime and its generated OpenAPI as one
server contract, followed by the frontend snapshot/types and rendering change;
there is no claim that an old frontend understands the target envelope.

## Contract Implementation

The implementation contract is the generated result of the following mapping:

| Target concept | Backend source | HTTP location |
| --- | --- | --- |
| Last confirmed result or no complete result | Existing local resource reader | `data` |
| Result freshness and completeness | Resource mapper from confirmed result/coverage | `meta.freshness`, `meta.completeness` |
| Original confirmed-result time | Saved source timestamp | `meta.lastSuccessfulAt` |
| Relevant hourly result | Durable attempt plus resource section projection | `meta.lastAttempt` |
| Whether complete data can be used | Connection/access/read result | `meta.availability` |
| Complete current-connection walk | Durable connection-wide attempt | Accounts `sync` |

The target schema's `problemSection` shape is used for both scopes. For a
connection-wide Accounts walk, its list may include all affected sections; for
a resource-level attempt, it is restricted to failures relevant to that
resource. A running attempt has `finishedAt: null`; terminal attempts have a
completion time; `none` has null timestamps and no problems. These invariants
are validated at runtime tests and in the generated OpenAPI, rather than by
shipping the JSON Schema to application consumers.

## Developer Verification Strategy

Backend verification proves the contract at DTO, mapper, controller, and
SQLite boundaries: successful empty data; first load with `data: null`; stale
fallback after a failed walk; partial walk with preserved data; resource versus
Accounts attempt scoping; events forecast-window filtering; invalid-token and
connection-revision disclosure fences; no upstream personal call on ordinary
GET; and both first-load exceptions staying narrow. It regenerates OpenAPI and
runs the existing OpenAPI contract, type, lint, and relevant E2E suites.

Frontend verification updates the supplied snapshot, runs `openapi:check`, and
uses MSW/page-VM tests to prove generated metadata projection, no automatic
retry, HTTP-only retry, 30-minute open-screen reread, cache retention after a
background error, connection invalidation, and distinct server stale/hourly
attempt/client transport states. UI checks cover keyboard access, accessible
status text, loading/error/empty/loaded states, and narrow viewport wrapping.

Before release, verify the backend-generated OpenAPI and frontend snapshot are
the same published artifact, then run each repository's configured verification
commands. In this specs store, run both v2 schema validation and strict change
validation; structural success is supplemented by a semantic comparison against
the approved deltas and System Analysis.

## UI/UX Package

**Applicability and scope:** `new/changed`. This is an inline package for the
existing Accounts, account Portfolio, operations, and events screens. It adds
the approved status vocabulary and preserves 30-minute local rereads; it does
not introduce new composition, graphics, navigation, or controls.

**Package revision:** `1` (this Design). No visual files, external references,

**Screens, states and interactions:**

| Screen/state | Presentation | Interaction |
| --- | --- | --- |
| No complete result, initial attempt running | Existing structural loading state, labelled as loading | No manual sync action |
| No complete result, attempt failed/unavailable | Safe unavailable/error state with target status text | Retry rereads HTTP only when recovery allows |
| Complete empty result | Normal domain empty state and target status text | Existing navigation/actions only |
| Complete data with running/succeeded/partial/failed attempt | Content and original success time remain visible; attempt appears as separate accessible status | No source-work action |
| Client Cached Snapshot during/after HTTP failure | Existing content remains visible; transport warning remains distinct from server state | Existing retry rereads HTTP only |

**Responsive behavior:** reuse existing Mantine/theme components and page
layouts. Status text wraps without hiding account content or money values on
narrow screens; color supplements an accessible label. Existing skeletons
remain noninteractive and unfocusable. Existing cards, buttons, and icons are
used through their public package exports; no wrapper, SVG asset, token, or
component package revision is introduced.

**Reused source and differences:** `N/A` for approved visual reuse. This is a
new inline status package revision `1`, not an immutable reuse of a prior
approved visual package. Existing cards, layouts, hooks, and UI packages are
implementation inputs only; they are not the approved visual source. The
package adds semantic presentation of the five target metadata fields and
separate hourly-attempt/client-transport warnings. No local visual export is
required.

## Risks / Trade-offs

- Envelope evolution is a coordinated backend/frontend release risk. Generated
  artifacts prevent hidden type drift but cannot make an older client interpret
  new enum meanings.
- Mapping resource relevance from connection-wide attempts can over-report or
  under-report a failure if it is not constrained by resource/section scope;
  mapper tests must cover each affected resource.
- A durable `none` projection for old rows is intentionally less informative
  than inventing a historic success; it preserves truthfulness.
- The 30-minute client timer can issue local reads while an hourly walk runs;
  this is acceptable because reads do not contend for upstream work and retain
  the last confirmed result.
- Single-process scheduling remains unsuitable for multi-process deployment;
  adding leases, a worker, or distributed coordination requires a separate
  decision.
- Documentation currently contains legacy request-driven/manual-refresh
  descriptions. Until the identified durable references are updated, they must
  not be used as implementation authority over the approved behavior.

## Migration / Rollout / Rollback

Rollout is backend-first at the contract source: apply only additive persistence
changes if required, implement and test runtime projection/fences, regenerate
and publish backend OpenAPI, then update the frontend snapshot/generated types
and renderers. Do not enable the new projection alongside a second personal
write owner. A frontend built from the old snapshot is not the compatibility
target; rollout uses the matched generated artifact pair.

| Field | Decision |
| --- | --- |
| Status | `required` |
| Trigger | Personal-data disclosure across connection revisions, false complete/fresh state, malformed generated contract, failure to preserve confirmed fallback, or a frontend unable to read the published contract. |
| Actions | Stop the affected rollout, preserve a SQLite copy, return to the last compatible backend/frontend pair, and fix forward. Do not restore screen-triggered/manual synchronization while the hourly owner remains enabled. |
| Data safety | Never reset production SQLite, delete additive attempt metadata, delete snapshots/coverage/forecasts/value history, or use a destructive downgrade. Old rows without target details safely project `none`. |
| Verification | On a representative copy, verify migration integrity, revision/access fences, ordinary local GET behavior, preserved fallback timestamps, generated OpenAPI/snapshot identity, and frontend rendering before traffic resumes. |

## Solution Assessment

| Option | Assessment |
| --- | --- |
| Evolve the existing shared DTO/mapper and generated OpenAPI | Chosen. It is the narrowest path: one resource-meta projection already owns all affected envelope metadata, Swagger already produces the contract, and frontend types already derive from the supplied snapshot. |
| Add a target adapter/parallel endpoint or versioned envelope | Rejected. It duplicates mapping and cache paths, delays removal of ambiguous states, and creates an unnecessary second compatibility surface. |
| Handwrite frontend target types or runtime validation | Rejected. Generated `paths`/schemas already own transport typing; manual copies would drift and a validator adds no trust boundary. |
| Add a scheduler/queue/cache service to express status | Rejected. Durable single-process attempts and SQLite records already express the required state. A new service cannot improve this contract without changing the explicitly retained deployment model. |
| Add a new design-system package/component family | Rejected. Existing Mantine and `@wealthledger/ui` states cover the textual, accessible status presentation; a new package would be abstraction without a new visual primitive. |

The chosen solution is intentionally a contract-and-projection reconciliation,
not a new synchronization system. It minimizes code and migration surface while
making the approved observable model expressible through the existing owners.