C4/D11 follow-up - the integration review's seven confirmed findings

An adversarial review (four lenses, every finding independently verified by two
skeptics told to refute it) ran over 2a5f6b3 and 8cf8c0f. Seven findings
survived; all seven are fixed here.

Against the C4 fix:
- help.js: the nav hover was renamed onto its own surface token, keeping a
  no-op T9.9 had introduced (two different grays had been mapped to one name).
  Hover is now --cds-layer-hover, the token that exists for exactly this.
- wp-creation-app.js: the drawer's critical CSS pre-painted --cds-layer-accent
  while the stylesheet paints --wp-nav-bg; now both paint --wp-nav-bg.

Against D11:
- wp-sections.js: the Assets toggle note still described the pre-D11 card
  ('Asset tags and controls.dev links') with a rationale the picker inverts.
- runAssetSearch: the result cap counted contains-matches before the exact and
  prefix tiers finished, so 500 alphabetically-early substring hits could evict
  the exact match - and Enter then added the wrong asset, ID-locked. The cap
  now bounds each tier; the scan always sees the whole catalog.
- addCatalogAsset: the one mutation in the section with no announced outcome
  was the successful pick. It now toasts (role=status), matching every sibling
  path (C1).
- assets_db.py: failures are remembered for FAIL_CACHE_SECONDS (default 30s)
  and a stale catalog is served over an error, so a Micron outage costs one
  CONNECT_TIMEOUT per window instead of one per page load stacking up in the
  shared sync threadpool until login itself stalls.
- assets_db.py: MICRON_ASSETS_CACHE_SECONDS='5m' no longer crashes the boot -
  a malformed knob on an OPTIONAL feature degrades to its default, loudly.

assets_check grows four regressions for these (27 -> 31): per-tier cap against
600 decoys, the announced pick, boot with a malformed knob, and the stable
cached 503. Battery: assets_check 31/31, color_check 5/5, sections_check ALL
PASS.

Items: C4, D11.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

server/assets_db.py

@@ -57,8 +57,26 @@ ASSET_QUERY = """
      ORDER BY a.AssetID
 """
 
+def _env_int(name: str, default: int) -> int:
+    """A malformed tuning knob degrades to its default; it must never keep the
+    suite from booting. app.py imports this module unconditionally, so a bare
+    int() here would turn "300s" in someone's .env into a crash-looping API -
+    the total-outage switch an OPTIONAL feature is not allowed to own."""
+    raw = os.getenv(name, "")
+    try:
+        return int(raw.strip()) if raw.strip() else default
+    except ValueError:
+        log.warning("%s=%r is not an integer; using the default %d.", name, raw, default)
+        return default
+
+
 # How long a fetched catalog is reused before the next page load re-queries.
-CACHE_SECONDS = int(os.getenv("MICRON_ASSETS_CACHE_SECONDS", "300"))    # 5 min
+CACHE_SECONDS = _env_int("MICRON_ASSETS_CACHE_SECONDS", 300)    # 5 min
+# How long a FAILURE is remembered before the next request retries the source.
+# Without this, every page load during a Micron outage spends CONNECT_TIMEOUT
+# seconds inside a worker thread; enough concurrent loads exhaust the app's
+# shared sync threadpool and take unrelated endpoints down with the picker.
+FAIL_CACHE_SECONDS = _env_int("MICRON_ASSETS_FAIL_CACHE_SECONDS", 30)
 CONNECT_TIMEOUT = 8
 
 
@@ -149,37 +167,66 @@ def _get_engine():
 # would be one full-table query per person. Held per worker process.
 _cache: list[dict] | None = None
 _cached_at = 0.0
+_error: str | None = None      # negative cache: the last failure's user-safe text
+_error_at = 0.0
 _cache_lock = threading.Lock()
 
 
 def load(force: bool = False) -> list[dict]:
-    """Return the whole catalog as [{'tag': …}, …]. Never writes."""
-    global _cache, _cached_at
+    """Return the whole catalog as [{'tag': …}, …]. Never writes.
+
+    Failures are handled in two tiers so a Micron outage stays the picker's
+    problem and never the suite's (the module contract above):
+      * a previously fetched catalog is served STALE - it is slow-moving
+        reference data, and old-but-real beats an error;
+      * with nothing to serve, the failure itself is cached for
+        FAIL_CACHE_SECONDS, so an outage costs one CONNECT_TIMEOUT per window
+        instead of one per page load stacking up in the shared threadpool."""
+    global _cache, _cached_at, _error, _error_at
     with _cache_lock:
         if _cache is not None and not force and (time.monotonic() - _cached_at) < CACHE_SECONDS:
             return _cache
+        if (_error is not None and not force
+                and (time.monotonic() - _error_at) < FAIL_CACHE_SECONDS
+                and _cache is None):
+            raise AssetSourceError(_error)
 
-    engine = _get_engine()
     try:
+        engine = _get_engine()
         with engine.connect() as conn:
             result = conn.execute(text(ASSET_QUERY)).mappings().all()
+    except AssetSourceError as exc:
+        # _get_engine already logged and sanitised; remember or stale-serve.
+        with _cache_lock:
+            if _cache is not None:
+                log.warning("Micron asset catalog unavailable; serving the cached "
+                            "catalog (%d rows).", len(_cache))
+                return _cache
+            _error, _error_at = str(exc), time.monotonic()
+        raise
     except SQLAlchemyError as exc:
         # The driver's message is NOT propagated. AssetSourceError text reaches the
         # browser, and connection errors quote the host, the login, and — when the
         # URL is malformed — fragments of the password. Operators get the detail
         # from the API log, where it belongs; users get a message they can act on.
         log.error("Micron asset catalog query failed: %s", exc)
-        raise AssetSourceError(
-            "The Micron DB could not be read. Check that the host is "
-            "reachable, that the login has SELECT on the asset table, and that "
-            "ASSET_QUERY matches the real schema — the API log has the driver error."
-        ) from exc
+        msg = ("The Micron DB could not be read. Check that the host is "
+               "reachable, that the login has SELECT on the asset table, and that "
+               "ASSET_QUERY matches the real schema — the API log has the driver error.")
+        with _cache_lock:
+            if _cache is not None:
+                log.warning("Micron asset catalog unavailable; serving the cached "
+                            "catalog (%d rows).", len(_cache))
+                return _cache
+            _error, _error_at = msg, time.monotonic()
+        raise AssetSourceError(msg) from exc
 
     # Drop rows with no identifier — an asset with no tag is not selectable and
     # would render as a blank line in the picker.
     rows = [{"tag": str(r["tag"])} for r in result if r.get("tag") not in (None, "")]
     with _cache_lock:
         _cache, _cached_at = rows, time.monotonic()
+        _error = None
     return rows