docs: explain the Sentinel WAF saga, retire completed plans, commit the needs-warming spec
PROGRESS.md and CLAUDE.md now document the full investigation: three weekly checkpoints confirming the backend's direct link-fetch is 100% dead, the crowdsourced CLI-contribution cache carrying the real load (286 accepted, 0 rejected), the two merged backend fixes (lockdown TTL, /proxy product_id validation), and the CLI TLS fingerprint hardening in this branch (functionally verified, not yet validated against the Sentinel-rejection-rate trend). Removed two plan+spec pairs under docs/superpowers/ (the original CLI build plan and the telemetry/update-check plan+design) -- both fully shipped, redundant with PROGRESS.md's own record of what happened. Committed docs/superpowers/specs/2026-07-13-needs-warming-design.md for the first time -- it's been sitting untracked in the working tree for weeks. Unlike the two removed docs, this one is a live, unbuilt proposal (a public page surfacing which products are currently Sentinel-locked with no cached fallback), not completed work, so it's kept and now tracked in the backlog instead of discarded. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Shekhar Vaidya
parent
9e22cd8bf8
commit
78be837a4a
6 changed files with 264 additions and 2722 deletions
35
CLAUDE.md
35
CLAUDE.md
|
|
@ -37,6 +37,41 @@ negative cache (60s), dynamic TTL, stale-on-failure, jitter. Verified in product
|
|||
- [x] **Recently viewed** — localStorage only. Shows on homepage. Consumer + eval pages both tracked. Expired state shown when link expiry known.
|
||||
- ~~**File size**~~ — not feasible. Microsoft CDN does not return file size in the API response.
|
||||
|
||||
### Sentinel WAF resilience (ongoing)
|
||||
|
||||
Confirmed via three weekly `/metrics` + docker-log checkpoints (2026-07-14 → 07-21 → 07-31):
|
||||
the backend's own direct link-fetch (`/proxy` → Microsoft) is **100% blocked**, permanently —
|
||||
`link.ms_fetches: 0` every time, and zero even-attempted-and-failed fetches in the raw logs
|
||||
(the lockdown gate short-circuits before reaching that code path). The site stays alive
|
||||
entirely on cached/stale entries plus CLI-contributed links (286 accepted, 0 rejected,
|
||||
confirmed 2026-07-31). `/skuinfo` and `/evallinks` are unaffected — only the download-link
|
||||
endpoint is targeted.
|
||||
|
||||
The CLI's own residential-IP requests also see a stable ~20% Sentinel rejection rate across
|
||||
all three checkpoints — not improving, not worsening. Microsoft's own API error response
|
||||
literally names the system (`"Sentinel marked this request as rejected."`, `Type: 9` in
|
||||
`Errors[]`), confirming it's a real, named product, not our guess. The clean structured-JSON
|
||||
deny (not an HTML/JS challenge page) suggests a signature/reputation gate rather than full
|
||||
interactive bot-management — TLS ClientHello fingerprinting is a plausible contributing
|
||||
signal, since Go's stdlib `crypto/tls` doesn't look like any real browser, independent of IP.
|
||||
|
||||
- [x] **Back off Sentinel retries** — bumped `lockdownTTL` 90min → 5h (`backend/main.go`).
|
||||
181 retries over 17 days, 0 successes; retrying that often was pure noise.
|
||||
(fix/sentinel-lockdown-and-proxy-validation, merged)
|
||||
- [x] **Validate `product_id` in `/proxy`** — was passing unknown IDs straight through to a
|
||||
real Microsoft session attempt (found via a stray `product_id=2861`, never a real
|
||||
product, in the logs). Now rejected with 404 before any outbound call. (same PR, merged)
|
||||
- [ ] **CLI TLS/HTTP2 fingerprint hardening** — swapped the CLI's transport from stdlib
|
||||
`net/http` to `github.com/bogdanfinn/tls-client` (wraps `utls` with a maintained Chrome
|
||||
profile). Verified functionally correct (real link fetched + contributed end-to-end),
|
||||
but does NOT yet prove the fingerprint theory — the old client already succeeded ~80%
|
||||
of the time. Need to watch the Sentinel-rejection-rate telemetry over a comparable
|
||||
multi-day window post-merge. (feat/cli-tls-fingerprint-hardening, not yet merged)
|
||||
- [ ] **`/needs-warming` community page** — proposed, not built. Surfaces products currently
|
||||
failing web users (active Sentinel/rate-limit lockdown, no cached/stale link available)
|
||||
with a one-click CLI command to fix it. See
|
||||
`docs/superpowers/specs/2026-07-13-needs-warming-design.md`.
|
||||
|
||||
### Known bugs / open issues
|
||||
|
||||
- [x] **`_redirects` Cloudflare Pages** — investigated 2026-07-21. The rule was indeed ignored
|
||||
|
|
|
|||
46
PROGRESS.md
46
PROGRESS.md
|
|
@ -2,6 +2,51 @@
|
|||
|
||||
## Shipped
|
||||
|
||||
### Sentinel WAF investigation & resilience hardening — backend merged, CLI fix pending
|
||||
|
||||
Three weekly `/metrics` + raw docker-log checkpoints (2026-07-14 → 07-21 → 07-31) turned the
|
||||
"Microsoft blocks our server IP" assumption from a one-off observation into a confirmed,
|
||||
stable fact — and revealed the crowdsourcing architecture below is now doing all the real work.
|
||||
|
||||
**What the data actually showed:**
|
||||
- `link.ms_fetches: 0` at every checkpoint — the backend's own direct link-fetch to Microsoft
|
||||
has not succeeded once, ever, across 17+ days.
|
||||
- Raw docker logs showed not just zero successes but zero *attempted-and-failed* fetches
|
||||
either — the existing Sentinel lockdown gate short-circuits the request before it even
|
||||
reaches Microsoft, serving cache/stale silently instead. 181 blocked session attempts over
|
||||
the same window, all 181 unsuccessful.
|
||||
- The site stayed online anyway: 286 CLI contributions accepted, 0 rejected, in that same
|
||||
window — confirming the crowdsourced cache from the CLI (see architecture entry below) is
|
||||
the thing actually keeping links fresh now, not the backend's own fetch.
|
||||
- `/skuinfo` (125 successful Microsoft fetches) and `/evallinks` (126 cache hits, 0 failures)
|
||||
are both unaffected — Sentinel targets the download-link endpoint specifically.
|
||||
- The CLI's own residential-IP requests held a stable ~20% Sentinel-rejection rate across all
|
||||
three checkpoints too — not improving, not worsening, despite running from IPs that
|
||||
shouldn't trip ASN-based blocking. Traced the exact error text
|
||||
(`"Sentinel marked this request as rejected."`) to Microsoft's own API response
|
||||
(`Errors[0].Value` with `Type: 9`) — confirming "Sentinel" is Microsoft's real name for this
|
||||
system, not our guess, and that it returns a clean structured JSON deny rather than an
|
||||
HTML/JS challenge page (suggesting a signature/reputation gate rather than full interactive
|
||||
bot management).
|
||||
|
||||
**Backend fixes (`fix/sentinel-lockdown-and-proxy-validation`, merged):**
|
||||
- `lockdownTTL` bumped 90min → 5h. 181 retries, 0 successes — retrying that often was pure
|
||||
noise against Microsoft (and our own IP's reputation) for zero return.
|
||||
- `/proxy` now validates `product_id` against the existing catalog allow-list before
|
||||
attempting a Microsoft session. Found via a stray `product_id=2861` (never a real product)
|
||||
in the logs — the backend was burning a full outbound attempt and a Sentinel-block hit on
|
||||
IDs that could never succeed anyway.
|
||||
|
||||
**CLI fix (`feat/cli-tls-fingerprint-hardening`, not yet merged):** swapped the CLI's HTTP
|
||||
transport from stdlib `net/http` to `github.com/bogdanfinn/tls-client` (wraps `utls` with a
|
||||
maintained Chrome TLS+HTTP2 fingerprint), on the theory that Go's default TLS handshake is
|
||||
itself a detectable non-browser signal, independent of IP. Verified functionally correct
|
||||
end-to-end (real link fetched and contributed), but that does *not* prove the theory — the
|
||||
old client already succeeded ~80% of the time. Real validation is watching the
|
||||
Sentinel-rejection-rate telemetry over a comparable multi-day window post-merge.
|
||||
|
||||
---
|
||||
|
||||
### CLI + resilience architecture (formerly `IMPLEMENTATION_PLAN.md`, Phases 1–5) — merged
|
||||
|
||||
Microsoft's Azure Sentinel WAF started blocking the backend's data-center IP (Hetzner) on
|
||||
|
|
@ -120,4 +165,3 @@ CF Worker ensures Hetzner IP is never exposed to Microsoft — rate-limit block
|
|||
| Item | Notes |
|
||||
|---|---|
|
||||
| Per-IP / per-product rate limiter | Revisit after 1 month of production traffic data |
|
||||
| `_redirects` Cloudflare Pages bug | SPA fallback rule flagged as infinite loop, may cause 404s on direct nav |
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,200 +0,0 @@
|
|||
# CLI Telemetry, Update Check & Help — Design Spec
|
||||
|
||||
**Date:** 2026-06-21
|
||||
**Status:** Approved
|
||||
**Scope:** CLI (cli/) + Backend (backend/main.go)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Three related additions to the msdl CLI and backend:
|
||||
|
||||
1. **Telemetry** — anonymous usage counters sent on every CLI run, stored durably in Redis
|
||||
2. **Update check** — CLI checks our backend for the latest version on every run, notifies if behind
|
||||
3. **Help** — `msdl --help` / `-h` prints usage with all flags and examples
|
||||
|
||||
No opt-in required. `MSDL_NO_TELEMETRY=1` skips both telemetry and update check silently.
|
||||
Disclosed in README and `/cli` page with one line each.
|
||||
|
||||
---
|
||||
|
||||
## Backend Changes
|
||||
|
||||
### 1. New constant: `latestCLIVersion`
|
||||
|
||||
```go
|
||||
const latestCLIVersion = "0.3.0"
|
||||
```
|
||||
|
||||
Bumped manually in `backend/main.go` whenever a new CLI release is cut.
|
||||
|
||||
### 2. `GET /cli/version` — public, no auth
|
||||
|
||||
Returns the latest CLI version. The CLI calls this on every run.
|
||||
|
||||
```json
|
||||
{ "latest": "0.3.0" }
|
||||
```
|
||||
|
||||
No rate limiting needed — response is trivial and cacheable by the client.
|
||||
|
||||
### 3. `POST /telemetry` — public, no auth
|
||||
|
||||
Accepts one event per CLI run. Body:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "fetch" | "eval" | "list" | "interactive",
|
||||
"product_id": "2618",
|
||||
"eval_slug": "server-2025",
|
||||
"platform": "windows" | "darwin" | "linux",
|
||||
"version": "0.3.0",
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
- `product_id` present only for `fetch` action
|
||||
- `eval_slug` present only for `eval` action
|
||||
- All other fields always present
|
||||
- Returns `200 OK` with `{}` — CLI ignores the response
|
||||
- Per-IP rate limit: ~10 req/min (same token bucket pattern as `/contribute`)
|
||||
- Increments Redis counters via `HINCRBY` directly (no in-memory layer)
|
||||
- If Redis unavailable: silently drop (telemetry is best-effort)
|
||||
|
||||
### 4. Redis telemetry keys
|
||||
|
||||
```
|
||||
msdl:telemetry:actions hash { fetch, eval, list, interactive }
|
||||
msdl:telemetry:platforms hash { windows, darwin, linux }
|
||||
msdl:telemetry:versions hash { "0.2.0", "0.3.0", ... }
|
||||
msdl:telemetry:products hash { "2618", "3262", "3113", ... }
|
||||
msdl:telemetry:results hash { success, failed }
|
||||
```
|
||||
|
||||
### 5. Existing metrics — Redis persistence
|
||||
|
||||
Current in-memory `atomic.Int64` counters are preserved for fast per-request increments. Added durability:
|
||||
|
||||
- **On startup**: seed in-memory counters from Redis (`HGETALL msdl:metrics:*`)
|
||||
- **Every 5 minutes**: flush in-memory counters to Redis (`HSET msdl:metrics:*`)
|
||||
- **On graceful shutdown**: final flush before exit
|
||||
|
||||
Redis keys for existing metrics:
|
||||
|
||||
```
|
||||
msdl:metrics:sku hash { requests, hits, fetches, neg_hits }
|
||||
msdl:metrics:link hash { requests, hits, fetches, neg_hits, stale }
|
||||
msdl:metrics:eval hash { requests, hits, stale }
|
||||
```
|
||||
|
||||
### 6. `/metrics` response — updated
|
||||
|
||||
Adds telemetry section to existing cache stats:
|
||||
|
||||
```json
|
||||
{
|
||||
"sku": { ... },
|
||||
"link": { ... },
|
||||
"eval": { ... },
|
||||
"telemetry": {
|
||||
"actions": { "fetch": 142, "eval": 31, "list": 8, "interactive": 67 },
|
||||
"platforms": { "windows": 180, "darwin": 42, "linux": 18 },
|
||||
"versions": { "0.3.0": 145, "0.2.0": 95 },
|
||||
"products": { "3262": 134, "2618": 89, "3113": 67 },
|
||||
"results": { "success": 390, "failed": 48 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Changes
|
||||
|
||||
### 1. Version constant
|
||||
|
||||
Injected at build time via ldflags:
|
||||
|
||||
```
|
||||
-ldflags "-X main.Version=0.3.0"
|
||||
```
|
||||
|
||||
Added to the existing GitHub Actions release workflow (`cli-release.yml`). Fallback: `const Version = "dev"` in source.
|
||||
|
||||
### 2. Update check + telemetry — concurrent goroutines
|
||||
|
||||
On every run, immediately after flag parsing, two goroutines are launched:
|
||||
|
||||
```
|
||||
main()
|
||||
├── go updateCheck() — GET /cli/version, 500ms timeout, print notice if behind
|
||||
└── go sendTelemetry() — POST /telemetry, fire-and-forget, result ignored
|
||||
```
|
||||
|
||||
Both are non-blocking. The update check result is printed before any other output if it arrives within 500ms. If the timeout fires, it's silently skipped for that run.
|
||||
|
||||
**Update notice format** (printed before picker or output):
|
||||
```
|
||||
A new version of msdl is available: v0.3.0
|
||||
Download: https://github.com/starkSV/windows-iso-downloader/releases/latest
|
||||
```
|
||||
|
||||
Both goroutines are skipped entirely when `MSDL_NO_TELEMETRY=1` is set.
|
||||
|
||||
### 3. Telemetry payload construction
|
||||
|
||||
| Scenario | action | product_id | eval_slug | success |
|
||||
|---|---|---|---|---|
|
||||
| `msdl` (interactive) | `interactive` | set after pick | — | true/false |
|
||||
| `msdl --id X --lang Y` | `fetch` | X | — | true/false |
|
||||
| `msdl --eval slug` | `eval` | — | slug | true/false |
|
||||
| `msdl --list` | `list` | — | — | true |
|
||||
|
||||
`platform` is set from `runtime.GOOS` at runtime (not build time — same result, simpler).
|
||||
`version` is set from the `Version` constant.
|
||||
|
||||
Telemetry is sent **after** the main action completes so `success` reflects the actual outcome.
|
||||
|
||||
### 4. `--help` / `-h`
|
||||
|
||||
Prints a formatted usage block and exits 0. Shown automatically by Go's `flag` package or via a custom print if the existing CLI uses manual arg parsing.
|
||||
|
||||
```
|
||||
msdl — Windows ISO downloader
|
||||
|
||||
Usage:
|
||||
msdl Interactive mode — pick product and language
|
||||
msdl --id <id> --lang <lang> Fetch link directly (skip picker)
|
||||
msdl --eval <slug> Evaluation / Server ISO
|
||||
msdl --list List all available products
|
||||
|
||||
Flags:
|
||||
--id <id> Product ID (e.g. 3262 for Windows 11 25H2)
|
||||
--lang <language> Language name (e.g. "English")
|
||||
--eval <slug> Eval ISO slug (server-2025, server-2022, win11-ent, ...)
|
||||
--list List all products and exit
|
||||
--no-contribute Skip contributing link back to msdl web cache
|
||||
-h, --help Show this help
|
||||
|
||||
Environment:
|
||||
MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks
|
||||
MSDL_NO_CONTRIBUTE=1 Disable cache contribution
|
||||
|
||||
More info: https://msdl.tech-latest.com/cli
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- **README** — one line under CLI section: "msdl sends anonymous usage counts (action, platform, version) to help us understand which products are popular. Set `MSDL_NO_TELEMETRY=1` to opt out."
|
||||
- **`/cli` page** — same one-liner in the contribute section, next to the `--no-contribute` note.
|
||||
|
||||
---
|
||||
|
||||
## What's Not In Scope
|
||||
|
||||
- No per-user tracking, no IP storage, no session IDs
|
||||
- No telemetry dashboard UI (raw numbers via `/metrics` is sufficient)
|
||||
- No forced update or auto-update — notification only
|
||||
- No telemetry for the web frontend
|
||||
184
docs/superpowers/specs/2026-07-13-needs-warming-design.md
Normal file
184
docs/superpowers/specs/2026-07-13-needs-warming-design.md
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
# Spec: "Needs Warming" Community Help Page
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-07-13
|
||||
**Author:** Shekhar + Grok (original idea), refined against live codebase by Claude
|
||||
**Related:** Crowdsourced cache warming (`/contribute`), Sentinel lockdown resilience, CLI contribution flow
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Give visible, honest signal for products currently failing web users because Microsoft's Sentinel WAF (or a rate limit) has blocked the backend and no cached link is available to fall back on. Point people at the CLI as the fix, and let a successful fetch/contribution clear the item automatically.
|
||||
|
||||
This is not a new mechanism bolted onto the project — it's a public window into resilience state (`negCache` / `linkCache`) that already exists and already drives the per-product "blocked, use CLI" banner. This just makes that signal visible in aggregate, with a stronger call to action.
|
||||
|
||||
**Change from the original draft:** a dedicated page, not a homepage section. Original draft (written by Grok from the GitHub repo alone, without runtime visibility into cache state) proposed a parallel tracking system; this version derives the list from state that already exists, which is both less code and immune to drift between two copies of "is this broken."
|
||||
|
||||
---
|
||||
|
||||
## 2. What Actually Needs Tracking (verified against `backend/main.go`)
|
||||
|
||||
`handleProxy` (main.go:1189-1404) has exactly three outcomes when a product/SKU is under an active Sentinel or rate-limit block:
|
||||
|
||||
| Outcome | Code path | User experience | Needs warming? |
|
||||
|---|---|---|---|
|
||||
| `serving cached` | negCache active, `linkCache` entry still within its own TTL (e.g. just contributed) | Gets a working link | **No** |
|
||||
| `serving stale` | negCache active, `linkCache` entry expired but still returned | Gets a (possibly near-expired) link | **No** |
|
||||
| `no stale available` | negCache active, nothing in `linkCache` at all | Gets a 429, no link | **Yes** — this is the actual failure |
|
||||
|
||||
So "needs warming" = **an active `negCache` entry with no corresponding valid-or-stale `linkCache` entry** — computed on read, not tracked as separate state. No new manager struct, no `Record()`/`Remove()` calls to keep in sync, no new Redis schema.
|
||||
|
||||
**Known gap, scoped out of MVP:** a fetch that fails with a non-rate-limit error (network blip, malformed response) and has no stale cache to fall back on returns an error directly (main.go:1377) without ever touching `negCache`. In practice this hasn't been the dominant failure mode this month — Sentinel/rate-limit blocks have been — so the MVP doesn't cover it. If it turns out to matter, closing the gap is a two-line addition (write a `negCacheEntry` with `IsSentinel: false` at that call site) rather than a redesign.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Needed That Doesn't Exist Yet
|
||||
|
||||
Two small additions, both minimal:
|
||||
|
||||
1. **A request counter per locked product/SKU.** `negCache`/`linkCache` don't track "how many times did this get hit while broken" — add a small `map[string]int` (or extend `negCacheEntry` with a counter field) incremented at the existing `"no stale available"` log call sites.
|
||||
2. **Consumer product display names on the backend.** `validContributeProducts` (main.go:476) is `map[string]bool` — just an ID allow-list, no names. Eval products already have names (`{Name, EvalURL}` map at main.go:207-211). Converting `validContributeProducts` to `map[string]string` (ID → Name) serves both its existing validation role and this new lookup need, with no behavior change to `/contribute`.
|
||||
|
||||
Language name hydration for consumer products reuses the existing `skuCache` (already populated per-product from the normal SKU-fetch flow) — look up the SKU by ID within that product's cached SKU list to get `Language`/`LocalizedLanguage`. No new structure needed; if the product isn't in `skuCache` yet, just omit the language field rather than blocking the response.
|
||||
|
||||
---
|
||||
|
||||
## 4. New Endpoint
|
||||
|
||||
### `GET /needs-warming`
|
||||
|
||||
Computed on each request by iterating `negCache`, filtering to entries where `IsSentinel` (or rate-limited) is still active **and** no valid `linkCache`/eval-cache entry exists for that key. No background job, no persistence — it's a live view.
|
||||
|
||||
**Query params:** `limit` (default 10, max 50)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"product_id": "3262",
|
||||
"sku_id": "0x0409",
|
||||
"product_name": "Windows 11 25H2",
|
||||
"language": "English (United States)",
|
||||
"is_eval": false,
|
||||
"reason": "waf_blocked",
|
||||
"last_seen": "2026-07-13T17:42:00Z",
|
||||
"request_count": 47,
|
||||
"cli_command": "msdl --id 3262 --lang \"English (United States)\""
|
||||
}
|
||||
],
|
||||
"total": 3
|
||||
}
|
||||
```
|
||||
|
||||
- Public, unauthenticated, rate-limited (30 req/min per IP — matches the existing pattern used by `/contribute` at 5/min and `/telemetry` at 10/min)
|
||||
- Sorted by `request_count` descending (surfaces the highest-impact items first, more useful than recency alone)
|
||||
- Eval products included (`is_eval: true`, `cli_command` uses `msdl --eval <slug>` instead of `--id`)
|
||||
|
||||
---
|
||||
|
||||
## 5. Backend Implementation Outline
|
||||
|
||||
No new package-level state beyond the two additions in §3. The handler:
|
||||
|
||||
```go
|
||||
func handleNeedsWarming(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
if !needsWarmingRL.allow(clientIP(r)) {
|
||||
respondJSONError(w, http.StatusTooManyRequests, "rate limit exceeded")
|
||||
return
|
||||
}
|
||||
limit := parseLimitParam(r, 10, 50)
|
||||
|
||||
negCacheMu.RLock()
|
||||
defer negCacheMu.RUnlock()
|
||||
linkCacheMu.RLock()
|
||||
defer linkCacheMu.RUnlock()
|
||||
|
||||
var items []needsWarmingItem
|
||||
for key, neg := range negCache {
|
||||
if time.Now().After(neg.ExpiresAt) {
|
||||
continue // lockdown/rate-limit window itself already expired
|
||||
}
|
||||
if _, hasValidLink := resolveValidLink(key, linkCache); hasValidLink {
|
||||
continue // already resolved via cached/stale link -- not a failure state
|
||||
}
|
||||
items = append(items, buildNeedsWarmingItem(key, neg))
|
||||
}
|
||||
sort.Slice(items, func(i, j int) bool { return items[i].RequestCount > items[j].RequestCount })
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
}
|
||||
json.NewEncoder(w).Encode(map[string]interface{}{"items": items, "total": len(items)})
|
||||
}
|
||||
```
|
||||
|
||||
Integration points:
|
||||
- Increment the new request counter at the existing `"lockdown active, no stale available"` and first-time-Sentinel/rate-limit-with-no-stale log sites (main.go:1225, ~1367)
|
||||
- No changes needed to `/contribute` or the fresh-fetch success path — an item stops appearing automatically once `linkCache` has a valid entry, which those paths already populate
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend: Dedicated Page
|
||||
|
||||
**Route:** `/needs-warming` (or `/community` — bikeshed-able, not load-bearing)
|
||||
|
||||
Linked from the footer and from the per-product "blocked" banner (`CliHandoff.tsx`'s `highlight` variant) — "See what else needs help →" — so people already primed to help via the CLI can see the fuller picture instead of just their one product.
|
||||
|
||||
**Empty state:** hide the page's list section entirely (or show a short "Everything's healthy right now" line) — no "0 items" placeholder box. Most visits will likely find this empty, which is a good sign, not a gap to fill with UI.
|
||||
|
||||
Each card:
|
||||
- Product name + language (or eval product name)
|
||||
- Request count as a small badge (social proof — "47 people hit this")
|
||||
- One-click "Copy CLI command" button (same interaction pattern as the existing `CliHandoff` component — reuse its copy-button styling)
|
||||
- No per-item "reason" jargon exposed in the UI (WAF/rate-limit distinction is backend detail); just "temporarily unavailable"
|
||||
|
||||
Tone: matches the project's existing calm, transparent voice (README already explains the caching layer and Sentinel lockdown openly) — helpful, not alarmist.
|
||||
|
||||
---
|
||||
|
||||
## 7. CLI Impact
|
||||
|
||||
None required for MVP. Optional future addition: `msdl --needs-warming` prints the same list from the terminal and offers to fetch+contribute the top item directly — natural fit for the homepage-screen work already shipped (`cli/homepage.go`), but a separate phase.
|
||||
|
||||
---
|
||||
|
||||
## 8. Privacy & Safety
|
||||
|
||||
- Only ever stores/exposes `product_id`, `sku_id`/slug, reason, counters, timestamps — no IPs, no user identifiers (matches the existing telemetry/contribute philosophy already documented in README)
|
||||
- Rate-limited public endpoint, same pattern as existing public endpoints
|
||||
- Nothing new to abuse: the list is derived from state Microsoft's own WAF already put the backend into, not something a client can inject
|
||||
|
||||
---
|
||||
|
||||
## 9. Implementation Phases
|
||||
|
||||
| Phase | Description | Effort |
|
||||
|---|---|---|
|
||||
| MVP | Request counter + product-name map conversion + `/needs-warming` handler + dedicated page | Small — most of the state already exists |
|
||||
| 2 | Link from per-product blocked banner; polish empty/loading states | Small |
|
||||
| 3 | Close the non-rate-limit fetch-error gap (§2) if it turns out to matter in practice | Small, only if needed |
|
||||
| 4 | `msdl --needs-warming` CLI command | Small |
|
||||
|
||||
(Dropped the original draft's "Redis-backed persistence" phase — this is inherently live/transient state mirroring `negCache`'s own in-memory, reset-on-restart behavior; persisting it would add a Redis key namespace for no real benefit.)
|
||||
|
||||
---
|
||||
|
||||
## 10. Open Questions Resolved
|
||||
|
||||
1. **Eval products included?** Yes.
|
||||
2. **Max age before auto-removal?** No separate TTL — inherits the underlying `negCache` entry's own expiry (60s rate-limit / 90min Sentinel), since the list is computed live rather than tracked separately.
|
||||
3. **Empty state?** Hide the list, don't show a "0 items" box.
|
||||
4. **Tone?** Helpful/neutral, matching existing project voice.
|
||||
|
||||
---
|
||||
|
||||
## 11. Success Metrics (Future)
|
||||
|
||||
- Contributions triggered from this page specifically (could tag `/contribute` calls with a `source=needs-warming` query param to measure this)
|
||||
- Reduction in time-to-resolution for locked-down products
|
||||
- CLI download/usage bump correlated with items appearing on the page
|
||||
Loading…
Add table
Reference in a new issue