diff --git a/CLAUDE.md b/CLAUDE.md
index b876e3c..e421101 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -37,6 +37,41 @@ negative cache (60s), dynamic TTL, stale-on-failure, jitter. Verified in product
- [x] **Recently viewed** — localStorage only. Shows on homepage. Consumer + eval pages both tracked. Expired state shown when link expiry known.
- ~~**File size**~~ — not feasible. Microsoft CDN does not return file size in the API response.
+### Sentinel WAF resilience (ongoing)
+
+Confirmed via three weekly `/metrics` + docker-log checkpoints (2026-07-14 → 07-21 → 07-31):
+the backend's own direct link-fetch (`/proxy` → Microsoft) is **100% blocked**, permanently —
+`link.ms_fetches: 0` every time, and zero even-attempted-and-failed fetches in the raw logs
+(the lockdown gate short-circuits before reaching that code path). The site stays alive
+entirely on cached/stale entries plus CLI-contributed links (286 accepted, 0 rejected,
+confirmed 2026-07-31). `/skuinfo` and `/evallinks` are unaffected — only the download-link
+endpoint is targeted.
+
+The CLI's own residential-IP requests also see a stable ~20% Sentinel rejection rate across
+all three checkpoints — not improving, not worsening. Microsoft's own API error response
+literally names the system (`"Sentinel marked this request as rejected."`, `Type: 9` in
+`Errors[]`), confirming it's a real, named product, not our guess. The clean structured-JSON
+deny (not an HTML/JS challenge page) suggests a signature/reputation gate rather than full
+interactive bot-management — TLS ClientHello fingerprinting is a plausible contributing
+signal, since Go's stdlib `crypto/tls` doesn't look like any real browser, independent of IP.
+
+- [x] **Back off Sentinel retries** — bumped `lockdownTTL` 90min → 5h (`backend/main.go`).
+ 181 retries over 17 days, 0 successes; retrying that often was pure noise.
+ (fix/sentinel-lockdown-and-proxy-validation, merged)
+- [x] **Validate `product_id` in `/proxy`** — was passing unknown IDs straight through to a
+ real Microsoft session attempt (found via a stray `product_id=2861`, never a real
+ product, in the logs). Now rejected with 404 before any outbound call. (same PR, merged)
+- [ ] **CLI TLS/HTTP2 fingerprint hardening** — swapped the CLI's transport from stdlib
+ `net/http` to `github.com/bogdanfinn/tls-client` (wraps `utls` with a maintained Chrome
+ profile). Verified functionally correct (real link fetched + contributed end-to-end),
+ but does NOT yet prove the fingerprint theory — the old client already succeeded ~80%
+ of the time. Need to watch the Sentinel-rejection-rate telemetry over a comparable
+ multi-day window post-merge. (feat/cli-tls-fingerprint-hardening, not yet merged)
+- [ ] **`/needs-warming` community page** — proposed, not built. Surfaces products currently
+ failing web users (active Sentinel/rate-limit lockdown, no cached/stale link available)
+ with a one-click CLI command to fix it. See
+ `docs/superpowers/specs/2026-07-13-needs-warming-design.md`.
+
### Known bugs / open issues
- [x] **`_redirects` Cloudflare Pages** — investigated 2026-07-21. The rule was indeed ignored
diff --git a/PROGRESS.md b/PROGRESS.md
index 4e0458f..3707b99 100644
--- a/PROGRESS.md
+++ b/PROGRESS.md
@@ -2,6 +2,51 @@
## Shipped
+### Sentinel WAF investigation & resilience hardening — backend merged, CLI fix pending
+
+Three weekly `/metrics` + raw docker-log checkpoints (2026-07-14 → 07-21 → 07-31) turned the
+"Microsoft blocks our server IP" assumption from a one-off observation into a confirmed,
+stable fact — and revealed the crowdsourcing architecture below is now doing all the real work.
+
+**What the data actually showed:**
+- `link.ms_fetches: 0` at every checkpoint — the backend's own direct link-fetch to Microsoft
+ has not succeeded once, ever, across 17+ days.
+- Raw docker logs showed not just zero successes but zero *attempted-and-failed* fetches
+ either — the existing Sentinel lockdown gate short-circuits the request before it even
+ reaches Microsoft, serving cache/stale silently instead. 181 blocked session attempts over
+ the same window, all 181 unsuccessful.
+- The site stayed online anyway: 286 CLI contributions accepted, 0 rejected, in that same
+ window — confirming the crowdsourced cache from the CLI (see architecture entry below) is
+ the thing actually keeping links fresh now, not the backend's own fetch.
+- `/skuinfo` (125 successful Microsoft fetches) and `/evallinks` (126 cache hits, 0 failures)
+ are both unaffected — Sentinel targets the download-link endpoint specifically.
+- The CLI's own residential-IP requests held a stable ~20% Sentinel-rejection rate across all
+ three checkpoints too — not improving, not worsening, despite running from IPs that
+ shouldn't trip ASN-based blocking. Traced the exact error text
+ (`"Sentinel marked this request as rejected."`) to Microsoft's own API response
+ (`Errors[0].Value` with `Type: 9`) — confirming "Sentinel" is Microsoft's real name for this
+ system, not our guess, and that it returns a clean structured JSON deny rather than an
+ HTML/JS challenge page (suggesting a signature/reputation gate rather than full interactive
+ bot management).
+
+**Backend fixes (`fix/sentinel-lockdown-and-proxy-validation`, merged):**
+- `lockdownTTL` bumped 90min → 5h. 181 retries, 0 successes — retrying that often was pure
+ noise against Microsoft (and our own IP's reputation) for zero return.
+- `/proxy` now validates `product_id` against the existing catalog allow-list before
+ attempting a Microsoft session. Found via a stray `product_id=2861` (never a real product)
+ in the logs — the backend was burning a full outbound attempt and a Sentinel-block hit on
+ IDs that could never succeed anyway.
+
+**CLI fix (`feat/cli-tls-fingerprint-hardening`, not yet merged):** swapped the CLI's HTTP
+transport from stdlib `net/http` to `github.com/bogdanfinn/tls-client` (wraps `utls` with a
+maintained Chrome TLS+HTTP2 fingerprint), on the theory that Go's default TLS handshake is
+itself a detectable non-browser signal, independent of IP. Verified functionally correct
+end-to-end (real link fetched and contributed), but that does *not* prove the theory — the
+old client already succeeded ~80% of the time. Real validation is watching the
+Sentinel-rejection-rate telemetry over a comparable multi-day window post-merge.
+
+---
+
### CLI + resilience architecture (formerly `IMPLEMENTATION_PLAN.md`, Phases 1–5) — merged
Microsoft's Azure Sentinel WAF started blocking the backend's data-center IP (Hetzner) on
@@ -120,4 +165,3 @@ CF Worker ensures Hetzner IP is never exposed to Microsoft — rate-limit block
| Item | Notes |
|---|---|
| Per-IP / per-product rate limiter | Revisit after 1 month of production traffic data |
-| `_redirects` Cloudflare Pages bug | SPA fallback rule flagged as infinite loop, may cause 404s on direct nav |
diff --git a/docs/superpowers/plans/2026-06-15-msdl-cli.md b/docs/superpowers/plans/2026-06-15-msdl-cli.md
deleted file mode 100644
index 48c0d51..0000000
--- a/docs/superpowers/plans/2026-06-15-msdl-cli.md
+++ /dev/null
@@ -1,1498 +0,0 @@
-# MSDL CLI Implementation Plan
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Standalone `msdl` Go binary that fetches Windows ISO download URLs **directly from Microsoft** (from the user's own machine/IP) and prints them to stdout, with both interactive and named-arg modes.
-
-**Architecture:** The CLI performs the entire Microsoft flow locally — session setup (vlscppe → ov-df fingerprinting chain with a domain-less cookie jar), SKU/language lookup, and download-link fetch — ported from `backend/main.go`. **It does NOT call the MSDL backend for fetching.** Because each request originates from the user's residential/commercial IP, it bypasses the Azure Sentinel WAF that blocks the data-center backend (the Rufus/Fido model). Eval builds are likewise scraped directly from Microsoft's Eval Center. Interactive mode presents numbered menus; named args skip the relevant pickers. URLs go to stdout, all prompts to stderr (so `msdl ... | xargs wget` works).
-
-> **Why this changed:** This plan originally routed the CLI through the MSDL backend. Microsoft's Sentinel WAF now blocks the backend's data-center IP (Hetzner) on the link endpoint. See [`IMPLEMENTATION_PLAN.md`](../../../IMPLEMENTATION_PLAN.md) §2 for the full architecture decision. Crowdsourced contribution back to the backend (`--contribute`) is a separate phase covered there, not in this plan.
-
-**Tech Stack:** Go 1.21+, **zero external dependencies** — standard library only (`net/http` with a custom domain-less cookie jar, `crypto/rand`+`encoding/hex` for the session UUID, `encoding/json`, `regexp`, `html`, `flag`, `bufio`, `os`, `strings`, `sync`, `time`). Standalone module in `cli/` directory.
-
----
-
-## File Map
-
-| File | Responsibility |
-|------|---------------|
-| `cli/go.mod` | Module definition |
-| `cli/main.go` | Entry point, flag parsing, top-level flow dispatch |
-| `cli/catalog.go` | Hardcoded product catalog: ID/slug → name, word-based search |
-| `cli/catalog_test.go` | Tests for product lookup and search |
-| `cli/microsoft.go` | Direct Microsoft client: session setup (cookie jar + fingerprinting), SKU/language lookup, link fetch, eval-center scrape |
-| `cli/microsoft_test.go` | Tests for Microsoft response parsing via mock HTTP server |
-| `cli/picker.go` | Interactive numbered menus reading from stdin |
-| `cli/picker_test.go` | Tests for picker input parsing |
-| `.github/workflows/cli-release.yml` | Release workflow: build 4 binaries, attach to GitHub Release on `cli/v*` tag, auto-submit winget update PR |
-| `winget/manifests/starkSV.msdl.yaml` | Winget version manifest |
-| `winget/manifests/starkSV.msdl.installer.yaml` | Winget installer manifest (portable, x64) |
-| `winget/manifests/starkSV.msdl.locale.en-US.yaml` | Winget default locale manifest |
-
----
-
-## Usage (target UX)
-
-```
-msdl # full interactive: pick product → pick language → print URL
-msdl "windows 11 25h2" # filtered product list → pick language → print URL
-msdl --id 3262 # skip product picker, pick language
-msdl --id 3262 --lang "English (United States)" # no prompts, prints URL directly
-msdl --eval # pick eval product → print all download URLs
-msdl --eval server-2025 # print all eval download URLs directly
-msdl --list # list all products and exit
-```
-
-> Note: there is no `--api` flag — the CLI talks to Microsoft directly. The `--contribute` / `--no-contribute` flags and the backend contribution endpoint are added in a later phase (see [`IMPLEMENTATION_PLAN.md`](../../../IMPLEMENTATION_PLAN.md) Phase 3).
-
----
-
-### Task 1: Scaffold — go.mod and stub main.go
-
-**Files:**
-- Create: `cli/go.mod`
-- Create: `cli/main.go`
-
-- [ ] **Step 1: Create the module**
-
-```bash
-cd "C:\Users\shekh\OneDrive\Documents\GitHub\windows-iso-downloader"
-mkdir cli
-cd cli
-go mod init github.com/starkSV/msdl-cli
-```
-
-Expected output: `go: creating new go.mod: module github.com/starkSV/msdl-cli`
-
-- [ ] **Step 2: Create stub main.go**
-
-`cli/main.go`:
-```go
-package main
-
-import "fmt"
-
-func main() {
- fmt.Println("msdl — Windows ISO downloader")
-}
-```
-
-- [ ] **Step 3: Verify it builds**
-
-```bash
-cd cli && go build -o msdl . && ./msdl
-```
-
-Expected output: `msdl — Windows ISO downloader`
-
-- [ ] **Step 4: Commit**
-
-```bash
-git add cli/
-git commit -m "feat(cli): scaffold Go module"
-```
-
----
-
-### Task 2: Product catalog
-
-**Files:**
-- Create: `cli/catalog.go`
-- Create: `cli/catalog_test.go`
-
-The catalog is hardcoded from `frontend/public/data/products.json` (18 consumer products) and `frontend/src/data/evalProducts.ts` (5 eval slugs). Hardcoded instead of embedded because the catalog rarely changes and cross-module `//go:embed` paths are awkward.
-
-- [ ] **Step 1: Write failing tests**
-
-`cli/catalog_test.go`:
-```go
-package main
-
-import "testing"
-
-func TestFindProductByID_found(t *testing.T) {
- p, ok := findProductByID("3262")
- if !ok {
- t.Fatal("expected product 3262 to exist")
- }
- if p.ID != "3262" {
- t.Errorf("got ID %s, want 3262", p.ID)
- }
-}
-
-func TestFindProductByID_notFound(t *testing.T) {
- _, ok := findProductByID("9999")
- if ok {
- t.Fatal("expected product 9999 to not exist")
- }
-}
-
-func TestSearchProducts_match(t *testing.T) {
- results := searchProducts("windows 11 25h2")
- if len(results) == 0 {
- t.Fatal("expected at least one result for 'windows 11 25h2'")
- }
-}
-
-func TestSearchProducts_noMatch(t *testing.T) {
- results := searchProducts("zzznomatch")
- if len(results) != 0 {
- t.Errorf("expected no results, got %d", len(results))
- }
-}
-
-func TestSearchProducts_caseInsensitive(t *testing.T) {
- results := searchProducts("WINDOWS 11")
- if len(results) == 0 {
- t.Fatal("expected results for uppercase query")
- }
-}
-
-func TestFindEvalProduct_found(t *testing.T) {
- p, ok := findEvalProduct("server-2025")
- if !ok {
- t.Fatal("expected server-2025 to exist")
- }
- if p.Slug != "server-2025" {
- t.Errorf("got slug %s, want server-2025", p.Slug)
- }
-}
-
-func TestFindEvalProduct_notFound(t *testing.T) {
- _, ok := findEvalProduct("zzz-nope")
- if ok {
- t.Fatal("expected zzz-nope to not exist")
- }
-}
-```
-
-- [ ] **Step 2: Run tests to confirm they fail**
-
-```bash
-cd cli && go test ./... -v
-```
-
-Expected: FAIL with `undefined: findProductByID` (or similar)
-
-- [ ] **Step 3: Implement catalog.go**
-
-`cli/catalog.go`:
-```go
-package main
-
-import "strings"
-
-type Product struct {
- ID string
- Name string
-}
-
-type EvalProduct struct {
- Slug string
- Name string
- EvalURL string // Microsoft Eval Center page scraped directly by the CLI
-}
-
-var consumerProducts = []Product{
- {"48", "Windows 8.1 Single Language (9600.17415)"},
- {"52", "Windows 8.1 (9600.17415)"},
- {"2378", "Windows 10 22H2 Home China (19045.2006)"},
- {"2618", "Windows 10 22H2 v1 (19045.2965)"},
- {"3113", "Windows 11 24H2 (26100.1742)"},
- {"3114", "Windows 11 24H2 Home China (26100.1742)"},
- {"3115", "Windows 11 24H2 Pro China (26100.1742)"},
- {"3131", "Windows 11 Arm64 24H2 (26100.1742)"},
- {"3132", "Windows 11 Arm64 24H2 Home China (26100.1742)"},
- {"3133", "Windows 11 Arm64 24H2 Pro China (26100.1742)"},
- {"3262", "Windows 11 25H2 (26200.6584)"},
- {"3263", "Windows 11 25H2 Home China (26200.6584)"},
- {"3264", "Windows 11 25H2 Pro China (26200.6584)"},
- {"3265", "Windows 11 Arm64 25H2 (26200.6584)"},
- {"3266", "Windows 11 Arm64 25H2 Home China (26200.6584)"},
- {"3267", "Windows 11 Arm64 25H2 Pro China (26200.6584)"},
- {"3321", "Windows 11 25H2 (Updated Oct)"},
- {"3324", "Windows 11 Arm64 25H2 (Updated Oct)"},
-}
-
-var evalProducts = []EvalProduct{
- {"server-2025", "Windows Server 2025", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2025"},
- {"server-2022", "Windows Server 2022", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2022"},
- {"server-2019", "Windows Server 2019", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2019"},
- {"server-2016", "Windows Server 2016", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2016"},
- {"win11-ent", "Windows 11 Enterprise", "https://www.microsoft.com/en-us/evalcenter/download-windows-11-enterprise"},
-}
-
-func findProductByID(id string) (Product, bool) {
- for _, p := range consumerProducts {
- if p.ID == id {
- return p, true
- }
- }
- return Product{}, false
-}
-
-// searchProducts returns products whose name contains all words in query (case-insensitive).
-func searchProducts(query string) []Product {
- words := strings.Fields(strings.ToLower(query))
- if len(words) == 0 {
- return consumerProducts
- }
- var results []Product
- for _, p := range consumerProducts {
- name := strings.ToLower(p.Name)
- match := true
- for _, w := range words {
- if !strings.Contains(name, w) {
- match = false
- break
- }
- }
- if match {
- results = append(results, p)
- }
- }
- return results
-}
-
-func findEvalProduct(slug string) (EvalProduct, bool) {
- for _, p := range evalProducts {
- if p.Slug == slug {
- return p, true
- }
- }
- return EvalProduct{}, false
-}
-```
-
-- [ ] **Step 4: Run tests to confirm they pass**
-
-```bash
-cd cli && go test ./... -v
-```
-
-Expected: all PASS
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add cli/catalog.go cli/catalog_test.go
-git commit -m "feat(cli): product catalog with word-based search"
-```
-
----
-
-### Task 3: Microsoft client (direct, from the user's machine)
-
-**Files:**
-- Create: `cli/microsoft.go`
-- Create: `cli/microsoft_test.go`
-
-This ports the Microsoft flow from `backend/main.go` so the CLI calls Microsoft directly — no MSDL backend. Source references:
-- `setupSession()` — `backend/main.go:394` (vlscppe → ov-df fingerprinting + cookie accumulation)
-- `simpleCookieJar` — `backend/main.go:45` (domain-less cookie jar)
-- `fetchSkuInfoFromMS()` — `backend/main.go:470`
-- `fetchDownloadLinksFromMS()` — `backend/main.go:566` (note the SKU-info "warmup" call before the link fetch)
-- `fetchEvalLinks()` / `detectArch` / `detectLang` / `fwlinkRe` / `isoLangRe` — `backend/main.go:278`, `256`, `270`, `191`, `192`
-
-**Design for testability:** the network functions (`fetchLanguages`, `fetchDownloadLinks`, `fetchEvalLinks`) hit hardcoded `microsoft.com` URLs, so they're covered by Task 5 smoke tests against the real API. The **parsing** is factored into pure functions (`parseSkuInfo`, `parseDownloadLinks`, `extractFwlinks`) that take bytes/strings and are unit-tested here.
-
-- [ ] **Step 1: Write failing tests for the pure parsers**
-
-`cli/microsoft_test.go`:
-```go
-package main
-
-import "testing"
-
-func TestParseSkuInfo_success(t *testing.T) {
- raw := []byte(`{"Skus":[{"Id":"19675","Language":"English (United States)"},{"Id":"19676","Language":"French"}]}`)
- langs, err := parseSkuInfo(raw)
- if err != nil {
- t.Fatalf("unexpected error: %v", err)
- }
- if len(langs) != 2 {
- t.Fatalf("want 2 languages, got %d", len(langs))
- }
- if langs[0].ID != "19675" || langs[0].Language != "English (United States)" {
- t.Errorf("unexpected first lang: %+v", langs[0])
- }
-}
-
-func TestParseSkuInfo_rateLimitError(t *testing.T) {
- raw := []byte(`{"Errors":[{"Type":9,"Value":"715-123130 blocked"}]}`)
- _, err := parseSkuInfo(raw)
- if err == nil {
- t.Fatal("expected error for Errors array with Type 9")
- }
-}
-
-func TestParseSkuInfo_empty(t *testing.T) {
- _, err := parseSkuInfo([]byte(`{"Skus":[]}`))
- if err == nil {
- t.Fatal("expected error for empty Skus")
- }
-}
-
-func TestParseDownloadLinks_architectureString(t *testing.T) {
- raw := []byte(`{"ProductDownloadOptions":[{"Uri":"https://example.com/win11.iso?t=abc","Architecture":"x64"}]}`)
- links, err := parseDownloadLinks(raw)
- if err != nil {
- t.Fatalf("unexpected error: %v", err)
- }
- if len(links) != 1 {
- t.Fatalf("want 1 link, got %d", len(links))
- }
- if links[0].URI != "https://example.com/win11.iso?t=abc" || links[0].Architecture != "x64" {
- t.Errorf("unexpected link: %+v", links[0])
- }
-}
-
-func TestParseDownloadLinks_downloadTypeFallback(t *testing.T) {
- // No Architecture field; DownloadType 1 maps to x64.
- raw := []byte(`{"ProductDownloadOptions":[{"Uri":"https://example.com/a.iso","DownloadType":1}]}`)
- links, err := parseDownloadLinks(raw)
- if err != nil {
- t.Fatalf("unexpected error: %v", err)
- }
- if links[0].Architecture != "x64" {
- t.Errorf("want x64 from DownloadType 1, got %q", links[0].Architecture)
- }
-}
-
-func TestParseDownloadLinks_error(t *testing.T) {
- raw := []byte(`{"Errors":[{"Type":4,"Value":"no download links found for this SKU"}]}`)
- _, err := parseDownloadLinks(raw)
- if err == nil {
- t.Fatal("expected error for Errors array")
- }
-}
-
-func TestParseDownloadLinks_empty(t *testing.T) {
- _, err := parseDownloadLinks([]byte(`{"ProductDownloadOptions":[]}`))
- if err == nil {
- t.Fatal("expected error for empty options")
- }
-}
-
-func TestExtractFwlinks_dedupAndUnescape(t *testing.T) {
- html := `
- x64
- dup
- arm
- `
- links := extractFwlinks(html)
- if len(links) != 2 {
- t.Fatalf("want 2 unique fwlinks, got %d: %v", len(links), links)
- }
- if links[0] != "https://go.microsoft.com/fwlink/?linkid=111&clcid=0x409" {
- t.Errorf("expected unescaped &, got %q", links[0])
- }
-}
-
-func TestDetectArchLang(t *testing.T) {
- url := "https://software-static.download.prss.microsoft.com/.../server2025_arm64_en-us.iso"
- if got := detectArch(url); got != "ARM64" {
- t.Errorf("arch: want ARM64, got %s", got)
- }
- if got := detectLang(url); got != "en-us" {
- t.Errorf("lang: want en-us, got %s", got)
- }
-}
-```
-
-- [ ] **Step 2: Run tests to confirm they fail**
-
-```bash
-cd cli && go test ./... -run "TestParseSku|TestParseDownload|TestExtractFwlinks|TestDetectArchLang" -v
-```
-
-Expected: FAIL with `undefined: parseSkuInfo` (or similar)
-
-- [ ] **Step 3: Implement microsoft.go**
-
-`cli/microsoft.go`:
-```go
-package main
-
-import (
- "crypto/rand"
- "encoding/hex"
- "encoding/json"
- "fmt"
- "html"
- "io"
- "net/http"
- "net/url"
- "regexp"
- "strconv"
- "strings"
- "sync"
- "time"
-)
-
-// Constants ported from backend/main.go. These mimic the official download page.
-const (
- msUA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
- msProfile = "606624d44113"
- msLocale = "en-US"
- msOrgID = "y6jn8c31"
- msCustomer = "560dc9f3-1aa5-4a2f-b63c-9e18f8d0e175"
-)
-
-var (
- fwlinkRe = regexp.MustCompile(`https://go\.microsoft\.com/fwlink/[^"'\s<>]+`)
- isoLangRe = regexp.MustCompile(`_([a-z]{2}-[a-z]{2})\.iso$`)
- reW = regexp.MustCompile(`[&?]w=([^&"'\s]+)`)
- reRt = regexp.MustCompile(`rticks[="]+\+?\s*(\d{10,})`)
-)
-
-type Language struct {
- ID string `json:"Id"`
- Language string `json:"Language"`
-}
-
-type DownloadLink struct {
- URI string
- Architecture string
-}
-
-type EvalLink struct {
- Arch string
- Lang string
- URL string
-}
-
-// simpleCookieJar stores cookies without domain scoping so cookies from
-// Microsoft's fingerprinting hosts (vlscppe, ov-df) replay on the
-// download-connector call. Ported from backend/main.go:45.
-type simpleCookieJar struct {
- mu sync.Mutex
- cookies []*http.Cookie
-}
-
-func (j *simpleCookieJar) SetCookies(_ *url.URL, cookies []*http.Cookie) {
- j.mu.Lock()
- defer j.mu.Unlock()
- for _, c := range cookies {
- found := false
- for i, existing := range j.cookies {
- if existing.Name == c.Name {
- j.cookies[i] = c
- found = true
- break
- }
- }
- if !found {
- j.cookies = append(j.cookies, c)
- }
- }
-}
-
-func (j *simpleCookieJar) Cookies(_ *url.URL) []*http.Cookie {
- j.mu.Lock()
- defer j.mu.Unlock()
- out := make([]*http.Cookie, len(j.cookies))
- copy(out, j.cookies)
- return out
-}
-
-// newSessionID returns a v4 UUID without an external dependency.
-func newSessionID() string {
- b := make([]byte, 16)
- rand.Read(b)
- b[6] = (b[6] & 0x0f) | 0x40
- b[8] = (b[8] & 0x3f) | 0x80
- return fmt.Sprintf("%s-%s-%s-%s-%s",
- hex.EncodeToString(b[0:4]), hex.EncodeToString(b[4:6]),
- hex.EncodeToString(b[6:8]), hex.EncodeToString(b[8:10]),
- hex.EncodeToString(b[10:16]))
-}
-
-func referer(productID string) string {
- id, err := strconv.Atoi(productID)
- if err != nil {
- return "https://www.microsoft.com/en-us/software-download/windows8ISO"
- }
- if id >= 2935 {
- return "https://www.microsoft.com/en-us/software-download/windows11"
- }
- if id >= 2618 {
- return "https://www.microsoft.com/en-us/software-download/windows10ISO"
- }
- return "https://www.microsoft.com/en-us/software-download/windows8ISO"
-}
-
-func mapDownloadType(n int) string {
- switch n {
- case 0:
- return "x86"
- case 1:
- return "x64"
- case 2:
- return "ARM64"
- default:
- return fmt.Sprintf("type_%d", n)
- }
-}
-
-// newSession runs the vlscppe → ov-df fingerprinting chain and returns a client
-// whose cookie jar carries the accumulated session cookies, plus the session ID.
-func newSession() (*http.Client, string) {
- sessionID := newSessionID()
- jar := &simpleCookieJar{}
- client := &http.Client{Timeout: 15 * time.Second, Jar: jar}
-
- q1 := url.Values{}
- q1.Set("org_id", msOrgID)
- q1.Set("session_id", sessionID)
- req1, _ := http.NewRequest("GET", "https://vlscppe.microsoft.com/tags?"+q1.Encode(), nil)
- req1.Header.Set("User-Agent", msUA)
- client.Do(req1)
-
- q2 := url.Values{}
- q2.Set("instanceId", msCustomer)
- q2.Set("PageId", "si")
- q2.Set("session_id", sessionID)
- req2, _ := http.NewRequest("GET", "https://ov-df.microsoft.com/mdt.js?"+q2.Encode(), nil)
- req2.Header.Set("User-Agent", msUA)
- resp2, err := client.Do(req2)
- if err != nil {
- return client, sessionID
- }
- body, _ := io.ReadAll(resp2.Body)
- resp2.Body.Close()
-
- wMatch := reW.FindStringSubmatch(string(body))
- rtMatch := reRt.FindStringSubmatch(string(body))
- if len(wMatch) > 1 && len(rtMatch) > 1 {
- q3 := url.Values{}
- q3.Set("session_id", sessionID)
- q3.Set("CustomerId", msCustomer)
- q3.Set("PageId", "si")
- q3.Set("w", wMatch[1])
- q3.Set("mdt", fmt.Sprintf("%d", time.Now().UnixMilli()))
- q3.Set("rticks", rtMatch[1])
- req3, _ := http.NewRequest("GET", "https://ov-df.microsoft.com/?"+q3.Encode(), nil)
- req3.Header.Set("User-Agent", msUA)
- client.Do(req3)
- }
- return client, sessionID
-}
-
-func msGet(client *http.Client, reqURL, productID string) ([]byte, error) {
- req, _ := http.NewRequest("GET", reqURL, nil)
- req.Header.Set("User-Agent", msUA)
- req.Header.Set("Referer", referer(productID))
- req.Header.Set("Accept", "application/json")
- resp, err := client.Do(req)
- if err != nil {
- return nil, fmt.Errorf("request failed: %w", err)
- }
- defer resp.Body.Close()
- body, _ := io.ReadAll(resp.Body)
- if resp.StatusCode != http.StatusOK {
- return nil, fmt.Errorf("Microsoft returned HTTP %d", resp.StatusCode)
- }
- // Some responses are a JSON-encoded string; unwrap one level.
- if len(body) > 0 && body[0] == '"' {
- var unquoted string
- json.Unmarshal(body, &unquoted)
- body = []byte(unquoted)
- }
- return body, nil
-}
-
-type msErrorEntry struct {
- Type float64 `json:"Type"`
- Value string `json:"Value"`
-}
-
-func firstError(errs []msErrorEntry) string {
- if len(errs) == 0 {
- return ""
- }
- if int(errs[0].Type) == 9 {
- if errs[0].Value != "" {
- return errs[0].Value
- }
- return "Your IP has been temporarily blocked by Microsoft (Code 715-123130)"
- }
- if errs[0].Value != "" {
- return errs[0].Value
- }
- return "Microsoft API error"
-}
-
-// parseSkuInfo extracts the language list from a getskuinformation response.
-func parseSkuInfo(raw []byte) ([]Language, error) {
- var data struct {
- Skus []Language `json:"Skus"`
- Errors []msErrorEntry `json:"Errors"`
- }
- if err := json.Unmarshal(raw, &data); err != nil {
- return nil, fmt.Errorf("invalid SKU response: %w", err)
- }
- if msg := firstError(data.Errors); msg != "" {
- return nil, fmt.Errorf("%s", msg)
- }
- if len(data.Skus) == 0 {
- return nil, fmt.Errorf("no languages found for this product")
- }
- return data.Skus, nil
-}
-
-// parseDownloadLinks extracts download options, filling Architecture from
-// DownloadType when the Architecture field is absent.
-func parseDownloadLinks(raw []byte) ([]DownloadLink, error) {
- var data struct {
- ProductDownloadOptions []struct {
- Uri string `json:"Uri"`
- Architecture interface{} `json:"Architecture"`
- DownloadType *float64 `json:"DownloadType"`
- } `json:"ProductDownloadOptions"`
- Errors []msErrorEntry `json:"Errors"`
- }
- if err := json.Unmarshal(raw, &data); err != nil {
- return nil, fmt.Errorf("invalid download response: %w", err)
- }
- if msg := firstError(data.Errors); msg != "" {
- return nil, fmt.Errorf("%s", msg)
- }
- if len(data.ProductDownloadOptions) == 0 {
- return nil, fmt.Errorf("no download links found for this SKU")
- }
- var links []DownloadLink
- for _, o := range data.ProductDownloadOptions {
- arch := ""
- if s, ok := o.Architecture.(string); ok && s != "" {
- arch = s
- } else if o.DownloadType != nil {
- arch = mapDownloadType(int(*o.DownloadType))
- }
- links = append(links, DownloadLink{URI: o.Uri, Architecture: arch})
- }
- return links, nil
-}
-
-func fetchLanguages(client *http.Client, sessionID, productID string) ([]Language, error) {
- q := url.Values{}
- q.Set("profile", msProfile)
- q.Set("productEditionId", productID)
- q.Set("SKU", "undefined")
- q.Set("friendlyFileName", "undefined")
- q.Set("Locale", msLocale)
- q.Set("sessionID", sessionID)
- raw, err := msGet(client,
- "https://www.microsoft.com/software-download-connector/api/getskuinformationbyproductedition?"+q.Encode(),
- productID)
- if err != nil {
- return nil, err
- }
- return parseSkuInfo(raw)
-}
-
-func fetchDownloadLinks(client *http.Client, sessionID, productID, skuID string) ([]DownloadLink, error) {
- // Warmup: a SKU-info call so setup cookies carry into the link fetch
- // (mirrors backend/main.go:581).
- wq := url.Values{}
- wq.Set("profile", msProfile)
- wq.Set("productEditionId", productID)
- wq.Set("SKU", "undefined")
- wq.Set("friendlyFileName", "undefined")
- wq.Set("Locale", msLocale)
- wq.Set("sessionID", sessionID)
- msGet(client,
- "https://www.microsoft.com/software-download-connector/api/getskuinformationbyproductedition?"+wq.Encode(),
- productID)
-
- q := url.Values{}
- q.Set("profile", msProfile)
- q.Set("productEditionId", "undefined")
- q.Set("SKU", skuID)
- q.Set("friendlyFileName", "undefined")
- q.Set("Locale", msLocale)
- q.Set("sessionID", sessionID)
- raw, err := msGet(client,
- "https://www.microsoft.com/software-download-connector/api/GetProductDownloadLinksBySku?"+q.Encode(),
- productID)
- if err != nil {
- return nil, err
- }
- return parseDownloadLinks(raw)
-}
-
-// --- Eval Center (direct scrape) ---
-
-func extractFwlinks(pageHTML string) []string {
- matches := fwlinkRe.FindAllString(pageHTML, -1)
- seen := map[string]bool{}
- var out []string
- for _, m := range matches {
- link := html.UnescapeString(m)
- if !seen[link] {
- seen[link] = true
- out = append(out, link)
- }
- }
- return out
-}
-
-func detectArch(rawURL string) string {
- l := strings.ToLower(rawURL)
- switch {
- case strings.Contains(l, "arm64"):
- return "ARM64"
- case strings.Contains(l, "x64"):
- return "x64"
- case strings.Contains(l, "x86"):
- return "x86"
- }
- return "ISO"
-}
-
-func detectLang(rawURL string) string {
- m := isoLangRe.FindStringSubmatch(strings.ToLower(rawURL))
- if len(m) > 1 {
- return m[1]
- }
- return ""
-}
-
-// fetchEvalLinks scrapes a Microsoft Eval Center page, follows each fwlink to
-// its final .iso URL, and returns the resolved links. The Eval Center is a
-// separate endpoint and is fetched directly (no session/cookie dance needed).
-func fetchEvalLinks(evalURL string) ([]EvalLink, error) {
- client := &http.Client{Timeout: 20 * time.Second}
- req, _ := http.NewRequest("GET", evalURL, nil)
- req.Header.Set("User-Agent", msUA)
- req.Header.Set("Accept", "text/html,application/xhtml+xml")
- resp, err := client.Do(req)
- if err != nil {
- return nil, fmt.Errorf("fetching eval page: %w", err)
- }
- body, _ := io.ReadAll(resp.Body)
- resp.Body.Close()
-
- fwlinks := extractFwlinks(string(body))
- var (
- links []EvalLink
- mu sync.Mutex
- wg sync.WaitGroup
- )
- for _, fw := range fwlinks {
- wg.Add(1)
- go func(fw string) {
- defer wg.Done()
- r, _ := http.NewRequest("GET", fw, nil)
- r.Header.Set("User-Agent", msUA)
- resp, err := client.Do(r)
- if err != nil {
- return
- }
- resp.Body.Close()
- final := resp.Request.URL.String()
- if !strings.Contains(strings.ToLower(final), ".iso") {
- return
- }
- mu.Lock()
- links = append(links, EvalLink{Arch: detectArch(final), Lang: detectLang(final), URL: final})
- mu.Unlock()
- }(fw)
- }
- wg.Wait()
- if len(links) == 0 {
- return nil, fmt.Errorf("no eval ISO links found")
- }
- return links, nil
-}
-```
-
-- [ ] **Step 4: Run tests to confirm they pass**
-
-```bash
-cd cli && go test ./... -run "TestParseSku|TestParseDownload|TestExtractFwlinks|TestDetectArchLang" -v
-```
-
-Expected: all PASS
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add cli/microsoft.go cli/microsoft_test.go
-git commit -m "feat(cli): direct Microsoft client (session, SKU, links, eval scrape)"
-```
-
----
-
-### Task 4: Interactive picker
-
-**Files:**
-- Create: `cli/picker.go`
-- Create: `cli/picker_test.go`
-
-All prompts print to stderr so stdout stays clean for piping. `parseChoice` is pure (takes an `io.Reader`) so it's directly testable without capturing stdin.
-
-- [ ] **Step 1: Write failing tests**
-
-`cli/picker_test.go`:
-```go
-package main
-
-import (
- "strings"
- "testing"
-)
-
-func TestParseChoice_valid(t *testing.T) {
- cases := []struct {
- input string
- max int
- want int
- }{
- {"1\n", 3, 1},
- {"3\n", 3, 3},
- {" 2 \n", 5, 2},
- }
- for _, c := range cases {
- got, err := parseChoice(strings.NewReader(c.input), c.max)
- if err != nil {
- t.Errorf("input %q: unexpected error: %v", c.input, err)
- continue
- }
- if got != c.want {
- t.Errorf("input %q: got %d, want %d", c.input, got, c.want)
- }
- }
-}
-
-func TestParseChoice_outOfRange(t *testing.T) {
- cases := []string{"0\n", "4\n"}
- for _, input := range cases {
- _, err := parseChoice(strings.NewReader(input), 3)
- if err == nil {
- t.Errorf("input %q: expected error for out-of-range, got nil", input)
- }
- }
-}
-
-func TestParseChoice_nonNumeric(t *testing.T) {
- _, err := parseChoice(strings.NewReader("abc\n"), 3)
- if err == nil {
- t.Error("expected error for non-numeric input, got nil")
- }
-}
-
-func TestParseChoice_emptyInput(t *testing.T) {
- _, err := parseChoice(strings.NewReader("\n"), 3)
- if err == nil {
- t.Error("expected error for empty input, got nil")
- }
-}
-```
-
-- [ ] **Step 2: Run tests to confirm they fail**
-
-```bash
-cd cli && go test ./... -run "TestParse" -v
-```
-
-Expected: FAIL with `undefined: parseChoice`
-
-- [ ] **Step 3: Implement picker.go**
-
-`cli/picker.go`:
-```go
-package main
-
-import (
- "bufio"
- "fmt"
- "io"
- "os"
- "strconv"
- "strings"
-)
-
-// parseChoice reads a 1-based integer from r, validated in [1, max].
-func parseChoice(r io.Reader, max int) (int, error) {
- scanner := bufio.NewScanner(r)
- if !scanner.Scan() {
- return 0, fmt.Errorf("no input")
- }
- text := strings.TrimSpace(scanner.Text())
- if text == "" {
- return 0, fmt.Errorf("enter a number between 1 and %d", max)
- }
- n, err := strconv.Atoi(text)
- if err != nil || n < 1 || n > max {
- return 0, fmt.Errorf("enter a number between 1 and %d", max)
- }
- return n, nil
-}
-
-func pickProduct(products []Product) (Product, error) {
- fmt.Fprintln(os.Stderr, "\nSelect a product:")
- for i, p := range products {
- fmt.Fprintf(os.Stderr, " %2d. %s\n", i+1, p.Name)
- }
- fmt.Fprint(os.Stderr, "\nChoice: ")
- n, err := parseChoice(os.Stdin, len(products))
- if err != nil {
- return Product{}, err
- }
- return products[n-1], nil
-}
-
-func pickLanguage(langs []Language) (Language, error) {
- fmt.Fprintln(os.Stderr, "\nSelect a language:")
- for i, l := range langs {
- fmt.Fprintf(os.Stderr, " %2d. %s\n", i+1, l.Language)
- }
- fmt.Fprint(os.Stderr, "\nChoice: ")
- n, err := parseChoice(os.Stdin, len(langs))
- if err != nil {
- return Language{}, err
- }
- return langs[n-1], nil
-}
-
-func pickEvalProduct(products []EvalProduct) (EvalProduct, error) {
- fmt.Fprintln(os.Stderr, "\nSelect an evaluation product:")
- for i, p := range products {
- fmt.Fprintf(os.Stderr, " %2d. %s\n", i+1, p.Name)
- }
- fmt.Fprint(os.Stderr, "\nChoice: ")
- n, err := parseChoice(os.Stdin, len(products))
- if err != nil {
- return EvalProduct{}, err
- }
- return products[n-1], nil
-}
-```
-
-- [ ] **Step 4: Run tests to confirm they pass**
-
-```bash
-cd cli && go test ./... -run "TestParse" -v
-```
-
-Expected: all PASS
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add cli/picker.go cli/picker_test.go
-git commit -m "feat(cli): interactive numbered picker (stdin/stderr separated)"
-```
-
----
-
-### Task 5: Main flow
-
-**Files:**
-- Modify: `cli/main.go` (replace stub)
-
-This is the glue layer: parse flags, dispatch to `runConsumer` or `runEval`, threading named-arg overrides through each stage.
-
-- [ ] **Step 1: Replace stub main.go with the full implementation**
-
-`cli/main.go`:
-```go
-package main
-
-import (
- "flag"
- "fmt"
- "os"
- "strings"
-)
-
-func main() {
- if err := run(os.Args[1:]); err != nil {
- fmt.Fprintln(os.Stderr, "error:", err)
- os.Exit(1)
- }
-}
-
-func run(args []string) error {
- fs := flag.NewFlagSet("msdl", flag.ContinueOnError)
- fs.SetOutput(os.Stderr)
- fs.Usage = func() {
- fmt.Fprintln(os.Stderr, `msdl — Windows ISO downloader
-
-Usage:
- msdl [search terms] interactive: filter + pick product, pick language
- msdl --id 3262 skip product picker
- msdl --id 3262 --lang "English (United States)" no prompts, print URL directly
- msdl --eval [slug] evaluation ISOs (server-2025, win11-ent, ...)
- msdl --list list all products and exit
-
-Flags:`)
- fs.PrintDefaults()
- }
-
- productID := fs.String("id", "", "consumer product ID (skips product picker)")
- langFlag := fs.String("lang", "", `language name, e.g. "English (United States)"`)
- evalMode := fs.Bool("eval", false, "fetch evaluation ISOs")
- listMode := fs.Bool("list", false, "list all products and exit")
-
- if err := fs.Parse(args); err != nil {
- return err
- }
- query := strings.Join(fs.Args(), " ")
-
- if *listMode {
- fmt.Fprintln(os.Stderr, "Consumer products:")
- for _, p := range consumerProducts {
- fmt.Fprintf(os.Stderr, " %-6s %s\n", p.ID, p.Name)
- }
- fmt.Fprintln(os.Stderr, "\nEvaluation products:")
- for _, p := range evalProducts {
- fmt.Fprintf(os.Stderr, " %-20s %s\n", p.Slug, p.Name)
- }
- return nil
- }
-
- if *evalMode {
- return runEval(query)
- }
- return runConsumer(*productID, query, *langFlag)
-}
-
-func runConsumer(productID, query, langName string) error {
- var product Product
-
- if productID != "" {
- p, ok := findProductByID(productID)
- if !ok {
- return fmt.Errorf("unknown product ID %q — run msdl --list to see all products", productID)
- }
- product = p
- } else {
- candidates := consumerProducts
- if query != "" {
- candidates = searchProducts(query)
- if len(candidates) == 0 {
- return fmt.Errorf("no products match %q — run msdl --list to see all products", query)
- }
- }
- if len(candidates) == 1 {
- product = candidates[0]
- fmt.Fprintf(os.Stderr, "Selected: %s\n", product.Name)
- } else {
- var err error
- product, err = pickProduct(candidates)
- if err != nil {
- return err
- }
- }
- }
-
- fmt.Fprintf(os.Stderr, "Setting up Microsoft session...\n")
- client, sessionID := newSession()
-
- langs, err := fetchLanguages(client, sessionID, product.ID)
- if err != nil {
- return fmt.Errorf("fetching languages for %s: %w", product.Name, err)
- }
-
- var lang Language
- if langName != "" {
- for _, l := range langs {
- if strings.EqualFold(l.Language, langName) {
- lang = l
- break
- }
- }
- if lang.ID == "" {
- return fmt.Errorf("language %q not available for %s", langName, product.Name)
- }
- } else {
- lang, err = pickLanguage(langs)
- if err != nil {
- return err
- }
- }
-
- fmt.Fprintf(os.Stderr, "Fetching download link...\n")
- links, err := fetchDownloadLinks(client, sessionID, product.ID, lang.ID)
- if err != nil {
- return fmt.Errorf("fetching download links: %w", err)
- }
-
- for _, link := range links {
- fmt.Println(link.URI)
- }
- return nil
-}
-
-func runEval(slug string) error {
- var ep EvalProduct
-
- if slug != "" {
- p, ok := findEvalProduct(slug)
- if !ok {
- return fmt.Errorf("unknown eval product %q — valid slugs: server-2025, server-2022, server-2019, server-2016, win11-ent", slug)
- }
- ep = p
- } else {
- var err error
- ep, err = pickEvalProduct(evalProducts)
- if err != nil {
- return err
- }
- }
-
- fmt.Fprintf(os.Stderr, "Fetching eval links for %s...\n", ep.Name)
- links, err := fetchEvalLinks(ep.EvalURL)
- if err != nil {
- return fmt.Errorf("fetching eval links for %s: %w", ep.Name, err)
- }
- if len(links) == 0 {
- return fmt.Errorf("no eval links returned for %s", ep.Slug)
- }
-
- for _, link := range links {
- if link.Arch != "" {
- fmt.Printf("%s\t%s\n", link.Arch, link.URL)
- } else {
- fmt.Println(link.URL)
- }
- }
- return nil
-}
-```
-
-- [ ] **Step 2: Build and run smoke tests**
-
-```bash
-cd cli && go build -o msdl .
-
-# List mode
-./msdl --list
-
-# Interactive (will prompt — confirm prompts appear on stderr, URLs on stdout)
-echo "1" | ./msdl --id 3262 2>/dev/null # non-interactive language pick
-
-# Full pipeline test
-./msdl --id 3262 --lang "English (United States)" 2>/dev/null
-```
-
-Expected for the last command: a single Microsoft CDN URL printed to stdout (no prompts), something like:
-```
-https://software.download.prss.microsoft.com/dbazure/Win11_25H2_...iso?t=...
-```
-
-> **Run these from a residential/commercial connection, not a data center.** The whole point of the CLI is that the call originates from the user's IP — running the smoke test from a VPS/CI runner may itself be Sentinel-blocked and is not a valid test of the code. A successful fetch of product 3262/3321 (currently backend-blocked) from a home connection is the proof the architecture works.
-
-If Microsoft returns an error, check that the error prints to stderr and exit code is 1:
-```bash
-./msdl --id 9999 --lang "English (United States)"; echo "exit: $?"
-```
-Expected: `error: unknown product ID "9999" — run msdl --list to see all products` + `exit: 1`
-
-- [ ] **Step 3: Run the full test suite**
-
-```bash
-cd cli && go test ./... -v
-```
-
-Expected: all PASS (Tasks 2–4 tests still pass, no regressions)
-
-- [ ] **Step 4: Commit**
-
-```bash
-git add cli/main.go
-git commit -m "feat(cli): main flow with interactive and named-arg modes"
-```
-
----
-
-### Task 6: GitHub Actions release workflow
-
-**Files:**
-- Create: `.github/workflows/cli-release.yml`
-
-Triggered by `cli/v*` tags. Builds 4 binaries (linux-amd64, darwin-amd64, darwin-arm64, windows-amd64), attaches them to a GitHub Release, then auto-submits a winget update PR using `wingetcreate`.
-
-**Prerequisite:** Add a `WINGET_TOKEN` repository secret — a GitHub PAT (classic) with `public_repo` scope. `wingetcreate` uses this to open a PR on `microsoft/winget-pkgs` on your behalf. Go to GitHub → Settings → Developer Settings → Personal access tokens → Generate new token (classic), check `public_repo`, copy the token, then add it at `https://github.com/starkSV/windows-iso-downloader/settings/secrets/actions` as `WINGET_TOKEN`.
-
-- [ ] **Step 1: Create the workflow file**
-
-`.github/workflows/cli-release.yml`:
-```yaml
-name: CLI Release
-
-on:
- push:
- tags:
- - 'cli/v*'
-
-jobs:
- release:
- runs-on: ubuntu-latest
- permissions:
- contents: write
- steps:
- - uses: actions/checkout@v4
-
- - uses: actions/setup-go@v5
- with:
- go-version: '1.21'
-
- - name: Build binaries
- working-directory: cli
- run: |
- mkdir -p ../dist
- GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o ../dist/msdl-linux-amd64 .
- GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w" -o ../dist/msdl-darwin-amd64 .
- GOOS=darwin GOARCH=arm64 go build -ldflags="-s -w" -o ../dist/msdl-darwin-arm64 .
- GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o ../dist/msdl-windows-amd64.exe .
-
- - name: Create GitHub Release
- uses: softprops/action-gh-release@v2
- with:
- tag_name: ${{ github.ref_name }}
- name: "msdl CLI ${{ github.ref_name }}"
- files: dist/*
- body: |
- ## Install
-
- **Windows (winget):**
- ```
- winget install starkSV.msdl
- ```
-
- **Linux/macOS:**
- ```bash
- # Linux x86_64
- curl -L https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-linux-amd64 -o msdl
- chmod +x msdl && sudo mv msdl /usr/local/bin/
-
- # macOS Apple Silicon
- curl -L https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-darwin-arm64 -o msdl
- chmod +x msdl && sudo mv msdl /usr/local/bin/
-
- # macOS Intel
- curl -L https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-darwin-amd64 -o msdl
- chmod +x msdl && sudo mv msdl /usr/local/bin/
- ```
-
- **Windows (manual):** Download `msdl-windows-amd64.exe`, rename to `msdl.exe`, place in a directory on your PATH.
-
- ## Usage
- ```
- msdl # fully interactive
- msdl "windows 11 25h2" # filter products, pick language
- msdl --id 3262 --lang "English (United States)" # direct, no prompts
- msdl --eval server-2025 # evaluation ISOs
- msdl --list # list all products
- ```
-
- - name: Submit winget update
- if: success() && secrets.WINGET_TOKEN != ''
- env:
- WINGET_TOKEN: ${{ secrets.WINGET_TOKEN }}
- run: |
- dotnet tool install --global wingetcreate
- VERSION="${{ github.ref_name }}"
- VERSION="${VERSION#cli/v}"
- wingetcreate update starkSV.msdl \
- --version "$VERSION" \
- --urls "https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-windows-amd64.exe" \
- --submit \
- --token "$WINGET_TOKEN"
-```
-
-> **Note:** The `wingetcreate update` step downloads the `.exe`, computes its SHA256, updates the manifest, and opens a PR on `microsoft/winget-pkgs` automatically. It is skipped if `WINGET_TOKEN` is not set. It will only succeed after the package is accepted into winget-pkgs (Task 7, Step 6). For `cli/v0.1.0` the step will be skipped or fail gracefully — that's expected.
-
-- [ ] **Step 2: Commit**
-
-```bash
-git add .github/workflows/cli-release.yml
-git commit -m "ci: GitHub Actions release workflow for msdl CLI binaries"
-```
-
-- [ ] **Step 3: Tag and push to trigger release**
-
-```bash
-git tag cli/v0.1.0
-git push origin cli/v0.1.0
-```
-
-Then check GitHub → Actions → CLI Release workflow. Confirm it completes and 4 binary files are attached to the release at `https://github.com/starkSV/windows-iso-downloader/releases/tag/cli%2Fv0.1.0`. The winget step will be skipped on this first release — that's expected (Task 7 handles the initial submission).
-
----
-
-### Task 7: Winget manifests and initial submission
-
-**Files:**
-- Create: `winget/manifests/starkSV.msdl.yaml`
-- Create: `winget/manifests/starkSV.msdl.installer.yaml`
-- Create: `winget/manifests/starkSV.msdl.locale.en-US.yaml`
-
-These 3 files are the winget package manifest for the initial submission to `microsoft/winget-pkgs`. Once accepted, all future version bumps are handled automatically by the `wingetcreate update` step in the CI workflow (Task 6). The files in this repo serve as the source of truth / reference copy.
-
-**Important:** Complete Task 6 (release `cli/v0.1.0`) **before** this task. You need the real SHA256 of the published `.exe` to fill into the installer manifest.
-
-- [ ] **Step 1: Get the SHA256 of the published Windows binary**
-
-After `cli/v0.1.0` is live on GitHub Releases, download the binary and compute its hash:
-
-```bash
-# Linux/macOS
-curl -sL "https://github.com/starkSV/windows-iso-downloader/releases/download/cli%2Fv0.1.0/msdl-windows-amd64.exe" -o msdl-windows-amd64.exe
-sha256sum msdl-windows-amd64.exe
-```
-
-On Windows PowerShell:
-```powershell
-(Get-FileHash "msdl-windows-amd64.exe" -Algorithm SHA256).Hash
-```
-
-Copy the output hash — you'll paste it into the installer manifest below. winget requires it in **UPPERCASE**.
-
-- [ ] **Step 2: Create the version manifest**
-
-`winget/manifests/starkSV.msdl.yaml`:
-```yaml
-PackageIdentifier: starkSV.msdl
-PackageVersion: 0.1.0
-DefaultLocale: en-US
-ManifestType: version
-ManifestVersion: 1.6.0
-```
-
-- [ ] **Step 3: Create the installer manifest**
-
-`winget/manifests/starkSV.msdl.installer.yaml`:
-```yaml
-PackageIdentifier: starkSV.msdl
-PackageVersion: 0.1.0
-Platform:
- - Windows.Desktop
-MinimumOSVersion: 10.0.0.0
-InstallerType: portable
-Commands:
- - msdl
-Installers:
- - Architecture: x64
- InstallerUrl: https://github.com/starkSV/windows-iso-downloader/releases/download/cli%2Fv0.1.0/msdl-windows-amd64.exe
- InstallerSha256: PASTE_SHA256_FROM_STEP_1_HERE
-ManifestType: installer
-ManifestVersion: 1.6.0
-```
-
-Replace `PASTE_SHA256_FROM_STEP_1_HERE` with the actual SHA256 hash from Step 1 (uppercase, no spaces).
-
-- [ ] **Step 4: Create the locale manifest**
-
-`winget/manifests/starkSV.msdl.locale.en-US.yaml`:
-```yaml
-PackageIdentifier: starkSV.msdl
-PackageVersion: 0.1.0
-PackageLocale: en-US
-Publisher: starkSV
-PublisherUrl: https://github.com/starkSV
-PublisherSupportUrl: https://github.com/starkSV/windows-iso-downloader/issues
-PackageName: msdl
-PackageUrl: https://msdl.tech-latest.com
-License: MIT
-LicenseUrl: https://github.com/starkSV/windows-iso-downloader/blob/main/LICENSE
-ShortDescription: Download Windows ISOs directly from Microsoft
-Description: |-
- msdl is a command-line tool for downloading Windows ISO files directly from Microsoft's servers.
- Supports interactive product and language selection, or fully automated with named arguments.
- Covers consumer Windows (10, 11, 8.1) and evaluation builds (Server 2025/2022/2019/2016, Windows 11 Enterprise).
-Tags:
- - windows
- - iso
- - download
- - microsoft
-ManifestType: defaultLocale
-ManifestVersion: 1.6.0
-```
-
-- [ ] **Step 5: Validate manifests locally**
-
-Install winget-create or use `winget validate` if you have winget on your machine:
-
-```powershell
-# Validate locally (Windows, winget installed)
-winget install --manifest winget\manifests\ --accept-source-agreements
-```
-
-This should install `msdl` and make `msdl --list` work from any terminal. Uninstall after:
-
-```powershell
-winget uninstall starkSV.msdl
-```
-
-- [ ] **Step 6: Commit the manifests**
-
-```bash
-git add winget/manifests/
-git commit -m "chore(winget): initial manifest for starkSV.msdl v0.1.0"
-git push origin main
-```
-
-- [ ] **Step 7: Submit the initial PR to winget-pkgs**
-
-Fork `microsoft/winget-pkgs` on GitHub (one-time). Then copy the 3 manifest files into the correct location in the fork and submit:
-
-```bash
-# Clone your fork
-git clone https://github.com/starkSV/winget-pkgs.git
-cd winget-pkgs
-
-# Create manifest directory matching the winget-pkgs layout
-mkdir -p manifests/s/starkSV/msdl/0.1.0
-
-# Copy manifests from the msdl repo
-cp /path/to/windows-iso-downloader/winget/manifests/starkSV.msdl.yaml manifests/s/starkSV/msdl/0.1.0/
-cp /path/to/windows-iso-downloader/winget/manifests/starkSV.msdl.installer.yaml manifests/s/starkSV/msdl/0.1.0/
-cp /path/to/windows-iso-downloader/winget/manifests/starkSV.msdl.locale.en-US.yaml manifests/s/starkSV/msdl/0.1.0/
-
-git checkout -b add-starkSV.msdl-0.1.0
-git add manifests/s/starkSV/msdl/
-git commit -m "New package: starkSV.msdl version 0.1.0"
-git push origin add-starkSV.msdl-0.1.0
-```
-
-Then open a PR from `starkSV/winget-pkgs:add-starkSV.msdl-0.1.0` → `microsoft/winget-pkgs:master`.
-
-The automated bot runs within minutes; human review typically takes 1–7 days for new packages. Once merged, `winget install starkSV.msdl` works globally and the CI's `wingetcreate update` step will auto-submit all future version PRs.
-
----
-
-## Self-Review
-
-**Spec coverage:**
-- ✅ Standalone Go binary — `cli/go.mod`, standard library only (zero external deps; UUID via `crypto/rand`)
-- ✅ Calls Microsoft directly from the user's machine — `microsoft.go` (NOT the MSDL backend), bypassing the Sentinel WAF that blocks the data-center backend
-- ✅ Full session flow ported — `newSession` (fingerprinting + `simpleCookieJar`), `fetchLanguages`, `fetchDownloadLinks` (with warmup)
-- ✅ Eval ISOs fetched directly — `runEval` + `fetchEvalLinks` scrapes the Eval Center page and resolves fwlinks
-- ✅ Interactive mode — `picker.go` + `pickProduct` / `pickLanguage` / `pickEvalProduct`
-- ✅ Named-arg mode — `--id`, `--lang`, `--eval slug`
-- ✅ Falls back to interactive when args omitted — `runConsumer` / `runEval`
-- ✅ Output URL to stdout, prompts to stderr — `fmt.Println(link.URI)` / `fmt.Fprintln(os.Stderr, ...)`
-- ✅ Search/filter by name — `searchProducts` in `catalog.go`
-- ✅ Auto-select when only one match — `if len(candidates) == 1` in `runConsumer`
-- ✅ Distribution — GitHub Actions workflow with install instructions
-- ✅ Winget publishing — manifests in `winget/manifests/`, initial PR documented step-by-step, CI auto-submits future version PRs via `wingetcreate update`
-
-**Out of scope here (covered in `IMPLEMENTATION_PLAN.md`):** crowdsourced contribution (`--contribute` flag + backend `/contribute` endpoint, Phase 3) and the web-to-CLI handoff (Phase 4).
-
-**Placeholder scan:** `PASTE_SHA256_FROM_STEP_1_HERE` in Task 7 Step 3 is intentional — it cannot be filled until the binary is built and published in Task 6. All other steps have complete, runnable content.
-
-**Ordering constraint:** Task 7 depends on Task 6 (needs the published binary URL and real SHA256). Tasks 1–5 are independent of Task 7 and can be completed before Task 6.
-
-**Type consistency:**
-- `Product.ID` defined Task 2 → used as `product.ID` in Task 5 `fetchLanguages(client, sessionID, product.ID)` ✅
-- `Language.ID` defined Task 3 → used as `lang.ID` in Task 5 `fetchDownloadLinks(client, sessionID, product.ID, lang.ID)` ✅
-- `newSession()` returns `(*http.Client, string)` (Task 3) → consumed as `client, sessionID := newSession()` in Task 5 `runConsumer` ✅
-- `EvalProduct.EvalURL` defined Task 2 → used as `ep.EvalURL` in Task 5 `fetchEvalLinks(ep.EvalURL)` ✅
-- `DownloadLink.URI` defined Task 3 → printed as `link.URI` in Task 5 ✅
-- `EvalLink.Arch` / `EvalLink.URL` defined Task 3 → printed in Task 5 `runEval` ✅
-- `parseChoice` defined Task 4 → called in `pickProduct`, `pickLanguage`, `pickEvalProduct` in Task 4 ✅
-- `consumerProducts`, `evalProducts` defined Task 2 → iterated in Task 5 `--list` mode ✅
diff --git a/docs/superpowers/plans/2026-06-21-cli-telemetry-update-check.md b/docs/superpowers/plans/2026-06-21-cli-telemetry-update-check.md
deleted file mode 100644
index 161cec8..0000000
--- a/docs/superpowers/plans/2026-06-21-cli-telemetry-update-check.md
+++ /dev/null
@@ -1,1023 +0,0 @@
-# CLI Telemetry + Update Check Implementation Plan
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Add anonymous CLI usage telemetry, an update check on every run, and Redis durability for existing in-memory metrics — all on the `feat/cli-telemetry` branch.
-
-**Architecture:** Three additions: (1) two new backend endpoints (`GET /cli/version`, `POST /telemetry`) that write counters to Redis; (2) Redis persistence for the existing in-memory metrics (seed on startup, flush every 5 min, final flush on graceful shutdown); (3) CLI goroutines that call those endpoints on every run. All CLI side-channel work uses `MSDL_NO_TELEMETRY=1` as a single opt-out.
-
-**Tech Stack:** Go 1.21+, `backend/main.go` (monolith, no framework), `github.com/redis/go-redis/v9`, CLI uses `runtime.GOOS` for platform, `flag` package already in place.
-
-## Global Constraints
-
-- REDIS_URL must NEVER be committed to any file — it is injected as an env var in Coolify only.
-- All changes go to feature branch `feat/cli-telemetry`. Do NOT push until winget PR #390908 is approved.
-- Backend: `latestCLIVersion = "0.3.0"` is a constant in `backend/main.go` — bumped manually per release.
-- Telemetry is best-effort throughout: if Redis is nil or HTTP fails, silently skip — never return an error to the user.
-- Per-IP rate limit on `/telemetry`: 10 req/min, burst 10 (reuse existing `newIPRateLimiter` pattern).
-- `MSDL_NO_TELEMETRY=1` env var skips both update check and telemetry in the CLI.
-- CLI `Version` var defaults to `"dev"` in source; injected as `0.3.0` via `-X main.Version=0.3.0` in CI.
-- No unit tests for backend handlers (existing codebase has none — test via curl). CLI: add tests only where deterministic pure functions are introduced.
-
----
-
-### Task 1: Backend — `GET /cli/version` endpoint
-
-**Files:**
-- Modify: `backend/main.go`
-
-**Interfaces:**
-- Produces: `GET /cli/version` → `{"latest": "0.3.0"}` (200 OK, no auth)
-
-- [ ] **Step 1: Create feature branch**
-
-```bash
-git checkout -b feat/cli-telemetry
-```
-
-- [ ] **Step 2: Add `latestCLIVersion` constant**
-
-Find the existing `const (` block at the top of `backend/main.go` (lines ~32-39). Add `latestCLIVersion` as the last entry before the closing `)`:
-
-```go
-const (
- UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
- PROFILE = "606624d44113"
- LOCALE = "en-US"
- ORG_ID = "y6jn8c31"
- CUSTOMER_ID = "560dc9f3-1aa5-4a2f-b63c-9e18f8d0e175"
- PORT = ":3002"
-
- latestCLIVersion = "0.3.0"
-)
-```
-
-- [ ] **Step 3: Add `handleCLIVersion` function**
-
-Add this function just before `func handleMetrics`:
-
-```go
-func handleCLIVersion(w http.ResponseWriter, r *http.Request) {
- w.Header().Set("Content-Type", "application/json")
- json.NewEncoder(w).Encode(map[string]string{"latest": latestCLIVersion})
-}
-```
-
-- [ ] **Step 4: Register route in `main()`**
-
-In the `main()` function, inside the `mux.HandleFunc(...)` block, add after the `/health` route:
-
-```go
-mux.HandleFunc("/cli/version", handleCLIVersion)
-```
-
-- [ ] **Step 5: Build to verify compilation**
-
-```bash
-cd backend && go build -o /dev/null . && echo "OK"
-```
-
-Expected: `OK` with no errors.
-
-- [ ] **Step 6: Smoke-test with the running backend**
-
-```bash
-curl -s http://localhost:3002/cli/version
-```
-
-Expected: `{"latest":"0.3.0"}`
-
-- [ ] **Step 7: Commit**
-
-```bash
-git add backend/main.go
-git commit -m "feat(backend): GET /cli/version returns latest CLI version"
-```
-
----
-
-### Task 2: Backend — `POST /telemetry` endpoint
-
-**Files:**
-- Modify: `backend/main.go`
-
-**Interfaces:**
-- Consumes: JSON body `{ action, product_id?, eval_slug?, platform, version, success }`
-- Produces: `POST /telemetry` → `{}` (200 OK, no auth); 429 on rate limit; 400 on bad JSON/action
-
-Redis keys written:
-```
-msdl:telemetry:actions hash { fetch, eval, list, interactive }
-msdl:telemetry:platforms hash { windows, darwin, linux }
-msdl:telemetry:versions hash { "0.2.0", "0.3.0", ... }
-msdl:telemetry:products hash { "2618", "3262", ... }
-msdl:telemetry:results hash { success, failed }
-```
-
-- [ ] **Step 1: Add `TelemetryPayload` struct and rate limiter**
-
-Add just after the `var contributeRL` line (~line 472 in current file):
-
-```go
-var telemetryRL = newIPRateLimiter(10, 10) // 10 requests/min, burst 10
-
-type telemetryPayload struct {
- Action string `json:"action"`
- ProductID string `json:"product_id"`
- EvalSlug string `json:"eval_slug"`
- Platform string `json:"platform"`
- Version string `json:"version"`
- Success bool `json:"success"`
-}
-```
-
-- [ ] **Step 2: Add `handleTelemetry` function**
-
-Add this function just before `handleCLIVersion`:
-
-```go
-func handleTelemetry(w http.ResponseWriter, r *http.Request) {
- if r.Method != http.MethodPost {
- http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
- return
- }
- w.Header().Set("Content-Type", "application/json")
-
- ip := r.RemoteAddr
- if fwd := r.Header.Get("X-Forwarded-For"); fwd != "" {
- ip = strings.TrimSpace(strings.SplitN(fwd, ",", 2)[0])
- }
- if !telemetryRL.allow(ip) {
- respondJSONError(w, http.StatusTooManyRequests, "rate limit exceeded")
- return
- }
-
- var p telemetryPayload
- if err := json.NewDecoder(r.Body).Decode(&p); err != nil {
- respondJSONError(w, http.StatusBadRequest, "invalid JSON")
- return
- }
-
- validActions := map[string]bool{"fetch": true, "eval": true, "list": true, "interactive": true}
- if !validActions[p.Action] {
- respondJSONError(w, http.StatusBadRequest, "invalid action")
- return
- }
-
- if rdb != nil {
- ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
- defer cancel()
- rdb.HIncrBy(ctx, "msdl:telemetry:actions", p.Action, 1)
- if p.Platform != "" {
- rdb.HIncrBy(ctx, "msdl:telemetry:platforms", p.Platform, 1)
- }
- if p.Version != "" {
- rdb.HIncrBy(ctx, "msdl:telemetry:versions", p.Version, 1)
- }
- if p.ProductID != "" {
- rdb.HIncrBy(ctx, "msdl:telemetry:products", p.ProductID, 1)
- }
- result := "success"
- if !p.Success {
- result = "failed"
- }
- rdb.HIncrBy(ctx, "msdl:telemetry:results", result, 1)
- }
-
- w.WriteHeader(http.StatusOK)
- w.Write([]byte("{}"))
-}
-```
-
-- [ ] **Step 3: Register route in `main()`**
-
-```go
-mux.HandleFunc("/telemetry", handleTelemetry)
-```
-
-- [ ] **Step 4: Build to verify compilation**
-
-```bash
-cd backend && go build -o /dev/null . && echo "OK"
-```
-
-- [ ] **Step 5: Smoke-test `/telemetry`**
-
-```bash
-curl -s -X POST http://localhost:3002/telemetry \
- -H "Content-Type: application/json" \
- -d '{"action":"fetch","product_id":"3262","platform":"windows","version":"0.3.0","success":true}'
-```
-
-Expected: `{}`
-
-```bash
-curl -s -X POST http://localhost:3002/telemetry \
- -H "Content-Type: application/json" \
- -d '{"action":"bad_action","platform":"linux","version":"0.3.0","success":false}'
-```
-
-Expected: `{"error":"invalid action"}` (400)
-
-- [ ] **Step 6: Commit**
-
-```bash
-git add backend/main.go
-git commit -m "feat(backend): POST /telemetry endpoint with Redis HINCRBY counters"
-```
-
----
-
-### Task 3: Backend — Redis metrics persistence + graceful shutdown
-
-**Files:**
-- Modify: `backend/main.go`
-
-**Interfaces:**
-- Consumes: existing `atomic.Int64` vars (`mSkuRequests`, `mLinkRequests`, etc.)
-- Produces: Redis hashes `msdl:metrics:sku`, `msdl:metrics:link`, `msdl:metrics:eval`
-- Produces: graceful shutdown that calls `flushMetricsToRedis()` before exit
-
-Redis hash schemas:
-```
-msdl:metrics:sku → { requests, cache_hits, ms_fetches, neg_hits }
-msdl:metrics:link → { requests, cache_hits, ms_fetches, neg_hits, stale }
-msdl:metrics:eval → { requests, cache_hits, stale }
-```
-
-- [ ] **Step 1: Add `"os/signal"` and `"syscall"` to imports**
-
-The existing import block in `backend/main.go` starts on line 3. Add the two new packages:
-
-```go
-import (
- "bytes"
- "context"
- "encoding/json"
- "fmt"
- "html"
- "io"
- "log"
- "math/rand"
- "net/http"
- "net/url"
- "os"
- "os/signal"
- "regexp"
- "sort"
- "strconv"
- "strings"
- "sync"
- "sync/atomic"
- "syscall"
- "time"
-
- "github.com/google/uuid"
- "github.com/redis/go-redis/v9"
- "golang.org/x/sync/singleflight"
-)
-```
-
-- [ ] **Step 2: Add `seedMetricsFromRedis` and `flushMetricsToRedis` functions**
-
-Add both functions just after the `redisSeedLinkCache` function (~line 370):
-
-```go
-// seedMetricsFromRedis loads persisted metric counters from Redis into the
-// in-memory atomic vars. Called once on startup after Redis connects.
-func seedMetricsFromRedis() {
- if rdb == nil {
- return
- }
- ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
- defer cancel()
-
- loadHash := func(key string) map[string]string {
- vals, err := rdb.HGetAll(ctx, key).Result()
- if err != nil {
- return nil
- }
- return vals
- }
- parseI64 := func(m map[string]string, field string) int64 {
- if m == nil {
- return 0
- }
- v, _ := strconv.ParseInt(m[field], 10, 64)
- return v
- }
-
- sku := loadHash("msdl:metrics:sku")
- atomic.StoreInt64(&mSkuRequests, parseI64(sku, "requests"))
- atomic.StoreInt64(&mSkuCacheHits, parseI64(sku, "cache_hits"))
- atomic.StoreInt64(&mSkuFetches, parseI64(sku, "ms_fetches"))
- atomic.StoreInt64(&mSkuNegHits, parseI64(sku, "neg_hits"))
-
- link := loadHash("msdl:metrics:link")
- atomic.StoreInt64(&mLinkRequests, parseI64(link, "requests"))
- atomic.StoreInt64(&mLinkCacheHits, parseI64(link, "cache_hits"))
- atomic.StoreInt64(&mLinkFetches, parseI64(link, "ms_fetches"))
- atomic.StoreInt64(&mLinkNegHits, parseI64(link, "neg_hits"))
- atomic.StoreInt64(&mLinkStale, parseI64(link, "stale"))
-
- eval := loadHash("msdl:metrics:eval")
- atomic.StoreInt64(&mEvalRequests, parseI64(eval, "requests"))
- atomic.StoreInt64(&mEvalCacheHits, parseI64(eval, "cache_hits"))
- atomic.StoreInt64(&mEvalStale, parseI64(eval, "stale"))
-
- log.Println("redis: seeded in-memory metrics from Redis")
-}
-
-// flushMetricsToRedis writes current in-memory metric counters to Redis.
-// Called periodically (every 5 min) and on graceful shutdown.
-func flushMetricsToRedis() {
- if rdb == nil {
- return
- }
- ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
- defer cancel()
-
- rdb.HSet(ctx, "msdl:metrics:sku",
- "requests", atomic.LoadInt64(&mSkuRequests),
- "cache_hits", atomic.LoadInt64(&mSkuCacheHits),
- "ms_fetches", atomic.LoadInt64(&mSkuFetches),
- "neg_hits", atomic.LoadInt64(&mSkuNegHits),
- )
- rdb.HSet(ctx, "msdl:metrics:link",
- "requests", atomic.LoadInt64(&mLinkRequests),
- "cache_hits", atomic.LoadInt64(&mLinkCacheHits),
- "ms_fetches", atomic.LoadInt64(&mLinkFetches),
- "neg_hits", atomic.LoadInt64(&mLinkNegHits),
- "stale", atomic.LoadInt64(&mLinkStale),
- )
- rdb.HSet(ctx, "msdl:metrics:eval",
- "requests", atomic.LoadInt64(&mEvalRequests),
- "cache_hits", atomic.LoadInt64(&mEvalCacheHits),
- "stale", atomic.LoadInt64(&mEvalStale),
- )
- log.Println("redis: flushed in-memory metrics to Redis")
-}
-```
-
-- [ ] **Step 3: Add `startMetricsFlusher` goroutine function**
-
-Add after `flushMetricsToRedis`:
-
-```go
-// startMetricsFlusher flushes in-memory metrics to Redis every 5 minutes.
-func startMetricsFlusher() {
- ticker := time.NewTicker(5 * time.Minute)
- for range ticker.C {
- flushMetricsToRedis()
- }
-}
-```
-
-- [ ] **Step 4: Call `seedMetricsFromRedis` in `initRedis`**
-
-Find the `initRedis` function. After `rdb = client` and `log.Println("redis: connected")`, add:
-
-```go
- rdb = client
- log.Println("redis: connected")
- seedMetricsFromRedis()
-```
-
-- [ ] **Step 5: Rewrite `main()` to use `http.Server` with graceful shutdown**
-
-Replace the existing `main()` function body:
-
-```go
-func main() {
- initRedis()
- go redisSeedLinkCache()
- go cleanupSessions()
- go cleanupCaches()
- go warmEvalCache()
- go startMetricsFlusher()
-
- mux := http.NewServeMux()
- mux.HandleFunc("/skuinfo", handleSkuInfo)
- mux.HandleFunc("/proxy", handleProxy)
- mux.HandleFunc("/contribute", handleContribute)
- mux.HandleFunc("/evallinks", handleEvalLinks)
- mux.HandleFunc("/metrics", handleMetrics)
- mux.HandleFunc("/telemetry", handleTelemetry)
- mux.HandleFunc("/cli/version", handleCLIVersion)
- mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
- w.Header().Set("Content-Type", "application/json")
- w.Write([]byte(`{"status":"ok"}`))
- })
- mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
- if r.URL.Path != "/" {
- http.NotFound(w, r)
- return
- }
- w.Write([]byte("MSDL API v3 is running"))
- })
-
- handler := enableCORS(mux)
- srv := &http.Server{Addr: PORT, Handler: handler}
-
- sigCh := make(chan os.Signal, 1)
- signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
- go func() {
- <-sigCh
- log.Println("shutdown: flushing metrics to Redis...")
- flushMetricsToRedis()
- ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
- defer cancel()
- srv.Shutdown(ctx)
- }()
-
- log.Printf("Go Backend running on http://localhost%s\n", PORT)
- if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
- log.Fatal(err)
- }
-}
-```
-
-- [ ] **Step 6: Build to verify compilation**
-
-```bash
-cd backend && go build -o /dev/null . && echo "OK"
-```
-
-- [ ] **Step 7: Verify Redis flush manually**
-
-Start the backend locally with `REDIS_URL` set, run a few requests, then check Redis:
-
-```bash
-redis-cli -u "$REDIS_URL" HGETALL msdl:metrics:sku
-```
-
-(Values will be 0 on a fresh Redis; after a request they increment.)
-
-- [ ] **Step 8: Commit**
-
-```bash
-git add backend/main.go
-git commit -m "feat(backend): Redis persistence for metrics (seed + 5min flush + graceful shutdown)"
-```
-
----
-
-### Task 4: Backend — Telemetry section in `/metrics` response
-
-**Files:**
-- Modify: `backend/main.go`
-
-**Interfaces:**
-- Consumes: `msdl:telemetry:*` Redis hashes (written by `handleTelemetry`)
-- Produces: adds `"telemetry"` key to existing `handleMetrics` JSON response
-
-- [ ] **Step 1: Add `loadTelemetryFromRedis` helper**
-
-Add just before `handleMetrics`:
-
-```go
-// loadTelemetryFromRedis reads all telemetry counters from Redis.
-// Returns nil maps when Redis is unavailable.
-func loadTelemetryFromRedis() map[string]map[string]string {
- if rdb == nil {
- return nil
- }
- ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
- defer cancel()
- keys := []string{
- "msdl:telemetry:actions",
- "msdl:telemetry:platforms",
- "msdl:telemetry:versions",
- "msdl:telemetry:products",
- "msdl:telemetry:results",
- }
- out := make(map[string]map[string]string, len(keys))
- for _, k := range keys {
- vals, err := rdb.HGetAll(ctx, k).Result()
- if err == nil {
- // Strip the "msdl:telemetry:" prefix for the JSON key
- name := strings.TrimPrefix(k, "msdl:telemetry:")
- out[name] = vals
- }
- }
- return out
-}
-```
-
-- [ ] **Step 2: Update `handleMetrics` to include telemetry**
-
-Find the `json.NewEncoder(w).Encode(map[string]interface{}{` call in `handleMetrics`. Replace the map literal to add a `"telemetry"` field:
-
-```go
- telemetry := loadTelemetryFromRedis()
-
- w.Header().Set("Content-Type", "application/json")
- json.NewEncoder(w).Encode(map[string]interface{}{
- "sku": map[string]interface{}{
- "requests": skuReqs,
- "cache_hits": skuHits,
- "ms_fetches": skuFetches,
- "neg_hits": skuNeg,
- "hit_rate": hitRate(skuHits, skuReqs),
- "cache_size": skuSize,
- },
- "link": map[string]interface{}{
- "requests": linkReqs,
- "cache_hits": linkHits,
- "ms_fetches": linkFetches,
- "neg_hits": linkNeg,
- "stale": linkStale,
- "hit_rate": hitRate(linkHits, linkReqs),
- "cache_size": linkSize,
- },
- "eval": map[string]interface{}{
- "requests": evalReqs,
- "cache_hits": evalHits,
- "stale": evalStale,
- "hit_rate": hitRate(evalHits, evalReqs),
- "cache_size": evalSize,
- },
- "neg_cache_size": negSize,
- "total_ms_fetches": skuFetches + linkFetches,
- "telemetry": telemetry,
- })
-```
-
-- [ ] **Step 3: Build to verify compilation**
-
-```bash
-cd backend && go build -o /dev/null . && echo "OK"
-```
-
-- [ ] **Step 4: Smoke-test `/metrics`**
-
-```bash
-curl -s "http://localhost:3002/metrics?secret=$METRICS_SECRET" | python -m json.tool
-```
-
-Expected: existing fields still present plus `"telemetry": { "actions": {}, ... }` (empty until telemetry events arrive).
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add backend/main.go
-git commit -m "feat(backend): include telemetry counters in /metrics response"
-```
-
----
-
-### Task 5: CLI — `Version` var + GitHub Actions ldflags + `--help` update
-
-**Files:**
-- Modify: `cli/main.go`
-- Modify: `.github/workflows/cli-release.yml`
-
-**Interfaces:**
-- Produces: `Version` package-level variable accessible from all CLI functions
-- Produces: CI builds inject `-X main.Version=` so the binary reports its real version
-
-- [ ] **Step 1: Add `Version` variable to `cli/main.go`**
-
-At the top of `cli/main.go`, after the `import` block, add:
-
-```go
-// Version is injected at build time via -ldflags "-X main.Version=0.3.0".
-// Falls back to "dev" for local builds.
-var Version = "dev"
-```
-
-- [ ] **Step 2: Update `fs.Usage` in `run()` to include env vars section**
-
-Replace the existing `fs.Usage` function in `run()`:
-
-```go
-fs.Usage = func() {
- fmt.Fprintln(os.Stderr, `msdl — Windows ISO downloader
-
-Usage:
- msdl [search terms] interactive: filter + pick product, pick language
- msdl --id 3262 skip product picker
- msdl --id 3262 --lang "English (United States)" no prompts, print URL directly
- msdl --eval [slug] evaluation ISOs (server-2025, win11-ent, ...)
- msdl --list list all products and exit
-
-Flags:`)
- fs.PrintDefaults()
- fmt.Fprintln(os.Stderr, `
-Environment:
- MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks
- MSDL_NO_CONTRIBUTE=1 Disable cache contribution
- MSDL_API_URL= Override backend URL (default: https://api.msdl.tech-latest.com)
-
-More info: https://msdl.tech-latest.com/cli`)
-}
-```
-
-- [ ] **Step 3: Handle `flag.ErrHelp` cleanly in `run()`**
-
-Find `if err := fs.Parse(args); err != nil {` and update it:
-
-```go
-if err := fs.Parse(args); err != nil {
- if err == flag.ErrHelp {
- return nil // fs.Usage already printed
- }
- return err
-}
-```
-
-- [ ] **Step 4: Update GitHub Actions workflow to inject version ldflags**
-
-In `.github/workflows/cli-release.yml`, find the `Build binaries` step. Replace the `go build` commands to inject version:
-
-```yaml
- - name: Build binaries
- working-directory: cli
- run: |
- VERSION="${{ github.ref_name }}"
- VERSION="${VERSION#cli/v}"
- mkdir -p ../dist
- GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-linux-amd64 .
- GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-darwin-amd64 .
- GOOS=darwin GOARCH=arm64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-darwin-arm64 .
- GOOS=windows GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-windows-amd64.exe .
-```
-
-- [ ] **Step 5: Build CLI locally to verify**
-
-```bash
-cd cli && go build -o /dev/null . && echo "OK"
-```
-
-- [ ] **Step 6: Verify --help flag works**
-
-```bash
-cd cli && go run . --help
-```
-
-Expected: prints usage block + env var section, exits 0 (no "error:" prefix).
-
-- [ ] **Step 7: Commit**
-
-```bash
-git add cli/main.go .github/workflows/cli-release.yml
-git commit -m "feat(cli): Version var injected via ldflags, updated --help text"
-```
-
----
-
-### Task 6: CLI — Update check + telemetry goroutines
-
-**Files:**
-- Modify: `cli/main.go`
-
-**Interfaces:**
-- Consumes: `Version` (from Task 5), `apiBaseURL()` helper
-- Consumes: `GET /cli/version` backend endpoint (from Task 1)
-- Consumes: `POST /telemetry` backend endpoint (from Task 2)
-- Produces: update notice printed to stderr before main output when a newer version is available
-- Produces: telemetry payload sent fire-and-forget after every CLI run
-
-- [ ] **Step 1: Add `runtime` to imports in `cli/main.go`**
-
-The existing import block has `"sync"` and other packages. Add `"runtime"`:
-
-```go
-import (
- "bytes"
- "context"
- "encoding/json"
- "flag"
- "fmt"
- "net/http"
- "os"
- "runtime"
- "strings"
- "sync"
- "time"
-)
-```
-
-- [ ] **Step 2: Add `apiBaseURL` helper**
-
-Add after the `contributeSecret` const near the top of the file (before `urlFilename`):
-
-```go
-// apiBaseURL returns the backend base URL, with MSDL_API_URL override.
-func apiBaseURL() string {
- if u := os.Getenv("MSDL_API_URL"); u != "" {
- return strings.TrimRight(u, "/")
- }
- return "https://api.msdl.tech-latest.com"
-}
-```
-
-Note: `contributeURL()` already does similar logic but for `/contribute` specifically. `apiBaseURL` is the shared base — update `contributeURL` to use it:
-
-```go
-func contributeURL() string {
- return apiBaseURL() + "/contribute"
-}
-```
-
-- [ ] **Step 3: Add `printUpdateNotice` function**
-
-Add after `apiBaseURL`:
-
-```go
-// printUpdateNotice checks /cli/version and prints a notice if a newer version
-// is available. Blocks up to 500ms; silently no-ops on timeout or any error.
-func printUpdateNotice() {
- ch := make(chan string, 1)
- go func() {
- ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
- defer cancel()
- req, err := http.NewRequestWithContext(ctx, http.MethodGet, apiBaseURL()+"/cli/version", nil)
- if err != nil {
- return
- }
- resp, err := http.DefaultClient.Do(req)
- if err != nil {
- return
- }
- defer resp.Body.Close()
- var body struct {
- Latest string `json:"latest"`
- }
- if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
- return
- }
- ch <- body.Latest
- }()
- select {
- case latest := <-ch:
- if latest != "" && latest != Version && Version != "dev" {
- fmt.Fprintf(os.Stderr, "\n A new version of msdl is available: v%s\n", latest)
- fmt.Fprintf(os.Stderr, " Download: https://github.com/starkSV/windows-iso-downloader/releases/latest\n\n")
- }
- case <-time.After(500 * time.Millisecond):
- }
-}
-```
-
-- [ ] **Step 4: Add `cliTelemetryPayload` struct and `sendTelemetry` function**
-
-Add after `printUpdateNotice`:
-
-```go
-type cliTelemetryPayload struct {
- Action string `json:"action"`
- ProductID string `json:"product_id,omitempty"`
- EvalSlug string `json:"eval_slug,omitempty"`
- Platform string `json:"platform"`
- Version string `json:"version"`
- Success bool `json:"success"`
-}
-
-// sendTelemetry posts a single telemetry event. Fire-and-forget: all errors
-// are silently ignored so telemetry never affects the user experience.
-func sendTelemetry(p cliTelemetryPayload) {
- body, err := json.Marshal(p)
- if err != nil {
- return
- }
- ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
- defer cancel()
- req, err := http.NewRequestWithContext(ctx, http.MethodPost, apiBaseURL()+"/telemetry", bytes.NewReader(body))
- if err != nil {
- return
- }
- req.Header.Set("Content-Type", "application/json")
- resp, err := http.DefaultClient.Do(req)
- if err != nil {
- return
- }
- resp.Body.Close()
-}
-```
-
-- [ ] **Step 5: Thread update check and telemetry into `run()`**
-
-Replace the entire `run()` function with the complete version below. This incorporates the Task 5 changes (Usage block, ErrHelp handling) plus the new telemetry logic:
-
-```go
-func run(args []string) error {
- fs := flag.NewFlagSet("msdl", flag.ContinueOnError)
- fs.SetOutput(os.Stderr)
- fs.Usage = func() {
- fmt.Fprintln(os.Stderr, `msdl — Windows ISO downloader
-
-Usage:
- msdl [search terms] interactive: filter + pick product, pick language
- msdl --id 3262 skip product picker
- msdl --id 3262 --lang "English (United States)" no prompts, print URL directly
- msdl --eval [slug] evaluation ISOs (server-2025, win11-ent, ...)
- msdl --list list all products and exit
-
-Flags:`)
- fs.PrintDefaults()
- fmt.Fprintln(os.Stderr, `
-Environment:
- MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks
- MSDL_NO_CONTRIBUTE=1 Disable cache contribution
- MSDL_API_URL= Override backend URL (default: https://api.msdl.tech-latest.com)
-
-More info: https://msdl.tech-latest.com/cli`)
- }
-
- productID := fs.String("id", "", "consumer product ID (skips product picker)")
- langFlag := fs.String("lang", "", `language name, e.g. "English (United States)"`)
- evalMode := fs.Bool("eval", false, "fetch evaluation ISOs")
- listMode := fs.Bool("list", false, "list all products and exit")
- noContributeFlag := fs.Bool("no-contribute", false, "skip sharing the link with the msdl.tech cache")
-
- if err := fs.Parse(args); err != nil {
- if err == flag.ErrHelp {
- return nil
- }
- return err
- }
- query := strings.Join(fs.Args(), " ")
- noContribute := *noContributeFlag || os.Getenv("MSDL_NO_CONTRIBUTE") == "1"
- noTelemetry := os.Getenv("MSDL_NO_TELEMETRY") == "1"
-
- // Update check — blocks up to 500ms, then continues regardless
- if !noTelemetry {
- printUpdateNotice()
- }
-
- if *listMode {
- fmt.Fprintln(os.Stderr, "Consumer products:")
- for _, p := range consumerProducts {
- fmt.Fprintf(os.Stderr, " %-6s %s\n", p.ID, p.Name)
- }
- fmt.Fprintln(os.Stderr, "\nEvaluation products:")
- for _, p := range evalProducts {
- fmt.Fprintf(os.Stderr, " %-20s %s\n", p.Slug, p.Name)
- }
- if !noTelemetry {
- go sendTelemetry(cliTelemetryPayload{
- Action: "list",
- Platform: runtime.GOOS,
- Version: Version,
- Success: true,
- })
- }
- return nil
- }
-
- // Determine telemetry fields before running (product_id / eval_slug known at this point)
- telAction := "interactive"
- telProductID := ""
- telEvalSlug := ""
- switch {
- case *evalMode:
- telAction = "eval"
- telEvalSlug = query
- case *productID != "":
- telAction = "fetch"
- telProductID = *productID
- }
-
- var err error
- if *evalMode {
- err = runEval(query)
- } else {
- err = runConsumer(*productID, query, *langFlag, noContribute)
- }
-
- if !noTelemetry {
- go sendTelemetry(cliTelemetryPayload{
- Action: telAction,
- ProductID: telProductID,
- EvalSlug: telEvalSlug,
- Platform: runtime.GOOS,
- Version: Version,
- Success: err == nil,
- })
- }
-
- return err
-}
-```
-
-- [ ] **Step 6: Build to verify compilation**
-
-```bash
-cd cli && go build -o /dev/null . && echo "OK"
-```
-
-- [ ] **Step 7: Test update check with local backend**
-
-Start the backend, then:
-
-```bash
-cd cli && MSDL_API_URL=http://localhost:3002 Version=0.2.0 go run . --list
-```
-
-Expected: update notice printed ("A new version of msdl is available: v0.3.0") before the product list. (Note: `Version` env var does NOT override the Go variable — use a built binary with `-ldflags "-X main.Version=0.2.0"` to test this properly.)
-
-Correct test:
-
-```bash
-cd cli && go build -ldflags "-X main.Version=0.2.0" -o msdl-test . && \
- MSDL_API_URL=http://localhost:3002 ./msdl-test --list
-```
-
-Expected first lines of stderr:
-
-```
- A new version of msdl is available: v0.3.0
- Download: https://github.com/starkSV/windows-iso-downloader/releases/latest
-```
-
-- [ ] **Step 8: Test telemetry fires**
-
-After the above run, check Redis:
-
-```bash
-redis-cli -u "$REDIS_URL" HGETALL msdl:telemetry:actions
-```
-
-Expected: `list` field with value `1`.
-
-```bash
-redis-cli -u "$REDIS_URL" HGETALL msdl:telemetry:platforms
-redis-cli -u "$REDIS_URL" HGETALL msdl:telemetry:versions
-```
-
-Expected: platform and version populated.
-
-- [ ] **Step 9: Test opt-out**
-
-```bash
-cd cli && MSDL_NO_TELEMETRY=1 MSDL_API_URL=http://localhost:3002 ./msdl-test --list
-```
-
-Expected: no update notice, no new telemetry entries in Redis.
-
-- [ ] **Step 10: Clean up test binary**
-
-```bash
-rm cli/msdl-test
-```
-
-- [ ] **Step 11: Run existing CLI tests**
-
-```bash
-cd cli && go test ./... -v
-```
-
-Expected: all existing tests pass (no new tests required — goroutines are integration-tested manually above).
-
-- [ ] **Step 12: Commit**
-
-```bash
-git add cli/main.go
-git commit -m "feat(cli): update check and anonymous telemetry on every run"
-```
-
----
-
-### Task 7: Documentation updates
-
-**Files:**
-- Modify: `README.md`
-
-**Interfaces:** None — doc-only.
-
-- [ ] **Step 1: Add telemetry disclosure to README**
-
-Find the `### Crowdsourced cache` section in `README.md`. Add a new `### Usage telemetry` section immediately after it:
-
-```markdown
-### Usage telemetry
-
-By default, each run sends an anonymous event to help us understand which products are popular and which platforms are used. No personal data is sent — only action type (`fetch`, `eval`, `list`, `interactive`), platform (`windows`, `darwin`, `linux`), CLI version, and whether the run succeeded. To opt out:
-
-```bash
-MSDL_NO_TELEMETRY=1 msdl --id 3262 --lang "English"
-```
-```
-
-- [ ] **Step 2: Commit**
-
-```bash
-git add README.md
-git commit -m "docs: add CLI telemetry disclosure to README"
-```
-
----
-
-## Post-Implementation Checklist
-
-After all tasks are committed to `feat/cli-telemetry`:
-
-- [ ] Run `cd backend && go build -o /dev/null . && echo OK` — must pass
-- [ ] Run `cd cli && go test ./... && echo OK` — must pass
-- [ ] Verify `/cli/version` endpoint returns correct version
-- [ ] Verify `/telemetry` endpoint writes to Redis
-- [ ] Verify `/metrics` includes `telemetry` section
-- [ ] Verify update notice shows for stale version, suppressed for current version
-- [ ] Verify `MSDL_NO_TELEMETRY=1` skips both update check and telemetry
-- [ ] **DO NOT push until winget PR #390908 is merged**
diff --git a/docs/superpowers/specs/2026-06-21-cli-telemetry-update-check-design.md b/docs/superpowers/specs/2026-06-21-cli-telemetry-update-check-design.md
deleted file mode 100644
index 31ca3fd..0000000
--- a/docs/superpowers/specs/2026-06-21-cli-telemetry-update-check-design.md
+++ /dev/null
@@ -1,200 +0,0 @@
-# CLI Telemetry, Update Check & Help — Design Spec
-
-**Date:** 2026-06-21
-**Status:** Approved
-**Scope:** CLI (cli/) + Backend (backend/main.go)
-
----
-
-## Overview
-
-Three related additions to the msdl CLI and backend:
-
-1. **Telemetry** — anonymous usage counters sent on every CLI run, stored durably in Redis
-2. **Update check** — CLI checks our backend for the latest version on every run, notifies if behind
-3. **Help** — `msdl --help` / `-h` prints usage with all flags and examples
-
-No opt-in required. `MSDL_NO_TELEMETRY=1` skips both telemetry and update check silently.
-Disclosed in README and `/cli` page with one line each.
-
----
-
-## Backend Changes
-
-### 1. New constant: `latestCLIVersion`
-
-```go
-const latestCLIVersion = "0.3.0"
-```
-
-Bumped manually in `backend/main.go` whenever a new CLI release is cut.
-
-### 2. `GET /cli/version` — public, no auth
-
-Returns the latest CLI version. The CLI calls this on every run.
-
-```json
-{ "latest": "0.3.0" }
-```
-
-No rate limiting needed — response is trivial and cacheable by the client.
-
-### 3. `POST /telemetry` — public, no auth
-
-Accepts one event per CLI run. Body:
-
-```json
-{
- "action": "fetch" | "eval" | "list" | "interactive",
- "product_id": "2618",
- "eval_slug": "server-2025",
- "platform": "windows" | "darwin" | "linux",
- "version": "0.3.0",
- "success": true
-}
-```
-
-- `product_id` present only for `fetch` action
-- `eval_slug` present only for `eval` action
-- All other fields always present
-- Returns `200 OK` with `{}` — CLI ignores the response
-- Per-IP rate limit: ~10 req/min (same token bucket pattern as `/contribute`)
-- Increments Redis counters via `HINCRBY` directly (no in-memory layer)
-- If Redis unavailable: silently drop (telemetry is best-effort)
-
-### 4. Redis telemetry keys
-
-```
-msdl:telemetry:actions hash { fetch, eval, list, interactive }
-msdl:telemetry:platforms hash { windows, darwin, linux }
-msdl:telemetry:versions hash { "0.2.0", "0.3.0", ... }
-msdl:telemetry:products hash { "2618", "3262", "3113", ... }
-msdl:telemetry:results hash { success, failed }
-```
-
-### 5. Existing metrics — Redis persistence
-
-Current in-memory `atomic.Int64` counters are preserved for fast per-request increments. Added durability:
-
-- **On startup**: seed in-memory counters from Redis (`HGETALL msdl:metrics:*`)
-- **Every 5 minutes**: flush in-memory counters to Redis (`HSET msdl:metrics:*`)
-- **On graceful shutdown**: final flush before exit
-
-Redis keys for existing metrics:
-
-```
-msdl:metrics:sku hash { requests, hits, fetches, neg_hits }
-msdl:metrics:link hash { requests, hits, fetches, neg_hits, stale }
-msdl:metrics:eval hash { requests, hits, stale }
-```
-
-### 6. `/metrics` response — updated
-
-Adds telemetry section to existing cache stats:
-
-```json
-{
- "sku": { ... },
- "link": { ... },
- "eval": { ... },
- "telemetry": {
- "actions": { "fetch": 142, "eval": 31, "list": 8, "interactive": 67 },
- "platforms": { "windows": 180, "darwin": 42, "linux": 18 },
- "versions": { "0.3.0": 145, "0.2.0": 95 },
- "products": { "3262": 134, "2618": 89, "3113": 67 },
- "results": { "success": 390, "failed": 48 }
- }
-}
-```
-
----
-
-## CLI Changes
-
-### 1. Version constant
-
-Injected at build time via ldflags:
-
-```
--ldflags "-X main.Version=0.3.0"
-```
-
-Added to the existing GitHub Actions release workflow (`cli-release.yml`). Fallback: `const Version = "dev"` in source.
-
-### 2. Update check + telemetry — concurrent goroutines
-
-On every run, immediately after flag parsing, two goroutines are launched:
-
-```
-main()
- ├── go updateCheck() — GET /cli/version, 500ms timeout, print notice if behind
- └── go sendTelemetry() — POST /telemetry, fire-and-forget, result ignored
-```
-
-Both are non-blocking. The update check result is printed before any other output if it arrives within 500ms. If the timeout fires, it's silently skipped for that run.
-
-**Update notice format** (printed before picker or output):
-```
- A new version of msdl is available: v0.3.0
- Download: https://github.com/starkSV/windows-iso-downloader/releases/latest
-```
-
-Both goroutines are skipped entirely when `MSDL_NO_TELEMETRY=1` is set.
-
-### 3. Telemetry payload construction
-
-| Scenario | action | product_id | eval_slug | success |
-|---|---|---|---|---|
-| `msdl` (interactive) | `interactive` | set after pick | — | true/false |
-| `msdl --id X --lang Y` | `fetch` | X | — | true/false |
-| `msdl --eval slug` | `eval` | — | slug | true/false |
-| `msdl --list` | `list` | — | — | true |
-
-`platform` is set from `runtime.GOOS` at runtime (not build time — same result, simpler).
-`version` is set from the `Version` constant.
-
-Telemetry is sent **after** the main action completes so `success` reflects the actual outcome.
-
-### 4. `--help` / `-h`
-
-Prints a formatted usage block and exits 0. Shown automatically by Go's `flag` package or via a custom print if the existing CLI uses manual arg parsing.
-
-```
-msdl — Windows ISO downloader
-
-Usage:
- msdl Interactive mode — pick product and language
- msdl --id --lang Fetch link directly (skip picker)
- msdl --eval Evaluation / Server ISO
- msdl --list List all available products
-
-Flags:
- --id Product ID (e.g. 3262 for Windows 11 25H2)
- --lang Language name (e.g. "English")
- --eval Eval ISO slug (server-2025, server-2022, win11-ent, ...)
- --list List all products and exit
- --no-contribute Skip contributing link back to msdl web cache
- -h, --help Show this help
-
-Environment:
- MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks
- MSDL_NO_CONTRIBUTE=1 Disable cache contribution
-
-More info: https://msdl.tech-latest.com/cli
-```
-
----
-
-## Documentation Updates
-
-- **README** — one line under CLI section: "msdl sends anonymous usage counts (action, platform, version) to help us understand which products are popular. Set `MSDL_NO_TELEMETRY=1` to opt out."
-- **`/cli` page** — same one-liner in the contribute section, next to the `--no-contribute` note.
-
----
-
-## What's Not In Scope
-
-- No per-user tracking, no IP storage, no session IDs
-- No telemetry dashboard UI (raw numbers via `/metrics` is sufficient)
-- No forced update or auto-update — notification only
-- No telemetry for the web frontend
diff --git a/docs/superpowers/specs/2026-07-13-needs-warming-design.md b/docs/superpowers/specs/2026-07-13-needs-warming-design.md
new file mode 100644
index 0000000..6190573
--- /dev/null
+++ b/docs/superpowers/specs/2026-07-13-needs-warming-design.md
@@ -0,0 +1,184 @@
+# Spec: "Needs Warming" Community Help Page
+
+**Status:** Proposed
+**Date:** 2026-07-13
+**Author:** Shekhar + Grok (original idea), refined against live codebase by Claude
+**Related:** Crowdsourced cache warming (`/contribute`), Sentinel lockdown resilience, CLI contribution flow
+
+---
+
+## 1. Goal
+
+Give visible, honest signal for products currently failing web users because Microsoft's Sentinel WAF (or a rate limit) has blocked the backend and no cached link is available to fall back on. Point people at the CLI as the fix, and let a successful fetch/contribution clear the item automatically.
+
+This is not a new mechanism bolted onto the project — it's a public window into resilience state (`negCache` / `linkCache`) that already exists and already drives the per-product "blocked, use CLI" banner. This just makes that signal visible in aggregate, with a stronger call to action.
+
+**Change from the original draft:** a dedicated page, not a homepage section. Original draft (written by Grok from the GitHub repo alone, without runtime visibility into cache state) proposed a parallel tracking system; this version derives the list from state that already exists, which is both less code and immune to drift between two copies of "is this broken."
+
+---
+
+## 2. What Actually Needs Tracking (verified against `backend/main.go`)
+
+`handleProxy` (main.go:1189-1404) has exactly three outcomes when a product/SKU is under an active Sentinel or rate-limit block:
+
+| Outcome | Code path | User experience | Needs warming? |
+|---|---|---|---|
+| `serving cached` | negCache active, `linkCache` entry still within its own TTL (e.g. just contributed) | Gets a working link | **No** |
+| `serving stale` | negCache active, `linkCache` entry expired but still returned | Gets a (possibly near-expired) link | **No** |
+| `no stale available` | negCache active, nothing in `linkCache` at all | Gets a 429, no link | **Yes** — this is the actual failure |
+
+So "needs warming" = **an active `negCache` entry with no corresponding valid-or-stale `linkCache` entry** — computed on read, not tracked as separate state. No new manager struct, no `Record()`/`Remove()` calls to keep in sync, no new Redis schema.
+
+**Known gap, scoped out of MVP:** a fetch that fails with a non-rate-limit error (network blip, malformed response) and has no stale cache to fall back on returns an error directly (main.go:1377) without ever touching `negCache`. In practice this hasn't been the dominant failure mode this month — Sentinel/rate-limit blocks have been — so the MVP doesn't cover it. If it turns out to matter, closing the gap is a two-line addition (write a `negCacheEntry` with `IsSentinel: false` at that call site) rather than a redesign.
+
+---
+
+## 3. Data Needed That Doesn't Exist Yet
+
+Two small additions, both minimal:
+
+1. **A request counter per locked product/SKU.** `negCache`/`linkCache` don't track "how many times did this get hit while broken" — add a small `map[string]int` (or extend `negCacheEntry` with a counter field) incremented at the existing `"no stale available"` log call sites.
+2. **Consumer product display names on the backend.** `validContributeProducts` (main.go:476) is `map[string]bool` — just an ID allow-list, no names. Eval products already have names (`{Name, EvalURL}` map at main.go:207-211). Converting `validContributeProducts` to `map[string]string` (ID → Name) serves both its existing validation role and this new lookup need, with no behavior change to `/contribute`.
+
+Language name hydration for consumer products reuses the existing `skuCache` (already populated per-product from the normal SKU-fetch flow) — look up the SKU by ID within that product's cached SKU list to get `Language`/`LocalizedLanguage`. No new structure needed; if the product isn't in `skuCache` yet, just omit the language field rather than blocking the response.
+
+---
+
+## 4. New Endpoint
+
+### `GET /needs-warming`
+
+Computed on each request by iterating `negCache`, filtering to entries where `IsSentinel` (or rate-limited) is still active **and** no valid `linkCache`/eval-cache entry exists for that key. No background job, no persistence — it's a live view.
+
+**Query params:** `limit` (default 10, max 50)
+
+**Response:**
+```json
+{
+ "items": [
+ {
+ "product_id": "3262",
+ "sku_id": "0x0409",
+ "product_name": "Windows 11 25H2",
+ "language": "English (United States)",
+ "is_eval": false,
+ "reason": "waf_blocked",
+ "last_seen": "2026-07-13T17:42:00Z",
+ "request_count": 47,
+ "cli_command": "msdl --id 3262 --lang \"English (United States)\""
+ }
+ ],
+ "total": 3
+}
+```
+
+- Public, unauthenticated, rate-limited (30 req/min per IP — matches the existing pattern used by `/contribute` at 5/min and `/telemetry` at 10/min)
+- Sorted by `request_count` descending (surfaces the highest-impact items first, more useful than recency alone)
+- Eval products included (`is_eval: true`, `cli_command` uses `msdl --eval ` instead of `--id`)
+
+---
+
+## 5. Backend Implementation Outline
+
+No new package-level state beyond the two additions in §3. The handler:
+
+```go
+func handleNeedsWarming(w http.ResponseWriter, r *http.Request) {
+ if r.Method != http.MethodGet {
+ http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+ return
+ }
+ if !needsWarmingRL.allow(clientIP(r)) {
+ respondJSONError(w, http.StatusTooManyRequests, "rate limit exceeded")
+ return
+ }
+ limit := parseLimitParam(r, 10, 50)
+
+ negCacheMu.RLock()
+ defer negCacheMu.RUnlock()
+ linkCacheMu.RLock()
+ defer linkCacheMu.RUnlock()
+
+ var items []needsWarmingItem
+ for key, neg := range negCache {
+ if time.Now().After(neg.ExpiresAt) {
+ continue // lockdown/rate-limit window itself already expired
+ }
+ if _, hasValidLink := resolveValidLink(key, linkCache); hasValidLink {
+ continue // already resolved via cached/stale link -- not a failure state
+ }
+ items = append(items, buildNeedsWarmingItem(key, neg))
+ }
+ sort.Slice(items, func(i, j int) bool { return items[i].RequestCount > items[j].RequestCount })
+ if len(items) > limit {
+ items = items[:limit]
+ }
+ json.NewEncoder(w).Encode(map[string]interface{}{"items": items, "total": len(items)})
+}
+```
+
+Integration points:
+- Increment the new request counter at the existing `"lockdown active, no stale available"` and first-time-Sentinel/rate-limit-with-no-stale log sites (main.go:1225, ~1367)
+- No changes needed to `/contribute` or the fresh-fetch success path — an item stops appearing automatically once `linkCache` has a valid entry, which those paths already populate
+
+---
+
+## 6. Frontend: Dedicated Page
+
+**Route:** `/needs-warming` (or `/community` — bikeshed-able, not load-bearing)
+
+Linked from the footer and from the per-product "blocked" banner (`CliHandoff.tsx`'s `highlight` variant) — "See what else needs help →" — so people already primed to help via the CLI can see the fuller picture instead of just their one product.
+
+**Empty state:** hide the page's list section entirely (or show a short "Everything's healthy right now" line) — no "0 items" placeholder box. Most visits will likely find this empty, which is a good sign, not a gap to fill with UI.
+
+Each card:
+- Product name + language (or eval product name)
+- Request count as a small badge (social proof — "47 people hit this")
+- One-click "Copy CLI command" button (same interaction pattern as the existing `CliHandoff` component — reuse its copy-button styling)
+- No per-item "reason" jargon exposed in the UI (WAF/rate-limit distinction is backend detail); just "temporarily unavailable"
+
+Tone: matches the project's existing calm, transparent voice (README already explains the caching layer and Sentinel lockdown openly) — helpful, not alarmist.
+
+---
+
+## 7. CLI Impact
+
+None required for MVP. Optional future addition: `msdl --needs-warming` prints the same list from the terminal and offers to fetch+contribute the top item directly — natural fit for the homepage-screen work already shipped (`cli/homepage.go`), but a separate phase.
+
+---
+
+## 8. Privacy & Safety
+
+- Only ever stores/exposes `product_id`, `sku_id`/slug, reason, counters, timestamps — no IPs, no user identifiers (matches the existing telemetry/contribute philosophy already documented in README)
+- Rate-limited public endpoint, same pattern as existing public endpoints
+- Nothing new to abuse: the list is derived from state Microsoft's own WAF already put the backend into, not something a client can inject
+
+---
+
+## 9. Implementation Phases
+
+| Phase | Description | Effort |
+|---|---|---|
+| MVP | Request counter + product-name map conversion + `/needs-warming` handler + dedicated page | Small — most of the state already exists |
+| 2 | Link from per-product blocked banner; polish empty/loading states | Small |
+| 3 | Close the non-rate-limit fetch-error gap (§2) if it turns out to matter in practice | Small, only if needed |
+| 4 | `msdl --needs-warming` CLI command | Small |
+
+(Dropped the original draft's "Redis-backed persistence" phase — this is inherently live/transient state mirroring `negCache`'s own in-memory, reset-on-restart behavior; persisting it would add a Redis key namespace for no real benefit.)
+
+---
+
+## 10. Open Questions Resolved
+
+1. **Eval products included?** Yes.
+2. **Max age before auto-removal?** No separate TTL — inherits the underlying `negCache` entry's own expiry (60s rate-limit / 90min Sentinel), since the list is computed live rather than tracked separately.
+3. **Empty state?** Hide the list, don't show a "0 items" box.
+4. **Tone?** Helpful/neutral, matching existing project voice.
+
+---
+
+## 11. Success Metrics (Future)
+
+- Contributions triggered from this page specifically (could tag `/contribute` calls with a `source=needs-warming` query param to measure this)
+- Reduction in time-to-resolution for locked-down products
+- CLI download/usage bump correlated with items appearing on the page