shekhar/windows-iso-downloader
1
0
Fork 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions

Merge pull request #37 from starkSV/chore/cleanup-stale-docs

Browse source 
chore: remove superseded implementation plan and stale issue draft
This commit is contained in:
Shekhar 2026-07-14 11:27:20 +05:30 • committed by GitHub
commit da1e894e6b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 38 additions and 326 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

284
IMPLEMENTATION_PLAN.md
View file

@ -1,284 +0,0 @@
# MSDL Resilience & CLI Implementation Plan
> **Status:** Proposed — 2026-06-17
> **Context:** Microsoft's Azure Sentinel WAF is blocking data-center IPs (Hetzner, Cloudflare Workers) on the `GetProductDownloadLinksBySku` endpoint at the ASN level. Two senior engineers independently concluded that server-side calls from hosting IPs are not sustainable, and that moving the Microsoft call onto the end-user's machine (the Rufus/Fido model) is the only durable fix.
---
## 1. Problem Statement
MSDL fetches signed Windows ISO download URLs from Microsoft's internal download API. That endpoint is now protected by Azure Sentinel WAF, which blocks by **ASN reputation** — data-center ranges are cleanly distinguishable from residential/commercial ISPs, so Microsoft can throttle hosting traffic with zero impact on real browser users.
**Observed escalation:**
- 2026-06-15: ~2 Sentinel rejections/day, intermittent
- 2026-06-16: ~15 rejections clustered in the evening, consistently hitting Win11 25H2 (products 3262, 3321)
- Windows 10 (2618) and eval builds still succeed; newer/high-traffic products are targeted first
**Current blast radius:** The caching layer (95% hit rate) absorbs most of it. The failure mode is: cache expires → backend refresh is WAF-blocked → stale-on-failure serves the old URL → eventually the URL passes its real `se` expiry → **user gets a dead Microsoft CDN link.**
**Confirmed not fixable server-side:** IP rotation buys weeks; CF Workers already blocked; residential proxies cost money and add a volatile dependency unjustifiable for a free tool. This is Microsoft policy, not a glitch.
---
## 2. Architecture Decision
**Invert the network footprint.** Today: `backend (Hetzner IP) → Microsoft → users`. The hard call originates from a blocked ASN.
New model: **`users (residential IP) → Microsoft → backend cache → all users`**
Three coordinated changes:
1. **CLI calls Microsoft directly** from the user's own machine. Their residential/commercial IP is one Microsoft cannot block without breaking downloads for real customers. This is exactly why Rufus/Fido work.
2. **Crowdsourced cache warming.** When a CLI user fetches a fresh link, the CLI optionally contributes that URL back to the MSDL backend. Microsoft's signed URLs are *not* user-specific (the `se` param is just an expiry timestamp — MSDL already serves one cached URL to many users), so a link fetched by one user is valid for everyone. The backend's Microsoft API calls trend toward zero; Sentinel becomes irrelevant.
3. **Web graceful degradation.** When the cache is dead and the backend is blocked, the web stops silently serving dead links — it flags expiry honestly and offers both the official Microsoft fallback and the web-to-CLI handoff.
**What stays unchanged:** The Go backend keeps serving the React frontend. Eval builds (separate Eval Center endpoint, unaffected) remain a first-class, reliable offering. The two-layer caching system is retained and becomes the distribution layer for crowdsourced links.
```
┌─────────────────────────────────────────┐
│ User visits msdl.tech-latest.com │
└───────────────────┬───────────────────────┘
│
Cached link valid & fresh?
──────────┬──────────────┬──────────
(Yes) │ │ (No / expired / WAF-blocked)
▼ ▼
Serve signed link Graceful degradation:
1. Show "likely expired" flag
2. Official Microsoft fallback (guided)
3. Web-to-CLI handoff command
│
▼
User runs: msdl --id 3262 --lang "..."
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
CLI fetches from MS Prints URL to user POST /contribute
(their residential IP) (immediate result) (warms shared cache)
│
▼
Next web visitor: cache hit
```
---
## 3. Phasing Overview
| Phase | Deliverable | Why this order |
|-------|-------------|----------------|
| **1** | Web graceful degradation (expired flag + official fallback) | Immediate — stops users hitting silent dead links today. Pure frontend + small backend signal. No dependency on the CLI. |
| **2** | CLI rework: direct-to-Microsoft | The core durable fix. Replaces the backend-routed CLI plan. |
| **3** | Crowdsourced cache contribution (`/contribute` + CLI flag) | Depends on Phase 2 CLI existing. Turns every CLI run into cache warming. |
| **4** | Web-to-CLI handoff UX | Depends on Phases 2–3. Wires the website's expired state to the CLI install/command. |
| **5** | Distribution & winget | Ships the CLI to users (covered in detail in the existing TDD plan). |
> The granular, TDD-level, step-by-step build instructions for the CLI binary itself (module scaffold, catalog, picker, flags, release workflow, winget manifests) already live in
> [`docs/superpowers/plans/2026-06-15-msdl-cli.md`](docs/superpowers/plans/2026-06-15-msdl-cli.md).
> **That plan must be amended for Phase 2** (it currently routes through `api.msdl.tech-latest.com`; it must instead port the Microsoft session flow into the binary). This document is the strategic master; that document is the implementation detail.
---
## Phase 1 — Web Graceful Degradation (Immediate)
**Goal:** No user is ever silently handed a dead Microsoft CDN link. When the link is at-risk or unavailable, say so and give a guaranteed path forward.
### 1.1 Backend: expose link freshness in the proxy response
**Files:** `backend/main.go` (`handleProxy`, ~line 817; `linkCacheEntry` line 105)
The frontend already parses `se` for the countdown, but it can't distinguish "fresh from Microsoft" from "stale-on-failure served past a failed refresh." Surface that explicitly.
- Add a response header on `/proxy` responses indicating provenance and confidence:
- `X-MSDL-Link-Status: fresh` — fetched live from Microsoft this request
- `X-MSDL-Link-Status: cached` — served from a valid (not-yet-soft-expired) cache entry
- `X-MSDL-Link-Status: stale` — served via stale-on-failure after a WAF block; **may be expired**
- The `stale` branch is the existing `serving stale` path (the `fetch failed (...), serving stale` log line). Set the header there.
- Also emit `X-MSDL-Link-Expires: <RFC3339>` (the `se`-derived expiry) so the frontend has authoritative metadata for the countdown without re-parsing the URL itself. *(Engg-2 suggestion.)*
- Ensure `enableCORS` (line 708) exposes the headers: add `Access-Control-Expose-Headers: X-MSDL-Link-Status, X-MSDL-Link-Expires`.
### 1.2 Frontend: "likely expired" flag
**Files:** `frontend/src/pages/ProductDetailPage.tsx`
- Read `X-MSDL-Link-Status` from the `/proxy` fetch response.
- When `stale`, render an amber inline warning above the download button:
> ⚠️ This link is cached and may have expired. Microsoft is currently limiting automated requests. Use the options below to get a guaranteed fresh download.
- When `fresh`/`cached`, keep the existing expiry countdown unchanged.
### 1.3 Frontend: official Microsoft fallback (guided)
**Files:** `frontend/src/pages/ProductDetailPage.tsx`, new `frontend/src/components/OfficialFallback.tsx`
- Below the download card, a collapsible "Get it directly from Microsoft" section that:
- Links to the official page (`https://www.microsoft.com/software-download/windows11`, or the matching page per product family).
- Lists the exact dropdown selections the user must make (edition + the language matching their chosen SKU).
- **Data-driven, not hardcoded** *(Engg-2 suggestion):* derive the official-page URL and edition label from the product catalog (`frontend/public/data/products.json`) keyed by product family, and use the SKU's own language string for the language step — so adding a product doesn't require touching the fallback component.
- This keeps MSDL useful as an **information directory** even when automation is fully blocked.
### 1.4 Acceptance
- Trigger a stale serve (point the backend at a WAF-blocked product or mock the stale path); confirm the amber warning + official fallback both render.
- Confirm `fresh`/`cached` responses show no warning and the countdown still works.
- `git commit -m "feat: graceful degradation for WAF-blocked links (expiry flag + official fallback)"`
---
## Phase 2 — CLI Direct-to-Microsoft (Core Durable Fix)
**Goal:** A standalone `msdl` binary that performs the **entire** Microsoft flow locally — session setup, SKU lookup, link fetch — so the request originates from the user's residential IP.
### 2.1 Amend the existing CLI plan
The detailed plan at [`docs/superpowers/plans/2026-06-15-msdl-cli.md`](docs/superpowers/plans/2026-06-15-msdl-cli.md) currently has `cli/api.go` calling `api.msdl.tech-latest.com`. **Replace that approach:** the CLI must own the Microsoft logic. Keep from that plan: module scaffold (Task 1), product catalog (Task 2), interactive picker (Task 4), main flow/flags (Task 5), release workflow + winget (Tasks 6–7). **Rewrite the API layer (Task 3).**
### 2.2 Port the Microsoft session flow into the CLI
**Source (backend):** these functions are the reference implementation to port —
- `setupSession() (string, *simpleCookieJar, error)` — `backend/main.go:394` (vlscppe → ov-df fingerprinting chain + cookie accumulation)
- `simpleCookieJar` — custom `http.CookieJar` with domain scoping disabled
- `fetchSkuInfoFromMS(productID string) ([]byte, int, error)` — `backend/main.go:470`
- `fetchDownloadLinksFromMS(productID, skuID string) ([]byte, error)` — `backend/main.go:566`
**Target (CLI):** create `cli/microsoft.go` with the ported equivalents. Strip server concerns (no shared session map — the CLI creates a fresh session per invocation, which is fine and more browser-like). Reuse the catalog and picker from the existing plan.
> **Shared package vs. copy** *(Engg-2 raised `pkg/microsoft/`):* the CLI is a separate Go module (`cli/go.mod`), so sharing a package across the backend and CLI modules requires a Go workspace or `replace` directives — added friction for ~250 lines of stable code. **Decision: copy/port for v1** (the logic changes rarely; divergence risk is low). Revisit a shared module only if the fingerprinting flow starts changing frequently and the two copies drift.
**New `cli/microsoft.go` responsibilities:**
- `newSession() (*http.Client, string)` — build the `simpleCookieJar`-backed client, run the fingerprinting chain, return the client + session ID.
- `fetchLanguages(client, sessionID, productID) ([]Language, error)` — GET SKU info, parse `Skus`.
- `fetchDownloadLinks(client, sessionID, productID, skuID) ([]DownloadLink, error)` — link fetch (with SKU-info warmup), parse `ProductDownloadOptions`.
- `fetchEvalLinks(evalURL) ([]EvalLink, error)` — scrape the Eval Center page directly and resolve fwlinks.
**Eval builds run directly too.** For consistency and future-proofing, the CLI scrapes the Microsoft Eval Center itself rather than proxying evals through the backend. The Eval Center isn't WAF-blocked today, but "today" is exactly the assumption that just broke for the consumer endpoint — keeping the CLI fully backend-independent means it never inherits a future Eval Center block.
### 2.3 TLS fingerprint hardening (note, not blocking)
Go's default `crypto/tls` ClientHello produces a JA3 fingerprint unlike any real browser. On user residential IPs this is a secondary signal (IP diversity is the primary protection), so ship with stdlib first. **If Sentinel starts fingerprinting the CLI,** swap `net/http`'s transport for [`utls`](https://github.com/refraction-networking/utls) with a Chrome ClientHello profile. Track as a follow-up, not a launch blocker.
### 2.4 Acceptance
- On a residential connection: `msdl --id 3262 --lang "English (United States)"` prints a valid signed URL with no backend involved.
- `msdl --id 3321` (a product the backend currently gets WAF-blocked on) succeeds from a home IP — this is the proof the architecture works.
- Eval builds: `msdl --eval server-2025` scrapes the Eval Center directly and prints the resolved `.iso` URLs — no backend involved.
---
## Phase 3 — Crowdsourced Cache Contribution
**Goal:** Every CLI link fetch optionally warms the shared backend cache, so web visitors get hits without the backend ever calling Microsoft.
### 3.0 Redis/Valkey cache persistence (prerequisite to 3.1)
**Why:** Without persistence, a backend restart wipes the link cache. If the backend is also WAF-blocked it cannot rebuild from Microsoft — crowdsourced contributions would be the only source of fresh links, so losing them on restart defeats Phase 3 entirely.
**Stack:** Redis/Valkey on Coolify (same Hetzner instance). TTL-native, Go has a first-class client (`github.com/redis/go-redis/v9`), zero added ops cost. Preferred over PostgreSQL (overkill for key-value TTL cache) and CF KV (requires routing through the Worker from Go, awkward).
**Architecture:** Two-layer cache — Redis as L2 persistent store; the existing in-memory `linkCache` map stays as L1 for hot reads.
**Files:** `backend/main.go`, `backend/go.mod`
- Add `REDIS_URL` env var (e.g. `redis://localhost:6379`). If unset, skip Redis silently — backend runs in memory-only mode for local dev.
- Key format: `msdl:link:<productID>:<skuID>` (namespaced).
- On startup: seed in-memory cache from Redis (so a restart recovers immediately).
- On link cache write (fresh fetch or contribution): `SET key value EX <ttl_seconds>` using `time.Until(entry.ExpiresAt)`.
- On in-memory cache miss in `handleProxy`: check Redis before calling Microsoft.
- If Redis is unavailable at runtime: log a warning and fall through to in-memory-only (keeps the existing failure behaviour, no hard dependency).
**Deploy order:** Provision a Valkey service on Coolify and set `REDIS_URL` before Phase 3 goes to production. Add `github.com/redis/go-redis/v9` to `backend/go.mod`.
---
### 3.1 Backend: `/contribute` endpoint
**Files:** `backend/main.go` (register in `main()` ~line 1147; cache write mirrors `handleProxy` line 922)
- `POST /contribute` accepting JSON: `{ "product_id": "3262", "sku_id": "19675", "raw_json": <the exact GetProductDownloadLinksBySku response bytes> }`.
- **Validation (reject untrusted contributions):**
1. `product_id` must be in the known catalog (reuse the allow-list logic the proxy uses).
2. Parse `raw_json` with the existing `parseLinkExpiry(rawJSON []byte) time.Time` (line 208). Reject if expiry is in the past or < 1 hour out.
3. Extract the URL(s) from `ProductDownloadOptions`; reject unless the host matches the Microsoft CDN pattern (e.g. `*.download.prss.microsoft.com` / `software-static.download.prss.microsoft.com`).
4. **No `HEAD` validation by default** *(both engineers):* parsing + host-allow-list + expiry are the real defense, and an extra request to Microsoft's CDN on every contribution adds traffic Microsoft may be sensitive to. Rely on parsing first; only add a `HEAD` check later if junk slips through.
- On success, write to the shared cache exactly as `handleProxy` does:
```go
cacheKey := productID + ":" + skuID
expiresAt := parseLinkExpiry(raw)
linkCacheMu.Lock()
linkCache[cacheKey] = linkCacheEntry{RawJSON: raw, ExpiresAt: expiresAt, FetchedAt: time.Now()}
linkCacheMu.Unlock()
```
- Respond `204 No Content` on accept, `400`/`422` on validation failure.
- **Logging** *(Engg-2):* log accepts as `contribute: product_id=%s sku_id=%s -> accepted, cached until %s` AND rejections clearly as `contribute: product_id=%s sku_id=%s -> rejected (<reason>)` so contribution patterns/abuse are visible in ops logs from day one.
- **Abuse protection:** since this writes to a cache served to all users —
- **Per-IP rate limit** on `/contribute`: ~5/min *(Engg-2's concrete number)*. Build it as small middleware (a per-IP token bucket in a `map[string]…` with mutex) — there's no existing rate-limit middleware to reuse.
- **`CONTRIBUTE_SECRET`** env var, baked into released CLI builds, checked as a request header. Cheap obfuscation that blocks casual abuse; not real security.
- The host allow-list + expiry validation remain the real defense: the worst a bad actor can do is submit a *valid Microsoft link*, which is harmless.
### 3.2 CLI: contribution flag
**Files:** `cli/main.go`, `cli/microsoft.go`
- After a successful link fetch, POST the raw response to `https://api.msdl.tech-latest.com/contribute` in the background. **Truly non-blocking** *(Engg-2):* fire it in a goroutine with a `context.WithTimeout(ctx, 5*time.Second)`, send the `CONTRIBUTE_SECRET` as a header, ignore all errors. The user already has their URL printed before this runs — and `main` must wait on it only briefly (or the process may exit first; spawn it before printing the result and `wg.Wait()` with the same short timeout, or simply accept best-effort).
- **Opt-out, transparent by default:** print to stderr `✓ Shared with msdl.tech cache to help other users (disable with --no-contribute)`. Add `--no-contribute` flag. Honor a `MSDL_NO_CONTRIBUTE=1` env var too.
### 3.3 Acceptance
- Run the CLI for a product whose backend cache is empty/expired; confirm `/contribute` logs an accept and a subsequent web request to `/proxy` for that product is a **cache hit** with `X-MSDL-Link-Status: cached`.
- Submit a junk/non-Microsoft URL to `/contribute`; confirm rejection.
- Submit a near-expired link; confirm rejection.
---
## Phase 4 — Web-to-CLI Handoff
**Goal:** When the web cache is dead and the backend is WAF-blocked, the site offers the CLI as the guaranteed path — and that CLI run heals the cache for the next visitor (Phase 3).
### 4.1 Frontend handoff UI
**Files:** `frontend/src/pages/ProductDetailPage.tsx`, `frontend/src/components/CliHandoff.tsx`
- In the `stale` state (from Phase 1.1), in addition to the official fallback, show:
> **Get a fresh link instantly with the MSDL CLI** — runs from your computer, always works.
- Install one-liner per platform (winget for Windows, curl|sh for Linux/macOS — final commands from the release in Phase 5).
- The exact command pre-filled for this product: `msdl --id 3262 --lang "English (United States)"`
- One-line explanation: "Runs the download lookup from your own connection, so Microsoft's rate limits don't apply. It also shares the fresh link back to help other visitors."
- Copy-to-clipboard button (reuse the existing CLI command tab component pattern).
### 4.2 Acceptance
- Force a stale state; confirm the handoff block shows the correct install command + pre-filled product command, and copy works.
---
## Phase 5 — Distribution
Covered in full by [`docs/superpowers/plans/2026-06-15-msdl-cli.md`](docs/superpowers/plans/2026-06-15-msdl-cli.md) Tasks 6–7:
- GitHub Actions release workflow (4-platform binaries, `cli/v*` tag trigger)
- Winget publishing (`starkSV.msdl`) with `wingetcreate` auto-PR on future releases
- `curl | sh` install script for Linux/macOS
No changes needed here beyond ensuring the `--no-contribute` flag and contribute endpoint URL are baked into release builds.
---
## 4. Cleanup / Catalog Note (do ASAP — independent of the above)
- **Product 48 (Windows 8.1 Single Language)** returns `ERROR [502]: no download links found for this SKU` on every attempt — this is a Microsoft-side removal, not a WAF block. **Both reviewers flagged this as do-ASAP.** Remove it from `frontend/public/data/products.json` now (and from `cli/catalog.go` when the CLI is built). This is a one-line config change → straight to main.
---
## 5. Risk & Open Questions
- **Contribution trust model:** the CDN-host allow-list + expiry validation make malicious contributions essentially harmless (worst case: a valid Microsoft link). Revisit if abuse appears.
- **Eval Center longevity:** the CLI scrapes Eval Center directly (decided). If Microsoft changes that page's markup, `fwlinkRe`/`isoLangRe` in `cli/microsoft.go` (and the backend) need updating — low risk, easy fix.
- **TLS fingerprinting (`utls`):** only if Sentinel begins fingerprinting the CLI on residential IPs. Not a launch blocker.
- **Backend's own Microsoft calls:** once crowdsourced contributions are flowing, consider whether the backend should still attempt live fetches at all, or become contribution-only + stale-serving. Likely keep live fetch as a fallback for cold products with no contributor yet.
---
## 6. Sequencing Recommendation
1. **Phase 1 now** — independent, immediate user benefit, low risk.
2. **Catalog cleanup (§4)** — trivial, do alongside Phase 1.
3. **Phase 2** — amend & execute the CLI TDD plan with the direct-to-Microsoft rewrite.
4. **Phase 3** — once the CLI fetches links, add contribution.
5. **Phase 4 + 5** — wire the web handoff and ship distribution together.

38
PROGRESS.md
View file

@ -2,6 +2,44 @@
## Shipped
### CLI + resilience architecture (formerly `IMPLEMENTATION_PLAN.md`, Phases 1–5) — merged
Microsoft's Azure Sentinel WAF started blocking the backend's data-center IP (Hetzner) on
the link endpoint. Rather than fight it server-side, inverted the network footprint: the
CLI now talks to Microsoft directly from the user's own residential IP, and feeds fresh
links back into the shared backend cache.
- **`msdl` CLI (`cli/`)** — standalone Go binary, ports the full Microsoft session flow
(`cli/microsoft.go`: vlscppe permit → ov-df fingerprint → SKU lookup → signed link fetch)
so requests originate from the user's machine, not a blockable ASN. Also scrapes the
Eval Center directly for eval builds — no backend dependency either way.
- **Crowdsourced cache contribution** — after a successful CLI fetch, the raw Microsoft
response is POSTed to the backend's `POST /contribute` (validated against a product
allow-list, CDN-host allow-list, and expiry check; rate-limited per IP; gated by
`CONTRIBUTE_SECRET`). Opt out with `--no-contribute` or `MSDL_NO_CONTRIBUTE=1`.
- **Redis/Valkey L2 cache** — `REDIS_URL` env var; in-memory cache is seeded from Redis on
startup and written through on every fresh fetch, so a backend restart no longer loses
crowdsourced links. Falls back to memory-only mode silently if unset/unreachable.
Added `github.com/redis/go-redis/v9`.
- **Web graceful degradation** — `/proxy` responses carry `X-MSDL-Link-Status`
(`fresh` / `cached` / `stale`) and `X-MSDL-Link-Expires` headers. A stale (WAF-blocked
refresh) response now shows an explicit warning plus an `OfficialFallback` component
(data-driven from `products.json`, links to Microsoft's own download page) instead of
silently handing out a possibly-dead link.
- **CLI handoff UI** — `CliHandoff` component on the product page offers the install
one-liner + pre-filled `msdl` command as a guaranteed-fresh alternative when the cached
link is stale.
- **CLI telemetry** — `GET /cli/version` (update check) and `POST /telemetry` (anonymous
action/platform/version counts, no personal data) endpoints; opt out with
`--no-telemetry` / `MSDL_NO_TELEMETRY=1`. Counts surfaced in `/metrics`.
- **Catalog cleanup** — removed product `48` (Windows 8.1 Single Language), which
Microsoft had already pulled server-side (`ERROR [502]: no download links found`).
- **Distribution** — GitHub Actions release workflow (4-platform binaries on `cli/vX.Y.Z`
tags), winget manifest (`starkSV.msdl`), Homebrew tap (`msdl-cli` formula), AUR package
(`msdl-bin`), and a `curl | bash` installer for Linux/macOS without Homebrew.
---
### eval ISOs — PR #13 `feat/evalcenter-isos`
**Windows Server 2016–2025 + Windows 11 Enterprise evaluation editions**

42
issues.md
View file

@ -1,42 +0,0 @@
# Issue Drafts
## Inactive products still trigger backend requests
### Summary
The new `active: false` catalog state is not fully enforced on the product detail page.
Inactive products still pass validation and trigger `/skuinfo` requests before the UI shows the discontinued-product panel.
### Affected file
- `frontend/src/pages/ProductDetailPage.tsx`
### Current behavior
For a catalog entry with `active: false`:
- the page loads the product from `products.json`
- `setIsValidated(true)` still runs
- the follow-up effect calls `/skuinfo?product_id=...`
- the backend and Microsoft endpoints are still hit
- only afterward does the page render the `Product Discontinued` state
### Expected behavior
Inactive products should fail fast on the frontend and should not trigger backend or Microsoft API requests.
### Risk
The inactive-product flow still consumes backend capacity and external request budget even though the user cannot download that release.
### Suggested fix
- gate validation on `product.active !== false`
- avoid setting `isValidated` for inactive products
- ensure the `/skuinfo` effect only runs for active products
### Prevention
- add a test covering an inactive catalog entry and assert that no backend request is made
- model the route state explicitly so `active`, `not_found`, and `error` paths cannot fall through into fetch logic