Version bump: backend/main.go latestCLIVersion and the local
winget/manifests/*.yaml mirror to 0.3.7, with the real
InstallerUrl/InstallerSha256 for the cli/v0.3.7 release asset.
CI fix: the first real winget-submission attempt (v0.3.7, now that
WINGET_TOKEN is finally set) failed immediately -- wingetcreate isn't
published as a NuGet/dotnet-tool package at all, it's a native Windows
binary requiring .NET 6 + VC++ Redistributable. `dotnet tool install
--global wingetcreate` on ubuntu-latest could never have worked; this
path was just never exercised before since the empty-token check
always short-circuited it first. Split winget submission into its own
job on windows-latest, downloading wingetcreate.exe directly from its
GitHub release instead of dotnet tool install.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Log analysis over 17 days showed 181 Sentinel-block retry attempts on
/proxy, all 181 failed (0% success). Retrying every 90 minutes yields
zero value right now and just adds more blocked-request noise against
our own IP for nothing in return. Bumped lockdownTTL to 5h so it backs
off harder while still recovering same-day if Microsoft's block ever
eases.
Also found /proxy never validated product_id against the known
catalog, so a request for a nonexistent ID (e.g. product_id=2861,
never a real product) still burned a full outbound Microsoft session
attempt and a Sentinel-block hit for something that could never
succeed anyway. Now rejected with 404 before any Microsoft call,
reusing the existing validContributeProducts allow-list.
Verified locally: unknown product_id -> 404 (no MS fetch attempted in
logs), missing params -> 400 unchanged, known product_id -> proceeds
to a real MS session attempt as before.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Moves msdls_v3.py from repo root into scripts/check-new-releases/,
alongside a new Go tool, documented as Method 1 (Python brute-force
range scan, can write results to a catalog-shaped JSON) and Method 2
(Go auto-discovery via page-scrape + bounded adjacent-ID probe, no
range-guessing needed). Only Windows 11 is checked -- 8.1 is fully
frozen and Windows 10 is past end-of-life, so neither will produce a
new consumer ISO again.
Fixed msdls_v3.py's release-name extraction: it checked EditionName/
ReleaseName/FriendlyName, none of which exist in the real API
response -- the actual field is ProductDisplayName, confirmed live.
Also added the vlscppe session-permit call to setup_session(), which
was skipped entirely; higher-risk gap for this script's brute-force
range-scan usage pattern than for a single lookup.
Both tools' live runs surfaced a real catalog gap: product IDs
3322/3323/3325/3326 (Home/Pro China variants of the 25H2 "V2" refresh)
exist on Microsoft's side but weren't in our catalog. Added them to
cli/catalog.go, products.json, validContributeProducts, and
sitemap.xml, and renamed 3321/3324 from our own guessed "(Updated
Oct)" label to Microsoft's actual name for this refresh, "(V2)"
(their internal name is "Windows 11 25H2__V2"; cleaned up the
double-underscore for user-facing display).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a GET /install.sh endpoint on the backend (embedded via Go's
embed package, so the script stays a real, shellcheck-able .sh file
rather than a Go string literal). Auto-detects OS/arch (including
Termux on Android via $PREFIX) and always resolves the latest GitHub
release via the /releases/latest/download/ redirect, so unlike the
winget/brew/AUR manifests it needs no per-release maintenance.
Verified end-to-end in an ephemeral Ubuntu container (both the normal
/usr/local/bin path and the Termux $PREFIX path), and locally against
a running backend instance.
Documented alongside the existing winget/Homebrew instructions in
README.md, the release notes template, the web CLI page, and the
per-product CliHandoff card.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Matches the shape actually asked for and how every other telemetry
counter in this file already works (all-time cumulative via HIncrBy,
no expiry, no per-day key rotation): /metrics now exposes flat
sentinel_errors and sentinel_distinct_sources_est fields instead of a
nested, date-scoped sentinel_today object.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CLI telemetry showed a growing count of Sentinel rejections but no way
to tell whether they come from one user retrying repeatedly or many
distinct users independently getting blocked.
Adds a per-day Redis HyperLogLog (msdl:telemetry:sentinel_hll:<date>)
seeded from the reporting IP whenever a CLI error contains "Sentinel",
plus a parallel daily counter. /metrics now exposes sentinel_today:
{date, errors_today, distinct_sources_est}. No raw IPs are stored --
HLL is a one-way cardinality estimator, and both keys expire after 48h.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Sentinel lockdown gate served any linkCache hit as "stale"
regardless of whether the entry was still within its own TTL. A link
just contributed via CLI (still valid for ~24h) would get labeled
stale, causing the frontend to hide it and show the CLI-only banner
again instead of the working link it just received.
Now checks the entry's actual expiry: still-fresh entries are labeled
"cached" (frontend shows the link normally); only genuinely expired
entries are labeled "stale".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Failure telemetry only recorded success=false with no context. Adds
a truncated error string (150 chars, CLI-side) forwarded to a new
msdl:telemetry:errors Redis hash so failure causes are diagnosable
from /metrics instead of guessed at.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Microsoft's 403 capacity page was logging hundreds of lines of HTML
per error. Both non-200 error paths now cap the body at 120 chars.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backend (main.go):
- negCacheEntry gains IsSentinel bool; lockdownTTL = 90min constant
- Lockdown gate at top of handleProxy: serves stale with X-MSDL-Lockdown:1
header or 429 {"error":"...","lockdown":true} — zero MS API calls made
- Sentinel stale path: sets 90min lockdown neg cache, adds lockdown header,
suppresses background refresh (retrying extends the Sentinel block timer)
- Non-Sentinel stale path: unchanged (60s neg cache, bg refresh fires)
- Expose X-MSDL-Lockdown in Access-Control-Expose-Headers
Frontend (ProductDetailPage.tsx):
- isLockdown state: set from X-MSDL-Lockdown header or data.lockdown field
- Stale state hides expired download links entirely; shows CLI recommendation
inside the card instead of sending users to look for options elsewhere
- Lockdown: red banner explaining 90min cooldown + highlighted CLI card,
no Refresh button (it cannot help while Sentinel block is active)
- Normal stale: amber banner + highlighted CLI card + secondary Refresh button
- Outer CliHandoff suppressed when already shown inside card (stale states)
Frontend (CliHandoff.tsx):
- CLI command uses msdl.exe (primary audience is Windows users)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Microsoft's WAF returns HTTP 200 with {"Errors":[{"Value":"Sentinel marked
this request as rejected."}]}. The existing Sentinel check only ran for
non-200 responses, so the session was never evicted — every subsequent
request reused the poisoned session and hit Sentinel again.
Add the same eviction + rateLimitError logic inside the errMap["Value"]
branch so the session is evicted regardless of the WAF's HTTP status code.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add platform allowlist validation (windows, darwin, linux) and length caps for version (≤20) and product_id (≤10) in handleTelemetry to prevent unbounded Redis hash growth from attacker-controlled field values.
- Add HTTP method guard to handleCLIVersion to reject non-GET requests with 405 Method Not Allowed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- seedMetricsFromRedis: loads persisted counters into atomics on startup
- flushMetricsToRedis: writes all 12 counters to 3 Redis hashes on demand
- startMetricsFlusher: goroutine that flushes every 5 minutes
- Graceful shutdown: SIGINT/SIGTERM flushes metrics then drains HTTP in 10s
- http.ListenAndServe replaced with http.Server for Shutdown support
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When Microsoft Sentinel rejects a request (HTTP 502 with 'Sentinel'
in body), the session was being reused on every retry — hammering the
same poisoned session and making the block worse.
Now: detect Sentinel response, evict the session from cache, and
return a rateLimitError so stale-on-failure kicks in and users get
the last cached link instead of an error.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
parseLinkExpiry has a 1h safety floor that causes it to treat already-
expired URLs as valid. Replace with extractRawExpiry + direct se
timestamp comparison so any URL expiring < 1h out is rejected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Redis L2 persistent cache: initRedis() from REDIS_URL env, seed
in-memory linkCache on startup, write-through on every fresh fetch
and background refresh (silent fallback to memory-only if unavailable)
- POST /contribute endpoint: validates product catalog, expiry > 1h,
Microsoft CDN host allow-list, per-IP token bucket (~5/min),
CONTRIBUTE_SECRET header; writes to in-memory + Redis; logs all
accepts and rejections
- CLI --no-contribute flag (also MSDL_NO_CONTRIBUTE=1 env var):
background goroutine posts raw MS JSON to /contribute after every
consumer fetch; best-effort 5s timeout; opt-out message to stderr
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Backend: emit X-MSDL-Link-Status (fresh/cached/stale) and
X-MSDL-Link-Expires headers on every /proxy response; expose
both via Access-Control-Expose-Headers
- Frontend: read headers in handleGetLinks/handleRefreshLinks;
show amber stale warning when backend is WAF-blocked
- Frontend: new OfficialFallback component (collapsible) with
step-by-step guide to Microsoft's official download page,
pre-filled with the user's selected language; shown on stale
- Plan: add Redis/Valkey persistence section to Phase 3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Temporary diagnostic — tells us whether Set-Cookie headers from
Microsoft are actually reaching the backend through the CF Worker.
If count is 0, the Worker is stripping them or MS sets cookies via JS.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sentinel started rejecting all /proxy requests because cookies set during
the fingerprinting flow (vlscppe/ov-df) were never reaching the
GetProductDownloadLinksBySku call.
Two gaps closed:
- CF Worker now forwards Cookie header to Microsoft and Set-Cookie headers
back to the backend
- Backend uses a simpleCookieJar (domain-scoping disabled) shared across
the full session: setup → warmup → download link fetch. Standard
cookiejar rejects microsoft.com cookies when the request URL is the
Worker domain.
Tested locally without Worker (direct MS calls): download links returned,
no Sentinel error.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Backend: ?force=true on /proxy bypasses cache for fresh link fetch
- Frontend: parse `se` param for live expiry countdown on download links
- Frontend: show Refresh button when link expires in <6h, force-fetches fresh URL
- Frontend: CliCommand component replaces Aria2Tip — wget/curl/aria2 tabs, persists selection in localStorage
- Frontend: RecentlyViewed component — localStorage, shows active countdown or expired state
- Frontend: FileSize shown alongside arch if returned by Microsoft CDN
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Background refresh on stale serve now goes through sfGroup.Do so
concurrent stale hits collapse into one Microsoft call instead of
hammering the API from multiple goroutines simultaneously.
- cleanupCaches: only logs when entries are actually evicted (was
logging every 30min regardless of whether anything changed).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- /metrics: atomic counters for requests, cache hits, MS fetches, neg
hits, stale serves, and cache sizes per endpoint. Auth via
METRICS_SECRET env var (query param or Authorization: Bearer).
- Cache eviction: cleanupCaches() goroutine runs every 30min, removes
expired SKU/neg entries and link entries past ExpiresAt + 4h stale
window. Prevents unbounded map growth under sustained traffic.
- rand: replace global rand with package-level rand.Rand to avoid
mutex contention at high QPS (jitter() uses local source).
- Singleflight keys: separate sfKey from negKey in both handlers for
clarity — "sku:<id>" and "link:<id>:<sku>" vs neg cache keys.
- Background refresh on stale: when /proxy serves a stale entry after
a failed refresh, fires a goroutine to retry and update the cache
so the next request gets fresh data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reduces Microsoft API traffic from thousands/day to ~50-100/day,
eliminating the 715-123130 rate-limit block under real traffic.
1. Singleflight (golang.org/x/sync/singleflight) — wraps all Microsoft
fetches so 500 concurrent cache misses collapse into 1 API hit.
2. SKU info cache — 7-day TTL keyed by product_id. Language lists are
stable; no need to hit Microsoft on every product page load.
3. Download link cache — dynamic TTL keyed by product_id:sku_id. Expiry
is derived from the signed URL's `se` query param minus 30min buffer,
so we never serve an expired signed URL.
4. Negative response caching — 60s TTL for 429 / 715-123130 failures.
Prevents thundering herd from retries worsening an existing block.
Only stored when no stale data is available to fall back on.
5. Dynamic TTL — parses `se` (signed expiry) from Microsoft CDN URL:
TTL = (se - now) - 30min. Falls back to 22h if unparseable.
6. Stale-on-failure — if a refresh fails (rate-limited or transient),
the expired cache entry is served temporarily. Applies to both
/proxy (download links) and /evallinks (eval ISOs).
7. Jitter — ±5min random offset on all TTLs prevents synchronised
mass-expiry spikes when many entries are warmed at the same time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds /eval listing page and /product/<slug> detail pages for Windows
Server 2025/2022/2019/2016 and Windows 11 Enterprise evaluation ISOs,
sourced from Microsoft evalcenter without any session flow.
Backend: new /evallinks endpoint fetches evalcenter page, extracts
fwlinks, follows all redirects in parallel, detects arch + locale.
24h cache per product with startup warming so first hit is instant.
Frontend: EvalPage listing at /eval, EvalDetailPage at /product/<slug>
matching consumer structure with downloads, aria2 template, system
requirements, and also-available section. Smart ProductRouter in
App.tsx dispatches slug vs numeric productId. Dock highlights Products
tab for /product/* and /eval routes. CTA banner on home page and
footer link added. sitemap.xml updated with 6 new URLs.
Closes#13
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replaced strconv.Atoi on sku_id with a permissive allowlist regex
(alphanumeric, hyphens, underscores). Older products like Windows 8.1
(#48, #52) return non-integer SKU IDs from Microsoft API, causing the
Get Download Links button to fail with 'sku_id must be a numeric value'.
Closes#9
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add cloudflare-worker/worker.js: edge proxy forwarding requests to allowed
Microsoft hosts from Cloudflare distributed IPs. Set CF_WORKER_URL env var
to activate; omit to keep direct-to-Microsoft behaviour (self-hosters unaffected).
- Wrap all outbound Microsoft URLs in proxyURL() in main.go so the Worker
sits transparently in front of every session and download-link request.
- Fix Back button navigating to external referrer when user lands directly
on a product URL (#5) — use window.history.state?.idx to detect whether
there is prior in-app history before calling navigate(-1).
Closes#5, Closes#6
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Reject non-numeric product_id and sku_id with HTTP 400 before any
upstream request is made, preventing parameter injection into
Microsoft API query strings.
- Replace all fmt.Sprintf URL construction with url.Values.Set +
.Encode() across handleSkuInfo, handleProxy, and setupSession,
ensuring values are always percent-encoded correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Migrated products.json to structured metadata format.
- Refactored ProductDetailPage to eagerly reset state and validate product IDs against local catalog.
- Dynamic data fetching in HomePage and StatsBar for total release counts.
- Adjusted UI layout in App.tsx and UX states (404/Loading) for missing dependencies.
- Shortened backend tracking HTTP timeouts to prevent local Pi-Hole hangs.
