Merge pull request #40 from starkSV/feat/cli-tls-fingerprint-hardening
feat(cli): TLS/HTTP2 fingerprint hardening via tls-client
This commit is contained in:
Shekhar
GitHub
commit
cf318eb15c
9 changed files with 363 additions and 2763 deletions
35
CLAUDE.md
35
CLAUDE.md
|
|
@ -37,6 +37,41 @@ negative cache (60s), dynamic TTL, stale-on-failure, jitter. Verified in product
|
|||
- [x] **Recently viewed** — localStorage only. Shows on homepage. Consumer + eval pages both tracked. Expired state shown when link expiry known.
|
||||
- ~~**File size**~~ — not feasible. Microsoft CDN does not return file size in the API response.
|
||||
|
||||
### Sentinel WAF resilience (ongoing)
|
||||
|
||||
Confirmed via three weekly `/metrics` + docker-log checkpoints (2026-07-14 → 07-21 → 07-31):
|
||||
the backend's own direct link-fetch (`/proxy` → Microsoft) is **100% blocked**, permanently —
|
||||
`link.ms_fetches: 0` every time, and zero even-attempted-and-failed fetches in the raw logs
|
||||
(the lockdown gate short-circuits before reaching that code path). The site stays alive
|
||||
entirely on cached/stale entries plus CLI-contributed links (286 accepted, 0 rejected,
|
||||
confirmed 2026-07-31). `/skuinfo` and `/evallinks` are unaffected — only the download-link
|
||||
endpoint is targeted.
|
||||
|
||||
The CLI's own residential-IP requests also see a stable ~20% Sentinel rejection rate across
|
||||
all three checkpoints — not improving, not worsening. Microsoft's own API error response
|
||||
literally names the system (`"Sentinel marked this request as rejected."`, `Type: 9` in
|
||||
`Errors[]`), confirming it's a real, named product, not our guess. The clean structured-JSON
|
||||
deny (not an HTML/JS challenge page) suggests a signature/reputation gate rather than full
|
||||
interactive bot-management — TLS ClientHello fingerprinting is a plausible contributing
|
||||
signal, since Go's stdlib `crypto/tls` doesn't look like any real browser, independent of IP.
|
||||
|
||||
- [x] **Back off Sentinel retries** — bumped `lockdownTTL` 90min → 5h (`backend/main.go`).
|
||||
181 retries over 17 days, 0 successes; retrying that often was pure noise.
|
||||
(fix/sentinel-lockdown-and-proxy-validation, merged)
|
||||
- [x] **Validate `product_id` in `/proxy`** — was passing unknown IDs straight through to a
|
||||
real Microsoft session attempt (found via a stray `product_id=2861`, never a real
|
||||
product, in the logs). Now rejected with 404 before any outbound call. (same PR, merged)
|
||||
- [ ] **CLI TLS/HTTP2 fingerprint hardening** — swapped the CLI's transport from stdlib
|
||||
`net/http` to `github.com/bogdanfinn/tls-client` (wraps `utls` with a maintained Chrome
|
||||
profile). Verified functionally correct (real link fetched + contributed end-to-end),
|
||||
but does NOT yet prove the fingerprint theory — the old client already succeeded ~80%
|
||||
of the time. Need to watch the Sentinel-rejection-rate telemetry over a comparable
|
||||
multi-day window post-merge. (feat/cli-tls-fingerprint-hardening, not yet merged)
|
||||
- [ ] **`/needs-warming` community page** — proposed, not built. Surfaces products currently
|
||||
failing web users (active Sentinel/rate-limit lockdown, no cached/stale link available)
|
||||
with a one-click CLI command to fix it. See
|
||||
`docs/superpowers/specs/2026-07-13-needs-warming-design.md`.
|
||||
|
||||
### Known bugs / open issues
|
||||
|
||||
- [x] **`_redirects` Cloudflare Pages** — investigated 2026-07-21. The rule was indeed ignored
|
||||
|
|
|
|||
46
PROGRESS.md
46
PROGRESS.md
|
|
@ -2,6 +2,51 @@
|
|||
|
||||
## Shipped
|
||||
|
||||
### Sentinel WAF investigation & resilience hardening — backend merged, CLI fix pending
|
||||
|
||||
Three weekly `/metrics` + raw docker-log checkpoints (2026-07-14 → 07-21 → 07-31) turned the
|
||||
"Microsoft blocks our server IP" assumption from a one-off observation into a confirmed,
|
||||
stable fact — and revealed the crowdsourcing architecture below is now doing all the real work.
|
||||
|
||||
**What the data actually showed:**
|
||||
- `link.ms_fetches: 0` at every checkpoint — the backend's own direct link-fetch to Microsoft
|
||||
has not succeeded once, ever, across 17+ days.
|
||||
- Raw docker logs showed not just zero successes but zero *attempted-and-failed* fetches
|
||||
either — the existing Sentinel lockdown gate short-circuits the request before it even
|
||||
reaches Microsoft, serving cache/stale silently instead. 181 blocked session attempts over
|
||||
the same window, all 181 unsuccessful.
|
||||
- The site stayed online anyway: 286 CLI contributions accepted, 0 rejected, in that same
|
||||
window — confirming the crowdsourced cache from the CLI (see architecture entry below) is
|
||||
the thing actually keeping links fresh now, not the backend's own fetch.
|
||||
- `/skuinfo` (125 successful Microsoft fetches) and `/evallinks` (126 cache hits, 0 failures)
|
||||
are both unaffected — Sentinel targets the download-link endpoint specifically.
|
||||
- The CLI's own residential-IP requests held a stable ~20% Sentinel-rejection rate across all
|
||||
three checkpoints too — not improving, not worsening, despite running from IPs that
|
||||
shouldn't trip ASN-based blocking. Traced the exact error text
|
||||
(`"Sentinel marked this request as rejected."`) to Microsoft's own API response
|
||||
(`Errors[0].Value` with `Type: 9`) — confirming "Sentinel" is Microsoft's real name for this
|
||||
system, not our guess, and that it returns a clean structured JSON deny rather than an
|
||||
HTML/JS challenge page (suggesting a signature/reputation gate rather than full interactive
|
||||
bot management).
|
||||
|
||||
**Backend fixes (`fix/sentinel-lockdown-and-proxy-validation`, merged):**
|
||||
- `lockdownTTL` bumped 90min → 5h. 181 retries, 0 successes — retrying that often was pure
|
||||
noise against Microsoft (and our own IP's reputation) for zero return.
|
||||
- `/proxy` now validates `product_id` against the existing catalog allow-list before
|
||||
attempting a Microsoft session. Found via a stray `product_id=2861` (never a real product)
|
||||
in the logs — the backend was burning a full outbound attempt and a Sentinel-block hit on
|
||||
IDs that could never succeed anyway.
|
||||
|
||||
**CLI fix (`feat/cli-tls-fingerprint-hardening`, not yet merged):** swapped the CLI's HTTP
|
||||
transport from stdlib `net/http` to `github.com/bogdanfinn/tls-client` (wraps `utls` with a
|
||||
maintained Chrome TLS+HTTP2 fingerprint), on the theory that Go's default TLS handshake is
|
||||
itself a detectable non-browser signal, independent of IP. Verified functionally correct
|
||||
end-to-end (real link fetched and contributed), but that does *not* prove the theory — the
|
||||
old client already succeeded ~80% of the time. Real validation is watching the
|
||||
Sentinel-rejection-rate telemetry over a comparable multi-day window post-merge.
|
||||
|
||||
---
|
||||
|
||||
### CLI + resilience architecture (formerly `IMPLEMENTATION_PLAN.md`, Phases 1–5) — merged
|
||||
|
||||
Microsoft's Azure Sentinel WAF started blocking the backend's data-center IP (Hetzner) on
|
||||
|
|
@ -120,4 +165,3 @@ CF Worker ensures Hetzner IP is never exposed to Microsoft — rate-limit block
|
|||
| Item | Notes |
|
||||
|---|---|
|
||||
| Per-IP / per-product rate limiter | Revisit after 1 month of production traffic data |
|
||||
| `_redirects` Cloudflare Pages bug | SPA fallback rule flagged as infinite loop, may cause 404s on direct nav |
|
||||
|
|
|
|||
23
cli/go.mod
23
cli/go.mod
|
|
@ -1,3 +1,24 @@
|
|||
module github.com/starkSV/msdl-cli
|
||||
|
||||
go 1.21
|
||||
go 1.24.1
|
||||
|
||||
require (
|
||||
github.com/bogdanfinn/fhttp v0.6.8
|
||||
github.com/bogdanfinn/tls-client v1.15.1
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/andybalholm/brotli v1.2.0 // indirect
|
||||
github.com/bdandy/go-errors v1.2.2 // indirect
|
||||
github.com/bdandy/go-socks4 v1.2.3 // indirect
|
||||
github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect
|
||||
github.com/bogdanfinn/utls v1.7.7-barnius // indirect
|
||||
github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
|
||||
github.com/klauspost/compress v1.18.2 // indirect
|
||||
github.com/quic-go/qpack v0.6.0 // indirect
|
||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
|
||||
golang.org/x/crypto v0.46.0 // indirect
|
||||
golang.org/x/net v0.48.0 // indirect
|
||||
golang.org/x/sys v0.39.0 // indirect
|
||||
golang.org/x/text v0.32.0 // indirect
|
||||
)
|
||||
|
|
|
|||
48
cli/go.sum
Normal file
48
cli/go.sum
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
|
||||
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/bdandy/go-errors v1.2.2 h1:WdFv/oukjTJCLa79UfkGmwX7ZxONAihKu4V0mLIs11Q=
|
||||
github.com/bdandy/go-errors v1.2.2/go.mod h1:NkYHl4Fey9oRRdbB1CoC6e84tuqQHiqrOcZpqFEkBxM=
|
||||
github.com/bdandy/go-socks4 v1.2.3 h1:Q6Y2heY1GRjCtHbmlKfnwrKVU/k81LS8mRGLRlmDlic=
|
||||
github.com/bdandy/go-socks4 v1.2.3/go.mod h1:98kiVFgpdogR8aIGLWLvjDVZ8XcKPsSI/ypGrO+bqHI=
|
||||
github.com/bogdanfinn/fhttp v0.6.8 h1:LiQyHOY3i0QoxxNB7nq27/nGNNbtPj0fuBPozhR7Ws4=
|
||||
github.com/bogdanfinn/fhttp v0.6.8/go.mod h1:A+EKDzMx2hb4IUbMx4TlkoHnaJEiLl8r/1Ss1Y+5e5M=
|
||||
github.com/bogdanfinn/quic-go-utls v1.0.9-utls h1:tV6eDEiRbRCcepALSzxR94JUVD3N3ACIiRLgyc2Ep8s=
|
||||
github.com/bogdanfinn/quic-go-utls v1.0.9-utls/go.mod h1:aHph9B9H9yPOt5xnhWKSOum27DJAqpiHzwX+gjvaXcg=
|
||||
github.com/bogdanfinn/tls-client v1.15.1 h1:KiFAlED55DJ8Fcocn+/1nX6PrDFcttIHAf/GDkV6KN8=
|
||||
github.com/bogdanfinn/tls-client v1.15.1/go.mod h1:LsU6mXVn8MOFDwTkyRfI7V1BZM1p0wf2ZfZsICW/1fM=
|
||||
github.com/bogdanfinn/utls v1.7.7-barnius h1:OuJ497cc7F3yKNVHRsYPQdGggmk5x6+V5ZlrCR7fOLU=
|
||||
github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg=
|
||||
github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI=
|
||||
github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
|
||||
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
|
||||
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
|
||||
github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng=
|
||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||
go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
|
||||
go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
|
||||
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
|
||||
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
|
||||
golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
|
||||
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
|
||||
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
|
||||
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
|
||||
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
|
@ -7,17 +7,25 @@ import (
|
|||
"fmt"
|
||||
"html"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
http "github.com/bogdanfinn/fhttp"
|
||||
tls_client "github.com/bogdanfinn/tls-client"
|
||||
"github.com/bogdanfinn/tls-client/profiles"
|
||||
)
|
||||
|
||||
// msTLSProfile is the browser TLS/HTTP2 fingerprint the CLI presents to Microsoft.
|
||||
// Kept in step with msUA's claimed Chrome version -- a mismatched UA vs. TLS
|
||||
// fingerprint is itself a detectable signal, so if one changes, change both.
|
||||
var msTLSProfile = profiles.Chrome_133
|
||||
|
||||
const (
|
||||
msUA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
|
||||
msUA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
|
||||
msProfile = "606624d44113"
|
||||
msLocale = "en-US"
|
||||
msOrgID = "y6jn8c31"
|
||||
|
|
@ -47,37 +55,6 @@ type EvalLink struct {
|
|||
URL string
|
||||
}
|
||||
|
||||
type simpleCookieJar struct {
|
||||
mu sync.Mutex
|
||||
cookies []*http.Cookie
|
||||
}
|
||||
|
||||
func (j *simpleCookieJar) SetCookies(_ *url.URL, cookies []*http.Cookie) {
|
||||
j.mu.Lock()
|
||||
defer j.mu.Unlock()
|
||||
for _, c := range cookies {
|
||||
found := false
|
||||
for i, existing := range j.cookies {
|
||||
if existing.Name == c.Name {
|
||||
j.cookies[i] = c
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
j.cookies = append(j.cookies, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (j *simpleCookieJar) Cookies(_ *url.URL) []*http.Cookie {
|
||||
j.mu.Lock()
|
||||
defer j.mu.Unlock()
|
||||
out := make([]*http.Cookie, len(j.cookies))
|
||||
copy(out, j.cookies)
|
||||
return out
|
||||
}
|
||||
|
||||
func newSessionID() string {
|
||||
b := make([]byte, 16)
|
||||
rand.Read(b)
|
||||
|
|
@ -116,10 +93,16 @@ func mapDownloadType(n int) string {
|
|||
}
|
||||
}
|
||||
|
||||
func newSession() (*http.Client, string) {
|
||||
func newSession() (tls_client.HttpClient, string) {
|
||||
sessionID := newSessionID()
|
||||
jar := &simpleCookieJar{}
|
||||
client := &http.Client{Timeout: 15 * time.Second, Jar: jar}
|
||||
client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(),
|
||||
tls_client.WithTimeoutSeconds(15),
|
||||
tls_client.WithClientProfile(msTLSProfile),
|
||||
tls_client.WithCookieJar(tls_client.NewCookieJar()),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, sessionID
|
||||
}
|
||||
|
||||
q1 := url.Values{}
|
||||
q1.Set("org_id", msOrgID)
|
||||
|
|
@ -158,7 +141,7 @@ func newSession() (*http.Client, string) {
|
|||
return client, sessionID
|
||||
}
|
||||
|
||||
func msGet(client *http.Client, reqURL, productID string) ([]byte, error) {
|
||||
func msGet(client tls_client.HttpClient, reqURL, productID string) ([]byte, error) {
|
||||
req, _ := http.NewRequest("GET", reqURL, nil)
|
||||
req.Header.Set("User-Agent", msUA)
|
||||
req.Header.Set("Referer", referer(productID))
|
||||
|
|
@ -249,7 +232,7 @@ func parseDownloadLinks(raw []byte) ([]DownloadLink, error) {
|
|||
return links, nil
|
||||
}
|
||||
|
||||
func fetchLanguages(client *http.Client, sessionID, productID string) ([]Language, error) {
|
||||
func fetchLanguages(client tls_client.HttpClient, sessionID, productID string) ([]Language, error) {
|
||||
q := url.Values{}
|
||||
q.Set("profile", msProfile)
|
||||
q.Set("productEditionId", productID)
|
||||
|
|
@ -267,7 +250,7 @@ func fetchLanguages(client *http.Client, sessionID, productID string) ([]Languag
|
|||
}
|
||||
|
||||
// fetchDownloadLinks returns parsed links and the raw Microsoft JSON (for cache contribution).
|
||||
func fetchDownloadLinks(client *http.Client, sessionID, productID, skuID string) ([]DownloadLink, []byte, error) {
|
||||
func fetchDownloadLinks(client tls_client.HttpClient, sessionID, productID, skuID string) ([]DownloadLink, []byte, error) {
|
||||
wq := url.Values{}
|
||||
wq.Set("profile", msProfile)
|
||||
wq.Set("productEditionId", productID)
|
||||
|
|
@ -335,7 +318,13 @@ func detectLang(rawURL string) string {
|
|||
}
|
||||
|
||||
func fetchEvalLinks(evalURL string) ([]EvalLink, error) {
|
||||
client := &http.Client{Timeout: 20 * time.Second}
|
||||
client, err := tls_client.NewHttpClient(tls_client.NewNoopLogger(),
|
||||
tls_client.WithTimeoutSeconds(20),
|
||||
tls_client.WithClientProfile(msTLSProfile),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating http client: %w", err)
|
||||
}
|
||||
req, _ := http.NewRequest("GET", evalURL, nil)
|
||||
req.Header.Set("User-Agent", msUA)
|
||||
req.Header.Set("Accept", "text/html,application/xhtml+xml")
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,200 +0,0 @@
|
|||
# CLI Telemetry, Update Check & Help — Design Spec
|
||||
|
||||
**Date:** 2026-06-21
|
||||
**Status:** Approved
|
||||
**Scope:** CLI (cli/) + Backend (backend/main.go)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Three related additions to the msdl CLI and backend:
|
||||
|
||||
1. **Telemetry** — anonymous usage counters sent on every CLI run, stored durably in Redis
|
||||
2. **Update check** — CLI checks our backend for the latest version on every run, notifies if behind
|
||||
3. **Help** — `msdl --help` / `-h` prints usage with all flags and examples
|
||||
|
||||
No opt-in required. `MSDL_NO_TELEMETRY=1` skips both telemetry and update check silently.
|
||||
Disclosed in README and `/cli` page with one line each.
|
||||
|
||||
---
|
||||
|
||||
## Backend Changes
|
||||
|
||||
### 1. New constant: `latestCLIVersion`
|
||||
|
||||
```go
|
||||
const latestCLIVersion = "0.3.0"
|
||||
```
|
||||
|
||||
Bumped manually in `backend/main.go` whenever a new CLI release is cut.
|
||||
|
||||
### 2. `GET /cli/version` — public, no auth
|
||||
|
||||
Returns the latest CLI version. The CLI calls this on every run.
|
||||
|
||||
```json
|
||||
{ "latest": "0.3.0" }
|
||||
```
|
||||
|
||||
No rate limiting needed — response is trivial and cacheable by the client.
|
||||
|
||||
### 3. `POST /telemetry` — public, no auth
|
||||
|
||||
Accepts one event per CLI run. Body:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "fetch" | "eval" | "list" | "interactive",
|
||||
"product_id": "2618",
|
||||
"eval_slug": "server-2025",
|
||||
"platform": "windows" | "darwin" | "linux",
|
||||
"version": "0.3.0",
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
- `product_id` present only for `fetch` action
|
||||
- `eval_slug` present only for `eval` action
|
||||
- All other fields always present
|
||||
- Returns `200 OK` with `{}` — CLI ignores the response
|
||||
- Per-IP rate limit: ~10 req/min (same token bucket pattern as `/contribute`)
|
||||
- Increments Redis counters via `HINCRBY` directly (no in-memory layer)
|
||||
- If Redis unavailable: silently drop (telemetry is best-effort)
|
||||
|
||||
### 4. Redis telemetry keys
|
||||
|
||||
```
|
||||
msdl:telemetry:actions hash { fetch, eval, list, interactive }
|
||||
msdl:telemetry:platforms hash { windows, darwin, linux }
|
||||
msdl:telemetry:versions hash { "0.2.0", "0.3.0", ... }
|
||||
msdl:telemetry:products hash { "2618", "3262", "3113", ... }
|
||||
msdl:telemetry:results hash { success, failed }
|
||||
```
|
||||
|
||||
### 5. Existing metrics — Redis persistence
|
||||
|
||||
Current in-memory `atomic.Int64` counters are preserved for fast per-request increments. Added durability:
|
||||
|
||||
- **On startup**: seed in-memory counters from Redis (`HGETALL msdl:metrics:*`)
|
||||
- **Every 5 minutes**: flush in-memory counters to Redis (`HSET msdl:metrics:*`)
|
||||
- **On graceful shutdown**: final flush before exit
|
||||
|
||||
Redis keys for existing metrics:
|
||||
|
||||
```
|
||||
msdl:metrics:sku hash { requests, hits, fetches, neg_hits }
|
||||
msdl:metrics:link hash { requests, hits, fetches, neg_hits, stale }
|
||||
msdl:metrics:eval hash { requests, hits, stale }
|
||||
```
|
||||
|
||||
### 6. `/metrics` response — updated
|
||||
|
||||
Adds telemetry section to existing cache stats:
|
||||
|
||||
```json
|
||||
{
|
||||
"sku": { ... },
|
||||
"link": { ... },
|
||||
"eval": { ... },
|
||||
"telemetry": {
|
||||
"actions": { "fetch": 142, "eval": 31, "list": 8, "interactive": 67 },
|
||||
"platforms": { "windows": 180, "darwin": 42, "linux": 18 },
|
||||
"versions": { "0.3.0": 145, "0.2.0": 95 },
|
||||
"products": { "3262": 134, "2618": 89, "3113": 67 },
|
||||
"results": { "success": 390, "failed": 48 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Changes
|
||||
|
||||
### 1. Version constant
|
||||
|
||||
Injected at build time via ldflags:
|
||||
|
||||
```
|
||||
-ldflags "-X main.Version=0.3.0"
|
||||
```
|
||||
|
||||
Added to the existing GitHub Actions release workflow (`cli-release.yml`). Fallback: `const Version = "dev"` in source.
|
||||
|
||||
### 2. Update check + telemetry — concurrent goroutines
|
||||
|
||||
On every run, immediately after flag parsing, two goroutines are launched:
|
||||
|
||||
```
|
||||
main()
|
||||
├── go updateCheck() — GET /cli/version, 500ms timeout, print notice if behind
|
||||
└── go sendTelemetry() — POST /telemetry, fire-and-forget, result ignored
|
||||
```
|
||||
|
||||
Both are non-blocking. The update check result is printed before any other output if it arrives within 500ms. If the timeout fires, it's silently skipped for that run.
|
||||
|
||||
**Update notice format** (printed before picker or output):
|
||||
```
|
||||
A new version of msdl is available: v0.3.0
|
||||
Download: https://github.com/starkSV/windows-iso-downloader/releases/latest
|
||||
```
|
||||
|
||||
Both goroutines are skipped entirely when `MSDL_NO_TELEMETRY=1` is set.
|
||||
|
||||
### 3. Telemetry payload construction
|
||||
|
||||
| Scenario | action | product_id | eval_slug | success |
|
||||
|---|---|---|---|---|
|
||||
| `msdl` (interactive) | `interactive` | set after pick | — | true/false |
|
||||
| `msdl --id X --lang Y` | `fetch` | X | — | true/false |
|
||||
| `msdl --eval slug` | `eval` | — | slug | true/false |
|
||||
| `msdl --list` | `list` | — | — | true |
|
||||
|
||||
`platform` is set from `runtime.GOOS` at runtime (not build time — same result, simpler).
|
||||
`version` is set from the `Version` constant.
|
||||
|
||||
Telemetry is sent **after** the main action completes so `success` reflects the actual outcome.
|
||||
|
||||
### 4. `--help` / `-h`
|
||||
|
||||
Prints a formatted usage block and exits 0. Shown automatically by Go's `flag` package or via a custom print if the existing CLI uses manual arg parsing.
|
||||
|
||||
```
|
||||
msdl — Windows ISO downloader
|
||||
|
||||
Usage:
|
||||
msdl Interactive mode — pick product and language
|
||||
msdl --id <id> --lang <lang> Fetch link directly (skip picker)
|
||||
msdl --eval <slug> Evaluation / Server ISO
|
||||
msdl --list List all available products
|
||||
|
||||
Flags:
|
||||
--id <id> Product ID (e.g. 3262 for Windows 11 25H2)
|
||||
--lang <language> Language name (e.g. "English")
|
||||
--eval <slug> Eval ISO slug (server-2025, server-2022, win11-ent, ...)
|
||||
--list List all products and exit
|
||||
--no-contribute Skip contributing link back to msdl web cache
|
||||
-h, --help Show this help
|
||||
|
||||
Environment:
|
||||
MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks
|
||||
MSDL_NO_CONTRIBUTE=1 Disable cache contribution
|
||||
|
||||
More info: https://msdl.tech-latest.com/cli
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- **README** — one line under CLI section: "msdl sends anonymous usage counts (action, platform, version) to help us understand which products are popular. Set `MSDL_NO_TELEMETRY=1` to opt out."
|
||||
- **`/cli` page** — same one-liner in the contribute section, next to the `--no-contribute` note.
|
||||
|
||||
---
|
||||
|
||||
## What's Not In Scope
|
||||
|
||||
- No per-user tracking, no IP storage, no session IDs
|
||||
- No telemetry dashboard UI (raw numbers via `/metrics` is sufficient)
|
||||
- No forced update or auto-update — notification only
|
||||
- No telemetry for the web frontend
|
||||
184
docs/superpowers/specs/2026-07-13-needs-warming-design.md
Normal file
184
docs/superpowers/specs/2026-07-13-needs-warming-design.md
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
# Spec: "Needs Warming" Community Help Page
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-07-13
|
||||
**Author:** Shekhar + Grok (original idea), refined against live codebase by Claude
|
||||
**Related:** Crowdsourced cache warming (`/contribute`), Sentinel lockdown resilience, CLI contribution flow
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Give visible, honest signal for products currently failing web users because Microsoft's Sentinel WAF (or a rate limit) has blocked the backend and no cached link is available to fall back on. Point people at the CLI as the fix, and let a successful fetch/contribution clear the item automatically.
|
||||
|
||||
This is not a new mechanism bolted onto the project — it's a public window into resilience state (`negCache` / `linkCache`) that already exists and already drives the per-product "blocked, use CLI" banner. This just makes that signal visible in aggregate, with a stronger call to action.
|
||||
|
||||
**Change from the original draft:** a dedicated page, not a homepage section. Original draft (written by Grok from the GitHub repo alone, without runtime visibility into cache state) proposed a parallel tracking system; this version derives the list from state that already exists, which is both less code and immune to drift between two copies of "is this broken."
|
||||
|
||||
---
|
||||
|
||||
## 2. What Actually Needs Tracking (verified against `backend/main.go`)
|
||||
|
||||
`handleProxy` (main.go:1189-1404) has exactly three outcomes when a product/SKU is under an active Sentinel or rate-limit block:
|
||||
|
||||
| Outcome | Code path | User experience | Needs warming? |
|
||||
|---|---|---|---|
|
||||
| `serving cached` | negCache active, `linkCache` entry still within its own TTL (e.g. just contributed) | Gets a working link | **No** |
|
||||
| `serving stale` | negCache active, `linkCache` entry expired but still returned | Gets a (possibly near-expired) link | **No** |
|
||||
| `no stale available` | negCache active, nothing in `linkCache` at all | Gets a 429, no link | **Yes** — this is the actual failure |
|
||||
|
||||
So "needs warming" = **an active `negCache` entry with no corresponding valid-or-stale `linkCache` entry** — computed on read, not tracked as separate state. No new manager struct, no `Record()`/`Remove()` calls to keep in sync, no new Redis schema.
|
||||
|
||||
**Known gap, scoped out of MVP:** a fetch that fails with a non-rate-limit error (network blip, malformed response) and has no stale cache to fall back on returns an error directly (main.go:1377) without ever touching `negCache`. In practice this hasn't been the dominant failure mode this month — Sentinel/rate-limit blocks have been — so the MVP doesn't cover it. If it turns out to matter, closing the gap is a two-line addition (write a `negCacheEntry` with `IsSentinel: false` at that call site) rather than a redesign.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Needed That Doesn't Exist Yet
|
||||
|
||||
Two small additions, both minimal:
|
||||
|
||||
1. **A request counter per locked product/SKU.** `negCache`/`linkCache` don't track "how many times did this get hit while broken" — add a small `map[string]int` (or extend `negCacheEntry` with a counter field) incremented at the existing `"no stale available"` log call sites.
|
||||
2. **Consumer product display names on the backend.** `validContributeProducts` (main.go:476) is `map[string]bool` — just an ID allow-list, no names. Eval products already have names (`{Name, EvalURL}` map at main.go:207-211). Converting `validContributeProducts` to `map[string]string` (ID → Name) serves both its existing validation role and this new lookup need, with no behavior change to `/contribute`.
|
||||
|
||||
Language name hydration for consumer products reuses the existing `skuCache` (already populated per-product from the normal SKU-fetch flow) — look up the SKU by ID within that product's cached SKU list to get `Language`/`LocalizedLanguage`. No new structure needed; if the product isn't in `skuCache` yet, just omit the language field rather than blocking the response.
|
||||
|
||||
---
|
||||
|
||||
## 4. New Endpoint
|
||||
|
||||
### `GET /needs-warming`
|
||||
|
||||
Computed on each request by iterating `negCache`, filtering to entries where `IsSentinel` (or rate-limited) is still active **and** no valid `linkCache`/eval-cache entry exists for that key. No background job, no persistence — it's a live view.
|
||||
|
||||
**Query params:** `limit` (default 10, max 50)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"product_id": "3262",
|
||||
"sku_id": "0x0409",
|
||||
"product_name": "Windows 11 25H2",
|
||||
"language": "English (United States)",
|
||||
"is_eval": false,
|
||||
"reason": "waf_blocked",
|
||||
"last_seen": "2026-07-13T17:42:00Z",
|
||||
"request_count": 47,
|
||||
"cli_command": "msdl --id 3262 --lang \"English (United States)\""
|
||||
}
|
||||
],
|
||||
"total": 3
|
||||
}
|
||||
```
|
||||
|
||||
- Public, unauthenticated, rate-limited (30 req/min per IP — matches the existing pattern used by `/contribute` at 5/min and `/telemetry` at 10/min)
|
||||
- Sorted by `request_count` descending (surfaces the highest-impact items first, more useful than recency alone)
|
||||
- Eval products included (`is_eval: true`, `cli_command` uses `msdl --eval <slug>` instead of `--id`)
|
||||
|
||||
---
|
||||
|
||||
## 5. Backend Implementation Outline
|
||||
|
||||
No new package-level state beyond the two additions in §3. The handler:
|
||||
|
||||
```go
|
||||
func handleNeedsWarming(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
if !needsWarmingRL.allow(clientIP(r)) {
|
||||
respondJSONError(w, http.StatusTooManyRequests, "rate limit exceeded")
|
||||
return
|
||||
}
|
||||
limit := parseLimitParam(r, 10, 50)
|
||||
|
||||
negCacheMu.RLock()
|
||||
defer negCacheMu.RUnlock()
|
||||
linkCacheMu.RLock()
|
||||
defer linkCacheMu.RUnlock()
|
||||
|
||||
var items []needsWarmingItem
|
||||
for key, neg := range negCache {
|
||||
if time.Now().After(neg.ExpiresAt) {
|
||||
continue // lockdown/rate-limit window itself already expired
|
||||
}
|
||||
if _, hasValidLink := resolveValidLink(key, linkCache); hasValidLink {
|
||||
continue // already resolved via cached/stale link -- not a failure state
|
||||
}
|
||||
items = append(items, buildNeedsWarmingItem(key, neg))
|
||||
}
|
||||
sort.Slice(items, func(i, j int) bool { return items[i].RequestCount > items[j].RequestCount })
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
}
|
||||
json.NewEncoder(w).Encode(map[string]interface{}{"items": items, "total": len(items)})
|
||||
}
|
||||
```
|
||||
|
||||
Integration points:
|
||||
- Increment the new request counter at the existing `"lockdown active, no stale available"` and first-time-Sentinel/rate-limit-with-no-stale log sites (main.go:1225, ~1367)
|
||||
- No changes needed to `/contribute` or the fresh-fetch success path — an item stops appearing automatically once `linkCache` has a valid entry, which those paths already populate
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend: Dedicated Page
|
||||
|
||||
**Route:** `/needs-warming` (or `/community` — bikeshed-able, not load-bearing)
|
||||
|
||||
Linked from the footer and from the per-product "blocked" banner (`CliHandoff.tsx`'s `highlight` variant) — "See what else needs help →" — so people already primed to help via the CLI can see the fuller picture instead of just their one product.
|
||||
|
||||
**Empty state:** hide the page's list section entirely (or show a short "Everything's healthy right now" line) — no "0 items" placeholder box. Most visits will likely find this empty, which is a good sign, not a gap to fill with UI.
|
||||
|
||||
Each card:
|
||||
- Product name + language (or eval product name)
|
||||
- Request count as a small badge (social proof — "47 people hit this")
|
||||
- One-click "Copy CLI command" button (same interaction pattern as the existing `CliHandoff` component — reuse its copy-button styling)
|
||||
- No per-item "reason" jargon exposed in the UI (WAF/rate-limit distinction is backend detail); just "temporarily unavailable"
|
||||
|
||||
Tone: matches the project's existing calm, transparent voice (README already explains the caching layer and Sentinel lockdown openly) — helpful, not alarmist.
|
||||
|
||||
---
|
||||
|
||||
## 7. CLI Impact
|
||||
|
||||
None required for MVP. Optional future addition: `msdl --needs-warming` prints the same list from the terminal and offers to fetch+contribute the top item directly — natural fit for the homepage-screen work already shipped (`cli/homepage.go`), but a separate phase.
|
||||
|
||||
---
|
||||
|
||||
## 8. Privacy & Safety
|
||||
|
||||
- Only ever stores/exposes `product_id`, `sku_id`/slug, reason, counters, timestamps — no IPs, no user identifiers (matches the existing telemetry/contribute philosophy already documented in README)
|
||||
- Rate-limited public endpoint, same pattern as existing public endpoints
|
||||
- Nothing new to abuse: the list is derived from state Microsoft's own WAF already put the backend into, not something a client can inject
|
||||
|
||||
---
|
||||
|
||||
## 9. Implementation Phases
|
||||
|
||||
| Phase | Description | Effort |
|
||||
|---|---|---|
|
||||
| MVP | Request counter + product-name map conversion + `/needs-warming` handler + dedicated page | Small — most of the state already exists |
|
||||
| 2 | Link from per-product blocked banner; polish empty/loading states | Small |
|
||||
| 3 | Close the non-rate-limit fetch-error gap (§2) if it turns out to matter in practice | Small, only if needed |
|
||||
| 4 | `msdl --needs-warming` CLI command | Small |
|
||||
|
||||
(Dropped the original draft's "Redis-backed persistence" phase — this is inherently live/transient state mirroring `negCache`'s own in-memory, reset-on-restart behavior; persisting it would add a Redis key namespace for no real benefit.)
|
||||
|
||||
---
|
||||
|
||||
## 10. Open Questions Resolved
|
||||
|
||||
1. **Eval products included?** Yes.
|
||||
2. **Max age before auto-removal?** No separate TTL — inherits the underlying `negCache` entry's own expiry (60s rate-limit / 90min Sentinel), since the list is computed live rather than tracked separately.
|
||||
3. **Empty state?** Hide the list, don't show a "0 items" box.
|
||||
4. **Tone?** Helpful/neutral, matching existing project voice.
|
||||
|
||||
---
|
||||
|
||||
## 11. Success Metrics (Future)
|
||||
|
||||
- Contributions triggered from this page specifically (could tag `/contribute` calls with a `source=needs-warming` query param to measure this)
|
||||
- Reduction in time-to-resolution for locked-down products
|
||||
- CLI download/usage bump correlated with items appearing on the page
|
||||
Loading…
Add table
Reference in a new issue