diff --git a/docs/superpowers/plans/2026-06-15-msdl-cli.md b/docs/superpowers/plans/2026-06-15-msdl-cli.md new file mode 100644 index 0000000..48c0d51 --- /dev/null +++ b/docs/superpowers/plans/2026-06-15-msdl-cli.md @@ -0,0 +1,1498 @@ +# MSDL CLI Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Standalone `msdl` Go binary that fetches Windows ISO download URLs **directly from Microsoft** (from the user's own machine/IP) and prints them to stdout, with both interactive and named-arg modes. + +**Architecture:** The CLI performs the entire Microsoft flow locally — session setup (vlscppe → ov-df fingerprinting chain with a domain-less cookie jar), SKU/language lookup, and download-link fetch — ported from `backend/main.go`. **It does NOT call the MSDL backend for fetching.** Because each request originates from the user's residential/commercial IP, it bypasses the Azure Sentinel WAF that blocks the data-center backend (the Rufus/Fido model). Eval builds are likewise scraped directly from Microsoft's Eval Center. Interactive mode presents numbered menus; named args skip the relevant pickers. URLs go to stdout, all prompts to stderr (so `msdl ... | xargs wget` works). + +> **Why this changed:** This plan originally routed the CLI through the MSDL backend. Microsoft's Sentinel WAF now blocks the backend's data-center IP (Hetzner) on the link endpoint. See [`IMPLEMENTATION_PLAN.md`](../../../IMPLEMENTATION_PLAN.md) §2 for the full architecture decision. Crowdsourced contribution back to the backend (`--contribute`) is a separate phase covered there, not in this plan. + +**Tech Stack:** Go 1.21+, **zero external dependencies** — standard library only (`net/http` with a custom domain-less cookie jar, `crypto/rand`+`encoding/hex` for the session UUID, `encoding/json`, `regexp`, `html`, `flag`, `bufio`, `os`, `strings`, `sync`, `time`). Standalone module in `cli/` directory. + +--- + +## File Map + +| File | Responsibility | +|------|---------------| +| `cli/go.mod` | Module definition | +| `cli/main.go` | Entry point, flag parsing, top-level flow dispatch | +| `cli/catalog.go` | Hardcoded product catalog: ID/slug → name, word-based search | +| `cli/catalog_test.go` | Tests for product lookup and search | +| `cli/microsoft.go` | Direct Microsoft client: session setup (cookie jar + fingerprinting), SKU/language lookup, link fetch, eval-center scrape | +| `cli/microsoft_test.go` | Tests for Microsoft response parsing via mock HTTP server | +| `cli/picker.go` | Interactive numbered menus reading from stdin | +| `cli/picker_test.go` | Tests for picker input parsing | +| `.github/workflows/cli-release.yml` | Release workflow: build 4 binaries, attach to GitHub Release on `cli/v*` tag, auto-submit winget update PR | +| `winget/manifests/starkSV.msdl.yaml` | Winget version manifest | +| `winget/manifests/starkSV.msdl.installer.yaml` | Winget installer manifest (portable, x64) | +| `winget/manifests/starkSV.msdl.locale.en-US.yaml` | Winget default locale manifest | + +--- + +## Usage (target UX) + +``` +msdl # full interactive: pick product → pick language → print URL +msdl "windows 11 25h2" # filtered product list → pick language → print URL +msdl --id 3262 # skip product picker, pick language +msdl --id 3262 --lang "English (United States)" # no prompts, prints URL directly +msdl --eval # pick eval product → print all download URLs +msdl --eval server-2025 # print all eval download URLs directly +msdl --list # list all products and exit +``` + +> Note: there is no `--api` flag — the CLI talks to Microsoft directly. The `--contribute` / `--no-contribute` flags and the backend contribution endpoint are added in a later phase (see [`IMPLEMENTATION_PLAN.md`](../../../IMPLEMENTATION_PLAN.md) Phase 3). + +--- + +### Task 1: Scaffold — go.mod and stub main.go + +**Files:** +- Create: `cli/go.mod` +- Create: `cli/main.go` + +- [ ] **Step 1: Create the module** + +```bash +cd "C:\Users\shekh\OneDrive\Documents\GitHub\windows-iso-downloader" +mkdir cli +cd cli +go mod init github.com/starkSV/msdl-cli +``` + +Expected output: `go: creating new go.mod: module github.com/starkSV/msdl-cli` + +- [ ] **Step 2: Create stub main.go** + +`cli/main.go`: +```go +package main + +import "fmt" + +func main() { + fmt.Println("msdl — Windows ISO downloader") +} +``` + +- [ ] **Step 3: Verify it builds** + +```bash +cd cli && go build -o msdl . && ./msdl +``` + +Expected output: `msdl — Windows ISO downloader` + +- [ ] **Step 4: Commit** + +```bash +git add cli/ +git commit -m "feat(cli): scaffold Go module" +``` + +--- + +### Task 2: Product catalog + +**Files:** +- Create: `cli/catalog.go` +- Create: `cli/catalog_test.go` + +The catalog is hardcoded from `frontend/public/data/products.json` (18 consumer products) and `frontend/src/data/evalProducts.ts` (5 eval slugs). Hardcoded instead of embedded because the catalog rarely changes and cross-module `//go:embed` paths are awkward. + +- [ ] **Step 1: Write failing tests** + +`cli/catalog_test.go`: +```go +package main + +import "testing" + +func TestFindProductByID_found(t *testing.T) { + p, ok := findProductByID("3262") + if !ok { + t.Fatal("expected product 3262 to exist") + } + if p.ID != "3262" { + t.Errorf("got ID %s, want 3262", p.ID) + } +} + +func TestFindProductByID_notFound(t *testing.T) { + _, ok := findProductByID("9999") + if ok { + t.Fatal("expected product 9999 to not exist") + } +} + +func TestSearchProducts_match(t *testing.T) { + results := searchProducts("windows 11 25h2") + if len(results) == 0 { + t.Fatal("expected at least one result for 'windows 11 25h2'") + } +} + +func TestSearchProducts_noMatch(t *testing.T) { + results := searchProducts("zzznomatch") + if len(results) != 0 { + t.Errorf("expected no results, got %d", len(results)) + } +} + +func TestSearchProducts_caseInsensitive(t *testing.T) { + results := searchProducts("WINDOWS 11") + if len(results) == 0 { + t.Fatal("expected results for uppercase query") + } +} + +func TestFindEvalProduct_found(t *testing.T) { + p, ok := findEvalProduct("server-2025") + if !ok { + t.Fatal("expected server-2025 to exist") + } + if p.Slug != "server-2025" { + t.Errorf("got slug %s, want server-2025", p.Slug) + } +} + +func TestFindEvalProduct_notFound(t *testing.T) { + _, ok := findEvalProduct("zzz-nope") + if ok { + t.Fatal("expected zzz-nope to not exist") + } +} +``` + +- [ ] **Step 2: Run tests to confirm they fail** + +```bash +cd cli && go test ./... -v +``` + +Expected: FAIL with `undefined: findProductByID` (or similar) + +- [ ] **Step 3: Implement catalog.go** + +`cli/catalog.go`: +```go +package main + +import "strings" + +type Product struct { + ID string + Name string +} + +type EvalProduct struct { + Slug string + Name string + EvalURL string // Microsoft Eval Center page scraped directly by the CLI +} + +var consumerProducts = []Product{ + {"48", "Windows 8.1 Single Language (9600.17415)"}, + {"52", "Windows 8.1 (9600.17415)"}, + {"2378", "Windows 10 22H2 Home China (19045.2006)"}, + {"2618", "Windows 10 22H2 v1 (19045.2965)"}, + {"3113", "Windows 11 24H2 (26100.1742)"}, + {"3114", "Windows 11 24H2 Home China (26100.1742)"}, + {"3115", "Windows 11 24H2 Pro China (26100.1742)"}, + {"3131", "Windows 11 Arm64 24H2 (26100.1742)"}, + {"3132", "Windows 11 Arm64 24H2 Home China (26100.1742)"}, + {"3133", "Windows 11 Arm64 24H2 Pro China (26100.1742)"}, + {"3262", "Windows 11 25H2 (26200.6584)"}, + {"3263", "Windows 11 25H2 Home China (26200.6584)"}, + {"3264", "Windows 11 25H2 Pro China (26200.6584)"}, + {"3265", "Windows 11 Arm64 25H2 (26200.6584)"}, + {"3266", "Windows 11 Arm64 25H2 Home China (26200.6584)"}, + {"3267", "Windows 11 Arm64 25H2 Pro China (26200.6584)"}, + {"3321", "Windows 11 25H2 (Updated Oct)"}, + {"3324", "Windows 11 Arm64 25H2 (Updated Oct)"}, +} + +var evalProducts = []EvalProduct{ + {"server-2025", "Windows Server 2025", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2025"}, + {"server-2022", "Windows Server 2022", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2022"}, + {"server-2019", "Windows Server 2019", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2019"}, + {"server-2016", "Windows Server 2016", "https://www.microsoft.com/en-us/evalcenter/download-windows-server-2016"}, + {"win11-ent", "Windows 11 Enterprise", "https://www.microsoft.com/en-us/evalcenter/download-windows-11-enterprise"}, +} + +func findProductByID(id string) (Product, bool) { + for _, p := range consumerProducts { + if p.ID == id { + return p, true + } + } + return Product{}, false +} + +// searchProducts returns products whose name contains all words in query (case-insensitive). +func searchProducts(query string) []Product { + words := strings.Fields(strings.ToLower(query)) + if len(words) == 0 { + return consumerProducts + } + var results []Product + for _, p := range consumerProducts { + name := strings.ToLower(p.Name) + match := true + for _, w := range words { + if !strings.Contains(name, w) { + match = false + break + } + } + if match { + results = append(results, p) + } + } + return results +} + +func findEvalProduct(slug string) (EvalProduct, bool) { + for _, p := range evalProducts { + if p.Slug == slug { + return p, true + } + } + return EvalProduct{}, false +} +``` + +- [ ] **Step 4: Run tests to confirm they pass** + +```bash +cd cli && go test ./... -v +``` + +Expected: all PASS + +- [ ] **Step 5: Commit** + +```bash +git add cli/catalog.go cli/catalog_test.go +git commit -m "feat(cli): product catalog with word-based search" +``` + +--- + +### Task 3: Microsoft client (direct, from the user's machine) + +**Files:** +- Create: `cli/microsoft.go` +- Create: `cli/microsoft_test.go` + +This ports the Microsoft flow from `backend/main.go` so the CLI calls Microsoft directly — no MSDL backend. Source references: +- `setupSession()` — `backend/main.go:394` (vlscppe → ov-df fingerprinting + cookie accumulation) +- `simpleCookieJar` — `backend/main.go:45` (domain-less cookie jar) +- `fetchSkuInfoFromMS()` — `backend/main.go:470` +- `fetchDownloadLinksFromMS()` — `backend/main.go:566` (note the SKU-info "warmup" call before the link fetch) +- `fetchEvalLinks()` / `detectArch` / `detectLang` / `fwlinkRe` / `isoLangRe` — `backend/main.go:278`, `256`, `270`, `191`, `192` + +**Design for testability:** the network functions (`fetchLanguages`, `fetchDownloadLinks`, `fetchEvalLinks`) hit hardcoded `microsoft.com` URLs, so they're covered by Task 5 smoke tests against the real API. The **parsing** is factored into pure functions (`parseSkuInfo`, `parseDownloadLinks`, `extractFwlinks`) that take bytes/strings and are unit-tested here. + +- [ ] **Step 1: Write failing tests for the pure parsers** + +`cli/microsoft_test.go`: +```go +package main + +import "testing" + +func TestParseSkuInfo_success(t *testing.T) { + raw := []byte(`{"Skus":[{"Id":"19675","Language":"English (United States)"},{"Id":"19676","Language":"French"}]}`) + langs, err := parseSkuInfo(raw) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(langs) != 2 { + t.Fatalf("want 2 languages, got %d", len(langs)) + } + if langs[0].ID != "19675" || langs[0].Language != "English (United States)" { + t.Errorf("unexpected first lang: %+v", langs[0]) + } +} + +func TestParseSkuInfo_rateLimitError(t *testing.T) { + raw := []byte(`{"Errors":[{"Type":9,"Value":"715-123130 blocked"}]}`) + _, err := parseSkuInfo(raw) + if err == nil { + t.Fatal("expected error for Errors array with Type 9") + } +} + +func TestParseSkuInfo_empty(t *testing.T) { + _, err := parseSkuInfo([]byte(`{"Skus":[]}`)) + if err == nil { + t.Fatal("expected error for empty Skus") + } +} + +func TestParseDownloadLinks_architectureString(t *testing.T) { + raw := []byte(`{"ProductDownloadOptions":[{"Uri":"https://example.com/win11.iso?t=abc","Architecture":"x64"}]}`) + links, err := parseDownloadLinks(raw) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(links) != 1 { + t.Fatalf("want 1 link, got %d", len(links)) + } + if links[0].URI != "https://example.com/win11.iso?t=abc" || links[0].Architecture != "x64" { + t.Errorf("unexpected link: %+v", links[0]) + } +} + +func TestParseDownloadLinks_downloadTypeFallback(t *testing.T) { + // No Architecture field; DownloadType 1 maps to x64. + raw := []byte(`{"ProductDownloadOptions":[{"Uri":"https://example.com/a.iso","DownloadType":1}]}`) + links, err := parseDownloadLinks(raw) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if links[0].Architecture != "x64" { + t.Errorf("want x64 from DownloadType 1, got %q", links[0].Architecture) + } +} + +func TestParseDownloadLinks_error(t *testing.T) { + raw := []byte(`{"Errors":[{"Type":4,"Value":"no download links found for this SKU"}]}`) + _, err := parseDownloadLinks(raw) + if err == nil { + t.Fatal("expected error for Errors array") + } +} + +func TestParseDownloadLinks_empty(t *testing.T) { + _, err := parseDownloadLinks([]byte(`{"ProductDownloadOptions":[]}`)) + if err == nil { + t.Fatal("expected error for empty options") + } +} + +func TestExtractFwlinks_dedupAndUnescape(t *testing.T) { + html := ` + x64 + dup + arm + ` + links := extractFwlinks(html) + if len(links) != 2 { + t.Fatalf("want 2 unique fwlinks, got %d: %v", len(links), links) + } + if links[0] != "https://go.microsoft.com/fwlink/?linkid=111&clcid=0x409" { + t.Errorf("expected unescaped &, got %q", links[0]) + } +} + +func TestDetectArchLang(t *testing.T) { + url := "https://software-static.download.prss.microsoft.com/.../server2025_arm64_en-us.iso" + if got := detectArch(url); got != "ARM64" { + t.Errorf("arch: want ARM64, got %s", got) + } + if got := detectLang(url); got != "en-us" { + t.Errorf("lang: want en-us, got %s", got) + } +} +``` + +- [ ] **Step 2: Run tests to confirm they fail** + +```bash +cd cli && go test ./... -run "TestParseSku|TestParseDownload|TestExtractFwlinks|TestDetectArchLang" -v +``` + +Expected: FAIL with `undefined: parseSkuInfo` (or similar) + +- [ ] **Step 3: Implement microsoft.go** + +`cli/microsoft.go`: +```go +package main + +import ( + "crypto/rand" + "encoding/hex" + "encoding/json" + "fmt" + "html" + "io" + "net/http" + "net/url" + "regexp" + "strconv" + "strings" + "sync" + "time" +) + +// Constants ported from backend/main.go. These mimic the official download page. +const ( + msUA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + msProfile = "606624d44113" + msLocale = "en-US" + msOrgID = "y6jn8c31" + msCustomer = "560dc9f3-1aa5-4a2f-b63c-9e18f8d0e175" +) + +var ( + fwlinkRe = regexp.MustCompile(`https://go\.microsoft\.com/fwlink/[^"'\s<>]+`) + isoLangRe = regexp.MustCompile(`_([a-z]{2}-[a-z]{2})\.iso$`) + reW = regexp.MustCompile(`[&?]w=([^&"'\s]+)`) + reRt = regexp.MustCompile(`rticks[="]+\+?\s*(\d{10,})`) +) + +type Language struct { + ID string `json:"Id"` + Language string `json:"Language"` +} + +type DownloadLink struct { + URI string + Architecture string +} + +type EvalLink struct { + Arch string + Lang string + URL string +} + +// simpleCookieJar stores cookies without domain scoping so cookies from +// Microsoft's fingerprinting hosts (vlscppe, ov-df) replay on the +// download-connector call. Ported from backend/main.go:45. +type simpleCookieJar struct { + mu sync.Mutex + cookies []*http.Cookie +} + +func (j *simpleCookieJar) SetCookies(_ *url.URL, cookies []*http.Cookie) { + j.mu.Lock() + defer j.mu.Unlock() + for _, c := range cookies { + found := false + for i, existing := range j.cookies { + if existing.Name == c.Name { + j.cookies[i] = c + found = true + break + } + } + if !found { + j.cookies = append(j.cookies, c) + } + } +} + +func (j *simpleCookieJar) Cookies(_ *url.URL) []*http.Cookie { + j.mu.Lock() + defer j.mu.Unlock() + out := make([]*http.Cookie, len(j.cookies)) + copy(out, j.cookies) + return out +} + +// newSessionID returns a v4 UUID without an external dependency. +func newSessionID() string { + b := make([]byte, 16) + rand.Read(b) + b[6] = (b[6] & 0x0f) | 0x40 + b[8] = (b[8] & 0x3f) | 0x80 + return fmt.Sprintf("%s-%s-%s-%s-%s", + hex.EncodeToString(b[0:4]), hex.EncodeToString(b[4:6]), + hex.EncodeToString(b[6:8]), hex.EncodeToString(b[8:10]), + hex.EncodeToString(b[10:16])) +} + +func referer(productID string) string { + id, err := strconv.Atoi(productID) + if err != nil { + return "https://www.microsoft.com/en-us/software-download/windows8ISO" + } + if id >= 2935 { + return "https://www.microsoft.com/en-us/software-download/windows11" + } + if id >= 2618 { + return "https://www.microsoft.com/en-us/software-download/windows10ISO" + } + return "https://www.microsoft.com/en-us/software-download/windows8ISO" +} + +func mapDownloadType(n int) string { + switch n { + case 0: + return "x86" + case 1: + return "x64" + case 2: + return "ARM64" + default: + return fmt.Sprintf("type_%d", n) + } +} + +// newSession runs the vlscppe → ov-df fingerprinting chain and returns a client +// whose cookie jar carries the accumulated session cookies, plus the session ID. +func newSession() (*http.Client, string) { + sessionID := newSessionID() + jar := &simpleCookieJar{} + client := &http.Client{Timeout: 15 * time.Second, Jar: jar} + + q1 := url.Values{} + q1.Set("org_id", msOrgID) + q1.Set("session_id", sessionID) + req1, _ := http.NewRequest("GET", "https://vlscppe.microsoft.com/tags?"+q1.Encode(), nil) + req1.Header.Set("User-Agent", msUA) + client.Do(req1) + + q2 := url.Values{} + q2.Set("instanceId", msCustomer) + q2.Set("PageId", "si") + q2.Set("session_id", sessionID) + req2, _ := http.NewRequest("GET", "https://ov-df.microsoft.com/mdt.js?"+q2.Encode(), nil) + req2.Header.Set("User-Agent", msUA) + resp2, err := client.Do(req2) + if err != nil { + return client, sessionID + } + body, _ := io.ReadAll(resp2.Body) + resp2.Body.Close() + + wMatch := reW.FindStringSubmatch(string(body)) + rtMatch := reRt.FindStringSubmatch(string(body)) + if len(wMatch) > 1 && len(rtMatch) > 1 { + q3 := url.Values{} + q3.Set("session_id", sessionID) + q3.Set("CustomerId", msCustomer) + q3.Set("PageId", "si") + q3.Set("w", wMatch[1]) + q3.Set("mdt", fmt.Sprintf("%d", time.Now().UnixMilli())) + q3.Set("rticks", rtMatch[1]) + req3, _ := http.NewRequest("GET", "https://ov-df.microsoft.com/?"+q3.Encode(), nil) + req3.Header.Set("User-Agent", msUA) + client.Do(req3) + } + return client, sessionID +} + +func msGet(client *http.Client, reqURL, productID string) ([]byte, error) { + req, _ := http.NewRequest("GET", reqURL, nil) + req.Header.Set("User-Agent", msUA) + req.Header.Set("Referer", referer(productID)) + req.Header.Set("Accept", "application/json") + resp, err := client.Do(req) + if err != nil { + return nil, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + body, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("Microsoft returned HTTP %d", resp.StatusCode) + } + // Some responses are a JSON-encoded string; unwrap one level. + if len(body) > 0 && body[0] == '"' { + var unquoted string + json.Unmarshal(body, &unquoted) + body = []byte(unquoted) + } + return body, nil +} + +type msErrorEntry struct { + Type float64 `json:"Type"` + Value string `json:"Value"` +} + +func firstError(errs []msErrorEntry) string { + if len(errs) == 0 { + return "" + } + if int(errs[0].Type) == 9 { + if errs[0].Value != "" { + return errs[0].Value + } + return "Your IP has been temporarily blocked by Microsoft (Code 715-123130)" + } + if errs[0].Value != "" { + return errs[0].Value + } + return "Microsoft API error" +} + +// parseSkuInfo extracts the language list from a getskuinformation response. +func parseSkuInfo(raw []byte) ([]Language, error) { + var data struct { + Skus []Language `json:"Skus"` + Errors []msErrorEntry `json:"Errors"` + } + if err := json.Unmarshal(raw, &data); err != nil { + return nil, fmt.Errorf("invalid SKU response: %w", err) + } + if msg := firstError(data.Errors); msg != "" { + return nil, fmt.Errorf("%s", msg) + } + if len(data.Skus) == 0 { + return nil, fmt.Errorf("no languages found for this product") + } + return data.Skus, nil +} + +// parseDownloadLinks extracts download options, filling Architecture from +// DownloadType when the Architecture field is absent. +func parseDownloadLinks(raw []byte) ([]DownloadLink, error) { + var data struct { + ProductDownloadOptions []struct { + Uri string `json:"Uri"` + Architecture interface{} `json:"Architecture"` + DownloadType *float64 `json:"DownloadType"` + } `json:"ProductDownloadOptions"` + Errors []msErrorEntry `json:"Errors"` + } + if err := json.Unmarshal(raw, &data); err != nil { + return nil, fmt.Errorf("invalid download response: %w", err) + } + if msg := firstError(data.Errors); msg != "" { + return nil, fmt.Errorf("%s", msg) + } + if len(data.ProductDownloadOptions) == 0 { + return nil, fmt.Errorf("no download links found for this SKU") + } + var links []DownloadLink + for _, o := range data.ProductDownloadOptions { + arch := "" + if s, ok := o.Architecture.(string); ok && s != "" { + arch = s + } else if o.DownloadType != nil { + arch = mapDownloadType(int(*o.DownloadType)) + } + links = append(links, DownloadLink{URI: o.Uri, Architecture: arch}) + } + return links, nil +} + +func fetchLanguages(client *http.Client, sessionID, productID string) ([]Language, error) { + q := url.Values{} + q.Set("profile", msProfile) + q.Set("productEditionId", productID) + q.Set("SKU", "undefined") + q.Set("friendlyFileName", "undefined") + q.Set("Locale", msLocale) + q.Set("sessionID", sessionID) + raw, err := msGet(client, + "https://www.microsoft.com/software-download-connector/api/getskuinformationbyproductedition?"+q.Encode(), + productID) + if err != nil { + return nil, err + } + return parseSkuInfo(raw) +} + +func fetchDownloadLinks(client *http.Client, sessionID, productID, skuID string) ([]DownloadLink, error) { + // Warmup: a SKU-info call so setup cookies carry into the link fetch + // (mirrors backend/main.go:581). + wq := url.Values{} + wq.Set("profile", msProfile) + wq.Set("productEditionId", productID) + wq.Set("SKU", "undefined") + wq.Set("friendlyFileName", "undefined") + wq.Set("Locale", msLocale) + wq.Set("sessionID", sessionID) + msGet(client, + "https://www.microsoft.com/software-download-connector/api/getskuinformationbyproductedition?"+wq.Encode(), + productID) + + q := url.Values{} + q.Set("profile", msProfile) + q.Set("productEditionId", "undefined") + q.Set("SKU", skuID) + q.Set("friendlyFileName", "undefined") + q.Set("Locale", msLocale) + q.Set("sessionID", sessionID) + raw, err := msGet(client, + "https://www.microsoft.com/software-download-connector/api/GetProductDownloadLinksBySku?"+q.Encode(), + productID) + if err != nil { + return nil, err + } + return parseDownloadLinks(raw) +} + +// --- Eval Center (direct scrape) --- + +func extractFwlinks(pageHTML string) []string { + matches := fwlinkRe.FindAllString(pageHTML, -1) + seen := map[string]bool{} + var out []string + for _, m := range matches { + link := html.UnescapeString(m) + if !seen[link] { + seen[link] = true + out = append(out, link) + } + } + return out +} + +func detectArch(rawURL string) string { + l := strings.ToLower(rawURL) + switch { + case strings.Contains(l, "arm64"): + return "ARM64" + case strings.Contains(l, "x64"): + return "x64" + case strings.Contains(l, "x86"): + return "x86" + } + return "ISO" +} + +func detectLang(rawURL string) string { + m := isoLangRe.FindStringSubmatch(strings.ToLower(rawURL)) + if len(m) > 1 { + return m[1] + } + return "" +} + +// fetchEvalLinks scrapes a Microsoft Eval Center page, follows each fwlink to +// its final .iso URL, and returns the resolved links. The Eval Center is a +// separate endpoint and is fetched directly (no session/cookie dance needed). +func fetchEvalLinks(evalURL string) ([]EvalLink, error) { + client := &http.Client{Timeout: 20 * time.Second} + req, _ := http.NewRequest("GET", evalURL, nil) + req.Header.Set("User-Agent", msUA) + req.Header.Set("Accept", "text/html,application/xhtml+xml") + resp, err := client.Do(req) + if err != nil { + return nil, fmt.Errorf("fetching eval page: %w", err) + } + body, _ := io.ReadAll(resp.Body) + resp.Body.Close() + + fwlinks := extractFwlinks(string(body)) + var ( + links []EvalLink + mu sync.Mutex + wg sync.WaitGroup + ) + for _, fw := range fwlinks { + wg.Add(1) + go func(fw string) { + defer wg.Done() + r, _ := http.NewRequest("GET", fw, nil) + r.Header.Set("User-Agent", msUA) + resp, err := client.Do(r) + if err != nil { + return + } + resp.Body.Close() + final := resp.Request.URL.String() + if !strings.Contains(strings.ToLower(final), ".iso") { + return + } + mu.Lock() + links = append(links, EvalLink{Arch: detectArch(final), Lang: detectLang(final), URL: final}) + mu.Unlock() + }(fw) + } + wg.Wait() + if len(links) == 0 { + return nil, fmt.Errorf("no eval ISO links found") + } + return links, nil +} +``` + +- [ ] **Step 4: Run tests to confirm they pass** + +```bash +cd cli && go test ./... -run "TestParseSku|TestParseDownload|TestExtractFwlinks|TestDetectArchLang" -v +``` + +Expected: all PASS + +- [ ] **Step 5: Commit** + +```bash +git add cli/microsoft.go cli/microsoft_test.go +git commit -m "feat(cli): direct Microsoft client (session, SKU, links, eval scrape)" +``` + +--- + +### Task 4: Interactive picker + +**Files:** +- Create: `cli/picker.go` +- Create: `cli/picker_test.go` + +All prompts print to stderr so stdout stays clean for piping. `parseChoice` is pure (takes an `io.Reader`) so it's directly testable without capturing stdin. + +- [ ] **Step 1: Write failing tests** + +`cli/picker_test.go`: +```go +package main + +import ( + "strings" + "testing" +) + +func TestParseChoice_valid(t *testing.T) { + cases := []struct { + input string + max int + want int + }{ + {"1\n", 3, 1}, + {"3\n", 3, 3}, + {" 2 \n", 5, 2}, + } + for _, c := range cases { + got, err := parseChoice(strings.NewReader(c.input), c.max) + if err != nil { + t.Errorf("input %q: unexpected error: %v", c.input, err) + continue + } + if got != c.want { + t.Errorf("input %q: got %d, want %d", c.input, got, c.want) + } + } +} + +func TestParseChoice_outOfRange(t *testing.T) { + cases := []string{"0\n", "4\n"} + for _, input := range cases { + _, err := parseChoice(strings.NewReader(input), 3) + if err == nil { + t.Errorf("input %q: expected error for out-of-range, got nil", input) + } + } +} + +func TestParseChoice_nonNumeric(t *testing.T) { + _, err := parseChoice(strings.NewReader("abc\n"), 3) + if err == nil { + t.Error("expected error for non-numeric input, got nil") + } +} + +func TestParseChoice_emptyInput(t *testing.T) { + _, err := parseChoice(strings.NewReader("\n"), 3) + if err == nil { + t.Error("expected error for empty input, got nil") + } +} +``` + +- [ ] **Step 2: Run tests to confirm they fail** + +```bash +cd cli && go test ./... -run "TestParse" -v +``` + +Expected: FAIL with `undefined: parseChoice` + +- [ ] **Step 3: Implement picker.go** + +`cli/picker.go`: +```go +package main + +import ( + "bufio" + "fmt" + "io" + "os" + "strconv" + "strings" +) + +// parseChoice reads a 1-based integer from r, validated in [1, max]. +func parseChoice(r io.Reader, max int) (int, error) { + scanner := bufio.NewScanner(r) + if !scanner.Scan() { + return 0, fmt.Errorf("no input") + } + text := strings.TrimSpace(scanner.Text()) + if text == "" { + return 0, fmt.Errorf("enter a number between 1 and %d", max) + } + n, err := strconv.Atoi(text) + if err != nil || n < 1 || n > max { + return 0, fmt.Errorf("enter a number between 1 and %d", max) + } + return n, nil +} + +func pickProduct(products []Product) (Product, error) { + fmt.Fprintln(os.Stderr, "\nSelect a product:") + for i, p := range products { + fmt.Fprintf(os.Stderr, " %2d. %s\n", i+1, p.Name) + } + fmt.Fprint(os.Stderr, "\nChoice: ") + n, err := parseChoice(os.Stdin, len(products)) + if err != nil { + return Product{}, err + } + return products[n-1], nil +} + +func pickLanguage(langs []Language) (Language, error) { + fmt.Fprintln(os.Stderr, "\nSelect a language:") + for i, l := range langs { + fmt.Fprintf(os.Stderr, " %2d. %s\n", i+1, l.Language) + } + fmt.Fprint(os.Stderr, "\nChoice: ") + n, err := parseChoice(os.Stdin, len(langs)) + if err != nil { + return Language{}, err + } + return langs[n-1], nil +} + +func pickEvalProduct(products []EvalProduct) (EvalProduct, error) { + fmt.Fprintln(os.Stderr, "\nSelect an evaluation product:") + for i, p := range products { + fmt.Fprintf(os.Stderr, " %2d. %s\n", i+1, p.Name) + } + fmt.Fprint(os.Stderr, "\nChoice: ") + n, err := parseChoice(os.Stdin, len(products)) + if err != nil { + return EvalProduct{}, err + } + return products[n-1], nil +} +``` + +- [ ] **Step 4: Run tests to confirm they pass** + +```bash +cd cli && go test ./... -run "TestParse" -v +``` + +Expected: all PASS + +- [ ] **Step 5: Commit** + +```bash +git add cli/picker.go cli/picker_test.go +git commit -m "feat(cli): interactive numbered picker (stdin/stderr separated)" +``` + +--- + +### Task 5: Main flow + +**Files:** +- Modify: `cli/main.go` (replace stub) + +This is the glue layer: parse flags, dispatch to `runConsumer` or `runEval`, threading named-arg overrides through each stage. + +- [ ] **Step 1: Replace stub main.go with the full implementation** + +`cli/main.go`: +```go +package main + +import ( + "flag" + "fmt" + "os" + "strings" +) + +func main() { + if err := run(os.Args[1:]); err != nil { + fmt.Fprintln(os.Stderr, "error:", err) + os.Exit(1) + } +} + +func run(args []string) error { + fs := flag.NewFlagSet("msdl", flag.ContinueOnError) + fs.SetOutput(os.Stderr) + fs.Usage = func() { + fmt.Fprintln(os.Stderr, `msdl — Windows ISO downloader + +Usage: + msdl [search terms] interactive: filter + pick product, pick language + msdl --id 3262 skip product picker + msdl --id 3262 --lang "English (United States)" no prompts, print URL directly + msdl --eval [slug] evaluation ISOs (server-2025, win11-ent, ...) + msdl --list list all products and exit + +Flags:`) + fs.PrintDefaults() + } + + productID := fs.String("id", "", "consumer product ID (skips product picker)") + langFlag := fs.String("lang", "", `language name, e.g. "English (United States)"`) + evalMode := fs.Bool("eval", false, "fetch evaluation ISOs") + listMode := fs.Bool("list", false, "list all products and exit") + + if err := fs.Parse(args); err != nil { + return err + } + query := strings.Join(fs.Args(), " ") + + if *listMode { + fmt.Fprintln(os.Stderr, "Consumer products:") + for _, p := range consumerProducts { + fmt.Fprintf(os.Stderr, " %-6s %s\n", p.ID, p.Name) + } + fmt.Fprintln(os.Stderr, "\nEvaluation products:") + for _, p := range evalProducts { + fmt.Fprintf(os.Stderr, " %-20s %s\n", p.Slug, p.Name) + } + return nil + } + + if *evalMode { + return runEval(query) + } + return runConsumer(*productID, query, *langFlag) +} + +func runConsumer(productID, query, langName string) error { + var product Product + + if productID != "" { + p, ok := findProductByID(productID) + if !ok { + return fmt.Errorf("unknown product ID %q — run msdl --list to see all products", productID) + } + product = p + } else { + candidates := consumerProducts + if query != "" { + candidates = searchProducts(query) + if len(candidates) == 0 { + return fmt.Errorf("no products match %q — run msdl --list to see all products", query) + } + } + if len(candidates) == 1 { + product = candidates[0] + fmt.Fprintf(os.Stderr, "Selected: %s\n", product.Name) + } else { + var err error + product, err = pickProduct(candidates) + if err != nil { + return err + } + } + } + + fmt.Fprintf(os.Stderr, "Setting up Microsoft session...\n") + client, sessionID := newSession() + + langs, err := fetchLanguages(client, sessionID, product.ID) + if err != nil { + return fmt.Errorf("fetching languages for %s: %w", product.Name, err) + } + + var lang Language + if langName != "" { + for _, l := range langs { + if strings.EqualFold(l.Language, langName) { + lang = l + break + } + } + if lang.ID == "" { + return fmt.Errorf("language %q not available for %s", langName, product.Name) + } + } else { + lang, err = pickLanguage(langs) + if err != nil { + return err + } + } + + fmt.Fprintf(os.Stderr, "Fetching download link...\n") + links, err := fetchDownloadLinks(client, sessionID, product.ID, lang.ID) + if err != nil { + return fmt.Errorf("fetching download links: %w", err) + } + + for _, link := range links { + fmt.Println(link.URI) + } + return nil +} + +func runEval(slug string) error { + var ep EvalProduct + + if slug != "" { + p, ok := findEvalProduct(slug) + if !ok { + return fmt.Errorf("unknown eval product %q — valid slugs: server-2025, server-2022, server-2019, server-2016, win11-ent", slug) + } + ep = p + } else { + var err error + ep, err = pickEvalProduct(evalProducts) + if err != nil { + return err + } + } + + fmt.Fprintf(os.Stderr, "Fetching eval links for %s...\n", ep.Name) + links, err := fetchEvalLinks(ep.EvalURL) + if err != nil { + return fmt.Errorf("fetching eval links for %s: %w", ep.Name, err) + } + if len(links) == 0 { + return fmt.Errorf("no eval links returned for %s", ep.Slug) + } + + for _, link := range links { + if link.Arch != "" { + fmt.Printf("%s\t%s\n", link.Arch, link.URL) + } else { + fmt.Println(link.URL) + } + } + return nil +} +``` + +- [ ] **Step 2: Build and run smoke tests** + +```bash +cd cli && go build -o msdl . + +# List mode +./msdl --list + +# Interactive (will prompt — confirm prompts appear on stderr, URLs on stdout) +echo "1" | ./msdl --id 3262 2>/dev/null # non-interactive language pick + +# Full pipeline test +./msdl --id 3262 --lang "English (United States)" 2>/dev/null +``` + +Expected for the last command: a single Microsoft CDN URL printed to stdout (no prompts), something like: +``` +https://software.download.prss.microsoft.com/dbazure/Win11_25H2_...iso?t=... +``` + +> **Run these from a residential/commercial connection, not a data center.** The whole point of the CLI is that the call originates from the user's IP — running the smoke test from a VPS/CI runner may itself be Sentinel-blocked and is not a valid test of the code. A successful fetch of product 3262/3321 (currently backend-blocked) from a home connection is the proof the architecture works. + +If Microsoft returns an error, check that the error prints to stderr and exit code is 1: +```bash +./msdl --id 9999 --lang "English (United States)"; echo "exit: $?" +``` +Expected: `error: unknown product ID "9999" — run msdl --list to see all products` + `exit: 1` + +- [ ] **Step 3: Run the full test suite** + +```bash +cd cli && go test ./... -v +``` + +Expected: all PASS (Tasks 2–4 tests still pass, no regressions) + +- [ ] **Step 4: Commit** + +```bash +git add cli/main.go +git commit -m "feat(cli): main flow with interactive and named-arg modes" +``` + +--- + +### Task 6: GitHub Actions release workflow + +**Files:** +- Create: `.github/workflows/cli-release.yml` + +Triggered by `cli/v*` tags. Builds 4 binaries (linux-amd64, darwin-amd64, darwin-arm64, windows-amd64), attaches them to a GitHub Release, then auto-submits a winget update PR using `wingetcreate`. + +**Prerequisite:** Add a `WINGET_TOKEN` repository secret — a GitHub PAT (classic) with `public_repo` scope. `wingetcreate` uses this to open a PR on `microsoft/winget-pkgs` on your behalf. Go to GitHub → Settings → Developer Settings → Personal access tokens → Generate new token (classic), check `public_repo`, copy the token, then add it at `https://github.com/starkSV/windows-iso-downloader/settings/secrets/actions` as `WINGET_TOKEN`. + +- [ ] **Step 1: Create the workflow file** + +`.github/workflows/cli-release.yml`: +```yaml +name: CLI Release + +on: + push: + tags: + - 'cli/v*' + +jobs: + release: + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-go@v5 + with: + go-version: '1.21' + + - name: Build binaries + working-directory: cli + run: | + mkdir -p ../dist + GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o ../dist/msdl-linux-amd64 . + GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w" -o ../dist/msdl-darwin-amd64 . + GOOS=darwin GOARCH=arm64 go build -ldflags="-s -w" -o ../dist/msdl-darwin-arm64 . + GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o ../dist/msdl-windows-amd64.exe . + + - name: Create GitHub Release + uses: softprops/action-gh-release@v2 + with: + tag_name: ${{ github.ref_name }} + name: "msdl CLI ${{ github.ref_name }}" + files: dist/* + body: | + ## Install + + **Windows (winget):** + ``` + winget install starkSV.msdl + ``` + + **Linux/macOS:** + ```bash + # Linux x86_64 + curl -L https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-linux-amd64 -o msdl + chmod +x msdl && sudo mv msdl /usr/local/bin/ + + # macOS Apple Silicon + curl -L https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-darwin-arm64 -o msdl + chmod +x msdl && sudo mv msdl /usr/local/bin/ + + # macOS Intel + curl -L https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-darwin-amd64 -o msdl + chmod +x msdl && sudo mv msdl /usr/local/bin/ + ``` + + **Windows (manual):** Download `msdl-windows-amd64.exe`, rename to `msdl.exe`, place in a directory on your PATH. + + ## Usage + ``` + msdl # fully interactive + msdl "windows 11 25h2" # filter products, pick language + msdl --id 3262 --lang "English (United States)" # direct, no prompts + msdl --eval server-2025 # evaluation ISOs + msdl --list # list all products + ``` + + - name: Submit winget update + if: success() && secrets.WINGET_TOKEN != '' + env: + WINGET_TOKEN: ${{ secrets.WINGET_TOKEN }} + run: | + dotnet tool install --global wingetcreate + VERSION="${{ github.ref_name }}" + VERSION="${VERSION#cli/v}" + wingetcreate update starkSV.msdl \ + --version "$VERSION" \ + --urls "https://github.com/starkSV/windows-iso-downloader/releases/download/${{ github.ref_name }}/msdl-windows-amd64.exe" \ + --submit \ + --token "$WINGET_TOKEN" +``` + +> **Note:** The `wingetcreate update` step downloads the `.exe`, computes its SHA256, updates the manifest, and opens a PR on `microsoft/winget-pkgs` automatically. It is skipped if `WINGET_TOKEN` is not set. It will only succeed after the package is accepted into winget-pkgs (Task 7, Step 6). For `cli/v0.1.0` the step will be skipped or fail gracefully — that's expected. + +- [ ] **Step 2: Commit** + +```bash +git add .github/workflows/cli-release.yml +git commit -m "ci: GitHub Actions release workflow for msdl CLI binaries" +``` + +- [ ] **Step 3: Tag and push to trigger release** + +```bash +git tag cli/v0.1.0 +git push origin cli/v0.1.0 +``` + +Then check GitHub → Actions → CLI Release workflow. Confirm it completes and 4 binary files are attached to the release at `https://github.com/starkSV/windows-iso-downloader/releases/tag/cli%2Fv0.1.0`. The winget step will be skipped on this first release — that's expected (Task 7 handles the initial submission). + +--- + +### Task 7: Winget manifests and initial submission + +**Files:** +- Create: `winget/manifests/starkSV.msdl.yaml` +- Create: `winget/manifests/starkSV.msdl.installer.yaml` +- Create: `winget/manifests/starkSV.msdl.locale.en-US.yaml` + +These 3 files are the winget package manifest for the initial submission to `microsoft/winget-pkgs`. Once accepted, all future version bumps are handled automatically by the `wingetcreate update` step in the CI workflow (Task 6). The files in this repo serve as the source of truth / reference copy. + +**Important:** Complete Task 6 (release `cli/v0.1.0`) **before** this task. You need the real SHA256 of the published `.exe` to fill into the installer manifest. + +- [ ] **Step 1: Get the SHA256 of the published Windows binary** + +After `cli/v0.1.0` is live on GitHub Releases, download the binary and compute its hash: + +```bash +# Linux/macOS +curl -sL "https://github.com/starkSV/windows-iso-downloader/releases/download/cli%2Fv0.1.0/msdl-windows-amd64.exe" -o msdl-windows-amd64.exe +sha256sum msdl-windows-amd64.exe +``` + +On Windows PowerShell: +```powershell +(Get-FileHash "msdl-windows-amd64.exe" -Algorithm SHA256).Hash +``` + +Copy the output hash — you'll paste it into the installer manifest below. winget requires it in **UPPERCASE**. + +- [ ] **Step 2: Create the version manifest** + +`winget/manifests/starkSV.msdl.yaml`: +```yaml +PackageIdentifier: starkSV.msdl +PackageVersion: 0.1.0 +DefaultLocale: en-US +ManifestType: version +ManifestVersion: 1.6.0 +``` + +- [ ] **Step 3: Create the installer manifest** + +`winget/manifests/starkSV.msdl.installer.yaml`: +```yaml +PackageIdentifier: starkSV.msdl +PackageVersion: 0.1.0 +Platform: + - Windows.Desktop +MinimumOSVersion: 10.0.0.0 +InstallerType: portable +Commands: + - msdl +Installers: + - Architecture: x64 + InstallerUrl: https://github.com/starkSV/windows-iso-downloader/releases/download/cli%2Fv0.1.0/msdl-windows-amd64.exe + InstallerSha256: PASTE_SHA256_FROM_STEP_1_HERE +ManifestType: installer +ManifestVersion: 1.6.0 +``` + +Replace `PASTE_SHA256_FROM_STEP_1_HERE` with the actual SHA256 hash from Step 1 (uppercase, no spaces). + +- [ ] **Step 4: Create the locale manifest** + +`winget/manifests/starkSV.msdl.locale.en-US.yaml`: +```yaml +PackageIdentifier: starkSV.msdl +PackageVersion: 0.1.0 +PackageLocale: en-US +Publisher: starkSV +PublisherUrl: https://github.com/starkSV +PublisherSupportUrl: https://github.com/starkSV/windows-iso-downloader/issues +PackageName: msdl +PackageUrl: https://msdl.tech-latest.com +License: MIT +LicenseUrl: https://github.com/starkSV/windows-iso-downloader/blob/main/LICENSE +ShortDescription: Download Windows ISOs directly from Microsoft +Description: |- + msdl is a command-line tool for downloading Windows ISO files directly from Microsoft's servers. + Supports interactive product and language selection, or fully automated with named arguments. + Covers consumer Windows (10, 11, 8.1) and evaluation builds (Server 2025/2022/2019/2016, Windows 11 Enterprise). +Tags: + - windows + - iso + - download + - microsoft +ManifestType: defaultLocale +ManifestVersion: 1.6.0 +``` + +- [ ] **Step 5: Validate manifests locally** + +Install winget-create or use `winget validate` if you have winget on your machine: + +```powershell +# Validate locally (Windows, winget installed) +winget install --manifest winget\manifests\ --accept-source-agreements +``` + +This should install `msdl` and make `msdl --list` work from any terminal. Uninstall after: + +```powershell +winget uninstall starkSV.msdl +``` + +- [ ] **Step 6: Commit the manifests** + +```bash +git add winget/manifests/ +git commit -m "chore(winget): initial manifest for starkSV.msdl v0.1.0" +git push origin main +``` + +- [ ] **Step 7: Submit the initial PR to winget-pkgs** + +Fork `microsoft/winget-pkgs` on GitHub (one-time). Then copy the 3 manifest files into the correct location in the fork and submit: + +```bash +# Clone your fork +git clone https://github.com/starkSV/winget-pkgs.git +cd winget-pkgs + +# Create manifest directory matching the winget-pkgs layout +mkdir -p manifests/s/starkSV/msdl/0.1.0 + +# Copy manifests from the msdl repo +cp /path/to/windows-iso-downloader/winget/manifests/starkSV.msdl.yaml manifests/s/starkSV/msdl/0.1.0/ +cp /path/to/windows-iso-downloader/winget/manifests/starkSV.msdl.installer.yaml manifests/s/starkSV/msdl/0.1.0/ +cp /path/to/windows-iso-downloader/winget/manifests/starkSV.msdl.locale.en-US.yaml manifests/s/starkSV/msdl/0.1.0/ + +git checkout -b add-starkSV.msdl-0.1.0 +git add manifests/s/starkSV/msdl/ +git commit -m "New package: starkSV.msdl version 0.1.0" +git push origin add-starkSV.msdl-0.1.0 +``` + +Then open a PR from `starkSV/winget-pkgs:add-starkSV.msdl-0.1.0` → `microsoft/winget-pkgs:master`. + +The automated bot runs within minutes; human review typically takes 1–7 days for new packages. Once merged, `winget install starkSV.msdl` works globally and the CI's `wingetcreate update` step will auto-submit all future version PRs. + +--- + +## Self-Review + +**Spec coverage:** +- ✅ Standalone Go binary — `cli/go.mod`, standard library only (zero external deps; UUID via `crypto/rand`) +- ✅ Calls Microsoft directly from the user's machine — `microsoft.go` (NOT the MSDL backend), bypassing the Sentinel WAF that blocks the data-center backend +- ✅ Full session flow ported — `newSession` (fingerprinting + `simpleCookieJar`), `fetchLanguages`, `fetchDownloadLinks` (with warmup) +- ✅ Eval ISOs fetched directly — `runEval` + `fetchEvalLinks` scrapes the Eval Center page and resolves fwlinks +- ✅ Interactive mode — `picker.go` + `pickProduct` / `pickLanguage` / `pickEvalProduct` +- ✅ Named-arg mode — `--id`, `--lang`, `--eval slug` +- ✅ Falls back to interactive when args omitted — `runConsumer` / `runEval` +- ✅ Output URL to stdout, prompts to stderr — `fmt.Println(link.URI)` / `fmt.Fprintln(os.Stderr, ...)` +- ✅ Search/filter by name — `searchProducts` in `catalog.go` +- ✅ Auto-select when only one match — `if len(candidates) == 1` in `runConsumer` +- ✅ Distribution — GitHub Actions workflow with install instructions +- ✅ Winget publishing — manifests in `winget/manifests/`, initial PR documented step-by-step, CI auto-submits future version PRs via `wingetcreate update` + +**Out of scope here (covered in `IMPLEMENTATION_PLAN.md`):** crowdsourced contribution (`--contribute` flag + backend `/contribute` endpoint, Phase 3) and the web-to-CLI handoff (Phase 4). + +**Placeholder scan:** `PASTE_SHA256_FROM_STEP_1_HERE` in Task 7 Step 3 is intentional — it cannot be filled until the binary is built and published in Task 6. All other steps have complete, runnable content. + +**Ordering constraint:** Task 7 depends on Task 6 (needs the published binary URL and real SHA256). Tasks 1–5 are independent of Task 7 and can be completed before Task 6. + +**Type consistency:** +- `Product.ID` defined Task 2 → used as `product.ID` in Task 5 `fetchLanguages(client, sessionID, product.ID)` ✅ +- `Language.ID` defined Task 3 → used as `lang.ID` in Task 5 `fetchDownloadLinks(client, sessionID, product.ID, lang.ID)` ✅ +- `newSession()` returns `(*http.Client, string)` (Task 3) → consumed as `client, sessionID := newSession()` in Task 5 `runConsumer` ✅ +- `EvalProduct.EvalURL` defined Task 2 → used as `ep.EvalURL` in Task 5 `fetchEvalLinks(ep.EvalURL)` ✅ +- `DownloadLink.URI` defined Task 3 → printed as `link.URI` in Task 5 ✅ +- `EvalLink.Arch` / `EvalLink.URL` defined Task 3 → printed in Task 5 `runEval` ✅ +- `parseChoice` defined Task 4 → called in `pickProduct`, `pickLanguage`, `pickEvalProduct` in Task 4 ✅ +- `consumerProducts`, `evalProducts` defined Task 2 → iterated in Task 5 `--list` mode ✅ diff --git a/docs/superpowers/plans/2026-06-21-cli-telemetry-update-check.md b/docs/superpowers/plans/2026-06-21-cli-telemetry-update-check.md new file mode 100644 index 0000000..161cec8 --- /dev/null +++ b/docs/superpowers/plans/2026-06-21-cli-telemetry-update-check.md @@ -0,0 +1,1023 @@ +# CLI Telemetry + Update Check Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add anonymous CLI usage telemetry, an update check on every run, and Redis durability for existing in-memory metrics — all on the `feat/cli-telemetry` branch. + +**Architecture:** Three additions: (1) two new backend endpoints (`GET /cli/version`, `POST /telemetry`) that write counters to Redis; (2) Redis persistence for the existing in-memory metrics (seed on startup, flush every 5 min, final flush on graceful shutdown); (3) CLI goroutines that call those endpoints on every run. All CLI side-channel work uses `MSDL_NO_TELEMETRY=1` as a single opt-out. + +**Tech Stack:** Go 1.21+, `backend/main.go` (monolith, no framework), `github.com/redis/go-redis/v9`, CLI uses `runtime.GOOS` for platform, `flag` package already in place. + +## Global Constraints + +- REDIS_URL must NEVER be committed to any file — it is injected as an env var in Coolify only. +- All changes go to feature branch `feat/cli-telemetry`. Do NOT push until winget PR #390908 is approved. +- Backend: `latestCLIVersion = "0.3.0"` is a constant in `backend/main.go` — bumped manually per release. +- Telemetry is best-effort throughout: if Redis is nil or HTTP fails, silently skip — never return an error to the user. +- Per-IP rate limit on `/telemetry`: 10 req/min, burst 10 (reuse existing `newIPRateLimiter` pattern). +- `MSDL_NO_TELEMETRY=1` env var skips both update check and telemetry in the CLI. +- CLI `Version` var defaults to `"dev"` in source; injected as `0.3.0` via `-X main.Version=0.3.0` in CI. +- No unit tests for backend handlers (existing codebase has none — test via curl). CLI: add tests only where deterministic pure functions are introduced. + +--- + +### Task 1: Backend — `GET /cli/version` endpoint + +**Files:** +- Modify: `backend/main.go` + +**Interfaces:** +- Produces: `GET /cli/version` → `{"latest": "0.3.0"}` (200 OK, no auth) + +- [ ] **Step 1: Create feature branch** + +```bash +git checkout -b feat/cli-telemetry +``` + +- [ ] **Step 2: Add `latestCLIVersion` constant** + +Find the existing `const (` block at the top of `backend/main.go` (lines ~32-39). Add `latestCLIVersion` as the last entry before the closing `)`: + +```go +const ( + UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + PROFILE = "606624d44113" + LOCALE = "en-US" + ORG_ID = "y6jn8c31" + CUSTOMER_ID = "560dc9f3-1aa5-4a2f-b63c-9e18f8d0e175" + PORT = ":3002" + + latestCLIVersion = "0.3.0" +) +``` + +- [ ] **Step 3: Add `handleCLIVersion` function** + +Add this function just before `func handleMetrics`: + +```go +func handleCLIVersion(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(map[string]string{"latest": latestCLIVersion}) +} +``` + +- [ ] **Step 4: Register route in `main()`** + +In the `main()` function, inside the `mux.HandleFunc(...)` block, add after the `/health` route: + +```go +mux.HandleFunc("/cli/version", handleCLIVersion) +``` + +- [ ] **Step 5: Build to verify compilation** + +```bash +cd backend && go build -o /dev/null . && echo "OK" +``` + +Expected: `OK` with no errors. + +- [ ] **Step 6: Smoke-test with the running backend** + +```bash +curl -s http://localhost:3002/cli/version +``` + +Expected: `{"latest":"0.3.0"}` + +- [ ] **Step 7: Commit** + +```bash +git add backend/main.go +git commit -m "feat(backend): GET /cli/version returns latest CLI version" +``` + +--- + +### Task 2: Backend — `POST /telemetry` endpoint + +**Files:** +- Modify: `backend/main.go` + +**Interfaces:** +- Consumes: JSON body `{ action, product_id?, eval_slug?, platform, version, success }` +- Produces: `POST /telemetry` → `{}` (200 OK, no auth); 429 on rate limit; 400 on bad JSON/action + +Redis keys written: +``` +msdl:telemetry:actions hash { fetch, eval, list, interactive } +msdl:telemetry:platforms hash { windows, darwin, linux } +msdl:telemetry:versions hash { "0.2.0", "0.3.0", ... } +msdl:telemetry:products hash { "2618", "3262", ... } +msdl:telemetry:results hash { success, failed } +``` + +- [ ] **Step 1: Add `TelemetryPayload` struct and rate limiter** + +Add just after the `var contributeRL` line (~line 472 in current file): + +```go +var telemetryRL = newIPRateLimiter(10, 10) // 10 requests/min, burst 10 + +type telemetryPayload struct { + Action string `json:"action"` + ProductID string `json:"product_id"` + EvalSlug string `json:"eval_slug"` + Platform string `json:"platform"` + Version string `json:"version"` + Success bool `json:"success"` +} +``` + +- [ ] **Step 2: Add `handleTelemetry` function** + +Add this function just before `handleCLIVersion`: + +```go +func handleTelemetry(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + w.Header().Set("Content-Type", "application/json") + + ip := r.RemoteAddr + if fwd := r.Header.Get("X-Forwarded-For"); fwd != "" { + ip = strings.TrimSpace(strings.SplitN(fwd, ",", 2)[0]) + } + if !telemetryRL.allow(ip) { + respondJSONError(w, http.StatusTooManyRequests, "rate limit exceeded") + return + } + + var p telemetryPayload + if err := json.NewDecoder(r.Body).Decode(&p); err != nil { + respondJSONError(w, http.StatusBadRequest, "invalid JSON") + return + } + + validActions := map[string]bool{"fetch": true, "eval": true, "list": true, "interactive": true} + if !validActions[p.Action] { + respondJSONError(w, http.StatusBadRequest, "invalid action") + return + } + + if rdb != nil { + ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) + defer cancel() + rdb.HIncrBy(ctx, "msdl:telemetry:actions", p.Action, 1) + if p.Platform != "" { + rdb.HIncrBy(ctx, "msdl:telemetry:platforms", p.Platform, 1) + } + if p.Version != "" { + rdb.HIncrBy(ctx, "msdl:telemetry:versions", p.Version, 1) + } + if p.ProductID != "" { + rdb.HIncrBy(ctx, "msdl:telemetry:products", p.ProductID, 1) + } + result := "success" + if !p.Success { + result = "failed" + } + rdb.HIncrBy(ctx, "msdl:telemetry:results", result, 1) + } + + w.WriteHeader(http.StatusOK) + w.Write([]byte("{}")) +} +``` + +- [ ] **Step 3: Register route in `main()`** + +```go +mux.HandleFunc("/telemetry", handleTelemetry) +``` + +- [ ] **Step 4: Build to verify compilation** + +```bash +cd backend && go build -o /dev/null . && echo "OK" +``` + +- [ ] **Step 5: Smoke-test `/telemetry`** + +```bash +curl -s -X POST http://localhost:3002/telemetry \ + -H "Content-Type: application/json" \ + -d '{"action":"fetch","product_id":"3262","platform":"windows","version":"0.3.0","success":true}' +``` + +Expected: `{}` + +```bash +curl -s -X POST http://localhost:3002/telemetry \ + -H "Content-Type: application/json" \ + -d '{"action":"bad_action","platform":"linux","version":"0.3.0","success":false}' +``` + +Expected: `{"error":"invalid action"}` (400) + +- [ ] **Step 6: Commit** + +```bash +git add backend/main.go +git commit -m "feat(backend): POST /telemetry endpoint with Redis HINCRBY counters" +``` + +--- + +### Task 3: Backend — Redis metrics persistence + graceful shutdown + +**Files:** +- Modify: `backend/main.go` + +**Interfaces:** +- Consumes: existing `atomic.Int64` vars (`mSkuRequests`, `mLinkRequests`, etc.) +- Produces: Redis hashes `msdl:metrics:sku`, `msdl:metrics:link`, `msdl:metrics:eval` +- Produces: graceful shutdown that calls `flushMetricsToRedis()` before exit + +Redis hash schemas: +``` +msdl:metrics:sku → { requests, cache_hits, ms_fetches, neg_hits } +msdl:metrics:link → { requests, cache_hits, ms_fetches, neg_hits, stale } +msdl:metrics:eval → { requests, cache_hits, stale } +``` + +- [ ] **Step 1: Add `"os/signal"` and `"syscall"` to imports** + +The existing import block in `backend/main.go` starts on line 3. Add the two new packages: + +```go +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "html" + "io" + "log" + "math/rand" + "net/http" + "net/url" + "os" + "os/signal" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "sync/atomic" + "syscall" + "time" + + "github.com/google/uuid" + "github.com/redis/go-redis/v9" + "golang.org/x/sync/singleflight" +) +``` + +- [ ] **Step 2: Add `seedMetricsFromRedis` and `flushMetricsToRedis` functions** + +Add both functions just after the `redisSeedLinkCache` function (~line 370): + +```go +// seedMetricsFromRedis loads persisted metric counters from Redis into the +// in-memory atomic vars. Called once on startup after Redis connects. +func seedMetricsFromRedis() { + if rdb == nil { + return + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + + loadHash := func(key string) map[string]string { + vals, err := rdb.HGetAll(ctx, key).Result() + if err != nil { + return nil + } + return vals + } + parseI64 := func(m map[string]string, field string) int64 { + if m == nil { + return 0 + } + v, _ := strconv.ParseInt(m[field], 10, 64) + return v + } + + sku := loadHash("msdl:metrics:sku") + atomic.StoreInt64(&mSkuRequests, parseI64(sku, "requests")) + atomic.StoreInt64(&mSkuCacheHits, parseI64(sku, "cache_hits")) + atomic.StoreInt64(&mSkuFetches, parseI64(sku, "ms_fetches")) + atomic.StoreInt64(&mSkuNegHits, parseI64(sku, "neg_hits")) + + link := loadHash("msdl:metrics:link") + atomic.StoreInt64(&mLinkRequests, parseI64(link, "requests")) + atomic.StoreInt64(&mLinkCacheHits, parseI64(link, "cache_hits")) + atomic.StoreInt64(&mLinkFetches, parseI64(link, "ms_fetches")) + atomic.StoreInt64(&mLinkNegHits, parseI64(link, "neg_hits")) + atomic.StoreInt64(&mLinkStale, parseI64(link, "stale")) + + eval := loadHash("msdl:metrics:eval") + atomic.StoreInt64(&mEvalRequests, parseI64(eval, "requests")) + atomic.StoreInt64(&mEvalCacheHits, parseI64(eval, "cache_hits")) + atomic.StoreInt64(&mEvalStale, parseI64(eval, "stale")) + + log.Println("redis: seeded in-memory metrics from Redis") +} + +// flushMetricsToRedis writes current in-memory metric counters to Redis. +// Called periodically (every 5 min) and on graceful shutdown. +func flushMetricsToRedis() { + if rdb == nil { + return + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + + rdb.HSet(ctx, "msdl:metrics:sku", + "requests", atomic.LoadInt64(&mSkuRequests), + "cache_hits", atomic.LoadInt64(&mSkuCacheHits), + "ms_fetches", atomic.LoadInt64(&mSkuFetches), + "neg_hits", atomic.LoadInt64(&mSkuNegHits), + ) + rdb.HSet(ctx, "msdl:metrics:link", + "requests", atomic.LoadInt64(&mLinkRequests), + "cache_hits", atomic.LoadInt64(&mLinkCacheHits), + "ms_fetches", atomic.LoadInt64(&mLinkFetches), + "neg_hits", atomic.LoadInt64(&mLinkNegHits), + "stale", atomic.LoadInt64(&mLinkStale), + ) + rdb.HSet(ctx, "msdl:metrics:eval", + "requests", atomic.LoadInt64(&mEvalRequests), + "cache_hits", atomic.LoadInt64(&mEvalCacheHits), + "stale", atomic.LoadInt64(&mEvalStale), + ) + log.Println("redis: flushed in-memory metrics to Redis") +} +``` + +- [ ] **Step 3: Add `startMetricsFlusher` goroutine function** + +Add after `flushMetricsToRedis`: + +```go +// startMetricsFlusher flushes in-memory metrics to Redis every 5 minutes. +func startMetricsFlusher() { + ticker := time.NewTicker(5 * time.Minute) + for range ticker.C { + flushMetricsToRedis() + } +} +``` + +- [ ] **Step 4: Call `seedMetricsFromRedis` in `initRedis`** + +Find the `initRedis` function. After `rdb = client` and `log.Println("redis: connected")`, add: + +```go + rdb = client + log.Println("redis: connected") + seedMetricsFromRedis() +``` + +- [ ] **Step 5: Rewrite `main()` to use `http.Server` with graceful shutdown** + +Replace the existing `main()` function body: + +```go +func main() { + initRedis() + go redisSeedLinkCache() + go cleanupSessions() + go cleanupCaches() + go warmEvalCache() + go startMetricsFlusher() + + mux := http.NewServeMux() + mux.HandleFunc("/skuinfo", handleSkuInfo) + mux.HandleFunc("/proxy", handleProxy) + mux.HandleFunc("/contribute", handleContribute) + mux.HandleFunc("/evallinks", handleEvalLinks) + mux.HandleFunc("/metrics", handleMetrics) + mux.HandleFunc("/telemetry", handleTelemetry) + mux.HandleFunc("/cli/version", handleCLIVersion) + mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.Write([]byte(`{"status":"ok"}`)) + }) + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/" { + http.NotFound(w, r) + return + } + w.Write([]byte("MSDL API v3 is running")) + }) + + handler := enableCORS(mux) + srv := &http.Server{Addr: PORT, Handler: handler} + + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + go func() { + <-sigCh + log.Println("shutdown: flushing metrics to Redis...") + flushMetricsToRedis() + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + srv.Shutdown(ctx) + }() + + log.Printf("Go Backend running on http://localhost%s\n", PORT) + if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { + log.Fatal(err) + } +} +``` + +- [ ] **Step 6: Build to verify compilation** + +```bash +cd backend && go build -o /dev/null . && echo "OK" +``` + +- [ ] **Step 7: Verify Redis flush manually** + +Start the backend locally with `REDIS_URL` set, run a few requests, then check Redis: + +```bash +redis-cli -u "$REDIS_URL" HGETALL msdl:metrics:sku +``` + +(Values will be 0 on a fresh Redis; after a request they increment.) + +- [ ] **Step 8: Commit** + +```bash +git add backend/main.go +git commit -m "feat(backend): Redis persistence for metrics (seed + 5min flush + graceful shutdown)" +``` + +--- + +### Task 4: Backend — Telemetry section in `/metrics` response + +**Files:** +- Modify: `backend/main.go` + +**Interfaces:** +- Consumes: `msdl:telemetry:*` Redis hashes (written by `handleTelemetry`) +- Produces: adds `"telemetry"` key to existing `handleMetrics` JSON response + +- [ ] **Step 1: Add `loadTelemetryFromRedis` helper** + +Add just before `handleMetrics`: + +```go +// loadTelemetryFromRedis reads all telemetry counters from Redis. +// Returns nil maps when Redis is unavailable. +func loadTelemetryFromRedis() map[string]map[string]string { + if rdb == nil { + return nil + } + ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) + defer cancel() + keys := []string{ + "msdl:telemetry:actions", + "msdl:telemetry:platforms", + "msdl:telemetry:versions", + "msdl:telemetry:products", + "msdl:telemetry:results", + } + out := make(map[string]map[string]string, len(keys)) + for _, k := range keys { + vals, err := rdb.HGetAll(ctx, k).Result() + if err == nil { + // Strip the "msdl:telemetry:" prefix for the JSON key + name := strings.TrimPrefix(k, "msdl:telemetry:") + out[name] = vals + } + } + return out +} +``` + +- [ ] **Step 2: Update `handleMetrics` to include telemetry** + +Find the `json.NewEncoder(w).Encode(map[string]interface{}{` call in `handleMetrics`. Replace the map literal to add a `"telemetry"` field: + +```go + telemetry := loadTelemetryFromRedis() + + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(map[string]interface{}{ + "sku": map[string]interface{}{ + "requests": skuReqs, + "cache_hits": skuHits, + "ms_fetches": skuFetches, + "neg_hits": skuNeg, + "hit_rate": hitRate(skuHits, skuReqs), + "cache_size": skuSize, + }, + "link": map[string]interface{}{ + "requests": linkReqs, + "cache_hits": linkHits, + "ms_fetches": linkFetches, + "neg_hits": linkNeg, + "stale": linkStale, + "hit_rate": hitRate(linkHits, linkReqs), + "cache_size": linkSize, + }, + "eval": map[string]interface{}{ + "requests": evalReqs, + "cache_hits": evalHits, + "stale": evalStale, + "hit_rate": hitRate(evalHits, evalReqs), + "cache_size": evalSize, + }, + "neg_cache_size": negSize, + "total_ms_fetches": skuFetches + linkFetches, + "telemetry": telemetry, + }) +``` + +- [ ] **Step 3: Build to verify compilation** + +```bash +cd backend && go build -o /dev/null . && echo "OK" +``` + +- [ ] **Step 4: Smoke-test `/metrics`** + +```bash +curl -s "http://localhost:3002/metrics?secret=$METRICS_SECRET" | python -m json.tool +``` + +Expected: existing fields still present plus `"telemetry": { "actions": {}, ... }` (empty until telemetry events arrive). + +- [ ] **Step 5: Commit** + +```bash +git add backend/main.go +git commit -m "feat(backend): include telemetry counters in /metrics response" +``` + +--- + +### Task 5: CLI — `Version` var + GitHub Actions ldflags + `--help` update + +**Files:** +- Modify: `cli/main.go` +- Modify: `.github/workflows/cli-release.yml` + +**Interfaces:** +- Produces: `Version` package-level variable accessible from all CLI functions +- Produces: CI builds inject `-X main.Version=` so the binary reports its real version + +- [ ] **Step 1: Add `Version` variable to `cli/main.go`** + +At the top of `cli/main.go`, after the `import` block, add: + +```go +// Version is injected at build time via -ldflags "-X main.Version=0.3.0". +// Falls back to "dev" for local builds. +var Version = "dev" +``` + +- [ ] **Step 2: Update `fs.Usage` in `run()` to include env vars section** + +Replace the existing `fs.Usage` function in `run()`: + +```go +fs.Usage = func() { + fmt.Fprintln(os.Stderr, `msdl — Windows ISO downloader + +Usage: + msdl [search terms] interactive: filter + pick product, pick language + msdl --id 3262 skip product picker + msdl --id 3262 --lang "English (United States)" no prompts, print URL directly + msdl --eval [slug] evaluation ISOs (server-2025, win11-ent, ...) + msdl --list list all products and exit + +Flags:`) + fs.PrintDefaults() + fmt.Fprintln(os.Stderr, ` +Environment: + MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks + MSDL_NO_CONTRIBUTE=1 Disable cache contribution + MSDL_API_URL= Override backend URL (default: https://api.msdl.tech-latest.com) + +More info: https://msdl.tech-latest.com/cli`) +} +``` + +- [ ] **Step 3: Handle `flag.ErrHelp` cleanly in `run()`** + +Find `if err := fs.Parse(args); err != nil {` and update it: + +```go +if err := fs.Parse(args); err != nil { + if err == flag.ErrHelp { + return nil // fs.Usage already printed + } + return err +} +``` + +- [ ] **Step 4: Update GitHub Actions workflow to inject version ldflags** + +In `.github/workflows/cli-release.yml`, find the `Build binaries` step. Replace the `go build` commands to inject version: + +```yaml + - name: Build binaries + working-directory: cli + run: | + VERSION="${{ github.ref_name }}" + VERSION="${VERSION#cli/v}" + mkdir -p ../dist + GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-linux-amd64 . + GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-darwin-amd64 . + GOOS=darwin GOARCH=arm64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-darwin-arm64 . + GOOS=windows GOARCH=amd64 go build -ldflags="-s -w -X main.Version=${VERSION}" -o ../dist/msdl-windows-amd64.exe . +``` + +- [ ] **Step 5: Build CLI locally to verify** + +```bash +cd cli && go build -o /dev/null . && echo "OK" +``` + +- [ ] **Step 6: Verify --help flag works** + +```bash +cd cli && go run . --help +``` + +Expected: prints usage block + env var section, exits 0 (no "error:" prefix). + +- [ ] **Step 7: Commit** + +```bash +git add cli/main.go .github/workflows/cli-release.yml +git commit -m "feat(cli): Version var injected via ldflags, updated --help text" +``` + +--- + +### Task 6: CLI — Update check + telemetry goroutines + +**Files:** +- Modify: `cli/main.go` + +**Interfaces:** +- Consumes: `Version` (from Task 5), `apiBaseURL()` helper +- Consumes: `GET /cli/version` backend endpoint (from Task 1) +- Consumes: `POST /telemetry` backend endpoint (from Task 2) +- Produces: update notice printed to stderr before main output when a newer version is available +- Produces: telemetry payload sent fire-and-forget after every CLI run + +- [ ] **Step 1: Add `runtime` to imports in `cli/main.go`** + +The existing import block has `"sync"` and other packages. Add `"runtime"`: + +```go +import ( + "bytes" + "context" + "encoding/json" + "flag" + "fmt" + "net/http" + "os" + "runtime" + "strings" + "sync" + "time" +) +``` + +- [ ] **Step 2: Add `apiBaseURL` helper** + +Add after the `contributeSecret` const near the top of the file (before `urlFilename`): + +```go +// apiBaseURL returns the backend base URL, with MSDL_API_URL override. +func apiBaseURL() string { + if u := os.Getenv("MSDL_API_URL"); u != "" { + return strings.TrimRight(u, "/") + } + return "https://api.msdl.tech-latest.com" +} +``` + +Note: `contributeURL()` already does similar logic but for `/contribute` specifically. `apiBaseURL` is the shared base — update `contributeURL` to use it: + +```go +func contributeURL() string { + return apiBaseURL() + "/contribute" +} +``` + +- [ ] **Step 3: Add `printUpdateNotice` function** + +Add after `apiBaseURL`: + +```go +// printUpdateNotice checks /cli/version and prints a notice if a newer version +// is available. Blocks up to 500ms; silently no-ops on timeout or any error. +func printUpdateNotice() { + ch := make(chan string, 1) + go func() { + ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, apiBaseURL()+"/cli/version", nil) + if err != nil { + return + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + return + } + defer resp.Body.Close() + var body struct { + Latest string `json:"latest"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return + } + ch <- body.Latest + }() + select { + case latest := <-ch: + if latest != "" && latest != Version && Version != "dev" { + fmt.Fprintf(os.Stderr, "\n A new version of msdl is available: v%s\n", latest) + fmt.Fprintf(os.Stderr, " Download: https://github.com/starkSV/windows-iso-downloader/releases/latest\n\n") + } + case <-time.After(500 * time.Millisecond): + } +} +``` + +- [ ] **Step 4: Add `cliTelemetryPayload` struct and `sendTelemetry` function** + +Add after `printUpdateNotice`: + +```go +type cliTelemetryPayload struct { + Action string `json:"action"` + ProductID string `json:"product_id,omitempty"` + EvalSlug string `json:"eval_slug,omitempty"` + Platform string `json:"platform"` + Version string `json:"version"` + Success bool `json:"success"` +} + +// sendTelemetry posts a single telemetry event. Fire-and-forget: all errors +// are silently ignored so telemetry never affects the user experience. +func sendTelemetry(p cliTelemetryPayload) { + body, err := json.Marshal(p) + if err != nil { + return + } + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodPost, apiBaseURL()+"/telemetry", bytes.NewReader(body)) + if err != nil { + return + } + req.Header.Set("Content-Type", "application/json") + resp, err := http.DefaultClient.Do(req) + if err != nil { + return + } + resp.Body.Close() +} +``` + +- [ ] **Step 5: Thread update check and telemetry into `run()`** + +Replace the entire `run()` function with the complete version below. This incorporates the Task 5 changes (Usage block, ErrHelp handling) plus the new telemetry logic: + +```go +func run(args []string) error { + fs := flag.NewFlagSet("msdl", flag.ContinueOnError) + fs.SetOutput(os.Stderr) + fs.Usage = func() { + fmt.Fprintln(os.Stderr, `msdl — Windows ISO downloader + +Usage: + msdl [search terms] interactive: filter + pick product, pick language + msdl --id 3262 skip product picker + msdl --id 3262 --lang "English (United States)" no prompts, print URL directly + msdl --eval [slug] evaluation ISOs (server-2025, win11-ent, ...) + msdl --list list all products and exit + +Flags:`) + fs.PrintDefaults() + fmt.Fprintln(os.Stderr, ` +Environment: + MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks + MSDL_NO_CONTRIBUTE=1 Disable cache contribution + MSDL_API_URL= Override backend URL (default: https://api.msdl.tech-latest.com) + +More info: https://msdl.tech-latest.com/cli`) + } + + productID := fs.String("id", "", "consumer product ID (skips product picker)") + langFlag := fs.String("lang", "", `language name, e.g. "English (United States)"`) + evalMode := fs.Bool("eval", false, "fetch evaluation ISOs") + listMode := fs.Bool("list", false, "list all products and exit") + noContributeFlag := fs.Bool("no-contribute", false, "skip sharing the link with the msdl.tech cache") + + if err := fs.Parse(args); err != nil { + if err == flag.ErrHelp { + return nil + } + return err + } + query := strings.Join(fs.Args(), " ") + noContribute := *noContributeFlag || os.Getenv("MSDL_NO_CONTRIBUTE") == "1" + noTelemetry := os.Getenv("MSDL_NO_TELEMETRY") == "1" + + // Update check — blocks up to 500ms, then continues regardless + if !noTelemetry { + printUpdateNotice() + } + + if *listMode { + fmt.Fprintln(os.Stderr, "Consumer products:") + for _, p := range consumerProducts { + fmt.Fprintf(os.Stderr, " %-6s %s\n", p.ID, p.Name) + } + fmt.Fprintln(os.Stderr, "\nEvaluation products:") + for _, p := range evalProducts { + fmt.Fprintf(os.Stderr, " %-20s %s\n", p.Slug, p.Name) + } + if !noTelemetry { + go sendTelemetry(cliTelemetryPayload{ + Action: "list", + Platform: runtime.GOOS, + Version: Version, + Success: true, + }) + } + return nil + } + + // Determine telemetry fields before running (product_id / eval_slug known at this point) + telAction := "interactive" + telProductID := "" + telEvalSlug := "" + switch { + case *evalMode: + telAction = "eval" + telEvalSlug = query + case *productID != "": + telAction = "fetch" + telProductID = *productID + } + + var err error + if *evalMode { + err = runEval(query) + } else { + err = runConsumer(*productID, query, *langFlag, noContribute) + } + + if !noTelemetry { + go sendTelemetry(cliTelemetryPayload{ + Action: telAction, + ProductID: telProductID, + EvalSlug: telEvalSlug, + Platform: runtime.GOOS, + Version: Version, + Success: err == nil, + }) + } + + return err +} +``` + +- [ ] **Step 6: Build to verify compilation** + +```bash +cd cli && go build -o /dev/null . && echo "OK" +``` + +- [ ] **Step 7: Test update check with local backend** + +Start the backend, then: + +```bash +cd cli && MSDL_API_URL=http://localhost:3002 Version=0.2.0 go run . --list +``` + +Expected: update notice printed ("A new version of msdl is available: v0.3.0") before the product list. (Note: `Version` env var does NOT override the Go variable — use a built binary with `-ldflags "-X main.Version=0.2.0"` to test this properly.) + +Correct test: + +```bash +cd cli && go build -ldflags "-X main.Version=0.2.0" -o msdl-test . && \ + MSDL_API_URL=http://localhost:3002 ./msdl-test --list +``` + +Expected first lines of stderr: + +``` + A new version of msdl is available: v0.3.0 + Download: https://github.com/starkSV/windows-iso-downloader/releases/latest +``` + +- [ ] **Step 8: Test telemetry fires** + +After the above run, check Redis: + +```bash +redis-cli -u "$REDIS_URL" HGETALL msdl:telemetry:actions +``` + +Expected: `list` field with value `1`. + +```bash +redis-cli -u "$REDIS_URL" HGETALL msdl:telemetry:platforms +redis-cli -u "$REDIS_URL" HGETALL msdl:telemetry:versions +``` + +Expected: platform and version populated. + +- [ ] **Step 9: Test opt-out** + +```bash +cd cli && MSDL_NO_TELEMETRY=1 MSDL_API_URL=http://localhost:3002 ./msdl-test --list +``` + +Expected: no update notice, no new telemetry entries in Redis. + +- [ ] **Step 10: Clean up test binary** + +```bash +rm cli/msdl-test +``` + +- [ ] **Step 11: Run existing CLI tests** + +```bash +cd cli && go test ./... -v +``` + +Expected: all existing tests pass (no new tests required — goroutines are integration-tested manually above). + +- [ ] **Step 12: Commit** + +```bash +git add cli/main.go +git commit -m "feat(cli): update check and anonymous telemetry on every run" +``` + +--- + +### Task 7: Documentation updates + +**Files:** +- Modify: `README.md` + +**Interfaces:** None — doc-only. + +- [ ] **Step 1: Add telemetry disclosure to README** + +Find the `### Crowdsourced cache` section in `README.md`. Add a new `### Usage telemetry` section immediately after it: + +```markdown +### Usage telemetry + +By default, each run sends an anonymous event to help us understand which products are popular and which platforms are used. No personal data is sent — only action type (`fetch`, `eval`, `list`, `interactive`), platform (`windows`, `darwin`, `linux`), CLI version, and whether the run succeeded. To opt out: + +```bash +MSDL_NO_TELEMETRY=1 msdl --id 3262 --lang "English" +``` +``` + +- [ ] **Step 2: Commit** + +```bash +git add README.md +git commit -m "docs: add CLI telemetry disclosure to README" +``` + +--- + +## Post-Implementation Checklist + +After all tasks are committed to `feat/cli-telemetry`: + +- [ ] Run `cd backend && go build -o /dev/null . && echo OK` — must pass +- [ ] Run `cd cli && go test ./... && echo OK` — must pass +- [ ] Verify `/cli/version` endpoint returns correct version +- [ ] Verify `/telemetry` endpoint writes to Redis +- [ ] Verify `/metrics` includes `telemetry` section +- [ ] Verify update notice shows for stale version, suppressed for current version +- [ ] Verify `MSDL_NO_TELEMETRY=1` skips both update check and telemetry +- [ ] **DO NOT push until winget PR #390908 is merged** diff --git a/docs/superpowers/specs/2026-06-21-cli-telemetry-update-check-design.md b/docs/superpowers/specs/2026-06-21-cli-telemetry-update-check-design.md new file mode 100644 index 0000000..31ca3fd --- /dev/null +++ b/docs/superpowers/specs/2026-06-21-cli-telemetry-update-check-design.md @@ -0,0 +1,200 @@ +# CLI Telemetry, Update Check & Help — Design Spec + +**Date:** 2026-06-21 +**Status:** Approved +**Scope:** CLI (cli/) + Backend (backend/main.go) + +--- + +## Overview + +Three related additions to the msdl CLI and backend: + +1. **Telemetry** — anonymous usage counters sent on every CLI run, stored durably in Redis +2. **Update check** — CLI checks our backend for the latest version on every run, notifies if behind +3. **Help** — `msdl --help` / `-h` prints usage with all flags and examples + +No opt-in required. `MSDL_NO_TELEMETRY=1` skips both telemetry and update check silently. +Disclosed in README and `/cli` page with one line each. + +--- + +## Backend Changes + +### 1. New constant: `latestCLIVersion` + +```go +const latestCLIVersion = "0.3.0" +``` + +Bumped manually in `backend/main.go` whenever a new CLI release is cut. + +### 2. `GET /cli/version` — public, no auth + +Returns the latest CLI version. The CLI calls this on every run. + +```json +{ "latest": "0.3.0" } +``` + +No rate limiting needed — response is trivial and cacheable by the client. + +### 3. `POST /telemetry` — public, no auth + +Accepts one event per CLI run. Body: + +```json +{ + "action": "fetch" | "eval" | "list" | "interactive", + "product_id": "2618", + "eval_slug": "server-2025", + "platform": "windows" | "darwin" | "linux", + "version": "0.3.0", + "success": true +} +``` + +- `product_id` present only for `fetch` action +- `eval_slug` present only for `eval` action +- All other fields always present +- Returns `200 OK` with `{}` — CLI ignores the response +- Per-IP rate limit: ~10 req/min (same token bucket pattern as `/contribute`) +- Increments Redis counters via `HINCRBY` directly (no in-memory layer) +- If Redis unavailable: silently drop (telemetry is best-effort) + +### 4. Redis telemetry keys + +``` +msdl:telemetry:actions hash { fetch, eval, list, interactive } +msdl:telemetry:platforms hash { windows, darwin, linux } +msdl:telemetry:versions hash { "0.2.0", "0.3.0", ... } +msdl:telemetry:products hash { "2618", "3262", "3113", ... } +msdl:telemetry:results hash { success, failed } +``` + +### 5. Existing metrics — Redis persistence + +Current in-memory `atomic.Int64` counters are preserved for fast per-request increments. Added durability: + +- **On startup**: seed in-memory counters from Redis (`HGETALL msdl:metrics:*`) +- **Every 5 minutes**: flush in-memory counters to Redis (`HSET msdl:metrics:*`) +- **On graceful shutdown**: final flush before exit + +Redis keys for existing metrics: + +``` +msdl:metrics:sku hash { requests, hits, fetches, neg_hits } +msdl:metrics:link hash { requests, hits, fetches, neg_hits, stale } +msdl:metrics:eval hash { requests, hits, stale } +``` + +### 6. `/metrics` response — updated + +Adds telemetry section to existing cache stats: + +```json +{ + "sku": { ... }, + "link": { ... }, + "eval": { ... }, + "telemetry": { + "actions": { "fetch": 142, "eval": 31, "list": 8, "interactive": 67 }, + "platforms": { "windows": 180, "darwin": 42, "linux": 18 }, + "versions": { "0.3.0": 145, "0.2.0": 95 }, + "products": { "3262": 134, "2618": 89, "3113": 67 }, + "results": { "success": 390, "failed": 48 } + } +} +``` + +--- + +## CLI Changes + +### 1. Version constant + +Injected at build time via ldflags: + +``` +-ldflags "-X main.Version=0.3.0" +``` + +Added to the existing GitHub Actions release workflow (`cli-release.yml`). Fallback: `const Version = "dev"` in source. + +### 2. Update check + telemetry — concurrent goroutines + +On every run, immediately after flag parsing, two goroutines are launched: + +``` +main() + ├── go updateCheck() — GET /cli/version, 500ms timeout, print notice if behind + └── go sendTelemetry() — POST /telemetry, fire-and-forget, result ignored +``` + +Both are non-blocking. The update check result is printed before any other output if it arrives within 500ms. If the timeout fires, it's silently skipped for that run. + +**Update notice format** (printed before picker or output): +``` + A new version of msdl is available: v0.3.0 + Download: https://github.com/starkSV/windows-iso-downloader/releases/latest +``` + +Both goroutines are skipped entirely when `MSDL_NO_TELEMETRY=1` is set. + +### 3. Telemetry payload construction + +| Scenario | action | product_id | eval_slug | success | +|---|---|---|---|---| +| `msdl` (interactive) | `interactive` | set after pick | — | true/false | +| `msdl --id X --lang Y` | `fetch` | X | — | true/false | +| `msdl --eval slug` | `eval` | — | slug | true/false | +| `msdl --list` | `list` | — | — | true | + +`platform` is set from `runtime.GOOS` at runtime (not build time — same result, simpler). +`version` is set from the `Version` constant. + +Telemetry is sent **after** the main action completes so `success` reflects the actual outcome. + +### 4. `--help` / `-h` + +Prints a formatted usage block and exits 0. Shown automatically by Go's `flag` package or via a custom print if the existing CLI uses manual arg parsing. + +``` +msdl — Windows ISO downloader + +Usage: + msdl Interactive mode — pick product and language + msdl --id --lang Fetch link directly (skip picker) + msdl --eval Evaluation / Server ISO + msdl --list List all available products + +Flags: + --id Product ID (e.g. 3262 for Windows 11 25H2) + --lang Language name (e.g. "English") + --eval Eval ISO slug (server-2025, server-2022, win11-ent, ...) + --list List all products and exit + --no-contribute Skip contributing link back to msdl web cache + -h, --help Show this help + +Environment: + MSDL_NO_TELEMETRY=1 Disable anonymous usage reporting and update checks + MSDL_NO_CONTRIBUTE=1 Disable cache contribution + +More info: https://msdl.tech-latest.com/cli +``` + +--- + +## Documentation Updates + +- **README** — one line under CLI section: "msdl sends anonymous usage counts (action, platform, version) to help us understand which products are popular. Set `MSDL_NO_TELEMETRY=1` to opt out." +- **`/cli` page** — same one-liner in the contribute section, next to the `--no-contribute` note. + +--- + +## What's Not In Scope + +- No per-user tracking, no IP storage, no session IDs +- No telemetry dashboard UI (raw numbers via `/metrics` is sufficient) +- No forced update or auto-update — notification only +- No telemetry for the web frontend