feat(backend): two-layer in-memory caching to eliminate Microsoft rate-limit blocks #12

New issue

Closed

opened 2026-05-18 17:46:46 +05:30 by starkSV · 0 comments

starkSV commented

2026-05-18 17:46:46 +05:30

(Migrated from github.com)

Background

Our Go backend currently hits Microsoft's software download API on every
user interaction — once on page load (SKU/language list) and once on
button click (signed CDN download URL). Under sufficient traffic, Microsoft
temporarily blocks our server IP with error 715-123130, making the
"Get Download Links" button fail for all users simultaneously.

A Cloudflare Worker is currently in place as a mitigation (see #6), routing
outbound requests through Cloudflare's distributed edge IPs. This issue
proposes the proper fix: an in-memory two-layer cache that reduces Microsoft
API traffic from potentially thousands of requests per day to ~50–100.

Confirmed: Signed URLs are not IP-bound

Testing confirmed that Microsoft's signed CDN URLs (software.download.prss.microsoft.com/...)
are not tied to the requesting IP. Any user can download from a cached
link. This changes the architecture from per-user API proxying to periodic
upstream metadata refresh — a massive scalability improvement.

Proposed Solution: Two-Layer In-Memory Cache

Layer 1 — SKU Info Cache (7-day TTL)

The language/SKU list for a released Windows product is extremely stable.
Microsoft does not add or remove languages after a product ships.

Cache key: product_id
TTL: 7 days
On miss: fetch from Microsoft, store, return
On hit: return instantly, zero Microsoft calls
Self-healing: expired cache repopulates from Microsoft automatically

Layer 2 — Download Link Cache (22-hour TTL)

Microsoft's signed URLs expire after 24 hours. Caching for 22 hours safely
serves the same link to every user who requests the same product+language
combination within a day.

Cache key: product_id:sku_id
TTL: 22 hours ± random jitter (see below)
On miss: fetch from Microsoft, store, return
On hit: return instantly, zero Microsoft calls

Implementation Details

1. Singleflight (critical)

Without this, a cache miss under concurrent load causes a stampede:

cache miss → 500 concurrent requests → 500 Microsoft hits

With golang.org/x/sync/singleflight:

cache miss → 1 Microsoft hit → all 500 requests share the result

var group singleflight.Group

v, err, _ := group.Do(cacheKey, func() (interface{}, error) {
    return fetchFromMicrosoft()
})

2. Dynamic TTL

Instead of a fixed 22h TTL, parse the actual expiry from Microsoft's
response URL (the signed URL contains an expiry timestamp). Derive TTL as:

cache_ttl = (link_expiry - now) - 30min safety margin

Safer against clock skew, edge propagation delays, and timezone issues.

3. Stale-on-failure

If a cache entry is expired and the Microsoft refresh attempt fails
(rate-limited, transient error), serve the stale cached link temporarily
and retry the refresh in the background. Prevents global outages caused
by a single failed refresh.

4. Negative response caching (30–60s)

If Microsoft returns 715-123130 or 429, cache the failure briefly.
Without this, every concurrent user retries immediately, worsening the
block into a thundering herd.

5. TTL jitter

Add a small random offset (±few minutes) to cache expiry times to prevent
synchronized mass-expiry spikes:

Bad:  all Win11 English links expire at exactly 21:00 UTC → spike
Good: 21h ± random few minutes → smooth refresh distribution

6. Cache entry metadata

Store cached_at and expires_at alongside each cached value for
observability and dynamic TTL calculation.

Expected Impact

Metric	Before	After
SKU info requests/day to Microsoft	N per page load	~1 per product per week
Download link requests/day to Microsoft	N per button click	~1 per product/language per day
Total Microsoft API traffic	Thousands/day	~50–100/day
Rate-limit risk	High under traffic	Effectively zero

Relationship to Cloudflare Worker (#6)

The CF Worker stays in place as defense-in-depth — it masks the server's
egress IP and provides a fallback layer. At this cache volume, the Worker's
IP distribution role becomes largely redundant, but the zero-cost protection
is worth keeping.

Implementation scope

All changes are confined to backend/main.go. No frontend changes, no
database, no new infrastructure. Same in-memory pattern as the existing
sessionCache.

New dependency: golang.org/x/sync/singleflight

## Background Our Go backend currently hits Microsoft's software download API on every user interaction — once on page load (SKU/language list) and once on button click (signed CDN download URL). Under sufficient traffic, Microsoft temporarily blocks our server IP with error `715-123130`, making the "Get Download Links" button fail for all users simultaneously. A Cloudflare Worker is currently in place as a mitigation (see #6), routing outbound requests through Cloudflare's distributed edge IPs. This issue proposes the proper fix: an in-memory two-layer cache that reduces Microsoft API traffic from potentially thousands of requests per day to ~50–100. ## Confirmed: Signed URLs are not IP-bound Testing confirmed that Microsoft's signed CDN URLs (`software.download.prss.microsoft.com/...`) are **not tied to the requesting IP**. Any user can download from a cached link. This changes the architecture from per-user API proxying to periodic upstream metadata refresh — a massive scalability improvement. ## Proposed Solution: Two-Layer In-Memory Cache ### Layer 1 — SKU Info Cache (7-day TTL) The language/SKU list for a released Windows product is extremely stable. Microsoft does not add or remove languages after a product ships. - Cache key: `product_id` - TTL: 7 days - On miss: fetch from Microsoft, store, return - On hit: return instantly, zero Microsoft calls - Self-healing: expired cache repopulates from Microsoft automatically ### Layer 2 — Download Link Cache (22-hour TTL) Microsoft's signed URLs expire after 24 hours. Caching for 22 hours safely serves the same link to every user who requests the same product+language combination within a day. - Cache key: `product_id:sku_id` - TTL: 22 hours ± random jitter (see below) - On miss: fetch from Microsoft, store, return - On hit: return instantly, zero Microsoft calls ## Implementation Details ### 1. Singleflight (critical) Without this, a cache miss under concurrent load causes a stampede: ``` cache miss → 500 concurrent requests → 500 Microsoft hits ``` With `golang.org/x/sync/singleflight`: ``` cache miss → 1 Microsoft hit → all 500 requests share the result ``` ```go var group singleflight.Group v, err, _ := group.Do(cacheKey, func() (interface{}, error) { return fetchFromMicrosoft() }) ``` ### 2. Dynamic TTL Instead of a fixed 22h TTL, parse the actual expiry from Microsoft's response URL (the signed URL contains an expiry timestamp). Derive TTL as: ``` cache_ttl = (link_expiry - now) - 30min safety margin ``` Safer against clock skew, edge propagation delays, and timezone issues. ### 3. Stale-on-failure If a cache entry is expired and the Microsoft refresh attempt fails (rate-limited, transient error), serve the stale cached link temporarily and retry the refresh in the background. Prevents global outages caused by a single failed refresh. ### 4. Negative response caching (30–60s) If Microsoft returns 715-123130 or 429, cache the failure briefly. Without this, every concurrent user retries immediately, worsening the block into a thundering herd. ### 5. TTL jitter Add a small random offset (±few minutes) to cache expiry times to prevent synchronized mass-expiry spikes: ``` Bad: all Win11 English links expire at exactly 21:00 UTC → spike Good: 21h ± random few minutes → smooth refresh distribution ``` ### 6. Cache entry metadata Store `cached_at` and `expires_at` alongside each cached value for observability and dynamic TTL calculation. ## Expected Impact | Metric | Before | After | |---|---|---| | SKU info requests/day to Microsoft | N per page load | ~1 per product per week | | Download link requests/day to Microsoft | N per button click | ~1 per product/language per day | | Total Microsoft API traffic | Thousands/day | ~50–100/day | | Rate-limit risk | High under traffic | Effectively zero | ## Relationship to Cloudflare Worker (#6) The CF Worker stays in place as defense-in-depth — it masks the server's egress IP and provides a fallback layer. At this cache volume, the Worker's IP distribution role becomes largely redundant, but the zero-cost protection is worth keeping. ## Implementation scope All changes are confined to `backend/main.go`. No frontend changes, no database, no new infrastructure. Same in-memory pattern as the existing `sessionCache`. New dependency: `golang.org/x/sync/singleflight`