Repairing Fastly / Pulumi State Drift

What to do when pulumi up reports no changes but the live Fastly service is missing configuration that the code clearly declares.

The symptom

You merge a change that adds a VCL snippet (or any other named child object) to a fastly.ServiceVcl. The deploy pipeline goes green. The feature is not live. You run pulumi up again, by hand, and Pulumi tells you there is nothing to do — while the Fastly UI plainly does not have your snippet.

Nothing is broken enough to alarm. Every subsequent deploy will also be green, and the feature will stay missing indefinitely. This has happened twice:

In the second case mit-learn CI build 610 reported SUCCESS while creating none of the snippets. A green build is not evidence that the config is live.

Why it happens

Fastly's snippet endpoint requires a name to start with a letter and contain only alphanumeric, underscore, hyphen, period, and space characters. An illegal name is rejected with a 400:

Name must start with a letter and contain only alphanumeric, underscore,
hyphen, period, and space characters.

That rejection arrives mid-apply, after Pulumi has already cloned a new Fastly service version. The failed run nonetheless persists the full desired snippets set into the resource's state outputs — including the snippet that was never created.

From then on the state is lying, and the provider's diff logic believes it. SetDiff.Diff() keys set elements on name and computes unmodified := oldSet.Intersection(newSet). Process() only ever iterates the Deleted, Added and Modified buckets. An element that is byte-identical between the lying state and the config lands in Unmodified, so it generates no API call — forever. No amount of re-running pulumi up will heal it, because as far as Pulumi is concerned there is nothing to heal.

This is why the repair below leads with a refresh. Refresh is the only thing that can break the tie, because the provider's snippet Read() re-lists the children from the live active version and overwrites the fiction in state.

Prevention (already in place)

Two layers, because they fire at different times:

• tests/ol_infrastructure/lib/test_fastly.py walks src/ with ast and fails if any snippet name literal is illegal, or if a raw fastly.ServiceVclSnippetArgs is constructed outside lib/fastly.py. This is the one that runs before merge, on the PR.
• vcl_snippet() in src/ol_infrastructure/lib/fastly.py validates at construction time and raises, aborting the program before the ServiceVcl resource is registered — so an illegal name never reaches the Fastly API and state cannot be corrupted. This is the runtime backstop that catches a name the static scan cannot see (one built at runtime rather than written as a literal).

The validator is deliberately scoped to snippets only. Fastly validates conditions and request settings more loosely, and we have live ones that this pattern would reject: xpro has a condition named path starts with /images cache condition, and mit-learn a request setting name containing a comma. Do not widen the rule to those endpoints.

Detecting it

There is currently no automated detection. Every Fastly-bearing stack (mitxonline, micromasters, xpro, learn-ai, mit-learn, ocw-site, fastly-redirector, edxapp) runs with refresh_stack=False, precisely because it has Fastly resources — see Why refresh is off. Coverage is zero exactly where the risk lives.

To check a service by hand, compare the names in state against the names live. This needs no admin credential and cannot mutate anything:

# State side -- what Pulumi believes exists.
cd src/ol_infrastructure/applications/mit_learn
pulumi stack export --stack CI | python3 -c '
import json, sys
for r in json.load(sys.stdin)["deployment"]["resources"]:
    if r["type"] == "fastly:index/serviceVcl:ServiceVcl":
        o = r["outputs"]
        print("service", o["id"], "active version", o["activeVersion"])
        for s in sorted(o.get("snippets", []), key=lambda x: x["name"]):
            print(" ", s["name"])
'

# Live side -- what Fastly actually serves. Use the read-only token.
curl -s -H "Fastly-Key: $FASTLY_READ_KEY" \
  "https://api.fastly.com/service/<service-id>/version/<active-version>/snippet" \
  | python3 -c 'import json,sys; [print(" ", s["name"]) for s in sorted(json.load(sys.stdin), key=lambda x: x["name"])]'

Use global_read_api_key from src/bridge/secrets/fastly.yaml, not admin_api_key. A name present in the first list and absent from the second is this bug.

Why refresh is off on these pipelines

refresh_stack=False is set on every Fastly-bearing pipeline because a refresh calls the Fastly API, and during an admin API token rotation it calls it with the stale token and fails the entire deploy job. Turning refresh back on is tracked separately; until then, refresh is a deliberate manual step, which is what makes this runbook necessary.

The repair

Validated on both QA and Production during hq#12449 (mitxonline QA v563→v564, Production v207→v208) and again on mit-learn (QA v1435→v1437, CI v1902→v1904).

Work one stack at a time, and target the Fastly resource explicitly so an unrelated in-flight change cannot ride along.

1. Get the URN of the Fastly service

cd src/ol_infrastructure/applications/<app>
pulumi stack --stack <STACK> --show-urns | grep serviceVcl

It looks like:

urn:pulumi:CI::ol-application-mit-learn::fastly:index/serviceVcl:ServiceVcl::fastly-mit_learn-ci

2. Refresh just that resource

This rewrites state from the live service, and is the step that actually breaks the deadlock.

pulumi refresh --yes --stack <STACK> --target '<urn>'

pulumi refresh does not execute the Pulumi program, so it needs none of the image environment variables the program demands. In the refresh output, a backends: [secret] line is a state-representation change, not a config change — ignore it.

3. Verify state now matches live

Re-run the two commands from Detecting it. State should now be missing the snippet too. That is the point: state has stopped lying, so the next up has real work to do.

4. Preview, and gate on what you see

pulumi preview --diff --stack <STACK> --target '<urn>'

You should see the missing snippet being created and nothing else of substance. If you see anything you did not expect — especially a deletion — stop and work out why before applying.

The program does run for preview and up, and it refuses to start without <APP>_DOCKER_TAG or <APP>_DOCKER_SHA:

OSError: Either MIT_LEARN_DOCKER_TAG or MIT_LEARN_DOCKER_SHA must be set.

Read the currently-deployed value out of the stack rather than inventing one, otherwise an unrelated image roll rides along with your Fastly fix. Read it off the running Deployments, not with a bare grep of the export — the export also contains superseded digests from earlier revisions, and a grep returns all of them with no way to tell which is live:

pulumi stack export --stack <STACK> | python3 -c '
import json, sys
for r in json.load(sys.stdin)["deployment"]["resources"]:
    if r["type"] == "kubernetes:apps/v1:Deployment":
        for c in r["outputs"]["spec"]["template"]["spec"]["containers"]:
            print(r["outputs"]["metadata"]["name"], c["image"])
' | sort -u

All of an app's Deployments normally share one digest. export it rather than prefixing a single command, so the apply in step 5 runs against the same digest you previewed:

export MIT_LEARN_DOCKER_SHA=sha256:...

A one-shot MIT_LEARN_DOCKER_SHA=... pulumi preview ... leaves the variable unset for the next command, and the bare pulumi up below then fails with the OSError above.

5. Apply

pulumi up --yes --stack <STACK> --target '<urn>'

6. Validate against the live service, not against Pulumi

Allow roughly a minute for Fastly edge propagation first. Confirm the active version number advanced and the snippet is present, then exercise the behaviour end to end. For a cache-key change that means checking that a non-whitelisted query parameter collapses to one cache key while a whitelisted one still splits:

for i in 1 2 3; do
  curl -so /dev/null -D - "https://learn.mit.edu/search?q=python&utm_source=t$i" \
    | grep -iE '^(x-cache|age):'
  sleep 1
done
# expect MISS, then HIT with a rising age -- one shared key

Rollback

Fastly versions are immutable, so rollback is instant and does not involve Pulumi: reactivate the previous version in the Fastly UI or API. Then run step 2 again — otherwise Pulumi's state now describes the version you just rolled away from, and you have re-created the same lying-state condition by hand.

That makes the rollback true, not durable. The source still declares the change you just rolled back, so the next ordinary deploy of this stack reapplies it — reverting a bad snippet in the Fastly UI buys you time, nothing more. Revert or fix the declaration in ol-infrastructure before the next deploy lands, or expect the rollback to be silently undone by someone else's unrelated merge.