zipdeploy with WEBSITE_RUN_FROM_PACKAGE=1 removes the active package before validating the replacement — a failed deploy leaves the host unable to start

[roboro-tom]

Summary

With WEBSITE_RUN_FROM_PACKAGE=1, a failed zipdeploy can leave the Function App unable to start, because the previously-active package is removed from SitePackages before the replacement has been validated. A deploy that fails should be non-destructive; this one turned into a 90-minute production outage.

Environment

• Windows Function App, Elastic Premium plan
• WEBSITE_RUN_FROM_PACKAGE=1 (local SitePackages, not a blob URL)
• Deployed via Kudu zipdeploy (through Azure/functions-action)
• ~218 MB deployment package (Node, large node_modules plus bundled binaries)

What happened

Three consecutive deploys failed with HTTP 500 from /api/zipdeploy. After all three, the app crash-looped on:

Shutting down host due to presence of
C:\home\site\wwwroot\FAILED TO INITIALIZE RUN FROM PACKAGE.txt
File content: Run From Package Initialization failed.

Inspecting data/SitePackages explained why:

packagename.txt   (mtime = the LAST SUCCESSFUL deploy)  ->  "<previous>.zip"

SitePackages/ actual contents:
  <attempt-1>.zip   49,708,689 bytes   truncated
  <attempt-2>.zip   50,078,965 bytes   truncated
  <attempt-3>.zip   50,250,385 bytes   truncated
  packagename.txt           18 bytes
  ^^ "<previous>.zip" — the package the pointer names — is ABSENT

So the pointer still referenced the package that had been serving successfully, but that file was gone, while none of the three replacements ever completed. Each partial zip has a valid PK\003\004 local header and no end-of-central-directory record — i.e. cut off mid-write.

The app kept running for ~77 minutes after the first failed deploy (the already-mounted package stayed live), then a worker recycle tried to re-mount, found nothing, and wrote the failure marker. Every restart from then on hit the same marker.

Why the uploads failed (contributing, but not the defect being reported)

Writes through the Kudu path were extremely slow at the time. Measured against the same file share, from the same client, minutes apart:

Path	Payload	Throughput	Result
Kudu VFS	5 MB	1.06 MB/s	201
Kudu VFS	20 MB	0.83 MB/s	201
Kudu VFS	218 MB	~1 MB/s	502, truncated
Azure Files (direct, same share)	218 MB	~38 MB/s	complete

Small writes succeeded, so the path wasn't broken — just ~40× slower than the storage behind it, which meant anything over ~80 MB could not finish inside the SCM gateway's ~90s limit. Azure Resource Health reported the storage account healthy throughout.

That slowness is presumably environment-specific and I'm not asking for it to be diagnosed here. The defect is that a deploy which cannot complete destroys the working artifact.

Expected behaviour

A zipdeploy that fails to land its new package should leave the app running the package it already had. Concretely: don't prune/remove the currently-referenced package until the replacement has been written completely, validated, and packagename.txt flipped.

Actual behaviour

The active package is gone, packagename.txt still names it, and the host has nothing to mount on its next recycle. There is also no supported recovery from that state short of manually writing a package to the share and repointing packagename.txt, which is what I ended up doing.

Notes

• Nothing was dead-lettered or logged as an error at the moment of removal — the loss is silent until the next worker recycle, which can be over an hour later. That delay makes the deploy failure and the outage look unrelated.
• Switching this app to blob-backed WEBSITE_RUN_FROM_PACKAGE=<url> removes the exposure entirely, since each deploy is then an immutable blob and the swap is a single setting flip with the previous package still intact. That's the workaround I've adopted, and it may be worth documenting as the recommended pattern for large packages.