Skip to content

Unrecovered panic in WASM plugin loading crashes entire BFE process #1287

Description

@dhruthan

Describe the bug

A malformed .wasm plugin file causes an unrecovered panic that kills the entire BFE process. This happens both during startup (when mod_wasm loads plugins) and during config reload via /reload/mod_wasm. A single bad plugin file causes a full outage — all traffic is dropped.

There are two issues in the WASM plugin loading path:

  1. bfe_wasmplugin/plugin.go:251 — uses panic(err) on RegisterImports failure, while every other error in the same function uses log.Error + continue/break
  2. proxy-wasm-go-host/wazero runtime can panic on malformed WASM binaries during Module.Init(), and nothing in BFE's call chain catches it

To Reproduce

  1. Build BFE with mod_wasm enabled
  2. Create a corrupted WASM plugin:
mkdir -p conf/wasm_plugin/bad_plugin
cp conf/wasm_plugin/headers/headers.wasm conf/wasm_plugin/bad_plugin/bad_plugin.wasm
printf '\xff\xff\xff\xff' | dd of=conf/wasm_plugin/bad_plugin/bad_plugin.wasm bs=1 seek=200 conv=notrunc
echo '{}' > conf/wasm_plugin/bad_plugin/bad_plugin.conf
md5sum conf/wasm_plugin/bad_plugin/bad_plugin.wasm | awk '{print $1}' > conf/wasm_plugin/bad_plugin/bad_plugin.md5
  1. Configure mod_wasm to load it in conf/mod_wasm/mod_wasm.data:
{
    "Version": "999",
    "BeforeLocationRules": [],
    "ProductRules": {},
    "PluginMap": {
        "bad_plugin": {
            "Name": "bad_plugin",
            "WasmVersion": "1",
            "ConfVersion": "1",
            "InstanceNum": 1
        }
    }
}
  1. Start BFE:
./bfe -c conf/ -l log/

Result is BFE crashes immediately:

panic: section type: read 19-th type: could not read parameter types: invalid value type: 255

goroutine 1 [running]:
github.com/bfenetworks/proxy-wasm-go-host/wazero.(*Module).Init(...)
    wazero/module.go:49
github.com/bfenetworks/bfe/bfe_wasmplugin.NewWasmPlugin(...)
    bfe_wasmplugin/plugin.go:173
github.com/bfenetworks/bfe/bfe_modules/mod_wasmplugin.buildNewPluginMap(...)
    bfe_modules/mod_wasmplugin/plugin_rule_load.go:120
...
main.main()
    bfe.go:124

exit status 2

The same crash also happens via hot reload while BFE is running:

curl http://localhost:8421/reload/mod_wasm
# BFE process dies, all traffic dropped

Expected behavior

BFE should log an error and reject the bad plugin gracefully - not crash. The call chain has no recover(), so the panic propagates all the way up and kills the process.

Proposed Fix

I have a working fix ready to PR:

  1. plugin.go: Replace panic(err) with log.Logger.Error + break (matching the existing pattern in EnsureInstanceNum)
  2. plugin_rule_load.go: Add safeNewWasmPlugin() wrapper with defer recover() to catch panics from the wasm runtime

After the fix, BFE handles it gracefully:

PRODUCTION DEMO: BFE with WASM panic recovery

1. BFE running with valid WASM plugin:  ALIVE
3. Loading MALFORMED .wasm via hot reload: {"error":"panic loading wasm plugin bad_plugin: ..."}
4. After bad reload -  STILL ALIVE
6. Restoring valid WASM plugin:   {"error":null}
7. BFE recovered?   Yes its ALIVE AND HEALTHY

Additional context

  • The proxy-wasm-go-host/wazero runtime panics in Module.Init() on invalid WASM bytecode - this is upstream behavior that BFE should defend against
  • The panic(err) in plugin.go:251 is inconsistent with all adjacent error handling in the same function

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions