Skip to content

Fix WebSocket exception leaks after kill -9 termination #60

Description

@yhl-cs

Describe the Bug

When establishing a WebSocket connection over SSL, if the server process is abruptly terminated via kill -9, the reverse proxy cannot gracefully handle the exception.

To Reproduce

  1. Run the server:
# ws-server.py
import asyncio

from fastapi import FastAPI, WebSocket

app = FastAPI()


@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    print("Client connected")

    response = await websocket.receive()
    print(f"Received: {response}")

    await asyncio.sleep(100)

    await websocket.send_text("Success: The request completed after 100 seconds.")
    print("Response sent to client")


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        app, host="0.0.0.0", port=8766, ssl_certfile="cert.pem", ssl_keyfile="key.pem"
    )
$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=localhost"
$ python ./ws-server.py 
INFO:     Started server process [32042]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on https://0.0.0.0:8766 (Press CTRL+C to quit)
  1. Run the reverse proxy:
# ws-rs.py
from fastapi_proxy_lib.fastapi.app import reverse_ws_app
from httpx import AsyncClient


app = reverse_ws_app(AsyncClient(verify=False), base_url="https://127.0.0.1:8766/")

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8765)
$ python ./ws-rs.py 
INFO:     Started server process [31689]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8765 (Press CTRL+C to quit)
  1. Run the Client:
# ws-client.py
import asyncio
import httpx
import httpx_ws

async def websocket_client():
    url = "ws://127.0.0.1:8765/ws"
    async with httpx.AsyncClient() as client:
        async with httpx_ws.aconnect_ws(url, client) as websocket:
            print("Connected to WebSocket server")

            message = "Hello, WebSocket Server!"
            await websocket.send_text(message)
            print(f"Sent: {message}")

            response = await websocket.receive_text()
            print(f"Received: {response}")


async def main():
    await websocket_client()


if __name__ == "__main__":
    asyncio.run(main())
$ python ./ws-client.py 
Connected to WebSocket server
Sent: Hello, WebSocket Server!
  1. Terminate the server process:
$ ps aux | grep python
root        1560  0.0  0.6 481600 21056 ?        Ssl  15:54   0:01 /usr/bin/python3 -Es /usr/sbin/tuned -l -P
root       31689  1.2  1.7 277760 53376 pts/0    S+   18:17   0:02 python ./ws-rs.py
root       32042  3.0  1.6 274880 49984 pts/1    S+   18:19   0:02 python ./ws-server.py
root       32122  1.0  1.1 256320 35776 pts/3    S+   18:19   0:00 python ./ws-client.py
root       32267  0.0  0.0 214080  1536 pts/2    S+   18:20   0:00 grep python
$ kill -9 32042
  1. Check the reverse proxy status(throw out unexcepted error):
$ python ./ws-rs.py 
INFO:     Started server process [31689]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8765 (Press CTRL+C to quit)
INFO:     ('127.0.0.1', 56556) - "WebSocket /ws" [accepted]
Exception is not set. when close ws connection. client: <Task pending name='client_to_server_task' coro=<_wait_client_then_send_to_server() running at /root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py:281> wait_for=<Future pending cb=[Task.task_wakeup()]>>, server:<Task pending name='server_to_client_task' coro=<_wait_server_then_send_to_client() running at /root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py:306> wait_for=<Future pending cb=[Task.task_wakeup()]>>
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/root/venv/lib/python3.12/site-packages/uvicorn/protocols/websockets/wsproto_impl.py", line 235, in run_asgi
  |     result = await self.app(self.scope, self.receive, self.send)  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 152, in __call__
  |     await self.app(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 362, in handle
  |     await self.app(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 95, in app
  |     await wrap_app_handling_exceptions(app, session)(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 93, in app
  |     await func(session)
  |   File "/root/venv/lib/python3.12/site-packages/fastapi/routing.py", line 383, in app
  |     await dependant.call(**solved_result.values)
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/fastapi/router.py", line 112, in ws_proxy
  |     return await proxy.proxy(websocket=websocket, path=path)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py", line 806, in proxy
  |     return await self.send_request_to_target(
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py", line 567, in send_request_to_target
  |     async with stack:
  |   File "/opt/python/lib/python3.12/contextlib.py", line 754, in __aexit__
  |     raise exc_details[1]
  |   File "/opt/python/lib/python3.12/contextlib.py", line 737, in __aexit__
  |     cb_suppress = await cb(*exc_details)
  |                   ^^^^^^^^^^^^^^^^^^^^^^
  |   File "/opt/python/lib/python3.12/contextlib.py", line 231, in __aexit__
  |     await self.gen.athrow(value)
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1349, in aconnect_ws
  |     async with _aconnect_ws(
  |   File "/opt/python/lib/python3.12/contextlib.py", line 231, in __aexit__
  |     await self.gen.athrow(value)
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1254, in _aconnect_ws
  |     async with session:
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 658, in __aexit__
  |     await self._exit_stack.aclose()
  |   File "/opt/python/lib/python3.12/contextlib.py", line 696, in aclose
  |     await self.__aexit__(None, None, None)
  |   File "/opt/python/lib/python3.12/contextlib.py", line 754, in __aexit__
  |     raise exc_details[1]
  |   File "/opt/python/lib/python3.12/contextlib.py", line 737, in __aexit__
  |     cb_suppress = await cb(*exc_details)
  |                   ^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1041, in _background_receive
    |     await self.close(CloseReason.INTERNAL_ERROR, "Stream error")
    |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 985, in close
    |     await self.stream.write(data)
    |   File "/root/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 365, in write
    |     await self._stream.write(buffer, timeout)
    |   File "/root/venv/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 50, in write
    |     await self._stream.send(item=buffer)
    |   File "/root/venv/lib/python3.12/site-packages/anyio/streams/tls.py", line 212, in send
    |     await self._call_sslobject_method(self._ssl_object.write, item)
    |   File "/root/venv/lib/python3.12/site-packages/anyio/streams/tls.py", line 172, in _call_sslobject_method
    |     raise EndOfStream from None
    | anyio.EndOfStream
    +------------------------------------

Expected Behavior

Exceptions should be gracefully handled, not thrown out (or exposed).

Configuration

  • Python 3.12.3
  • httpx 0.28.1
  • httpx-ws 0.7.2
  • FastAPI 0.115.6
  • fastAPI-proxy-lib 0.3.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions