Skip to content

vllm.entrypoints.serve.elastic_ep.middleware

Classes:

  • ScalingMiddleware –

    Pause serving while scaling or after the local rank is retired.

ScalingMiddleware

Pause serving while scaling or after the local rank is retired.

Health and scaling-status requests remain available so an external orchestrator can observe the operation and remove a retired process.

Source code in vllm/entrypoints/serve/elastic_ep/middleware.py 
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
class ScalingMiddleware:
    """Pause serving while scaling or after the local rank is retired.

    Health and scaling-status requests remain available so an external
    orchestrator can observe the operation and remove a retired process.
    """

    def __init__(self, app: ASGIApp) -> None:
        self.app = app

    def __call__(self, scope: Scope, receive: Receive, send: Send) -> Awaitable[None]:
        if scope["type"] != "http":
            return self.app(scope, receive, send)

        # Keep observability available while serving is paused or this rank is
        # waiting for the external orchestrator to remove its process.
        rank_retired = get_elastic_ep_rank_retired()
        serving_paused = get_scaling_elastic_ep() or rank_retired
        if serving_paused and scope["path"] not in _SCALING_OBSERVABILITY_PATHS:
            error = (
                "This data-parallel rank has been retired and is awaiting shutdown."
                if rank_retired
                else "The model is currently scaling. Please try again later."
            )
            response = JSONResponse(
                content={"error": error},
                status_code=503,
            )
            return response(scope, receive, send)

        return self.app(scope, receive, send)