200 OK from Gateway Means Nothing

Hello, Habr! I'm Mikhail Shpakov, and I manage Statuser, a website availability monitoring service. Previously, I shared the journey of its development and insights gained from monitoring 6,500 sites over a year. Recently, a Statuser user reached out regarding an issue with their product. Their AI model was responsible for interpreting customer inquiries and generating draft responses for operators. At one point, this feature ceased to function, yet a standard HTTP monitor pointed to a green status for the AI gateway. The support operators reported the issue, not the monitoring system. The root cause was straightforward: the provider had taken the selected model offline, but the gateway remained operational. The HTTP monitor was making a typical GET request, receiving a successful response without checking the model's status. Diagnosing such a failure is easy if you know where to look, but recognizing it initially is more challenging. There exists an additional layer between the accessible gateway and the functioning product feature—the model itself. When the model stops responding, the entire service usually remains active, but specific functionalities, like suggestions and chat responses, fail. This incident highlighted the need for a distinct type of monitoring that not only checks the gateway address but also sends a genuine request to the selected model to verify its response. Thus, AI model monitoring was introduced in Statuser.