diff --git a/source/administration-guide/scale/common-configure-mattermost-for-enterprise-search.rst b/source/administration-guide/scale/common-configure-mattermost-for-enterprise-search.rst index 753588f02ab..25ee58efb8b 100644 --- a/source/administration-guide/scale/common-configure-mattermost-for-enterprise-search.rst +++ b/source/administration-guide/scale/common-configure-mattermost-for-enterprise-search.rst @@ -35,7 +35,7 @@ Set **Enable Elasticsearch for search queries** to ``true``, and setting **Enabl .. warning:: - For high post volume deployments, we also strongly recommend *disabling* Database Search once Elasticsearch or AWS OpenSearch is fully configured and running. The Mattermost Server will fall back on Database search if ElasticSearch or OpenSearch are unavailable which can lead to performance degradation on high post volume deployments. + For high post volume deployments, we also strongly recommend *disabling* Database Search once Elasticsearch or AWS OpenSearch is fully configured and running. The Mattermost Server will fall back on Database search if Elasticsearch or OpenSearch are unavailable, which can lead to performance degradation on high post volume deployments. From Mattermost v11.7, the server proactively detects outages through periodic health checks and falls back to database search on the first health check failure, rather than waiting for requests to time out. See the :ref:`outage handling FAQ ` for details. Once the configuration is saved, new posts made to the database are automatically indexed on the Elasticsearch or AWS OpenSearch server. diff --git a/source/administration-guide/scale/enterprise-search.rst b/source/administration-guide/scale/enterprise-search.rst index b239b2cfff4..49109a638f8 100644 --- a/source/administration-guide/scale/enterprise-search.rst +++ b/source/administration-guide/scale/enterprise-search.rst @@ -129,6 +129,49 @@ From Mattermost v11, :doc:`Support Packet generation ` for search engine outages. See :doc:`performance monitoring metrics ` for details. + +During an outage, you may see the following log messages: + +.. list-table:: + :widths: 15 45 40 + :header-rows: 1 + + * - Level + - Log message + - Meaning + * - Error + - ``Search engine health check failed repeatedly; stopping engine`` + - The failure threshold was reached and the engine has been stopped. Search falls back to the database. + * - Warn + - ``Search engine health check failed`` + - An individual health check failed. Includes a ``consecutive_failures`` count. + * - Warn + - ``Search engine health check failed: it is now marked as unhealthy`` + - A previously healthy engine failed a health check and has been marked unhealthy. Search requests will fall back to the database immediately. + * - Warn + - ``Search engine watcher: Start() failed, will retry`` + - A reconnection attempt failed. Includes a ``next_backoff`` field indicating the time until the next retry. + * - Info + - ``Search engine health check succeeded: it is now marked as healthy`` + - The engine passed a health check after being unhealthy and is now handling search requests again. + * - Info + - ``Search engine watcher: engine started successfully`` + - The engine has recovered and is active again. + * - Info + - ``Search engine watcher: engine disabled, parking`` + - The health monitor is idle because the search engine is disabled in configuration. + My search indexes won't complete, what should I do? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/administration-guide/scale/performance-monitoring-metrics.rst b/source/administration-guide/scale/performance-monitoring-metrics.rst index dda3ceb64f2..14e89995d2a 100644 --- a/source/administration-guide/scale/performance-monitoring-metrics.rst +++ b/source/administration-guide/scale/performance-monitoring-metrics.rst @@ -118,6 +118,7 @@ Search metrics - ``mattermost_search_post_index_total``: The total number of posts indexes carried out. - ``mattermost_search_posts_searches_total``: The total number of post searches carried out. - ``mattermost_search_user_index_total``: The total number of user indexes carried out. +- ``mattermost_search_engine_status``: Status of the configured search engine: ``1`` = healthy or not configured, ``0`` = configured but unavailable. Use this metric to set up alerts for search engine outages. WebSocket metrics ~~~~~~~~~~~~~~~~~