Graph-Based Dependency Intelligence for Predictive Failure Analysis in Large-Scale Distributed Systems

Abstract

Distributed systems with many services fail via the distributed processes of dependent services, and most monitoring systems do not respond to problems until the first symptom occur. This article introduces a graph-based dependency intelligence architecture, which is used to predict failures that are going to impact users. The framework constructs a weighted service dependency graph using traces, metrics and logs, and trains the graph in each time window, and uses a graph neural network to make a prediction about the likelihood of each service failing within the next fifteen minutes. The edge weight is defined by three equations, the message passing rule and the probability of failure and a fourth equation of the evaluation score. The paper provides a survey of the literature on microservice faults, root cause localization and graph learning, and a comparison of six recent diagnosis studies with the prediction gap. A benchmark of 41 services and 126 dependency edges on a cluster of 12 nodes was utilised in the experiment. The team infused 300 failure instances between four classes over 30 days and evaluated the models on 120 unknown instances. The proposed model achieved precision that was equal to 0.89, a recall of 0.87 and an F1 score that was 0.88. A model that was a static graph model achieved 0.79, service level LSTM reached 0.69 and threshold alerting reached 0.56. The suggested model provided a mean warning lead time of 8.7 minutes, reduced the false alarms to 2.1 per day and the true root service was ranked in top three ranked services in 81 percent of episodes. Findings indicate that the dynamic dependency weights provide predictive power over and above the service level signals.