International Business Machines Corporation
Anomaly identification in log files

Abstract:

Described are techniques for identifying anomalies in log files. The techniques including a method encompassing mapping a plurality of log entries into a bag-of-words matrix and determining respective cosine similarities between consecutive log entries in the bag-of-words matrix. The method further includes converting respective cosine similarities to respective states and determining respective state transitions between consecutive log entries. The method further includes generating a stochastic matrix based on the respective state transitions and identifying an anomalous log entry based on an anomalous state transition. The anomalous state transition has a probability below a probability threshold, and the probability is based on the stochastic matrix.