International Business Machines Corporation
ANOMALY IDENTIFICATION IN LOG FILES

Abstract:

Described are techniques for identifying anomalies in log files. The techniques including a method comprising mapping a plurality of log entries into a bag-of-words matrix and determining respective cosine similarities between consecutive log entries in the bag-of-words matrix. The method further comprises converting respective cosine similarities to respective states and determining respective state transitions between consecutive log entries. The method further comprises generating a stochastic matrix based on the respective state transitions and identifying an anomalous log entry based on an anomalous state transition. The anomalous state transition has a probability below a probability threshold, and the probability is based on the stochastic matrix.