VMware, Inc.
HIERARCHICAL NOVELTY DETECTION USING INTENDED STATES FOR NETWORK SECURITY

Abstract:

The disclosure provides an approach for detecting and preventing attacks in a network. Embodiments include determining a plurality of network behaviors of a process by monitoring the process. Embodiments include generating a plurality of intended states for the process based on subsets of the plurality of network behaviors. Embodiments include determining a plurality of intended state clusters by applying a clustering technique to the plurality of intended states. Embodiments include determining a state of the process. Embodiments include identifying a given cluster of the plurality of intended state clusters that corresponds to the state of the process. Embodiments include selecting a novelty detection technique based on a size of the given cluster. Embodiments include using the novelty detection technique to determine, based on the given cluster and the state of the process, whether to generate a security alert for the process.