Zscaler, Inc.
Web crawler systems and methods to efficiently detect malicious sites

Abstract:

Systems and methods include receiving a list of web sites; anonymously browsing to each web site in the list; receiving a response based on the browsing; and analyzing the response to classify each web site as malicious or not based on a plurality of techniques including JavaScript (JS) obfuscation detection based on de-obfuscation. The systems and methods can further include providing a blacklist of web sites classified as malicious. The systems and methods can further include determining the list of web sites periodically based on a plurality of factors. The JS obfuscation detection can be performed by de-obfuscating JS content and utilizing heuristics to determine if the de-obfuscated JS content is malicious, and the heuristics can include a presence of any of a new JS function and a domain in the de-obfuscated JS content.