Morgan Stanley
File indexing and retrospective malware detection system

Abstract:

A computer-implemented method for indexing a stream of files is disclosed. The method comprises receiving a file; generating a set of atomic indexes based on the file contents; storing the atomic indexes in a current index; and if the current index reaches a threshold criterion, freezing the current index into a read-only form, propagating the current index to one or more distributed databases, and generating a new index for future insertions. In some embodiments, the method further comprises one or more of providing an interface to query the databases for files matching a particular signature; searching the databases using a YARA-specified signature; converting a user-provided signature in the YARA format to an index-acceleratable format; retrieving a set of files that are possible matches of the particular signature; verifying that each file of the set of files is a match of the particular signature; and providing the files through the interface.