Intuit Inc.
System and method for automated domain-extensible web scraping

Abstract:

An automated extensible scraping script is generated for web scraping that is extensible to a plurality of domains. Web sites are classified based on common extracted domain data, further clustering the data based on common navigation structures, and using such commonalities to automate the generation of scraping code based on predefined and reusable code snippets for specific parts of the web sites. Scraping services include a mapper module and a script generator module. Building blocks include a data model updater, a navigation model generator and a navigation model matcher. An administrative module includes domain clustering and configuration file maintenance.