User-assisted synergic crawling and wrapping for entity discovery in vertical domains

Abstract

Large data-intensive web sites publish considerable quantities of information stored in their structured repositories. Data is usually rendered in numerous data- rich pages using templates or automatic scripts. This wealth of information is of wide interest to many applications and online services that do not have direct access to the structured data repositories. Therefore, there’s a great need to lo- cate such pages, accurately and efficiently extract data on them, and store it in a structured format more adapted to automatic processing than HTML. In this context, we exploit intra- and inter-web site information redundancy to address the problem of locating relevant data-rich pages and inferring wrappers on them, while incurring a minimum user overhead. In the first part, we propose to model large data-intensive web sites, to crawl only the subset of pages pertaining to one vertical domain, and then build effective wrappers for attributes of interest on them, with minimum user effort. Our methodology for synergic specification and execution of crawlers and wrappers is supported by a working system devoted to non-expert users, built over an active-learning inference engine. In the second part, we use the information gathered during inference on the training site, to automatically discover new similar sources on the same type of entities of the vertical domain, which can be useful to complement, enrich, or ver- ify the collected data. Our proposed approach performs an automated search and filter operation by generating specific queries and analyzing the returned search engines results. It combines exploiting existing attributes, template, and page information with a semantic, syntactic, and structural evaluation of newly discov- ered pages to identify relevant semi-structured sources. Both techniques are validated with extensive testing on a variety of sources from different vertical domains.