Dr. rer. nat. Axel Uhl

Object-based Internet Search

Internet search engines compute their centralized index by crawling web contents. This approach implies two major problems; large and relevant parts of the Internet content are notreachable by crawling and thus remain inaccessible for search engines, bandwidth and its growth impose harsh limits on central index currency and indexable share of vastly growing available information. The obvious solution is a distributed approach to information retrieval that better leverages the available bandwidth in order to achieve higher index currency and improved coverage, including deep web contents. Forward knowledge -like keyword indices – have to be stored closer to the searchable information sources than the central index approach currently does. Furthermore, their updating has to happen in a more bandwidth-efficient manner as compared to the change detection heuristics and brute force crawling methods used today. Metrics collected have shown that the amount of publicly accessible information on the Internet grows much faster than the available Internet backbone bandwidth. In particular it turns out that the amount of application generated content grows especially fast. Crawling-based technologies for global Internet search won’t be able to scale with these growth factors. Furthermore, many of the application-generated contents are unreachably hidden from the crawlers in the deep web. The solution lies in reversing the paradigm of Internet search. Content providers will have to contribute to the searchability of their information space, thus making search more bandwidth-efficient and making deep web contents accessible to search.