DIGIMIND

Discover and exploit the Invisible Web for competitive intelligence

The 01/03/2006

Download : "Discover and exploit the Invisible Web for competitive intelligence"
What is Invisible Web (Deep Web) : If you use simply general (as opposed to specialist) search engines and directories to conduct research and intelligence projects on the web, you will be missing out on hundreds of thousands of valuable sources….

Why?
This is because search engines such as Google, MSN or Yahoo! Search and directories such as Yahoo! Directory only give you access to a small part (less than 10%) of the web, the Visible Web. The technology of these conventional search engines does not provide a gateway into a vast area of the web, the Invisible web, much larger than the Visible Web.

Let us think of Antarctic expeditions where experts wish to take ice samples from icebergs: if they focus only on the part above the surface they are failing to take advantage of the huge mass beneath the surface, on average 50 times larger.
It’s the same with the web! If you remain within the confines of the Visible Web for your intelligence, you are ignoring an invisible area representing 500 times the volume and containing hundreds of thousands of high value resources.

Indeed the resources on the Invisible web are generally of better quality and more relevant than those on the Surface Web. Why should this be?
They have been written or validated by experts, authorities in their particular area of expertise.

This document presents the concept and particularities of the Invisible web. Furthermore, it explains how to find sites that belong to this deep web.
Finally, it demonstrates why Digimind’s approach can help you implement an industrial process for monitoring Invisible Web resources, thanks to its cutting-edge technologies and consultancy services.

Table of Contents :

• The Invisible Web: concepts and definition
  • the internet is not the Web
  • the visible Web
  • the Invisible Web: definition and structure
  • Invisible Web: hidden or deep?
• Characteristics of the Invisible Web: size and quality
  • the size of the visible and Invisible Webs: estimates
  • content and quality of the Invisible Web
• How to identify sites on the Invisible Web
  • identifying sources via catalogs and specialist directories
  • searching for specialized resources using your own queries
  • example of Invisible Web resources
• Digimind and monitoring the Invisible Web
  • particularities of automated monitoring of the Invisible Web
  • Comparative table of principle Invisible Web search / surveillance tools
  • advantages of digimind’s approach for monitoring the Invisible Web