Skip to Main Content

Beyond Google: The Invisible Web: Characterisistics

Characteristics of Invisible Web Content or Why Search Engines Can’t Find this Information

  • Content found in databases – Database content that is dynamically generated as the result of a query cannot be found by general-purpose search engines. Example: ERIC database, Library catalogs.
  • Subscription database content – Fee-based database content is only accessible to those who have subscribed. (Many libraries offer their members free access to subscription databases.) Examples: EBSCOhost databases, LexisNexis Academic.
  • Information offered on very content rich websites – General-purpose search engines only partially index very large (deep) websites. The parts of the website that they do not index become part of the Invisible Web. Examples: Library of Congress, U. S. Census Bureau.
  • Real time content – Information about events currently taking place may not yet be indexed by general-purpose search engines.
  • Formats – Information occurs in various formats, some of which are not indexed by general-purpose search engines. It also takes time for new formats to appear in search engines. Example: Any new format.
  • Sites requiring login authorization – These sites require users to login or identify themselves as having the right to access and use content. Examples: Blackboard, membership sites.
  • Sites with interactive content – These sites require information from the user before they can generate an answer. Examples: Travel direction sites, job hunting sites.
  • New content – It may take time for a search engine to find and include new websites and newly added website content.
  • Sites that are not linked to by other sites - Search engines index websites by following links from one website to another, if there aren't any links to a site it might not be found or included.
  • Sites blocked by Robot Exclusion Protocols – These sites are not intended for open access use.
  • Gated social media communities – Social networking sites such as Facebook, LinkedIn, etc.
  • Dark Web – Sometimes called Deep Web, Tor, or Hidden Wiki, this Dark Web is not to be confused with the Invisible Web as presented in this website. The Dark Web is usually used by those who seek anonymity for their web activities. Can be used by governments, whistleblowers, political protestors and the like but can also be exploited by criminals for illicit purposes.
Characteristics of Invisible Web Content Examples
Database content (dynamically generated for a particular inquiry) ERIC
Library catalogs
Subscription databases EBSCOhost, LexisNexis academic
Deep websites Library of Congress, U.S. Census Bureau
Real time content  
Formats Any new format
Sites that require login Blackboard
Membership sites
Sites that require that forms be filled out Sites offering travel directions, job hunting sites
New content Any new websites or content newly added to an existing website
Sites with a no-index protocol Private websites
Social networking sites Facebook, LinkedIn, etc.
Dark Web sites Tor (The Onion Router), Hidden Wiki, Serval

 

Library Media Resources Center
LaGuardia Community College
31-10 Thomson Avenue, room E101
Long Island City, NY 11101
Email the Library