LibGuides: Beyond Google: The Invisible Web: Characterisistics

Characteristics of Invisible Web Content or Why Search Engines Can’t Find this Information

Content found in databases – Database content that is dynamically generated as the result of a query cannot be found by general-purpose search engines. Example: ERIC database, Library catalogs.
Subscription database content – Fee-based database content is only accessible to those who have subscribed. (Many libraries offer their members free access to subscription databases.) Examples: EBSCOhost databases, LexisNexis Academic.
Information offered on very content rich websites – General-purpose search engines only partially index very large (deep) websites. The parts of the website that they do not index become part of the Invisible Web. Examples: Library of Congress, U. S. Census Bureau.
Real time content – Information about events currently taking place may not yet be indexed by general-purpose search engines.
Formats – Information occurs in various formats, some of which are not indexed by general-purpose search engines. It also takes time for new formats to appear in search engines. Example: Any new format.
Sites requiring login authorization – These sites require users to login or identify themselves as having the right to access and use content. Examples: Blackboard, membership sites.
Sites with interactive content – These sites require information from the user before they can generate an answer. Examples: Travel direction sites, job hunting sites.
New content – It may take time for a search engine to find and include new websites and newly added website content.
Sites that are not linked to by other sites - Search engines index websites by following links from one website to another, if there aren't any links to a site it might not be found or included.
Sites blocked by Robot Exclusion Protocols – These sites are not intended for open access use.
Gated social media communities – Social networking sites such as Facebook, LinkedIn, etc.
Dark Web – Sometimes called Deep Web, Tor, or Hidden Wiki, this Dark Web is not to be confused with the Invisible Web as presented in this website. The Dark Web is usually used by those who seek anonymity for their web activities. Can be used by governments, whistleblowers, political protestors and the like but can also be exploited by criminals for illicit purposes.

Characteristics of Invisible Web Content	Examples
Database content (dynamically generated for a particular inquiry)	ERIC Library catalogs
Subscription databases	EBSCOhost, LexisNexis academic
Deep websites	Library of Congress, U.S. Census Bureau
Real time content
Formats	Any new format
Sites that require login	Blackboard Membership sites
Sites that require that forms be filled out	Sites offering travel directions, job hunting sites
New content	Any new websites or content newly added to an existing website
Sites with a no-index protocol	Private websites
Social networking sites	Facebook, LinkedIn, etc.
Dark Web sites	Tor (The Onion Router), Hidden Wiki, Serval