I wonder if there are further developments than are given in this Guardian
article? the date of my archived file is 27 February 2004.
I would be interested to know what, if any, use people have made of the
resources indicated in this article, up to now, I had totally forgotten this
file, it happens to be in an archive saved from my Windows98SE desktop,
which effectively died back in 2007.
There are more websites than those seen with the naked eye. Chris
Sherman explores cyberspace
More internet news
Chris Sherman
Guardian
Thursday September 6, 2001
Like the universe, the web is full of dark matter. Astronomers tell us
that even using the world's most powerful telescope, less than 25% of the
matter in the universe is visible. Likewise, the vast majority of
information on the web is invisible to even the most powerful search
engines. This "invisible web" is not only huge, it is likely growing at a
much faster rate than the visible web with which you are familiar.
What exactly is the invisible web? Quite simply, it is made up of
information that search engines either cannot or will not add to their web
indexes.
We are not talking about information of dubious value, such as spam,
porn or Uncle Edwin's trout fishing gallery. A big chunk of the invisible
web consists of first-rate information maintained by reputable,
authoritative sources - treasure troves for the users that can find it.
There are both technical and business reasons why some web content is
excluded from search engines. Search engines make use of programs called
"crawlers" to find pages to add to their indexes. Crawlers aren't very smart
- they are designed to find simple, text-based web pages. They generally
cannot handle other file types, such as images, sounds or video clips. These
types of non-text files make up a small portion of the invisible web.
Other file types are crawlable, but user demand is so small it is not
worth the expense for the search engine to fetch and store them. For
example, many academic papers are available on the web in PostScript or PDF
format. But, of the major search engines, only Google bothers to index this
kind of file. They are invisible to most of the other engines.
The bulk of the invisible web is made up of databases that can be
accessed via the web, but that search engine crawlers cannot penetrate. To
access information from a database, you need to interact with a form that
tells the database what you want. Unfortunately, search engine crawlers
cannot type. When a crawler runs into a web page that is the gateway to a
database, it is as if it has run smack into the front door of a vast library
with securely bolted doors.
There are hundreds of thousands of web-accessible databases, together
containing terabytes of information completely hidden from the search
engines. New techniques are emerging that will gradually allow search
engines to peer into invisible web databases, but once these technical
problems are solved, we will be faced with a new problem. Most searchers are
already overwhelmed when presented with millions of results in response to a
query. When currently invisible resources are added into the mix and the
number of search results potentially reaches into the billions or trillions,
information overload is all but guaranteed.
Does size really matter?
How big is the visible web, compared to the invisible web? Google, the
search engine with the largest web index, allows you to search about 1.4bn
pages - equivalent to a stack of paper more than 100 miles tall. But Google
has not actually fetched and indexed all of those pages, relying instead on
a certain amount of wizardry to know what some pages are about without
actually reading them.
Officials from Google and the other major engines readily admit they
have not indexed the entire web. Aggregating estimates from several
reputable sources puts the size of the visible web at somewhere between 2.5
and 4bn pages, growing at the rate of about 7m a day.
What about the invisible web? A widely quoted study purporting to look
at the invisible web makes Google look like a veritable slacker when it
comes to charting the web.
The study, performed by a company called BrightPlanet to help market
its search products, asserts that what it calls the "deep" web is actually
400 to 550 times larger than the visible web, containing more than a
half-trillion documents.
These numbers seem exaggerated, as the study equates raw data from
earth-observing satellites and oceanographic measurements with the contents
of text-oriented databases, using the convenient device of simply dividing
the size of the database by the size of an average web page. While raw data
is undoubtedly valuable to some, it is noise to the majority of web users,
and should not be counted in the same way as refined information.
Indeed, the whole concept of a web "page" is somewhat questionable
when you are dealing with content that is dynamically generated in response
to a database query. Even when you are dealing with static information, the
same "page" has a larger file size in Microsoft Word format than in plain
text, and larger still when encoded as a PostScript or PDF file.
Nevertheless, normalising for both data type and format yields a more
reasonable esti mate that puts the size of the invisible web at between two
and 50 times larger than the visible web. No matter how you measure it, the
invisible web is much larger than the visible web.
The upshot is that using a search engine to find some kinds of
information is much like the drunk looking for his lost keys under the
streetlamp not because he expects to find his keys there, but because the
light is better. No matter how hard you look or how cleverly you manipulate
the controls, a vast amount of information on the web is simply unfindable
with a general purpose search engine.
How to navigate the invisible web
What types of things can you find on the invisible web that cannot be
found by search engines? The range of information is boundless. The
invisible web is home to patent records, telephone directories, interactive
maps and directions, airline arrival information, stock prices, census data,
art galleries, current news_ the list goes on and on. To really get a sense
of what is available, you need to spend time exploring.
Fortunately, there are pathfinders that can guide you through the
realm of the invisible web.
Search engines are indispensable tools for navigating the web. But
they are almost completely blind when it comes to helping you explore the
web's own terra incognita, the hidden realms of the invisible web.
To fully appreciate the riches available on the invisible web, to
paraphrase the famous slogan, you must boldly go where no search engine has
gone before.
Path finders
Direct search
http://gwu.edu/~gprice/direct.htm
Direct search is a growing compilation of links to the search
interfaces of resources that contain data not easily or entirely
searchable/accessible from general search tools such as Alta Vista, Google,
and HotBot. The goal of direct search is to get as close as possible to the
search form offered by a Web resource (rather than having to click through
one or two pages to get there); hence the name "direct search".
InvisibleWeb.com
www.invisibleWeb.com
The InvisibleWeb Catalog contains more than10,000 databases and
searchable sources frequently overlooked by traditional searching. Each
source is analysed and described by editors to ensure that every user of the
InvisibleWeb Catalog will find reliable information on hundreds of topics,
from airfares to Yellow Pages. All of this material can be accessed easily
by Quick or Advanced Search features or a browsable index of the
InvisibleWeb Catalog. Unlike other search engines, it takes you directly to
the searchable source within a web site, even generating a search form for
you to perform your query.
Librarians' Index to the Internet
www.lii.org
The Librarians' Index to the Internet is a searchable, annotated
subject directory of more than 7,000 internet resources selected and
evaluated by librarians for their usefulness to users of public libraries.
LII only includes links to the very best net content. While not a "pure"
Invisible Web pathfinder, LII categorises each resource as best of,
directories, databases, and specific resources. Databases, of course, are
Invisible Web resources. By using LII's advanced search feature, you can
limit your search to return only databases in the results list. Advanced
search also lets you restrict your results to specific fields of the
directory (author name, description, title, URL). The Librarians' Index to
the Internet is a laser-sharp searching tool for finding Invisible Web
databases.
ProFusion
www.profusion.com
ProFusion is a meta search engine from Intelliseek, the same company
that runs InvisibleWeb.com. In addition to providing a sophisticated
simultaneous search capability for the major search engines, ProFusion
provides access to the Invisible Web with the ability to search more than
1,000 targeted sources of information, including TerraServer, Adobe PDF
Search, Britannica.com, NY Times and the US Patent database.
Alpha Search
www.profusion.com
The primary purpose of AlphaSearch is to access the finest internet
"gateway" sites. The authors of these "gateway" sites have spent significant
time gathering into one place all relevant sites related to a discipline,
subject or idea. You have instant access to hundreds of sites by entering
just one gateway site.
Chris Sherman is associate editor of SearchEngine Watch.com and
co-author of The Invisible Web: Uncovering Information Sources Search
Engines Can't See.
Comments to online.feedback@xxxxxxxxxxxxxx
Guardian Unlimited © Guardian Newspapers Limited 2001
---
This email has been checked for viruses by AVG.
https://www.avg.com
** To leave the list, click on the immediately-following link:-
** [mailto:access-uk-request@xxxxxxxxxxxxx?subject=unsubscribe]
** If this link doesn't work then send a message to:
** access-uk-request@xxxxxxxxxxxxx
** and in the Subject line type
** unsubscribe
** For other list commands such as vacation mode, click on the
** immediately-following link:-
** [mailto:access-uk-request@xxxxxxxxxxxxx?subject=faq]
** or send a message, to
** access-uk-request@xxxxxxxxxxxxx with the Subject:- faq
