These crawls are part of an effort to archive pages as
they are created
and archive the pages that they refer to. That way, as
the pages that
are referenced are changed or taken from the web, a
link to the version
that was live when the page was written will be
preserved.
Then the Internet Archive hopes that references to
these archived pages
will be put in place of a link that would be otherwise
be broken, or a
companion link to allow people to see what was
originally intended by a
page's authors.
The goal is to
fix
all
broken links on the web. Crawls of supported "No
More 404"
sites.
