Web archiving is the process of collecting, preserving, and providing access to material from theWorld Wide Web. The aim is to ensure that information ispreserved in anarchival format for research and the public.[1] The process of platformizing archives, digitizing historical records via interfaces patterned on social media platforms, can reshape collective memory by privileging content that aligns with social-media logic such as popularity, connectivity, and programmability.[2]
Web archivists typically employ automatedweb crawlers to capturing the massive amount of information on the Web. A widely known web archive service is theWayback Machine, run by theInternet Archive.
The growing portion of human culture created and recorded on the web makes it inevitable that more and more libraries and archives will have to face the challenges of web archiving.[3]National libraries,national archives, and various consortia of organizations are also involved in archiving Web content to prevent its loss.
Commercial web archiving software and services are also available to organizations that need to archive their own web content for corporate heritage, regulatory, or legal purposes.
While curation and organization of the web has been prevalent since the mid- to late-1990s, one of the first large-scale web archiving projects was theInternet Archive, a non-profit organization created byBrewster Kahle in 1996.[4] The Internet Archive released its own search engine for viewing archived web content, theWayback Machine, in 2001.[4] As of 2018, the Internet Archive was home to 40 petabytes of data.[5] The Internet Archive also developed many of its own tools for collecting and storing its data, includingPetaBox for storing large amounts of data efficiently and safely, andHeritrix, a web crawler developed in conjunction with the Nordic national libraries.[4] Other projects launched around the same time included a web archiving project by theNational Library of Canada, Australia'sPandora, Tasmanian web archives and Sweden's Kulturarw3.[6][7]
From 2001to 2010,[failed verification] the International Web Archiving Workshop (IWAW) provided a platform to share experiences and exchange ideas.[8][9] TheInternational Internet Preservation Consortium (IIPC), established in 2003, has facilitated international collaboration in developing standards and open source tools for the creation of web archives.[10]
The now-defunctInternet Memory Foundation was founded in 2004 and founded by theEuropean Commission in order to archive the web in Europe.[4] This project developed and released many open source tools, such as "rich media capturing, temporal coherence analysis, spam assessment, and terminology evolution detection."[4] The data from the foundation is now housed by the Internet Archive, but not currently publicly accessible.[11]
Despite the fact that there is no centralized responsibility for its preservation, web content is rapidly becoming the official record. For example, in 2017, theUnited States Department of Justice affirmed that the government treats the President'stweets as official statements.[12]
Web archivists generally archive various types of web content includingHTML web pages,style sheets,JavaScript,images, andvideo. They also archivemetadata about the collected resources such as access time,MIME type, and content length. This metadata is useful in establishingauthenticity andprovenance of the archived collection.
Transactional archiving is an event-driven approach, which collects the actual transactions which take place between aweb server and aweb browser. It is primarily used as a means of preserving evidence of the content which was actually viewed on a particularwebsite, on a given date. This may be particularly important for organizations which need to comply with legal or regulatory requirements for disclosing and retaining information.[13]
A transactional archiving system typically operates by intercepting everyHTTP request to, and response from, the web server, filtering each response to eliminate duplicate content, and permanently storing the responses as bitstreams.
Web archives which rely on web crawling as their primary means of collecting the Web are influenced by the difficulties of web crawling:
However, it is important to note that a native format web archive, i.e., a fully browsable web archive, with working links, media, etc., is only really possible using crawler technology.
The Web is so large that crawling a significant portion of it takes a large number of technical resources. Also, the Web is changing so fast that portions of a website may suffer modifications before a crawler has even finished crawling it.
Some web servers are configured to return different pages to web archiver requests than they would in response to regular browser requests. This is typically done to fool search engines into directing more user traffic to a website and is often done to avoid accountability or to provide enhanced content only to those browsers that can display it.
Not only must web archivists deal with the technical challenges of web archiving, they must also contend with intellectual property laws. Peter Lyman[14] states that "although the Web is popularly regarded as apublic domain resource, it iscopyrighted; thus, archivists have no legal right to copy the Web". Howevernational libraries in some countries[15] have a legal right to copy portions of the web under an extension of alegal deposit.
Some private non-profit web archives that are made publicly accessible likeWebCite, theInternet Archive or theInternet Memory Foundation allow content owners to hide or remove archived content that they do not want the public to have access to. Other web archives are only accessible from certain locations or have regulated usage. WebCite cites a recent lawsuit against Google's caching, whichGoogle won.[16]
In 2017 theFinancial Industry Regulatory Authority, Inc. (FINRA), a United States financial regulatory organization, released a notice stating all the businesses doing digital communications are required to keep a record. This includes website data, social media posts, and messages.[17] Somecopyright laws may inhibit Web archiving. For instance, academic archiving bySci-Hub falls outside the bounds of contemporary copyright law. The site provides enduring access to academic works including those that do not have anopen access license and thereby contributes to the archival of scientific research which may otherwise be lost.[18][19]
| Concepts | |
|---|---|
| Techniques | |
| By type | |
| Organizations | |
| Lists | |
| International | |
|---|---|
| National | |
