 | |
| Type of business | 501(c)(3) non-profit |
|---|---|
| Founded | 2007 |
| Headquarters | San Francisco, California;Los Angeles, California, United States |
| Founder | Gil Elbaz |
| Key people | Peter Norvig,Rich Skrenta,Eva Ho |
| URL | commoncrawl |
Content license | Apache 2.0 (software)[clarification needed] |
Common Crawl is anonprofit501(c)(3) organization thatcrawls the web and freely provides its archives and datasets to the public.[1][2] Common Crawl'sweb archive consists ofpetabytes of data collected since 2008.[3] It completes crawls approximately once a month.[4]
Common Crawl was founded byGil Elbaz.[5] Advisors to the non-profit includePeter Norvig andJoi Ito.[6] It is funded by the Elbaz Family Foundation Trust and significant donations from the AI industry.[7] The organization's crawlers respectnofollow androbots.txt policies.[citation needed] Open source code for processing Common Crawl's data set is publicly available.
The Common Crawl dataset includes copyrighted work and is distributed from the US underfair use claims. Researchers in other countries have made use of techniques such as shuffling sentences or referencing the Common Crawl dataset to work around copyright law in otherlegal jurisdictions.[8]
Contents archived by Common Crawl are mirrored[9] and made available online[10] in theWayback Machine. They are used by researchers, as well as AI companies to trainlarge language models.[7]
English is the primary language for 46% of documents in the March 2023 version of the Common Crawl dataset. The next most common primary languages are German, Russian, Japanese, French, Spanish and Chinese, each with less than 6% of documents.[11]
In November 2025, an investigation byThe Atlantic revealed that Common Crawl lied when it claimed it respectedpaywalls in its scraping and requests from publishers to have their content removed from its databases.[7]
Amazon Web Services began hosting Common Crawl's archive through its Public Data Sets program in 2012.[12]
The organization began releasingmetadata files and the text output of the crawlers alongside.arc files in July 2012.[13] Common Crawl's archives had only included .arc files previously.[13]
In December 2012,blekko donated to Common Crawl search enginemetadata blekko had gathered from crawls it conducted from February to October 2012.[14] The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessiveSEO."[14]
In 2013, Common Crawl began using theApache Software Foundation'sNutch webcrawler instead of a custom crawler.[15] Common Crawl switched from using .arc files to.warc files with its November 2013 crawl.[16]
A filtered version of Common Crawl was used to train OpenAI'sGPT-3 language model, announced in 2020.[17] In 2023, it began receiving significant financial support from AI companies, includingAnthropic andOpenAI, each of which donated $250,000.[7]
In November 2025, an investigation by technology journalist Alex Reisner forThe Atlantic revealed that Common Crawl lied when it claimed it respected paywalls in its scraping and requests from publishers to have their content removed from its databases.[7] It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies.[7]
 | This sectionis in a table format thatmay be better if supplemented withprose. You can help by adding a prose summary of the table(s).Editing help is available.(November 2025) |
The following data have been collected from the official Common Crawl Blog[18]and Common Crawl's API.[19]
| Crawl date | Size inTiB | Billions of pages | Comments |
|---|---|---|---|
| 2008-2009 | ? | ? | Crawl conducted from May 2008 through January 2009 |
| 2009-2010 | ? | ? | Crawl conducted from July 2009 through September 2010 |
| 2012 | ? | ? | Crawl conducted from January 2012 through June 2012. Final ARC crawl |
| Summer 2013 | ? | ? | Crawl conducted from May 2013 through June 2013. First WARC crawl |
| Winter 2013 | 148 | 2.3 | Crawl conducted from December 4 through December 22, 2013 |
| March 2014 | 223 | 2.8 | First Nutch crawl |
| April 2014 | 183 | 2.6 | |
| July 2014 | 266 | 3.6 | |
| August 2014 | 200 | 2.8 | |
| September 2014 | 220 | 2.8 | |
| October 2014 | 254 | 3.7 | |
| November 2014 | 135 | 1.95 | |
| December 2014 | 160 | 2.08 | |
| January 2015 | 139 | 1.82 | |
| February 2015 | 145 | 1.9 | |
| March 2015 | 124 | 1.64 | |
| April 2015 | 168 | 2.11 | |
| May 2015 | 159 | 2.05 | |
| June 2015 | 131 | 1.67 | |
| July 2015 | 145 | 1.81 | |
| August 2015 | 149 | 1.84 | |
| September 2015 | 106 | 1.32 | |
| November 2015 | 151 | 1.82 | |
| February 2016 | — | 1.73 | |
| April 2016 | — | 1.33 | |
| May 2016 | — | 1.46 | |
| June 2016 | — | 1.23 | |
| July 2016 | — | 1.73 | |
| August 2016 | — | 1.61 | |
| September 2016 | — | 1.72 | |
| October 2016 | — | 3.25 | |
| December 2016 | — | 2.85 | |
| January 2017 | 250 | 3.14 | |
| February 2017 | 250 | 3.08 | |
| March 2017 | 250 | 3.07 | |
| April 2017 | 250 | 2.94 | |
| May 2017 | 250 | 2.96 | |
| June 2017 | 260 | 3.16 | |
| July 2017 | 240 | 2.89 | |
| August 2017 | 280 | 3.28 | |
| September 2017 | 250 | 3.01 | |
| October 2017 | 300 | 3.65 | |
| November 2017 | 260 | 3.2 | |
| December 2017 | 240 | 2.9 | |
| January 2018 | 270 | 3.4 | |
| February 2018 | 270 | 3.4 | |
| March 2018 | 250 | 3.2 | |
| April 2018 | 230 | 3.1 | |
| May 2018 | 215 | 2.75 | |
| June 2018 | 235 | 3.05 | |
| July 2018 | 255 | 3.25 | |
| August 2018 | 220 | 2.65 | |
| September 2018 | 220 | 2.8 | |
| October 2018 | 240 | 3.0 | |
| November 2018 | 220 | 2.6 | |
| April 2021 | 320 | 3.1 | |
| October 2022 | 380 | 3.15 | Crawl conducted in September and October 2022 |
| December 2022 | 420 | 3.35 | Crawl conducted from November 26 to December 10, 2022 |
| February 2023 | 400 | 3.15 | Crawl conducted from January 26 to February 9, 2023 |
| April 2023 | 400 | 3.1 | Crawl conducted from March 20 to April 2, 2023 |
| June 2023 | 390 | 3.1 | Crawl conducted from May 27 to June 11, 2023 |
| December 2023 | 454 | 3.35 | Crawl conducted from November 28 to December 12, 2023 |
| February/March 2024 | 425 | 3.16 | Crawl conducted from February 20 to March 5, 2024 |
| April 2024 | 386 | 2.7 | Crawl conducted from April 12 to April 24, 2024 |
In corroboration withSURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers inBenelux.[20][21] The award is named forPeter Norvig who also chairs the judging committee for the award.[20]
Google's version of the Common Crawl is called theColossal Clean Crawled Corpus, orC4 for short. It was constructed for the training of theT5 language model series in 2019.[22] There are some concerns over copyrighted content in the C4.[23]
the majority of our data is derived from raw Common Crawl with only quality-based filtering.