Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done somefurther analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.



![]() | This update applies to theAndroid Dev Phone for developers only. The Android Dev Phone with its unique color and pattern can be seen here. This update will not work on the T-Mobile G1 or any other Android powered phone. |
This page provides instructions on how you can flash your Android Dev Phone with a factory Android platform system image and associated radio image. From here, you can also download the actual image binaries, as well as relevant tools to help you flash the images to your device.
With these system image files, you can:
Note that you can flash a system image only to a supported device, as described in the next section.
In this document:
For discussions and support about updating your Android Dev Phone, please head over to the Android-DevPhone-Updating group.
The sections below describe two alternative methods that you use to flash your device with a factory Android system image:
This approach involves using the adb tool, available in the Android SDK, to copy updated radio and recovery images to the device's recovery partition on the SD card. It's a straightforward operation that you can run in a Windows, OS X, or Linux environment. If you are an Android application developer, this method of flashing your device is recommended. Note that to use this method, the device must have the original factory bootloader that came with the device (hboot 0.95.3000).
This approach is slightly more complicated and involves using the adb tool to update the radio image and the fastboot tool to flash a system image to the device. Flashing your device in this way is useful if you need to install the radio and system images independently or you are already familiar with fastboot and are using it successfully in your environment. To use this method, your device must have a fastboot-compatible bootloader. Having the original factory bootloader included on ADP devices is not required.
Before you get started, decide which of these two methods is appropriate for you, then follow the instructions below, in the order given. The flashing process for both methods is similar ??the main difference is that the recovery-image method does not require the fastboot tool. Rather than using fastboot to flash a system image to the device, you use adb to copy an updated recovery image to the device. All other steps are the same.
The system images provided on this page are designed only for installation on devices that meet the following requirements:
You cannot flash these system images to other hardware platforms or to a version of HTC Dream hardware that doesn't have an original factory bootloader, unless the device bootloader supports fastboot (fastboot is described in the next section). Additionally, you cannot flash these images to a retail devices such as the T-Mobile G1 because the images do not have the appropriate cryptographic signatures.
To flash a system image to your device, you need to have access to the proper tools.
Adb Tool
The adb tool is included in the Android SDK. If you have already installed the Android SDK on your local machine, you will find adb in the<sdk>/tools/ directory.
If you do not have the Android SDK, download the latest version from this location and install it:
Once you install the SDK, the adb tool will be available to you.
Fastboot Tool
If you are flashing your device using the fastboot method, you need the command-line tool calledfastboot.
The fastboot source code is available in the Android open source repository and is compiled whenever you build from that tree. If you've built from the Android open source tree, you will find the fastboot executable in this location:
<build-dir>/out/host/<platform>/bin/
For your convenience, you can also download the fastboot executable directly, using the links below.
|
On OS X and Linux, you can store the fastboot executable anywhere on your machine. On Windows, you must store the fastboot executable where it can find the AdbWinApi.dll file (included with the Android SDK), otherwise it will not work properly. The recommended place to store the fastboot executable on Windows is with the other tools in the Android SDK, in the<sdk>/tools directory. On all platforms, you may want to add the location to your PATH, for convenience.
USB Driver
If your development machine is running Windows, you should make sure to update your USB driver before continuing. For your convenience, the Android SDK (1.0 r2 or later) includes an updated USB driver that you can install. For more information, see Setting up a Device for Development on the Android Developers Site.
To flash a new system image to your device, you need to download two image packages to your local machine.
The table below provides links to the downloadable image packages. The sections provide more information about the contents of each package and how you should handle it after download.
|
Radio Image Download
For both methods of flashing your device, you need to download a radio image package from the list of links in the table above.
Each Android platform version is designed to run on top of a specifc version of the radio firmware. Typically, each successive release of the Android platform relies on a newer version of the radio firmware, so whenever you are planning to flash a new system image to your device, whether upgrade or downgrade, you should also update the radio firmware to the version expected by the platform. If you do not update your radio image, the system may encounter problems.
Use the links provided in the previous section to download the radio image package that corresponds to the system image you downloaded.
Once you've downloaded a radio image package, you do not need to extract it - you can reference the image package directly from adb commands, as described in the sections below. You can store the radio image package anywhere on your machine, but make sure to store to maintain its version-specific naming.
System Image Download
If you are flashing your device using the fastboot method, download the signed system image package(s) that you need from the list of links in the table above. Multiple versions of the Android platform are available, so make sure that you understand what version(s) you need and what version(s) you are downloading.
For development, you should select the lowest possible Android platform version that meets the needs of your applications. For example, if you are working in the Android 1.1 SDK and your application is using APIs introduced in Android 1.1, then you should download the Android 1.1 system image. If you are using the Android 1.1 SDK but your application does not use Android 1.1 APIs, then using Android 1.0 image is sufficient.
For testing, consider downloading all platform versions with which your application is compatible, then running your applications on those platform versions to ensure that they work as designed.
Each system image package is a .zip archive that includes several images that you will flash to your device:
Note that the images in each package are mutually dependent, so you must flash them all to your device together. For example, you should not flash a system image to the device without also flashing the boot and recovery images included in the same image package.
Once you've downloaded a system image package, you do not need to extract it ??you can reference the image package directly from fastboot commands, as described in the sections below. You can store the system image package anywhere on your machine, but make sure to store to maintain its version-specific naming.
Recovery Image Download
If you are flashing your device using the recovery-image method, download the recovery image package(s) that you need from the list of links in the table above. Currently, a recovery image packages is available only for the Android 1.1 platform version.
Each recovery image package is a .zip archive containing the full contents of a recovery partition, including:
Note that the images in each recovery package are mutually dependent, so you must copy them all to your device together. For example, you should not copy a system image to the device without also copying the boot and recovery images included in the same image package.
Once you've downloaded a recovery image package, you do not need to extract it - you can reference the image package directly from adb commands, as described in the sections below. You can store the recovery image package anywhere on your machine, but make sure to store to maintain its version-specific naming.
For both the recovery-image and fastboot methods, you need to update the radio firmware on the device before you go on to update the system. This step is necessary both to ensure that a compatible radio image exists on the device and to satisfy version prerequisites enforced by the bootloader when loading the updated recovery image.
After you've installed the tools and downloaded a system/recovery image package and corresponding radio image package, follow the steps below to update the radio firmware on the device.
$ adb devices
You should see your device in the console output generated by the command
$ adb push <radio-image-package>.zip /sdcard/update.zip
The command prints the number of bytes copied and the elapsed time, then completes. However, note that the files are still being copied in the background, so wait a minute or two before continuing. If you continue too soon and terminate the operation before it is complete, you will get a "bad update.zip" error later, when you try to install the update.
To ensure that the entire radio image package is copied before you continue, use this command:
$ adb shell sync
Whensync returns, all of the data has been copied to the device.
After the device reboots normally, you can update the system image or recovery image, as described in the next sections. To check the radio image version, you can start the device infastboot mode.
For the recovery-image method only, after you've successfully updated the radio firmware, the next (and final) step is using the adb tool to copy the recovery image package to the device SD card. (If you are using the fastboot method, you do not need to copy a recovery image to the device - seeFlash the System Image Package to the Device instead.)
To copy the recovery image to the device SD card, follow these steps:
$ adb devices
You should see your device in the console output generated by the command
$ adb push <recovery-image-package>.zip /sdcard/update.zip
The command prints the number of bytes copied and the elapsed time, then completes. However, note that the files are still being copied in the background, so wait a minute or two before continuing. If you continue too soon and terminate the operation before it is complete, you will get a "bad update.zip" error later, when you try to install the update.
To ensure that the entire recovery image package is copied before you continue, use this command:
$ adb shell sync
Whensync returns, all of the data has been copied to the device.
After the device reboots normally, the operation is complete.
For discussions and support about updating your Android Dev Phone, please head over to the Android-DevPhone-Updating group.
For the fastboot method only, after you've successfully updated the radio firmware, the next (and final) step is using the fastboot tool to flash the system image package to the device. Note that you should always update the radio image whenever you update the Android system image on your device. (If you are using the recovery-image method, you do not need to flash the system image package to the device - seeCopy the Recovery Image Package to the Device instead.)
Follow the steps below to flash a system image package to your device:
To enter fastboot mode, power up the device (or reboot it) while holding down the BACK key. Hold the BACK key down until the bootloader screen is visible and shows "FASTBOOT". The device is now in fastboot mode and is ready to receive fastboot commands. If you want to exit fastboot mode at this point, you can hold down the keys MENU+SEND+END (on the Android Dev Phone, SEND is the "Call" key and END is the "End call" key).
Note that the bootloader screen may vary across devices. For Android Dev Phone devices, the bootloader screen shows an image of skateboarding robots. Other devices may show a different image or color pattern. In all cases, the bootloader screen shows the text "FASTBOOT" when in fastboot mode. The bootloader also shows the radio version.
$ fastboot erase userdata
This will remove all of the installed applications and their data, but it is a useful step when debugging because it ensures a clean execution environment for the application. Erasing the user data may also be necessary when you are flashing a lower (downgrade) platform version to your device. However, it is not strictly necessary to erase the userdata partition when upgrading to a higher (upgrade) platform version, which is the typical use-case in consumer devices.
$ fastboot erase cache
Note that, rather than using separate fastboot commands to erase the userdata and cache partitions, as described in the previous steps, you can simply supply the-w option when flashing the system image, which has the same result.
$ fastboot update <imagepackage>.zip [-w]
With this command, fastboot copies all of the images in<system-image-package>.zip to the appropriate partitions on the device. During the process, it prints status messages to the command shell. Depending on your bootloader version, fastboot may also print status messages to the bootloader screen.
When all image are copied to the device successfully, fastboot displays a success message.
For information about other fastboot commands, use
$ fastboot
If necessary, you can reboot the device manually using these fastboot commands:
$ fastboot reboot
or
$ fastboot reboot-bootloader
You can also reboot from the device by holding down the key combination MENU+SEND+END.
After rebooting, the device should start normally. If the device does not start normally, you can try flashing it again. If you did not erase the userdata partition and the device hangs during boot, try erasing the userdata partition and then reboot.
For discussions and support about updating your Android Dev Phone, please head over to the Android-DevPhone-Updating group.
fastboot.exe (on Windows) doesn't detect the ADP device properly
In some cases, Windows detects the ADP1 bootloader as a USB mass storage device, preventing fastboot.exe from accessing it. To resolve the issue, you can dissociate the USB Mass Storage driver from your ADP1 on your Windows machine and install the proper driver instead, as described in the steps below:
If you are unable to use fastboot successfully, note that you can use the recovery-image method to flash an updated system image to the device, provided the device has the original factory bootloader. See theOverview for more information, then follow the instructions starting from the top of this document.
When installing updated radio firmware, the recovery console fails to open the update.zip file
As described inUpdate the Device Radio Firmware, you need to copy the contents of the radio image package to the update.zip archive on the device's SD card, then install the update using the recovery console. In some cases, when you try to install the update, the operation fails because of a problem with the update.zip file. This type of failure usually occurs when you start the installation before all of the radio files are copied to the update.zip archive. To resolve the issue, copy the files again to update.zip, this time waiting a minute or two before starting the installation, to ensure that all of the files are copied successfully.
Cannot downgrade to Android 1.1 with recovery package 142608
If you are using the recovery-image method and had previously downloaded and the Android 1.1 recovery packagesigned-dream_devphone-ota-142608.zip, you should make sure to download the replacement recovery packagesigned-dream_devphone-ota-142608-b.zip (see the download link at the top of this page).
This is necessary because the recovery packagesigned-dream_devphone-ota-142608.zip prevents you from downgradingto it from ahigher version of the recovery image. Therefore, if you upgrade to the Android 1.5 recovery image, you won't be able to downgrade to Android 1.1 unless you use the replacement packagesigned-dream_devphone-ota-142608-b.zip.
