Repository files navigation

This public repository archives data over time from various public sources on the web.

Data is presented as timestamped CSV files, for maximum compatibility.

It is hoped that this data will be useful to those producing visualizations or analyses.

Code is included.

Please note this data is no longer updated. It substantially covers the significant period of growth of the virus in China and should be useful for historical analysis.

Shown here in GIF format. There is a better (smaller/higher resolution) webm format also generated.

Generates static SVGs.

Source images wereChina_blank_province_map.svg(link) andBlankMap-World.svg(link).

Unix-like OS with the dependencies installed (seeSoftware Dependencies). In practice that means macOS with brew, Linux or a BSD. Windows is unsupported.

For a China map, the following command sequence will grab data from DXY and render it.

./build china

You now have timestamped JSON, CSV and SVG files in thedata-sources/dxy/data/ subdirectory.

For a world map, the process is similar.Note that the BNO world data parser is currently broken and we have no plan to fix it.

./build world

You now have timestamped CSV and SVG files indata-sources/bno/data.

Probably an incomplete list:

• bash
• perl
• php
• imagemagick
• gifsicle
• ffmpeg
• wget

• Add key to global view
• Convert disparate data sources in to singular SQLite database
• Add cities (needs an improved visualization system)
• More visualization options:
  • amcharts (icky licensing but looks cool)
  • d3.js andd3-china-map (good option but possibly overkill)
  • jvectormap andjvectormap china (good option but no cities)
  • kartograph (good option)
• Add other sources, eg.
  • WHO situation reports
  • National Health Commission daily reports
  • Tencent live monitoring page
  • English Wikipedia (although it generally lags DXY on updates, the summary table is good for international data citations)
  • Fuuuuuuu's map which seems to have many international cases missed by BNO
• Potential for different source SVGs, eg.naturalearthdata

• Coronavirus tally in epicentre Wuhan may be ‘just the tip of the iceberg’ (2020-02-03)
  • A doctor at the Union Hospital in Wuhan, who declined to be identified, said staff could only test about 100 patients a day, and they had to wait 48 hours for the results. “When the National Health Commission announces the numbers, they’re already two days old,” the doctor said. “We also have to turn away patients with mild symptoms, knowing that many of them will return later [when their condition worsens]. But we don’t have the space in the testing centre, or the hospital beds.”
  • “There have also been many patients who died of undifferentiated respiratory and undiagnosed pneumonia symptoms in Wuhan since December – before the virus testing kits were made available,” Tsang said. “These cases should have been investigated and counted [in the tally] if confirmed. These are factors pointing to inaccurate reporting of the official figures,” he said.
  • A doctor at the Tongji Hospital in Wuhan, speaking on condition of anonymity, said the kits were still in short supply. “I don’t know what’s gone wrong – we only have a very limited number of testing kits every day, there’s been no increase yet,” the doctor said.
• Potential for global spread of a novel coronavirus from China (2020-01-27)
  • Ranks Burma, Cambodia, India, Indonesia, Philippines as relatively vulnerable.
• Real-time nowcast and forecast on the extent of the Wuhan CoV outbreak, domestic and international spread (2020-01-27)
  • Hong Kong University professors estimate 43,590 infections as of 2020-01-25. (ie. ~20x 'confirmed cases')
  • 2019-nCoV may be about to become a global epidemic
  • Self-sustaining human-to-human spread is already present in all major Chinese cities
• Pattern of early human-to-human transmission of Wuhan 2019-nCoV (2020-01-24)
  • We found the basic reproduction number, R0, to be around 2.2 (90% high density interval 1.4–3.8), indicating the potential for sustained human-to-human transmission. Transmission characteristics appear to be of similar magnitude tosevere acute respiratory syndrome-related coronavirus (SARS-CoV) and the1918 pandemic influenza.
• Novel coronavirus 2019-nCoV: early estimation of epidemiological parameters and epidemic predictions (2020-01-23)
  • Only 5% of cases are likely reported in official figures of confirmed cases.

• Open source Wuhan (list of projects related to the virus in Chinese)
• BlankerL's DXY-2019-nCoV-Crawler andAPI
• yitao94's 2019-nCoV python-based DXY crawler

This section is written for the curious / non-technical user.

The general approach to problems such as these is as follows:

1. Gather the data
2. Modify and store it
3. Do something with it.

The area of programming surrounding gathering data from websites that were not explicitly designed for it is calledweb scraping.

In general, web scraping consists of making anHTTP (web) request to the website in question,parsing (or interpreting) the response, andextracting the data of interest. Thereafter some modification may be required.

We translate some Chinese and English information (toponyms or geographic region names) in to a known format by matching against astatic database file for countries and asimilar file for regions in or near China.

We then store the data in various formats, mostlyCSV andJSON, which aretimestamped in amost to least significance format, inspired by theISO 8601 standard, to aid insorting.

Finally, we further interpret and process the data in two stages.

First, we transform some referenceSVG maps gathered fromWikimedia Commons by applying the data we have captured.

Finally we animate multiple such resulting images in to two formats,animated GIF and the greatly superior and far more modernwebm container format withVP9 encoding. This is done using the open source toolsimagemagick andffmpeg.