Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Data Analyze and Visualization of Coronavirus Data of Korea

NotificationsYou must be signed in to change notification settings

tao-sun2/Data_Analyze_and_Visualization

Repository files navigation

1. Background introduction:

2019 Novel Coronavirus (COVID-19) has infected more than 1 million people in theworld. We want to do something against this pestilence by using what we have learnt.This datasets consists of 11 CSV files from the KCDC (Korea Centers for Disease Control & Prevention), a structureddataset based on the report materials of KCDC and local governments was provided by Korean researchers.

2. Data Analysis

2.1 Data Statistics and Visualization

2.1.1 Data Preprocessing

# needed importsimportpandasaspdimportnumpyasnpimportseabornassnsimportpandasaspdfromsklearnimportmanifoldimportnumpyasnpfromsklearn.linear_modelimportLassoLarsCVfromsklearn.ensembleimportGradientBoostingRegressorfromsklearn.model_selectionimporttrain_test_splitfromsklearn.linear_modelimportLinearRegressionfromsklearn.metricsimportr2_scorefromsklearn.linear_modelimportRidge,RidgeCVfromsklearn.metricsimportmean_squared_errorfrommatplotlibimportpyplotaspltfromscipy.cluster.hierarchyimportfclusterfromscipy.cluster.hierarchyimportdendrogram,linkagefromscipy.spatialimportdistancefromsklearn.clusterimportKMeansfromsklearnimportmanifoldfromsklearn.decompositionimportPCA
PatientInfo=pd.read_csv('PatientInfo.csv')PatientInfo.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
patient_idglobal_numsexbirth_yearagecountryprovincecitydiseaseinfection_caseinfection_orderinfected_bycontact_numbersymptom_onset_dateconfirmed_datereleased_datedeceased_datestate
010000000012.0male1964.050sKoreaSeoulGangseo-guNaNoverseas inflow1.0NaN75.02020-01-222020-01-232020-02-05NaNreleased
110000000025.0male1987.030sKoreaSeoulJungnang-guNaNoverseas inflow1.0NaN31.0NaN2020-01-302020-03-02NaNreleased
210000000036.0male1964.050sKoreaSeoulJongno-guNaNcontact with patient2.02.002000e+0917.0NaN2020-01-302020-02-19NaNreleased
310000000047.0male1991.020sKoreaSeoulMapo-guNaNoverseas inflow1.0NaN9.02020-01-262020-01-302020-02-15NaNreleased
410000000059.0female1992.020sKoreaSeoulSeongbuk-guNaNcontact with patient2.01.000000e+092.0NaN2020-01-312020-02-24NaNreleased
  1. Using the method duplicated in Pandas to detect duplicate values in the dataset
PatientInfo[PatientInfo.duplicated()]
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
patient_idglobal_numsexbirth_yearagecountryprovincecitydiseaseinfection_caseinfection_orderinfected_bycontact_numbersymptom_onset_dateconfirmed_datereleased_datedeceased_datestate

The result shows there is no duplicate values.2. See if there are missing values for each attribute of the data set

index_array=np.sum(PatientInfo.isnull()==True,axis=0)print(index_array)print(PatientInfo.shape[0])
patient_id               0global_num            1160sex                     94birth_year             464age                    105country                 90province                 0city                    76disease               3110infection_case         819infection_order       3097infected_by           2393contact_number        2539symptom_onset_date    2682confirmed_date           0released_date         2147deceased_date         3072state                    0dtype: int643128

There are 3128 rows in the dataset PatientInfo.csvI notice that there are many miss values in many colums, and I decide first observe characteristics of the data by visualization and then deal with the miss value.

  1. Outlier detecting
PatientInfo.describe().astype(np.int64).T
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
countmeanstdmin25%50%75%max
patient_id31283562811488221016822710000000011300000026200000057660010005197000000009
global_num196862343297136247437908510329
birth_year266419742019161960197419932020
infection_order312111236
infected_by7352557285853164593084810000000021200000031200000020541000000066100000384
contact_number5891876024141160

The results show that there is no outliers

2.1.2 Visualization of PatientInfo

  1. Gender distribution
PatientInfo['sex'].value_counts().plot(kind='bar',title='Gender distribution',fontsize=15,rot=360)
<matplotlib.axes._subplots.AxesSubplot at 0x2296931a548>

png

  1. Age distribution
PatientInfo['age'].value_counts().plot(kind='bar',title='Age distribution',fontsize=15,rot=360)
<matplotlib.axes._subplots.AxesSubplot at 0x229693ed588>

png

  1. Top ten cities with the most patients
city=PatientInfo['city'].value_counts()city=city.sort_values(ascending=False)[:10]city.sort_values(ascending=True).plot.barh(fontsize=15)plt.title('Top ten cities with the most patients',size=15)plt.show()

png

  1. Top ten province with the most patients
province=PatientInfo['province'].value_counts()province=province.sort_values(ascending=False)[:10]province.sort_values(ascending=True).plot.barh(fontsize=15)plt.title('Top ten provinces with the most patients',size=15)plt.show()

png

  1. Top ten infection_case of patients
infection_case=PatientInfo['infection_case'].value_counts()infection_case=infection_case.sort_values(ascending=False)[:10]infection_case.sort_values(ascending=True).plot.barh(fontsize=15)plt.title('Top ten infection_case of patients',size=15)plt.show()

png

  1. The relation between sex and state among patients.
sns.countplot(x="state",hue="sex",data=PatientInfo)
<matplotlib.axes._subplots.AxesSubplot at 0x22968175c48>

png

  1. The relation between age and state among patients.
sns.countplot(x="state",hue="age",data=PatientInfo)
<matplotlib.axes._subplots.AxesSubplot at 0x229694a0048>

png

  1. The relation between the top five cities with most patients and state among patients.
top_five_cities=PatientInfo[PatientInfo['city'].isin(['Gyeongsan-si','Seongnam-si','Bonghwa-gun','Bucheon-si','Gumi-si'])]sns.countplot(x="state",hue="city",data=top_five_cities)
<matplotlib.axes._subplots.AxesSubplot at 0x2296a829608>

png

  1. The relation between the top five provinces with most patients and state among patients.
top_five_provinces=PatientInfo[PatientInfo['province'].isin(['Gyeongsangbuk-do','Gyeonggi-do','Seoul','Chungcheongnam-do','Busan'])]sns.countplot(x="state",hue="province",data=top_five_provinces)
<matplotlib.axes._subplots.AxesSubplot at 0x2296a8ba348>

png

  1. The relation between the top five infection_case with most patients and state among patients.
top_five_infection_case=PatientInfo[PatientInfo['infection_case'].isin(['contact with patient','etc','overseas inflow','Guro-gu Call Center','Shincheonji Church'])]sns.countplot(x="state",hue="infection_case",data=top_five_infection_case)
<matplotlib.axes._subplots.AxesSubplot at 0x2296b922708>

png

Conclusion:

After preliminary data visualization analysis, we can find that the number of female patients is about 300 larger than the number of male patients, most patients are aged from 20 to 60, most patients are form the city Gyeongsan-si, most patients are form the provinces Gyeongsangbuk-do, Gyeonggi-do and Seoul and most infection cases are contact with patient, etc and overseas inflow.

By analysing the states of patients(isolated / released / deceased) and other attributes, we can find that male patients may be a little easier to be deceased than female patients because the number of female patients is larger than the number of male patients but the number of deceased female patients is less than the number of deceased male patients.

We can also find that younger patients may be easier to be released than older patients and older patients may be easier to be deceased than older patients.

We can also find that contacting with patients around you in your city and province is the main reason people get infected.

2.1.3 Visualize the Time Series Data

  1. Visualize the File Time
Time=pd.read_csv('Time.csv')Time.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
datetimetestnegativeconfirmedreleaseddeceased
02020-01-201610100
12020-01-211610100
22020-01-221643100
32020-01-23162221100
42020-01-24162725200
#We do not care the update time so we drop it.Time.drop(['time'],axis=1,inplace=True)Time.set_index('date',inplace=True)Time.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
testnegativeconfirmedreleaseddeceased
date
2020-01-2010100
2020-01-2110100
2020-01-2243100
2020-01-232221100
2020-01-242725200
fig,ax=plt.subplots(figsize=(12,6))Time.plot(marker='o',ms=2,lw=1,ax=ax)fig.autofmt_xdate()plt.legend(bbox_to_anchor= [1,1])plt.title('COVID-19 Data of South Korea',size=15)plt.ylabel('number')plt.grid(axis='y')plt.show()

png

The figure is not very clear because the accumulated test and negative number is much larger that the rest numbers.Hence we will just focus on the number of confirmed, released and deceased patients.

Time.drop(['test','negative'],axis=1,inplace=True)fig,ax=plt.subplots(figsize=(12,6))Time.plot(marker='o',ms=2,lw=1,ax=ax)fig.autofmt_xdate()plt.legend(bbox_to_anchor= [1,1])plt.title('COVID-19 Data of South Korea',size=15)plt.ylabel('number')plt.grid(axis='y')plt.show()

png

  1. Visualize the File TimeAge
TimeAge=pd.read_csv('TimeAge.csv')TimeAge.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
datetimeageconfirmeddeceased
02020-03-0200s320
12020-03-02010s1690
22020-03-02020s12350
32020-03-02030s5061
42020-03-02040s6331
#We do not care the update time so we drop it.TimeAge.drop(['time'],axis=1,inplace=True)TimeAge.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
dateageconfirmeddeceased
02020-03-020s320
12020-03-0210s1690
22020-03-0220s12350
32020-03-0230s5061
42020-03-0240s6331
#processing of Dataframe TimeAge to make it easy to draw clear figures.Age0=TimeAge[TimeAge['age']=='0s']Age0=Age0.rename(columns={'confirmed':'confirmed_0s'})Age0=Age0.rename(columns={'deceased':'deceased_0s'})Age1=TimeAge[TimeAge['age']=='10s']Age1=Age1.rename(columns={'confirmed':'confirmed_10s'})Age1=Age1.rename(columns={'deceased':'deceased_10s'})Age2=TimeAge[TimeAge['age']=='20s']Age2=Age2.rename(columns={'confirmed':'confirmed_20s'})Age2=Age2.rename(columns={'deceased':'deceased_20s'})Age3=TimeAge[TimeAge['age']=='30s']Age3=Age3.rename(columns={'confirmed':'confirmed_30s'})Age3=Age3.rename(columns={'deceased':'deceased_30s'})Age4=TimeAge[TimeAge['age']=='40s']Age4=Age4.rename(columns={'confirmed':'confirmed_40s'})Age4=Age4.rename(columns={'deceased':'deceased_40s'})Age5=TimeAge[TimeAge['age']=='50s']Age5=Age5.rename(columns={'confirmed':'confirmed_50s'})Age5=Age5.rename(columns={'deceased':'deceased_50s'})Age6=TimeAge[TimeAge['age']=='60s']Age6=Age6.rename(columns={'confirmed':'confirmed_60s'})Age6=Age6.rename(columns={'deceased':'deceased_60s'})Age7=TimeAge[TimeAge['age']=='70s']Age7=Age7.rename(columns={'confirmed':'confirmed_70s'})Age7=Age7.rename(columns={'deceased':'deceased_70s'})Age8=TimeAge[TimeAge['age']=='80s']Age8=Age8.rename(columns={'confirmed':'confirmed_80s'})Age8=Age8.rename(columns={'deceased':'deceased_80s'})result=pd.merge(Age0,Age1,on='date')result=pd.merge(result,Age2,on='date')result=pd.merge(result,Age3,on='date')result=pd.merge(result,Age4,on='date')result=pd.merge(result,Age5,on='date')result=pd.merge(result,Age6,on='date')result=pd.merge(result,Age7,on='date')result=pd.merge(result,Age8,on='date')result.set_index(['date'],inplace=True)result.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
age_xconfirmed_0sdeceased_0sage_yconfirmed_10sdeceased_10sage_xconfirmed_20sdeceased_20sage_y...deceased_50sage_xconfirmed_60sdeceased_60sage_yconfirmed_70sdeceased_70sageconfirmed_80sdeceased_80s
date
2020-03-020s32010s169020s1235030s...560s530670s192680s813
2020-03-030s34010s204020s1417030s...560s597770s224980s935
2020-03-040s34010s233020s1575030s...560s646770s2601280s1086
2020-03-050s38010s257020s1727030s...560s699870s2881380s1247
2020-03-060s45010s292020s1877030s...560s7631170s3401480s16810

5 rows × 27 columns

fig,ax=plt.subplots(figsize=(16,10))result.plot(marker='o',ms=2,lw=1,ax=ax)fig.autofmt_xdate()plt.legend(bbox_to_anchor= [1,1])plt.title('COVID-19 Data of Different Ages',size=15)plt.ylabel('number')plt.grid(axis='y')plt.show()

png

  1. Visualize the File TimeGender
TimeGender=pd.read_csv('TimeGender.csv')TimeGender.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
datetimesexconfirmeddeceased
02020-03-020male159113
12020-03-020female26219
22020-03-030male181016
32020-03-030female300212
42020-03-040male199620
#The processing is the same as the one above.TimeGender.drop(['time'],axis=1,inplace=True)male=TimeGender[TimeGender['sex']=='male']male=male.rename(columns={'confirmed':'confirmed_male'})male=male.rename(columns={'deceased':'deceased_male'})female=TimeGender[TimeGender['sex']=='female']female=female.rename(columns={'confirmed':'confirmed_female'})female=female.rename(columns={'deceased':'deceased_female'})result=pd.merge(female ,male,on='date')result.set_index(['date'],inplace=True)result.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
sex_xconfirmed_femaledeceased_femalesex_yconfirmed_maledeceased_male
date
2020-03-02female26219male159113
2020-03-03female300212male181016
2020-03-04female333212male199620
2020-03-05female361714male214921
2020-03-06female393917male234525
fig,ax=plt.subplots(figsize=(16,8))result.plot(marker='o',ms=2,lw=1,ax=ax)fig.autofmt_xdate()plt.legend(bbox_to_anchor= [1,1])plt.title('COVID-19 Data of Different Genders',size=15)plt.ylabel('number')plt.grid(axis='y')plt.show()

png

  1. Visualize the File TimeProvince
TimeProvince=pd.read_csv('TimeProvince.csv')TimeProvince.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
datetimeprovinceconfirmedreleaseddeceased
02020-01-2016Seoul000
12020-01-2016Busan000
22020-01-2016Daegu000
32020-01-2016Incheon100
42020-01-2016Gwangju000
#It will be a mass if we draw the imformatin of all provinces, so we choose to draw the figure of the top 5 provinces with most patients.TimeProvince.drop(['time'],axis=1,inplace=True)p1=TimeProvince[TimeProvince['province']=='Gyeongsangbuk-do']p1=p1.rename(columns={'confirmed':'confirmed_Gyeongsangbuk-do'})p1=p1.rename(columns={'released':'released_Gyeongsangbuk-do'})p1=p1.rename(columns={'deceased':'deceased_Gyeongsangbuk-do'})p2=TimeProvince[TimeProvince['province']=='Gyeonggi-do']p2=p2.rename(columns={'confirmed':'confirmed_Gyeonggi-do'})p2=p2.rename(columns={'released':'released_Gyeonggi-do'})p2=p2.rename(columns={'deceased':'deceased_Gyeonggi-do'})p3=TimeProvince[TimeProvince['province']=='Seoul']p3=p3.rename(columns={'confirmed':'confirmed_Seoul'})p3=p3.rename(columns={'released':'released_Seoul'})p3=p3.rename(columns={'deceased':'deceased_Seoul'})p4=TimeProvince[TimeProvince['province']=='Chungcheongnam-do']p4=p4.rename(columns={'confirmed':'confirmed_Chungcheongnam-do'})p4=p4.rename(columns={'released':'released_Chungcheongnam-do'})p4=p4.rename(columns={'deceased':'deceased_Chungcheongnam-do'})p5=TimeProvince[TimeProvince['province']=='Busan']p5=p5.rename(columns={'confirmed':'confirmed_Busan'})p5=p5.rename(columns={'released':'released_Busan'})p5=p5.rename(columns={'deceased':'deceased_Busan'})result=pd.merge(p1,p2,on='date')result=pd.merge(result,p3,on='date')result=pd.merge(result,p4,on='date')result=pd.merge(result,p5,on='date')result.set_index(['date'],inplace=True)result.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
province_xconfirmed_Gyeongsangbuk-doreleased_Gyeongsangbuk-dodeceased_Gyeongsangbuk-doprovince_yconfirmed_Gyeonggi-doreleased_Gyeonggi-dodeceased_Gyeonggi-doprovince_xconfirmed_Seoulreleased_Seouldeceased_Seoulprovince_yconfirmed_Chungcheongnam-doreleased_Chungcheongnam-dodeceased_Chungcheongnam-doprovinceconfirmed_Busanreleased_Busandeceased_Busan
date
2020-01-20Gyeongsangbuk-do000Gyeonggi-do000Seoul000Chungcheongnam-do000Busan000
2020-01-21Gyeongsangbuk-do000Gyeonggi-do000Seoul000Chungcheongnam-do000Busan000
2020-01-22Gyeongsangbuk-do000Gyeonggi-do000Seoul000Chungcheongnam-do000Busan000
2020-01-23Gyeongsangbuk-do000Gyeonggi-do000Seoul000Chungcheongnam-do000Busan000
2020-01-24Gyeongsangbuk-do000Gyeonggi-do100Seoul000Chungcheongnam-do000Busan000
fig,ax=plt.subplots(figsize=(16,10))result.plot(marker='o',ms=2,lw=1,ax=ax)fig.autofmt_xdate()plt.legend(bbox_to_anchor= [1,1])plt.title('COVID-19 Data of Different Provinces',size=15)plt.ylabel('number')plt.grid(axis='y')plt.show()

png

Conclusion:

Through these time series figures, I find that from February 22, 2020, South Korea began to increase the number of daily tests, and the number of newly diagnosed people in South Korea also increased rapidly. The most newly confirmed patients are in the age group 20s and 50s. There are more newly diagnosed female patients than male patients.A large number of newly confirmed cases came from the province of Gyeongsangbuk-do, and the number of newly confirmed diagnoses in other provinces gradually increased a few days later.

2.2 Prediction of the Recovery Time of the Patients

2.2.1 Data Preprocessing

#count miss value of each columnindex_array=np.sum(PatientInfo.isnull()==True,axis=0)print(index_array)print(PatientInfo.shape[0])
patient_id               0global_num            1160sex                     94birth_year             464age                    105country                 90province                 0city                    76disease               3110infection_case         819infection_order       3097infected_by           2393contact_number        2539symptom_onset_date    2682confirmed_date           0released_date         2147deceased_date         3072state                    0dtype: int643128

I want to build a regression model to predict of the recovery time of the patients.

Firstly I do data preprocessing to the data.It is easy to define the ‘recovery time’ as the difference between ‘released_date’ and ‘confirmed_date’.For the ‘deceased’patients,I choose to simply delete it in this task and I will find the rows that released_date is not nulland calculte the recovery time and join the table with the table Region to get more information.

Finally I will delete some useless column and do some data coding and discretization and normalization.

data_released=PatientInfo[PatientInfo['released_date'].notnull()]data_released=data_released[data_released['state']=='released']data_released['recover_days']=(pd.to_datetime(data_released['released_date'])-pd.to_datetime(data_released['confirmed_date'])).dt.daysRegion=pd.read_csv('Region.csv')result=pd.merge(data_released,Region,on=['city','province'])#count miss value of each column of the new tableindex_array=np.sum(result.isnull()==True,axis=0)print(index_array)
patient_id                    0global_num                  357sex                           0birth_year                  130age                           2country                       0province                      0city                          0disease                     952infection_case              451infection_order             926infected_by                 759contact_number              724symptom_onset_date          857confirmed_date                0released_date                 0deceased_date               952state                         0recover_days                  0code                          0latitude                      0longitude                     0elementary_school_count       0kindergarten_count            0university_count              0academy_ratio                 0elderly_population_ratio      0elderly_alone_ratio           0nursing_home_count            0dtype: int64

We notice that some columns with too much miss value should be ignored. Since birth_year is more accurate than age, we choose to fill the miss value in age and then use age to fill birth_year with media value i.e. 20s will be considered as 25 years old and thenwe use 2020-25 to calculate his birth_year.

age=result['age'].value_counts()age=age.sort_values(ascending=False)[:10]age
20s    25150s    19540s    16530s    11960s     9510s     4370s     3980s     210s      1390s      9Name: age, dtype: int64
#fill the miss value of the column age with mode '20s'result['age']=result['age'].fillna('20s')result['age_int']=result['age'].apply(lambdax: (int)(x[0:1]))result['birth_year']=result['birth_year'].fillna(2020-5-result['age_int']*10)index_array=np.sum(result.isnull()==True,axis=0)print(index_array)
patient_id                    0global_num                  357sex                           0birth_year                    0age                           0country                       0province                      0city                          0disease                     952infection_case              451infection_order             926infected_by                 759contact_number              724symptom_onset_date          857confirmed_date                0released_date                 0deceased_date               952state                         0recover_days                  0code                          0latitude                      0longitude                     0elementary_school_count       0kindergarten_count            0university_count              0academy_ratio                 0elderly_population_ratio      0elderly_alone_ratio           0nursing_home_count            0age_int                       0dtype: int64

I find that there is no miss value in the columns that will be used later. Then I will normalize the data and encode the column sex.

col=['birth_year','elementary_school_count','kindergarten_count','university_count','academy_ratio','elderly_population_ratio','latitude','longitude','elderly_alone_ratio','nursing_home_count','age_int']temp=result[col]#using min-max normalizationdata_pca=temptemp= (temp-temp.min())/(temp.max()-temp.min())Map={'male':0,'female':1}temp['sex']=result['sex']temp['sex']=temp['sex'].map(Map)temp['recover_days']=result['recover_days']temp['birth_year']=temp['birth_year']temp.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
birth_yearelementary_school_countkindergarten_countuniversity_countacademy_ratioelderly_population_ratiolatitudelongitudeelderly_alone_rationursing_home_countage_intsexrecover_days
00.4105260.2935780.2684210.10.2120420.2430180.8609910.1345850.1437130.3396910.555556013
10.4526320.2935780.2684210.10.2120420.2430180.8609910.1345850.1437130.3396910.555556010
20.6526320.1743120.1368420.10.0890050.3249910.8727900.2165560.2155690.2111150.333333032
30.4105260.0825690.0631580.30.3534030.3837500.8656190.1783040.2095810.2042090.555556020
40.4315790.0825690.0631580.30.3534030.3837500.8656190.1783040.2095810.2042090.555556119

2.2.2 Data Prediction

It is well known that how long a confirmed patient will recover mainly depends on his physical health condition and whether he has other underlying diseases. We do not have any data on the health indicators of patients, so it is very hard to build an accurate model.

I will compare some models to see how they work on this dataset.

Firstly I will divide the samples into training set and test set and the test set accounted for 30%.

y=temp['recover_days']X=temp.copy().drop(['recover_days'],axis=1)X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.3,random_state=33)
#1.LinearRegressionlr=LinearRegression()lr.fit(X_train,y_train)predicted=lr.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),3))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),3))
R_square_value: 0.077MSE_value: 52.682
#2.RidgeRegressionmodel=RidgeCV(alphas=[0.5,0.6,0.7,0.8,0.9,0.4,0.3,0.1,0.2,1])model.fit(X_train,y_train)predicted=model.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),4))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),4))
R_square_value: 0.0785MSE_value: 52.6262
#3.LassoRegressionmodel=LassoLarsCV()# LassoLarsCV自动调节alpha可以实现选择最佳的alphamodel.fit(X_train,y_train)predicted=model.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),4))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),4))
R_square_value: 0.0804MSE_value: 52.5148
#4.Assembled model: GradientBoostingRegressorgbr=GradientBoostingRegressor()gbr.fit(X_train,y_train)predicted=gbr.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),4))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),4))
R_square_value: 0.055MSE_value: 53.9639

Conclusion:

I find that no model is accurate because the most important information i.e. physical health condition and whether he has other underlying diseases is unknown.

Comparing the four models, I find that the Assembled model did worst because it may be overfitting in this dataset.

In the other three models, LassoRegression did best because it uses LASSO Regression with L1 form which has a very good variable selection effect.

RidgeRegression uses Ridge Regression with L2 form which has a good variable selection effect so it did better than LinearRegressionbut it did worse than LassoRegression.

The LinearRegression did worst in the three models without assembling because it did not use Regression to solve multicollinearity.

2.2.3 Feature Selection and Model Selection

We know that not all features in the dataset will affect the recover time of a patient and some columns may have multicollinearityand some columns maybe unrelated to the recover time.

A method to choose feature is using iteration and the greedy algorithm. In other words, we choose subset of the features and builda model on the training set and then test the model on the test set. If the model works better than the prvious one, we keep theselection, otherwise we reselect features and build models aga until the model will not be impoved by selecting features.

However, greedy algorithm can find a better model but the result may be not a global optimum solution because it may bestuck in a local optimum solution.

The followings are the new model I built after variable selection.

# Divide the samples into training set and test set.col1=['elementary_school_count','elderly_population_ratio','latitude','elderly_alone_ratio','nursing_home_count','age_int','recover_days']temp1=result[col1]y=temp1['recover_days']X=temp1.drop(['recover_days'],axis=1)X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.3,random_state=33)
#1.LinearRegressionlr=LinearRegression()lr.fit(X_train,y_train)predicted=lr.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),3))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),3))
R_square_value: 0.09MSE_value: 51.953
#2.RidgeRegressionmodel=RidgeCV(alphas=[0.5,0.6,0.7,0.8,0.9,0.4,0.3,0.1,0.2,1])model.fit(X_train,y_train)predicted=model.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),4))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),4))
R_square_value: 0.0902MSE_value: 51.9557
#3.LassoRegressionmodel=LassoLarsCV()# LassoLarsCV自动调节alpha可以实现选择最佳的alphamodel.fit(X_train,y_train)predicted=model.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),4))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),4))
R_square_value: 0.0903MSE_value: 51.9525
#Assembled model: GradientBoostingRegressorcol2=['elementary_school_count','latitude','elderly_alone_ratio','age_int','recover_days']temp2=result[col2]y=temp2['recover_days']X=temp2.drop(['recover_days'],axis=1)X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.3,random_state=33)gbr=GradientBoostingRegressor()gbr.fit(X_train,y_train)predicted=gbr.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),4))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),4))
R_square_value: 0.1109MSE_value: 50.7721

Conclusion:

By using feature selection, It is obvious that the R_square_value of the four models all increase and the MSE_value of the four modelsall decrease which means the four models are all improved.

The best model is the Assembled model: GradientBoostingRegressor because its R_square_value is maximal and its MSE_value is minimal.However, the assembled model has poor interpretability.

2.2.4 Using PCA to Do Data Preprocessing

data_pca
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
birth_yearelementary_school_countkindergarten_countuniversity_countacademy_ratioelderly_population_ratiolatitudelongitudeelderly_alone_rationursing_home_countage_int
01964.0365611.1714.3937.551166126.8495065.710805
11968.0365611.1714.3937.551166126.8495065.710805
21987.0233110.7016.6537.606832127.0926566.96893
31964.0131731.7118.2737.572999126.9791896.86685
41966.0131731.7118.2737.572999126.9791896.86685
....................................
9471969.0171621.2527.0135.686526127.91002117.41275
9481963.0171621.2527.0135.686526127.91002117.41275
9491998.011312341.5315.1033.488936126.5004236.412452
9501998.011312341.5315.1033.488936126.5004236.412452
9511974.011312341.5315.1033.488936126.5004236.412454

952 rows × 11 columns

#The dimension of the final data is 3pca=PCA(n_components=3)X_pca=pca.fit_transform(data_pca)df=pd.DataFrame(X_pca)df= (df-df.min())/(df.max()-df.min())df
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
012
00.3404370.4394500.577860
10.3404450.4378600.536146
20.2113650.4363800.309048
30.2040340.5023860.531499
40.2040380.5015910.510642
............
9470.0262610.3658370.490679
9480.0262490.3682220.553250
9490.3968940.1211350.330485
9500.3968940.1211350.330485
9510.3968460.1307520.582834

952 rows × 3 columns

y=temp['recover_days']X=dfX_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.3,random_state=33)
#1.LinearRegressionlr=LinearRegression()lr.fit(X_train,y_train)predicted=lr.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),3))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),3))
R_square_value: 0.051MSE_value: 54.216
#2.Assembled model: GradientBoostingRegressorgbr=GradientBoostingRegressor()gbr.fit(X_train,y_train)predicted=gbr.predict(X_test)print('R_square_value: ')print(round(r2_score(y_test,predicted),4))print('MSE_value: ')print(round(mean_squared_error(y_test,predicted),4))
R_square_value: 0.0117MSE_value: 56.4408

Conclusion:

After using PCA to Do Do Data Preprocessing, we did not get a better model as we have expected. The result is much worse than the result of using the Greedy Algorithm to do the Feature Selection.

I guess the reason is that maybe the dataset is not suitable for the method PCA to do data preprocessing.

2.4 Clustering of Risk Levels of the Cities

t=result['city'].value_counts()df=t.to_frame()df=df.reset_index()df=df.rename(columns={'city':'cnt'})df=df.rename(columns={'index':'city'})df.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
citycnt
0Gyeongsan-si439
1Cheonan-si89
2Gumi-si49
3Chilgok-gun35
4Yeongcheon-si22
t=pd.merge(df,Region,on='city')t.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
citycntcodeprovincelatitudelongitudeelementary_school_countkindergarten_countuniversity_countacademy_ratioelderly_population_ratioelderly_alone_rationursing_home_count
0Gyeongsan-si43960010Gyeongsangbuk-do35.825056128.7415443161101.3416.187.0427
1Cheonan-si8941120Chungcheongnam-do36.814980127.1138687511261.9110.424.51069
2Gumi-si4960040Gyeongsangbuk-do36.119641128.3442955010431.969.084.3616
3Chilgok-gun3560220Gyeongsangbuk-do35.995529128.401735213221.4815.176.7151
4Yeongcheon-si2260150Gyeongsangbuk-do35.973268128.938603182310.8327.3215.3192
col=['cnt','latitude','longitude','elementary_school_count','kindergarten_count','university_count','academy_ratio'     ,'elderly_population_ratio','elderly_alone_ratio','nursing_home_count','city']a=t[col]a.set_index(['city'],inplace=True)#min-max normalizationa= (a-a.min())/(a.max()-a.min())b=aa.head(5)
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
cntlatitudelongitudeelementary_school_countkindergarten_countuniversity_countacademy_ratioelderly_population_ratioelderly_alone_rationursing_home_count
city
Gyeongsan-si1.0000000.4951410.7724320.2477060.2984291.00.2565450.3079430.2215570.124959
Cheonan-si0.2009130.7049560.2237070.6513760.5654450.60.4057590.0990210.0718560.336074
Gumi-si0.1095890.5575790.6385110.4220180.5235600.30.4188480.0504170.0598800.187110
Chilgok-gun0.0776260.5312730.6578750.1559630.1465970.20.2931940.2713090.2035930.034199
Yeongcheon-si0.0479450.5265550.8388650.1284400.0994760.10.1230370.7120060.7185630.047682
# generate the linkage matrixZ=linkage(a,'complete','minkowski')
# calculate full dendrogramplt.figure(figsize=(25,10))plt.title('Hierarchical Clustering Dendrogram')plt.xlabel('sample index')plt.ylabel('distance')dendrogram(Z,no_labels=True)plt.show()

png

plt.title('Hierarchical Clustering Dendrogram (truncated)')plt.xlabel('sample index')plt.ylabel('distance')dendrogram(Z,truncate_mode='lastp',# show only the last p merged clustersp=3,# show only the last p merged clustersshow_leaf_counts=False,# otherwise numbers in brackets are countsshow_contracted=True,# to get a distribution impression in truncated branches)plt.show()

png

#We use the MDS method to reduce the dimension of the distance matrix, and use different colors to represent the clustering results.seed=np.random.RandomState(seed=3)mds=manifold.MDS(n_components=2,max_iter=3000,eps=1e-9,random_state=seed,n_jobs=1)pos=mds.fit(a).embedding_
k=3clusters=fcluster(Z,k,criterion='maxclust')plt.figure(figsize=(10,8))plt.scatter(pos[:,0],pos[:,1],c=clusters,cmap='prism')# plot points with cluster dependent colorsplt.show()

png

b=b.copy()b['cluster']=clustersb.groupby('cluster').mean()
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }
.dataframe tbody tr th {    vertical-align: top;}.dataframe thead th {    text-align: right;}
</style>
cntlatitudelongitudeelementary_school_countkindergarten_countuniversity_countacademy_ratioelderly_population_ratioelderly_alone_rationursing_home_count
cluster
116.16666736.474173127.45796576.000000135.1666674.9166671.91250012.2033335.4250001249.666667
24.09803937.150067127.06709229.09803943.8431371.9803921.48392215.0572556.333333664.137255
318.69444435.938680128.59165920.75000029.2500001.2500001.38861121.55055611.011111279.305556

Conclusion:

We use the MDS method to reduce the dimension of the distance matrix, and use different colors to represent the clustering results.The figure above shows that the results is good.

The cities are divided into three different risk levels: high, middle, or low.

In the high risk level cities(cluster 3), the number of confirmed patients, the elderly_alone_ratio and the elderly_population_ratiois the largest but the nursing_home_count and the academy_ratio is the smallest. It is very reasonable to classify the places with high number of diagnosed patients, serious aging, small number of nursing homes and people with low education level as high-risk areas.

In the mid risk level cities(cluster 1), although the number of confirmed patients is large, the nursing_home_count andthe academy_ratio are maximal and the elderly_population_ratio and the elderly_alone_ratio is minimal which means there aremore medical resources for every patient.

In the low risk level cities(cluster 2), the number of confirmed patients is fewest and the academy_ratio, elderly_population_ratio,elderly_alone_ratio and nursing_home_count are all middle valuse.

If I were the policy maker, I would distribute more medical resources to cities with high level of risk, and send medical teams to assist high-risk cities.

3. Conclusion:

By analyzing these data of South Korea, I found COVID-19 is highly transmitted and people of all ages are easily infected. The main transmission route is contact transmission, so reducing aggregation and paying attention to wearing masks are good protective measures. At the same time, we should assist medical resources and send medical teams to high-risk areas to help them fight the epidemic together and overcome difficulties.

About

Data Analyze and Visualization of Coronavirus Data of Korea

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

[8]ページ先頭

©2009-2025 Movatter.jp