Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit5cd2d18

Browse files
authored
Merge pull request#55 from OidaTiftla/onedrive
Onedrive
2 parentsd79295c +8287e1c commit5cd2d18

File tree

10 files changed

+238
-58
lines changed

10 files changed

+238
-58
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
config/prod.cfg
1010
config/client_secrets.json
1111
config/auth_token.json
12+
config/session.onedrive.pickle
1213
config/lastNewsletterUrl
1314

1415
ebooks/

‎README.md‎

Lines changed: 61 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,15 @@ This crawler automates the following step:
99
* parse title, description and useful information
1010
* download favorite format*.pdf .epub .mobi*
1111
* download source code and book cover
12-
* upload files to Google Drive or via scp
12+
* upload files to Google Drive, OneDrive or via scp
1313
* store data on Firebase
1414
* notify via email, IFTTT or Join (on success and errors)
1515
* schedule daily job on Heroku or with Docker
1616

1717
###Default command
1818
```bash
19-
# upload pdf todrive, store data and notify via email
20-
python script/spider.py -c config/prod.cfg -udrive -s firebase -n gmail
19+
# upload pdf togoogledrive, store data and notify via email
20+
python script/spider.py -c config/prod.cfg -ugoogledrive -s firebase -n gmail
2121
```
2222

2323
###Other options
@@ -33,9 +33,13 @@ python script/spider.py --config config/prod.cfg -t pdf --extras
3333
# equivalent (default is pdf)
3434
python script/spider.py -c config/prod.cfg -e
3535

36-
# download and then upload to Drive (given the download url anyone can download it)
37-
python script/spider.py -c config/prod.cfg -t epub --upload drive
38-
python script/spider.py --config config/prod.cfg --all --extras --upload drive
36+
# download and then upload to Google Drive (given the download url anyone can download it)
37+
python script/spider.py -c config/prod.cfg -t epub --upload googledrive
38+
python script/spider.py --config config/prod.cfg --all --extras --upload googledrive
39+
40+
# download and then upload to OneDrive (given the download url anyone can download it)
41+
python script/spider.py -c config/prod.cfg -t epub --upload onedrive
42+
python script/spider.py --config config/prod.cfg --all --extras --upload onedrive
3943

4044
# download and notify: gmail|ifttt|join
4145
python script/spider.py -c config/prod.cfg --notify gmail
@@ -64,42 +68,80 @@ Now you should be able to claim and download your first eBook
6468
python script/spider.py --config config/prod.cfg
6569
```
6670

67-
###Drive
71+
###GoogleDrive
6872

69-
From the documentation, Drive API requires OAuth2.0 for authentication, so to upload files you should:
73+
From the documentation,GoogleDrive API requires OAuth2.0 for authentication, so to upload files you should:
7074

71-
* Go to[Google APIs Console](https://code.google.com/apis/console) and create a new[Drive](https://console.developers.google.com/apis/api/drive/overview) project named**PacktpubDrive**
75+
* Go to[Google APIs Console](https://code.google.com/apis/console) and create a new[GoogleDrive](https://console.developers.google.com/apis/api/drive/overview) project named**PacktpubDrive**
7276
* On*API manager > Overview* menu
7377
* Enable Google Drive API
7478
* On*API manager > Credentials* menu
7579
* In*OAuth consent screen* tab set**PacktpubDrive** as the product name shown to users
7680
* In*Credentials* tab create credentials of type*OAuth client ID* and choose Application type*Other* named**PacktpubDriveCredentials**
7781
* Click*Download JSON* and save the file`config/client_secrets.json`
78-
* Change your Drive credentials in the config file
82+
* Change yourGoogleDrive credentials in the config file
7983

8084
```
81-
[drive]
85+
[googledrive]
8286
...
83-
drive.client_secrets=config/client_secrets.json
84-
drive.gmail=GOOGLE_DRIVE@gmail.com
87+
googledrive.client_secrets=config/client_secrets.json
88+
googledrive.gmail=GOOGLE_DRIVE@gmail.com
8589
```
8690

87-
Now you should be able to upload your eBook to Drive
91+
Now you should be able to upload your eBook toGoogleDrive
8892
```
89-
python script/spider.py --config config/prod.cfg --uploaddrive
93+
python script/spider.py --config config/prod.cfg --uploadgoogledrive
9094
```
9195

9296
Only the first time you will be prompted to login in a browser which has javascript enabled (no text-based browser) to generate`config/auth_token.json`.
9397
You should also copy and paste in the config the*FOLDER_ID*, otherwise every time a new folder with the same name will be created.
9498
```
95-
[drive]
99+
[googledrive]
96100
...
97-
drive.default_folder=packtpub
98-
drive.upload_folder=FOLDER_ID
101+
googledrive.default_folder=packtpub
102+
googledrive.upload_folder=FOLDER_ID
99103
```
100104

101105
Documentation:[OAuth](https://developers.google.com/api-client-library/python/guide/aaa_oauth),[Quickstart](https://developers.google.com/drive/v3/web/quickstart/python),[example](https://github.com/googledrive/python-quickstart) and[permissions](https://developers.google.com/drive/v2/reference/permissions)
102106

107+
###OneDrive
108+
109+
From the documentation, OneDrive API requires OAuth2.0 for authentication, so to upload files you should:
110+
111+
112+
* Go to the[Microsoft Application Registration Portal](https://apps.dev.microsoft.com/?referrer=https%3A%2F%2Fdev.onedrive.com%2Fapp-registration.htm).
113+
* When prompted, sign in with your Microsoft account credentials.
114+
* Find**My applications** and click**Add an app**.
115+
* Enter**PacktpubDrive** as the app's name and click**Create application**.
116+
* Scroll to the bottom of the page and check the**Live SDK support** box.
117+
* Change your OneDrive credentials in the config file
118+
* Copy your**Application Id** into the config file to**onedrive.client_id**
119+
* Click**Generate New Password** and copy the password shown into the config file to**onedrive.client_secret**
120+
* Click**Add Platform** and select**Web**
121+
* Enter**http://localhost:8080/** as the**Redirect URL**
122+
* Click**Save** at the bottom of the page
123+
124+
```
125+
[onedrive]
126+
...
127+
onedrive.client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
128+
onedrive.client_secret=XxXxXxXxXxXxXxXxXxXxXxX
129+
```
130+
131+
Now you should be able to upload your eBook to OneDrive
132+
```
133+
python script/spider.py --config config/prod.cfg --upload onedrive
134+
```
135+
136+
Only the first time you will be prompted to login in a browser which has javascript enabled (no text-based browser) to generate`config/session.onedrive.pickle`.
137+
```
138+
[onedrive]
139+
...
140+
onedrive.folder=packtpub
141+
```
142+
143+
Documentation:[Registration](https://dev.onedrive.com/app-registration.htm),[Python API](https://github.com/OneDrive/onedrive-sdk-python)
144+
103145
###Scp
104146

105147
To upload your eBook via`scp` on a remote server update the configs
@@ -136,7 +178,7 @@ firebase.url=https://PROJECT_NAME.firebaseio.com
136178

137179
Now you should be able to store your eBook details on Firebase
138180
```
139-
python script/spider.py --config config/prod.cfg --uploaddrive --store firebase
181+
python script/spider.py --config config/prod.cfg --uploadgoogledrive --store firebase
140182
```
141183

142184
###Gmail notification

‎config/dev.cfg‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@ path.ebooks=ebooks
2020
path.extras=ebooks/extras
2121
#path.group=true
2222

23-
[drive]
24-
drive.oauth2_scope=https://www.googleapis.com/auth/drive
25-
drive.client_secrets=config/client_secrets.json
26-
drive.auth_token=config/auth_token.json
27-
drive.gmail=g00gler@gmail.com
23+
[googledrive]
24+
googledrive.oauth2_scope=https://www.googleapis.com/auth/drive
25+
googledrive.client_secrets=config/client_secrets.json
26+
googledrive.auth_token=config/auth_token.json
27+
googledrive.gmail=g00gler@gmail.com

‎config/prod_example.cfg‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,20 @@ path.ebooks=ebooks
1818
path.extras=ebooks/extras
1919
#path.group=true
2020

21-
[drive]
22-
drive.oauth2_scope=https://www.googleapis.com/auth/drive
23-
drive.client_secrets=config/client_secrets.json
24-
drive.auth_token=config/auth_token.json
25-
drive.gmail=GOOGLE_DRIVE@gmail.com
26-
drive.default_folder=packtpub
27-
#drive.upload_folder=FOLDER_ID
21+
[googledrive]
22+
googledrive.oauth2_scope=https://www.googleapis.com/auth/drive
23+
googledrive.client_secrets=config/client_secrets.json
24+
googledrive.auth_token=config/auth_token.json
25+
googledrive.gmail=GOOGLE_DRIVE@gmail.com
26+
googledrive.default_folder=packtpub
27+
#googledrive.upload_folder=FOLDER_ID
28+
29+
[onedrive]
30+
onedrive.api_base_url=https://api.onedrive.com/v1.0/
31+
onedrive.client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
32+
onedrive.client_secret=XxXxXxXxXxXxXxXxXxXxXxX
33+
onedrive.session_file=config/session.onedrive.pickle
34+
onedrive.folder=packtpub
2835

2936
[scp]
3037
scp.host=SCP_HOST

‎requirements.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,5 @@ python-firebase==1.2
1212
paramiko==2.0.2
1313
cryptography==1.6
1414
scp==0.10.2
15+
onedrivesdk==1.1.8
16+

‎script/drive.py‎renamed to ‎script/googledrive.py‎

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,13 @@
99
fromutilsimportthread_loader
1010
fromlogsimport*
1111

12-
classDrive(object):
12+
classGoogleDrive(object):
1313
"""
1414
"""
1515

1616
def__init__(self,config):
1717
self.__config=config
18-
self.__drive_service=None
18+
self.__googledrive_service=None
1919
self.info= {}
2020

2121
def__guess_info(self,file_path):
@@ -27,11 +27,11 @@ def __guess_info(self, file_path):
2727
'name':file_path.split('/')[-1],
2828
'mime_type':magic.from_file(file_path,mime=True),
2929
}
30-
log_info('[+] new file upload on Drive:')
30+
log_info('[+] new file upload onGoogleDrive:')
3131
# log_dict(self.file_info)
3232

3333
def__init_service(self):
34-
auth_token=self.__config.get('drive','drive.auth_token')
34+
auth_token=self.__config.get('googledrive','googledrive.auth_token')
3535

3636
ifnotexists(auth_token):
3737
self.__save_credentials(auth_token)
@@ -41,12 +41,12 @@ def __init_service(self):
4141

4242
http=httplib2.Http()
4343
http=credentials.authorize(http)
44-
self.__drive_service=build('drive','v2',http=http)
44+
self.__googledrive_service=build('drive','v2',http=http)
4545

4646
def__save_credentials(self,auth_token):
4747
flow=flow_from_clientsecrets(
48-
self.__config.get('drive','drive.client_secrets'),
49-
self.__config.get('drive','drive.oauth2_scope'),
48+
self.__config.get('googledrive','googledrive.client_secrets'),
49+
self.__config.get('googledrive','googledrive.oauth2_scope'),
5050
OOB_CALLBACK_URN)
5151

5252
authorize_url=flow.step1_get_authorize_url()
@@ -63,35 +63,35 @@ def __save_credentials(self, auth_token):
6363

6464
def__create_folder(self):#Create folder with provided name
6565
try:#Check default folder name
66-
default_folder_name=self.__config.get('drive','drive.default_folder')
66+
default_folder_name=self.__config.get('googledrive','googledrive.default_folder')
6767
except:
6868
default_folder_name='packtpub'
6969

7070
metadata= {
7171
'title':default_folder_name,
7272
'mimeType' :'application/vnd.google-apps.folder'
7373
}
74-
folder=self.__drive_service.files().insert(body=metadata).execute()
75-
self.__config.set('drive','drive.upload_folder',folder['id'])
74+
folder=self.__googledrive_service.files().insert(body=metadata).execute()
75+
self.__config.set('googledrive','googledrive.upload_folder',folder['id'])
7676
log_success('[+] creating new directory...')
7777
print'[+] updating folder permissions...'
7878
permissions= {
7979
'role':'reader',
8080
'type':'anyone',
81-
'value':self.__config.get('drive','drive.gmail')
81+
'value':self.__config.get('googledrive','googledrive.gmail')
8282
}
83-
self.__drive_service.permissions().insert(fileId=folder['id'],body=permissions).execute()
83+
self.__googledrive_service.permissions().insert(fileId=folder['id'],body=permissions).execute()
8484
log_dict({'folder_name':default_folder_name,
8585
'id':folder['id'],})
8686
#'share_link': folder['webContentLink']}) #TODO Fix
87-
log_success('[+] Please add this line after [drive] in your configuration file:')
88-
log_info('drive.upload_folder='+folder.get('id'))
87+
log_success('[+] Please add this line after [googledrive] in your configuration file:')
88+
log_info('googledrive.upload_folder='+folder.get('id'))
8989

9090
returnfolder.get('id')#Return folder object ID
9191

9292
def__get_folder(self):#Get folder name settings
9393
try:
94-
returnself.__config.get('drive','drive.upload_folder')
94+
returnself.__config.get('googledrive','googledrive.upload_folder')
9595
except:
9696
returnself.__create_folder()#new folder ID
9797

@@ -105,18 +105,18 @@ def __insert_file(self):
105105
'mimeType':self.info['mime_type'],
106106
'parents': [{'id':self.__get_folder()}]
107107
}
108-
file=self.__drive_service.files().insert(body=body,media_body=media_body).execute()
108+
file=self.__googledrive_service.files().insert(body=body,media_body=media_body).execute()
109109
# log_dict(file)
110110

111111
print'[+] updating file permissions...'
112112
permissions= {
113113
'role':'reader',
114114
'type':'anyone',
115-
'value':self.__config.get('drive','drive.gmail')
115+
'value':self.__config.get('googledrive','googledrive.gmail')
116116
}
117-
self.__drive_service.permissions().insert(fileId=file['id'],body=permissions).execute()
117+
self.__googledrive_service.permissions().insert(fileId=file['id'],body=permissions).execute()
118118

119-
# self.__drive_service.files().get(fileId=file['id']).execute()
119+
# self.__googledrive_service.files().get(fileId=file['id']).execute()
120120

121121
self.info['id']=file['id']
122122
self.info['download_url']=file['webContentLink']

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp