- Notifications
You must be signed in to change notification settings - Fork0
oxylabs/python-script-service-guide
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A service (also known as a "daemon") is a process that performs tasks in the background and responds to system events.
Services can be written using any language. We use Python in these examples as it is one of the most versatile languages out there.
For more information, be sure to readour blogpost on the subject.
To run any of the examples, you will need Python 3. We also recommendusing a virtual environment.
python3 -m venv venv
First, create a script that scrapes a website. Make sure the script also handles OS signals so that it exits gracefully.
linux_scrape.py:
importjsonimportreimportsignalfrompathlibimportPathimportrequestsfrombs4importBeautifulSoupclassSignalHandler:shutdown_requested=Falsedef__init__(self):signal.signal(signal.SIGINT,self.request_shutdown)signal.signal(signal.SIGTERM,self.request_shutdown)defrequest_shutdown(self,*args):print('Request to shutdown received, stopping')self.shutdown_requested=Truedefcan_run(self):returnnotself.shutdown_requestedsignal_handler=SignalHandler()urls= ['https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html','https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html','https://books.toscrape.com/catalogue/sharp-objects_997/index.html',]index=0whilesignal_handler.can_run():url=urls[index%len(urls)]index+=1print('Scraping url',url)response=requests.get(url)soup=BeautifulSoup(response.content,'html.parser')book_name=soup.select_one('.product_main').h1.textrows=soup.select('.table.table-striped tr')product_info= {row.th.text:row.td.textforrowinrows}data_folder=Path('./data')data_folder.mkdir(parents=True,exist_ok=True)json_file_name=re.sub('[\': ]','-',book_name)json_file_path=data_folder/f'{json_file_name}.json'withopen(json_file_path,'w')asbook_file:json.dump(product_info,book_file)
Then, create a systemd configuration file.
/etc/systemd/system/book-scraper.service:
[Unit]Description=A script for scraping the book informationAfter=syslog.target network.target[Service]WorkingDirectory=/home/oxylabs/python-script-service/src/systemdExecStart=/home/oxylabs/python-script-service/venv/bin/python3 main.pyRestart=alwaysRestartSec=120[Install]WantedBy=multi-user.targetMake sure to adjust the paths based on your actual script location.
A fully working example can be foundhere.
To create a Windows service, you will need to implement methods such asSvcDoRun andSvcStop and handle events sent by the operating system.
windows_scrape.py:
importsysimportservicemanagerimportwin32eventimportwin32serviceimportwin32serviceutilimportjsonimportrefrompathlibimportPathimportrequestsfrombs4importBeautifulSoupclassBookScraperService(win32serviceutil.ServiceFramework):_svc_name_='BookScraperService'_svc_display_name_='BookScraperService'_svc_description_='Constantly updates the info about books'def__init__(self,args):win32serviceutil.ServiceFramework.__init__(self,args)self.event=win32event.CreateEvent(None,0,0,None)defGetAcceptedControls(self):result=win32serviceutil.ServiceFramework.GetAcceptedControls(self)result|=win32service.SERVICE_ACCEPT_PRESHUTDOWNreturnresultdefSvcDoRun(self):urls= ['https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html','https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html','https://books.toscrape.com/catalogue/sharp-objects_997/index.html', ]index=0whileTrue:result=win32event.WaitForSingleObject(self.event,5000)ifresult==win32event.WAIT_OBJECT_0:breakurl=urls[index%len(urls)]index+=1print('Scraping url',url)response=requests.get(url)soup=BeautifulSoup(response.content,'html.parser')book_name=soup.select_one('.product_main').h1.textrows=soup.select('.table.table-striped tr')product_info= {row.th.text:row.td.textforrowinrows}data_folder=Path('C:\\Users\\User\\Scraper\\dist\\scrape\\data')data_folder.mkdir(parents=True,exist_ok=True)json_file_name=re.sub('[\': ]','-',book_name)json_file_path=data_folder/f'{json_file_name}.json'withopen(json_file_path,'w')asbook_file:json.dump(product_info,book_file)defSvcStop(self):self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)win32event.SetEvent(self.event)if__name__=='__main__':iflen(sys.argv)==1:servicemanager.Initialize()servicemanager.PrepareToHostSingle(BookScraperService)servicemanager.StartServiceCtrlDispatcher()else:win32serviceutil.HandleCommandLine(BookScraperService)
Next, install dependencies and run a post-install script.
PS C:\> cd C:\Users\User\ScraperPS C:\Users\User\Scraper> .\venv\Scripts\pip install pypiwin32PS C:\Users\User\Scraper> .\venv\Scripts\pywin32_postinstall.py -installBundle your script into an executable.
PS C:\Users\User\Scraper> venv\Scripts\pyinstaller --hiddenimport win32timezone -F scrape.pyAnd finally, install your newly-created service.
PS C:\Users\User\Scraper> .\dist\scrape.exe installInstalling service BookScraperChanging service configurationService updatedPS C:\Users\User\Scraper> .\dist\scrape.exe startStarting service BookScraperPS C:\Users\User\Scripts>A fully working example can be foundhere.
Instead of dealing with the Windows service layer directly, you can use the NSSM (Non-Sucking Service Manager).
Install NSSM by visitingthe official website. Extract it to a folder of your choice and add the folder to your PATH environment variable for convenience.
Once you have NSSM installed, simplify your script by getting rid of all Windows-specific methods and definitions.
simple_scrape.py:
importjsonimportrefrompathlibimportPathimportrequestsfrombs4importBeautifulSoupurls= ['https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html','https://books.toscrape.com/catalogue/shakespeares-sonnets_989/index.html','https://books.toscrape.com/catalogue/sharp-objects_997/index.html', ]index=0whileTrue:url=urls[index%len(urls)]index+=1print('Scraping url',url)response=requests.get(url)soup=BeautifulSoup(response.content,'html.parser')book_name=soup.select_one('.product_main').h1.textrows=soup.select('.table.table-striped tr')product_info= {row.th.text:row.td.textforrowinrows}data_folder=Path('C:\\Users\\User\\Scraper\\data')data_folder.mkdir(parents=True,exist_ok=True)json_file_name=re.sub('[\': ]','-',book_name)json_file_path=data_folder/f'{json_file_name}.json'withopen(json_file_path,'w')asbook_file:json.dump(product_info,book_file)
Bundle your script into an executable.
PS C:\Users\User\Scraper> venv\Scripts\pyinstaller -F simple_scrape.pyAnd finally, install the script using NSSM.
PS C:\> nssm.exe install SimpleScrape C:\Users\User\Scraper\dist\simple_scrape.exePS C:\Users\User\Scraper> .\venv\Scripts\pip install pypiwin32A fully working script can be foundhere.
About
A guide on running a Python script as a service on Windows & Linux.
Topics
Resources
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Releases
Packages0
Uh oh!
There was an error while loading.Please reload this page.