Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit4d214be

Browse files
authored
scraper: POVR Scraper Fix (#1825)
* Fix for POVR scraperThey are now putting porn star suggestions on the same page as scenes. This has the same search parameters as the scenes. The easist way is to make sure that links only in the `/vr-porn` directory are scraped. As there really isn't any identifiable way to target only the scene links through Query Selectors* Typo fix* Add trailing slashJust to make sure it matches a directory
1 parent7ca0840 commit4d214be

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

‎pkg/scrape/povr.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -124,8 +124,8 @@ func POVR(wg *sync.WaitGroup, updateSite bool, knownScenes []string, out chan<-
124124
siteCollector.OnHTML(`div.thumbnail-wrap div.thumbnail a.thumbnail__link`,func(e*colly.HTMLElement) {
125125
sceneURL:=e.Request.AbsoluteURL(e.Attr("href"))
126126

127-
// If scene exists in database, or the slternate source exists, there's no need to scrape
128-
if!funk.ContainsString(knownScenes,sceneURL)&&!strings.Contains(sceneURL,"/join") {
127+
// If scene exists in database, or the slternate source exists, there's no need to scrape. Also make sure we only grab valid scene links in the vr-porn directory
128+
if!funk.ContainsString(knownScenes,sceneURL)&&strings.Contains(sceneURL,"/vr-porn/")&&!strings.Contains(sceneURL,"/join") {
129129
WaitBeforeVisit("povr.com",sceneCollector.Visit,sceneURL)
130130
}
131131
})

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp