NotificationsYou must be signed in to change notification settings
Fork0
Star0

Commit925990e

authored

Merge pull requestflairNLP#798 from bresslem/add-publisher-der-freitag

New publisher added: Der Freitag

2 parentsba61e0f +a70ce32 commit925990eCopy full SHA for 925990e

File tree

6 files changed

+232

-0

lines changed

docs
- supported_publishers.md
src/fundus/publishers/de
- __init__.py
- der_freitag.py
tests/resources/parser/test_data/de

6 files changed

+232

-0

lines changed

`‎docs/supported_publishers.md‎`

Lines changed: 19 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -1361,6 +1361,25 @@`
`1361`	`1361`	`<td> </td>`
`1362`	`1362`	`<td> </td>`
`1363`	`1363`	`</tr>`
	`1364`	`+<tr>`
	`1365`	`+ <td>`
	`1366`	`+ <code>DerFreitag</code>`
	`1367`	`+ </td>`
	`1368`	`+ <td>`
	`1369`	`+ <div>der Freitag</div>`
	`1370`	`+ </td>`
	`1371`	`+ <td>`
	`1372`	`+ <a href="https://www.freitag.de/">`
	`1373`	`+ <span>www.freitag.de</span>`
	`1374`	`+ </a>`
	`1375`	`+ </td>`
	`1376`	`+ <td>`
	`1377`	`+ <code>de</code>`
	`1378`	`+ </td>`
	`1379`	`+ <td> </td>`
	`1380`	`+ <td> </td>`
	`1381`	`+ <td> </td>`
	`1382`	`+</tr>`
`1364`	`1383`	`<tr>`
`1365`	`1384`	`<td>`
`1366`	`1385`	`<code>NetzpolitikOrg</code>`

`‎src/fundus/publishers/de/init.py‎`

Lines changed: 11 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -12,6 +12,7 @@`
`12`	`12`	`from .boersenzeitungimportBoersenZeitungParser`
`13`	`13`	`from .brimportBRParser`
`14`	`14`	`from .business_insider_deimportBusinessInsiderDEParser`
	`15`	`+from .der_freitagimportDerFreitagParser`
`15`	`16`	`from .die_weltimportDieWeltParser`
`16`	`17`	`from .die_zeitimportDieZeitParser`
`17`	`18`	`from .dwimportDWParser`
`@@ -605,3 +606,13 @@ class DE(metaclass=PublisherGroup):`
`605`	`606`	`Sitemap("https://www.gamestar.de/artikel_archiv_index.xml"),`
`606`	`607`	`],`
`607`	`608`	`)`
	`609`	`+`
	`610`	`+DerFreitag=Publisher(`
	`611`	`+name="der Freitag",`
	`612`	`+domain="https://www.freitag.de/",`
	`613`	`+parser=DerFreitagParser,`
	`614`	`+sources=[`
	`615`	`+RSSFeed("https://www.freitag.de/@@RSS"),`
	`616`	`+Sitemap("https://www.freitag.de/sitemap.xml",sitemap_filter=inverse(regex_filter("sitemap-articles"))),`
	`617`	`+ ],`
	`618`	`+ )`

`‎src/fundus/publishers/de/der_freitag.py‎`

Lines changed: 57 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,57 @@`
	`1`	`+fromdatetimeimportdate,datetime`
	`2`	`+fromtypingimportList,Optional`
	`3`	`+`
	`4`	`+fromlxml.cssselectimportCSSSelector`
	`5`	`+fromlxml.etreeimportXPath`
	`6`	`+`
	`7`	`+fromfundus.parserimportArticleBody,BaseParser,Image,ParserProxy,attribute`
	`8`	`+fromfundus.parser.utilityimport (`
	`9`	`+extract_article_body_with_selector,`
	`10`	`+generic_author_parsing,`
	`11`	`+generic_date_parsing,`
	`12`	`+image_extraction,`
	`13`	`+)`
	`14`	`+`
	`15`	`+`
	`16`	`+classDerFreitagParser(ParserProxy):`
	`17`	`+classV1(BaseParser):`
	`18`	`+_summary_selector=CSSSelector("header > p.bc-article-intro__text")`
	`19`	`+_paragraph_selector=CSSSelector("div.bo-article-text > p")`
	`20`	`+_subheadline_selector=CSSSelector("div.bo-article-text > h2")`
	`21`	`+`
	`22`	`+@attribute`
	`23`	`+deftitle(self)->Optional[str]:`
	`24`	`+returnself.precomputed.meta.get("og:title")`
	`25`	`+`
	`26`	`+@attribute`
	`27`	`+defbody(self)->Optional[ArticleBody]:`
	`28`	`+returnextract_article_body_with_selector(`
	`29`	`+self.precomputed.doc,`
	`30`	`+summary_selector=self._summary_selector,`
	`31`	`+subheadline_selector=self._subheadline_selector,`
	`32`	`+paragraph_selector=self._paragraph_selector,`
	`33`	`+ )`
	`34`	`+`
	`35`	`+@attribute`
	`36`	`+defauthors(self)->List[str]:`
	`37`	`+returngeneric_author_parsing(self.precomputed.ld.bf_search("author"))`
	`38`	`+`
	`39`	`+@attribute`
	`40`	`+defpublishing_date(self)->Optional[datetime]:`
	`41`	`+returngeneric_date_parsing(self.precomputed.ld.bf_search("datePublished"))`
	`42`	`+`
	`43`	`+@attribute`
	`44`	`+deftopics(self)->List[str]:`
	`45`	`+returnself.precomputed.ld.bf_search("keywords")`
	`46`	`+`
	`47`	`+@attribute`
	`48`	`+defimages(self)->List[Image]:`
	`49`	`+returnimage_extraction(`
	`50`	`+doc=self.precomputed.doc,`
	`51`	`+paragraph_selector=self._paragraph_selector,`
	`52`	`+upper_boundary_selector=CSSSelector("header.bc-article-intro"),`
	`53`	`+lower_boundary_selector=CSSSelector("span.freitag-article-end"),`
	`54`	`+image_selector=CSSSelector("figure img,div[role='figure'] img"),`
	`55`	`+caption_selector=XPath("./ancestor::figure//figcaption//span[@class='bo-image__caption__desc']"),`
	`56`	`+author_selector=XPath("./ancestor::figure//figcaption//span[@class='bo-image__caption__credit']"),`
	`57`	`+ )`

0 commit comments

Comments

(0)

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Commit925990e

File tree

6 files changed

6 files changed

`‎docs/supported_publishers.md‎`

`‎src/fundus/publishers/de/init.py‎`

`‎src/fundus/publishers/de/der_freitag.py‎`

0 commit comments

Movatterモバイル変換

File tree

6 files changed

6 files changed

‎docs/supported_publishers.md‎

‎src/fundus/publishers/de/__init__.py‎

‎src/fundus/publishers/de/der_freitag.py‎

0 commit comments

`‎docs/supported_publishers.md‎`

`‎src/fundus/publishers/de/init.py‎`

`‎src/fundus/publishers/de/der_freitag.py‎`