Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Need for better webpage retrieval#188

mayurdb started this conversation inGeneral
Discussion options

Currently, FetchNode uses the AsyncChromiumLoader to load the page and the returned content is used further.

I think it would be good if we invested more effort into getting more structured and detailed webpage content:
I mean getting a dict out of the page that includes

  1. Static content
  2. Forms
  3. Image URLs
  4. Links
  5. Getting dynamically loaded content on the page (I mean on scroll etc)
You must be logged in to vote

Replies: 2 comments

Comment options

I think there should be a node that just saves the data scraped either to a vector store or to a local directory. Sometimes the need for the scraping is not just to generate answers.

You must be logged in to vote
0 replies
Comment options

There is the rag node

You must be logged in to vote
0 replies
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Category
General
Labels
None yet
3 participants
@mayurdb@vvkishere@VinciGit00
[8]ページ先頭
©2009-2025 Movatter.jp