- Notifications
You must be signed in to change notification settings - Fork845
Introduce Markdown readers#6969
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Pull Request Overview
This PR introduces Markdown reading capabilities to the data ingestion library through two new readers:MarkdownReader for native Markdown files andMarkItDownReader that leverages the external MarkItDown tool to convert various document formats to Markdown before parsing.
Key changes:
- Adds
MarkdownReaderfor parsing.mdfiles using the Markdig library - Adds
MarkItDownReaderthat wraps the MarkItDown CLI tool to convert documents (PDF, DOCX, etc.) to Markdown - Introduces shared
MarkdownParserto parse Markdig AST intoIngestionDocumentmodel - Implements comprehensive test suite with conformance tests and format-specific test cases
Reviewed Changes
Copilot reviewed 12 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
MarkdownReader.cs | Implements reader for native Markdown files using Markdig parser |
MarkdownParser.cs | Core parsing logic converting Markdig AST to IngestionDocument model |
MarkItDownReader.cs | Wraps MarkItDown CLI tool to convert various document formats to Markdown |
Microsoft.Extensions.DataIngestion.Markdown.csproj | Project file for MarkdownReader with Markdig dependency |
Microsoft.Extensions.DataIngestion.MarkItDown.csproj | Project file for MarkItDownReader, shares MarkdownParser code |
DocumentReaderConformanceTests.cs | Base test class defining conformance tests for document readers |
MarkdownReaderTests.cs | Tests specific to MarkdownReader functionality |
MarkItDownReaderTests.cs | Tests specific to MarkItDownReader with CLI availability checks |
ArrayUtils.cs | Test utility for mapping 2D arrays used in table assertions |
Microsoft.Extensions.DataIngestion.Tests.csproj | Updated project file adding references and test file configuration |
General.props | Adds Markdig package reference |
Versions.props | Specifies Markdig version 0.42.0 |
src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
...ies/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
…sions.DataIngestion.Markdig
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Readers/MarkItDownReaderTests.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Pull Request Overview
Copilot reviewed 13 out of 20 changed files in this pull request and generated 3 comments.
src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownReader.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.MarkItDown/MarkItDownReader.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Libraries/Microsoft.Extensions.DataIngestion.Markdig/MarkdownParser.csShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
# Conflicts:#test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj
- delete temporary file when .CopyToAsync fails- handle all image types
…rectory to a "safe location"
Uh oh!
There was an error while loading.Please reload this page.
Microsoft Reviewers:Open in CodeFlow