Repository files navigation

WIP

• Main Content Extraction
• Pure JS Impl (NO DOM API for Cloudflare)
• Pass original tests
• Custom Parser (now only supportshtmlparser2)
• Accessibility
• Markdown Serializer
• Navigation Detection
• Document Structure Analysis
• Test Coverage Support

$ npm install -g @mizchi/readability

$ readability https://example.com/article# Outputs extracted content as markdown

$ readability --helpUsage: @mizchi/readability [options]<url>Options:  -t, --threshold<number>        Character thresholdfor extraction (default: 250)  -h, --help                      Show thishelp message  -f, --format<format>           Output format: md, html, doc, ai-summary, ai-structured (default: md)  -o, --out<file>                Output file path (default: stdout)# Progressive Analysis Options  --analyze-structure             Analyze page structure without extracting content  --extract-nav                   Extract navigation information only  --extract-content               Extract main content only  --with-context                  Include context when extracting content  --full-analysis                 Performcomplete page analysis# Navigation Options  --nav-only                      Extract navigation information (legacy)  --nav-type<type>               Filter navigation bytype (global/local/breadcrumb/toc/sidebar/footer)  --nav-location<location>       Filter navigation by location (header/main/sidebar/footer)# Document Mode Options  --doc-mode                      Extract documentation structure

# Extract content as markdown$ readability https://example.com/article# Extract content with AI-friendly summary$ readability -f ai-summary https://example.com/article# Analyze page structure$ readability --analyze-structure https://example.com/article# Extract navigation only$ readability --extract-nav https://example.com/article# Extract content with context (breadcrumb, surrounding navigation)$ readability --extract-content --with-context https://example.com/article# Full page analysis (structure + navigation + content)$ readability --full-analysis https://example.com/article# Documentation mode (for technical documentation sites)$ readability --doc-mode https://docs.example.com/guide# Save output to file$ readability -o article.md https://example.com/article

• md (default): Markdown format
• html: HTML format
• doc: Documentation structure with navigation
• ai-summary: AI-optimized summary with metadata
• ai-structured: Structured format for AI processing

{"mcpServers": {"readability": {"command":"npx","args": ["-y","@mizchi/readability","--mcp"]    }  }}

Prompt

Given a URL, use read_url_content_as_markdown and summary contents

# Run tests with coveragenpm run test:coverage# Run all tests with coverage (including excluded tests)npm run test:coverage:all# View coverage report in browseropen coverage/index.html

Current coverage thresholds:

• Lines: 50%
• Functions: 60%
• Branches: 75%
• Statements: 50%

npm install --save @mizchi/readability

import{extract,toMarkdown}from"@mizchi/readability";consturl="https://zenn.dev/mizchi/articles/ts-using-sampling-logger";consthtml=awaitfetch(url).then((res)=>res.text());constextracted=extract(html,{charThreshold:100,});constparsed=toMarkdown(extracted.root);console.log(parsed);

import{extractAriaTree,ariaTreeToString}from"@mizchi/readability";consthtml=awaitfetch("https://zenn.dev").then((res)=>res.text());consttree=extractAriaTree(html);conststr=ariaTreeToString(tree);console.log(str);

A standalone version of the readability library used forFirefox Reader View.

Readability is available on npm:

npm install @mozilla/readability

You can thenrequire() it, or for web-based projects, load theReadability.js script from your webpage.

To parse a document, you must create a newReadability object from a DOMdocument object, and then call theparse() method. Here's anexample:

vararticle=newReadability(document).parse();

If you use Readability in a web browser, you will likely be able to use adocument reference from elsewhere (e.g. fetched via XMLHttpRequest, in asame-origin<iframe> you have access to, etc.). In Node.js, you canuse an external DOM library.

Theoptions object accepts a number of properties, all optional:

• debug (boolean, defaultfalse): whether to enable logging.
• maxElemsToParse (number, default0 i.e. no limit): the maximum number ofelements to parse.
• nbTopCandidates (number, default5): the number of top candidates toconsider when analysing how tight the competition is among candidates.
• charThreshold (number, default500): the number of characters an articlemust have in order to return a result.
• classesToPreserve (array): a set of classes to preserve on HTML elementswhen thekeepClasses options is set tofalse.
• keepClasses (boolean, defaultfalse): whether to preserve all classes onHTML elements. When set tofalse only classes specified in theclassesToPreserve array are kept.
• disableJSONLD (boolean, defaultfalse): when extracting page metadata,Readability gives precedence to Schema.org fields specified in the JSON-LDformat. Set this option totrue to skip JSON-LD parsing.
• serializer (function, defaultel => el.innerHTML) controls how thecontent property returned by theparse() method is produced from the rootDOM element. It may be useful to specify theserializer as the identityfunction (el => el) to obtain a DOM element instead of a string forcontent if you plan to process it further.
• allowedVideoRegex (RegExp, defaultundefined ): a regular expression thatmatches video URLs that should be allowed to be included in the articlecontent. Ifundefined, thedefault regexis applied.
• linkDensityModifier (number, default0): a number that is added to thebase link density threshold during the shadiness checks. This can be used topenalize nodes with a high link density or vice versa.

Returns an object containing the following properties:

• title: article title;
• content: HTML string of processed article content;
• textContent: text content of the article, with all the HTML tags removed;
• length: length of an article, in characters;
• excerpt: article description, or short excerpt from the content;
• byline: author metadata;
• dir: content direction;
• siteName: name of the site;
• lang: content language;
• publishedTime: published time;

Theparse() method works by modifying the DOM. This removes some elements inthe web page, which may be undesirable. You can avoid this by passing the cloneof thedocument object to theReadability constructor:

vardocumentClone=document.cloneNode(true);vararticle=newReadability(documentClone).parse();

A quick-and-dirty way of figuring out if it's plausible that the contents of agiven document are suitable for processing with Readability. It is likely toproduce both false positives and false negatives. The reason it exists is toavoid bogging down a time-sensitive process (like loading and showing the user awebpage) with the complex logic in the core of Readability. Improvements to itslogic (while not deteriorating its performance) are very welcome.

Theoptions object accepts a number of properties, all optional:

• minContentLength (number, default140): the minimum node content lengthused to decide if the document is readerable;
• minScore (number, default20): the minimum cumulated 'score' used todetermine if the document is readerable;
• visibilityChecker (function, defaultisNodeVisible): the function used todetermine if a node is visible;

The function returns a boolean corresponding to whether or not we suspectReadability.parse() will succeed at returning an article object. Here's anexample:

/*    Only instantiate Readability  if we suspect    the `parse()` method will produce a meaningful result.*/if(isProbablyReaderable(document)){letarticle=newReadability(document).parse();}

Since Node.js does not come with its own DOM implementation, we rely on externallibraries likejsdom. Here's an example usingjsdom to obtain a DOM document object:

var{ Readability}=require("@mozilla/readability");var{JSDOM}=require("jsdom");vardoc=newJSDOM("<body>Look at this cat: <img src='./cat.jpg'></body>",{url:"https://www.example.com/the-page-i-got-the-source-from",});letreader=newReadability(doc.window.document);letarticle=reader.parse();

Remember to pass the page's URI as theurl option in theJSDOM constructor(as shown in the example above), so that Readability can convert relative URLsfor images, hyperlinks, etc. to their absolute counterparts.

jsdom has the ability to run the scripts included in the HTML and fetch remoteresources. For security reasons these aredisabled by default, and westrongly recommend you keep them that way.

If you're going to use Readability with untrusted input (whether in HTML or DOMform), westrongly recommend you use a sanitizer library likeDOMPurify to avoid script injection whenyou use the output of Readability. We would also recommend usingCSP to add furtherdefense-in-depth restrictions to what you allow the resulting content to do. TheFirefox integration of reader mode uses both of these techniques itself.Sanitizing unsafe content out of the input is explicitly not something we aim todo as part of Readability itself - there are other good sanitizer libraries outthere, use them!

Please see ourContributing document.

Copyright (c) 2010 Arc90 IncLicensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at   http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.