Repository files navigation

A fast tool to convert any website into LLM-ready markdown data.

I'm building an AI app called Supermemory -https://git.new/memory. Where users can store website content in the app and then query it using AI. One thing I noticed was - when data is structured and predictable (in markdown format), the LLM responses aremuch better.

There are other solutions available for this -https://r.jina.ai,https://firecrawl.dev, etc. But they are either:

• too expensive / proprietary
• or too limited.
• very difficult to deploy

Here's a quote from my friend@nexxeln

So naturally, we fix it ourselves ⚡

• Convert any website into markdown
• LLM Filtering
• Detailed markdown mode
• Auto Crawler (without sitemap!)
• Text and JSON responses
• Easy to self-host
• ... All that and more, for FREE!

To use the API, just make GET a request tohttps://md.dhr.wtf

Usage example:

$ curl 'https://md.dhr.wtf/?url=https://example.com'

url (string) -> The website URL to convert into markdown.

enableDetailedResponse (boolean: false) -> Toggle for detailed response with full HTML content.crawlSubpages (boolean: false) -> Crawl and return markdown for up to 10 subpages.llmFilter (boolean: false) -> Filter out unnecessary information using LLM.

AddContent-Type: text/plain in headers for plain text response.AddContent-Type: application/json in headers for JSON response.

Under the hood, Markdowner utilises Cloudflare'sBrowser rendering andDurable objects to spin up browser instances and then convert it to markdown using Turndown.

You can easily self host this project. To use the browser rendering and Durable Objects, you need theWorkers paid plan

1. Clone the repo and download dependencies

git clone https://github.com/dhravya/markdownernpm i

2. Run this command:

   npx wrangler kv:namespace create md_cache

3. Open Wrangler.toml and change the IDs accordingly
4. Runnpm run deploy
5. That's it 👍

Support me by simply starring this repository! ⭐