Dataset Works

Web Page to Markdown & Clean Text for AI

Turn any web page into clean Markdown or plain text for LLMs, RAG and AI agents. Strips menus, ads and boilerplate. Fast, cheap at $1 per 1,000 pages, respects robots.txt.

Try it on Apify $1.00 per 1,000 pages

Turn any web page into clean Markdown or plain text, ready for an LLM, a RAG pipeline or an AI agent.

Give it URLs. It fetches each page, strips menus, cookie banners, sidebars and footers, and returns the main content as tidy Markdown (headings, lists, links and tables kept) and/or plain text. Fast and lightweight: no browser, results in seconds.

Sample output

url title wordCount markdown (start)
https://en.wikipedia.org/wiki/Web_scraping Web scraping 4083 **Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](...) used for...
https://www.bbc.co.uk/news Home - BBC News 3136 ## Top stories ...

Each result also has status, httpStatus, description, language, truncated, depth, fetchedAt and, if you ask for it, links.

How to use

  1. Paste one or more URLs into URLs.
  2. Choose Markdown, Plain text or both.
  3. Optional: Follow links to also convert linked pages in the same site section (depth 1, 2...), Max pages, and Max characters per page to fit a model's context window.
  4. Run, then export as JSON, CSV or Excel, or read results via the API.

Pages that can't be converted are listed with status: "failed" or "skipped" and a reason, and are not charged.

Pricing

$1 per 1,000 pages converted ($0.001 per page). Failed and skipped pages are free. Platform usage is included.

For AI agents

Call it through the Apify API or the Apify MCP server with {"startUrls": [{"url": "https://example.com/article"}], "maxCharsPerPage": 20000} and read markdown from the result. Every result has the same fields, and errors come back as data rather than crashes, so agents can handle them.

Limits

This tool reads the HTML a site sends. Pages that only show content after JavaScript runs, or that need a login, will return little or no text. Sites whose robots.txt disallows automated access are skipped.

Data and responsibility

The tool fetches only public pages, identifies itself honestly as DatasetWorksBot, follows robots.txt and crawl delays, and never logs in or gets around captchas or blocks. You are responsible for how you use the content you convert, including copyright and any personal data on the pages you choose.

Support

Found a bug or need a feature? Open an issue on this page and we'll reply within one working day.

Built and maintained by Dataset Works.

Open Web Page to Markdown & Clean Text for AI on Apify

Guides