Content extraction

This article is a stub. You can help the IndieWeb wiki by expanding it with relevant information.

Content extraction are techniques and tools to get the main/structured content from web pages.

Tools

if you have experience with any of these tools, please add your experience

• https://github.com/n1k0/readable-proxy – based on readability.js, the basis of

oh, content extraction service

• readability-lxml
  • limited
• breadability
  • limited
• newspaper3k
  • memory issues?
• https://www.file2markdown.ai/tools/website-to-markdown
  • hosted URL → clean Markdown extraction, free tier, no JavaScript rendering (disclosure: added by its maintainer)

Past

• Mercury Web Parser by Postlight (archived) shut down in 2019; the parser continues as open source at https://github.com/postlight/parser
  • Note: the postlight(.)com domain is now a spam zombie site

indieweb/silo specific tools

• XRay
• granary