One of the biggest releases this year: a reworked MCP server, RAG-ready Markdown in one line, an improved Cloudflare solver, and browser tabs that stay open for automation 🚀

🚀 New Stuff and quality of life changes

• Browser tabs now stay open and get reused across requests (Check the docs):
  • All browser sessions keep their tabs after a request, and the next request reuses a free tab instead of opening a new one.
  • Every request re-applies its own settings (timeouts, headers, resource blocking) to the tab it gets, so nothing leaks between requests.
  • Tabs that hit an error are closed and replaced, and the new close_pages() method closes every open tab.
  • The page you fetched stays loaded, so a page_setup function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
• Turn any page into clean, LLM-ready Markdown with Response.markdown() (Check the docs):

  from scrapling.fetchers import Fetcher
  
  markdown = Fetcher.get("https://example.com").markdown(main_content_only=True)

  • Scripts, styles, and hidden prompt-injection content are always stripped first, the same as the cleaning the MCP server does.
  • Pass css_selector to convert only the elements you need.
  • Available through the new rag extra (pip install "scrapling[rag]"), which the ai/shell/all extras include too.
• New SiteToMarkdownSpider template to crawl a whole website into a Markdown corpus for RAG pipelines (Check the docs):

  from scrapling.spiders import SiteToMarkdownSpider
  
  class DocsSpider(SiteToMarkdownSpider):
      name = "docs"
      start_urls = ["https://example.com/docs/"]
      allowed_domains = {"example.com"}
      output_dir = "docs_markdown"
  
  result = DocsSpider().start()
  result.items.to_jsonl("docs.jsonl")

  • Yields one item per page with url/title/markdown, and the optional output_dir writes one Markdown file per page.
  • max_pages caps the crawl, and since it builds on CrawlSpider, overriding rules() gives you full control over which links get followed.
• The MCP server is reworked (breaking) (Check the breaking changes and the docs):
  • The 13 tools are now split into two modes: one-shot tools (fetch, bulk_fetch, stealthy_fetch, bulk_stealthy_fetch) that always launch their own browser and show their real defaults, and session tools that work through a session opened once.
  • The new session_fetch tool fetches through a browser session, while open_session now holds the browser-level settings only and returns the session's effective settings for the AI agent.
  • The get tool is renamed to make_request, and it now supports any HTTP method.
  • The new open_request_session and session_make_request tools give the AI persistent HTTP sessions that keep cookies and the browser fingerprint between requests.
  • This also ends fetches resetting the session's settings, first fixed by @Yigtwxx in #418.
• The MCP server's HTTP transport now requires authentication and binds to localhost by default (breaking) by @yamantaka-singh in #414 (Fixes #413, check the docs):
  • Pass --auth-token (or the SCRAPLING_MCP_AUTH_TOKEN environment variable) to require a bearer token, or --no-auth to serve it unauthenticated on purpose.
  • Pass --host 0.0.0.0 to accept connections from the network.

🐛 Bug Fixes

• Cloudflare Turnstile/Interstitial solving now works regardless of the browser locale, and no longer loops forever on interactive challenges in headless mode. Stealth pages also stop crashing mid-solve, with contributions by @subediparas5 in #412. (Fixes #411 and #422)
• Fixed find/find_all with class_ silently missing multi-class elements by @yetval in #410, and blank class_ values and unescaped CSS string values by @yamantaka-singh in #417.
• Fixed cached responses in development_mode losing the request meta on replay by @Yigtwxx in #419.
• Fixed HTTP requests with retries below 1 failing without sending the request by @Yigtwxx in #420.

Docs

• The website sections are restructured: a new "Using with AI" section holds the MCP server, the new Agent skill page, and the Building RAG systems guide, and the BeautifulSoup migration guide moved next to the Scrapy integration under "Integrations and migrations".
• Added a CHANGELOG.md to the repository with the notes of every release so far.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A quick maintenance release to fix installation with uv 🔧

🐛 Bug Fixes

• Fixed uv refusing to install v0.4.13 by default and silently falling back to an older version. The previous release required a prerelease version of curl_cffi, which uv doesn't allow unless explicitly enabled.
  All dependencies now resolve to stable releases. (Fixes #407)

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A new update bringing feed spiders and a smarter MCP server 🎉

🚀 New Stuff and quality of life changes

• New feed spider templates. XMLFeedSpider iterates over the nodes of any XML feed (RSS, Atom, product feeds, etc.), and CSVFeedSpider iterates over CSV rows as dictionaries. Both decompress gzipped feeds automatically. (Check the docs)

  from scrapling.spiders import XMLFeedSpider
  
  class RSSSpider(XMLFeedSpider):
      name = "rss"
      start_urls = ["https://example.com/feed.xml"]
  
      async def parse_node(self, response, node):
          yield {"title": node.findtext("title"), "link": node.findtext("link")}
  
  result = RSSSpider().start()

• Upgraded the MCP server to MCP SDK v2 and made it smarter. The server now ships instructions that teach your AI agent how to use the tools efficiently; every tool declares annotations so clients like Claude Code can auto-approve the read-only ones; tool descriptions are leaner to save tokens; and the server advertises its version and logo to MCP clients. (Check the docs)
• Added a scrapling-mcp command that maps directly to scrapling mcp, so registering Scrapling with MCP clients and registries that expect a single command is now a one-liner.
• Unpinned Playwright/Patchright and browser versions. The generated browser User-Agent now always matches the exact Chromium version your installed Playwright/Patchright drives, so Scrapling no longer pins their versions and you can upgrade them freely. Run scrapling install --force after updating to refresh the browsers.

🐛 Bug Fixes

• Fixed importing Scrapling crashing with a browserforge ValueError when the fingerprints data package lags behind the browser versions. (Fixes #394, #396, and #400)
• Fixed the MCP bulk browser tools mis-sizing their page pools, which made bulk_fetch fail on batches of more than 50 URLs and bulk_stealthy_fetch fetch all URLs through a single tab, by @Yigtwxx in #393.

Project

• New AI Contribution Policy: AI-assisted contributions are welcome but must be disclosed in the PR or issue; submissions that look like undisclosed AI output get labeled and closed.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A release focused on making your spiders smarter about the websites they crawl

🚀 New Stuff and quality of life changes

• Spiders can now tune their own speed with AutoThrottle. Instead of guessing a download_delay that's either too slow or gets you banned, the spider measures how fast each website answers and adjusts the delay of every domain on its own. When a website starts blocking or rate-limiting you, it doubles the delay (or waits exactly what the Retry-After header asks for) until that stops, then speeds back up. Your download_delay and any robots.txt Crawl-delay are still respected as the minimum. (Check the docs)

  class MySpider(Spider):
      name = "adaptive"
      start_urls = ["https://example.com"]
      autothrottle_enabled = True
      autothrottle_start_delay = 2.0
      autothrottle_max_delay = 30.0
      autothrottle_block_backoff = True

• Export your results to CSV and XML, next to the JSON/JSONL exporters you already had. Items that don't all share the same keys are still exported without losing anything, and nested values are written as JSON. (Check the docs)

  result = MySpider().start()
  result.items.to_csv("products.csv")
  result.items.to_xml("products.xml")

• The MCP server can now require authentication, so you can safely expose it instead of keeping it on your own machine. Any request without the token is rejected, and you can also restrict which hostnames the server answers to. (Check the docs)

  scrapling mcp --http --auth-token "$(openssl rand -hex 32)"

• Browsers now accept CDP URLs over HTTP, not just WebSocket ones. So next to the wss:// endpoints managed browser providers hand out, you can now point any browser fetcher or MCP session at a Chrome you started yourself with --remote-debugging-port=9222.

• Published Docker images are now tagged with their release version instead of only latest, so you can pin the exact version you want, by @JanRK in #384.

🐛 Bug Fixes

• Fixed cached responses losing all their cookies when the response came from a browser engine, which silently broke any session or auth logic relying on them while using the spiders' development mode, by @amitvijapur in #379. (Fixes #376)

• Fixed StealthyFetcher forcing the en-US locale on every browser instead of following your system's, which made websites see a mismatch between your locale and your IP address and treat you as suspicious, like Google answering with 429s. (Fixes #381)

• Fixed a misleading error message in the storage system and removed a dead call left after inserts, by @fix2015 in #377.

Performance

• get_all_text() is now O(nodes) instead of walking up the ancestors of every single text node, which makes it around 5-6x faster on deeply nested pages, by @yetval in #378.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A solid update bringing the first platform spider template, a faster parser, and important fixes 🎉

🚀 New Stuff and quality of life changes

• Added ShopifySpider, the first platform spider template! Extract every product from any Shopify-powered store through its JSON API without touching the website's HTML. Subclass it, set the store's domain, and you are done (Check the docs)

  from scrapling.spiders import ShopifySpider
  
  class MyStore(ShopifySpider):
      target_website = "example.com"
  
  result = MyStore().start()

• Added --executable-path to the CLI browser commands. Both scrapling extract fetch and scrapling extract stealthy-fetch now accept a custom Chromium-compatible browser executable, and fall back to the SCRAPLING_EXECUTABLE_PATH environment variable when the option isn't passed, bringing full parity with the MCP server (Solves #371)

  scrapling extract fetch "https://example.com" page.html --executable-path "/path/to/chromium"

🤖 Quality of life changes

• Made find_by_text and find_by_regex up to ~2x faster when first_match is enabled (the default) by wrapping elements lazily so the search stops at the first match, by @yetval in #370
• Updated the benchmarks with the new numbers against the latest versions of all libraries.
• Updated contribution rules

🐛 Bug Fixes

• Fixed the MCP server's fetch tools crashing on pages containing control characters with the error All strings must be XML compatible, by @yetval in #368 (Fixes #366)

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors:

A new update with a brand-new Scrapy integration and a batch of community fixes 🎉

🚀 New Stuff and quality of life changes

• Added a Scrapy integration so you can use Scrapling's parsing API inside your existing Scrapy projects without rewriting them. Put the scrapling_response decorator on any spider callback, and the response it receives becomes a Scrapling Response while Scrapy keeps handling the crawling (Check the docs):

  import scrapy
  from scrapling.integrations.scrapy import scrapling_response
  
  
  class QuotesSpider(scrapy.Spider):
      name = "quotes"
      start_urls = ["https://quotes.toscrape.com"]
  
      @scrapling_response
      def parse(self, response):  # `response` is now a Scrapling Response
          first_quote = response.find_by_text("The world as we have created it", partial=True)
          for quote in [first_quote, *first_quote.find_similar()]:
              yield {"text": quote.get_all_text(strip=True)}

• The MCP server can now use a custom Chromium-compatible browser for all browser-based tools. Set it once with scrapling mcp --executable-path "/path/to/chromium" or the SCRAPLING_EXECUTABLE_PATH environment variable, or per request with the executable_path argument, by @samrusani in #360 (Solves #347)

• Updated all browsers and fingerprints. Run scrapling install --force after updating to refresh them.

🐛 Bug Fixes

• Fixed garbled text (mojibake) from browser fetchers on non-UTF-8 websites by @yehudalevy-collab in #365 (Fixes #364).
• Fixed LinkExtractor not filtering compound file extensions like .tar.gz by @renbkna in #359 (Fixes #349).
• Fixed paused crawls losing their in-flight requests from checkpoints, so resuming no longer skips them by @yetval in #358.
• Fixed spiders calculating wrong crawl delays from robots.txt Request-rate directives through the Protego upgrade, with tests aligned by @Disaster-Terminator in #355.

Docs

• Clarified how init_script interacts with Patchright's isolated execution context in stealth mode by @mturac in #353 (Solves #350).
• Added the skills.sh install method for the agent skill by @ob-aion in #363.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A maintenance update packed with community-reported fixes 🛠️

🚀 New Stuff and quality of life changes

• Updated all browsers and fingerprints. Run scrapling install --force after updating to refresh them.
• Added a --version flag to the CLI by @ETM-Code in #303 (Solves #299)

🐛 Bug Fixes

• Fixed the session-level proxy argument being silently ignored in HTTP sessions, which could leak your real IP (Solves #295). Note that mixing a session-level proxy with a per-request proxies argument (or vice versa) now raises an error instead of one being silently dropped.
• Fixed browser navigations failing when combining init_script with user_data_dir (Solves #294).
• Fixed encoding detection when websites quote the charset value in the Content-Type header by @Bortlesboat in #323.
• Fixed an IndexError in adaptive element relocation when auto_save is enabled by @Mubashirrrr in #340.
• Fixed spiders' checkpoint and cache saving crashing on Windows by @MrStarkEG in #344.
• Fixed incorrect similarity scoring in find_similar for elements with mismatched attribute counts (Solves #322).

Docs

• Clarified that the default installation includes the parser engine only, and the fetchers/spiders need the extras (Solves #343).
• Fixed the Docker image name in the remaining examples by @evanclan in #315.
• Fixed a broken link in the contribution guide by @Bortlesboat in #320.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A big spider update that takes the crawling framework to the next level 🕷️

🚀 New Stuff and quality of life changes

• Added a LinkExtractor primitive in scrapling.spiders.LinkExtractor to pull URLs out of a Response. There are a lot of controls (Check the docs)

  from scrapling.spiders import LinkExtractor
  
  extractor = LinkExtractor(allow=r"/posts/", deny_domains=["ads.example.com"])

• Added CrawlSpider and CrawlRule generic spider templates so you no longer have to hand-write the same "follow links matching this pattern" boilerplate. Override rules() to return a list of CrawlRule objects, each pairing a LinkExtractor. (Check the docs)

  from scrapling.spiders import CrawlSpider, CrawlRule, LinkExtractor
  
  class QuotesSpider(CrawlSpider):
      name = "blog"
      start_urls = ["https://quotes.toscrape.com/"]
  
      def rules(self):
          return [
              CrawlRule(LinkExtractor(allow=r"/author/"), callback=self.parse_author),
              CrawlRule(LinkExtractor(allow=r"/page/\d+/")),  # pagination, no callback
          ]
  
      async def parse_author(self, response):
          yield {
              "name": response.css(".author-title::text").get(),
              "birthday": response.css(".author-born-date::text").get(),
              "url": response.url,
          }

• Added a SitemapSpider template that seeds a crawl directly from a sitemap, or robots.txt URLs. Handles gzip-compressed sitemaps, and a lot of controls and options. URLs are dispatched via the crawl rules as shown above for CrawlSpider. (Check the docs)

  from scrapling.spiders import SitemapSpider, CrawlRule, LinkExtractor
  
  class NewsSitemap(SitemapSpider):
      name = "news"
      sitemap_urls = ["https://example.com/robots.txt"]
  
      def rules(self):
          return [
              CrawlRule(LinkExtractor(allow=r"/articles/"), callback=self.parse_article),
          ]
  
      async def parse_article(self, response):
          yield {"url": response.url, "title": response.css("h1::text").get()}

• Adaptive relocation now defaults to a 40% similarity threshold instead of 0 across all methods. This will make the adaptive feature work better. When nothing crosses the threshold, a warning now tells you the top score it did see, so you can lower percentage deliberately if needed.

• Updated all browsers and fingerprints. Run a new scrapling install --force after updating to refresh the browsers and fingerprints.

🐛 Bug Fixes

• Fixed Fetcher.configure(...) not applying to per-request calls. Same fix applied to AsyncFetcher.
• Fixed incorrect request fingerprinting that caused duplicate requests in spiders by @yetval in #255.
• Fixed the Adaptive scraping engine staying silent on weak matches. Combined with the threshold change above, you now get a warning instead of a misleading "best guess" element when relocation fails.

Docs

• Refreshed older code examples across the documentation to match the current version.
• Improved the code copy-paste experience on the docs site and trimmed the agent skill so it uses fewer tokens per invocation.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A focused update bringing eyes to your AI agents 📸

🚀 New Stuff and quality of life changes

• Added a screenshot MCP tool that captures a page and returns it as a real MCP ImageContent block so the model can actually see it. The tool requires an open browser session, so you call open_session first (either dynamic or stealthy) and pass the session_id here. Supports PNG and JPEG, full-page captures, JPEG quality, and the usual readiness controls (wait, wait_selector, network_idle, timeout). (implements #244)
• Added a custom session_id parameter to open_session so you can name sessions meaningfully ("search", "checkout") instead of the random 12-character hex default. By @hauntedhost in #243

🐛 Bug Fixes

• Fixed FetcherSession state corruption and a lazy session close crash. By @yetval in #245
• Fixed TypeError: Session.request() got an unexpected keyword argument 'block_ads' when using the CLI's --ai-targeted flag with HTTP commands. By @voidborne-d in #249 (Fixes #247)

Translations

• Added a Brazilian Portuguese README translation By @rgomids in #250

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors

A focused update on browser stealth, privacy, and developer experience 🔒

🚀 New Stuff and quality of life changes

• Added built-in ad blocking for browser fetchers. Pass block_ads=True to block requests to ~3,500 known ad and tracker domains at the route interception level -- no DNS, no TCP, instant abort. Can be combined with blocked_domains for custom lists. The MCP server and CLI --ai-targeted mode enable this automatically to save tokens and speed up page loads.

  page = StealthyFetcher.fetch('https://example.com', block_ads=True)

• Added DNS-over-HTTPS support to prevent DNS leaks when using proxies. Pass dns_over_https=True to route DNS queries through Cloudflare's DoH, so your real location isn't exposed through DNS resolution even when your HTTP traffic goes through a proxy.

  page = StealthyFetcher.fetch('https://example.com', proxy='http://proxy:8080', dns_over_https=True)

• Added page_setup callback for browser fetchers. A function that runs before page.goto(), letting you register event listeners, routes, or scripts that must be set up before the page navigates. Pairs with page_action (which runs after navigation). (Solves #237)

  def capture_websockets(page):
      page.on("websocket", lambda ws: print(f"WS: {ws.url}"))
  
  page = DynamicFetcher.fetch('https://example.com', page_setup=capture_websockets)

• Added --block-ads and --dns-over-https CLI options to both fetch and stealthy-fetch commands.

🐛 Bug Fixes

• Fixed Seconds type alias rejecting float values. Passing wait=1.5 or timeout=500.0 to browser fetchers would fail with a type error because the type alias incorrectly treated float as metadata instead of a type. by @kuishou68 in #240
• Fixed duplicate ID segments in full-path selector generation. Elements with id attributes had their selector appended twice when generating full CSS/XPath paths, producing selectors like body > #main > #main > #target > #target. Also fixed full-path XPath emitting bare [@id='x'] predicates (invalid XPath) instead of *[@id='x']. by @sjhddh in #241
• Fixed missing shell signature parameters. The interactive shell was missing blocked_domains, block_ads, retries, retry_delay, capture_xhr, executable_path, and dns_over_https from its function signatures.

🙏 Special thanks to the community for all the continuous testing and feedback

Big shoutout to our Platinum Sponsors