Extraction reliability release: six fixes for silent content loss or silent wrong answers, plus the first changelog generated from commit messages.

Highlights

• Scanned PDFs fail fast. An image-only PDF used to run the whole extraction chain — Docling across every page in technical mode — before failing with a message about Unicode sanitization. It now stops on the first pages in ~0.01s and tells you to run ocrmypdf (#130), and the docs finally state the limit (#132).
• Real words stopped being deleted. Page-edge cleanup treated any short word built from ivxlcdm as a page number, so "MIX", "CIVIL" and "VIVID" vanished when they were alone on a page's first or last line (#119).
• Table-of-contents detection no longer depends on input order. With multiple sources, has_toc was re-derived over the joined corpus and its 30k-char window only ever covered the first book — the same two books in the opposite order gave opposite answers (#114).
• A single unreadable file no longer aborts the batch (#120), and ~ in a quoted glob is expanded, so the README's own example works (#118).
• CJK tables of contents with inserted whitespace (目 录, including the ideographic space) are detected (#112).
• The generated-skill scanner reads nested chapters and reports Markdown it left out of scope instead of printing an unqualified pass (#115).

Thanks to @Zhengzhongjie, @Hotragn, @aryansk and @dex0shubham for these fixes.

Added

• Warn on unknown CLI flags instead of silently ignoring (#89)
• Make the site indexable and shareable
• Add Booklin banner and hand-made share card
• Apply pdftotext cleanup to pypdf and pdfminer paths too (#101)

Documentation

• Sync architecture, skill spec, and README with v1.3.0
• Reserve README project listings for sponsors
• Add Booklin mascot to the README (#106)
• Split README into pages and add Booklin mascot set (#110)
• State that scanned PDFs need OCR first (#132)

Fixed

• Gate lowercase Roman behind heading context, add FP coverage (#88)
• CJK-aware token estimate (rescued from #70) (#103)
• Support documented help flags (#97)
• Emit block boundaries at end tags in the stdlib parser (#107)
• Skip control destination groups in the dependency-free fallback (#109)
• Scrub bidirectional controls and remaining invisible code points (#108)
• Write metadata.json as UTF-8 (#105)
• Detect Markdown-prefixed chapter headings (#91) (#92)
• Reserve the banner's space on the site build
• Abort early on scanned PDFs with an OCR hint (#130)
• Stop edge-of-page cleanup deleting real one-word lines (#119)
• Expand ~ in input paths (#118)
• Skip an unreadable source instead of aborting the batch (#120)
• Detect a table of contents in any source, not just the first (#114)
• Scan nested chapters and report markdown left out of scope (#115)
• Detect whitespace-separated CJK ToC headers (#112)
• Stop duplicating the PR number in generated entries (#131)

Miscellaneous

• Untrack stale compiled bytecode (#94)
• Generate CHANGELOG from commits with git-cliff (#104)

Notes on selected changes

Written by hand before this file became generated; kept for the detail the
one-line entries above do not carry.

CJK-aware token estimate — estimate_tokens now counts CJK codepoints
directly (against CJK_CHARS_PER_TOKEN) instead of whitespace-delimited words,
fixing a ~1000× undercount for space-less Chinese/Japanese books in the cost
pre-flight. Latin behavior is unchanged and the estimate stays deterministic
and dependency-free (rescued from #70).

Stale tracked bytecode — scripts/__pycache__/extract.cpython-313.pyc was committed
alongside the EPUB-support change and has been tracked ever since, despite .gitignore
listing both *.pyc and __pycache__/ (ignore rules do not untrack already-committed
files). The file was inert — its timestamp-based header records a 11,321-byte source while
scripts/extract.py is now 871 bytes, so Python always invalidated it and recompiled — but
committed bytecode is not reviewable in a diff. A regression test now fails if any .pyc
or __pycache__/ path is tracked.

book-to-skill v1.3.0

36 commits since v1.2.0. This release is security-heavy and adds multilingual chapter detection.

🔒 Security

• Generated-skill prompt-injection scan — a dependency-free advisory scanner flags instruction-override phrases, model-control tags, invisible Unicode, authority-widening frontmatter, and exfiltration-shaped content before a generated skill is accepted or published. Findings never echo attacker-controlled text (#73, #74).
• Invisible-Unicode extraction hardening — every parser result now strips zero-width U+200B/200C/200D/2060/FEFF and the Unicode tag block U+E0000–E007F, reports the removal count, and rejects sources with no visible content after the scrub (#73, #75, #85).
• DOCX XXE / Billion Laughs and subprocess argument-injection hardening, plus dependency CVE review on PRs (#53, #54).

🌏 Multilingual chapter detection

• Korean 제N장 / 제N절 / 제N편 and the statutory 의N form — 제 prefix required so the everyday counter 장 never false-matches. Corpus-validated (precision 0.999 / recall 1.000) (#82).
• Thai บทที่ N / ตอนที่ N / ภาคที่ N with Thai or Arabic numerals.
• Roman Chapter I. (chapter word + Roman numeral) — Project Gutenberg #132 goes from 2 phantom "chapters" to the 13 real headings.

🧹 Extraction quality & fixes

• pdftotext cleanup — dehyphenate line-wraps, strip repeated headers/footers and edge page numbers (conservative: edges only, ≥3 pages) (#77).
• SOURCE-banner fix — chapter detection now runs on source text before the generated SOURCE: banners, so they no longer become phantom setext headings (#81, #83).
• Encoding fixes — pdftotext output decoded as UTF-8; UTF-16/UTF-32 text files decoded by BOM; RTF \uN escapes; HTML double-entity fix; DOCX table reconstruction; EPUB spine-order reading.

📦 Packaging

• The pdf extra now installs pypdf instead of the end-of-life PyPDF2.

Full changelog: https://github.com/virgiliojr94/book-to-skill/blob/master/CHANGELOG.md

book-to-skill v1.2.0 turns the project into an installable Python package and makes chapter detection genuinely multilingual.

Highlights

📦 Installable package + CLI

book-to-skill is now a real book_to_skill package: pip install it, run the book-to-skill console script or python -m book_to_skill, and pull only the extractors you need via extras (epub, pdf, docx, rtf, technical, all). The base install stays dependency-free with stdlib fallbacks, and python3 scripts/extract.py still works unchanged, so existing skill flows keep running.

🌍 Multilingual chapter detection

• Markdown / AsciiDoc ATX headings (#, ==) detected when no numeric "Chapter N" is present.
• setext / reStructuredText underline headings (Title over === / ---), guarded against thematic breaks, table borders, and YAML front matter.
• French, German, Italian, Dutch chapter words (Chapitre, Kapitel, Capitolo, Hoofdstuk) and umlaut titles (Überblick).
• Full-width Arabic digits in CJK headings (第１章), common in Japanese typesetting.
• Multilingual table-of-contents detection (CN, JP, FR, DE, IT, NL).

🔎 Diagnosable extraction

Unexpected parser errors are now logged to stderr (extractor name + exception type) instead of vanishing, while the fallback chain still continues. Corrupt files and encoding issues are finally visible.

🔒 Security & CI

CodeQL, Bandit (HIGH gate), Zizmor workflow audit, and grouped Dependabot. Test matrix now spans Python 3.9–3.13.

Thanks

Community contributions from @Marcelluxx, @dex0shubham, @RandMelville, @addy790, @yukaina, @wuji-labs, and everyone filing multilingual edge cases. 💖 Sponsor the project

Full changelog: https://github.com/virgiliojr94/book-to-skill/blob/master/CHANGELOG.md

[1.1.0] — 2026-06-12

Added

• GitHub Copilot CLI as a first-class target — the same SKILL.md now
  discovers, installs, and runs across GitHub Copilot CLI, Amp, and Claude Code
  via the open Agent Skills standard. Skill Locations cover 8 discovery paths and
  the script probe walks all of them (#30).
• validate_skill.py --lens claude|copilot|amp — audits a generated SKILL.md
  against each host's rules; claude stays the default for CI back-compat (#30).
• Attribution banner — scripts/banner.txt is printed at the start of each
  run (best-effort, never fails the run).

Changed

• SKILL.md frontmatter trimmed toward the open-standard minimum and the
  description now names all three hosts so each agent's auto-loader picks it up (#30).
• README headline + "Agent Skills" badge; install/usage sections cover all three
  hosts. docs/ARCHITECTURE.md shows per-host destination paths (#30).

Notes

• allowed-tools was dropped from the frontmatter for host-neutrality; the skill
  is conformant on all three hosts (validated with all three lenses). If Claude
  users hit permission-prompt friction, the Bash grant from #18 will be restored
  with Claude-native tokens (Copilot ignores the key either way).

[1.0.0] — 2026-06-08

First formally tagged release. The converter is stable, multi-format, and
validated on real books.

Added

• Multi-format extraction — PDF, EPUB, DOCX, HTML, Markdown, reStructuredText,
  AsciiDoc, RTF, and MOBI/AZW/AZW3 (via Calibre), through a modular extractor
  package with per-format parsers and graceful stdlib fallbacks.
• extract.py --check — preflight that reports which extractors are installed
  for every format and the exact command to install whatever is missing (#21).
• Adaptive per-chapter depth — token budget scales with BOOK_TYPE × DEPTH;
  study-depth chapters require a worked example, and the cheatsheet is generated as
  a decision/reasoning layer (decision rules, trees, trade-offs, thresholds, tells)
  rather than a keyword list (#20).
• tools/discovery_tax.py — measures the "Discovery Loop Tax": tokens a
  context-dump vs a discovery loop vs book-to-skill put into context to answer one
  question, on a real book (#23).
• Update / fold-in workflow — merge new sources into an existing skill, keeping
  chapter index, topic index, glossary, patterns, and cheatsheet in sync.
• GitHub Actions CI — lint (ruff), test matrix (py3.10–3.13), dependency-free
  smoke test, and SKILL.md Claude-conformance validation (#15, #18).

Changed

• README positioning — copyright & fair-use section, "Beyond books" use cases,
  context-dump / RAG / 1M-window FAQ, and a measured Discovery Loop Tax + real
  per-conversion cost table across four books (#19, #27).
• Default output target is ~/.claude/skills/ for Claude Code, with Amp skill
  directories also supported (#13, #14).

Fixed

• Chapter detection — scans the full text (was capped at 50k chars) and counts
  distinct explicit Chapter N / Capítulo N headings, rejecting numbered list
  items, inline cross-references, and years; adds Portuguese support (#26).
• Roman-numeral headings — I: Loomings, II. The Carpet-Bag are now detected
  with canonical-numeral validation (#28).
• EPUB extraction — resolve OPF-relative hrefs in the stdlib zipfile fallback (#11, #12).
• Batch resilience — one bad source is skipped with a warning instead of aborting
  the whole run; explicit input order is preserved (#7).

Known limitations

• Chapter auto-detection needs explicit Chapter N / Capítulo N or Roman-numeral
  headings. Books that head chapter bodies with bare titles (e.g. Moby-Dick, where
  numerals appear only in the table of contents) or use section titles (e.g. Pro Git)
  do not auto-segment.
• Technical PDFs extracted in text mode may lose heading structure; use technical
  mode (Docling) to preserve tables, code, and headings.