| 1 | # Conversion Tools |
| 2 | |
| 3 | > **Design boundary**: use native-Python converters for supported formats, |
| 4 | > invoke Pandoc only for explicit fallback formats, and let web conversion use |
| 5 | > `curl_cffi` when available for sites that reject Python's default TLS |
| 6 | > fingerprint. |
| 7 | |
| 8 | Source conversion tools turn PDFs, documents, slide decks, and web pages into Markdown before project creation. |
| 9 | |
| 10 | Default workflow entry: use `source_to_md.py` unless a backend-specific |
| 11 | diagnostic or forced route is needed. |
| 12 | |
| 13 | ## Shared Output Contract |
| 14 | |
| 15 | All `source_to_md` backends preserve their Markdown output and attempt a |
| 16 | sidecar profile after conversion. Direct calls treat the sidecar as |
| 17 | best-effort: an I/O failure warns without changing the Markdown result. The |
| 18 | unified `source_to_md.py` dispatcher writes a missing profile before success. |
| 19 | |
| 20 | | Output | Convention | |
| 21 | |---|---| |
| 22 | | Markdown | `<stem>.md` beside the local source unless `-o` selects another path | |
| 23 | | Asset directory | `<stem>_files/` when the backend extracts images or media | |
| 24 | | Image manifest | `<stem>_files/image_manifest.json` when image metadata is available | |
| 25 | | Conversion profile | `<stem>.conversion_profile.json` beside the Markdown output when written | |
| 26 | |
| 27 | When present, the conversion profile is metadata only: converter, source path, |
| 28 | Markdown structure counts, asset directory, image manifest path, and image |
| 29 | count. Downstream PPT workflows still use Markdown and the image manifest as |
| 30 | the content/asset contract; the profile is for inspection and debugging. |
| 31 | |
| 32 | ## `source_to_md.py` |
| 33 | |
| 34 | Unified dispatcher for ad hoc explicit-source conversion. It auto-detects each |
| 35 | listed input file or URL and calls the existing backend converter, so backend |
| 36 | behavior remains the source of truth. |
| 37 | |
| 38 | Routing is centralized in `source_to_md/_dispatcher.py` and reused by |
| 39 | `project_manager.py import-sources`; do not add a second type-to-backend table. |
| 40 | |
| 41 | ```bash |
| 42 | python3 scripts/source_to_md.py paper.pdf |
| 43 | python3 scripts/source_to_md.py paper.pdf report.docx deck.pptx |
| 44 | python3 scripts/source_to_md.py ./sources |
| 45 | python3 scripts/source_to_md.py ./pdfs/*.pdf |
| 46 | python3 scripts/source_to_md.py ./decks/*.pptx |
| 47 | python3 scripts/source_to_md.py report.docx -o report.md |
| 48 | python3 scripts/source_to_md.py ./sources -o ./markdown # explicit separate output directory |
| 49 | python3 scripts/source_to_md.py workbook.xlsx --json |
| 50 | python3 scripts/source_to_md.py deck.pptx |
| 51 | python3 scripts/source_to_md.py https://example.com/article -o article.md |
| 52 | ``` |
| 53 | |
| 54 | Useful options: |
| 55 | - `-t pdf|doc|excel|pptx|web|markdown|text` forces a route when extension |
| 56 | detection is not enough. |
| 57 | - `--json` prints a compact machine-readable result after success when the |
| 58 | output path is known. With multiple inputs, each successful conversion prints |
| 59 | its own JSON line after that source finishes. |
| 60 | - At the unified `source_to_md.py` entry, `--images all|filtered|none`, |
| 61 | `--no-images`, and `--filter-images` map to the PDF image mode. The web |
| 62 | backend exposes its own direct `--no-images` option described below. |
| 63 | - Unknown backend-specific flags are passed through to each selected converter. |
| 64 | - `-o/--output` selects one Markdown file for one input, or an output directory |
| 65 | for multiple inputs / directory inputs. |
| 66 | |
| 67 | For multi-source project intake, use `project_manager.py import-sources` with |
| 68 | all source paths / URLs. For local files, the default is to keep generated |
| 69 | Markdown and profile outputs beside the original source. |
| 70 | `source_to_md.py` and the backend converters support single files, explicit |
| 71 | multi-file inputs, and non-recursive directory inputs. |
| 72 | |
| 73 | ## `source_to_md/pdf_to_md.py` |
| 74 | |
| 75 | Recommended first choice for native PDFs. |
| 76 | |
| 77 | ```bash |
| 78 | python3 scripts/source_to_md/pdf_to_md.py book.pdf |
| 79 | python3 scripts/source_to_md/pdf_to_md.py book.pdf -o output.md |
| 80 | python3 scripts/source_to_md/pdf_to_md.py book.pdf appendix.pdf |
| 81 | python3 scripts/source_to_md/pdf_to_md.py ./pdfs |
| 82 | python3 scripts/source_to_md/pdf_to_md.py ./pdfs -o ./markdown # explicit separate output directory |
| 83 | |
| 84 | # Image extraction control (default: filtered) |
| 85 | python3 scripts/source_to_md/pdf_to_md.py book.pdf --images filtered # size/quality filters applied |
| 86 | python3 scripts/source_to_md/pdf_to_md.py book.pdf --images all # extract all images, no filtering |
| 87 | python3 scripts/source_to_md/pdf_to_md.py book.pdf --images none # skip all images (text only) |
| 88 | ``` |
| 89 | |
| 90 | Use cases: |
| 91 | - Native PDFs exported from Word, PowerPoint, LaTeX, or similar tools |
| 92 | - Privacy-sensitive documents that should stay local |
| 93 | - Fast first-pass extraction before falling back to OCR-heavy tools |
| 94 | |
| 95 | Prefer MinerU or another OCR/layout tool when: |
| 96 | - The PDF is scanned or image-based |
| 97 | - Multi-column layout parsing is poor |
| 98 | - Encoding is garbled |
| 99 | |
| 100 | Dependency: |
| 101 | |
| 102 | ```bash |
| 103 | pip install PyMuPDF |
| 104 | ``` |
| 105 | |
| 106 | ## `source_to_md/doc_to_md.py` |
| 107 | |
| 108 | Hybrid converter: pure-Python for the common formats, pandoc fallback for the rest. |
| 109 | |
| 110 | Native path (no external binary required): |
| 111 | - `.docx` — via `mammoth`; text-only tables are preserved as pipe Markdown, and OMML / Office Math equations (Word-native or MathType "Convert to Office Math") are rewritten to inline LaTeX. Classic MathType OLE objects carry no OMML and are kept only as their preview image. |
| 112 | - `.html` / `.htm` — via `markdownify` + `beautifulsoup4` |
| 113 | - `.epub` — via `ebooklib` + `markdownify` |
| 114 | - `.ipynb` — via `nbconvert` |
| 115 | |
| 116 | Pandoc fallback (only if you need these): |
| 117 | - `.doc`, `.odt`, `.rtf`, `.tex`/`.latex`, `.rst`, `.org`, `.typ` |
| 118 | |
| 119 | ```bash |
| 120 | python3 scripts/source_to_md/doc_to_md.py lecture.docx |
| 121 | python3 scripts/source_to_md/doc_to_md.py lecture.docx -o output.md |
| 122 | python3 scripts/source_to_md/doc_to_md.py lecture.docx notes.html |
| 123 | python3 scripts/source_to_md/doc_to_md.py ./docs |
| 124 | python3 scripts/source_to_md/doc_to_md.py ./docs -o ./markdown # explicit separate output directory |
| 125 | python3 scripts/source_to_md/doc_to_md.py notes.epub |
| 126 | python3 scripts/source_to_md/doc_to_md.py paper.tex -o paper.md # uses pandoc |
| 127 | ``` |
| 128 | |
| 129 | Dependencies: |
| 130 | |
| 131 | ```bash |
| 132 | # Native path — always required |
| 133 | pip install mammoth markdownify ebooklib nbconvert beautifulsoup4 |
| 134 | |
| 135 | # Fallback path — only for .doc/.odt/.rtf/.tex/.rst/.org/.typ |
| 136 | # macOS: brew install pandoc |
| 137 | # Ubuntu: sudo apt install pandoc |
| 138 | # Windows: https://pandoc.org/installing.html |
| 139 | ``` |
| 140 | |
| 141 | All paths produce `<input>.md`. Extracted assets use a sibling `<input>_files/` |
| 142 | directory with relative references. Without assets, that directory need not remain. |
| 143 | The direct backend then attempts `<input>.conversion_profile.json` under the |
| 144 | shared best-effort sidecar contract. |
| 145 | |
| 146 | ## `source_to_md/excel_to_md.py` |
| 147 | |
| 148 | Excel workbook converter for presentation source intake. |
| 149 | |
| 150 | Supported formats: |
| 151 | - `.xlsx` |
| 152 | - `.xlsm` |
| 153 | |
| 154 | Unsupported by default: |
| 155 | - `.xls` — resave as `.xlsx` first |
| 156 | |
| 157 | ```bash |
| 158 | python3 scripts/source_to_md/excel_to_md.py report.xlsx |
| 159 | python3 scripts/source_to_md/excel_to_md.py report.xlsx -o output.md |
| 160 | python3 scripts/source_to_md/excel_to_md.py report.xlsx budget.xlsm |
| 161 | python3 scripts/source_to_md/excel_to_md.py ./workbooks |
| 162 | python3 scripts/source_to_md/excel_to_md.py ./workbooks -o ./markdown # explicit separate output directory |
| 163 | python3 scripts/source_to_md/excel_to_md.py report.xlsm --max-rows 200 --max-cols 40 |
| 164 | ``` |
| 165 | |
| 166 | Behavior: |
| 167 | - preserves workbook and sheet structure in Markdown |
| 168 | - exports visible sheets only |
| 169 | - trims empty outer rows and columns |
| 170 | - propagates merged-cell labels for readable Markdown tables |
| 171 | - exports formula cells as cached values; it does not recalculate formulas |
| 172 | - uses the shared best-effort conversion-profile contract after success |
| 173 | |
| 174 | Dependency: |
| 175 | |
| 176 | ```bash |
| 177 | pip install openpyxl |
| 178 | ``` |
| 179 | |
| 180 | CSV/TSV files are already plain-text table sources and do not require this converter. |
| 181 | |
| 182 | ## `source_to_md/ppt_to_md.py` |
| 183 | |
| 184 | Structured PowerPoint-to-Markdown converter for Open XML slide decks. |
| 185 | |
| 186 | Supported formats include: |
| 187 | - `.pptx`, `.pptm` |
| 188 | - `.ppsx`, `.ppsm` |
| 189 | - `.potx`, `.potm` |
| 190 | |
| 191 | ```bash |
| 192 | python3 scripts/source_to_md/ppt_to_md.py sales_deck.pptx |
| 193 | python3 scripts/source_to_md/ppt_to_md.py sales_deck.pptx -o output.md |
| 194 | python3 scripts/source_to_md/ppt_to_md.py sales_deck.pptx appendix.pptx |
| 195 | python3 scripts/source_to_md/ppt_to_md.py ./decks |
| 196 | python3 scripts/source_to_md/ppt_to_md.py ./decks -o ./markdown # explicit separate output directory |
| 197 | python3 scripts/source_to_md/ppt_to_md.py template.ppsx -o notes/template.md |
| 198 | ``` |
| 199 | |
| 200 | Behavior: |
| 201 | - extracts slide text in reading order |
| 202 | - converts PowerPoint tables to Markdown tables; cell-internal line breaks become `<br>` so they cannot break the pipe-table row structure |
| 203 | - transcribes category charts as category × series tables and scatter/bubble charts as typed X/Y[/size] point tables |
| 204 | - preserves every readable chart dimension or series and emits `[Chart data warning: <reason>]` for missing caches/count mismatches; `[Chart data unavailable: <reason>]` is reserved for charts with no readable points (and unsupported ChartEx), so XY data is never flattened into a misleading category table |
| 205 | - transcribes SmartArt semantic nodes as hierarchical Markdown; unreadable diagram data emits an explicit placeholder and conversion warning |
| 206 | - exports embedded pictures to a sibling `_files/` directory |
| 207 | - preserves supported run, table-cell, picture, and text-shape links as Markdown links, including `#slide-N` jumps |
| 208 | - appends speaker notes when present |
| 209 | - uses the shared best-effort conversion-profile contract after success |
| 210 | |
| 211 | Dependency: |
| 212 | |
| 213 | ```bash |
| 214 | pip install python-pptx |
| 215 | ``` |
| 216 | |
| 217 | Legacy `.ppt` is not parsed directly. Resave it as `.pptx` or export it to PDF first. |
| 218 | |
| 219 | ## `pptx_intake.py` |
| 220 | |
| 221 | Standard enrichment layer for PPTX sources. It complements `ppt_to_md.py` rather |
| 222 | than replacing it: Markdown remains the normalized content source, while intake |
| 223 | artifacts provide source facts for Strategist and standalone PPTX workflows. |
| 224 | |
| 225 | ```bash |
| 226 | python3 scripts/pptx_intake.py deck.pptx -o projects/demo/analysis |
| 227 | ``` |
| 228 | |
| 229 | Outputs (per source deck, prefixed by file stem): |
| 230 | - `<stem>.identity.json` — canvas size/aspect, theme palette/fonts, observed colors/fonts |
| 231 | - `<stem>.slide_library.json` — text slots, geometry, native tables, native chart display caches, and SmartArt nodes/connections |
| 232 | - `source_profile.json` — the single multi-deck index: a compact Strategist-facing digest per deck (over identity, tables, charts, SmartArt, and page types) under `decks[]`, with prefixed artifact pointers |
| 233 | |
| 234 | `project_manager.py import-sources` runs this automatically for PPTX/PPTM/PPSX/PPSM/POTX/POTM inputs and stores the bundle directly under `analysis/`. Multi-deck per project: importing several PPTX files gives each its own `<stem>.*` artifacts and a `decks[]` entry in the shared `source_profile.json` index (re-importing the same stem replaces its entry). The beautify profile and Fill Native PPTX route stay single-deck and read one chosen deck's `<stem>.*` artifacts. |
| 235 | |
| 236 | Usage boundary: |
| 237 | - Standard generation uses these fields as facts and recommendation candidates; it does not inherit source slide coordinates or page order by default. |
| 238 | - Beautify promotes selected identity/content fields into locked constraints after confirmation and redraws SmartArt meaning with ordinary editable shapes. |
| 239 | - Template-fill uses the slide library as the native PPTX fill contract; SmartArt is inventory-only and remains unchanged. |
| 240 | |
| 241 | ## `pptx_to_svg.py` |
| 242 | |
| 243 | Reconstruct a PPTX package as editable SVG views by reading OOXML directly. |
| 244 | |
| 245 | ```bash |
| 246 | python3 scripts/pptx_to_svg.py deck.pptx --inheritance-mode both |
| 247 | python3 scripts/pptx_to_svg.py deck.pptx --inheritance-mode layered |
| 248 | python3 scripts/pptx_to_svg.py deck.pptx --inheritance-mode flat |
| 249 | python3 scripts/pptx_to_svg.py deck.pptx --strict |
| 250 | ``` |
| 251 | |
| 252 | | Mode | Output | |
| 253 | |---|---| |
| 254 | | `both` (default) | Layered master/layout/slide SVGs under `svg/`, plus self-contained slides under `svg-flat/` | |
| 255 | | `layered` | Only the layered `svg/` view and inheritance metadata | |
| 256 | | `flat` | One self-contained slide SVG per page under `svg/` | |
| 257 | |
| 258 | Every mode also writes a canonical `animations.json`. Its default transition |
| 259 | is `none`, so slides without a source transition stay transition-free when the |
| 260 | workspace is exported again. |
| 261 | |
| 262 | For Office pictures that carry both a raster compatibility preview on |
| 263 | `a:blip` and an editable SVG relationship in `asvg:svgBlip`, import resolves |
| 264 | the SVG relationship first. The raster relationship is used only when the SVG |
| 265 | relationship or media part cannot be read. The template manifest uses the same |
| 266 | relationship preference for asset identity; its existing missing-media gate |
| 267 | remains strict rather than silently treating the raster preview as the |
| 268 | template's canonical asset. |
| 269 | |
| 270 | Supported `a:hlinkClick` on shape/picture `p:cNvPr` and text `a:rPr` becomes |
| 271 | the shared SVG `<a href>` form for absolute external URIs and final-roster |
| 272 | `#slide-N` jumps. A source shape that also has linked inner runs uses the |
| 273 | importer-only `data-pptx-shape-hyperlink` transport to avoid nested SVG anchors. |
| 274 | Unsupported click actions produce a diagnostic; strict import stops. |
| 275 | |
| 276 | ### Import compatibility and recovery boundary |
| 277 | |
| 278 | Import is tolerant by default because the source deck is user-owned or comes |
| 279 | from third-party authoring tools. Recovery happens at the narrowest safe |
| 280 | boundary: first omit only an unsupported property or feature; if that is not |
| 281 | possible, replace only the affected object with a visible diagnostic |
| 282 | placeholder; omit a background without discarding its page. Corrupt ZIP/XML or |
| 283 | missing required package structure remains fatal because no safe local recovery |
| 284 | exists. Pass `--strict` for parser development or contract verification when |
| 285 | the first unsupported/malformed source construct should stop conversion. |
| 286 | |
| 287 | Every successful run writes `<output>/conversion-report.json`. Its stable |
| 288 | top-level fields are `schemaVersion`, `source`, `mode`, `summary`, `artifacts`, |
| 289 | and `diagnostics`; `artifacts.animationConfig` and |
| 290 | `artifacts.animationMedia` identify the converter-owned sidecar and transition |
| 291 | sounds. Each diagnostic records a reason `code`, source `message`, chosen |
| 292 | `fallback`, package `part_path`, and—when available—`slide_index`, `shape_id`, |
| 293 | `shape_name`, and `shape_kind`. The command also prints a bounded warning |
| 294 | summary instead of a raw Python traceback. |
| 295 | |
| 296 | In the detailed native-object notes below, “fails closed” or “error” describes |
| 297 | the native replacement claim or strict mode. Default tolerant deck import |
| 298 | retains the usable fallback/object and records the degradation; it does not |
| 299 | discard unrelated shapes, pages, or the entire deck. |
| 300 | |
| 301 | Source `p:transition` and `p:timing` nodes are never silently implied by the |
| 302 | static SVG view. Supported page transitions and finite object-animation |
| 303 | sequences are reconstructed in `animations.json`; source timing outside either |
| 304 | closed contract emits `transition-not-reconstructed` or |
| 305 | `animation-not-reconstructed` with the exact source slide. Direct PPTX |
| 306 | Fill/Enhance workflows remain the source-preserving route for all other timing; |
| 307 | `--strict` stops on the first unreconstructed node. |
| 308 | |
| 309 | ### Page-transition reverse import |
| 310 | |
| 311 | The importer accepts exactly the current generated-transition registry and |
| 312 | validates the source carrier with the same read-back contract used after |
| 313 | SVG-to-PPTX export. It reconstructs the canonical effect and all effective |
| 314 | options, exact `p14:dur`, optional `advTm`, and an internal WAV transition |
| 315 | sound. Sound bytes are extracted under the selected media directory with a |
| 316 | content-addressed filename and referenced from the sidecar. |
| 317 | |
| 318 | This is a PPT Master-owned semantic loop, not a general transition normalizer. |
| 319 | Unknown effects, legacy `p:transition@spd`, visual effects without exact |
| 320 | `p14:dur`, `advClick="0"`, malformed carriers, and unsupported or broken sound |
| 321 | relationships produce `transition-not-reconstructed` in tolerant mode; |
| 322 | `--strict` stops. The converter never substitutes `fade` for those cases. |
| 323 | |
| 324 | ### Finite object-animation reverse import |
| 325 | |
| 326 | The importer accepts only rows that pass the current generated-animation |
| 327 | behavior-tree validator and map both their target and optional click trigger to |
| 328 | one unique top-level slide SVG group. It reconstructs the canonical registry |
| 329 | effect, non-default effective options, Animation Pane order, Start trigger, |
| 330 | exact native duration, and relative delay. Repeated targets use `effects[]`; |
| 331 | shape-triggered rows restore `trigger_shape`. |
| 332 | |
| 333 | This exact-duration subset covers 199 of the 203 registered effects. The four |
| 334 | native rows without a readable behavior duration—`emphasis_change_font`, |
| 335 | `emphasis_change_font_style`, `emphasis_transparency`, and |
| 336 | `emphasis_bold_reveal`—remain diagnosed because their authored scheduling span |
| 337 | cannot be separated honestly from the following delay. Repeat/reverse/rewind, |
| 338 | acceleration/bounce/restart, after-effects, animation sounds, paragraph or |
| 339 | Chart/SmartArt builds, media commands, unknown behavior trees, and targets that |
| 340 | do not map to a top-level SVG group likewise produce |
| 341 | `animation-not-reconstructed`; `--strict` stops. The importer never invents a |
| 342 | replacement timing tree. |
| 343 | |
| 344 | ### Native formula reverse import |
| 345 | |
| 346 | The importer reconstructs formulas only from the closed OMML vocabulary owned |
| 347 | by the native formula compiler. One formula-only `a14:m > m:oMathPara` text |
| 348 | shape becomes a bounded `<g data-pptx-replace-with="formula">` with canonical |
| 349 | LaTeX JSON and a visible linear SVG preview when its carrier is an ungrouped, |
| 350 | unstyled, unrotated rectangular formula shape. Carrier styling, effects, |
| 351 | hyperlinks, or placeholder ownership force diagnosed fallback rather than |
| 352 | silent loss. |
| 353 | Supported `a14:m > m:oMath` zones inside an ordinary paragraph become leaf |
| 354 | `<tspan data-pptx-inline-formula="...">preview</tspan>` markers while retaining |
| 355 | their surrounding text runs. The generated markers pass the same native-object |
| 356 | and inline-formula validators used by SVG-to-PPTX export. |
| 357 | |
| 358 | This is normalized semantic read-back, not recovery of the author's original |
| 359 | LaTeX spelling and not a general Office Math converter. Every OMML root must |
| 360 | pass the compiler's namespace, element, attribute, structure, size, and depth |
| 361 | gates, and the reconstructed LaTeX must compile again under the same profile. |
| 362 | If any formula in one text body falls outside that boundary, tolerant import |
| 363 | keeps all formulas in that body as readable linear text and retains the |
| 364 | relationship-free source `txBody` as opaque metadata instead of partially |
| 365 | claiming native reconstruction. It records `formula-not-reconstructed`; |
| 366 | `--strict` stops on the same condition. |
| 367 | |
| 368 | ### Native table and chart import claims |
| 369 | |
| 370 | Supported text-grid tables and conservative classic-chart caches carry a |
| 371 | `data-pptx-replace-with` claim beside their SVG fallback, with the replacement |
| 372 | payload in a child `<metadata type="application/json">`. The parent claim |
| 373 | selects the table or chart schema. Table import requires |
| 374 | exact physical row/grid topology and accepts canonical rectangular merges, |
| 375 | safe solid/no-fill per-side borders, plain multi-paragraph cells, and a closed |
| 376 | run-rich paragraph schema. |
| 377 | Each run requires `text` and may use only `bold`, `italic`, `underline`, |
| 378 | `strike`, `color`, `font_size`, one `font_family`, `lang`, and `alt_lang`. |
| 379 | Presentation-only source run XML without a non-empty `effectLst` / `effectDag` |
| 380 | normalizes. A table-cell run effect disables native replacement and adds a |
| 381 | blocking effect diagnostic. Relationship-bearing text, extensions, |
| 382 | noncanonical/overlapping merges, nonblank merge slaves, unsafe border XML, |
| 383 | non-solid fills, structural line breaks/fields/tabs/bullets, and broken text |
| 384 | topology remain fallback-only. |
| 385 | Markers remain dormant |
| 386 | unless a later export uses `--native-charts-and-tables`. That opt-in is |
| 387 | data-object-first: the default fallback still exports as editable DrawingML |
| 388 | shapes, while the opt-in supplies a data source and PowerPoint's |
| 389 | chart/table-specific object model. |
| 390 | The native-object route may normalize styling or omit marker-local details not represented by the |
| 391 | payload, and export reports that risk without disabling an otherwise supported |
| 392 | active marker. Unsupported tables keep their |
| 393 | rendered SVG table; unsupported charts keep a baked preview when one exists. |
| 394 | For the currently supported parsed classic families (column/bar/line/area, |
| 395 | pie/doughnut, scatter, and bubble), a chart without a baked preview receives a |
| 396 | deterministic readable fallback marked |
| 397 | `data-pptx-fallback-kind="normalized"`. Unknown style XML disables the native |
| 398 | replacement claim or falls back to a diagnostic object in tolerant mode; |
| 399 | common solid/no-fill/line/marker forms and scheme colors are normalized for the |
| 400 | SVG fallback and core payload colors, while native opt-in may still normalize |
| 401 | unmodeled alpha, line, marker, or no-fill details. Common General, decimal, |
| 402 | grouped, percent, and simple currency-prefix data-label formats render |
| 403 | deterministically; an unknown Excel format program keeps the active payload but |
| 404 | does not claim a normalized fallback. Active types outside the current renderer |
| 405 | continue to use an explicit placeholder marked |
| 406 | `data-pptx-fallback-kind="placeholder"`. Validation and export report that |
| 407 | reconstruction-only fallback as a warning. Default export keeps the |
| 408 | placeholder; when the same group has a valid active chart replacement payload, |
| 409 | `--native-charts-and-tables` may still reconstruct the PowerPoint-native chart. |
| 410 | Invalid or contradictory fallback declarations remain errors. Fallback-only |
| 411 | replacement capability uses `data-pptx-replacement-status` and remains a |
| 412 | warning when the SVG fallback itself is complete. Imported table/chart groups |
| 413 | under this contract carry `data-pptx-import-source="pptx"`, whether active or |
| 414 | fallback-only; generated authoring omits this provenance attribute. |
| 415 | |
| 416 | Active imported markers also carry `data-pptx-fallback-sha256`, computed over |
| 417 | their canonical fallback plus reachable document-level SVG fragment definitions. |
| 418 | A later visible edit, reachable definition change, local reference-target |
| 419 | change, or marker transform makes the replacement metadata stale. The mandatory |
| 420 | quality checker reports the mismatch; default export keeps the edited fallback, |
| 421 | while `--native-charts-and-tables` fails before replacement so it cannot discard that edit. |
| 422 | `visibility:hidden` content, marker-local unused definitions, and explicitly |
| 423 | referenced document-level target roots (even when hidden) are included |
| 424 | conservatively; marker-local `display:none` subtrees are excluded, and external |
| 425 | file bytes are not read. |
| 426 | Generated authoring and reusable templates omit import provenance and do not |
| 427 | preseed a static fallback hash; that state is normal and does not warn. A legacy |
| 428 | imported marker that still carries PPTX import provenance but lacks the hash |
| 429 | remains native-compatible and warns in the checker/native route that stale |
| 430 | detection is unavailable. |
| 431 | |
| 432 | Legacy `data-pptx-native*`, `data-pptx-visual-status`, and |
| 433 | `data-pptx-route-status` spellings remain read-compatible. New importer output |
| 434 | and generated SVG use the replacement/fallback names above. The old |
| 435 | `--native-objects` option remains a compatibility alias for |
| 436 | `--native-charts-and-tables`. |
| 437 | |
| 438 | For table style `{5C22544A-7EE6-4342-B048-85BDC9FD1C3A}`, the importer resolves |
| 439 | the normalized `wholeTbl`, `firstRow`, `band1H`/`band2H`, theme color/font, and |
| 440 | direct-format override subset. Other built-in/custom style families remain |
| 441 | outside this guarantee. |
| 442 | |
| 443 | The chart importer also accepts the verified column/line/area combo subset, |
| 444 | canonical four-series OHLC stock charts with shared numeric date caches, area |
| 445 | charts with numeric date axes, and verified scatter/bubble charts with a closed |
| 446 | pair of `axes.x` / `axes.y` value axes. Combo primary/secondary plots may retain |
| 447 | independent category caches and workbook ranges. Both the category/value and XY |
| 448 | contracts read back the supported kind/position/visibility, label-position, |
| 449 | number-format, min/max/major-unit, reverse, and major-gridline fields; the native |
| 450 | writer emits every field in those closed contracts. Scatter style is derived |
| 451 | from uniform effective series line/marker/smooth state. The normalized XY |
| 452 | fallback newly consumes only the two major-gridline flags, not the remaining |
| 453 | axis fields. The importer also accepts radar, safe `of_pie` `serLines`, the |
| 454 | closed axis/title/legend normalization cases, and bar/column `gapWidth` / |
| 455 | `overlap`. `gapWidth` must be one integer in `0..500` and `overlap` one integer |
| 456 | in `-100..100`; both normalize in native output, while malformed, duplicate, or |
| 457 | out-of-range values disable the native replacement claim in tolerant mode and |
| 458 | stop strict import. These additions do not expand the normalized |
| 459 | renderer. |
| 460 | Safe stock series style may pass the structural gate, while stock series, |
| 461 | `hiLowLines`, and up-down bar local styling can still normalize under the |
| 462 | data-object-first contract. |
| 463 | ChartEx import accepts exactly the validated treemap, sunburst, histogram, |
| 464 | pareto, box-whisker, waterfall, and funnel data models. Their supported |
| 465 | hierarchy/category/value/series/subtotal topology round-trips to native output. |
| 466 | Numeric caches must be non-empty and finite, with canonical non-negative counts |
| 467 | and indexes and exact contiguous point topology. Source style, axes, labels, |
| 468 | and binning details may normalize. This is not full `AxisSpec`, arbitrary |
| 469 | ChartEx import or presentation fidelity, arbitrary stock variants, other |
| 470 | date-axis chart families, or unlisted axis semantics. ChartEx native output |
| 471 | still consumes valid payload colors in its color-style part. |
| 472 | |
| 473 | Exporter-canonical charts recover canonical solid series/slice colors and exact |
| 474 | one- or two-paragraph title styling; two paragraphs retain their `title` / |
| 475 | `subtitle` roles. This is not a general source-chart style round-trip guarantee. |
| 476 | |
| 477 | Concrete slide SVGs resolve `<a:fld type="slidenum">` using the presentation's |
| 478 | `firstSlideNum` display numbering. Standalone master/layout SVGs keep the |
| 479 | literal field fallback because one shared part can serve multiple slides. |
| 480 | |
| 481 | ### Maintenance smoke checks |
| 482 | |
| 483 | Run these checks from the repository root after changing `pptx_to_svg/` or its |
| 484 | CLI. They generate every required input under `/tmp`; do not replace them with |
| 485 | a committed `test_*.py` suite. |
| 486 | |
| 487 | #### Healthy generated deck |
| 488 | |
| 489 | ```bash |
| 490 | python3 - <<'PY' |
| 491 | from pptx import Presentation |
| 492 | from pptx.enum.shapes import MSO_SHAPE |
| 493 | from pptx.util import Inches |
| 494 | |
| 495 | presentation = Presentation() |
| 496 | slide = presentation.slides.add_slide(presentation.slide_layouts[6]) |
| 497 | shape = slide.shapes.add_shape( |
| 498 | MSO_SHAPE.RECTANGLE, |
| 499 | Inches(1), |
| 500 | Inches(1), |
| 501 | Inches(3), |
| 502 | Inches(1), |
| 503 | ) |
| 504 | shape.text = "PPTX import smoke check" |
| 505 | presentation.save("/tmp/ppt-master-smoke-healthy.pptx") |
| 506 | PY |
| 507 | |
| 508 | python3 "skills/ppt-master/scripts/pptx_to_svg.py" \ |
| 509 | "/tmp/ppt-master-smoke-healthy.pptx" \ |
| 510 | --inheritance-mode flat \ |
| 511 | -o "/tmp/ppt-master-smoke-healthy" |
| 512 | python3 -c "import json; from pathlib import Path; report = json.loads(Path('/tmp/ppt-master-smoke-healthy/conversion-report.json').read_text()); assert report['summary'] == {'slides': 1, 'warnings': 0}, report['summary']; print('OK: 1 slide, 0 warnings')" |
| 513 | ``` |
| 514 | |
| 515 | Expected: both commands exit `0`; the assertion prints |
| 516 | `OK: 1 slide, 0 warnings`. |
| 517 | |
| 518 | #### Tolerant/strict color-structure probe |
| 519 | |
| 520 | Generate a two-shape PPTX, then add one foreign attribute to the first shape's |
| 521 | valid `a:srgbClr` node: |
| 522 | |
| 523 | ```bash |
| 524 | python3 -c ' |
| 525 | from pathlib import Path |
| 526 | from zipfile import ZIP_DEFLATED, ZipFile |
| 527 | |
| 528 | from pptx import Presentation |
| 529 | from pptx.dml.color import RGBColor |
| 530 | from pptx.enum.shapes import MSO_SHAPE |
| 531 | from pptx.util import Inches |
| 532 | |
| 533 | base = Path("/tmp/ppt-master-color-smoke-base.pptx") |
| 534 | target = Path("/tmp/ppt-master-color-smoke.pptx") |
| 535 | presentation = Presentation() |
| 536 | slide = presentation.slides.add_slide(presentation.slide_layouts[6]) |
| 537 | for left, color in ((1, (0x44, 0x72, 0xC4)), (4, (0xED, 0x7D, 0x31))): |
| 538 | shape = slide.shapes.add_shape( |
| 539 | MSO_SHAPE.RECTANGLE, |
| 540 | Inches(left), |
| 541 | Inches(1), |
| 542 | Inches(2), |
| 543 | Inches(1), |
| 544 | ) |
| 545 | shape.fill.solid() |
| 546 | shape.fill.fore_color.rgb = RGBColor(*color) |
| 547 | presentation.save(base) |
| 548 | |
| 549 | with ZipFile(base) as source, ZipFile(target, "w", ZIP_DEFLATED) as destination: |
| 550 | patched = False |
| 551 | for member in source.infolist(): |
| 552 | payload = source.read(member) |
| 553 | if member.filename == "ppt/slides/slide1.xml": |
| 554 | old = b"<a:srgbClr val=\"4472C4\"/>" |
| 555 | new = b"<a:srgbClr val=\"4472C4\" legacy=\"1\"/>" |
| 556 | if old not in payload: |
| 557 | raise RuntimeError("probe color node was not generated") |
| 558 | payload = payload.replace(old, new, 1) |
| 559 | patched = True |
| 560 | destination.writestr(member, payload) |
| 561 | if not patched: |
| 562 | raise RuntimeError("slide XML was not patched") |
| 563 | print(target) |
| 564 | ' |
| 565 | ``` |
| 566 | |
| 567 | Run tolerant import and verify both the recovery report and the visible SVG: |
| 568 | |
| 569 | ```bash |
| 570 | python3 "skills/ppt-master/scripts/pptx_to_svg.py" \ |
| 571 | "/tmp/ppt-master-color-smoke.pptx" \ |
| 572 | --inheritance-mode flat \ |
| 573 | -o "/tmp/ppt-master-smoke-color-tolerant" |
| 574 | python3 -c ' |
| 575 | import json |
| 576 | from pathlib import Path |
| 577 | |
| 578 | output = Path("/tmp/ppt-master-smoke-color-tolerant") |
| 579 | report = json.loads((output / "conversion-report.json").read_text()) |
| 580 | diagnostics = report["diagnostics"] |
| 581 | svg = (output / "svg" / "slide_01.svg").read_text() |
| 582 | assert report["summary"] == {"slides": 1, "warnings": 1}, report["summary"] |
| 583 | assert len(diagnostics) == 1, diagnostics |
| 584 | assert diagnostics[0]["code"] == "color-structure-normalized", diagnostics[0] |
| 585 | assert diagnostics[0]["fallback"] == "retain recognized color attributes and modifiers", diagnostics[0] |
| 586 | assert diagnostics[0]["slide_index"] == 1, diagnostics[0] |
| 587 | assert diagnostics[0]["shape_name"] == "Rectangle 1", diagnostics[0] |
| 588 | assert "#4472C4" in svg and "#ED7D31" in svg |
| 589 | print("OK: tolerant import recovered #4472C4 and preserved #ED7D31") |
| 590 | ' |
| 591 | ``` |
| 592 | |
| 593 | Expected: both commands exit `0`; the importer reports one |
| 594 | `color-structure-normalized` warning and the assertion prints |
| 595 | `OK: tolerant import recovered #4472C4 and preserved #ED7D31`. |
| 596 | |
| 597 | Run the same probe in strict mode: |
| 598 | |
| 599 | ```bash |
| 600 | python3 "skills/ppt-master/scripts/pptx_to_svg.py" \ |
| 601 | "/tmp/ppt-master-color-smoke.pptx" \ |
| 602 | --inheritance-mode flat \ |
| 603 | --strict \ |
| 604 | -o "/tmp/ppt-master-smoke-color-strict" |
| 605 | ``` |
| 606 | |
| 607 | Expected: exit `1`, no traceback, and one error line: |
| 608 | |
| 609 | ```text |
| 610 | Error: PPTX-to-SVG conversion failed: Invalid DrawingML sRGB color structure |
| 611 | ``` |
| 612 | |
| 613 | ## `source_to_md/web_to_md.py` |
| 614 | |
| 615 | Convert web pages to Markdown and download images locally by default. Use |
| 616 | `--no-images` to retain remote image links without downloading their files. |
| 617 | |
| 618 | ```bash |
| 619 | python3 scripts/source_to_md/web_to_md.py https://example.com/article |
| 620 | python3 scripts/source_to_md/web_to_md.py https://url1.com https://url2.com |
| 621 | python3 scripts/source_to_md/web_to_md.py -f urls.txt |
| 622 | python3 scripts/source_to_md/web_to_md.py https://example.com -o output.md |
| 623 | python3 scripts/source_to_md/web_to_md.py https://example.com --emit-result /tmp/result.json |
| 624 | python3 scripts/source_to_md/web_to_md.py https://example.com -o evidence.md --no-images |
| 625 | ``` |
| 626 | |
| 627 | When `curl_cffi` is installed (included in `requirements.txt`), this script |
| 628 | automatically impersonates a modern Chrome TLS fingerprint, which lets it |
| 629 | fetch WeChat Official Accounts (`mp.weixin.qq.com`) and other sites that |
| 630 | block Python's default TLS fingerprint. No extra flags needed. If |
| 631 | `curl_cffi` is not available, it falls back to plain `requests`. |
| 632 | |
| 633 | On success, the converter uses the shared best-effort sidecar contract for |
| 634 | `<output>.conversion_profile.json` beside the Markdown output. |
| 635 | `--emit-result` is for wrapper scripts that need the actual saved Markdown path |
| 636 | when the converter derives a title-based filename. |
| 637 | |
| 638 | |
| 639 | ## Image Orientation Review |
| 640 | |
| 641 | Run this review when the user requests orientation correction, converted text |
| 642 | asks the reader to rotate the device, or a downloaded asset is visibly |
| 643 | sideways. EXIF and dimensions may trigger review, but they cannot determine the |
| 644 | semantic direction of pixels that are already stored sideways. |
| 645 | |
| 646 | Generate a labeled static contact sheet. This command previews the first frame |
| 647 | after EXIF normalization and does not modify source images: |
| 648 | |
| 649 | ```bash |
| 650 | python3 ${SKILL_DIR}/scripts/rotate_images.py sheet <images_directory> |
| 651 | ``` |
| 652 | |
| 653 | The default output is |
| 654 | `<images_directory>/../analysis/<directory>_orientation_contact_sheet.jpg`. |
| 655 | Inspect it with the current multimodal agent, identify only visually confirmed |
| 656 | rotations, and write a temporary JSON list. `rotation` is clockwise degrees and |
| 657 | must be `90`, `180`, or `270`: |
| 658 | |
| 659 | ```json |
| 660 | [ |
| 661 | {"path": "/absolute/path/to/sideways.jpg", "rotation": 270} |
| 662 | ] |
| 663 | ``` |
| 664 | |
| 665 | Apply the confirmed fixes and regenerate image facts: |
| 666 | |
| 667 | ```bash |
| 668 | python3 ${SKILL_DIR}/scripts/rotate_images.py fix /tmp/orientation_fixes.json |
| 669 | python3 ${SKILL_DIR}/scripts/analyze_images.py <images_directory> |
| 670 | ``` |
| 671 | |
| 672 | GIF files are excluded: `sheet` does not list them, and `fix` rejects a batch |
| 673 | that references one so all GIF files remain unchanged. |
| 674 | |
| 675 | Do not infer a rotation from prose, EXIF, or aspect ratio alone, and do not |
| 676 | launch the HTML `gen` command in source intake. `auto` remains an in-place EXIF |
| 677 | normalizer. `gen` is a compatibility UI that runs the same normalization before |
| 678 | writing HTML; neither belongs to source intake. |
| 679 |