mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-25 03:42:06 +00:00
ca3428fa07061a8eb966f25b5432d2df19dfd77f
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
8771f95ab2 |
feat(urlimport): OpenAPI 3.x → Markdown converter (TASK-1471) (#555)
* feat(urlimport): OpenAPI 3.x → Markdown converter (TASK-1471)
Adds ConvertOpenAPI to internal/urlimport — the "openapi" branch of
the v1 importer. Built on pb33f/libopenapi.
Layout:
- H1 with the API title + version + description
- Contact + License lines
- Servers list
- Endpoints section grouped by primary tag (or "Other" for the
untagged). Per operation: `METHOD /path` heading, summary,
description, deprecation marker, operation ID, parameter table,
request-body summary (with media-type fences and YAML-rendered
example), and response code table.
- Schemas section with component schemas as Property/Type/Required/
Description tables, schema names sorted for stable output.
Scope:
- OpenAPI 3.x only. Swagger 2.0 detection returns an explicit
"only 3.x" error so the import endpoint (TASK-1472) can fall
through to the generic converter.
- Recoverable libopenapi build errors (unresolved refs, etc.) are
swallowed when the model is still produced — partial spec >
no output.
Tests:
- testdata/petstore-openapi.yaml — full v3 fixture: tags, params,
requestBody example, deprecated op, ref-typed schema array, two
component schemas with required-field markers.
- TestConvertOpenAPI_Petstore — 30+ markdown-substring assertions
on the rendered output.
- TestConvertOpenAPI_RejectsSwagger2 — explicit v2 error.
- TestConvertOpenAPI_RejectsGarbage — non-spec input.
- TestConvertOpenAPI_MinimalSpec — empty paths short-circuits.
- Helpers: schemaTypeBrief(nil), escapeTableCell, singleLine.
Dependency: github.com/pb33f/libopenapi v0.36.3 (MIT-licensed).
Parent: PLAN-1467.
* fix(urlimport): merge path-level + operation-level parameters per Codex review (round 1)
MEDIUM: OpenAPI path-item-level parameters apply to every operation
on the path. Previously only slot.op.Parameters was rendered, so
common specs that hoist a shared {id} parameter to the path-item
level emitted operations with the path parameter missing from the
docs.
Now opSlot carries item.Parameters as pathParams, and a new
mergeParameters helper produces the spec-conformant union:
- Path-level parameters first, in declared order.
- Operation-level parameters with matching (name, in) override the
path-level entry in place.
- Operation-only parameters appended after.
Tests:
- TestConvertOpenAPI_PathLevelParametersMerged — inline fixture
with a path-level widgetId + trace and an operation-level trace
override + fields op-only param. Asserts widgetId survives, trace
shows op-level (required=yes), no duplicate path-level trace row,
fields appears.
- TestMergeParameters_EmptyInputs — nil/nil short-circuit.
* fix(urlimport): no double-backticks on array-of-ref schema types per Codex review (round 2)
MEDIUM: schemaTypeBrief() previously wrapped refs in inline backticks
("`Pet`"). For array-of-ref schemas the brief became "array of `Pet`",
and the table-cell call site (codeOrBlank) then wrapped the entire
value in another pair, producing broken markdown like
"`array of `Pet``". Schema properties whose type is an array of a
component schema are a normal OpenAPI shape — `Litter.pets: array of
Pet` — so this would have hit real specs immediately.
Fixes:
- schemaTypeBrief now returns plain text — ref names without
surrounding backticks. Docstring updated to make the contract
explicit ("never contains backticks; caller wraps").
- codeOrBlank strips any stray backticks from input before wrapping
so the resulting cell always carries exactly one balanced pair.
Defensive: the contract from schemaTypeBrief is plain text now,
but stray backticks from any future caller can't corrupt the
table.
Tests added:
- TestConvertOpenAPI_ArrayOfRefTypeCell — inline spec with a
`Litter.pets: array of Pet` property. Asserts the type cell is
exactly `` `array of Pet` `` and no malformed variants leak.
- TestCodeOrBlank — 7-case table covering empty, plain, whitespace,
pre-backticked, embedded-backtick, and backtick-only inputs.
|
||
|
xarmian
|
d1560606cb |
feat(urlimport): generic HTML→Markdown converter (TASK-1470) (#553)
* feat(urlimport): generic HTML→Markdown converter (TASK-1470)
Adds ConvertGeneric to internal/urlimport — the v1 catch-all converter
for "non-OpenAPI" URLs. Pipeline:
1. go-shiori/go-readability strips chrome/nav/ads/scripts and returns
the page's primary article.
2. JohannesKaufmann/html-to-markdown/v2 converts the cleaned HTML to
markdown.
3. cleanupMarkdown normalizes line endings, trims trailing whitespace,
collapses blank-line runs, and ensures a single trailing newline.
Fallback path: when Readability cannot identify an article (directory
listings, single paragraphs, pages with no clear content container),
the converter falls back to a whole-body conversion so callers still
get usable markdown.
Dependencies (license-checked):
- github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.1 (MIT)
- github.com/go-shiori/go-readability (Apache-2.0)
Fixtures + tests:
- testdata/availity-shape.html — Availity-style div soup with heavy
chrome (nav, ads, sidebar, footer, analytics script). Asserts the
article content survives and the chrome is stripped.
- testdata/mdn-shape.html — MDN-style semantic HTML (article/main +
proper heading levels + code fences). Asserts structure preserved.
- TestConvertGeneric_EmptyBody — empty-input rejection.
- TestConvertGeneric_PlainTextFallback — Readability-can't-find-article
fallback path.
- TestCleanupMarkdown — 5-case table-driven cleanup verification.
Parent: PLAN-1467.
* fix(urlimport): preserve hard-line-break markers + apply WithDomain on fallback per Codex review (round 1)
- MEDIUM: cleanupMarkdown was stripping the markdown two-trailing-
spaces hard-line-break idiom. html-to-markdown emits <br> as
" \n" — bulk-stripping trailing whitespace was demoting hard
breaks to soft wraps. Now the cleanup steps line-by-line, keeps
exactly-two trailing spaces (no tab), strips 1/3+/tab-mixed runs.
- MEDIUM: The raw-HTML fallback path (used when Readability cannot
identify an article) now passes converter.WithDomain(pageURL) so
relative links/images resolve against the source URL rather than
the Pad host where they'd 404.
Tests added:
- cleanupMarkdown: hard-line-break preserved, single/triple trailing
spaces stripped, tab-mixed spaces stripped, blank-line-with-spaces
collapsed (5 new cases).
- TestConvertGeneric_RelativeURLsResolvedOnFallback: relative href
in non-article HTML resolves to absolute URL via pageURL.
|