mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-25 03:42:06 +00:00
xarmian
d1560606cb
* feat(urlimport): generic HTML→Markdown converter (TASK-1470)
Adds ConvertGeneric to internal/urlimport — the v1 catch-all converter
for "non-OpenAPI" URLs. Pipeline:
1. go-shiori/go-readability strips chrome/nav/ads/scripts and returns
the page's primary article.
2. JohannesKaufmann/html-to-markdown/v2 converts the cleaned HTML to
markdown.
3. cleanupMarkdown normalizes line endings, trims trailing whitespace,
collapses blank-line runs, and ensures a single trailing newline.
Fallback path: when Readability cannot identify an article (directory
listings, single paragraphs, pages with no clear content container),
the converter falls back to a whole-body conversion so callers still
get usable markdown.
Dependencies (license-checked):
- github.com/JohannesKaufmann/html-to-markdown/v2 v2.5.1 (MIT)
- github.com/go-shiori/go-readability (Apache-2.0)
Fixtures + tests:
- testdata/availity-shape.html — Availity-style div soup with heavy
chrome (nav, ads, sidebar, footer, analytics script). Asserts the
article content survives and the chrome is stripped.
- testdata/mdn-shape.html — MDN-style semantic HTML (article/main +
proper heading levels + code fences). Asserts structure preserved.
- TestConvertGeneric_EmptyBody — empty-input rejection.
- TestConvertGeneric_PlainTextFallback — Readability-can't-find-article
fallback path.
- TestCleanupMarkdown — 5-case table-driven cleanup verification.
Parent: PLAN-1467.
* fix(urlimport): preserve hard-line-break markers + apply WithDomain on fallback per Codex review (round 1)
- MEDIUM: cleanupMarkdown was stripping the markdown two-trailing-
spaces hard-line-break idiom. html-to-markdown emits <br> as
" \n" — bulk-stripping trailing whitespace was demoting hard
breaks to soft wraps. Now the cleanup steps line-by-line, keeps
exactly-two trailing spaces (no tab), strips 1/3+/tab-mixed runs.
- MEDIUM: The raw-HTML fallback path (used when Readability cannot
identify an article) now passes converter.WithDomain(pageURL) so
relative links/images resolve against the source URL rather than
the Pad host where they'd 404.
Tests added:
- cleanupMarkdown: hard-line-break preserved, single/triple trailing
spaces stripped, tab-mixed spaces stripped, blank-line-with-spaces
collapsed (5 new cases).
- TestConvertGeneric_RelativeURLsResolvedOnFallback: relative href
in non-article HTML resolves to absolute URL via pageURL.
58 lines
1.6 KiB
HTML
58 lines
1.6 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Array.prototype.flat() — MDN-shape</title>
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<nav>
|
|
<ul>
|
|
<li><a href="/">Home</a></li>
|
|
<li><a href="/docs">Docs</a></li>
|
|
</ul>
|
|
</nav>
|
|
</header>
|
|
<main>
|
|
<article>
|
|
<h1>Array.prototype.flat()</h1>
|
|
<p>The <code>flat()</code> method of <a href="/docs/Array"><code>Array</code></a> instances creates a new array with all sub-array elements concatenated into it recursively up to the specified depth.</p>
|
|
|
|
<h2 id="syntax">Syntax</h2>
|
|
<pre><code class="language-js">flat()
|
|
flat(depth)</code></pre>
|
|
|
|
<h3 id="parameters">Parameters</h3>
|
|
<dl>
|
|
<dt><code>depth</code> <em>(optional)</em></dt>
|
|
<dd>The depth level specifying how deep a nested array structure should be flattened. Defaults to <code>1</code>.</dd>
|
|
</dl>
|
|
|
|
<h3 id="return_value">Return value</h3>
|
|
<p>A new array with the sub-array elements concatenated into it.</p>
|
|
|
|
<h2 id="examples">Examples</h2>
|
|
<h3 id="flattening_nested_arrays">Flattening nested arrays</h3>
|
|
<pre><code class="language-js">const arr1 = [1, 2, [3, 4]];
|
|
arr1.flat();
|
|
// [1, 2, 3, 4]
|
|
|
|
const arr2 = [1, 2, [3, 4, [5, 6]]];
|
|
arr2.flat();
|
|
// [1, 2, 3, 4, [5, 6]]
|
|
|
|
const arr3 = [1, 2, [3, 4, [5, 6]]];
|
|
arr3.flat(2);
|
|
// [1, 2, 3, 4, 5, 6]
|
|
</code></pre>
|
|
|
|
<h2 id="specifications">Specifications</h2>
|
|
<p>See <a href="https://tc39.es/ecma262/#sec-array.prototype.flat">ECMA-262, Array.prototype.flat</a>.</p>
|
|
</article>
|
|
</main>
|
|
<footer>
|
|
<p>© Mozilla and individual contributors</p>
|
|
</footer>
|
|
</body>
|
|
</html>
|