To harden Renovate from being able to access internal hosts, we can add
a level of enforcement on all outgoing requests.
We can split this into two types of traffic:
- where the response becomes configuration (i.e. `extends`)
- where the response does not become configuration (i.e.
`customDatasources`)
To harden this, we enforce on every outbound HTTP request:
- check the resolved URL before anything, including cache lookups, can
proceed
- catching IP-literal hosts in every spelling the WHATWG URL parser normalizes
- blocking known Cloud metadata IPs, too
- a `dnsLookup` wrapper validates every address a hostname resolves to,
on every new connection, which also defeats DNS rebinding
- a `beforeRedirect` hook re-runs the URL check on each redirect
target, with the new host getting its own grant lookup rather than
inheriting the original host's verdict
We default this enforcement to `warn`, to not break existing deployments
and instead give them visibility into required configuration before a
future major version (major+1) enables `block` by default.
We'll also remove allow in a future major version (major+2), to give
time for admins to migrate over.
When a request is blocked, this degrades like an unreachable registry,
so a Renovate run isn't aborted by a blocked request.
As part of this, we can also provide a worked example of the different
cases that do/do not need allowing via `hostRules`.
Requests through an admin-configured proxy resolve DNS at the proxy, so
`dnsLookup` never runs for them, so we can resolve the hostname locally
and validate every address before the request instead.
This doesn't catch everything, especially as the proxy may resolve the
name to a different result, and if we can't resolve the name - possible
in locked-down corporate environments - we fail open.
Child processes aren't currently able to route traffic through
Renovate's HTTP layer, so are unrestricted.
Co-authored-by: Claude Fable 5 <jamie.tanna+claude-code@mend.io>
Co-authored-by: Claude Opus 5 <jamie.tanna+claude-code@mend.io>
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
* ci(linter): add renovate/require-regex-util rule
Adds a custom oxlint rule that flags raw RegExp usage in lib/
production code: any `new RegExp(...)` call and any regex literal
that is not the direct first argument of `regEx(...)`. The regEx()
helper from lib/util/regex.ts uses the RE2 engine for ReDoS-safe
matching and caches compiled patterns. lib/util/regex.ts itself is
exempt.
The rule is disabled for **/*.spec.ts, test/**, and tools/** via
config overrides, because vitest APIs such as expect.stringMatching
need bare RegExp instances (a past commit reverted regEx() there).
Severity is set to "warn" instead of "error": the repo currently has
197 pre-existing violations (above the agreed mechanical-fix
threshold of 80), and a number of them are deliberately
RE2-incompatible patterns (lookahead/lookbehind/named backreferences,
see TODO #12872/#12875 markers in e.g. lib/modules/versioning/rez/
pattern.ts and lib/modules/manager/dockerfile/extract.ts) that need
per-site judgment before they can be wrapped in regEx() or suppressed
with a justified disable comment. No violation fixes are included in
this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci(linter): enforce renovate/require-regex-util at error
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: use regEx() for regex literals and RegExp construction in lib
Burn down all renovate/require-regex-util violations (197):
- wrap RE2-compatible regex literals and RegExp constructions in regEx()
- rewrite lookaround patterns where behavior is provably preserved:
goproxy separator split, sbt-package link extraction, leiningen
comment stripping and keyword search, npm GitHub owner validation
(unit-count bound replaced by an explicit length check)
- keep native RegExp with disable comments where RE2 cannot be used:
moo / good-enough-parser lexer patterns (moo rejects RE2 instances),
MigrationsService propertyName (matched via isRegExp, which RE2
instances fail), rez named backreferences and lookaheads, GitHub
markdown link negative lookbehind, emoji astral-plane matching
- drop stale TODO #12875 comments for patterns current re2 handles
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: reduce require-regex-util disable comments
Rewrite the github massage-markdown-links URL regex to capture an
optional `api.` prefix in a named group (skipped in code) instead of
relying on a negative lookbehind, so both the base and global variants
can use regEx()/RE2 and their two disable comments are removed. Add
spec cases for api-prefixed URLs that match the URL shape.
Consolidate the four related line disables in rez/pattern.ts (bound and
range patterns emulating Python conditional patterns via named
backreferences and lookaheads, which RE2 cannot express) into a single
block disable/enable pair, matching the treatment of the lexer files.
Disable comment count drops from 13 to 8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(datasource/sbt-package): use named capture group in link regex
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(manager/cake): restore original multiLineComment regex spelling
The [^] -> [\\s\\S] rewrite was only needed for an RE2 wrap that was
later reverted; keep the diff minimal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(datasource/go): skip empty GOPROXY segments
The switch from a lookahead split to a capture-group split (needed because
RE2 has no lookahead support) changed how empty GOPROXY segments parse:
`a|,b` yielded `{ url: ',', fallback: '|' }` before and
`{ url: '', fallback: ',' }` after. The comma fallback only continues on
404/410, so a failing proxy ahead of an empty segment silently stopped the
lookup instead of falling through to the remaining proxies.
Empty segments carry no url to query, so drop them.
* refactor(datasource/go): use isTruthy for empty segment filter
* fix(datasource/go): filter empty segments before mapping
Filtering after the map broke the contextual typing from the
GoproxyItem[] annotation, widening fallback to string.
* refactor(platform/github): use regEx() for merge queue query stripping
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Enable the type-aware typescript/no-unnecessary-type-arguments rule as
error and remove all 44 pre-existing redundant type arguments (type
arguments identical to the type parameter default), e.g.
HttpResponse<string>, ZodError<unknown>,
PackageDependency<Record<string, any>>.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Enable no-else-return (with allowElseIf: false) and unicorn/no-lonely-if
as errors in .oxlintrc.json and fix all 131 pre-existing violations
(91 no-else-return, 40 unicorn/no-lonely-if) across 98 files.
The no-else-return violations were fixed with the oxlint autofix
(iterated to a fixpoint, plus two else-if chains that required
--fix-dangerously) followed by Prettier formatting. One comment dropped
by the autofix in lib/modules/manager/pub/artifacts.ts was restored
manually. The unicorn/no-lonely-if violations were merged by hand,
preserving comments and coverage-ignore hints.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
In 01bcd4fd43 we introduced knowledge of
the Maven Central mirror, but haven't taken it into account when looking
at rate limit errors.
To prevent this in the future, we can introduce a new helper function,
`isMavenCentral`, for other usage.
* fix(datasource/maven): cache 404s on `maven-metadata.xml` for 24h
As part of efforts to reduce the general overload on Maven Central[0]
from Renovate, we can look to reduce unnecessary HTTP lookups.
The Maven Datasource will use the `registryStrategy=merge`, which leads
to looking up a package in each of our registries, and merging them if
present.
However, this means that when looking up a package that /will only ever
be in a given registry/ (such as in
01bcd4fd43) we're hitting other registries
with packages they are unlikely to have.
This is unnecessary, significantly increases traffic to Maven Central,
and leads to us re-fetching the same 404s which count towards rate
limits, and increased, unnecessary, server load on Maven Central.
We can instead introduce a new cache for 404s for the
`maven-metadata.xml`, for 24 hours, which should reduce traffic to Maven
Central (and other configured registries) if they only have a subset of
dependencies.
[0]: https://www.sonatype.com/blog/maven-central-and-the-tragedy-of-the-commons
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
* fixup! fix(datasource/maven): cache 404s on `maven-metadata.xml` for 24h
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
* cache-down
* perf(datasource/maven): add jitter
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
* sq
---------
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
fix(datasource/maven): cache `.pom` URLs for 7 days
As part of efforts to reduce the general overload on Maven Central[0]
from Renovate, we can look to reduce unnecessary HTTP lookups.
For instance, when a `.pom` URL is retrieved, it is re-checked every 15
minutes in case it has been updated.
This is unnecessary, significantly increases traffic to Maven Central,
and leads to us re-checking constantly for artifacts that are not
updating.
Each time we perform the lookup, we'll use the ETag result to do an
`If-None-Match`, but this still introduces an HTTP call upstream that
counts towards rate limits, and server load on Maven Central.
Instead, we can wrap the lookups for immutable pom.xml results with a 28
day soft cache. In a rare case where a `.pom` URL is updated, we'll then
re-fetch it after 28 days, but until then, it'll be safely returned from
cache.
[0]: https://www.sonatype.com/blog/maven-central-and-the-tragedy-of-the-commons
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
Renames `lookupName` to be `packageName`.
BREAKING CHANGE: Use `packageName` instead of `lookupName` if interacting with Renovate datasources directly.
Moves datasource, manager, platform and versioning code from lib/ into new lib/modules/
BREAKING CHANGE: External tools must update paths to datasource, manager, platform and versioning
Jamie Tanna