* fix(util/exec): pick tool releases compatible with resolved node version
* refactor(util/exec): pass resolved runtimes as an object
* refactor(util/exec): match tools against their containerbase parent
* test(util/exec): run real constraints filtering for parent versions
* fix(util/http): fall back to `go` host rules for `go-proxy` requests
Prior to 5665570f0b, all requests made by
the `go` Datasource - including those to a Go module proxy - used the
`go` host type.
When we refactored them into classes, this introduced the ability to
specify a `go-proxy` host type, which would only be used for direct
access to the Go module proxy.
As this isn't documented - and may change in the future - we can add a
fallback to the credentials for `hostType: go`, similar to how i.e.
`gitea-tags` and `gitea` do the same.
Co-Authored-By: Claude Opus 5.5 <jamie.tanna+claude-code@mend.io>
* fixup! fix(util/http): fall back to `go` host rules for `go-proxy` requests
Co-Authored-By: Claude Sonnet 5.5 <jamie.tanna+claude-code@mend.io>
* fixup! fix(util/http): fall back to `go` host rules for `go-proxy` requests
---------
Co-authored-by: Claude Opus 5.5 <jamie.tanna+claude-code@mend.io>
* fix(http): share concurrent identical cache provider requests
Identical requests that use a cache provider and run at the same time now
share one in-flight request instead of each fetching from the server.
Callers that join an in-flight request get a deep copy of the response.
This replaces the per-URL mutex, which only covered the cache check and
was released before the fetch finished.
Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
* fix(http): reuse cache providers for npm and typst lookups
The npm and typst datasources now use one module-level cache provider, so
concurrent lookups of the same package share one request. The request key
is only hashed for GET and HEAD requests and includes `throwHttpErrors`.
Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
Co-Authored-By: Claude Sonnet 5.5 <michael.kriese+claude-code@mend.io>
* fix(http): don't share requests that differ in response type
Add `responseType` and `responseBecomesConfig` to the key of shared
requests, and test that requests with different headers aren't shared
and that a shared response is copied for every caller.
Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
Co-Authored-By: Claude Sonnet 5.5 <michael.kriese+claude-code@mend.io>
* fix(http): keep in-flight requests per Http class and copy binary bodies
Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
---------
Co-authored-by: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
* chore(util/exec): add a strong type for Containerbase's known tools
As noted in #41849, we currently don't have a way of enforcing the usage
of a `ContainerbaseToolName` being a tool that Containerbase actually
knows.
To handle this, we can use the new `@containerbase/base` and
`@renovatebot/base-image` packages to see what Tools we currently have
support for, and make sure that our types use the tools supported by
`@renovatebot/base-image` (and therefore `@containerbase/base`).
As well as the Typescript compilation error a user will receive, we can
add a test that provides a better note to the user about how to add a
new tool.
Co-Authored-By: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
* fixup! chore(util/exec): add a strong type for Containerbase's known tools
* fixup! chore(util/exec): add a strong type for Containerbase's known tools
So we can get better confidence that we're not clashing with type names
between Containerbase tools and non-tool constraints.
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
---------
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
`install-tool` installs a version and also makes it the active one. The
memo skipped `install-tool` for any tool version already installed in the
run, so after A -> B -> A the last command ran on B. Track the active
version per tool instead and skip only when the requested version is
already active.
Co-authored-by: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
* feat(datasource): report the datasource which served a lookup
A datasource which delegates a lookup to another one, such as the go datasource to github-tags, returns the delegate's id as effectiveDatasource. The lookup copies it onto the update result, so that templates, matchJsonata and logs can tell which path answered.
* Update lib/util/template/index.ts
* style(template): format the effectiveDatasource description
As it's not an official tool Renovate supports, and was added for
Mend-internal usage.
However, as we work towards #41849, we should make sure that we don't
document a tool that isn't used by Renovate.
A deep copy of a `Uint8Array` body used `subarray()`, which returns a view
of the same memory, so changing the original also changed the copy. Use
`slice()`, which copies the bytes.
Co-authored-by: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
The Redis package cache client had no `error` listener. node-redis emits
connection errors, such as an idle socket closed by the server, as
`error` events and reconnects on its own, but an unhandled `error` event
exits the process. Log them instead: once as a warning, every time at
debug level.
Co-authored-by: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
Responses served by the cache provider's `bypassServer` no longer wait
for a `maxRequestsPerSecond` throttle slot or a concurrency queue slot.
Only requests that reach the server are throttled and queued.
Co-authored-by: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
As we're now managing them via a `afterEach` in `test/setup.ts`, we can
remove any entries in tests themselves.
We don't update overrides of an outer hook or cases where there are more
explicit `clear()`s.
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
Instead of needing to always make sure that we've cleared host rules in
tests that use them - or knowing when they're touched indirectly - we
can make sure that we have an `afterEach` that will clear our Host
Rules.
We can do this by making sure that we have a top-level `afterEach` that
handles this, which needs a dynamic import to avoid eagerly importing
and caching it, which would leave stale unmocked bindings for a given
file's test run.
We also need to make sure that `private-key.spec.ts`'s `sanitize.ts`
mock is updated to also stub `clearRepoSanitizedSecretsList`, since
`hostRules.clear()` now calls.
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
In a similar way to how `allowedEnv` works, we should allow a
self-hosted admin to specify whichever headers they wish in their
`hostRules` - whether in their global config, a `repositories[]` entry,
or a preset a `repositories[]` entry extends.
Previously, this would require a self-hosted admin to add any header
they wished to set to `allowedHeaders`, which would also allow any of
their users' repositories to set that same header.
We can align this behaviour with `allowedEnv`, which makes sure that
the admin's own `headers` are always applied regardless of
`allowedHeaders`, while a repository's own `hostRules` - and those of
any preset it extends - remain constrained by it as before.
We make sure to use an explicit `[]` to make it clear when there are no
headers to allow, opposed to using something less clear like
`undefined`.
We do not currently allow an `extends` (without a `repositories[]`
entry) to bypass, and they'll be followed-up with in #45670.
Co-authored-by: Claude Opus 5 <jamie.tanna+claude-code@mend.io>
Exposes the isBreaking flag that lookup already computes per upgrade as a
native packageRules matcher, so it can be combined with matchJsonata
selectors from extended presets instead of being inlined into them.
Exposes Renovate's string pattern matching (globs, regex, negation) to
matchJsonata expressions, so a single expression can OR conditions on
different fields such as packageName and registryUrls.
Every datasource that caches its releases hand-wrote the same
`withCache({ namespace: `datasource-${id}`, ... }, () => this._getReleases(config))`
wrapper, repeating the namespace literal and using underscore-prefixed
method names that `docs/development/best-practices.md` discourages.
Add `Datasource.cacheNamespace` (derived from the datasource id) and a
protected `Datasource.cached()` helper that defaults `namespace` to it,
exporting `CachedOptions` from `with-cache.ts`. Migrate 29 datasources
onto the helper, dropping their namespace literals, the static
`cacheNamespace` field in `github-digest`, and renaming the wrapped
`_getX` methods to `fetchX`.
Behaviour change: `golang-version` used the constant cache key `'all'`
while declaring `customRegistrySupport = true`, so a custom registryUrl
shared the cache entry with the default one. The key is now
`${config.registryUrl}`, which invalidates existing entries once.
`lib/util/http/auth.ts` and `lib/util/http/host-rules.ts` were the last
readers of the `*_API_USING_HOST_TYPES` constants, which the previous
commits left as aliases of `PLATFORM_FAMILIES`. Both now read the table
directly and the aliases are deleted, so each family's host types are
declared in exactly one place.
Also covers the `forgejo` fallback in `findMatchingRule()`, which had no
test: the shared setup only registers a `gitea` host rule, so the
forgejo branch was never entered.
Behaviour changes: none. The aliases pointed at the same arrays.
* refactor(constants): add a platform family capability table
Every consumer of `detectPlatform` carried its own mapping from the
detected platform to a tags datasource or a web UI path, and the seven
`*_API_USING_HOST_TYPES` arrays were checked one by one to produce that
platform in the first place.
Introduce `PLATFORM_FAMILIES`, one record keyed by family holding the
host types, the hostnames which identify the family outright, the tags
datasource, the API base URL derivation and the web UI path to a
directory at the default branch. A family is a group of platforms, so
the ids are checked against `PlatformId`: a family which is not a
platform is a compile error. Not every platform is a family, because
codecommit, gerrit, local and scm-manager are never recognized from a
URL.
`detectPlatform` loops the table for both its hostname and its hostType
matching, and the consumers look their answers up:
- `github-actions` and `pre-commit` read `tagsDatasource` instead of
importing the tags datasource classes.
- The PR body reads `webDirPath` instead of special-casing Bitbucket and
Bitbucket Server.
`apiBaseUrl` has no consumer yet; the changelog sources pick it up in
the next commit.
`lib/util/http/auth.ts` and `lib/util/http/host-rules.ts` still read six
of the old arrays by name, so those stay as aliases of the table and are
migrated in a follow-up. `AZURE_API_USING_HOST_TYPES` had no other
consumer and is dropped.
Behaviour changes: none. `detectPlatform` now tests every known hostname
before its name heuristics instead of interleaving the two, which is
equivalent because the hostname lists are pairwise disjoint and no known
hostname contains the name of another family.
* Apply suggestion from @secustor
* fix(datasource/crate): only query the crates.io web API for crates.io
Private sparse registries such as Artifactory or CodeArtifact advertise an
`api` URL in their `config.json` for publishing, but do not implement the
crates.io-only read endpoints `api/v1/crates/<name>` and
`api/v1/crates/<name>/<version>`. Since #41536 Renovate called both for
every registry with an `api` field, producing 404s and a
"Release interceptor failed" warning on each run.
Restore the crates.io flavor guard for the crate metadata lookup and the
release timestamp lookup. Reading `config.json` from a cloned registry is
now unreachable, so drop that branch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs(datasource/crate): cite the registry web API specification in the guards
* fix(datasource/crate): probe the registry web API once instead of guarding on flavor
Keep the crates.io read endpoints for registries that implement them.
Registries whose config.json points `api` at crates.io, such as index
mirrors, are used without probing. Any other registry is probed on the
first crate; a 404 for a crate that the index just returned means the
endpoint does not exist, so the registry is remembered as unsupported in
the memory cache and, for a day, in the package cache. The release
timestamp lookup handles a 404 the same way instead of surfacing the
generic "Release interceptor failed" warning.
Restores the cloned config.json branch, since cloned registries can now
advertise an API again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(datasource/crate): key the unsupported web API marker by the api URL
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
As part of follow-up changes in #45706, we want the `apk` / `apt`
parsing in the `dockerfile` manager to attempt to determine which
distribution - and therefore registry - to look up dependencies for.
When there is an image that we don't know how to resolve, we want to
treat them as a `skipReason: unknown-registry`.
However, this doesn't provide any way for the user to say "I know which
registry you should look up updates for".
To handle this, we can treat `unknown-registry` differently, allowing it
to un-skip itself when there are any `registryUrls`s defined.
This allows for general usage, where if Renovate doesn't know how to
interact with a given registry, but a user does, then the user can
override the registry with a `packageRules` entry.
We continue skipping a dependency when `unknown-registry` is set
alongside a dependency's `registryUrl`.
We add this un-skipping logic into `applyPackageRules()`, but need to
make sure that we also gate it inside `lookup()`, so a dependency that's
passed through `applyPackageRules()` is re-checked to ensure that the
check is enforced.
Co-authored-by: Claude Opus 5 <jamie.tanna+claude-code@mend.io>
feat(host-rules): ignore URL credentials when matching matchHost
`matchesHost()` compared the raw href of the searched URL against
`matchHost`, so a URL carrying `user:pass@` userinfo (often a literal
placeholder such as `${USER}:${PASS}@`) never matched a `matchHost`
without credentials, including path-scoped ones. Callers worked around
this by stripping `username`/`password` off the URL by hand before
calling `hostRules.find()`.
- Add `stripUrlCredentials()` to `lib/util/url.ts`, returning the URL's
`href` without userinfo (or `null` if unparseable), covered by
`lib/util/url.spec.ts`.
- `matchesHost()` in `lib/util/host-rules.ts` now strips credentials
from the searched URL before comparing it against an HTTP(S)
`matchHost`, so `hostRules.find()` ignores userinfo on every caller's
URL. Covered by new cases in `lib/util/host-rules.spec.ts` (a
`matchHost` rule matches a URL carrying `user:pass@`, including a
path-scoped `matchHost`).
- Remove the hand-rolled strip in
`lib/modules/datasource/pypi/host-rules.ts`'s
`findPypiIndexCredentials()`, now redundant since `find()` does it.
`lib/modules/datasource/pypi/host-rules.spec.ts` already covers this
behaviour and needed no changes.
This is strictly additive: more URLs match `matchHost` than before,
and nothing that matched before stops matching.
* test(util/git): skip the per-test clone for tests that need no repo
The spec's `beforeEach` rebuilds a bare clone and calls `initRepo()`
before every test, which costs about 0.7s each. Eight tests never touch
a repository at all -- `createSimpleGit`, `gitRetry`, `validateGitVersion`
and `getUrl` -- yet each paid for that clone. They now sit in a second
root describe with no clone in it, so the file drops from about 129s to
about 121s with the same 144 tests.
The two blocks that are pure functions are marked `describe.concurrent`.
The other two stay sequential: `createSimpleGit` spies on a prototype and
sets the custom env, and `gitRetry` stubs `NODE_ENV`.
This was the outcome of trying `it.concurrent` across the file. It cannot
help there, for two reasons stacked on each other. Vitest still runs
`beforeEach` for every concurrent test, so concurrency does not remove
the clone, it only interleaves several of them. And those interleaved
clones all write the same module singleton (`config`, `git`,
`gitInitialized`) and `GlobalConfig.localDir`, so one test's `initRepo()`
is overwritten by another's before its body runs. The only lever that
works is not running the clone for tests that do not need it.
Co-Authored-By: Claude Fable 5.1 <michael.kriese+claude-code@mend.io>
* test(util/git): share one clone across the read-only tests
The spec now has one root describe with three nested groups, split by how
much git setup a test needs:
- `without a repository`: pure functions, no setup at all.
- `shared clone`: 32 read-only tests. One bare clone of `base` and one
`initRepo()` in `beforeAll`, so each test costs only the git commands
it runs itself.
- `clone per test`: the tests that commit, push, check out or re-init.
They keep the fresh clone and `initRepo()` per test.
The `base` repository moves up to the root describe so both clone groups
can use it. Groups run in file order, and some of the writing tests change
`base` without restoring it, so the shared-clone group has to stay ahead.
`isBranchModified()` stays with the writing tests although it never touches
the repository: it caches per-branch results in `config.branchIsModified`,
which only `initRepo()` clears, and its tests query the same branch with
different ignored authors.
Three describes are split by test: `getFileList`, `getBranchUpdateDate`
and `diffCommitTree` each have one test that writes, which stays behind
under the same describe name with a pointer to the read-only half.
Same 144 tests; the file drops from about 121s to about 100s.
Co-Authored-By: Claude Fable 5.1 <michael.kriese+claude-code@mend.io>
* test(util/git): keep the 30s test timeout for every test
The `{ timeout: 30000 }` option used to sit on the single root describe
and so covered every test in the file. The split left it on the
`clone per test` group only, which put the 32 shared-clone tests on the
5s default even though they still run real git commands. CI is a lot
slower than a local machine, so the option moves up to the root describe
again; nested describes inherit it.
Hooks are unaffected either way: they run under `hookTimeout`, not the
suite option, and the shared `beforeAll` does the same work the old
`beforeEach` did.
Co-Authored-By: Claude Fable 5.1 <michael.kriese+claude-code@mend.io>
* test(util/git): reuse one clone for the local-only writers
A fourth nested group, `reused clone`, holds the 36 tests that write to
the local checkout but never to `origin` or `base`: `checkoutBranch`,
`isBranchModified`, `mergeToLocal`, `isFileModeEnabled`,
`getCommitMessages`, `setGitAuthor`, `getRepoStatus`, `fetchRevSpec` and
single tests of `getBranchUpdateDate`, `mergeBranch`, `deleteBranch` and
`diffCommitTree`.
The group clones once, but `initRepo()` and `syncGit()` run again before
every test. On an existing checkout `syncGit()` takes its fetch path:
`fetch --prune`, `reset --hard`, `checkout` of the default branch,
`clean -fd` and deletion of every other local branch. With the fresh
`config` from `initRepo()` that undoes everything such a test can do to
the repository or the module singleton, including the per-branch
`branchIsModified` cache that kept `isBranchModified()` out of the
shared-clone group. Local git config is the one thing it leaves alone, so
the `beforeEach` re-sets the author that `setGitAuthor()` writes.
That reset costs about 0.35s per test against about 0.87s for a clone, so
the file drops from about 100s to about 87s. Same 144 tests.
Co-Authored-By: Claude Fable 5.1 <michael.kriese+claude-code@mend.io>
* test(util/git): trim the describe comments to the essentials
Review feedback: the block comments read more like a narration of how
the layout came to be than like documentation. Each now states only
what a group's setup is, what a test may or may not do to belong there,
and the ordering constraint about `base`. The explanation of the
`isBranchModified()` cache is gone too, since that test now runs with a
fresh `initRepo()` in the reused-clone group.
Comments only, no test changes.
Co-Authored-By: Claude Fable 5.1 <michael.kriese+claude-code@mend.io>
* test(util/git): move isBranchModified() into the shared-clone group
`isBranchModified()` only reads the repository, but it stores results in
`config.branchIsModified`, which only `initRepo()` cleared. That kept its
16 tests in the reused-clone group at about 0.35s each.
A new `clearBranchIsModifiedCache()` helper, marked `@internal`, clears
that store. The shared-clone `beforeEach` calls it along with the cheap
config setters (`setUserRepoConfig`, `setGitAuthor`,
`setPlatformIgnoredAuthors`), so those tests now share one `initRepo()`
and cost a few milliseconds each. The file drops from about 87s to about
83s.
A dynamic import with `vi.resetModules()` was considered instead: a fresh
module instance has no `config` and `gitInitialized = false`, so every
test would need its own `initRepo()` and `syncGit()` on it, which is the
reused-clone cost again plus re-importing the module graph.
Co-Authored-By: Claude Fable 5.1 <michael.kriese+claude-code@mend.io>
---------
Co-authored-by: Claude Fable 5.1 <michael.kriese+claude-code@mend.io>
* feat(config): add minimumReleaseAgeBuffer option
Adds a new config option which extends the time Renovate waits before
suggesting an update by a percentage of minimumReleaseAge. This gives
companion packages that are published after the main package (e.g.
platform-specific binaries) time to pass the package manager's own
release age cooldown, avoiding lock file artifact errors.
* feat(config): make minimumReleaseAgeBuffer a duration with 10 minutes default
* feat(config): raise minimumReleaseAgeBuffer default to 30 minutes
Add `collectFileChanges()` next to the other repository helpers, plus a
thin `fileChangesToArtifactResults()` in `lib/modules/manager/util.ts`,
and migrate the managers which re-implemented the same status loops:
apm, cocoapods, composer, copier, gomod, helmv3, jsonnet-bundler,
kustomize and vendir.
Each site keeps its own status bucket selection and path filter, so the
emitted artifacts are unchanged. Two dead pieces of logic are dropped:
vendir's `vendorDir || isFileInVendorDir` condition, which was always
true, and kustomize's optional chaining on a non-nullable status, which
only ever mattered for an unset test mock (now set explicitly).
* fix(util/http): make the legacy `headers` getter configurable
Every other property defined on `HttpError.prototype` in this file is
`configurable`, but `headers` was not. That makes the module fail with
`TypeError: Cannot redefine property: headers` whenever a test resets the
module registry and re-imports anything that pulls in `got.ts`.
Co-authored-by: Claude Opus 5 <michael.kriese+claude-code@mend.io>
* test(manager/pre-commit): cover the suppress-warning env var
`defaultConfig.prBodyNotes` drops the unsupported-manager warning when
`RENOVATE_X_SUPPRESS_PRE_COMMIT_WARNING` is set. Since the value is
computed at module load, the new spec resets the module registry and
re-imports the manager with the variable stubbed.
The `istanbul ignore next` on that arm is dropped because the branch is
now exercised.
Co-authored-by: Claude Opus 5 <michael.kriese+claude-code@mend.io>
* refactor(manager): remove unreachable fallbacks
Drop three defensive fallbacks that can never be taken:
- `docker-compose`: `config.services ?? config` always yields an object,
so the `|| {}` guard behind it is dead
- `git-submodules`: `git.raw()` resolves to a string, so the `?? ''`
never applies
- `maven`: the only site that builds a `MavenProp` always passes a
`packageFile` string, so narrow the type and drop the `?? undefined`
Co-authored-by: Claude Opus 5 <michael.kriese+claude-code@mend.io>
---------
Co-authored-by: Claude Opus 5 <michael.kriese+claude-code@mend.io>
* fix(datasource/go): cache the publication time of each version
The Go proxy defines that the `.info` API call is for a specific release
version of a module, this file should never be modified after it's been
created.
However, because we previously had a 30m TTL on these entries, a
repository that runs infrequently can lead to re-fetching data that
doesn't ever update.
We can fix this by introducing a separate cache namespace for the
module's creation timestamp (`Time` in the `.info` API call).
We can make sure that:
- we always merge, not replace, the entries, as the `major < 2` filter
in `_getReleases` means a run may only see a subset of a module's
versions, and writing back only what we saw would evict the rest
- we re-read immediately before writing, as `foo` and `foo/v2` are
separate lookups which both walk the `v2` module, so one can overwrite
the other's timestamps
- the key is the proxy which served the data, minus any credentials,
since a private Artifactory `GOPROXY` is a different universe of
timestamps to `proxy.golang.org`
- only the timestamps we successfully fetched are stored, so a proxy
which serves no `Time` doesn't have that recorded for 100 days
- it uses its own namespace, so that a `cacheTtlOverride` on the release
cache doesn't drag the immutable data down with it
Co-authored-by: Claude Opus 5 <jamie.tanna+claude-code@mend.io>
* fixup! fix(datasource/go): cache the publication time of each version
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
* fixup! fixup! fix(datasource/go): cache the publication time of each version
Co-Authored-By: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
---------
Co-authored-by: Claude Opus 5 <jamie.tanna+claude-code@mend.io>
When running Renovate with `binarySource=install` on a repository with
many branches, the amount of time running `install-tool` - on already
installed tools - can take a significant percentage of the overall run.
For instance, on one repository we have instrumented with OpenTelemetry,
we see:
- initial `install-tool golang`: 8.7s
- follow-up `install-tool golang`: median 635ms, max 2.4s
However, because we run `install-tool golang` 640 times, this adds up to
24.8/29.2m (85%) of the repo's run time, which is an overwhelming
percentage when we're not even re-installing the tool.
To handle this more effectively, we can add an in-memory cache of all
the tools (and resolved versions) that we've installed in a given run.
We can't do this with `binarySource=docker`, as it has a fresh container
each time.
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
Michael Kriese