In 01bcd4fd43 we introduced knowledge of
the Maven Central mirror, but haven't taken it into account when looking
at rate limit errors.
To prevent this in the future, we can introduce a new helper function,
`isMavenCentral`, for other usage.
* fix(datasource/maven): cache 404s on `maven-metadata.xml` for 24h
As part of efforts to reduce the general overload on Maven Central[0]
from Renovate, we can look to reduce unnecessary HTTP lookups.
The Maven Datasource will use the `registryStrategy=merge`, which leads
to looking up a package in each of our registries, and merging them if
present.
However, this means that when looking up a package that /will only ever
be in a given registry/ (such as in
01bcd4fd43) we're hitting other registries
with packages they are unlikely to have.
This is unnecessary, significantly increases traffic to Maven Central,
and leads to us re-fetching the same 404s which count towards rate
limits, and increased, unnecessary, server load on Maven Central.
We can instead introduce a new cache for 404s for the
`maven-metadata.xml`, for 24 hours, which should reduce traffic to Maven
Central (and other configured registries) if they only have a subset of
dependencies.
[0]: https://www.sonatype.com/blog/maven-central-and-the-tragedy-of-the-commons
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
* fixup! fix(datasource/maven): cache 404s on `maven-metadata.xml` for 24h
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
* cache-down
* perf(datasource/maven): add jitter
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
* sq
---------
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
fix(datasource/maven): cache `.pom` URLs for 7 days
As part of efforts to reduce the general overload on Maven Central[0]
from Renovate, we can look to reduce unnecessary HTTP lookups.
For instance, when a `.pom` URL is retrieved, it is re-checked every 15
minutes in case it has been updated.
This is unnecessary, significantly increases traffic to Maven Central,
and leads to us re-checking constantly for artifacts that are not
updating.
Each time we perform the lookup, we'll use the ETag result to do an
`If-None-Match`, but this still introduces an HTTP call upstream that
counts towards rate limits, and server load on Maven Central.
Instead, we can wrap the lookups for immutable pom.xml results with a 28
day soft cache. In a rare case where a `.pom` URL is updated, we'll then
re-fetch it after 28 days, but until then, it'll be safely returned from
cache.
[0]: https://www.sonatype.com/blog/maven-central-and-the-tragedy-of-the-commons
Co-authored-by: Claude Sonnet 4.6 <jamie.tanna+claude-code@mend.io>
* test(noNewURL): add new rule enforcing parseUrl() from 'lib/util/url.ts'
* replace newURL with parseUrl
* remove non-null assertions marks if appropiate
* test: add additional tests
* test: add additional tests and ignore where it makes sense
* implement suggestions
* test: further improve coverage
* apply suggestion
Sebastian Poxhofer