mirror of
https://github.com/renovatebot/renovate.git
synced 2026-11-02 02:29:51 +00:00
James Griffiths
1afa293dda
* fix(docker): adjust tag caching for private registries Tag caching for private registries currently has two issues, observed in issues #45249 and #40734: 1. `getTags` has no `cacheable` guard (unlike `getReleases`) 2. `getTags` caches `null` tag results These two combined mean that regardless of `cachePrivatePackages` settings, private registry tag lookups which fail due to transient/auth errors are cached. This results in the cache being "poisoned" by misconfigured jobs, with well-configured jobs unable to recover. This is especially relevant for the hosted app where caching is shared by different tenants. This commit: 1. Aligns `getTags` caching with `getReleases`, ensuring caching behaviour is driven by `cachePrivatePackages`. 2. Prevents `null` tag results being cached, allowing for recovery regardless of `cachePrivatePackages` configuration. Note that I have _not_ applied (2) to `getReleases`, because 1. `getReleases` has a slightly different contract in that it sets `fallback: true` to intentionally cache failures 2. I have not observed any actual failures in `getReleases` due to (20, so it hasn't been empirically proven as requiring a fix 3. `getTags` is internal so is a lower-risk change Fixes - #45249 Relates to - #45249 - #40734 * refactor(docker): undefined for no-auth Previously `getAuthHeaders()` used `null` as an auth-failed sentinel, which was inconsistent with `withCache()`, and led to the original "fix" where nulls were explicitly omitted from the cache. Instead this implements the suggestion from https://github.com/renovatebot/renovate/pull/45348#issuecomment-5340524967 to uniformly return undefined for `getTags` auth errors, which allows the cache to recognise failures itself. Then cacheability ends up being conditional on dockerhub as always. This also rectifies a mismatch between auth-failure signals, where other functions like `_getImageConfig` were returning `undefined` when headers were unavailable. IMPORTANT: once this lands, since there aren't any cache invalidations, the existing issue will still be observed until the cache TTL expires and old `null` entries are evicted. * fix: whoops, sort out the vacuous tests I failed to realise caching isn't automatically orchestrated in these tests. As an eagle-eyed observer [pointed out][1], this meant the tests weren't actually exercising anything. This commit fixes that by adding a new test block which does the cache orchestration using a temp dir cache. I've also opted to add assertions against the mock server's http trace counter directly to ensure the mock really is/isn't being hit. This should also serve to make the tests fail fast if the underlying impl flow (auth probe -> get tags) is adjusted in the future. I've also brought in some caching dependencies into the test file, but those are used elsewhere too (e.g. `withCache` tests). This time, I have actually verified the tests which we'd expect to fail do fail against main. Note that the first 'caches tags for Docker Hub' is expected to pass on both branches (the control test). ``` ❯ getTags caching (3) ✓ caches tags for Docker Hub 21ms 249 MB heap used × does not cache tags for a private registry 20ms 212 MB heap used × does not cache failed lookups when cachePrivatePackages is enabled 15ms 215 MB heap used ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ FAIL lib/modules/datasource/docker/index.spec.ts > modules/datasource/docker/index > getTags caching > does not cache tags for a private registry AssertionError: expected [ '1.0.0' ] to deeply equal [ '1.0.0', '2.0.0' ] - Expected + Received [ "1.0.0", - "2.0.0", ] ❯ lib/modules/datasource/docker/index.spec.ts:3694:72 3692| expect(httpMock.getTrace()).toHaveLength(2); 3693| 3694| expect(await ds.getTags('https://registry.company.com', 'node')).toEqual([ | ^ 3695| '1.0.0', 3696| '2.0.0', ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/4]⎯ FAIL lib/modules/datasource/docker/index.spec.ts > modules/datasource/docker/index > getTags caching > does not cache tags for a private registry Error: *** Unused HTTP mocks *** - GET https://registry.company.com/v2/node/tags/list Requests done: - GET https://registry.company.com/v2/node/tags/list?n=10000 [200] - GET https://registry.company.com/v2/node/tags/list?n=10000 [200] --- Renovate testing strategy requires that every HTTP request has a corresponding mock. This error occurs because some of the created mocks are unused. In most cases, you simply need to remove them. +++ ❯ <test> lib/modules/datasource/docker/index.spec.ts:3675:9 3673| }); 3674| 3675| it('does not cache tags for a private registry', async () => { | ^ 3676| httpMock 3677| .scope('https://registry.company.com/v2') ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/4]⎯ FAIL lib/modules/datasource/docker/index.spec.ts > modules/datasource/docker/index > getTags caching > does not cache failed lookups when cachePrivatePackages is enabled AssertionError: expected null to be undefined - Expected: undefined + Received: null ❯ lib/modules/datasource/docker/index.spec.ts:3717:9 3715| expect( 3716| await ds.getTags('https://registry.company.com', 'node'), 3717| ).toBeUndefined(); | ^ 3718| expect(httpMock.getTrace()).toHaveLength(1); 3719| ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/4]⎯ FAIL lib/modules/datasource/docker/index.spec.ts > modules/datasource/docker/index > getTags caching > does not cache failed lookups when cachePrivatePackages is enabled Error: *** Unused HTTP mocks *** - GET https://registry.company.com/v2/node/tags/list Requests done: - GET https://registry.company.com/v2/node/tags/list?n=10000 [403] --- Renovate testing strategy requires that every HTTP request has a corresponding mock. This error occurs because some of the created mocks are unused. In most cases, you simply need to remove them. +++ ❯ <test> lib/modules/datasource/docker/index.spec.ts:3701:9 3699| }); 3700| 3701| it('does not cache failed lookups when cachePrivatePackages is enabled', async () => { | ^ 3702| GlobalConfig.set({ cachePrivatePackages: true }); 3703| ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[4/4]⎯ Test Files 1 failed (1) Tests 2 failed | 313 passed (315) ``` [1]: https://github.com/renovatebot/renovate/pull/45348#pullrequestreview-5092478059 * oxlint-fix --------- Co-authored-by: Jamie Tanna <jamie.tanna@mend.io>
Datasources
Datasources are used in Renovate primarily to fetch released versions of packages.
Follow the class-based programming style
New datasources must follow the class-based programming style.
Use the java-version datasource as a reference.
Add the datasource to the API in api.ts so that the new datasource is usable.
If you find Unused HTTP mocks errors in the Vitest tests and your mocked URLs are correct, make sure the datasource is correctly registered.
getReleases
The minimum exported interface for a datasource is a function called getReleases that takes a lookup config as input.
The config has:
packageName: the package's full name including scope if present (e.g.@foo/bar)registryUrls: an array of registry URLs to try
getReleases should return an object having:
releases: an array of strings of matched versions. This is the only mandatory fielddeprecationMessage: a string description of the package's deprecation notice, if applicablesourceUrl: a HTTP URL pointing to the source code (for example on GitHub)homepage: a HTTP URL for the package's homepage. Ideally should be empty if the homepage andsourceUrlare the samechangelogUrl: a URL pointing to the package's changelog (could be a Markdown file, for example). If not present then Renovate will search thesourceUrlfor a changelog filetags: an object mapping tag → version, for exampletags: { latest: '3.0.0' }. This is only used by thefollowTagsfunction
getDigest
Datasources that support digests (like Docker digests and Git commit hashes) can export a getDigest function.
The getDigest function has two inputs:
config: the Renovate config for the package being updated, has the same fields asgetReleasesnewValue: the version or value to retrieve the digest for
The getDigest function returns a string output representing the digest value.
If no digest is found, the getDigest function should return null.