Files
renovate/lib/modules/datasource/util.ts
T
Sebastian Poxhofer f0dd862bd5 fix(python): share PyPI index credential resolution (#46127)
* refactor(python): share PyPI index credential resolution

pipenv, poetry, uv, pip-compile and the pypi datasource each resolved
index credentials differently, so a host rule that worked for one Python
manager did not necessarily work for the next. Add
`findPypiIndexCredentials()` in `lib/modules/datasource/pypi/host-rules.ts`,
which strips the URL's own `user:password@` part, runs a `pypi`-scoped
`hostRules.find()`, and only falls back to Google Artifact Registry when
the matched rule has no username and no password. The repeated
`.pkg.dev` test moves into a single `isGoogleArtifactRegistry()`
predicate next to `getGoogleAuthHostRule()`.

Behaviour changes:

- pipenv now supports Google Artifact Registry; previously it only ever
  consulted host rules.
  Spec: manager/pipenv/artifacts > "passes Google Artifact Registry
  credential environment vars"
- pip-compile now supports Google Artifact Registry.
  Spec: manager/pip-compile/common > getRegistryCredVarsFromPackageFiles()
  > "supports Google Artifact Registry"
- pip-compile now matches `hostType: pypi` rules; its find call was
  unscoped, so such rules were skipped.
  Spec: manager/pip-compile/common > getRegistryCredVarsFromPackageFiles()
  > "ignores a rule scoped to another hostType"
- pip-compile now matches a `matchHost` that contains a path; it reduced
  every registry URL to its origin first, and `matchesHost()` compares
  with `href.startsWith()`, so a path-scoped rule could never match.
  Spec: manager/pip-compile/common > getRegistryCredVarsFromPackageFiles()
  > "matches a rule scoped to a path"
- poetry and uv now strip `${USER}:${PASS}@` style placeholders from the
  index URL before matching, so a `matchHost` can match those URLs too.
  Spec: datasource/pypi/host-rules > "strips credentials from the URL
  before matching"
- poetry now falls back to Google Artifact Registry when the matched rule
  carries no username or password; any non-empty rule used to suppress
  the fallback. Its dead unscoped `find({ url })` fallback is gone: a
  `pypi`-scoped find already matches every rule an unscoped one would.
  Spec: manager/poetry/artifacts > "falls back to Google auth when the
  matching rule has no credentials"
- the pypi datasource now prefers host rule credentials over the Google
  token for `*.pkg.dev`; it took the Google token unconditionally, and
  the resulting `authorization` header made `applyHostRule()` skip the
  configured credentials.
  Spec: datasource/pypi/index > getReleases > "prefers host rule
  credentials over Google Auth"

`getRegistryCredEnvVars()` also no longer logs the whole matched host
rule as the log merge object.

* test(pipenv): inline the Google Artifact Registry Pipfile

The fixture was used by a single spec case; inline it as requested in the
review of #46127.

* test(datasource/pypi): cover partial Google Artifact Registry credentials

A host rule that carries only a username or only a password still produces a
Basic header, but nothing exercised those two `?? ''` fallbacks, so the merge
queue failed the global branch threshold.
2026-09-18 10:53:11 +00:00

115 lines
3.7 KiB
TypeScript

import { isString } from '@sindresorhus/is';
import { GoogleAuth } from 'google-auth-library';
import { logger } from '../../logger/index.ts';
import type { HostRule } from '../../types/index.ts';
import { getEnv } from '../../util/env.ts';
import type { HttpResponse } from '../../util/http/types.ts';
import { addSecretForSanitizing } from '../../util/sanitize.ts';
import { parseUrl, resolveSameOriginUrl } from '../../util/url.ts';
const JFROG_ARTIFACTORY_RES_HEADER = 'x-jfrog-version';
/**
* Experimental, per-datasource env var that opts a datasource into following
* server-provided pagination links across origins. Each datasource has its own
* flag so relaxing the restriction for one registry cannot weaken the others.
*/
const crossOriginPaginationEnv: Record<string, string> = {
docker: 'RENOVATE_X_DOCKER_PAGINATION_ALLOW_CROSS_ORIGIN',
nuget: 'RENOVATE_X_NUGET_PAGINATION_ALLOW_CROSS_ORIGIN',
};
/**
* Whether a datasource is opted in to following server-provided pagination
* links across origins, via its experimental `RENOVATE_X_<DATASOURCE>_PAGINATION_ALLOW_CROSS_ORIGIN`
* env var.
*
* By default such links are restricted to the same origin as the request so a
* malicious or compromised registry cannot attempt to redirect our requests in an attempt for Server Side Request Forgery.
*/
export function isCrossOriginPaginationAllowed(datasource: string): boolean {
const envVar = crossOriginPaginationEnv[datasource];
return envVar ? !!getEnv()[envVar] : false;
}
/**
* Resolves a server-provided pagination `next` URL against `baseUrl`, returning
* the URL to follow or `null` if it must not be followed.
*
* When `allowCrossOrigin` is false (the default; see
* {@link isCrossOriginPaginationAllowed}) the link is dropped unless it stays on
* the same origin. When true, the link is followed regardless of origin.
*/
export function resolvePaginationUrl(
baseUrl: string,
nextUrl: string,
allowCrossOrigin: boolean,
): string | null {
if (!allowCrossOrigin) {
return resolveSameOriginUrl(baseUrl, nextUrl);
}
try {
const resolved = new URL(nextUrl, baseUrl).href;
if (parseUrl(baseUrl)?.origin !== parseUrl(resolved)?.origin) {
logger.once.warn(
{ baseUrl, nextUrl },
'Following cross-origin pagination link',
);
}
return resolved;
} catch {
return null;
}
}
export function isArtifactoryServer<T = unknown>(
res: HttpResponse<T> | undefined,
): boolean {
return isString(res?.headers[JFROG_ARTIFACTORY_RES_HEADER]);
}
/**
* Whether the host is a Google Artifact Registry endpoint, which authenticates with a short-lived Google access token instead of static credentials.
*/
export function isGoogleArtifactRegistry(hostname: string): boolean {
return hostname.endsWith('.pkg.dev');
}
export async function getGoogleAuthHostRule(): Promise<HostRule | null> {
try {
const googleAuth: GoogleAuth = new GoogleAuth({
scopes: 'https://www.googleapis.com/auth/cloud-platform',
});
const accessToken = await googleAuth.getAccessToken();
if (accessToken) {
// sanitize token
addSecretForSanitizing(accessToken);
return {
username: 'oauth2accesstoken',
password: accessToken,
};
}
logger.warn(
'Could not retrieve access token using google-auth-library getAccessToken',
);
} catch (err) {
if (err.message?.includes('Could not load the default credentials')) {
return null;
}
throw err;
}
return null;
}
export async function getGoogleAuthToken(): Promise<string | null> {
const rule = await getGoogleAuthHostRule();
if (rule) {
const token = Buffer.from(`${rule.username}:${rule.password}`).toString(
'base64',
);
addSecretForSanitizing(token);
return token;
}
return null;
}