Files
renovate/lib/modules/datasource/pypi/schema.ts
Felipe Santos 2606ae2bd3 feat(pypi): add support for JSON-based Simple API (PEP 691) (#44222)
* feat(pypi): add support for JSON-based Simple API (PEP 691)

* Fix some stuff

* Clean some things up

* Fix coverage

* Simplify JSON parse

* Remove useless tests

* Align terminology

* Incorporate some changes from PR #44182

Co-authored-by: Andreas Hippler <andreas.hippler@goto.com>

* Use isNonEmptyString to filter requires_python

* Address review feedback

* Warn once actually

* fixup! review comments

Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>

* fixup! fixup! review comments

Co-authored-by: Claude Opus 5 <jamie.tanna+claude-code@mend.io>

* fixup! review comments

As the list of status codes we retry on was based on an unverified
claim that Artifactory, Nexus and devpi answer a rejected `Accept`
header with `400`/`403`/`415`, we should only retry on `406`, which is
what RFC 9110 specifies for a request whose `Accept` cannot be
satisfied.

Our `Accept` header ends with `text/html; q=0.01`, so any conforming
registry can satisfy it, `415` refers to the request body's
`Content-Type` rather than `Accept`, and retrying on `403` doubles the
requests for a registry we're not authenticated against.

The unwrapping of `ExternalHostError` is unaffected, as that's needed
for the `abortOnError` case regardless of which status codes we retry.

Co-Authored-By: Claude Opus 5 <claude-code@jamietanna.co.uk>

* fixup! review comments

As the note about `upload-time` sat in the paragraph about the Simple
API, it read as if `minimumReleaseAge` needs a registry which supports
the JSON-based Simple API (PEP 691).

That's only true of the Simple API. The JSON API has always exposed
`upload_time`, and as the default registry of `https://pypi.org/pypi/`
doesn't end in `/simple/`, that's the lookup most users get, so
`minimumReleaseAge` already works for them.

We can split the two apart, so that the guarantee for each API is
stated separately.

Co-Authored-By: Claude Opus 5 <claude-code@jamietanna.co.uk>

* reword

* fixup! review comments

As the JSON-based Simple API's release timestamps were only covered by
hand-written responses, we didn't have the equivalent of the real world
coverage added for the JSON API in
e942788539.

We can add a `numpy` fixture from PyPI's JSON-based Simple API, trimmed
to 1.5.0 and 1.5.1, so we assert against a response a registry has
actually served. 1.5.1 is the interesting case, as it was released in
2010 but gained a wheel in 2014 which PyPI lists first.

This also covers the `upload-time` format the Simple API uses, of
`2010-11-18T14:16:58.801332Z`, rather than the JSON API's
`2010-11-18T14:16:58`, so that we know `asTimestamp` normalises both.

Unlike the JSON API, there's no previous behaviour to characterise
first, as before these changes the Simple API exposed no timestamps at
all.

Co-Authored-By: Claude Opus 5 <claude-code@jamietanna.co.uk>

* fixup! fixup! review comments

* fixups

* fixups

Co-authored-by: Claude Opus 5 <jamie.tanna+claude-code@mend.io>

* fixup! comments

Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>

---------

Co-authored-by: Andreas Hippler <andreas.hippler@goto.com>
Co-authored-by: Jamie Tanna <jamie.tanna@mend.io>
Co-authored-by: Claude Sonnet 5 <jamie.tanna+claude-code@mend.io>
Co-authored-by: Claude Opus 5 <claude-code@jamietanna.co.uk>
2026-08-18 09:03:12 +00:00

62 lines
2.0 KiB
TypeScript

import { isTruthy } from '@sindresorhus/is';
import { z } from 'zod/v4';
import { DeepNullish, LooseArray } from '../../../util/schema-utils/index.ts';
export const PypiRelease = DeepNullish(
z.object({
requires_python: z.string().optional(),
upload_time: z.string().optional(),
yanked: z.boolean().optional().default(false),
}),
);
export type PypiRelease = z.infer<typeof PypiRelease>;
export const PypiResponse = DeepNullish(
z.object({
info: z
.object({
name: z.string().optional(),
home_page: z.string().optional(),
project_urls: z.record(z.string(), z.string().optional()).optional(),
})
.optional(),
releases: z.record(z.string(), z.array(PypiRelease)).optional(),
}),
);
export type PypiResponse = z.infer<typeof PypiResponse>;
// JSON-based Simple API (PEP 691) schemas
// https://peps.python.org/pep-0691/
export const PypiSimpleFile = DeepNullish(
z.object({
filename: z.string(),
'requires-python': z.string().optional(),
yanked: z.union([z.boolean(), z.string()]).optional().default(false),
// `upload-time` is specified by PEP 700, which mandates an ISO 8601 string and lets a server omit the key even at `api-version` 1.1.
// A string which `asTimestamp` cannot parse downstream degrades to a missing timestamp; a value of any other type fails validation, and `LooseArray` then drops the file entry.
'upload-time': z.string().optional(),
}),
).transform(
({
filename,
'requires-python': requires_python,
yanked,
'upload-time': upload_time,
}) => ({
filename,
requires_python,
// Per the Simple API spec, `yanked` is yanked when truthy
// https://packaging.python.org/en/latest/specifications/simple-repository-api/#json-based-simple-api-for-python-package-indexes
yanked: isTruthy(yanked),
upload_time,
}),
);
export type PypiSimpleFile = z.infer<typeof PypiSimpleFile>;
export const PypiSimpleResponse = z.object({
files: LooseArray(PypiSimpleFile),
});
export type PypiSimpleResponse = z.infer<typeof PypiSimpleResponse>;