mirror of
https://github.com/renovatebot/renovate.git
synced 2026-11-01 18:19:51 +00:00
Jamie Tanna
128f446e08
* docs: skip validation on JSON files that aren't Renovate config
As part of future changes, we'll enable validation on all JSON code
snippets to run Config Validation.
Before we do that, we should opt-out JSON snippets that aren't anything
related to Renovate config.
* docs: don't validate invalid Renovate config
As it's intentionally referencing a manager that doesn't exist.
* docs: don't validate partial Renovate config
As part of future changes, we'll enable validation on all JSON code
snippets to run Config Validation.
There are several code snippets we have that are part of a Renovate JSON,
but it doesn't make sense to convert these to a fully passing config, as
it'd reduce clarity on what we're trying to explain.
* docs: convert `baseBranchPatterns` snippets to code blocks
So we can validate them.
* docs: make sure that `datasourceTemplate` is set
So this passes validation.
* docs: clarify that `force` is global config
As it may not be clear, so we can make it extra clear with a `.js` file,
and note that it's global self-hosted config.
* docs: make sure `packageRules` is an array
* docs: `matchBaseBranches` needs `baseBranchPatterns` to be set
* docs: correct typo for `prBodyColumns`
* docs: mark global config snippets as such
As part of future changes, we'll use this to detect whether it's global
self-hosted configuration.
* docs: make sure `packageRules` wraps its children
* docs: add `allowedHeaders`
* docs: ignore deprecation warnings
As part of future changes, we'll use this to note that we don't care
about the validation warnings.
* docs: make sure `customManagers` wraps its children
* docs: make sure `fleet` wraps its children
* docs: validate JSON snippets are valid Renovate config
To make sure that we're always presenting valid JSON configuration to
our users, we can validate them as part of `check-fenced-code`.
As well as configuration errors we can warn for warnings, but allow code
blocks to opt-out, for instance when there's a known deprecation
affecting the code block.
We need to make sure to set `allowedEnv` for various code snippets that
reference environment variables.
* docs(development): document fenced code block checks
* docs: wrap `configType` and `ignoreConfigWarnings` in `{ ... }`
As otherwise we see build failures i.e.
WARNING - Doc file 'modules/platform/gitlab/index.md' contains a link
'../../../self-hosted-configuration.md#autodiscoverrepoorder', but the doc
'self-hosted-configuration.md' does not contain an anchor '#autodiscoverrepoorder'.
This needs to be wrapped alongside all other options like `title`.
* fixup! docs: wrap `configType` and `ignoreConfigWarnings` in `{ ... }`
477 lines
13 KiB
Markdown
477 lines
13 KiB
Markdown
This `custom` datasource allows requesting version data from generic HTTP(S) endpoints.
|
|
|
|
## Usage
|
|
|
|
The `customDatasources` option takes a record of `customDatasource` configs.
|
|
This example shows how to update the `k3s.version` file with a custom datasource and a [regex](../../manager/regex/index.md) custom manager:
|
|
|
|
Options:
|
|
|
|
| option | default | description |
|
|
| -------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| defaultRegistryUrlTemplate | `""` | URL used if no `registryUrl` is provided when looking up new releases |
|
|
| format | `"json"` | format used by the API. Available values are: `html`, `json`, `plain`, `toml`, `yaml` |
|
|
| transformTemplates | `[]` | [JSONata rules](https://docs.jsonata.org/simple) to transform the API output. Each rule will be evaluated after another and the result will be used as input to the next |
|
|
|
|
!!! tip
|
|
Use [JSONata Exerciser](https://try.jsonata.org/) to test your JSONata rules.
|
|
|
|
Available template variables:
|
|
|
|
- `packageName`
|
|
- `currentValue`
|
|
|
|
```json
|
|
{
|
|
"customManagers": [
|
|
{
|
|
"customType": "regex",
|
|
"managerFilePatterns": ["/k3s.version/"],
|
|
"matchStrings": ["(?<currentValue>\\S+)"],
|
|
"depNameTemplate": "k3s",
|
|
"versioningTemplate": "semver-coerced",
|
|
"datasourceTemplate": "custom.k3s"
|
|
}
|
|
],
|
|
"customDatasources": {
|
|
"k3s": {
|
|
"defaultRegistryUrlTemplate": "https://update.k3s.io/v1-release/channels",
|
|
"transformTemplates": [
|
|
"{\"releases\":[{\"version\": $$.(data[id = 'stable'].latest),\"sourceUrl\":\"https://github.com/k3s-io/k3s\",\"changelogUrl\":$join([\"https://github.com/k3s-io/k3s/releases/tag/\",data[id = 'stable'].latest])}],\"sourceUrl\": \"https://github.com/k3s-io/k3s\",\"homepage\": \"https://k3s.io/\"}"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
After all transformations, the resulting JSON must match one of these formats:
|
|
|
|
Minimal-supported object:
|
|
|
|
```json {configType=none}
|
|
{
|
|
"releases": [
|
|
{
|
|
"version": "v1.1.0"
|
|
},
|
|
{
|
|
"version": "v1.2.0"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
All available options:
|
|
|
|
```json {configType=none}
|
|
{
|
|
"releases": [
|
|
{
|
|
"version": "v1.0.0",
|
|
"isDeprecated": true,
|
|
"releaseTimestamp": "2022-12-24T18:21Z",
|
|
"changelogUrl": "https://github.com/demo-org/demo/blob/main/CHANGELOG.md#v0710",
|
|
"sourceUrl": "https://github.com/demo-org/demo",
|
|
"sourceDirectory": "monorepo/folder",
|
|
"digest": "c667f758f9e46e1d8111698e8d3a181c0b10f430",
|
|
"isStable": true
|
|
}
|
|
],
|
|
"sourceUrl": "https://github.com/demo-org/demo",
|
|
"sourceDirectory": "monorepo/folder",
|
|
"changelogUrl": "https://github.com/demo-org/demo/blob/main/CHANGELOG.md",
|
|
"homepage": "https://demo.org"
|
|
}
|
|
```
|
|
|
|
### Debugging
|
|
|
|
Renovate writes tracing logs entries before transformation.
|
|
If Renovate finds an unexpected dataformat, it also writes a tracing log _after_ transformation.
|
|
|
|
#### Getting trace level logs on hosted app
|
|
|
|
If you use the Mend Renovate app, use the [`logLevelRemap` config option](../../../configuration-options.md#loglevelremap) to get the trace log.
|
|
|
|
```json title="Getting trace logs from the Mend Renovate app"
|
|
{
|
|
"logLevelRemap": [
|
|
{
|
|
"matchMessage": "/^Custom datasource/",
|
|
"newLogLevel": "info"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
#### Getting trace level logs when self-hosting
|
|
|
|
If you self-host Renovate, follow these steps to get the trace logs:
|
|
|
|
1. Set the [`LOG_FILE_LEVEL` env var](../../../config-overview.md#logging-variables) to `trace`
|
|
1. Run Renovate in [`dryRun` mode](../../../self-hosted-configuration.md#dryrun)
|
|
|
|
### Formats
|
|
|
|
#### JSON
|
|
|
|
If `json` is used processing works as described above.
|
|
The returned body will be directly interpreted as JSON and forwarded to the transformation rules.
|
|
|
|
#### Plain
|
|
|
|
If the format is set to `plain`, Renovate will call the HTTP endpoint with the `Accept` header value `text/plain`.
|
|
The body of the response will be treated as plain text and will be converted into JSON.
|
|
|
|
Suppose the body of the HTTP response is as follows:
|
|
|
|
```
|
|
1.0.0
|
|
2.0.0
|
|
3.0.0
|
|
```
|
|
|
|
When Renovate receives this response with the `plain` format, it will convert it into the following:
|
|
|
|
```json {configType=none}
|
|
{
|
|
"releases": [
|
|
{
|
|
"version": "1.0.0"
|
|
},
|
|
{
|
|
"version": "2.0.0"
|
|
},
|
|
{
|
|
"version": "3.0.0"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
After the conversion, any `jsonata` rules defined in the `transformTemplates` section will be applied as usual to further process the JSON data.
|
|
|
|
#### Yaml
|
|
|
|
If `yaml` is used, response is parsed and converted into JSON for further processing.
|
|
|
|
Suppose the body of the HTTP response is as follows:
|
|
|
|
```yaml
|
|
releases:
|
|
- version: 1.0.0
|
|
- version: 2.0.0
|
|
- version: 3.0.0
|
|
```
|
|
|
|
When Renovate receives this response with the `yaml` format, it will convert it into the following:
|
|
|
|
```json {configType=none}
|
|
{
|
|
"releases": [
|
|
{
|
|
"version": "1.0.0"
|
|
},
|
|
{
|
|
"version": "2.0.0"
|
|
},
|
|
{
|
|
"version": "3.0.0"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
After the conversion, any `jsonata` rules defined in the `transformTemplates` section will be applied as usual to further process the JSON data.
|
|
|
|
#### TOML
|
|
|
|
If `toml` is used, response is parsed and converted into TOML for further processing.
|
|
|
|
The below TOML document
|
|
|
|
```toml
|
|
[[releases]]
|
|
version = "1.0.0"
|
|
[[releases]]
|
|
version = "2.0.0"
|
|
[[releases]]
|
|
version = "3.0.0"
|
|
```
|
|
|
|
Will convert applying any `jsonata` rules defined in the `transformTemplates` section will be applied.
|
|
|
|
```json {configType=none}
|
|
{
|
|
"releases": [
|
|
{
|
|
"version": "1.0.0"
|
|
},
|
|
{
|
|
"version": "2.0.0"
|
|
},
|
|
{
|
|
"version": "3.0.0"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
#### HTML
|
|
|
|
If the format is set to `html`, Renovate will call the HTTP endpoint with the `Accept` header value `text/html`.
|
|
The body of the response will be treated as a HTML document, and all hyperlinks will be converted to versions.
|
|
|
|
For the following HTML document:
|
|
|
|
```html
|
|
<html>
|
|
<body>
|
|
<a href="package-1.0.tar.gz">package-1.0.tar.gz</a>
|
|
<a href="package-2.0.tar.gz">package-2.0.tar.gz</a>
|
|
</body>
|
|
</html>
|
|
```
|
|
|
|
The following JSON will be generated:
|
|
|
|
```json {configType=none}
|
|
{
|
|
"releases": [
|
|
{
|
|
"version": "package-1.0.tar.gz"
|
|
},
|
|
{
|
|
"version": "package-1.0.tar.gz"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
After the conversion, any `jsonata` rules defined in the `transformTemplates` section will be applied to process the JSON data.
|
|
|
|
To extract the version number, you may use [`extractVersion`](../../../configuration-options.md#extractversion) or JSONata rules.
|
|
|
|
## Examples
|
|
|
|
### K3s
|
|
|
|
You can use this configuration to request the newest version available to [K3s](https://k3s.io/)
|
|
|
|
```json
|
|
{
|
|
"customDatasources": {
|
|
"k3s": {
|
|
"defaultRegistryUrlTemplate": "https://update.k3s.io/v1-release/channels",
|
|
"transformTemplates": [
|
|
"{\"releases\":[{\"version\": $$.(data[id = 'stable'].latest),\"sourceUrl\":\"https://github.com/k3s-io/k3s\",\"changelogUrl\":$join([\"https://github.com/k3s-io/k3s/releases/tag/\",data[id = 'stable'].latest])}],\"sourceUrl\": \"https://github.com/k3s-io/k3s\",\"homepage\": \"https://k3s.io/\"}"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Hashicorp
|
|
|
|
You can use this configuration to request the newest versions of the Hashicorp products:
|
|
|
|
```json
|
|
{
|
|
"customManagers": [
|
|
{
|
|
"customType": "regex",
|
|
"managerFilePatterns": ["/\\.yml$/"],
|
|
"datasourceTemplate": "custom.hashicorp",
|
|
"matchStrings": [
|
|
"#\\s*renovate:\\s*(datasource=(?<datasource>.*?) )?depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\s*\\w*:\\s*(?<currentValue>.*)\\s"
|
|
],
|
|
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
|
}
|
|
],
|
|
"customDatasources": {
|
|
"hashicorp": {
|
|
"defaultRegistryUrlTemplate": "https://api.releases.hashicorp.com/v1/releases/{{packageName}}?license_class=oss",
|
|
"transformTemplates": [
|
|
"{ \"releases\": $map($, function($v) { { \"version\": $v.version, \"releaseTimestamp\": $v.timestamp_created, \"changelogUrl\": $v.url_changelog, \"sourceUrl\": $v.url_source_repository } }), \"homepage\": $[0].url_project_website, \"sourceUrl\": $[0].url_source_repository }"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
To have the latest Nomad version in your Ansible variables, use this snippet _after_ adding the above configuration:
|
|
|
|
```yaml
|
|
# renovate: depName=nomad
|
|
nomad_version: 1.6.0
|
|
```
|
|
|
|
### Grafana Dashboard
|
|
|
|
You can use the following configuration to upgrade the Grafana Dashboards versions in your [Grafana Helm chart](https://github.com/grafana/helm-charts/blob/e5f1e9c4a4a3b43a820dc4b9eb16f3daa0b6e74f/charts/grafana/values.yaml#L685-L688):
|
|
|
|
```json
|
|
{
|
|
"customManagers": [
|
|
{
|
|
"customType": "regex",
|
|
"managerFilePatterns": ["/\\.yml$/"],
|
|
"matchStrings": [
|
|
"#\\s+renovate:\\s+depName=\"(?<depName>.*)\"\\n\\s+gnetId:\\s+(?<packageName>.*?)\\n\\s+revision:\\s+(?<currentValue>.*)"
|
|
],
|
|
"versioningTemplate": "regex:^(?<major>\\d+)$",
|
|
"datasourceTemplate": "custom.grafana-dashboards"
|
|
}
|
|
],
|
|
"customDatasources": {
|
|
"grafana-dashboards": {
|
|
"defaultRegistryUrlTemplate": "https://grafana.com/api/dashboards/{{packageName}}",
|
|
"format": "json",
|
|
"transformTemplates": [
|
|
"{\"releases\":[{\"version\": $string(revision)}]}"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Grafana Helm chart `values.yaml` snippet:
|
|
|
|
```yml
|
|
dashboards:
|
|
default:
|
|
1860-node-exporter-full:
|
|
# renovate: depName="Node Exporter Full"
|
|
gnetId: 1860
|
|
revision: 31
|
|
datasource: Prometheus
|
|
15760-kubernetes-views-pods:
|
|
# renovate: depName="Kubernetes / Views / Pods"
|
|
gnetId: 15760
|
|
revision: 20
|
|
datasource: Prometheus
|
|
```
|
|
|
|
### Custom offline dependencies
|
|
|
|
Sometimes the "dependency version source" is _not_ available via an API.
|
|
To work around a missing API, you can create dependency "files".
|
|
These files are served via HTTP(S), so that Renovate can access them.
|
|
For example, imagine the following file `versiontracker.json` for the software `something`:
|
|
|
|
<!-- schema-validation-disable-next-block -->
|
|
|
|
```json
|
|
[
|
|
{
|
|
"version": "77"
|
|
},
|
|
{
|
|
"version": "76"
|
|
}
|
|
]
|
|
```
|
|
|
|
By writing a custom datasource, Renovate can process the `versiontracker.json` file, see below.
|
|
This example uses Nexus as the webserver.
|
|
|
|
```json
|
|
{
|
|
"customDatasources": {
|
|
"nexus_generic": {
|
|
"defaultRegistryUrlTemplate": "https://nexus.example.com/repository/versiontrackers/{{packageName}}/versiontracker.json",
|
|
"transformTemplates": [
|
|
"{ \"releases\": $map($, function($v) { { \"version\": $v.version, \"sourceUrl\": $v.filelink } }) }"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
This could be used to update Ansible YAML files with the latest version through a custom manager.
|
|
For example, with the following Ansible content:
|
|
|
|
```yaml
|
|
# renovate: datasource=custom.nexus_generic depName=something versioning=loose
|
|
something_version: '77'
|
|
```
|
|
|
|
And the following custom manager:
|
|
|
|
```json
|
|
{
|
|
"customManagers": [
|
|
{
|
|
"customType": "regex",
|
|
"managerFilePatterns": ["/\\.yml$/"],
|
|
"datasourceTemplate": "custom.nexus_generic",
|
|
"matchStrings": [
|
|
"#\\s*renovate:\\s*(datasource=(?<datasource>.*?)\\s*)?depName=(?<depName>.*?)(\\s*versioning=(?<versioning>.*?))?\\s*\\w*:\\s*[\"']?(?<currentValue>.+?)[\"']?\\s"
|
|
],
|
|
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Or if you have the datasource locally, you can also define your local registry by prefixing it with `file://`:
|
|
|
|
```json
|
|
{
|
|
"customDatasources": {
|
|
"local_generic": {
|
|
"defaultRegistryUrlTemplate": "file://dependencies/{{packageName}}/versiontracker.json",
|
|
"transformTemplates": [
|
|
"{ \"releases\": $map($, function($v) { { \"version\": $v.version, \"sourceUrl\": $v.filelink } }) }"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Renovate will then parse your file from your current folder to access it.
|
|
|
|
### nginx directory listing
|
|
|
|
Sometimes all you have is a directory with files, and a HTTP server that can generate directory listings.
|
|
|
|
Let's use nginx itself as an example:
|
|
|
|
```json
|
|
{
|
|
"customDatasources": {
|
|
"nginx": {
|
|
"defaultRegistryUrlTemplate": "https://nginx.org/download",
|
|
"format": "html"
|
|
}
|
|
},
|
|
"packageRules": [
|
|
{
|
|
"matchDatasources": ["custom.nginx"],
|
|
"extractVersion": "^nginx-(?<version>.+)\\.tar\\.gz$"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### HTML page
|
|
|
|
You can use the `html` format to extract versions from a typical "Downloads" page:
|
|
|
|
```json
|
|
{
|
|
"customDatasources": {
|
|
"curl": {
|
|
"defaultRegistryUrlTemplate": "https://curl.se/download.html",
|
|
"format": "html"
|
|
}
|
|
},
|
|
"packageRules": [
|
|
{
|
|
"matchDatasources": ["custom.curl"],
|
|
"extractVersion": "/curl-(?<version>.+)\\.tar\\.gz$"
|
|
}
|
|
]
|
|
}
|
|
```
|