gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ead2f2fe6c0689dda421b832d5e52c8c103fd0b4
spinifex/go.mod
T
Josh Siviour d632b607ff fix: decompress single-file zst/bz2/xz/gz disk images (#1048) 
* fix: decompress single-file zst/bz2/xz/gz disk images into tmpdir

ExtractDiskImageFromFile only handled tar archives and a broken plain
.xz branch that ran xz -dk beside the source and then tried to parse
tar chatter for a path, so it silently returned an empty string. Talos
(.raw.zst, .raw.xz), Fedora CoreOS (.qcow2.xz, .raw.xz) and Flatcar
(.img.bz2) publish only single-file compressed artifacts, so none of
them could be imported.

Add streaming decompression for .zst, .bz2, .xz and .gz that writes
into the caller's tmpdir and returns that path directly, instead of
inferring it from tar output. A decompressed .qcow2 payload now flows
through the existing qcow2-to-raw qemu-img conversion via a shared
finalizeDiskImage helper. zstd uses klauspost/compress, already a
transitive dependency via containerd, for a pure-Go streaming decoder.
bz2 and gzip use the standard library. xz shells out to the xz binary
since no pure-Go xz decoder is already a dependency, and the existing
.tar.xz branch already requires that binary on the host via tar's -J
flag, so this adds no new host requirement.

* fix: write sparse holes when decompressing single-file disk images

Verified live on env13: a Talos raw.zst imported successfully but the
decompressed output was written literally rather than sparsely, an
11 GiB file on disk for content that decompresses to 183 MiB with
xz -dc | dd conv=sparse. This is a regression relative to the
qcow2-to-raw path beside it, which already writes sparse via
qemu-img convert.

decompressToTemp now reads in 1 MiB blocks, seeks past all-zero
blocks instead of writing them, and truncates to the final size so a
trailing zero run is still recorded as a hole. 1 MiB amortises the
read/seek syscall overhead over a multi-GiB image while still being
small enough to catch the large contiguous zero runs that dominate a
sparse raw disk; a zero run shorter than one block is written
literally, which is the safe direction for a false negative to fail
in.

zstd, bzip2 and gzip all route through decompressToTemp, so all three
gain sparse output from this change. The plain-.xz path shells out to
the xz binary with output piped straight to a file and does not go
through decompressToTemp, so it is unaffected by this fix and remains
a known follow-up.

* fix: make plain-.xz decompression sparse too

decompressXz shelled its output straight to a file, bypassing the
sparse-hole handling just added to decompressToTemp for zstd, bzip2
and gzip. .raw.xz is the format Talos and Flatcar actually publish
alongside .zst, so this path is the one most likely hit in practice,
and left dense the exact disk-size regression this fix targets still
fires for it.

exec.Cmd already runs its own copier goroutine when Stdout is not an
*os.File, and Wait joins it. decompressXz now hands the xz process an
io.Pipe write end and reuses decompressToTemp on the read end, so
plain .xz gets identical sparse-hole handling to the other codecs. A
separate goroutine closes the pipe with the process's exit error once
Wait returns, so a read past the end of a failed decompression
surfaces that error, stderr included, instead of a bare EOF; the
error path also closes the read end explicitly so a stuck copier
cannot wedge Wait forever if decompressToTemp gives up early.
2026-09-16 16:54:51 +10:00

228 lines
11 KiB
Modula-2
Raw Blame History

module github.com/mulgadc/spinifex
go 1.27.0
require (
github.com/aws/aws-sdk-go v1.55.8
github.com/aws/aws-sdk-go-v2 v1.47.0
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20
github.com/aws/aws-sdk-go-v2/config v1.33.4
github.com/aws/aws-sdk-go-v2/credentials v1.20.4
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.0
github.com/cenkalti/backoff/v4 v4.3.0
github.com/charmbracelet/bubbles v1.0.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/containerd/containerd/v2 v2.3.5
github.com/distribution/reference v0.6.0
github.com/go-chi/chi/v5 v5.3.2
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/insomniacslk/dhcp v0.0.0-20260719225207-c76316d4aa82
github.com/jackc/pgx/v5 v5.11.0
github.com/klauspost/compress v1.19.2
github.com/klauspost/cpuid/v2 v2.4.0
github.com/miekg/dns v1.1.73
github.com/minio/highwayhash v1.0.4
github.com/mulgadc/bluebottle v1.20.1-0.20260916040751-5a5ca6f0fb33
github.com/mulgadc/northstar v1.20.0
github.com/mulgadc/predastore v1.20.0
github.com/mulgadc/viperblock v1.20.0
github.com/nats-io/nats-server/v2 v2.14.6
github.com/nats-io/nats.go v1.53.1
github.com/opencontainers/runtime-spec v1.3.0
github.com/ovn-kubernetes/libovsdb v0.8.1
github.com/pelletier/go-toml/v2 v2.4.3
github.com/pterm/pterm v0.12.83
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.12.1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0
go.opentelemetry.io/otel v1.46.0
go.opentelemetry.io/otel/metric v1.46.0
go.opentelemetry.io/otel/sdk v1.46.0
go.opentelemetry.io/otel/sdk/metric v1.46.0
go.opentelemetry.io/otel/trace v1.46.0
go.uber.org/goleak v1.3.1-0.20260602155401-5f791e3ef83b
go.yaml.in/yaml/v3 v3.0.5
golang.org/x/crypto v0.57.0
golang.org/x/sync v0.23.0
golang.org/x/sys v0.48.0
golang.org/x/time v0.16.0
golang.org/x/tools v0.50.0
gopkg.in/ini.v1 v1.67.3
)
require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
github.com/Microsoft/go-winio v0.6.3-0.20251027160822-ad3df93bed29 // indirect
github.com/Microsoft/hcsshim v0.15.0-rc.1 // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.2-default-no-op // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.3 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.113.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.10.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.38.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.50.0 // indirect
github.com/aws/smithy-go v1.28.1 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boltdb/bolt v1.3.1 // indirect
github.com/buraksezer/consistent v1.1.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cenkalti/hub v1.0.2 // indirect
github.com/cenkalti/rpc2 v1.0.5 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/charmbracelet/colorprofile v0.4.1 // indirect
github.com/charmbracelet/x/ansi v0.11.6 // indirect
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
github.com/charmbracelet/x/term v0.2.2 // indirect
github.com/clipperhouse/displaywidth v0.9.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/cgroups/v3 v3.1.3 // indirect
github.com/containerd/console v1.0.5 // indirect
github.com/containerd/containerd/api v1.11.1 // indirect
github.com/containerd/continuity v0.5.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/fifo v1.1.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v1.0.0-rc.5 // indirect
github.com/containerd/plugin v1.1.0 // indirect
github.com/containerd/ttrpc v1.2.8 // indirect
github.com/containerd/typeurl/v2 v2.2.3 // indirect
github.com/dgraph-io/badger/v4 v4.9.6 // indirect
github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
github.com/fatih/color v1.19.0 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gookit/color v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/raft v1.7.3 // indirect
github.com/hashicorp/raft-boltdb/v2 v2.3.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/native v1.1.0 // indirect
github.com/klauspost/reedsolomon v1.14.2 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/mdlayher/packet v1.1.2 // indirect
github.com/mdlayher/socket v0.5.1 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/moby/locker v1.0.1 // indirect
github.com/moby/sys/capability v0.4.0 // indirect
github.com/moby/sys/mountinfo v0.7.2 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/signal v0.7.1 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.2 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/runtime-tools v0.9.1-0.20251114084447-edf4cb3d2116 // indirect
github.com/pierrec/lz4/v4 v4.1.18 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/quic-go/quic-go v0.62.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/tidwall/btree v1.8.1 // indirect
github.com/u-root/uio v0.0.0-20230220225925-ffce2a382923 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/bridges/otelslog v0.20.1 // indirect
go.opentelemetry.io/contrib/instrumentation/runtime v0.71.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.22.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 // indirect
go.opentelemetry.io/otel/log v0.22.0 // indirect
go.opentelemetry.io/otel/sdk/log v0.22.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.uber.org/nilaway v0.0.0-20260721205819-4a0f7653488a // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
golang.org/x/exp/typeparams v0.0.0-20260611194520-c48552f49976 // indirect
golang.org/x/mod v0.41.0 // indirect
golang.org/x/net v0.59.0 // indirect
golang.org/x/telemetry v0.0.0-20260908163034-4bcc4b2ee518 // indirect
golang.org/x/term v0.46.0 // indirect
golang.org/x/text v0.42.0 // indirect
golang.org/x/vuln v1.6.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/grpc v1.83.2 // indirect
google.golang.org/protobuf v1.36.12 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
tags.cncf.io/container-device-interface v1.1.0 // indirect
tags.cncf.io/container-device-interface/specs-go v1.1.0 // indirect
)
// go.opencensus.io -> grpc@v1.33.2 drags in the pre-split monolithic genproto,
// which still ships google.golang.org/genproto/googleapis/rpc/status and so
// collides with the split genproto/googleapis/rpc module that containerd/ttrpc
// needs ("ambiguous import"). The main module never imports the monolith, so a
// plain require is dropped by `go mod tidy`; pin it via replace to a post-split
// version (no rpc/status) so the fix survives tidy. Remove if grpc/opencensus
// stop requiring the monolith.
replace google.golang.org/genproto => google.golang.org/genproto v0.0.0-20260622175928-b703f567277d
tool (
go.uber.org/nilaway/cmd/nilaway
golang.org/x/vuln/cmd/govulncheck
)
Reference in New Issue View Git Blame Copy Permalink