Preserve transitioned version data when lifecycle expiration only creates a delete marker, and apply suspended-versioning semantics consistently.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Forward FULL_OBJECT checksum type for multipart CRC replication so the target validates the whole-object checksum correctly. Restore the existing single-PUT replication path for server-side copies retaining these checksums.
Fixes#254
Signed-off-by: Feng Ruohang <rh@vonng.com>
Reuse deleteObjectReconciled for tiered bulk entries and unversioned single deletes. Its reference-aware cleanup avoids scheduling remote contents for garbage collection while another copy still needs them, including restored copies. Keep the normal local-data bulk path unchanged.
Extend existing tier GC regressions to bulk and unversioned deletes and include them in the multi-pool race CI step.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Resolve each addressed version under the shared write lock, keep bulk I/O parallel within cleanup and authoritative-deletion stages, and preserve replication response identity on errors. Treat reliable absence as successful physical cleanup on retries without accepting it for marker or replication-state writes.
Cover execution-time faults, partial cleanup recovery, mixed authoritative pools, response identity, null marker ordering, metadata uncertainty, cancellation, and concurrent PUT. Run the delete consistency regressions under race in CI.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Release the authentication-cache lock while retaining IAM write ordering. Carry request cancellation through IAM and SSE-S3 list decryption, and publish policy replacements only after a complete successful load.
Preserve caller deadlines for slow policy queries and keep the existing etcd read budget scoped to its RPC. Cover concurrent refresh, updates, revocation, cancellation, and deadline compatibility.
Validation: 36 focused and 23 adversarial scenarios each passed three race runs; real 32-second object/etcd probes passed. Full suite passed with the Go DNS resolver after reproducing native DNS failures on the baseline; existing etcd suites, make verifiers, and make build also passed.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Only classify a valid IAM key as absent when a length error accompanies an ASCII path component longer than 255 bytes. Normalized LDAP DNs satisfy this condition, while total-path errors and Unicode names can refer to existing revocation records.
Cover component boundaries, wrapped and storage errors, and real revocation records moved beneath a deep Darwin drive path.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Since RELEASE.2026-09-16T00-00-00Z, LDAP users whose normalized DN is
longer than 255 bytes cannot log in. NormDN hex-escapes every non-ASCII
byte (go-ldap DN.String()), so one Cyrillic letter takes six characters.
IAM looks up records named after the DN: the parent revision check added
with cmd/iam-revision.go reads config/iam/users/<DN>/identity.json, and
PolicyDBGet reads config/iam/policydb/sts-users/<DN>.json when the
mapping is not cached.
A drive rejects such an element in one of two ways. checkPathLength
refuses more than 255 counted characters, but outside Windows it never
counts a backslash, so an escaped DN usually passes it. The filesystem
then fails with ENAMETOOLONG once the parent directory exists.
toObjectErr reports both as ObjectNameInvalid, and the callers treat
only errConfigNotFound as "no record". As a result,
AssumeRoleWithLDAPIdentity, access-key creation and STS loading fail.
Such a record cannot have been stored. When the object layer's own name
validation (checkBucketAndObjectNames) accepts the path,
ObjectNameInvalid can only mean a length rejection, so report it as
errConfigNotFound. Invalid names and storage errors are returned
unchanged, and saving such a record still fails. The etcd store has no
such limit and is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: GL <g@lopatin.info>
Enumerate durable STS identities for parent/type revocation, retain the
operation cutoff, and notify sibling caches for each committed deletion.
Repair missed notifications during periodic object/etcd IAM refresh.
Carry immutable STS tombstones and expiry through the existing IAM revision
protocol and deletion healing. Delayed creation cannot undo revocation,
while unrelated and newly issued keys stay valid. Keep partial failures
visible while propagating the successfully committed deletions.
Cover uncached credentials, warm sibling caches, dropped notifications,
object/etcd backends, reordered replay, concurrent issuance and partial
persistence failure. No new dependencies or authentication-path I/O.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Move solitary and null delete markers instead of dropping them, including
idempotent UUID marker retries. Check that a reader's source version still
exists before moving it, and preserve the established read winner for
conflicting timestamp ties.
Retire only the versions that were successfully moved while holding the
pooled object lock. Recheck the source and retained copies before cleanup,
so concurrent deletes and uncertain destination reads cannot revive data.
Count remaining markers when verifying decommission completion.
Add two/three-pool walker, history, purge, concurrent-delete, read-fault,
and timestamp-tie regressions.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Extend the pending-directory collision test with absent recovery and a smaller
key. Add all six normal-object, directory, and pending-object channel orders,
checking complete-group recovery and preservation of trailing entries.
Test changes adapted from commit 557bc274a32330321fa5e5ac7636068c7bd98eb0.
Retain the complete-group directory handling from 3dd53eb81f.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Use the existing self-service authorization helper for policyless regular
users and normalize explicit service-account and STS aliases to their parent.
Keep explicit denials and cross-user authorization intact. Assert decrypted
identity contents and negative boundaries in the existing handler tests.
Adapted from commit 5d0b692d8cd387622a3219e72430dffaaf5a25ce.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Distinguish all-drive native absence from read uncertainty and confirm current pool targets before omitting a recovered name. Cover both existing single-pool writer lock domains, release before encoding or output, and preserve cache terminal errors without blocking the HTTP error response on remote publication.
Keep ordinary entries free of recovery I/O and locks. Bound the selective short lock-conflict wait, preserve existing retry intervals and quorum, and make retries cancelable with late-grant cleanup coverage. Restore the original raw listing callback and unchanged scanner/healing/migration callers now that recovery errors originate in the merger.
Add deterministic deletion, pool ordering, cache, lock-routing and cancellation regressions to race CI. Focused cmd/dsync/lsync race suites and verifiers pass; complete current-main integration and bare-binary performance validation are recorded in PR #239.
Refs: https://github.com/pgsty/silo/issues/218
Signed-off-by: Feng Ruohang <rh@vonng.com>
Honor self-service authorization denials, prevent session-limited credential delegation and updates, and protect root credential mutations under built-in IAM. Preserve empty-policy inheritance and damaged-account cleanup, normalize new service policy claims at issuance, and add signed Admin API and LDAP regressions.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Capture the metacache peer before launching cleanup and pass listing
options by value. The parent can clear its cache ID without changing the
background RPC lookup or racing with its reads of shared request state.
Publish complete immutable disk snapshots through atomic.Value in the
listing fixtures instead of replacing getDisks after requests start.
Add a real peer RPC regression for cleanup using the original cache ID;
reverting the options copy reproduces both the race and an empty RPC ID.
Verified the final 18-test listing race suite, 30 repetitions of the real
peer regression, formatting checks and make lint. Prior runtime evidence
retains its original e90cb206 source identity.
Refs: https://github.com/pgsty/silo/issues/218
Refs: https://github.com/pgsty/silo/pull/239
Signed-off-by: Feng Ruohang <rh@vonng.com>
Continue PR #221 while retaining its original commits and authorship.
Route latest-only fallback reads through the namespace lock used by normal
writes, propagate terminal scan errors before cache completion, and retain
delete markers until the cross-pool merge has resolved them.
Recount narrowly eligible latest null-version metadata when mixed erasure
layouts prevent selection. Keep each EC group separate and preserve the
requested quorum, strict resolution, and version-listing behavior.
Add regression coverage for locking, terminal errors, cache completion,
delete markers, mixed EC layouts, and HTTP behavior; run focused race tests
in CI. Verified with make test, focused race tests, and local distributed
single- and multi-pool rolling-restart workloads with exact final oracles.
Refs: https://github.com/pgsty/silo/issues/218
Refs: https://github.com/pgsty/silo/pull/221
Signed-off-by: Feng Ruohang <rh@vonng.com>
Keep the legacy header resets and clear the unused NSScanner response
field at each construction site. Remove the generic stream Recycler call:
Recycler transfers resource ownership and cannot be treated as a reset
hook followed by another insertion into the handler's pool.
Exercise the actual NSScanner handler across repeated scans with dirty
response allocations, so the regression test does not depend on sync.Pool
reusing an object. Check both response variants after wire serialization.
Cover legacy DecodeMsg and reused indexed metadata headers as well.
Retain the original contributor's commits and DCO sign-offs.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Return storage read failures before accepting an object overwrite. Keep the null-version and multipart target checks under the destination write lock, and add slow-body, read-fault, legacy-upload and recovery-collision regressions to the Object Lock race gate.
Signed-off-by: Feng Ruohang <rh@vonng.com>
Upload ID directories on disk are named as '<UUID>x<UnixNano>'.
Attempting to decode them as Base64 fails unconditionally, causing
the cleanup routine to bypass the timestamp check and fall back to
disk.ReadVersion on every active upload.
Directly parse the timestamp suffix from the plaintext directory
name to restore O(1) stale upload detection, while preserving Base64
decoding and ReadVersion fallback for legacy uploads.
Signed-off-by: clown15 <thsdhrhd15@naver.com>
Close SN-2026-015 by requiring PutObjectRetention with request-derived
condition keys before reading bucket or object state. Weakening unexpired
GOVERNANCE retention also requires the bypass header and permission. Treat
retention removal as zero remaining days and avoid duration saturation for
distant dates.
Close SN-2026-016 by requiring replication trust for caller-selected target
version IDs and checking persisted multipart targets and locked null
versions under the destination write lock, including across storage pools.
Preserve versioning transitions, trusted replication and batch Snowball
version IDs, and fail closed when lock state cannot be read.
Add authorization, concurrent lock-update, multipart transition, cross-pool,
configuration-failure and retention-boundary regression coverage. Validate
with make test, focused race tests and the original independent probes.
Developed with Codex (Astra).
Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
(cherry picked from commit fe3935476b929e03193d7496a7448d760e635096)
Drive renameData() with stub disks to assert that an in-quorum
errCommitMetaMissing is converted to errErasureWriteQuorum (HTTP 503)
and that an in-quorum errFileNotFound still surfaces unchanged so
PutObject() keeps returning IncompleteBody for client disconnects.
Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
The V1 header wire format has no Signature field, so decoding into a
reused header leaked the previous object's signature. Reset it along
with EcN/EcM so a decode fully replaces the header contents.
Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
NSScanner responses stream either a dataUsageEntry update or the
final dataUsageCache. The grid framework recycles stream responses
into a pool without resetting them, and the handler set only one
field per response - so a recycled response still carried the Final
(or Update) payload of an earlier scan. A remote node reading such a
response accepted the stale final cache, corrupting bucket usage
data across scanner cycles.
Honor Recycler types when pooling stream responses, implement it for
nsScannerResp, and clear the unset field at the response
construction sites for defense in depth.
Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
The legacy header decode paths zeroed EcN twice (z.EcN, z.EcN = 0, 0)
instead of EcN and EcM, so EcM retained its previous value when the
same structure was reused across decodes. Decoding an xl.meta written
by an older release would then leak the erasure metadata (data/parity
block counts) of a previously decoded object into it, potentially
corrupting quorum and healing decisions.
Also zero the fields on the V1 header path, which had no reset at
all.
Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
When the staged xl.meta disappears between the data rename and the
meta-file commit inside RenameData(), the affected drives return
errFileNotFound. An in-quorum errFileNotFound is reduced by
reduceWriteQuorumErrs() and previously surfaced as:
- IncompleteBody (400) from PutObject(), misattributing the failure
to a client disconnect, or
- ObjectNotFound (404 NoSuchKey) from CompleteMultipartUpload(),
telling clients the object is absent so they skip retries.
Map errFileNotFound at the commit step to a dedicated internal
errCommitMetaMissing, convert it to errErasureWriteQuorum in
renameData(), and map it to InsufficientWriteQuorum (HTTP 503) in
toObjectErr() so clients retry. Below-quorum outcomes already return
InsufficientWriteQuorum and are unchanged, as is the IncompleteBody
mapping for staging failures of client-disconnected uploads.
Signed-off-by: Julien Laurenceau <julien.laurenceau@pepitedata.com>
hash.ChecksumMismatch now carries the requested algorithm (CRC32, CRC32C,
SHA1, SHA256, CRC64NVME), and toAPIError renders it S3-style:
The CRC32 you specified did not match the calculated checksum.
instead of the generic "The provided 'x-amz-checksum' header does not match
what was computed.", for all four mismatch construction sites (streaming
reader, trailing-checksum decode, and both Checksum.Matches paths). Mirrors
the wording already shipped for CompleteMultipartUpload in #225.
Fixes#226
Feng Ruohang