* s3: a list marker that sorts before the prefix excludes nothing
ListObjects `marker` and ListObjectsV2 `start-after` are a plain key
cutoff: list the keys that sort after it. A marker that sorts before the
prefix and is not under it therefore excludes no key carrying the prefix,
and the listing must equal the one with no marker at all.
normalizePrefixMarker treated every marker that does not start with the
prefix as "something wrong" and the listing came back empty. Clients send
this shape routinely: docker/distribution's S3 storage driver walks
prefix "<root>/<path>/" with start-after "<root>" (its rootdirectory), so
on SeaweedFS a registry walk saw an empty bucket. zot read that as "no
repositories": /v2/_catalog was empty, GC/scrub/retention never saw a
repo, and on restart its storage parse deleted every repository's
metadata as "no longer in storage".
listFilerEntries now lists as if no marker were given when the marker
sorts before the prefix; the response still echoes the marker the client
sent. A marker that sorts after the prefix's subtree is left alone: it may
legitimately sit inside a partial-name prefix's match set, which
normalizePrefixMarker already handles, and otherwise correctly lists
nothing.
Reproduce on 4.44 and 4.47:
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot/zot/" # all keys
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=zot" # KeyCount 0
curl -s "$S/zot?list-type=2&prefix=zot/zot/&start-after=a" # KeyCount 0
* s3: keep the prefix's own key excluded by a marker that names it
Fold the before-prefix marker rule into normalizePrefixMarker, which now also
derives prefixEndsOnDelimiter from the effective marker instead of each cursor
rebuilding the expression.
A marker equal to the prefix is no longer trimmed to a subtree cutoff:
start-after "a/b/" with prefix "a/b/" excludes only the "a/b/" key, so the walk
starts inside that directory and its children still list.
Adds a listing-level test that walks the whole path for both start-after shapes
a registry sends, and covers the new normalization cases.
* s3: leading slashes do not hide a marker that names the prefix
* s3: echo the V1 marker the client sent, not the walk's cutoff
* s3: filter only the walk's cutoff from the V1 page, not the echoed marker
* s3: skip the key an exclusive marker names as it streams
---------
Co-authored-by: Zuse <be9c90a8-c104-4be2-b7a4-9f92eb833ac8@forge.local>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
see https://blog.aqwari.net/xml-schema-go/
1. go get aqwari.net/xml/cmd/xsdgen
2. Add EncodingType element for ListBucketResult in AmazonS3.xsd
3. xsdgen -o s3api_xsd_generated.go -pkg s3api AmazonS3.xsd
4. Remove empty Grantee struct in s3api_xsd_generated.go
5. Remove xmlns: sed s'/http:\/\/s3.amazonaws.com\/doc\/2006-03-01\/\ //' s3api_xsd_generated.go
robfischer1
1f037e48f9