Compare commits

...

13 Commits

Author SHA1 Message Date
安正超 08e1f4670b fix: restore default CORS fallback and STS object ACL ownership (#2053)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-03 01:08:50 +08:00
唐小鸭 fff96a0921 fix sse-options (#2056) 2026-03-03 01:08:37 +08:00
唐小鸭 f17725a2ea fix(sse): allow PUT/GET without KMS when no SSE or bucket default (#2054)
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-03 00:44:23 +08:00
houseme bf957e3523 remove rustflags target cpu (#2052) 2026-03-02 23:59:34 +08:00
Rafael Herrero a6090b98dc fix(iam): remove incorrect trailing slash from OIDC issuer URL (#2050) 2026-03-02 19:48:12 +08:00
houseme 2ac07c95a8 refactor(obs): enhance log cleanup and rotation (#2040) 2026-03-02 16:28:32 +08:00
安正超 e157a88f09 fix: support query-only presigned URL access (#2046) 2026-03-02 15:46:50 +08:00
安正超 01a75b5f58 Add env variable alias compatibility warnings (#2044) 2026-03-02 15:34:19 +08:00
GatewayJ 2cb8db36a5 fix(iam): user group policy and delete group (fixes #2028) (#2043)
Co-authored-by: GatewayJ <8352692332qq.com>
2026-03-02 14:19:01 +08:00
weisd e3815aa101 fix(ecstore): add etag fallback when mod_time unavailable in metadata (#2042) 2026-03-02 13:50:17 +08:00
dependabot[bot] fd32507ce5 build(deps): bump datafusion from 52.1.0 to 52.2.0 in the dependencies group (#2037)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-02 11:37:24 +08:00
GatewayJ ba32fd9d96 fix(s3): allow anonymous access when PublicAccessBlock config is miss… (#2039)
Co-authored-by: GatewayJ <8352692332qq.com>
2026-03-02 11:37:00 +08:00
安正超 273dbc9c38 feat(s3): return 409 BucketAlreadyExists when non-owner creates existing bucket (#2034) 2026-03-01 22:53:41 +08:00
47 changed files with 1793 additions and 1363 deletions
+4 -4
View File
@@ -174,7 +174,7 @@ jobs:
target: x86_64-unknown-linux-musl
cross: false
platform: linux
rustflags: '-C target-cpu=x86-64-v2'
rustflags: ''
- os: ubicloud-standard-2
target: aarch64-unknown-linux-musl
cross: true
@@ -184,7 +184,7 @@ jobs:
target: x86_64-unknown-linux-gnu
cross: false
platform: linux
rustflags: '-C target-cpu=x86-64-v2'
rustflags: ''
- os: ubicloud-standard-2
target: aarch64-unknown-linux-gnu
cross: true
@@ -200,13 +200,13 @@ jobs:
target: x86_64-apple-darwin
cross: false
platform: macos
rustflags: '-C target-cpu=x86-64-v2'
rustflags: ''
# Windows builds (temporarily disabled)
- os: windows-latest
target: x86_64-pc-windows-msvc
cross: false
platform: windows
rustflags: '-C target-cpu=x86-64-v2'
rustflags: ''
#- os: windows-latest
# target: aarch64-pc-windows-msvc
# cross: true
+1
View File
@@ -2,6 +2,7 @@
.DS_Store
.idea
.vscode
.cursor
.direnv/
/test
/logs
+4 -2
View File
@@ -71,8 +71,10 @@ Co-locate unit tests with their modules and give behavior-led names. Integration
## Environment Variables
- `RUSTFS_ENABLE_SCANNER` - Enable/disable background data scanner (default: true)
- `RUSTFS_ENABLE_HEAL` - Enable/disable auto-heal functionality (default: true)
- Global configuration environment variables must use flat `RUSTFS_*` names (no module segments), such as `RUSTFS_REGION`, `RUSTFS_ADDRESS`, `RUSTFS_VOLUMES`, and `RUSTFS_LICENSE`.
- `RUSTFS_SCANNER_ENABLED` - Enable/disable background data scanner (default: true)
- `RUSTFS_HEAL_ENABLED` - Enable/disable auto-heal functionality (default: true)
- Deprecated aliases (for pre-beta compatibility) are documented in [the config module README](crates/config/README.md#environment-variable-naming-conventions) and must log warnings when used.
- For KMS tests: `NO_PROXY=127.0.0.1,localhost` and clear proxy environment variables
## Commit & Pull Request Guidelines
Generated
+71 -71
View File
@@ -2183,9 +2183,9 @@ checksum = "d7a1e2f27636f116493b8b860f5546edb47c8d8f8ea73e1d2a20be88e28d1fea"
[[package]]
name = "datafusion"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d12ee9fdc6cdb5898c7691bb994f0ba606c4acc93a2258d78bb9f26ff8158bb3"
checksum = "503f1f4a9060ae6e650d3dff5dc7a21266fea1302d890768d45b4b28586e830f"
dependencies = [
"arrow",
"arrow-schema",
@@ -2238,9 +2238,9 @@ dependencies = [
[[package]]
name = "datafusion-catalog"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "462dc9ef45e5d688aeaae49a7e310587e81b6016b9d03bace5626ad0043e5a9e"
checksum = "14417a3ee4ae3d092b56cd6c1d32e8ff3e2c9ec130ecb2276ec91c89fd599399"
dependencies = [
"arrow",
"async-trait",
@@ -2263,9 +2263,9 @@ dependencies = [
[[package]]
name = "datafusion-catalog-listing"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1b96dbf1d728fc321817b744eb5080cdd75312faa6980b338817f68f3caa4208"
checksum = "9d0eba824adb45a4b3ac6f0251d40df3f6a9382371cad136f4f14ac9ebc6bc10"
dependencies = [
"arrow",
"async-trait",
@@ -2286,9 +2286,9 @@ dependencies = [
[[package]]
name = "datafusion-common"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3237a6ff0d2149af4631290074289cae548c9863c885d821315d54c6673a074a"
checksum = "0039deefbd00c56adf5168b7ca58568fb058e4ba4c5a03b09f8be371b4e434b6"
dependencies = [
"ahash",
"arrow",
@@ -2310,9 +2310,9 @@ dependencies = [
[[package]]
name = "datafusion-common-runtime"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "70b5e34026af55a1bfccb1ef0a763cf1f64e77c696ffcf5a128a278c31236528"
checksum = "1ec7e3e60b813048331f8fb9673583173e5d2dd8fef862834ee871fc98b57ca7"
dependencies = [
"futures",
"log",
@@ -2321,9 +2321,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1b2a6be734cc3785e18bbf2a7f2b22537f6b9fb960d79617775a51568c281842"
checksum = "802068957f620302ecf05f84ff4019601aeafd36f5f3f1334984af2e34265129"
dependencies = [
"arrow",
"async-compression",
@@ -2356,9 +2356,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-arrow"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1739b9b07c9236389e09c74f770e88aff7055250774e9def7d3f4f56b3dcc7be"
checksum = "90fc387d5067c62d494a6647d29c5ad4fcdd5a6e50ab4ea1d2568caa2d66f2cc"
dependencies = [
"arrow",
"arrow-ipc",
@@ -2380,9 +2380,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-csv"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c73bc54b518bbba7c7650299d07d58730293cfba4356f6f428cc94c20b7600"
checksum = "efd5e20579bb6c8bd4e6c620253972fb723822030c280dd6aa047f660d09eeba"
dependencies = [
"arrow",
"async-trait",
@@ -2403,9 +2403,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-json"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "37812c8494c698c4d889374ecfabbff780f1f26d9ec095dd1bddfc2a8ca12559"
checksum = "c0788b0d48fcef31880a02013ea3cc18e5a4e0eacc3b0abdd2cd0597b99dc96e"
dependencies = [
"arrow",
"async-trait",
@@ -2425,9 +2425,9 @@ dependencies = [
[[package]]
name = "datafusion-datasource-parquet"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2210937ecd9f0e824c397e73f4b5385c97cd1aff43ab2b5836fcfd2d321523fb"
checksum = "66639b70f1f363f5f0950733170100e588f1acfacac90c1894e231194aa35957"
dependencies = [
"arrow",
"async-trait",
@@ -2455,15 +2455,15 @@ dependencies = [
[[package]]
name = "datafusion-doc"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c825f969126bc2ef6a6a02d94b3c07abff871acf4d6dd759ce1255edb7923ce"
checksum = "e44b41f3e8267c6cf3eec982d63f34db9f1dd5f30abfd2e1f124f0871708952e"
[[package]]
name = "datafusion-execution"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fa03ef05a2c2f90dd6c743e3e111078e322f4b395d20d4b4d431a245d79521ae"
checksum = "9e456f60e5d38db45335e84617006d90af14a8c8c5b8e959add708b2daaa0e2c"
dependencies = [
"arrow",
"async-trait",
@@ -2482,9 +2482,9 @@ dependencies = [
[[package]]
name = "datafusion-expr"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef33934c1f98ee695cc51192cc5f9ed3a8febee84fdbcd9131bf9d3a9a78276f"
checksum = "6507c719804265a58043134580c1c20767e7c23ba450724393f03ec982769ad9"
dependencies = [
"arrow",
"async-trait",
@@ -2505,9 +2505,9 @@ dependencies = [
[[package]]
name = "datafusion-expr-common"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "000c98206e3dd47d2939a94b6c67af4bfa6732dd668ac4fafdbde408fd9134ea"
checksum = "a413caa9c5885072b539337aed68488f0291653e8edd7d676c92df2480f6cab0"
dependencies = [
"arrow",
"datafusion-common",
@@ -2518,9 +2518,9 @@ dependencies = [
[[package]]
name = "datafusion-functions"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "379b01418ab95ca947014066248c22139fe9af9289354de10b445bd000d5d276"
checksum = "189256495dc9cbbb8e20dbcf161f60422e628d201a78df8207e44bd4baefadb6"
dependencies = [
"arrow",
"arrow-buffer",
@@ -2549,9 +2549,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-aggregate"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd00d5454ba4c3f8ebbd04bd6a6a9dc7ced7c56d883f70f2076c188be8459e4c"
checksum = "12e73dfee4cd67c4a507ffff4c5a711d39983adf544adbc09c09bf06f789f413"
dependencies = [
"ahash",
"arrow",
@@ -2570,9 +2570,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-aggregate-common"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aec06b380729a87210a4e11f555ec2d729a328142253f8d557b87593622ecc9f"
checksum = "87727bd9e65f4f9ac6d608c9810b7da9eaa3b18b26a4a4b76520592d49020acf"
dependencies = [
"ahash",
"arrow",
@@ -2583,9 +2583,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-nested"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "904f48d45e0f1eb7d0eb5c0f80f2b5c6046a85454364a6b16a2e0b46f62e7dff"
checksum = "2e5ef761359224b7c2b5a1bfad6296ac63225f8583d08ad18af9ba1a89ac3887"
dependencies = [
"arrow",
"arrow-ord",
@@ -2606,9 +2606,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-table"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9a0d20e2b887e11bee24f7734d780a2588b925796ac741c3118dd06d5aa77f0"
checksum = "3b17dac25dfda2d2a90ff0ad1c054a11fb1523766226bec6e9bd8c410daee2ae"
dependencies = [
"arrow",
"async-trait",
@@ -2622,9 +2622,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-window"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3414b0a07e39b6979fe3a69c7aa79a9f1369f1d5c8e52146e66058be1b285ee"
checksum = "c594a29ddb22cbdbce500e4d99b5b2392c5cecb4c1086298b41d1ffec14dbb77"
dependencies = [
"arrow",
"datafusion-common",
@@ -2640,9 +2640,9 @@ dependencies = [
[[package]]
name = "datafusion-functions-window-common"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5bf2feae63cd4754e31add64ce75cae07d015bce4bb41cd09872f93add32523a"
checksum = "9aa1b15ed81c7543f62264a30dd49dec4b1b0b698053b968f53be32dfba4f729"
dependencies = [
"datafusion-common",
"datafusion-physical-expr-common",
@@ -2650,9 +2650,9 @@ dependencies = [
[[package]]
name = "datafusion-macros"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c4fe888aeb6a095c4bcbe8ac1874c4b9a4c7ffa2ba849db7922683ba20875aaf"
checksum = "c00c31c4795597aa25b74cab5174ac07a53051f27ce1e011ecaffa9eaeecef81"
dependencies = [
"datafusion-doc",
"quote",
@@ -2661,9 +2661,9 @@ dependencies = [
[[package]]
name = "datafusion-optimizer"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a6527c063ae305c11be397a86d8193936f4b84d137fe40bd706dfc178cf733c"
checksum = "80ccf60767c09302b2e0fc3afebb3761a6d508d07316fab8c5e93312728a21bb"
dependencies = [
"arrow",
"chrono",
@@ -2681,9 +2681,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-expr"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0bb028323dd4efd049dd8a78d78fe81b2b969447b39c51424167f973ac5811d9"
checksum = "c64b7f277556944e4edd3558da01d9e9ff9f5416f1c0aa7fee088e57bd141a7e"
dependencies = [
"ahash",
"arrow",
@@ -2705,9 +2705,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-expr-adapter"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78fe0826aef7eab6b4b61533d811234a7a9e5e458331ebbf94152a51fc8ab433"
checksum = "b7abaee372ea2d19c016ee9ef8629c4415257d291cdd152bc7f0b75f28af1b63"
dependencies = [
"arrow",
"datafusion-common",
@@ -2720,9 +2720,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-expr-common"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfccd388620734c661bd8b7ca93c44cdd59fecc9b550eea416a78ffcbb29475f"
checksum = "42237efe621f92adc22d111b531fdbc2cc38ca9b5e02327535628fb103ae2157"
dependencies = [
"ahash",
"arrow",
@@ -2737,9 +2737,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-optimizer"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bde5fa10e73259a03b705d5fddc136516814ab5f441b939525618a4070f5a059"
checksum = "fd093498bd1319c6e5c76e9dfa905e78486f01b34579ce97f2e3a49f84c37fac"
dependencies = [
"arrow",
"datafusion-common",
@@ -2756,9 +2756,9 @@ dependencies = [
[[package]]
name = "datafusion-physical-plan"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e1098760fb29127c24cc9ade3277051dc73c9ed0ac0131bd7bcd742e0ad7470"
checksum = "7cbe61b12daf81a9f20ba03bd3541165d51f86e004ef37426b11881330eed261"
dependencies = [
"ahash",
"arrow",
@@ -2787,9 +2787,9 @@ dependencies = [
[[package]]
name = "datafusion-pruning"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "64d0fef4201777b52951edec086c21a5b246f3c82621569ddb4a26f488bc38a9"
checksum = "0124331116db7f79df92ebfd2c3b11a8f90240f253555c9bb084f10b6fecf1dd"
dependencies = [
"arrow",
"datafusion-common",
@@ -2804,9 +2804,9 @@ dependencies = [
[[package]]
name = "datafusion-session"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f71f1e39e8f2acbf1c63b0e93756c2e970a64729dab70ac789587d6237c4fde0"
checksum = "1673e3c58ba618a6ea0568672f00664087b8982c581e9afd5aa6c3c79c9b431f"
dependencies = [
"async-trait",
"datafusion-common",
@@ -2818,9 +2818,9 @@ dependencies = [
[[package]]
name = "datafusion-sql"
version = "52.1.0"
version = "52.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f44693cfcaeb7a9f12d71d1c576c3a6dc025a12cef209375fa2d16fb3b5670ee"
checksum = "5272d256dab5347bb39d2040589f45d8c6b715b27edcb5fffe88cc8b9c3909cb"
dependencies = [
"arrow",
"bigdecimal",
@@ -4385,9 +4385,9 @@ dependencies = [
[[package]]
name = "inferno"
version = "0.12.5"
version = "0.12.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "20dd69640582458beceefcf045f8de34263d45194999c9a49fcd53e5b503d522"
checksum = "90807d610575744524d9bdc69f3885d96f0e6c3354565b0828354a7ff2a262b8"
dependencies = [
"ahash",
"clap",
@@ -5086,9 +5086,9 @@ dependencies = [
[[package]]
name = "moka"
version = "0.12.13"
version = "0.12.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4ac832c50ced444ef6be0767a008b02c106a909ba79d1d830501e94b96f6b7e"
checksum = "85f8024e1c8e71c778968af91d43700ce1d11b219d127d79fb2934153b82b42b"
dependencies = [
"async-lock",
"crossbeam-channel",
@@ -6109,7 +6109,7 @@ dependencies = [
"anyhow",
"backtrace",
"flate2",
"inferno 0.12.5",
"inferno 0.12.6",
"num",
"paste",
"prost",
@@ -8892,9 +8892,9 @@ dependencies = [
[[package]]
name = "sysinfo"
version = "0.38.2"
version = "0.38.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1efc19935b4b66baa6f654ac7924c192f55b175c00a7ab72410fc24284dacda8"
checksum = "d03c61d2a49c649a15c407338afe7accafde9dac869995dccb73e5f7ef7d9034"
dependencies = [
"libc",
"memchr",
@@ -10578,9 +10578,9 @@ dependencies = [
[[package]]
name = "zlib-rs"
version = "0.6.2"
version = "0.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c745c48e1007337ed136dc99df34128b9faa6ed542d80a1c673cf55a6d7236c8"
checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513"
[[package]]
name = "zmij"
+3 -3
View File
@@ -195,7 +195,7 @@ const-str = { version = "1.1.0", features = ["std", "proc"] }
convert_case = "0.11.0"
criterion = { version = "0.8", features = ["html_reports"] }
crossbeam-queue = "0.3.12"
datafusion = "52.1.0"
datafusion = "52.2.0"
derive_builder = "0.20.2"
enumset = "1.1.10"
faster-hex = "0.10.0"
@@ -216,7 +216,7 @@ matchit = "0.9.1"
md-5 = "0.11.0-rc.5"
md5 = "0.8.0"
mime_guess = "2.0.5"
moka = { version = "0.12.13", features = ["future"] }
moka = { version = "0.12.14", features = ["future"] }
netif = "0.1.6"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.12.0"
@@ -245,7 +245,7 @@ snafu = "0.8.9"
snap = "1.1.1"
starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
strum = { version = "0.28.0", features = ["derive"] }
sysinfo = "0.38.2"
sysinfo = "0.38.3"
temp-env = "0.3.6"
tempfile = "3.26.0"
test-case = "3.3.1"
+18
View File
@@ -32,6 +32,24 @@
For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## Environment Variable Naming Conventions
RustFS uses a flat naming style for top-level configuration: environment variables are `RUSTFS_*` without nested module segments.
Examples:
- `RUSTFS_REGION`
- `RUSTFS_ADDRESS`
- `RUSTFS_VOLUMES`
- `RUSTFS_LICENSE`
Current guidance:
- Prefer module-specific names only when they are not top-level product configuration.
- Renamed variables must keep backward-compatible aliases until before beta.
- Alias usage must emit deprecation warnings and be treated as transitional only.
- Deprecated example:
- `RUSTFS_ENABLE_SCANNER` -> `RUSTFS_SCANNER_ENABLED`
- `RUSTFS_ENABLE_HEAL` -> `RUSTFS_HEAL_ENABLED`
## 📄 License
This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
+9
View File
@@ -128,6 +128,9 @@ pub const RUSTFS_LICENSE_URL: &str = "https://www.apache.org/licenses/LICENSE-2.
/// Example: RUSTFS_ADDRESS=":9000"
pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
/// Environment variable for server volumes.
pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
/// Default port for rustfs
/// This is the default port for rustfs.
/// This is used to bind the server to a specific port.
@@ -153,6 +156,12 @@ pub const DEFAULT_CONSOLE_ADDRESS: &str = concat!(":", DEFAULT_CONSOLE_PORT);
/// Default value: us-east-1
pub const RUSTFS_REGION: &str = "us-east-1";
/// Environment variable for server region.
pub const ENV_RUSTFS_REGION: &str = "RUSTFS_REGION";
/// Environment variable for server license.
pub const ENV_RUSTFS_LICENSE: &str = "RUSTFS_LICENSE";
/// Default log filename for rustfs
/// This is the default log filename for rustfs.
/// It is used to store the logs of the application.
+4 -4
View File
@@ -15,6 +15,7 @@
// Observability Keys
mod metrics;
use const_str::concat;
pub use metrics::*;
pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
@@ -42,7 +43,6 @@ pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
/// Log cleanup related configurations
pub const ENV_OBS_LOG_KEEP_COUNT: &str = "RUSTFS_OBS_LOG_KEEP_COUNT";
pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES";
pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
pub const ENV_OBS_LOG_COMPRESS_OLD_FILES: &str = "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES";
@@ -56,15 +56,16 @@ pub const ENV_OBS_LOG_DRY_RUN: &str = "RUSTFS_OBS_LOG_DRY_RUN";
pub const ENV_OBS_LOG_MATCH_MODE: &str = "RUSTFS_OBS_LOG_MATCH_MODE";
/// Default values for log cleanup
pub const DEFAULT_OBS_LOG_KEEP_COUNT: usize = 10;
pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; // 2 GiB
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION: &str = "gz";
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION);
pub const DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: u64 = 30;
pub const DEFAULT_OBS_LOG_DELETE_EMPTY_FILES: bool = true;
pub const DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS: u64 = 3600; // 1 hour
pub const DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS: u64 = 6 * 3600; // 6 hours
pub const DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS: u64 = 1800; // 0.5 hours
pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
@@ -106,7 +107,6 @@ mod tests {
assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
// Test log cleanup related env keys
assert_eq!(ENV_OBS_LOG_KEEP_COUNT, "RUSTFS_OBS_LOG_KEEP_COUNT");
assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_COMPRESS_OLD_FILES, "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES");
@@ -0,0 +1,172 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for Issue #2036
//! Verifies that anonymous access works correctly with bucket policies
//! when PublicAccessBlock configuration is missing or explicitly set.
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::types::PublicAccessBlockConfiguration;
use serial_test::serial;
use tracing::info;
async fn setup_public_bucket(
env: &RustFSTestEnvironment,
bucket_name: &str,
) -> Result<aws_sdk_s3::Client, Box<dyn std::error::Error + Send + Sync>> {
let admin_client = env.create_s3_client();
admin_client.create_bucket().bucket(bucket_name).send().await?;
let policy_json = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAnonymousGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:GetObject"],
"Resource": [format!("arn:aws:s3:::{}/*", bucket_name)]
}
]
})
.to_string();
admin_client
.put_bucket_policy()
.bucket(bucket_name)
.policy(&policy_json)
.send()
.await?;
admin_client
.put_object()
.bucket(bucket_name)
.key("test.txt")
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"hello anonymous"))
.send()
.await?;
Ok(admin_client)
}
async fn anonymous_get_object(
env: &RustFSTestEnvironment,
bucket_name: &str,
key: &str,
) -> Result<reqwest::Response, reqwest::Error> {
let url = format!("{}/{}/{}", env.url, bucket_name, key);
reqwest::Client::new().get(&url).send().await
}
/// Issue #2036: Anonymous GetObject should succeed when bucket policy allows it
/// and no PublicAccessBlock configuration exists (ConfigNotFound).
#[tokio::test]
#[serial]
async fn test_anonymous_access_allowed_when_public_access_block_missing() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
info!("Starting test: anonymous access with missing PublicAccessBlock config...");
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let bucket_name = "anon-test-no-pab";
let admin_client = setup_public_bucket(&env, bucket_name).await?;
let _ = admin_client.delete_public_access_block().bucket(bucket_name).send().await;
let resp = anonymous_get_object(&env, bucket_name, "test.txt").await?;
assert_eq!(
resp.status().as_u16(),
200,
"Anonymous GetObject should succeed when no PublicAccessBlock config exists"
);
info!("Test passed: anonymous access allowed with missing PublicAccessBlock config");
Ok(())
}
/// Anonymous GetObject should be denied when RestrictPublicBuckets is true.
#[tokio::test]
#[serial]
async fn test_anonymous_access_denied_when_restrict_public_buckets_enabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting test: anonymous access denied with RestrictPublicBuckets=true...");
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let bucket_name = "anon-test-restrict";
let admin_client = setup_public_bucket(&env, bucket_name).await?;
admin_client
.put_public_access_block()
.bucket(bucket_name)
.public_access_block_configuration(
PublicAccessBlockConfiguration::builder()
.restrict_public_buckets(true)
.build(),
)
.send()
.await?;
let resp = anonymous_get_object(&env, bucket_name, "test.txt").await?;
assert_eq!(
resp.status().as_u16(),
403,
"Anonymous GetObject should be denied when RestrictPublicBuckets is true"
);
info!("Test passed: anonymous access denied with RestrictPublicBuckets=true");
Ok(())
}
/// Anonymous GetObject should succeed when PublicAccessBlock exists but
/// RestrictPublicBuckets is explicitly false.
#[tokio::test]
#[serial]
async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting test: anonymous access allowed with RestrictPublicBuckets=false...");
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let bucket_name = "anon-test-allow";
let admin_client = setup_public_bucket(&env, bucket_name).await?;
admin_client
.put_public_access_block()
.bucket(bucket_name)
.public_access_block_configuration(
PublicAccessBlockConfiguration::builder()
.restrict_public_buckets(false)
.build(),
)
.send()
.await?;
let resp = anonymous_get_object(&env, bucket_name, "test.txt").await?;
assert_eq!(
resp.status().as_u16(),
200,
"Anonymous GetObject should succeed when RestrictPublicBuckets is false"
);
info!("Test passed: anonymous access allowed with RestrictPublicBuckets=false");
Ok(())
}
+132
View File
@@ -12,10 +12,26 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! E2E tests for group management (fixes #2028).
use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_get, awscurl_put, init_logging};
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::{Client, Config};
use serial_test::serial;
use tracing::info;
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-group-test");
let config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
}
/// Test that deleting a group with members fails, and deleting an empty group succeeds.
#[tokio::test(flavor = "multi_thread")]
#[serial]
@@ -74,3 +90,119 @@ async fn test_delete_group_requires_empty_membership() -> Result<(), Box<dyn std
Ok(())
}
/// Test that a user with only group membership (no explicit user policy) gets group policies
/// and can perform actions allowed by the group (regression test for #2028.1).
#[tokio::test(flavor = "multi_thread")]
#[serial]
#[ignore = "requires awscurl and spawns a real RustFS server"]
async fn test_user_with_only_group_gets_group_policies() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let user_name = "grouponlyuser";
let user_secret = "grouponlysecret";
let group_name = "policygroup";
let policy_name = "ListBucketsOnlyPolicy";
// 1. Create canned policy that allows ListAllMyBuckets only
let policy_doc = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:ListAllMyBuckets"],
"Resource": ["*"]
}]
});
let add_policy_url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
awscurl_put(&add_policy_url, &policy_doc.to_string(), &env.access_key, &env.secret_key).await?;
info!("Created canned policy {}", policy_name);
// 2. Create user with no explicit policy
let add_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, user_name);
let user_body = serde_json::json!({
"secretKey": user_secret,
"status": "enabled"
});
awscurl_put(&add_user_url, &user_body.to_string(), &env.access_key, &env.secret_key).await?;
info!("Created user {} with no explicit policy", user_name);
// 3. Add user to group (creates group with this member; user_group_memberships must be updated)
let update_members_url = format!("{}/rustfs/admin/v3/update-group-members", env.url);
let add_member_body = serde_json::json!({
"group": group_name,
"members": [user_name],
"isRemove": false,
"groupStatus": "enabled"
});
awscurl_put(&update_members_url, &add_member_body.to_string(), &env.access_key, &env.secret_key).await?;
info!("Added {} to group {}", user_name, group_name);
// 4. Attach policy to group
let set_policy_url = format!(
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=true",
env.url, policy_name, group_name
);
awscurl_put(&set_policy_url, "", &env.access_key, &env.secret_key).await?;
info!("Attached policy {} to group {}", policy_name, group_name);
// 5. User with only group (no user policy) should be able to list buckets
let user_client = create_user_s3_client(&env, user_name, user_secret);
let list_result = user_client.list_buckets().send().await;
list_result.map_err(|e| -> Box<dyn std::error::Error + Send + Sync> {
format!("User with only group membership should get group policies and list buckets: {}", e).into()
})?;
info!("User with only group successfully listed buckets (group policies applied)");
Ok(())
}
/// Test that after deleting a user who was the only member of a group, the group can be deleted
/// (regression test for #2028.2: delete group uses backend membership, not stale cache).
#[tokio::test(flavor = "multi_thread")]
#[serial]
#[ignore = "requires awscurl and spawns a real RustFS server"]
async fn test_delete_group_after_deleting_user() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let user_name = "solemember";
let user_secret = "solemembersecret";
let group_name = "soledeletegroup";
// 1. Create user
let add_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, user_name);
let user_body = serde_json::json!({
"secretKey": user_secret,
"status": "enabled"
});
awscurl_put(&add_user_url, &user_body.to_string(), &env.access_key, &env.secret_key).await?;
info!("Created user {}", user_name);
// 2. Add user to group
let update_members_url = format!("{}/rustfs/admin/v3/update-group-members", env.url);
let add_member_body = serde_json::json!({
"group": group_name,
"members": [user_name],
"isRemove": false,
"groupStatus": "enabled"
});
awscurl_put(&update_members_url, &add_member_body.to_string(), &env.access_key, &env.secret_key).await?;
info!("Added {} to group {}", user_name, group_name);
// 3. Delete the user (backend and cache update so group membership becomes empty)
let remove_user_url = format!("{}/rustfs/admin/v3/remove-user?accessKey={}", env.url, user_name);
awscurl_delete(&remove_user_url, &env.access_key, &env.secret_key).await?;
info!("Deleted user {}", user_name);
// 4. Deleting the group should succeed (backend has empty members; no stale cache)
let delete_group_url = format!("{}/rustfs/admin/v3/group/{}", env.url, group_name);
awscurl_delete(&delete_group_url, &env.access_key, &env.secret_key).await?;
info!("Deleted group {} after user was removed", group_name);
Ok(())
}
+4
View File
@@ -40,6 +40,10 @@ mod quota_test;
#[cfg(test)]
mod bucket_policy_check_test;
// Regression tests for Issue #2036: anonymous access with PublicAccessBlock
#[cfg(test)]
mod anonymous_access_test;
// Special characters in path test modules
#[cfg(test)]
mod special_chars_test;
+22 -5
View File
@@ -277,10 +277,26 @@ impl SetDisks {
quorum: usize,
) -> (Vec<Option<DiskStore>>, Option<OffsetDateTime>, Option<String>) {
let mod_times = Self::list_object_modtimes(parts_metadata, errs);
let etags = Self::list_object_etags(parts_metadata, errs);
let mod_time = Self::common_time(&mod_times, quorum);
let etag = Self::common_etag(&etags, quorum);
if mod_time.is_none() {
let etags = Self::list_object_etags(parts_metadata, errs);
let etag_op = Self::common_etag(&etags, quorum);
if let Some(etag) = etag_op {
let mut new_disk = vec![None; disks.len()];
for (i, etag_item) in etags.iter().enumerate() {
if let Some(etag_item) = etag_item
&& etag_item == &etag
&& parts_metadata[i].is_valid()
{
new_disk[i].clone_from(&disks[i]);
}
}
return (new_disk, None, Some(etag));
}
}
let mut new_disk = vec![None; disks.len()];
@@ -290,7 +306,7 @@ impl SetDisks {
}
}
(new_disk, mod_time, etag)
(new_disk, mod_time, None)
}
pub(super) fn pick_valid_fileinfo(
@@ -403,8 +419,9 @@ impl SetDisks {
if let Some(hash) = op_hash
&& let Some(max_hash) = max_val
&& hash == max_hash
&& metas[i].is_valid()
{
if metas[i].is_valid() && !found {
if !found {
found_fi = Some(metas[i].clone());
found = true;
}
@@ -422,7 +439,7 @@ impl SetDisks {
let mut fi = found_fi.unwrap();
for (val, &count) in &valid_obj_map {
if count > quorum {
if count >= quorum {
fi.mod_time = val.mod_time;
fi.num_versions = val.num_versions;
fi.is_latest = val.mod_time.is_none();
+16 -12
View File
@@ -1386,11 +1386,9 @@ where
let user_group_memberships = self.cache.user_group_memberships.load();
members.iter().for_each(|member| {
if let Some(m) = user_group_memberships.get(member) {
let mut m = m.clone();
m.insert(group.to_string());
Cache::add_or_update(&self.cache.user_group_memberships, member, &m, OffsetDateTime::now_utc());
}
let mut m = user_group_memberships.get(member).cloned().unwrap_or_default();
m.insert(group.to_string());
Cache::add_or_update(&self.cache.user_group_memberships, member, &m, OffsetDateTime::now_utc());
});
Ok(OffsetDateTime::now_utc())
@@ -1518,13 +1516,19 @@ where
}
}
let gi = self
.cache
.groups
.load()
.get(group)
.cloned()
.ok_or(Error::NoSuchGroup(group.to_string()))?;
let gi = if members.is_empty() {
// Reload from backend so we see latest members (e.g. after user was deleted elsewhere)
let mut m = HashMap::new();
self.api.load_group(group, &mut m).await?;
m.get(group).cloned().ok_or(Error::NoSuchGroup(group.to_string()))?
} else {
self.cache
.groups
.load()
.get(group)
.cloned()
.ok_or(Error::NoSuchGroup(group.to_string()))?
};
if members.is_empty() && !gi.members.is_empty() {
return Err(Error::GroupNotEmpty);
+3 -7
View File
@@ -779,10 +779,6 @@ fn normalize_config_url(config_url: &str) -> Result<String, String> {
issuer.push_str(normalized_path);
}
if !issuer.ends_with('/') {
issuer.push('/');
}
Ok(issuer)
}
@@ -892,15 +888,15 @@ mod tests {
fn test_normalize_config_url() {
assert_eq!(
normalize_config_url("https://idp.example.com/.well-known/openid-configuration").unwrap(),
"https://idp.example.com/"
"https://idp.example.com"
);
assert_eq!(
normalize_config_url("https://idp.example.com/.well-known/openid-configuration/").unwrap(),
"https://idp.example.com/"
"https://idp.example.com"
);
assert_eq!(
normalize_config_url("https://idp.example.com/custom/realm").unwrap(),
"https://idp.example.com/custom/realm/"
"https://idp.example.com/custom/realm"
);
assert!(normalize_config_url("https://idp.example.com/.well-known/invalid").is_err());
assert!(normalize_config_url("gopher://idp.example.com").is_err());
+2 -2
View File
@@ -597,7 +597,7 @@ impl Store for ObjectStore {
async fn delete_group_info(&self, name: &str) -> Result<()> {
self.delete_iam_config(get_group_info_path(name)).await.map_err(|err| {
if is_err_config_not_found(&err) {
Error::NoSuchPolicy
Error::NoSuchGroup(name.to_string())
} else {
err
}
@@ -607,7 +607,7 @@ impl Store for ObjectStore {
async fn load_group(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
let u: GroupInfo = self.load_iam_config(get_group_info_path(name)).await.map_err(|err| {
if is_err_config_not_found(&err) {
Error::NoSuchPolicy
Error::NoSuchGroup(name.to_string())
} else {
err
}
@@ -13,7 +13,8 @@
// limitations under the License.
//! Disabled lock manager that bypasses all locking operations
//! Used when RUSTFS_ENABLE_LOCKS environment variable is set to false
//! Used when the lock feature is disabled via RUSTFS_LOCK_ENABLED
//! (or deprecated RUSTFS_ENABLE_LOCKS).
use std::sync::Arc;
+25 -5
View File
@@ -69,8 +69,12 @@ pub const BUILD_TIMESTAMP: &str = "unknown";
/// Maximum number of items in delete list
pub const MAX_DELETE_LIST: usize = 1000;
/// Default setting for RUSTFS_ENABLE_LOCKS environment variable
/// Default setting for lock enablement.
/// Canonical variable: RUSTFS_LOCK_ENABLED
/// Deprecated compatibility alias: RUSTFS_ENABLE_LOCKS
const DEFAULT_RUSTFS_LOCKS_ENABLED: bool = true;
const ENV_LOCK_ENABLED: &str = "RUSTFS_LOCK_ENABLED";
const ENV_LOCK_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_LOCKS";
// ============================================================================
// Global FastLock Manager
@@ -97,10 +101,26 @@ impl Default for GlobalLockManager {
impl GlobalLockManager {
/// Create a lock manager based on environment variable configuration
pub fn new() -> Self {
// Check RUSTFS_ENABLE_LOCKS environment variable
let locks_enabled = rustfs_utils::get_env_bool("RUSTFS_ENABLE_LOCKS", DEFAULT_RUSTFS_LOCKS_ENABLED);
// Check lock enablement env vars with deprecated compatibility support.
let locks_enabled = rustfs_utils::get_env_bool_with_aliases(
ENV_LOCK_ENABLED,
&[ENV_LOCK_ENABLED_DEPRECATED],
DEFAULT_RUSTFS_LOCKS_ENABLED,
);
if !locks_enabled {
tracing::info!("Lock system disabled via RUSTFS_ENABLE_LOCKS environment variable");
let disabled_by = if std::env::var(ENV_LOCK_ENABLED).is_ok() {
ENV_LOCK_ENABLED
} else if std::env::var(ENV_LOCK_ENABLED_DEPRECATED).is_ok() {
ENV_LOCK_ENABLED_DEPRECATED
} else {
"default setting"
};
if disabled_by == "default setting" {
tracing::info!("Lock system disabled via default setting");
} else {
tracing::info!("Lock system disabled via {} environment variable", disabled_by);
}
return Self::Disabled(DisabledLockManager::new());
}
tracing::info!("Lock system enabled");
@@ -248,7 +268,7 @@ static GLOBAL_LOCK_MANAGER: OnceLock<Arc<GlobalLockManager>> = OnceLock::new();
/// Get the global shared lock manager instance
///
/// Returns either FastObjectLockManager or DisabledLockManager based on
/// the RUSTFS_ENABLE_LOCKS environment variable.
/// the RUSTFS_LOCK_ENABLED environment variable.
pub fn get_global_lock_manager() -> Arc<GlobalLockManager> {
GLOBAL_LOCK_MANAGER.get_or_init(|| Arc::new(GlobalLockManager::new())).clone()
}
+9 -7
View File
@@ -132,15 +132,15 @@ All configuration is read from environment variables at startup.
| `RUSTFS_OBS_LOGGER_LEVEL` | `info` | Log level; `RUST_LOG` syntax supported |
| `RUSTFS_OBS_LOG_STDOUT_ENABLED` | `false` | When file logging is active, also mirror to stdout |
| `RUSTFS_OBS_LOG_DIRECTORY` | _(empty)_ | **Directory for rolling log files. When empty, logs go to stdout only** |
| `RUSTFS_OBS_LOG_FILENAME` | `rustfs` | Base filename for rolling logs (date suffix added automatically) |
| `RUSTFS_OBS_LOG_FILENAME` | `rustfs.log` | Base filename for rolling logs (date suffix added automatically) |
| `RUSTFS_OBS_LOG_ROTATION_TIME` | `hourly` | Rotation granularity: `minutely`, `hourly`, or `daily` |
| `RUSTFS_OBS_LOG_KEEP_FILES` | `30` | Number of rolling files to keep |
| `RUSTFS_OBS_LOG_KEEP_FILES` | `30` | Number of rolling files to keep (also used by cleaner) |
| `RUSTFS_OBS_LOG_MATCH_MODE` | `suffix` | File matching mode: `prefix` or `suffix` |
### Log cleanup
| Variable | Default | Description |
|----------|---------|-------------|
| `RUSTFS_OBS_LOG_KEEP_COUNT` | `10` | Minimum files the cleaner must always preserve |
| `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | `2147483648` | Hard cap on total log directory size (2 GiB) |
| `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` | `0` | Per-file size cap; `0` = unlimited |
| `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | `true` | Gzip-compress files before deleting |
@@ -149,7 +149,7 @@ All configuration is read from environment variables at startup.
| `RUSTFS_OBS_LOG_EXCLUDE_PATTERNS` | _(empty)_ | Comma-separated glob patterns to never clean up |
| `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | `true` | Remove zero-byte files |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | `3600` | Minimum file age (seconds) before cleanup |
| `RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS` | `21600` | How often the cleanup task runs (6 hours) |
| `RUSTFS_OBS_LOG_CLEANUP_INTERVAL_SECONDS` | `1800` | How often the cleanup task runs (0.5 hours) |
| `RUSTFS_OBS_LOG_DRY_RUN` | `false` | Report deletions without actually removing files |
@@ -223,12 +223,12 @@ rustfs-obs/src/
│ ├── otel.rs # Full OTLP/HTTP pipeline
│ └── recorder.rs # metrics-crate → OTel bridge (Recorder)
├── log_cleanup/ # Background log-file cleanup subsystem
├── cleaner/ # Background log-file cleanup subsystem
│ ├── mod.rs # LogCleaner public API + tests
│ ├── types.rs # FileInfo shared type
│ ├── scanner.rs # Filesystem discovery
│ ├── compress.rs # Gzip compression helper
│ └── cleaner.rs # Selection, compression, deletion logic
│ └── core.rs # Selection, compression, deletion logic
└── system/ # Host metrics (CPU, memory, disk, GPU)
├── mod.rs
@@ -245,10 +245,12 @@ rustfs-obs/src/
```rust
use std::path::PathBuf;
use rustfs_obs::LogCleaner;
use rustfs_obs::types::FileMatchMode;
let cleaner = LogCleaner::new(
PathBuf::from("/var/log/rustfs"),
"rustfs.log.".to_string(), // file_prefix
"rustfs.log.".to_string(), // file_pattern
FileMatchMode::Prefix, // match_mode
10, // keep_count
2 * 1024 * 1024 * 1024, // max_total_size_bytes (2 GiB)
0, // max_single_file_size_bytes (unlimited)
+2 -1
View File
@@ -20,6 +20,7 @@
use flate2::Compression;
use flate2::write::GzEncoder;
use rustfs_config::observability::DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION;
use std::fs::File;
use std::io::{BufReader, BufWriter, Write};
use std::path::Path;
@@ -39,7 +40,7 @@ use tracing::{debug, info};
/// Propagates any I/O error encountered while opening, reading, writing, or
/// flushing files.
pub(super) fn compress_file(path: &Path, level: u32, dry_run: bool) -> Result<(), std::io::Error> {
let gz_path = path.with_extension("gz");
let gz_path = path.with_extension(DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION);
if gz_path.exists() {
debug!("Compressed file already exists, skipping: {:?}", gz_path);
+11 -18
View File
@@ -47,7 +47,7 @@ pub struct LogCleaner {
pub(super) match_mode: FileMatchMode,
/// The cleaner will never delete files if doing so would leave fewer than
/// this many files in the directory.
pub(super) keep_count: usize,
pub(super) keep_files: usize,
/// Hard ceiling on the total bytes of all managed files; `0` = no limit.
pub(super) max_total_size_bytes: u64,
/// Hard ceiling on a single file's size; `0` = no per-file limit.
@@ -81,7 +81,7 @@ impl LogCleaner {
log_dir: PathBuf,
file_pattern: String,
match_mode: FileMatchMode,
keep_count: usize,
keep_files: usize,
max_total_size_bytes: u64,
max_single_file_size_bytes: u64,
compress_old_files: bool,
@@ -101,7 +101,7 @@ impl LogCleaner {
log_dir,
file_pattern,
match_mode,
keep_count,
keep_files,
max_total_size_bytes,
max_single_file_size_bytes,
compress_old_files,
@@ -207,7 +207,7 @@ impl LogCleaner {
/// Choose which files from `files` (sorted oldest-first) should be deleted or rotated.
///
/// The algorithm respects three constraints in order:
/// 1. Always keep at least `keep_count` files.
/// 1. Always keep at least `keep_files` files.
/// 2. Delete old files while the total size exceeds `max_total_size_bytes`.
/// 3. Delete any file whose individual size exceeds `max_single_file_size_bytes`.
///
@@ -226,27 +226,20 @@ impl LogCleaner {
// We will protect this file from size-based deletion.
let active_file_idx = files.len() - 1;
// The number of files we are allowed to delete.
// Any file with index >= max_deletable_count is protected by keep_count.
let max_deletable_count = files.len().saturating_sub(self.keep_count);
// Calculate how many files we *must* delete to satisfy keep_files.
let must_delete_count = files.len().saturating_sub(self.keep_files);
let mut current_size = total_size;
for (idx, file) in files.iter().enumerate() {
// If we are in the protected range, we stop deleting.
if idx >= max_deletable_count {
// However, if the active file is too large, we might rotate it.
if idx == active_file_idx {
let over_single = self.max_single_file_size_bytes > 0 && file.size > self.max_single_file_size_bytes;
if over_single {
to_rotate = Some(file.clone());
}
}
// Condition 1: Enforce keep_files.
// If we are in the range of files that exceed the count limit, delete them.
if idx < must_delete_count {
current_size = current_size.saturating_sub(file.size);
to_delete.push(file.clone());
continue;
}
// We are in the deletable range. Check if we *should* delete.
// Condition 2: Enforce max_total_size_bytes.
let over_total = self.max_total_size_bytes > 0 && current_size > self.max_total_size_bytes;
+6 -5
View File
@@ -37,7 +37,7 @@
//! PathBuf::from("/var/log/rustfs"),
//! "rustfs.log.".to_string(),
//! FileMatchMode::Prefix,
//! 10, // keep_count
//! 10, // keep_files
//! 2 * 1024 * 1024 * 1024, // max_total_size_bytes (2 GiB)
//! 0, // max_single_file_size_bytes (unlimited)
//! true, // compress_old_files
@@ -106,7 +106,7 @@ mod tests {
create_log_file(&dir, "app.log.2024-01-03", 1024)?;
create_log_file(&dir, "other.log", 1024)?; // not managed
// Total managed = 3 072 bytes; limit = 2 048; keep_count = 2 → must delete 1.
// Total managed = 3 072 bytes; limit = 2 048; keep_files = 2 → must delete 1.
let cleaner = make_cleaner(dir.clone(), 2, 2048);
let (deleted, freed) = cleaner.cleanup()?;
@@ -116,7 +116,7 @@ mod tests {
}
#[test]
fn test_cleanup_respects_keep_count() -> std::io::Result<()> {
fn test_cleanup_respects_keep_files() -> std::io::Result<()> {
let tmp = TempDir::new()?;
let dir = tmp.path().to_path_buf();
@@ -124,10 +124,11 @@ mod tests {
create_log_file(&dir, &format!("app.log.2024-01-0{i}"), 1024)?;
}
// No size limit, keep_count = 3 → nothing to delete (5 > 3 but size == 0 limit).
let cleaner = make_cleaner(dir.clone(), 3, 0);
let (deleted, _) = cleaner.cleanup()?;
assert_eq!(deleted, 0, "keep_count prevents deletion when no size limit");
// Updated expectation: keep_files acts as a limit (ceiling), so excess files are deleted.
assert_eq!(deleted, 2, "keep_files should enforce a maximum file count");
Ok(())
}
+3 -2
View File
@@ -18,6 +18,7 @@
//! compress any files — it only reports what it found.
use super::types::{FileInfo, FileMatchMode};
use rustfs_config::observability::DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION;
use std::path::Path;
use std::time::{Duration, SystemTime};
use tracing::debug;
@@ -81,7 +82,7 @@ pub(super) fn collect_log_files(
}
// Compressed files are handled by collect_compressed_files.
if filename.ends_with(".gz") {
if filename.ends_with(DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION) {
continue;
}
@@ -180,7 +181,7 @@ pub(super) fn collect_expired_compressed_files(
None => continue,
};
if !filename.ends_with(".gz") {
if !filename.ends_with(DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION) {
continue;
}
+19
View File
@@ -14,6 +14,7 @@
//! Shared types used across the log-cleanup sub-modules.
use std::fmt;
use std::path::PathBuf;
use std::time::SystemTime;
@@ -21,11 +22,29 @@ use std::time::SystemTime;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FileMatchMode {
/// The filename must start with the pattern (e.g. "app.log." matches "app.log.2024-01-01").
/// Corresponds to config value "prefix".
Prefix,
/// The filename must end with the pattern (e.g. ".log" matches "2024-01-01.log").
/// Corresponds to config value "suffix".
Suffix,
}
impl FileMatchMode {
/// Returns the string representation of the match mode.
pub fn as_str(&self) -> &'static str {
match self {
FileMatchMode::Prefix => "prefix",
FileMatchMode::Suffix => "suffix",
}
}
}
impl fmt::Display for FileMatchMode {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(self.as_str())
}
}
/// Metadata for a single log file discovered by the scanner.
///
/// Carries enough information to make cleanup decisions (sort by age, compare
+16 -22
View File
@@ -25,16 +25,16 @@
use rustfs_config::observability::{
DEFAULT_OBS_ENVIRONMENT_PRODUCTION, DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS, DEFAULT_OBS_LOG_COMPRESS_OLD_FILES,
DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS, DEFAULT_OBS_LOG_DELETE_EMPTY_FILES, DEFAULT_OBS_LOG_DRY_RUN,
DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_KEEP_COUNT, DEFAULT_OBS_LOG_MATCH_MODE,
DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS,
ENV_OBS_ENDPOINT, ENV_OBS_ENVIRONMENT, ENV_OBS_LOG_CLEANUP_INTERVAL_SECONDS, ENV_OBS_LOG_COMPRESS_OLD_FILES,
ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS, ENV_OBS_LOG_DELETE_EMPTY_FILES, ENV_OBS_LOG_DIRECTORY, ENV_OBS_LOG_DRY_RUN,
ENV_OBS_LOG_ENDPOINT, ENV_OBS_LOG_EXCLUDE_PATTERNS, ENV_OBS_LOG_FILENAME, ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL,
ENV_OBS_LOG_KEEP_COUNT, ENV_OBS_LOG_KEEP_FILES, ENV_OBS_LOG_MATCH_MODE, ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, ENV_OBS_LOG_MIN_FILE_AGE_SECONDS, ENV_OBS_LOG_ROTATION_TIME, ENV_OBS_LOG_STDOUT_ENABLED,
ENV_OBS_LOGGER_LEVEL, ENV_OBS_LOGS_EXPORT_ENABLED, ENV_OBS_METER_INTERVAL, ENV_OBS_METRIC_ENDPOINT,
ENV_OBS_METRICS_EXPORT_ENABLED, ENV_OBS_SAMPLE_RATIO, ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_TRACE_ENDPOINT,
ENV_OBS_TRACES_EXPORT_ENABLED, ENV_OBS_USE_STDOUT,
DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_MATCH_MODE, DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS, ENV_OBS_ENDPOINT, ENV_OBS_ENVIRONMENT,
ENV_OBS_LOG_CLEANUP_INTERVAL_SECONDS, ENV_OBS_LOG_COMPRESS_OLD_FILES, ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS,
ENV_OBS_LOG_DELETE_EMPTY_FILES, ENV_OBS_LOG_DIRECTORY, ENV_OBS_LOG_DRY_RUN, ENV_OBS_LOG_ENDPOINT,
ENV_OBS_LOG_EXCLUDE_PATTERNS, ENV_OBS_LOG_FILENAME, ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL, ENV_OBS_LOG_KEEP_FILES,
ENV_OBS_LOG_MATCH_MODE, ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES,
ENV_OBS_LOG_MIN_FILE_AGE_SECONDS, ENV_OBS_LOG_ROTATION_TIME, ENV_OBS_LOG_STDOUT_ENABLED, ENV_OBS_LOGGER_LEVEL,
ENV_OBS_LOGS_EXPORT_ENABLED, ENV_OBS_METER_INTERVAL, ENV_OBS_METRIC_ENDPOINT, ENV_OBS_METRICS_EXPORT_ENABLED,
ENV_OBS_SAMPLE_RATIO, ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_TRACE_ENDPOINT, ENV_OBS_TRACES_EXPORT_ENABLED,
ENV_OBS_USE_STDOUT,
};
use rustfs_config::{
APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_LEVEL, DEFAULT_LOG_ROTATION_TIME, DEFAULT_OBS_LOG_FILENAME,
@@ -65,8 +65,8 @@ use std::env;
///
/// - All fields are `Option<T>` to allow partial configuration via environment
/// variables with sensible defaults provided by constants in `rustfs-config`.
/// - `log_keep_count` represents the cleaner's minimum retention; `log_keep_files`
/// controls the rolling-appender's file limit (both typically set to the same value).
/// - `log_keep_files` is used to derive the rolling-appender's upper bound on
/// retained files (if enabled) and to set the cleaner's minimum retention count.
///
/// # Example
/// ```no_run
@@ -128,13 +128,11 @@ pub struct OtelConfig {
/// Rotation time granularity: `"hourly"` or `"daily"` (default: `"daily"`).
pub log_rotation_time: Option<String>,
/// Number of rolling log files to retain (default: `30`).
/// The rolling-appender will delete the oldest file when this limit is exceeded.
/// Used by both the rolling-appender (as a loose upper bound) and the
/// background cleaner (as the minimum retention count).
pub log_keep_files: Option<usize>,
// ── Log cleanup ───────────────────────────────────────────────────────────
/// Minimum number of files the cleaner must always preserve.
/// Typically set to the same value as `log_keep_files`.
pub log_keep_count: Option<usize>,
/// Hard ceiling on the total size (bytes) of all log files (default: 2 GiB).
pub log_max_total_size_bytes: Option<u64>,
/// Per-file size ceiling (bytes); `0` means unlimited (default: `0`).
@@ -205,12 +203,9 @@ impl OtelConfig {
_ => None,
};
// `log_keep_files` (legacy) and `log_keep_count` (new) share the same
// environment variables but have slightly different semantics.
// `log_keep_files` is the rolling-appender retention count; `log_keep_count`
// is the cleaner's minimum-keep threshold. Both default to the same value.
// `log_keep_files` is the single source of truth for file retention count.
// It defaults to `DEFAULT_LOG_KEEP_FILES` (30).
let log_keep_files = Some(get_env_usize(ENV_OBS_LOG_KEEP_FILES, DEFAULT_LOG_KEEP_FILES));
let log_keep_count = Some(get_env_usize(ENV_OBS_LOG_KEEP_COUNT, DEFAULT_OBS_LOG_KEEP_COUNT));
// `log_rotation_time` drives the rolling-appender rotation period.
let log_rotation_time = Some(get_env_str(ENV_OBS_LOG_ROTATION_TIME, DEFAULT_LOG_ROTATION_TIME));
@@ -238,7 +233,6 @@ impl OtelConfig {
log_rotation_time,
log_keep_files,
// Log cleanup
log_keep_count,
log_max_total_size_bytes: Some(get_env_u64(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES)),
log_max_single_file_size_bytes: Some(get_env_u64(
ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
+51 -8
View File
@@ -22,10 +22,11 @@ use tracing_subscriber::EnvFilter;
/// Build an `EnvFilter` from the given log level string.
///
/// If the `RUST_LOG` environment variable is set, it takes precedence over the
/// provided `logger_level`. For non-verbose levels (`info`, `warn`, `error`),
/// noisy internal crates (`hyper`, `tonic`, `h2`, `reqwest`, `tower`) are
/// automatically silenced to reduce log noise.
/// If `default_level` is provided, it is used directly. Otherwise, the
/// `RUST_LOG` environment variable takes precedence over `logger_level`.
/// For non-verbose levels (`info`, `warn`, `error`), noisy internal crates
/// (`hyper`, `tonic`, `h2`, `reqwest`, `tower`) are automatically silenced
/// based on the effective log configuration.
///
/// # Arguments
/// * `logger_level` - The desired log level string (e.g., `"info"`, `"debug"`).
@@ -35,13 +36,48 @@ use tracing_subscriber::EnvFilter;
///
/// # Returns
/// A configured `EnvFilter` ready to be attached to a `tracing_subscriber` registry.
fn is_verbose_level(level: &str) -> bool {
matches!(level.to_ascii_lowercase().as_str(), "trace" | "debug")
}
fn rust_log_requests_verbose(rust_log: &str) -> bool {
rust_log.split(',').any(|directive| {
let directive = directive.trim();
if directive.is_empty() {
return false;
}
let level = directive.rsplit('=').next().unwrap_or("").trim();
is_verbose_level(level)
})
}
fn should_suppress_noisy_crates(logger_level: &str, default_level: Option<&str>, rust_log: Option<&str>) -> bool {
if let Some(level) = default_level {
return !is_verbose_level(level);
}
if let Some(rust_log) = rust_log {
return !rust_log_requests_verbose(rust_log);
}
!is_verbose_level(logger_level)
}
pub(super) fn build_env_filter(logger_level: &str, default_level: Option<&str>) -> EnvFilter {
let level = default_level.unwrap_or(logger_level);
let mut filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(level));
let rust_log = if default_level.is_none() {
std::env::var("RUST_LOG").ok()
} else {
None
};
let mut filter = default_level
.map(EnvFilter::new)
.unwrap_or_else(|| EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(level)));
// Suppress chatty infrastructure crates unless the operator explicitly
// requests trace/debug output.
if !matches!(logger_level, "trace" | "debug") {
if should_suppress_noisy_crates(logger_level, default_level, rust_log.as_deref()) {
let directives: SmallVec<[&str; 5]> = smallvec::smallvec!["hyper", "tonic", "h2", "reqwest", "tower"];
for directive in directives {
filter = filter.add_directive(format!("{directive}=off").parse().unwrap());
@@ -70,7 +106,7 @@ mod tests {
#[test]
fn test_build_env_filter_suppresses_noisy_crates() {
// For info level, hyper/tonic/etc. should be suppressed with OFF.
let filter = build_env_filter("info", None);
let filter = build_env_filter("debug", Some("info"));
let dbg = format!("{filter:?}");
// The Debug output uses `LevelFilter::OFF` for suppressed crates.
assert!(
@@ -82,7 +118,7 @@ mod tests {
#[test]
fn test_build_env_filter_debug_no_suppression() {
// For debug level, our code does NOT inject any OFF directives.
let filter = build_env_filter("debug", None);
let filter = build_env_filter("info", Some("debug"));
let dbg = format!("{filter:?}");
// Verify the filter builds without panicking and contains the debug level.
assert!(!dbg.is_empty());
@@ -91,4 +127,11 @@ mod tests {
"Expected 'LevelFilter::DEBUG' in filter debug output: {dbg}"
);
}
#[test]
fn test_should_suppress_noisy_crates_respects_rust_log_debug() {
assert!(!should_suppress_noisy_crates("info", None, Some("debug")));
assert!(!should_suppress_noisy_crates("info", None, Some("s3=info,hyper=debug")));
assert!(should_suppress_noisy_crates("info", None, Some("info")));
}
}
+4 -3
View File
@@ -23,6 +23,7 @@
//! 4. Cleanup task — aborted to prevent lingering background work.
//! 5. Tracing worker guard — flushes buffered log lines written by
//! `tracing_appender`.
//! 6. Stdout worker guard — flushes buffered log lines written to stdout.
use opentelemetry_sdk::{logs::SdkLoggerProvider, metrics::SdkMeterProvider, trace::SdkTracerProvider};
@@ -39,13 +40,13 @@ pub struct OtelGuard {
pub(crate) meter_provider: Option<SdkMeterProvider>,
/// Optional logger provider for OTLP log export.
pub(crate) logger_provider: Option<SdkLoggerProvider>,
/// Handle to the background log-cleanup task; aborted on drop.
pub(crate) cleanup_handle: Option<tokio::task::JoinHandle<()>>,
/// Worker guard that keeps the non-blocking `tracing_appender` thread
/// alive. Dropping it blocks until all buffered records are flushed.
pub(crate) tracing_guard: Option<tracing_appender::non_blocking::WorkerGuard>,
/// Optional guard for stdout logging; kept separate to allow independent flushing and shutdown.
pub(crate) stdout_guard: Option<tracing_appender::non_blocking::WorkerGuard>,
/// Handle to the background log-cleanup task; aborted on drop.
pub(crate) cleanup_handle: Option<tokio::task::JoinHandle<()>>,
}
impl std::fmt::Debug for OtelGuard {
@@ -54,9 +55,9 @@ impl std::fmt::Debug for OtelGuard {
.field("tracer_provider", &self.tracer_provider.is_some())
.field("meter_provider", &self.meter_provider.is_some())
.field("logger_provider", &self.logger_provider.is_some())
.field("cleanup_handle", &self.cleanup_handle.is_some())
.field("tracing_guard", &self.tracing_guard.is_some())
.field("stdout_guard", &self.stdout_guard.is_some())
.field("cleanup_handle", &self.cleanup_handle.is_some())
.finish()
}
}
+3 -4
View File
@@ -195,7 +195,7 @@ fn init_file_logging_internal(
let mut builder = RollingFileAppender::builder()
.rotation(rotation)
.max_log_files(keep_files * 2); // Make sure there are some data files to archive to avoid premature deletion
.max_log_files(keep_files * 3); // Make sure there are some data files to archive to avoid premature deletion
match match_mode {
FileMatchMode::Prefix => builder = builder.filename_prefix(log_filename),
@@ -355,10 +355,9 @@ fn spawn_cleanup_task(
_ => FileMatchMode::Suffix,
};
let keep_count = config.log_keep_count.unwrap_or(keep_files);
let max_total_size = config
.log_max_total_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES * keep_count as u64);
.unwrap_or(DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES);
let max_single_file_size = config
.log_max_single_file_size_bytes
.unwrap_or(DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES);
@@ -387,7 +386,7 @@ fn spawn_cleanup_task(
log_dir,
file_pattern,
match_mode,
keep_count,
keep_files,
max_total_size,
max_single_file_size,
compress,
+65 -4
View File
@@ -12,7 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::env;
use std::{
collections::HashSet,
env,
sync::{Mutex, OnceLock},
};
use tracing::warn;
/// Retrieve an environment variable as a specific type, with a default value if not set or parsing fails.
/// 8-bit type: signed i8
@@ -68,6 +73,56 @@ pub fn get_env_opt_u8(key: &str) -> Option<u8> {
env::var(key).ok().and_then(|v| v.parse().ok())
}
static WARNED_ENV_MESSAGES: OnceLock<Mutex<HashSet<String>>> = OnceLock::new();
fn log_once(key: &str, message: impl FnOnce() -> String) {
let seen = WARNED_ENV_MESSAGES.get_or_init(|| Mutex::new(HashSet::new()));
let mut seen = match seen.lock() {
Ok(seen) => seen,
Err(poisoned) => poisoned.into_inner(),
};
if seen.insert(key.to_string()) {
warn!("{}", message());
}
}
fn resolve_env_with_aliases(key: &str, deprecated: &[&str]) -> Option<(String, String)> {
if let Ok(value) = env::var(key) {
return Some((key.to_string(), value));
}
let (alias, value) = deprecated
.iter()
.find_map(|alias| env::var(alias).ok().map(|value| (*alias, value)))?;
let deprecated_key = format!("env_alias:{alias}->{key}");
log_once(&deprecated_key, || {
format!("Environment variable {alias} is deprecated, use {key} instead")
});
Some((alias.to_string(), value))
}
pub fn get_env_str_with_aliases(key: &str, deprecated: &[&str], default: &str) -> String {
resolve_env_with_aliases(key, deprecated).map_or_else(|| default.to_string(), |(_, value)| value)
}
pub fn get_env_bool_with_aliases(key: &str, deprecated: &[&str], default: bool) -> bool {
let Some((used_key, value)) = resolve_env_with_aliases(key, deprecated) else {
return default;
};
parse_bool_str(&value).unwrap_or_else(|| {
log_once(
&format!("env_invalid_bool:{used_key}"),
|| {
format!(
"Invalid bool value for {used_key}: {value}. Supported values are true/false,1/0,yes/no,on/off. Treating as unset."
)
},
);
default
})
}
/// Retrieve an environment variable as a specific type, with a default value if not set or parsing fails.
/// 16-bit type: signed i16
///
@@ -338,7 +393,7 @@ pub fn get_env_opt_usize(key: &str) -> Option<usize> {
/// - `String`: The environment variable value if set, otherwise the default value.
///
pub fn get_env_str(key: &str, default: &str) -> String {
env::var(key).unwrap_or_else(|_| default.to_string())
get_env_str_with_aliases(key, &[], default)
}
/// Retrieve an environment variable as a String, returning None if not set.
@@ -363,7 +418,7 @@ pub fn get_env_opt_str(key: &str) -> Option<String> {
/// - `bool`: The parsed boolean value if successful, otherwise the default value.
///
pub fn get_env_bool(key: &str, default: bool) -> bool {
env::var(key).ok().and_then(|v| parse_bool_str(&v)).unwrap_or(default)
get_env_bool_with_aliases(key, &[], default)
}
/// Parse a string into a boolean value.
@@ -391,5 +446,11 @@ fn parse_bool_str(s: &str) -> Option<bool> {
/// - `Option<bool>`: The parsed boolean value if successful, otherwise None.
///
pub fn get_env_opt_bool(key: &str) -> Option<bool> {
env::var(key).ok().and_then(|v| parse_bool_str(&v))
let (used_key, value) = resolve_env_with_aliases(key, &[])?;
parse_bool_str(&value).or_else(|| {
log_once(&format!("env_invalid_bool_optional:{used_key}"), || {
format!("Invalid bool value for {used_key}: {value}. Supported values are true/false,1/0,yes/no,on/off.")
});
None
})
}
+31 -113
View File
@@ -18,13 +18,10 @@ use crate::app::context::{AppContext, default_notify_interface, get_global_app_c
use crate::auth::get_condition_values;
use crate::error::ApiError;
use crate::server::RemoteAddr;
use crate::storage::access::{ReqInfo, authorize_request};
use crate::storage::ecfs::{
RUSTFS_OWNER, default_owner, is_public_grant, parse_acl_json_or_canned_bucket, serialize_acl, stored_acl_from_canned_bucket,
stored_acl_from_grant_headers, stored_acl_from_policy, stored_grant_to_dto, stored_owner_to_dto,
};
use crate::storage::access::{ReqInfo, authorize_request, req_info_ref};
use crate::storage::ecfs::{RUSTFS_OWNER, default_owner};
use crate::storage::helper::OperationHelper;
use crate::storage::s3_api::{encryption, replication, tagging};
use crate::storage::s3_api::{acl, encryption, replication, tagging};
use crate::storage::*;
use futures::StreamExt;
use http::StatusCode;
@@ -33,7 +30,7 @@ use rustfs_config::RUSTFS_REGION;
use rustfs_ecstore::bucket::{
lifecycle::bucket_lifecycle_ops::validate_transition_tier,
metadata::{
BUCKET_ACL_CONFIG, BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG,
BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG,
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, BUCKET_REPLICATION_CONFIG, BUCKET_SSECONFIG, BUCKET_TAGGING_CONFIG,
BUCKET_VERSIONING_CONFIG,
},
@@ -150,14 +147,14 @@ impl DefaultBucketUsecase {
}
let helper = OperationHelper::new(&req, EventName::BucketCreated, "s3:CreateBucket");
let requester_id = match req_info_ref(&req) {
Ok(r) => r.cred.as_ref().map(|c| c.access_key.clone()),
Err(_) => {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Missing request info".to_string()));
}
};
let CreateBucketInput {
bucket,
acl,
grant_full_control,
grant_read,
grant_read_acp,
grant_write,
grant_write_acp,
object_lock_enabled_for_bucket,
..
} = req.input;
@@ -182,37 +179,26 @@ impl DefaultBucketUsecase {
match make_result {
Ok(()) => {}
Err(StorageError::BucketExists(_)) => {
// Per S3 spec: CreateBucket for a bucket you already own returns 200 OK
let output = CreateBucketOutput::default();
let result = Ok(S3Response::new(output));
// Per S3 spec: bucket namespace is global. Owner recreating returns 200 OK;
// non-owner gets 409 BucketAlreadyExists.
let is_owner = requester_id.as_deref().is_some_and(|req_id| req_id == default_owner().id);
if is_owner {
let output = CreateBucketOutput::default();
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
return result;
}
let result = Err(s3_error!(
BucketAlreadyExists,
"The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again."
));
let _ = helper.complete(&result);
return result;
}
Err(e) => return Err(ApiError::from(e).into()),
}
let owner = default_owner();
let mut stored_acl = stored_acl_from_grant_headers(
&owner,
grant_read.map(|v| v.to_string()),
grant_write.map(|v| v.to_string()),
grant_read_acp.map(|v| v.to_string()),
grant_write_acp.map(|v| v.to_string()),
grant_full_control.map(|v| v.to_string()),
)?;
if stored_acl.is_none()
&& let Some(canned) = acl
{
stored_acl = Some(stored_acl_from_canned_bucket(canned.as_str(), &owner));
}
let stored_acl = stored_acl.unwrap_or_else(|| stored_acl_from_canned_bucket(BucketCannedACL::PRIVATE, &owner));
let data = serialize_acl(&stored_acl)?;
metadata_sys::update(&bucket, BUCKET_ACL_CONFIG, data)
.await
.map_err(ApiError::from)?;
let output = CreateBucketOutput::default();
let result = Ok(S3Response::new(output));
@@ -227,13 +213,7 @@ impl DefaultBucketUsecase {
let PutBucketAclInput {
bucket,
acl,
access_control_policy,
grant_full_control,
grant_read,
grant_read_acp,
grant_write,
grant_write_acp,
..
} = req.input;
@@ -246,43 +226,13 @@ impl DefaultBucketUsecase {
.await
.map_err(ApiError::from)?;
let owner = default_owner();
let mut stored_acl = access_control_policy
.as_ref()
.map(|policy| stored_acl_from_policy(policy, &owner))
.transpose()?;
if stored_acl.is_none() {
stored_acl = stored_acl_from_grant_headers(
&owner,
grant_read.map(|v| v.to_string()),
grant_write.map(|v| v.to_string()),
grant_read_acp.map(|v| v.to_string()),
grant_write_acp.map(|v| v.to_string()),
grant_full_control.map(|v| v.to_string()),
)?;
if access_control_policy.is_some() {
return Err(s3_error!(
NotImplemented,
"ACL XML grants are not supported; use canned ACL headers or omit ACL"
));
}
if stored_acl.is_none()
&& let Some(canned) = acl
{
stored_acl = Some(stored_acl_from_canned_bucket(canned.as_str(), &owner));
}
let stored_acl = stored_acl.unwrap_or_else(|| stored_acl_from_canned_bucket(BucketCannedACL::PRIVATE, &owner));
if let Ok((config, _)) = metadata_sys::get_public_access_block_config(&bucket).await
&& config.block_public_acls.unwrap_or(false)
&& stored_acl.grants.iter().any(is_public_grant)
{
return Err(s3_error!(AccessDenied, "Access Denied"));
}
let data = serialize_acl(&stored_acl)?;
metadata_sys::update(&bucket, BUCKET_ACL_CONFIG, data)
.await
.map_err(ApiError::from)?;
Ok(S3Response::new(PutBucketAclOutput::default()))
}
@@ -369,25 +319,7 @@ impl DefaultBucketUsecase {
.await
.map_err(ApiError::from)?;
let owner = default_owner();
let stored_acl = match metadata_sys::get_bucket_acl_config(&bucket).await {
Ok((acl, _)) => parse_acl_json_or_canned_bucket(&acl, &owner),
Err(err) => {
if err != StorageError::ConfigNotFound {
return Err(ApiError::from(err).into());
}
stored_acl_from_canned_bucket(BucketCannedACL::PRIVATE, &owner)
}
};
let mut sorted_grants = stored_acl.grants.clone();
sorted_grants.sort_by_key(|grant| grant.grantee.grantee_type != "Group");
let grants = sorted_grants.iter().map(stored_grant_to_dto).collect();
Ok(S3Response::new(GetBucketAclOutput {
grants: Some(grants),
owner: Some(stored_owner_to_dto(&stored_acl.owner)),
}))
Ok(S3Response::new(acl::build_get_bucket_acl_output()))
}
#[instrument(level = "debug", skip(self, req))]
@@ -899,21 +831,7 @@ impl DefaultBucketUsecase {
Err(_) => false,
};
let owner = default_owner();
let acl_public = match metadata_sys::get_bucket_acl_config(&bucket).await {
Ok((acl, _)) => {
let stored_acl = parse_acl_json_or_canned_bucket(&acl, &owner);
stored_acl
.grants
.iter()
.any(|grant| is_public_grant(grant) && !ignore_public_acls)
}
Err(_) => false,
};
if acl_public {
is_public = true;
}
let _ = ignore_public_acls;
let policy_public = match metadata_sys::get_bucket_policy(&bucket).await {
Ok((cfg, _)) => cfg.statements.iter().any(|statement| {
+3 -2
View File
@@ -21,7 +21,8 @@ use crate::storage::ecfs::RUSTFS_OWNER;
use crate::storage::entity;
use crate::storage::helper::OperationHelper;
use crate::storage::options::{
copy_src_opts, extract_metadata, get_complete_multipart_upload_opts, get_content_sha256, parse_copy_source_range, put_opts,
copy_src_opts, extract_metadata, get_complete_multipart_upload_opts, get_content_sha256_with_query, parse_copy_source_range,
put_opts,
};
use crate::storage::*;
use bytes::Bytes;
@@ -615,7 +616,7 @@ impl DefaultMultipartUsecase {
None
};
let mut sha256hex = get_content_sha256(&req.headers);
let mut sha256hex = get_content_sha256_with_query(&req.headers, req.uri.query());
if is_compressible {
let mut hrd = HashReader::new(reader, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?;
+13 -130
View File
@@ -17,7 +17,7 @@
use crate::app::context::{AppContext, default_notify_interface, get_global_app_context};
use crate::config::workload_profiles::RustFSBufferConfig;
use crate::error::ApiError;
use crate::storage::access::{ReqInfo, authorize_request, has_bypass_governance_header, req_info_mut};
use crate::storage::access::{authorize_request, has_bypass_governance_header, req_info_mut};
use crate::storage::concurrency::{
CachedGetObject, ConcurrencyManager, GetObjectGuard, get_concurrency_aware_buffer_size, get_concurrency_manager,
};
@@ -26,10 +26,10 @@ use crate::storage::head_prefix::{head_prefix_not_found_message, probe_prefix_ha
use crate::storage::helper::OperationHelper;
use crate::storage::options::{
copy_dst_opts, copy_src_opts, del_opts, extract_metadata, extract_metadata_from_mime_with_object_name,
filter_object_metadata, get_content_sha256, get_opts, put_opts,
filter_object_metadata, get_content_sha256_with_query, get_opts, put_opts,
};
use crate::storage::s3_api::multipart::parse_list_parts_params;
use crate::storage::s3_api::{restore, select};
use crate::storage::s3_api::{acl, restore, select};
use crate::storage::*;
use bytes::Bytes;
use datafusion::arrow::{
@@ -287,11 +287,6 @@ impl DefaultObjectUsecase {
body,
bucket,
key,
acl,
grant_full_control,
grant_read,
grant_read_acp,
grant_write_acp,
content_length,
content_type,
tagging,
@@ -318,14 +313,6 @@ impl DefaultObjectUsecase {
// Validate object key
validate_object_key(&key, "PUT")?;
if let Some(acl) = &acl
&& let Ok((config, _)) = metadata_sys::get_public_access_block_config(&bucket).await
&& config.block_public_acls.unwrap_or(false)
&& is_public_canned_acl(acl.as_str())
{
return Err(s3_error!(AccessDenied, "Access Denied"));
}
if let Some(size) = content_length {
self.check_bucket_quota(&bucket, QuotaOperation::PutObject, size as u64)
.await?;
@@ -407,33 +394,6 @@ impl DefaultObjectUsecase {
)?;
let mut metadata = metadata.unwrap_or_default();
let owner = req
.extensions
.get::<ReqInfo>()
.and_then(|info| info.cred.as_ref())
.map(|cred| owner_from_access_key(&cred.access_key))
.unwrap_or_else(default_owner);
let bucket_owner = default_owner();
let mut stored_acl = stored_acl_from_grant_headers(
&owner,
grant_read.map(|v| v.to_string()),
None,
grant_read_acp.map(|v| v.to_string()),
grant_write_acp.map(|v| v.to_string()),
grant_full_control.map(|v| v.to_string()),
)?;
if stored_acl.is_none()
&& let Some(canned) = acl.as_ref()
{
stored_acl = Some(stored_acl_from_canned_object(canned.as_str(), &bucket_owner, &owner));
}
let stored_acl =
stored_acl.unwrap_or_else(|| stored_acl_from_canned_object(ObjectCannedACL::PRIVATE, &bucket_owner, &owner));
let acl_data = serialize_acl(&stored_acl)?;
metadata.insert(INTERNAL_ACL_METADATA_KEY.to_string(), String::from_utf8_lossy(&acl_data).to_string());
if let Some(content_type) = content_type {
metadata.insert("content-type".to_string(), content_type.to_string());
}
@@ -461,7 +421,7 @@ impl DefaultObjectUsecase {
None
};
let mut sha256hex = get_content_sha256(&req.headers);
let mut sha256hex = get_content_sha256_with_query(&req.headers, req.uri.query());
if is_compressible(&req.headers, &key) && size > MIN_COMPRESSIBLE_SIZE as i64 {
let algorithm = CompressionAlgorithm::default();
@@ -622,13 +582,7 @@ impl DefaultObjectUsecase {
let PutObjectAclInput {
bucket,
key,
acl,
access_control_policy,
grant_full_control,
grant_read,
grant_read_acp,
grant_write,
grant_write_acp,
version_id,
..
} = req.input;
@@ -640,64 +594,15 @@ impl DefaultObjectUsecase {
let opts: ObjectOptions = get_opts(&bucket, &key, version_id.clone(), None, &req.headers)
.await
.map_err(ApiError::from)?;
let info = store.get_object_info(&bucket, &key, &opts).await.map_err(ApiError::from)?;
store.get_object_info(&bucket, &key, &opts).await.map_err(ApiError::from)?;
let bucket_owner = default_owner();
let existing_owner = info
.user_defined
.get(INTERNAL_ACL_METADATA_KEY)
.and_then(|acl| serde_json::from_str::<StoredAcl>(acl).ok())
.map(|acl| acl.owner)
.unwrap_or_else(|| bucket_owner.clone());
let mut stored_acl = access_control_policy
.as_ref()
.map(|policy| stored_acl_from_policy(policy, &existing_owner))
.transpose()?;
if stored_acl.is_none() {
stored_acl = stored_acl_from_grant_headers(
&existing_owner,
grant_read.map(|v| v.to_string()),
grant_write.map(|v| v.to_string()),
grant_read_acp.map(|v| v.to_string()),
grant_write_acp.map(|v| v.to_string()),
grant_full_control.map(|v| v.to_string()),
)?;
if access_control_policy.is_some() {
return Err(s3_error!(
NotImplemented,
"ACL XML grants are not supported; use canned ACL headers or omit ACL"
));
}
if stored_acl.is_none()
&& let Some(canned) = acl
{
stored_acl = Some(stored_acl_from_canned_object(canned.as_str(), &bucket_owner, &existing_owner));
}
let stored_acl =
stored_acl.unwrap_or_else(|| stored_acl_from_canned_object(ObjectCannedACL::PRIVATE, &bucket_owner, &existing_owner));
if let Ok((config, _)) = metadata_sys::get_public_access_block_config(&bucket).await
&& config.block_public_acls.unwrap_or(false)
&& stored_acl.grants.iter().any(is_public_grant)
{
return Err(s3_error!(AccessDenied, "Access Denied"));
}
let acl_data = serialize_acl(&stored_acl)?;
let mut eval_metadata = HashMap::new();
eval_metadata.insert(INTERNAL_ACL_METADATA_KEY.to_string(), String::from_utf8_lossy(&acl_data).to_string());
let popts = ObjectOptions {
mod_time: info.mod_time,
version_id: opts.version_id,
eval_metadata: Some(eval_metadata),
..Default::default()
};
store.put_object_metadata(&bucket, &key, &popts).await.map_err(|e| {
error!("put_object_metadata failed, {}", e.to_string());
s3_error!(InternalError, "{}", e.to_string())
})?;
Ok(S3Response::new(PutObjectAclOutput::default()))
}
@@ -1606,31 +1511,9 @@ impl DefaultObjectUsecase {
let opts: ObjectOptions = get_opts(&bucket, &key, version_id.clone(), None, &req.headers)
.await
.map_err(ApiError::from)?;
let info = store.get_object_info(&bucket, &key, &opts).await.map_err(ApiError::from)?;
store.get_object_info(&bucket, &key, &opts).await.map_err(ApiError::from)?;
let bucket_owner = default_owner();
let object_owner = info
.user_defined
.get(INTERNAL_ACL_METADATA_KEY)
.and_then(|acl| serde_json::from_str::<StoredAcl>(acl).ok())
.map(|acl| acl.owner)
.unwrap_or_else(default_owner);
let stored_acl = info
.user_defined
.get(INTERNAL_ACL_METADATA_KEY)
.map(|acl| parse_acl_json_or_canned_object(acl, &bucket_owner, &object_owner))
.unwrap_or_else(|| stored_acl_from_canned_object(ObjectCannedACL::PRIVATE, &bucket_owner, &object_owner));
let mut sorted_grants = stored_acl.grants.clone();
sorted_grants.sort_by_key(|grant| grant.grantee.grantee_type != "Group");
let grants = sorted_grants.iter().map(stored_grant_to_dto).collect();
Ok(S3Response::new(GetObjectAclOutput {
grants: Some(grants),
owner: Some(stored_owner_to_dto(&stored_acl.owner)),
..Default::default()
}))
Ok(S3Response::new(acl::build_get_object_acl_output()))
}
pub async fn execute_get_object_attributes(
@@ -3521,7 +3404,7 @@ impl DefaultObjectUsecase {
None
};
let sha256hex = get_content_sha256(&req.headers);
let sha256hex = get_content_sha256_with_query(&req.headers, req.uri.query());
let actual_size = size;
let reader: Box<dyn Reader> = Box::new(WarpReader::new(body));
+90 -8
View File
@@ -397,6 +397,29 @@ pub fn get_condition_values(
version_id: Option<&str>,
region: Option<s3s::region::Region>,
remote_addr: Option<std::net::SocketAddr>,
) -> HashMap<String, Vec<String>> {
get_condition_values_with_query(header, cred, version_id, region, remote_addr, None)
}
/// Get condition values for policy evaluation with optional query-string values.
///
/// # Arguments
/// * `header` - HTTP headers of the request
/// * `cred` - User credentials
/// * `version_id` - Optional version ID of the object
/// * `region` - Optional region/location constraint
/// * `remote_addr` - Optional remote address of the connection
/// * `query` - Optional request query string
///
/// # Returns
/// * `HashMap<String, Vec<String>>` - Condition values for policy evaluation
pub fn get_condition_values_with_query(
header: &HeaderMap,
cred: &Credentials,
version_id: Option<&str>,
region: Option<s3s::region::Region>,
remote_addr: Option<std::net::SocketAddr>,
query: Option<&str>,
) -> HashMap<String, Vec<String>> {
let username = if cred.is_temp() || cred.is_service_account() {
cred.parent_user.clone()
@@ -427,8 +450,8 @@ pub fn get_condition_values(
// Use provided version ID or empty string
let vid = version_id.unwrap_or("");
// Determine auth type and signature version from headers
let (auth_type, signature_version) = determine_auth_type_and_version(header);
// Determine auth type and signature version from headers and query
let (auth_type, signature_version) = determine_auth_type_and_version_with_query(header, query);
// Get TLS status from header
let is_tls = header
@@ -582,10 +605,16 @@ pub fn get_condition_values(
/// # Returns
/// * `AuthType` - The determined authentication type
///
#[allow(dead_code)]
pub fn get_request_auth_type(header: &HeaderMap) -> AuthType {
get_request_auth_type_with_query(header, None)
}
#[allow(dead_code)]
pub(crate) fn get_request_auth_type_with_query(header: &HeaderMap, query: Option<&str>) -> AuthType {
if is_request_signature_v2(header) {
AuthType::SignedV2
} else if is_request_presigned_signature_v2(header) {
} else if is_request_presigned_signature_v2(header, query) {
AuthType::PresignedV2
} else if is_request_sign_streaming_v4(header) {
AuthType::StreamingSigned
@@ -595,7 +624,7 @@ pub fn get_request_auth_type(header: &HeaderMap) -> AuthType {
AuthType::StreamingUnsignedTrailer
} else if is_request_signature_v4(header) {
AuthType::Signed
} else if is_request_presigned_signature_v4(header) {
} else if is_request_presigned_signature_v4_with_query(header, query) {
AuthType::Presigned
} else if is_request_jwt(header) {
AuthType::JWT
@@ -618,8 +647,14 @@ pub fn get_request_auth_type(header: &HeaderMap) -> AuthType {
/// # Returns
/// * `(String, String)` - Tuple of auth type and signature version
///
#[allow(dead_code)]
fn determine_auth_type_and_version(header: &HeaderMap) -> (String, String) {
match get_request_auth_type(header) {
determine_auth_type_and_version_with_query(header, None)
}
#[allow(dead_code)]
fn determine_auth_type_and_version_with_query(header: &HeaderMap, query: Option<&str>) -> (String, String) {
match get_request_auth_type_with_query(header, query) {
AuthType::JWT => ("JWT".to_string(), String::new()),
AuthType::SignedV2 => ("REST-HEADER".to_string(), "AWS2".to_string()),
AuthType::PresignedV2 => ("REST-QUERY-STRING".to_string(), "AWS2".to_string()),
@@ -690,11 +725,18 @@ fn is_request_signature_v2(header: &HeaderMap) -> bool {
///
/// # Returns
/// * `bool` - True if request has AWS PreSign Version '4', false otherwise
#[allow(dead_code)]
pub(crate) fn is_request_presigned_signature_v4(header: &HeaderMap) -> bool {
is_request_presigned_signature_v4_with_query(header, None)
}
pub(crate) fn is_request_presigned_signature_v4_with_query(header: &HeaderMap, query: Option<&str>) -> bool {
if let Some(credential) = header.get(AMZ_CREDENTIAL) {
return !credential.to_str().unwrap_or("").is_empty();
}
false
query
.and_then(|query| get_query_param(query, "x-amz-credential"))
.is_some_and(|credential| !credential.is_empty())
}
/// Verify request has AWS PreSign Version '2'
@@ -704,11 +746,13 @@ pub(crate) fn is_request_presigned_signature_v4(header: &HeaderMap) -> bool {
///
/// # Returns
/// * `bool` - True if request has AWS PreSign Version '2', false otherwise
fn is_request_presigned_signature_v2(header: &HeaderMap) -> bool {
fn is_request_presigned_signature_v2(header: &HeaderMap, query: Option<&str>) -> bool {
if let Some(access_key) = header.get(AMZ_ACCESS_KEY_ID) {
return !access_key.to_str().unwrap_or("").is_empty();
}
false
query
.and_then(|query| get_query_param(query, "awsaccesskeyid"))
.is_some_and(|access_key| !access_key.is_empty())
}
/// Verify if request has AWS Post policy Signature Version '4'
@@ -1016,6 +1060,20 @@ mod tests {
assert_eq!(conditions.get("principaltype"), Some(&vec!["User".to_string()]));
}
#[test]
fn test_get_condition_values_with_presigned_query() {
let cred = create_test_credentials();
let headers = HeaderMap::new();
let uri: Uri = "https://example.com/?X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request"
.parse()
.unwrap();
let conditions = get_condition_values_with_query(&headers, &cred, None, None, None, uri.query());
assert_eq!(conditions.get("signatureversion"), Some(&vec!["AWS4-HMAC-SHA256".to_string()]));
assert_eq!(conditions.get("authType"), Some(&vec!["REST-QUERY-STRING".to_string()]));
}
#[test]
fn test_get_condition_values_temp_user() {
let cred = create_temp_credentials();
@@ -1284,6 +1342,18 @@ mod tests {
assert_eq!(auth_type, AuthType::PresignedV2);
}
#[test]
fn test_get_request_auth_type_presigned_v2_from_query() {
let headers = HeaderMap::new();
let uri: Uri = "https://example.com/?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Signature=example&Expires=1672531200"
.parse()
.unwrap();
let auth_type = get_request_auth_type_with_query(&headers, uri.query());
assert_eq!(auth_type, AuthType::PresignedV2);
}
#[test]
fn test_get_request_auth_type_presigned_v4() {
let mut headers = HeaderMap::new();
@@ -1297,6 +1367,18 @@ mod tests {
assert_eq!(auth_type, AuthType::Presigned);
}
#[test]
fn test_get_request_auth_type_presigned_v4_from_query() {
let headers = HeaderMap::new();
let uri: Uri = "https://example.com/?X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request"
.parse()
.unwrap();
let auth_type = get_request_auth_type_with_query(&headers, uri.query());
assert_eq!(auth_type, AuthType::Presigned);
}
#[test]
fn test_get_request_auth_type_post_policy() {
let mut headers = HeaderMap::new();
+10 -5
View File
@@ -66,12 +66,17 @@ use rustfs_iam::{init_iam_sys, init_oidc_sys};
use rustfs_metrics::init_metrics_system;
use rustfs_obs::{init_obs, set_global_guard};
use rustfs_scanner::init_data_scanner;
use rustfs_utils::net::parse_and_resolve_address;
use rustfs_utils::{get_env_bool_with_aliases, net::parse_and_resolve_address};
use std::io::{Error, Result};
use std::sync::Arc;
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, info, instrument, warn};
const ENV_SCANNER_ENABLED: &str = "RUSTFS_SCANNER_ENABLED";
const ENV_SCANNER_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_SCANNER";
const ENV_HEAL_ENABLED: &str = "RUSTFS_HEAL_ENABLED";
const ENV_HEAL_ENABLED_DEPRECATED: &str = "RUSTFS_ENABLE_HEAL";
#[cfg(all(target_os = "linux", target_env = "gnu", target_arch = "x86_64"))]
#[global_allocator]
static GLOBAL: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
@@ -404,8 +409,8 @@ async fn run(config: config::Config) -> Result<()> {
let _ = create_ahm_services_cancel_token();
// Check environment variables to determine if scanner and heal should be enabled
let enable_scanner = rustfs_utils::get_env_bool("RUSTFS_ENABLE_SCANNER", true);
let enable_heal = rustfs_utils::get_env_bool("RUSTFS_ENABLE_HEAL", true);
let enable_scanner = get_env_bool_with_aliases(ENV_SCANNER_ENABLED, &[ENV_SCANNER_ENABLED_DEPRECATED], true);
let enable_heal = get_env_bool_with_aliases(ENV_HEAL_ENABLED, &[ENV_HEAL_ENABLED_DEPRECATED], true);
info!(
target: "rustfs::main::run",
@@ -501,8 +506,8 @@ async fn handle_shutdown(
state_manager.update(ServiceState::Stopping);
// Check environment variables to determine what services need to be stopped
let enable_scanner = rustfs_utils::get_env_bool("RUSTFS_ENABLE_SCANNER", true);
let enable_heal = rustfs_utils::get_env_bool("RUSTFS_ENABLE_HEAL", true);
let enable_scanner = get_env_bool_with_aliases(ENV_SCANNER_ENABLED, &[ENV_SCANNER_ENABLED_DEPRECATED], true);
let enable_heal = get_env_bool_with_aliases(ENV_HEAL_ENABLED, &[ENV_HEAL_ENABLED_DEPRECATED], true);
// Stop background services based on what was enabled
if enable_scanner || enable_heal {
+149 -19
View File
@@ -343,6 +343,29 @@ pub struct ConditionalCorsService<S> {
cors_origins: Arc<Option<String>>,
}
async fn resolve_s3_options_cors_headers(bucket: &str, request_headers: &HeaderMap) -> Option<HeaderMap> {
apply_cors_headers(bucket, &http::Method::OPTIONS, request_headers).await
}
fn clear_cors_response_headers(headers: &mut HeaderMap) {
headers.remove(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN);
headers.remove(cors::response::ACCESS_CONTROL_ALLOW_METHODS);
headers.remove(cors::response::ACCESS_CONTROL_ALLOW_HEADERS);
headers.remove(cors::response::ACCESS_CONTROL_EXPOSE_HEADERS);
headers.remove(cors::response::ACCESS_CONTROL_ALLOW_CREDENTIALS);
headers.remove(cors::response::ACCESS_CONTROL_MAX_AGE);
}
fn apply_bucket_cors_result(response_headers: &mut HeaderMap, bucket_cors_headers: &HeaderMap) {
// Bucket-level CORS is authoritative for S3 object/bucket paths.
// Clear any previously-populated CORS response headers (e.g. generic/system defaults),
// then apply the evaluated bucket result (which may be intentionally empty).
clear_cors_response_headers(response_headers);
for (key, value) in bucket_cors_headers.iter() {
response_headers.insert(key, value.clone());
}
}
impl<S, ResBody> Service<HttpRequest<Incoming>> for ConditionalCorsService<S>
where
S: Service<HttpRequest<Incoming>, Response = Response<ResBody>> + Clone + Send + 'static,
@@ -364,12 +387,32 @@ where
let request_headers = req.headers().clone();
let cors_origins = self.cors_origins.clone();
let is_s3 = ConditionalCorsLayer::is_s3_path(&path);
let is_root = path == "/";
if method == Method::OPTIONS {
let has_acrm = request_headers.contains_key(cors::request::ACCESS_CONTROL_REQUEST_METHOD);
if is_root {
return Box::pin(async move {
if !has_acrm || !request_headers.contains_key(cors::standard::ORIGIN) {
return Ok(Response::builder()
.status(StatusCode::BAD_REQUEST)
.body(ResBody::default())
.unwrap());
}
let mut response = Response::builder().status(StatusCode::OK).body(ResBody::default()).unwrap();
let cors_layer = ConditionalCorsLayer {
cors_origins: (*cors_origins).clone(),
};
cors_layer.apply_cors_headers(&request_headers, response.headers_mut());
Ok(response)
});
}
if is_s3 {
let path_trimmed = path.trim_start_matches('/');
let bucket = path_trimmed.split('/').next().unwrap_or("").to_string();
let has_acrm = request_headers.contains_key(cors::request::ACCESS_CONTROL_REQUEST_METHOD);
return Box::pin(async move {
if !has_acrm || !request_headers.contains_key(cors::standard::ORIGIN) {
@@ -379,20 +422,27 @@ where
.unwrap());
}
if !bucket.is_empty()
&& let Some(cors_headers) = apply_cors_headers(&bucket, &method, &request_headers).await
{
let mut response = Response::builder().status(StatusCode::OK).body(ResBody::default()).unwrap();
for (key, value) in cors_headers.iter() {
response.headers_mut().insert(key, value.clone());
let cors_layer = ConditionalCorsLayer {
cors_origins: (*cors_origins).clone(),
};
if let Some(cors_headers) = resolve_s3_options_cors_headers(&bucket, &request_headers).await {
let cors_allowed = cors_headers.contains_key(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN);
let status = if cors_allowed { StatusCode::OK } else { StatusCode::FORBIDDEN };
let mut response = Response::builder().status(status).body(ResBody::default()).unwrap();
if cors_allowed {
for (key, value) in cors_headers.iter() {
response.headers_mut().insert(key, value.clone());
}
}
return Ok(response);
}
Ok(Response::builder()
.status(StatusCode::FORBIDDEN)
.body(ResBody::default())
.unwrap())
// No bucket-level CORS config: fall back to global/default CORS behavior.
let mut response = Response::builder().status(StatusCode::OK).body(ResBody::default()).unwrap();
cors_layer.apply_cors_headers(&request_headers, response.headers_mut());
Ok(response)
});
}
@@ -415,19 +465,30 @@ where
if request_headers.contains_key(cors::standard::ORIGIN)
&& !response.headers().contains_key(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN)
{
let cors_layer = ConditionalCorsLayer {
cors_origins: (*cors_origins).clone(),
};
if is_s3 {
let bucket = path.trim_start_matches('/').split('/').next().unwrap_or("");
if !bucket.is_empty()
&& let Some(cors_headers) = apply_cors_headers(bucket, &method, &request_headers).await
{
for (key, value) in cors_headers.iter() {
response.headers_mut().insert(key, value.clone());
if path == "/" {
cors_layer.apply_cors_headers(&request_headers, response.headers_mut());
} else if !bucket.is_empty() {
match apply_cors_headers(bucket, &method, &request_headers).await {
Some(bucket_cors_headers) => {
// Bucket-level CORS is authoritative when configured, even if it
// intentionally resolves to an empty header set (no rule match).
apply_bucket_cors_result(response.headers_mut(), &bucket_cors_headers);
}
None => {
// No bucket-level CORS config: fall back to global/default policy.
cors_layer.apply_cors_headers(&request_headers, response.headers_mut());
}
}
} else {
cors_layer.apply_cors_headers(&request_headers, response.headers_mut());
}
} else {
let cors_layer = ConditionalCorsLayer {
cors_origins: (*cors_origins).clone(),
};
cors_layer.apply_cors_headers(&request_headers, response.headers_mut());
}
}
@@ -532,4 +593,73 @@ mod tests {
"https://allowed.com"
);
}
#[tokio::test]
async fn test_resolve_s3_options_cors_headers_no_headers_without_match() {
let mut req_headers = HeaderMap::new();
req_headers.insert("origin", "https://example.com".parse().unwrap());
req_headers.insert("access-control-request-method", "GET".parse().unwrap());
let headers = resolve_s3_options_cors_headers("bbb", &req_headers).await;
assert!(headers.is_none());
}
#[test]
fn test_apply_bucket_cors_result_clears_existing_cors_headers_with_empty_result() {
let mut response_headers = HeaderMap::new();
response_headers.insert(
cors::response::ACCESS_CONTROL_ALLOW_ORIGIN,
HeaderValue::from_static("https://foo.example"),
);
response_headers.insert(
cors::response::ACCESS_CONTROL_ALLOW_METHODS,
HeaderValue::from_static("GET, POST, PUT, DELETE, OPTIONS, HEAD"),
);
response_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_HEADERS, HeaderValue::from_static("*"));
response_headers.insert(cors::response::ACCESS_CONTROL_EXPOSE_HEADERS, HeaderValue::from_static("etag"));
response_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_CREDENTIALS, HeaderValue::from_static("true"));
response_headers.insert(cors::response::ACCESS_CONTROL_MAX_AGE, HeaderValue::from_static("3600"));
let bucket_cors_headers = HeaderMap::new();
apply_bucket_cors_result(&mut response_headers, &bucket_cors_headers);
assert!(response_headers.get(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN).is_none());
assert!(response_headers.get(cors::response::ACCESS_CONTROL_ALLOW_METHODS).is_none());
assert!(response_headers.get(cors::response::ACCESS_CONTROL_ALLOW_HEADERS).is_none());
assert!(response_headers.get(cors::response::ACCESS_CONTROL_EXPOSE_HEADERS).is_none());
assert!(
response_headers
.get(cors::response::ACCESS_CONTROL_ALLOW_CREDENTIALS)
.is_none()
);
assert!(response_headers.get(cors::response::ACCESS_CONTROL_MAX_AGE).is_none());
}
#[test]
fn test_apply_bucket_cors_result_replaces_existing_cors_headers() {
let mut response_headers = HeaderMap::new();
response_headers.insert(
cors::response::ACCESS_CONTROL_ALLOW_ORIGIN,
HeaderValue::from_static("https://foo.example"),
);
response_headers.insert(
cors::response::ACCESS_CONTROL_ALLOW_METHODS,
HeaderValue::from_static("GET, POST, PUT, DELETE, OPTIONS, HEAD"),
);
let mut bucket_cors_headers = HeaderMap::new();
bucket_cors_headers.insert(
cors::response::ACCESS_CONTROL_ALLOW_ORIGIN,
HeaderValue::from_static("https://allowed.example"),
);
bucket_cors_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_METHODS, HeaderValue::from_static("GET"));
apply_bucket_cors_result(&mut response_headers, &bucket_cors_headers);
assert_eq!(
response_headers.get(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
"https://allowed.example"
);
assert_eq!(response_headers.get(cors::response::ACCESS_CONTROL_ALLOW_METHODS).unwrap(), "GET");
}
}
+18 -234
View File
@@ -13,14 +13,7 @@
// limitations under the License.
use super::ecfs::FS;
use super::ecfs::{
ACL_GROUP_ALL_USERS, ACL_GROUP_AUTHENTICATED_USERS, INTERNAL_ACL_METADATA_KEY, StoredAcl, default_owner,
parse_acl_json_or_canned_bucket, parse_acl_json_or_canned_object, stored_acl_from_canned_bucket,
stored_acl_from_canned_object,
};
use super::options::get_opts;
use crate::auth::{check_key_valid, get_condition_values, get_session_token};
use crate::error::ApiError;
use crate::auth::{check_key_valid, get_condition_values_with_query, get_session_token};
use crate::license::license_check;
use crate::server::RemoteAddr;
use metrics::counter;
@@ -28,7 +21,7 @@ use rustfs_ecstore::bucket::metadata_sys;
use rustfs_ecstore::bucket::policy_sys::PolicySys;
use rustfs_ecstore::error::{StorageError, is_err_bucket_not_found};
use rustfs_ecstore::new_object_layer_fn;
use rustfs_ecstore::store_api::{BucketOperations, ObjectOperations};
use rustfs_ecstore::store_api::BucketOperations;
use rustfs_iam::error::Error as IamError;
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_policy::policy::{Args, BucketPolicyArgs};
@@ -48,165 +41,6 @@ pub(crate) struct ReqInfo {
pub region: Option<s3s::region::Region>,
}
#[derive(Clone, Copy)]
enum AclTarget {
Bucket,
Object,
}
#[derive(Clone, Copy)]
enum AclPermission {
Read,
Write,
ReadAcp,
WriteAcp,
}
fn acl_permission_for_action(action: &Action) -> Option<(AclTarget, AclPermission)> {
match action {
Action::S3Action(S3Action::ListBucketAction) => Some((AclTarget::Bucket, AclPermission::Read)),
Action::S3Action(S3Action::PutObjectAction) => Some((AclTarget::Bucket, AclPermission::Write)),
Action::S3Action(S3Action::GetBucketAclAction) => Some((AclTarget::Bucket, AclPermission::ReadAcp)),
Action::S3Action(S3Action::PutBucketAclAction) => Some((AclTarget::Bucket, AclPermission::WriteAcp)),
Action::S3Action(S3Action::GetObjectAction) => Some((AclTarget::Object, AclPermission::Read)),
Action::S3Action(S3Action::GetObjectAclAction) => Some((AclTarget::Object, AclPermission::ReadAcp)),
Action::S3Action(S3Action::PutObjectAclAction) => Some((AclTarget::Object, AclPermission::WriteAcp)),
_ => None,
}
}
fn permission_matches(grant_perm: &str, required: AclPermission) -> bool {
if grant_perm == Permission::FULL_CONTROL {
return true;
}
match required {
AclPermission::Read => grant_perm == Permission::READ,
AclPermission::Write => grant_perm == Permission::WRITE,
AclPermission::ReadAcp => grant_perm == Permission::READ_ACP,
AclPermission::WriteAcp => grant_perm == Permission::WRITE_ACP,
}
}
fn acl_allows(
acl: &StoredAcl,
user_id: Option<&str>,
is_authenticated: bool,
required: AclPermission,
ignore_public_acls: bool,
) -> bool {
for grant in &acl.grants {
if !permission_matches(grant.permission.as_str(), required) {
continue;
}
match grant.grantee.grantee_type.as_str() {
"CanonicalUser" => {
if user_id.is_some_and(|id| grant.grantee.id.as_deref() == Some(id)) {
return true;
}
}
"Group" => {
if ignore_public_acls {
continue;
}
if let Some(uri) = grant.grantee.uri.as_deref() {
if uri == ACL_GROUP_ALL_USERS {
return true;
}
if uri == ACL_GROUP_AUTHENTICATED_USERS && is_authenticated {
return true;
}
}
}
_ => {}
}
}
false
}
async fn load_bucket_acl(bucket: &str) -> S3Result<StoredAcl> {
let owner = default_owner();
match metadata_sys::get_bucket_acl_config(bucket).await {
Ok((acl, _)) => Ok(parse_acl_json_or_canned_bucket(&acl, &owner)),
Err(err) => {
if err == StorageError::ConfigNotFound {
Ok(stored_acl_from_canned_bucket(BucketCannedACL::PRIVATE, &owner))
} else {
Err(S3Error::with_message(S3ErrorCode::InternalError, err.to_string()))
}
}
}
}
async fn load_object_acl(bucket: &str, object: &str, version_id: Option<&str>, headers: &http::HeaderMap) -> S3Result<StoredAcl> {
let Some(store) = new_object_layer_fn() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let opts = get_opts(bucket, object, version_id.map(|v| v.to_string()), None, headers)
.await
.map_err(ApiError::from)?;
let info = store.get_object_info(bucket, object, &opts).await.map_err(ApiError::from)?;
let bucket_owner = default_owner();
let object_owner = info
.user_defined
.get(INTERNAL_ACL_METADATA_KEY)
.and_then(|acl| serde_json::from_str::<StoredAcl>(acl).ok())
.map(|acl| acl.owner)
.unwrap_or_else(default_owner);
Ok(info
.user_defined
.get(INTERNAL_ACL_METADATA_KEY)
.map(|acl| parse_acl_json_or_canned_object(acl, &bucket_owner, &object_owner))
.unwrap_or_else(|| stored_acl_from_canned_object(ObjectCannedACL::PRIVATE, &bucket_owner, &object_owner)))
}
async fn check_acl_access<T>(req: &S3Request<T>, req_info: &ReqInfo, action: &Action, policy_allowed: bool) -> S3Result<bool> {
let Some((target, permission)) = acl_permission_for_action(action) else {
return Ok(true);
};
// For object-level operations, do not bypass on account-level is_owner.
// The bucket/account owner must have explicit ACL grant to access objects owned by others.
let bypass_owner = match target {
AclTarget::Bucket => req_info.is_owner,
AclTarget::Object => false,
};
if bypass_owner || policy_allowed {
return Ok(true);
}
let bucket = req_info.bucket.as_deref().unwrap_or("");
if bucket.is_empty() {
return Ok(true);
}
let ignore_public_acls = match metadata_sys::get_public_access_block_config(bucket).await {
Ok((config, _)) => config.ignore_public_acls.unwrap_or(false),
Err(_) => false,
};
let user_id = req_info.cred.as_ref().map(|cred| cred.access_key.as_str());
let is_authenticated = user_id.is_some();
let acl = match target {
AclTarget::Bucket => load_bucket_acl(bucket).await?,
AclTarget::Object => {
let object = req_info.object.as_deref().unwrap_or("");
if object.is_empty() {
return Ok(true);
}
load_object_acl(bucket, object, req_info.version_id.as_deref(), &req.headers).await?
}
};
Ok(acl_allows(&acl, user_id, is_authenticated, permission, ignore_public_acls))
}
pub(crate) fn req_info_ref<T>(req: &S3Request<T>) -> S3Result<&ReqInfo> {
req.extensions
.get::<ReqInfo>()
@@ -276,7 +110,14 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
let default_claims = HashMap::new();
let claims = cred.claims.as_ref().unwrap_or(&default_claims);
let mut conditions = get_condition_values(&req.headers, cred, req_info.version_id.as_deref(), None, remote_addr);
let mut conditions = get_condition_values_with_query(
&req.headers,
cred,
req_info.version_id.as_deref(),
None,
remote_addr,
req.uri.query(),
);
// Merge object tag conditions; extend existing values if the same key exists (e.g. from get_condition_values).
if let Some(ref tags) = object_tag_conditions {
for (k, v) in &tags.0 {
@@ -292,7 +133,7 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
&& !PolicySys::is_allowed(&BucketPolicyArgs {
bucket: bucket_name,
action,
// Run this early check in deny-only mode so ACL/IAM fallbacks can still grant access.
// Run this early check in deny-only mode so IAM fallback can still grant access.
is_owner: true,
account: &cred.access_key,
groups: &cred.groups,
@@ -348,26 +189,7 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
.await;
if iam_allowed {
let policy_allowed = if !bucket_name.is_empty() {
PolicySys::is_allowed(&BucketPolicyArgs {
bucket: bucket_name,
action,
is_owner: false,
account: &cred.access_key,
groups: &cred.groups,
conditions: &conditions,
object: req_info.object.as_deref().unwrap_or(""),
})
.await
} else {
false
};
if check_acl_access(req, req_info, &action, policy_allowed).await? {
return Ok(());
}
return Err(s3_error!(AccessDenied, "Access Denied"));
return Ok(());
}
let policy_allowed_fallback = PolicySys::is_allowed(&BucketPolicyArgs {
@@ -382,13 +204,6 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
.await;
if policy_allowed_fallback {
// For object-level operations, bucket owner (is_owner) must still pass ACL check.
// Policy may have allowed due to is_owner, but object owner is authoritative.
if let Some((AclTarget::Object, _)) = acl_permission_for_action(&action)
&& !check_acl_access(req, req_info, &action, false).await?
{
return Err(s3_error!(AccessDenied, "Access Denied"));
}
return Ok(());
}
@@ -425,12 +240,13 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
}
}
} else {
let mut conditions = get_condition_values(
let mut conditions = get_condition_values_with_query(
&req.headers,
&rustfs_credentials::Credentials::default(),
req_info.version_id.as_deref(),
req.region.clone(),
remote_addr,
req.uri.query(),
);
// Merge object tag conditions; extend existing values if the same key exists.
if let Some(ref tags) = object_tag_conditions {
@@ -447,7 +263,7 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
&& !PolicySys::is_allowed(&BucketPolicyArgs {
bucket: bucket_name,
action,
// Run this early check in deny-only mode so ACL checks are not bypassed.
// Run this early check in deny-only mode so later policy checks are not bypassed.
is_owner: true,
account: "",
groups: &None,
@@ -473,13 +289,13 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
if policy_allowed {
// RestrictPublicBuckets: when true, deny public access even if bucket policy allows it.
// Fail closed: if we cannot read the config, do not allow public access.
match metadata_sys::get_public_access_block_config(bucket_name).await {
Ok((config, _)) => {
if config.restrict_public_buckets.unwrap_or(false) {
return Err(s3_error!(AccessDenied, "Access Denied"));
}
}
Err(StorageError::ConfigNotFound) => {}
Err(_) => {
return Err(s3_error!(AccessDenied, "Access Denied"));
}
@@ -501,10 +317,6 @@ pub async fn authorize_request<T>(req: &mut S3Request<T>, action: Action) -> S3R
{
return Ok(());
}
if acl_permission_for_action(&action).is_some() && check_acl_access(req, req_info, &action, false).await? {
return Ok(());
}
}
}
@@ -524,18 +336,10 @@ fn get_bucket_policy_authorize_action() -> Action {
Action::S3Action(S3Action::GetBucketPolicyAction)
}
fn get_object_acl_authorize_action() -> Action {
Action::S3Action(S3Action::GetObjectAclAction)
}
fn put_bucket_policy_authorize_action() -> Action {
Action::S3Action(S3Action::PutBucketPolicyAction)
}
fn put_object_acl_authorize_action() -> Action {
Action::S3Action(S3Action::PutObjectAclAction)
}
#[async_trait::async_trait]
impl S3Access for FS {
// /// Checks whether the current request has accesses to the resources.
@@ -1096,12 +900,7 @@ impl S3Access for FS {
req_info.object = Some(req.input.key.clone());
req_info.version_id = req.input.version_id.clone();
let tag_conds = self
.fetch_tag_conditions(&req.input.bucket, &req.input.key, req.input.version_id.as_deref(), "get_object_acl")
.await?;
req.extensions.insert(tag_conds);
authorize_request(req, get_object_acl_authorize_action()).await
authorize_request(req, Action::S3Action(S3Action::GetObjectAclAction)).await
}
/// Checks whether the GetObjectAttributes request has accesses to the resources.
@@ -1518,12 +1317,7 @@ impl S3Access for FS {
req_info.object = Some(req.input.key.clone());
req_info.version_id = req.input.version_id.clone();
let tag_conds = self
.fetch_tag_conditions(&req.input.bucket, &req.input.key, req.input.version_id.as_deref(), "put_object_acl")
.await?;
req.extensions.insert(tag_conds);
authorize_request(req, put_object_acl_authorize_action()).await
authorize_request(req, Action::S3Action(S3Action::PutObjectAclAction)).await
}
/// Checks whether the PutObjectLegalHold request has accesses to the resources.
@@ -1651,21 +1445,11 @@ mod tests {
assert_eq!(get_bucket_policy_authorize_action(), Action::S3Action(S3Action::GetBucketPolicyAction));
}
#[test]
fn get_object_acl_uses_get_object_acl_action() {
assert_eq!(get_object_acl_authorize_action(), Action::S3Action(S3Action::GetObjectAclAction));
}
#[test]
fn put_bucket_policy_uses_put_bucket_policy_action() {
assert_eq!(put_bucket_policy_authorize_action(), Action::S3Action(S3Action::PutBucketPolicyAction));
}
#[test]
fn put_object_acl_uses_put_object_acl_action() {
assert_eq!(put_object_acl_authorize_action(), Action::S3Action(S3Action::PutObjectAclAction));
}
/// Object tag conditions must use keys like ExistingObjectTag/<tag-key> so that
/// bucket policy conditions (e.g. s3:ExistingObjectTag/security) are evaluated correctly.
#[test]
+1 -497
View File
@@ -22,7 +22,6 @@ use rustfs_ecstore::{
store_api::{BucketOperations, BucketOptions, ObjectOperations, ObjectOptions},
};
use s3s::{S3, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, dto::*, s3_error};
use serde::{Deserialize, Serialize};
use std::{fmt::Debug, sync::LazyLock};
use tokio::io::{AsyncRead, AsyncSeek};
use tracing::{debug, error, instrument, warn};
@@ -30,511 +29,23 @@ use uuid::Uuid;
const DEFAULT_OWNER_ID: &str = "rustfsadmin";
const DEFAULT_OWNER_DISPLAY_NAME: &str = "RustFS Tester";
const DEFAULT_OWNER_EMAIL: &str = "tester@rustfs.local";
const DEFAULT_ALT_ID: &str = "rustfsalt";
const DEFAULT_ALT_DISPLAY_NAME: &str = "RustFS Alt Tester";
const DEFAULT_ALT_EMAIL: &str = "alt@rustfs.local";
pub(crate) const INTERNAL_ACL_METADATA_KEY: &str = "x-rustfs-internal-acl";
pub(crate) static RUSTFS_OWNER: LazyLock<Owner> = LazyLock::new(|| Owner {
display_name: Some(DEFAULT_OWNER_DISPLAY_NAME.to_owned()),
id: Some(DEFAULT_OWNER_ID.to_owned()),
});
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[derive(Clone, Debug, Default)]
pub(crate) struct StoredOwner {
pub(crate) id: String,
pub(crate) display_name: String,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
pub(crate) struct StoredGrantee {
pub(crate) grantee_type: String,
pub(crate) id: Option<String>,
pub(crate) display_name: Option<String>,
pub(crate) uri: Option<String>,
pub(crate) email_address: Option<String>,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
pub(crate) struct StoredGrant {
pub(crate) grantee: StoredGrantee,
pub(crate) permission: String,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
pub(crate) struct StoredAcl {
pub(crate) owner: StoredOwner,
pub(crate) grants: Vec<StoredGrant>,
}
pub(crate) fn default_owner() -> StoredOwner {
StoredOwner {
id: DEFAULT_OWNER_ID.to_string(),
display_name: DEFAULT_OWNER_DISPLAY_NAME.to_string(),
}
}
pub(crate) fn owner_from_access_key(access_key: &str) -> StoredOwner {
if access_key == DEFAULT_OWNER_ID {
return default_owner();
}
if access_key == DEFAULT_ALT_ID {
return StoredOwner {
id: DEFAULT_ALT_ID.to_string(),
display_name: DEFAULT_ALT_DISPLAY_NAME.to_string(),
};
}
StoredOwner {
id: access_key.to_string(),
display_name: access_key.to_string(),
}
}
fn user_id_from_email(email: &str) -> Option<StoredOwner> {
if email.eq_ignore_ascii_case(DEFAULT_OWNER_EMAIL) {
return Some(default_owner());
}
if email.eq_ignore_ascii_case(DEFAULT_ALT_EMAIL) {
return Some(StoredOwner {
id: DEFAULT_ALT_ID.to_string(),
display_name: DEFAULT_ALT_DISPLAY_NAME.to_string(),
});
}
None
}
fn display_name_for_user_id(user_id: &str) -> Option<String> {
if user_id == DEFAULT_OWNER_ID {
return Some(DEFAULT_OWNER_DISPLAY_NAME.to_string());
}
if user_id == DEFAULT_ALT_ID {
return Some(DEFAULT_ALT_DISPLAY_NAME.to_string());
}
None
}
fn is_known_user_id(user_id: &str) -> bool {
user_id == DEFAULT_OWNER_ID || user_id == DEFAULT_ALT_ID
}
pub(crate) fn stored_owner_to_dto(owner: &StoredOwner) -> Owner {
Owner {
id: Some(owner.id.clone()),
display_name: Some(owner.display_name.clone()),
}
}
pub(crate) fn stored_grant_to_dto(grant: &StoredGrant) -> Grant {
let grantee = match grant.grantee.grantee_type.as_str() {
"Group" => Grantee {
type_: Type::from_static(Type::GROUP),
display_name: None,
email_address: None,
id: None,
uri: grant.grantee.uri.clone(),
},
_ => Grantee {
type_: Type::from_static(Type::CANONICAL_USER),
display_name: grant.grantee.display_name.clone(),
email_address: None,
id: grant.grantee.id.clone(),
uri: None,
},
};
Grant {
grantee: Some(grantee),
permission: Some(Permission::from(grant.permission.clone())),
}
}
pub(crate) fn is_public_grant(grant: &StoredGrant) -> bool {
matches!(grant.grantee.grantee_type.as_str(), "Group")
&& grant
.grantee
.uri
.as_deref()
.is_some_and(|uri| uri == ACL_GROUP_ALL_USERS || uri == ACL_GROUP_AUTHENTICATED_USERS)
}
fn grants_for_canned_bucket_acl(acl: &str, owner: &StoredOwner) -> Vec<StoredGrant> {
let mut grants = Vec::new();
match acl {
BucketCannedACL::PUBLIC_READ => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_ALL_USERS.to_string()),
email_address: None,
},
permission: Permission::READ.to_string(),
});
}
BucketCannedACL::PUBLIC_READ_WRITE => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_ALL_USERS.to_string()),
email_address: None,
},
permission: Permission::READ.to_string(),
});
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_ALL_USERS.to_string()),
email_address: None,
},
permission: Permission::WRITE.to_string(),
});
}
BucketCannedACL::AUTHENTICATED_READ => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_AUTHENTICATED_USERS.to_string()),
email_address: None,
},
permission: Permission::READ.to_string(),
});
}
_ => {}
}
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(owner.id.clone()),
display_name: Some(owner.display_name.clone()),
uri: None,
email_address: None,
},
permission: Permission::FULL_CONTROL.to_string(),
});
grants
}
fn grants_for_canned_object_acl(acl: &str, bucket_owner: &StoredOwner, object_owner: &StoredOwner) -> Vec<StoredGrant> {
let mut grants = Vec::new();
match acl {
ObjectCannedACL::PUBLIC_READ => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_ALL_USERS.to_string()),
email_address: None,
},
permission: Permission::READ.to_string(),
});
}
ObjectCannedACL::PUBLIC_READ_WRITE => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_ALL_USERS.to_string()),
email_address: None,
},
permission: Permission::READ.to_string(),
});
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_ALL_USERS.to_string()),
email_address: None,
},
permission: Permission::WRITE.to_string(),
});
}
ObjectCannedACL::AUTHENTICATED_READ => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(ACL_GROUP_AUTHENTICATED_USERS.to_string()),
email_address: None,
},
permission: Permission::READ.to_string(),
});
}
ObjectCannedACL::BUCKET_OWNER_READ => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(bucket_owner.id.clone()),
display_name: Some(bucket_owner.display_name.clone()),
uri: None,
email_address: None,
},
permission: Permission::READ.to_string(),
});
}
ObjectCannedACL::BUCKET_OWNER_FULL_CONTROL => {
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(bucket_owner.id.clone()),
display_name: Some(bucket_owner.display_name.clone()),
uri: None,
email_address: None,
},
permission: Permission::FULL_CONTROL.to_string(),
});
}
_ => {}
}
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(object_owner.id.clone()),
display_name: Some(object_owner.display_name.clone()),
uri: None,
email_address: None,
},
permission: Permission::FULL_CONTROL.to_string(),
});
grants
}
pub(crate) fn stored_acl_from_canned_bucket(acl: &str, owner: &StoredOwner) -> StoredAcl {
StoredAcl {
owner: owner.clone(),
grants: grants_for_canned_bucket_acl(acl, owner),
}
}
pub(crate) fn stored_acl_from_canned_object(acl: &str, bucket_owner: &StoredOwner, object_owner: &StoredOwner) -> StoredAcl {
StoredAcl {
owner: object_owner.clone(),
grants: grants_for_canned_object_acl(acl, bucket_owner, object_owner),
}
}
pub(crate) fn parse_acl_json_or_canned_bucket(data: &str, owner: &StoredOwner) -> StoredAcl {
match serde_json::from_str::<StoredAcl>(data) {
Ok(acl) => acl,
Err(_) => stored_acl_from_canned_bucket(data, owner),
}
}
pub(crate) fn parse_acl_json_or_canned_object(data: &str, bucket_owner: &StoredOwner, object_owner: &StoredOwner) -> StoredAcl {
match serde_json::from_str::<StoredAcl>(data) {
Ok(acl) => acl,
Err(_) => stored_acl_from_canned_object(data, bucket_owner, object_owner),
}
}
pub(crate) fn serialize_acl(acl: &StoredAcl) -> std::result::Result<Vec<u8>, S3Error> {
serde_json::to_vec(acl).map_err(|e| s3_error!(InternalError, "serialize acl failed {e}"))
}
fn grantee_from_grant(grantee: &Grantee) -> Result<StoredGrantee, S3Error> {
match grantee.type_.as_str() {
Type::GROUP => {
let uri = grantee
.uri
.clone()
.filter(|uri| uri == ACL_GROUP_ALL_USERS || uri == ACL_GROUP_AUTHENTICATED_USERS)
.ok_or_else(|| s3_error!(InvalidArgument))?;
Ok(StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(uri),
email_address: None,
})
}
Type::CANONICAL_USER => {
let id = grantee.id.clone().ok_or_else(|| s3_error!(InvalidArgument))?;
if !is_known_user_id(&id) {
return Err(s3_error!(InvalidArgument));
}
let display_name = display_name_for_user_id(&id).or_else(|| grantee.display_name.clone());
Ok(StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(id),
display_name,
uri: None,
email_address: None,
})
}
Type::AMAZON_CUSTOMER_BY_EMAIL => {
let email = grantee.email_address.clone().ok_or_else(|| s3_error!(InvalidArgument))?;
let owner = user_id_from_email(&email).ok_or_else(|| s3_error!(UnresolvableGrantByEmailAddress))?;
Ok(StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(owner.id),
display_name: Some(owner.display_name),
uri: None,
email_address: None,
})
}
_ => Err(s3_error!(InvalidArgument)),
}
}
pub(crate) fn stored_acl_from_policy(policy: &AccessControlPolicy, owner_fallback: &StoredOwner) -> Result<StoredAcl, S3Error> {
let owner = policy
.owner
.as_ref()
.and_then(|owner| {
let id = owner.id.clone()?;
let display_name = owner.display_name.clone()?;
Some(StoredOwner { id, display_name })
})
.unwrap_or_else(|| owner_fallback.clone());
let grants = policy
.grants
.clone()
.unwrap_or_default()
.into_iter()
.map(|grant| {
let permission = grant
.permission
.clone()
.ok_or_else(|| s3_error!(InvalidArgument))?
.as_str()
.to_string();
let grantee = grant
.grantee
.as_ref()
.ok_or_else(|| s3_error!(InvalidArgument))
.and_then(grantee_from_grant)?;
Ok(StoredGrant { grantee, permission })
})
.collect::<Result<Vec<_>, S3Error>>()?;
Ok(StoredAcl { owner, grants })
}
fn parse_grant_header(value: &str) -> Result<Vec<StoredGrantee>, S3Error> {
let mut grantees = Vec::new();
for entry in value.split(',') {
let entry = entry.trim();
if entry.is_empty() {
continue;
}
let mut parts = entry.splitn(2, '=');
let key = parts.next().unwrap_or_default().trim();
let val = parts.next().unwrap_or_default().trim();
match key {
"id" => {
if !is_known_user_id(val) {
return Err(s3_error!(InvalidArgument));
}
grantees.push(StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(val.to_string()),
display_name: display_name_for_user_id(val),
uri: None,
email_address: None,
});
}
"uri" => {
if val != ACL_GROUP_ALL_USERS && val != ACL_GROUP_AUTHENTICATED_USERS {
return Err(s3_error!(InvalidArgument));
}
grantees.push(StoredGrantee {
grantee_type: "Group".to_string(),
id: None,
display_name: None,
uri: Some(val.to_string()),
email_address: None,
});
}
"emailAddress" => {
let owner = user_id_from_email(val).ok_or_else(|| s3_error!(UnresolvableGrantByEmailAddress))?;
grantees.push(StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(owner.id),
display_name: Some(owner.display_name),
uri: None,
email_address: None,
});
}
_ => return Err(s3_error!(InvalidArgument)),
}
}
Ok(grantees)
}
pub(crate) fn stored_acl_from_grant_headers(
owner: &StoredOwner,
grant_read: Option<String>,
grant_write: Option<String>,
grant_read_acp: Option<String>,
grant_write_acp: Option<String>,
grant_full_control: Option<String>,
) -> Result<Option<StoredAcl>, S3Error> {
let mut grants = Vec::new();
let mut push_grants = |value: Option<String>, permission: &str| -> Result<(), S3Error> {
if let Some(value) = value {
for grantee in parse_grant_header(&value)? {
grants.push(StoredGrant {
grantee,
permission: permission.to_string(),
});
}
}
Ok(())
};
push_grants(grant_read, Permission::READ)?;
push_grants(grant_write, Permission::WRITE)?;
push_grants(grant_read_acp, Permission::READ_ACP)?;
push_grants(grant_write_acp, Permission::WRITE_ACP)?;
push_grants(grant_full_control, Permission::FULL_CONTROL)?;
if grants.is_empty() {
return Ok(None);
}
grants.push(StoredGrant {
grantee: StoredGrantee {
grantee_type: "CanonicalUser".to_string(),
id: Some(owner.id.clone()),
display_name: Some(owner.display_name.clone()),
uri: None,
email_address: None,
},
permission: Permission::FULL_CONTROL.to_string(),
});
Ok(Some(StoredAcl {
owner: owner.clone(),
grants,
}))
}
#[derive(Debug, Clone)]
pub struct FS {
// pub store: ECStore,
@@ -659,13 +170,6 @@ impl FS {
}
}
pub(crate) const ACL_GROUP_ALL_USERS: &str = "http://acs.amazonaws.com/groups/global/AllUsers";
pub(crate) const ACL_GROUP_AUTHENTICATED_USERS: &str = "http://acs.amazonaws.com/groups/global/AuthenticatedUsers";
pub(crate) fn is_public_canned_acl(acl: &str) -> bool {
matches!(acl, "public-read" | "public-read-write" | "authenticated-read")
}
pub(crate) fn parse_object_version_id(version_id: Option<String>) -> S3Result<Option<Uuid>> {
if let Some(vid) = version_id {
let uuid = Uuid::parse_str(&vid).map_err(|e| {
+36 -8
View File
@@ -641,13 +641,23 @@ pub(crate) fn needs_cors_processing(headers: &HeaderMap) -> bool {
/// 2. Retrieves the bucket's CORS configuration
/// 3. Matches the origin against CORS rules
/// 4. Validates AllowedHeaders if request headers are present
/// 5. Returns headers to add to the response if a match is found
/// 5. Returns one of:
/// - `None`: bucket has no CORS config (or request has no valid `Origin`)
/// - `Some(empty headers)`: bucket CORS exists but request is denied / no rule matched
/// - `Some(non-empty headers)`: bucket CORS exists and request matched
///
/// Note: This function should only be called if `needs_cors_processing()` returns true
/// to avoid unnecessary overhead for non-CORS requests.
pub(crate) async fn apply_cors_headers(bucket: &str, method: &http::Method, headers: &HeaderMap) -> Option<HeaderMap> {
use http::HeaderValue;
fn is_credentialed_request(headers: &HeaderMap) -> bool {
headers.contains_key(http::header::AUTHORIZATION)
|| headers.contains_key(http::header::COOKIE)
|| headers.contains_key("x-amz-security-token")
|| headers.contains_key("x-amz-content-sha256")
}
// Get Origin header from request
let origin = headers.get(cors::standard::ORIGIN)?.to_str().ok()?;
@@ -659,14 +669,14 @@ pub(crate) async fn apply_cors_headers(bucket: &str, method: &http::Method, head
// Early return if no CORS rules configured
if cors_config.cors_rules.is_empty() {
return None;
return Some(HeaderMap::new());
}
// Check if method is supported and get its string representation
const SUPPORTED_METHODS: &[&str] = &["GET", "PUT", "POST", "DELETE", "HEAD", "OPTIONS"];
let method_str = method.as_str();
if !SUPPORTED_METHODS.contains(&method_str) {
return None;
return Some(HeaderMap::new());
}
// Use Access-Control-Request-Method if present (for preflight and non-preflight requests),
@@ -740,19 +750,37 @@ pub(crate) async fn apply_cors_headers(bucket: &str, method: &http::Method, head
let mut response_headers = HeaderMap::new();
// Access-Control-Allow-Origin
// If origin is "*", use "*", otherwise echo back the origin
// Credentials mode + wildcard allow list requires echoing the request origin.
// Browsers reject `Access-Control-Allow-Origin: *` with `credentials: include`.
let has_wildcard_origin = rule.allowed_origins.iter().any(|o| o == "*");
let credentialed_request = is_credentialed_request(headers);
let mut origin_reflected = false;
if has_wildcard_origin {
response_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN, HeaderValue::from_static("*"));
if credentialed_request {
if let Ok(origin_value) = HeaderValue::from_str(origin) {
response_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN, origin_value);
origin_reflected = true;
}
} else {
response_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN, HeaderValue::from_static("*"));
}
} else if let Ok(origin_value) = HeaderValue::from_str(origin) {
response_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN, origin_value);
origin_reflected = true;
}
// Vary: Origin (required for caching, except when using wildcard)
if !has_wildcard_origin {
// Vary: Origin whenever origin is reflected (non-"*" allow-origin).
// This prevents proxy/browser caches from reusing CORS headers across different origins.
if origin_reflected {
response_headers.insert(cors::standard::VARY, HeaderValue::from_static("Origin"));
}
// Credentials mode requires explicit allow-credentials.
if credentialed_request {
response_headers.insert(cors::response::ACCESS_CONTROL_ALLOW_CREDENTIALS, HeaderValue::from_static("true"));
}
// Access-Control-Allow-Methods (required for preflight)
if is_preflight || !rule.allowed_methods.is_empty() {
let methods_str = rule.allowed_methods.iter().map(|m| m.as_str()).collect::<Vec<_>>().join(", ");
@@ -788,7 +816,7 @@ pub(crate) async fn apply_cors_headers(bucket: &str, method: &http::Method, head
return Some(response_headers);
}
None // No matching rule found
Some(HeaderMap::new()) // No matching rule found
}
/// Check if an origin matches a pattern (supports wildcards like https://*.example.com)
pub(crate) fn matches_origin_pattern(pattern: &str, origin: &str) -> bool {
+82 -2
View File
@@ -15,6 +15,7 @@
#[cfg(test)]
mod tests {
use crate::config::workload_profiles::WorkloadProfile;
use crate::server::cors;
use crate::storage::ecfs::FS;
use crate::storage::ecfs::RUSTFS_OWNER;
use crate::storage::{
@@ -25,13 +26,14 @@ mod tests {
};
use http::{HeaderMap, HeaderValue, StatusCode};
use rustfs_config::MI_B;
use rustfs_ecstore::bucket::{metadata::BucketMetadata, metadata_sys};
use rustfs_ecstore::set_disk::DEFAULT_READ_BUFFER_SIZE;
use rustfs_ecstore::store_api::ObjectInfo;
use rustfs_utils::http::{AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, RESERVED_METADATA_PREFIX_LOWER};
use rustfs_zip::CompressionFormat;
use s3s::dto::{
Delimiter, LambdaFunctionConfiguration, ObjectLockLegalHold, ObjectLockLegalHoldStatus, ObjectLockRetention,
ObjectLockRetentionMode, QueueConfiguration, TopicConfiguration,
CORSConfiguration, CORSRule, Delimiter, LambdaFunctionConfiguration, ObjectLockLegalHold, ObjectLockLegalHoldStatus,
ObjectLockRetention, ObjectLockRetentionMode, QueueConfiguration, TopicConfiguration,
};
use s3s::{S3Error, S3ErrorCode, s3_error};
use time::OffsetDateTime;
@@ -1126,6 +1128,84 @@ mod tests {
// This is expected behavior - we just verify it doesn't panic
}
#[tokio::test]
async fn test_apply_cors_headers_unmatched_origin_with_cors_config() {
if metadata_sys::get_global_bucket_metadata_sys().is_none() {
eprintln!("Skipping test: GLOBAL_BucketMetadataSys not initialized");
return;
}
let bucket = "test-bucket-no-match-cors";
let mut bm = BucketMetadata::new(bucket);
bm.cors_config = Some(CORSConfiguration {
cors_rules: vec![CORSRule {
allowed_headers: Some(vec!["*".to_string()]),
allowed_methods: vec!["GET".to_string()],
allowed_origins: vec!["https://allowed.example.com".to_string()],
expose_headers: None,
id: Some("non-match-origin".to_string()),
max_age_seconds: None,
}],
});
metadata_sys::set_bucket_metadata(bucket.to_string(), bm).await.unwrap();
let mut headers = HeaderMap::new();
headers.insert(cors::standard::ORIGIN, "https://disallowed.example.com".parse().unwrap());
let result = apply_cors_headers(bucket, &http::Method::GET, &headers).await;
assert!(
result.is_some(),
"Expected Some empty headers when bucket has CORS config but origin does not match any rule"
);
let result = result.unwrap();
assert!(result.get(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN).is_none());
assert!(result.get(cors::response::ACCESS_CONTROL_ALLOW_METHODS).is_none());
metadata_sys::set_bucket_metadata(bucket.to_string(), BucketMetadata::new(bucket))
.await
.unwrap();
}
#[tokio::test]
async fn test_apply_cors_headers_credentialed_request_with_wildcard_origin() {
if metadata_sys::get_global_bucket_metadata_sys().is_none() {
eprintln!("Skipping test: GLOBAL_BucketMetadataSys not initialized");
return;
}
let bucket = "test-bucket-credentialed-cors";
let mut bm = BucketMetadata::new(bucket);
bm.cors_config = Some(CORSConfiguration {
cors_rules: vec![CORSRule {
allowed_headers: Some(vec!["*".to_string()]),
allowed_methods: vec!["GET".to_string()],
allowed_origins: vec!["*".to_string()],
expose_headers: None,
id: Some("credentialed-unit".to_string()),
max_age_seconds: None,
}],
});
metadata_sys::set_bucket_metadata(bucket.to_string(), bm).await.unwrap();
let mut headers = HeaderMap::new();
headers.insert(cors::standard::ORIGIN, "https://console.localhost".parse().unwrap());
headers.insert(cors::request::ACCESS_CONTROL_REQUEST_METHOD, "GET".parse().unwrap());
headers.insert(cors::request::ACCESS_CONTROL_REQUEST_HEADERS, "x-amz-content-sha256".parse().unwrap());
headers.insert(http::header::AUTHORIZATION, "AWS4-HMAC-SHA256 Credential=test/20260302/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256, Signature=abc".parse().unwrap());
let result = apply_cors_headers(bucket, &http::Method::OPTIONS, &headers).await.unwrap();
assert_eq!(
result.get(cors::response::ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
"https://console.localhost",
);
assert_eq!(result.get(cors::response::ACCESS_CONTROL_ALLOW_CREDENTIALS).unwrap(), "true");
assert_eq!(result.get(cors::standard::VARY).unwrap(), "Origin");
metadata_sys::set_bucket_metadata(bucket.to_string(), BucketMetadata::new(bucket))
.await
.unwrap();
}
#[tokio::test]
async fn test_apply_cors_headers_unsupported_method() {
// Test with unsupported HTTP method
+58 -25
View File
@@ -44,8 +44,9 @@ use tracing::error;
use uuid::Uuid;
use crate::auth::AuthType;
use crate::auth::get_request_auth_type;
use crate::auth::is_request_presigned_signature_v4;
use crate::auth::get_query_param;
use crate::auth::get_request_auth_type_with_query;
use crate::auth::is_request_presigned_signature_v4_with_query;
const REPLICATION_REQUEST_TRUE: HeaderValue = HeaderValue::from_static("true");
@@ -580,13 +581,18 @@ pub fn parse_copy_source_range(range_str: &str) -> S3Result<HTTPRangeSpec> {
}
}
#[allow(dead_code)]
pub(crate) fn get_content_sha256(headers: &HeaderMap<HeaderValue>) -> Option<String> {
match get_request_auth_type(headers) {
get_content_sha256_with_query(headers, None)
}
pub(crate) fn get_content_sha256_with_query(headers: &HeaderMap<HeaderValue>, query: Option<&str>) -> Option<String> {
match get_request_auth_type_with_query(headers, query) {
AuthType::Presigned | AuthType::Signed => {
if skip_content_sha256_cksum(headers) {
if skip_content_sha256_cksum_with_query(headers, query) {
None
} else {
Some(get_content_sha256_cksum(headers, ServiceType::S3))
Some(get_content_sha256_cksum_with_query(headers, query, ServiceType::S3))
}
}
_ => None,
@@ -595,24 +601,21 @@ pub(crate) fn get_content_sha256(headers: &HeaderMap<HeaderValue>) -> Option<Str
/// skip_content_sha256_cksum returns true if caller needs to skip
/// payload checksum, false if not.
#[allow(dead_code)]
fn skip_content_sha256_cksum(headers: &HeaderMap<HeaderValue>) -> bool {
let content_sha256 = if is_request_presigned_signature_v4(headers) {
// For presigned requests, check query params first, then headers
// Note: In a real implementation, you would need to check query parameters
// For now, we'll just check headers
headers.get(AMZ_CONTENT_SHA256)
} else {
headers.get(AMZ_CONTENT_SHA256)
};
skip_content_sha256_cksum_with_query(headers, None)
}
// Skip if no header was set
fn skip_content_sha256_cksum_with_query(headers: &HeaderMap<HeaderValue>, query: Option<&str>) -> bool {
let include_query_values = matches!(get_request_auth_type_with_query(headers, query), AuthType::Presigned);
let content_sha256 = get_content_sha256_value(headers, query, include_query_values);
// Skip if no checksum value was set in header/query for query-presigned requests.
let Some(header_value) = content_sha256 else {
return true;
};
let Ok(value) = header_value.to_str() else {
return true;
};
let value = header_value;
// If x-amz-content-sha256 is set and the value is not
// 'UNSIGNED-PAYLOAD' we should validate the content sha256.
@@ -643,7 +646,11 @@ fn skip_content_sha256_cksum(headers: &HeaderMap<HeaderValue>) -> bool {
}
/// Returns SHA256 for calculating canonical-request.
fn get_content_sha256_cksum(headers: &HeaderMap<HeaderValue>, service_type: ServiceType) -> String {
fn get_content_sha256_cksum_with_query(
headers: &HeaderMap<HeaderValue>,
query: Option<&str>,
service_type: ServiceType,
) -> String {
if service_type == ServiceType::STS {
// For STS requests, we would need to read the body and calculate SHA256
// This is a simplified implementation - in practice you'd need access to the request body
@@ -651,28 +658,54 @@ fn get_content_sha256_cksum(headers: &HeaderMap<HeaderValue>, service_type: Serv
return "sts-body-sha256-placeholder".to_string();
}
let (default_sha256_cksum, content_sha256) = if is_request_presigned_signature_v4(headers) {
let (default_sha256_cksum, content_sha256) = if is_request_presigned_signature_v4_with_query(headers, query) {
// For a presigned request we look at the query param for sha256.
// X-Amz-Content-Sha256, if not set in presigned requests, checksum
// will default to 'UNSIGNED-PAYLOAD'.
(UNSIGNED_PAYLOAD.to_string(), headers.get(AMZ_CONTENT_SHA256))
(UNSIGNED_PAYLOAD.to_string(), get_content_sha256_value(headers, query, true))
} else {
// X-Amz-Content-Sha256, if not set in signed requests, checksum
// will default to sha256([]byte("")).
(EMPTY_STRING_SHA256_HASH.to_string(), headers.get(AMZ_CONTENT_SHA256))
(
EMPTY_STRING_SHA256_HASH.to_string(),
headers
.get(AMZ_CONTENT_SHA256)
.and_then(|v| v.to_str().ok().map(str::to_owned)),
)
};
// We found 'X-Amz-Content-Sha256' return the captured value.
if let Some(header_value) = content_sha256
&& let Ok(value) = header_value.to_str()
{
return value.to_string();
if let Some(header_value) = content_sha256 {
return header_value;
}
// We couldn't find 'X-Amz-Content-Sha256'.
default_sha256_cksum
}
fn get_content_sha256_value(
headers: &HeaderMap<HeaderValue>,
query: Option<&str>,
include_query_for_presigned: bool,
) -> Option<String> {
if include_query_for_presigned && is_request_presigned_signature_v4_with_query(headers, query) {
return query
.and_then(|q| get_query_param(q, "x-amz-content-sha256"))
.or_else(|| headers.get(AMZ_CONTENT_SHA256).and_then(|v| v.to_str().ok()))
.map(str::to_owned);
}
headers
.get(AMZ_CONTENT_SHA256)
.and_then(|v| v.to_str().ok())
.map(str::to_owned)
}
#[allow(dead_code)]
fn get_content_sha256_cksum(headers: &HeaderMap<HeaderValue>, service_type: ServiceType) -> String {
get_content_sha256_cksum_with_query(headers, None, service_type)
}
#[cfg(test)]
mod tests {
use super::*;
+159 -23
View File
@@ -167,27 +167,28 @@ async fn prepare_sse_configuration(
}));
}
// Get bucket default encryption configuration
// Get bucket default encryption configuration.
let bucket_sse_config_result = metadata_sys::get_sse_config(bucket).await;
debug!("bucket_sse_config_result={:?}", bucket_sse_config_result);
if let Ok((bucket_sse_config, _timestamp)) = bucket_sse_config_result {
let effective_sse = server_side_encryption
.clone()
.or_else(|| {
bucket_sse_config.rules.first().and_then(|rule| {
debug!("Processing SSE rule: {:?}", rule);
rule.apply_server_side_encryption_by_default.as_ref().map(|sse| {
debug!("Found SSE default: {:?}", sse);
match sse.sse_algorithm.as_str() {
"AES256" => ServerSideEncryption::from_static(ServerSideEncryption::AES256),
"aws:kms" => ServerSideEncryption::from_static(ServerSideEncryption::AWS_KMS),
_ => ServerSideEncryption::from_static(ServerSideEncryption::AES256), // fallback to AES256
}
})
let effective_sse = server_side_encryption.clone().or_else(|| {
bucket_sse_config.rules.first().and_then(|rule| {
debug!("Processing SSE rule: {:?}", rule);
rule.apply_server_side_encryption_by_default.as_ref().map(|sse| {
debug!("Found SSE default: {:?}", sse);
match sse.sse_algorithm.as_str() {
"AES256" => ServerSideEncryption::from_static(ServerSideEncryption::AES256),
"aws:kms" => ServerSideEncryption::from_static(ServerSideEncryption::AWS_KMS),
_ => ServerSideEncryption::from_static(ServerSideEncryption::AES256), // fallback
}
})
})
.unwrap_or_else(|| ServerSideEncryption::from_static(ServerSideEncryption::AES256));
});
if effective_sse.is_none() {
return Ok(None);
}
debug!("effective_sse={:?} (original={:?})", effective_sse, server_side_encryption);
let effective_kms_key_id = ssekms_key_id.or_else(|| {
@@ -199,7 +200,7 @@ async fn prepare_sse_configuration(
});
Ok(Some(SseConfiguration {
effective_sse,
effective_sse: effective_sse.unwrap(),
effective_kms_key_id,
}))
} else if let Err(e) = bucket_sse_config_result {
@@ -1028,9 +1029,11 @@ async fn apply_managed_encryption_material(
}
}
let kms_key_to_use = kms_key_candidate
.clone()
.ok_or_else(|| ApiError::from(StorageError::other("No KMS key available for managed server-side encryption")))?;
let kms_key_to_use = kms_key_candidate.clone().ok_or_else(|| {
ApiError::from(StorageError::other(
"No KMS key available for managed server-side encryption (required for SSE-KMS)",
))
})?;
let provider = get_sse_dek_provider().await?;
@@ -1390,6 +1393,33 @@ impl TestSseDekProvider {
Self { master_key }
}
/// Create a local SSE DEK provider for SSE-S3 when KMS is not configured.
/// Uses RUSTFS_SSE_S3_MASTER_KEY (base64 32-byte) if set; otherwise a built-in default.
/// Allows PUT/GET to work without KMS (backward compatible).
pub fn new_for_local_sse() -> Self {
let master_key = match std::env::var("RUSTFS_SSE_S3_MASTER_KEY") {
Ok(v) if !v.trim().is_empty() => match BASE64_STANDARD.decode(v.trim()) {
Ok(decoded) if decoded.len() == 32 => {
let mut arr = [0u8; 32];
arr.copy_from_slice(&decoded[..32]);
tracing::info!("Using RUSTFS_SSE_S3_MASTER_KEY for SSE-S3 (KMS not configured)");
arr
}
_ => {
tracing::warn!("RUSTFS_SSE_S3_MASTER_KEY invalid (expected base64 32 bytes); using default for SSE-S3");
[0u8; 32]
}
},
_ => {
tracing::debug!(
"KMS not configured; using built-in default key for SSE-S3 (set RUSTFS_SSE_S3_MASTER_KEY for production)"
);
[0u8; 32]
}
};
Self { master_key }
}
// Simple encryption of DEK
pub(crate) fn encrypt_dek(dek: [u8; 32], cmk_value: [u8; 32]) -> Result<String, ApiError> {
// Use AES-256-GCM to encrypt DEK
@@ -1501,13 +1531,16 @@ pub async fn get_sse_dek_provider() -> Result<Arc<dyn SseDekProvider>, ApiError>
return Ok(provider.clone());
}
// Determine provider based on environment variable
let provider: Arc<dyn SseDekProvider> = if std::env::var("__RUSTFS_SSE_SIMPLE_CMK").is_ok() {
// Determine provider: KMS when available, else test env, else local SSE-S3 fallback (no KMS)
let provider: Arc<dyn SseDekProvider> = if get_global_encryption_service().await.is_some() {
debug!("Using KmsSseDekProvider (KMS configured)");
Arc::new(KmsSseDekProvider::new().await?)
} else if std::env::var("__RUSTFS_SSE_SIMPLE_CMK").is_ok() {
debug!("Using SimpleSseDekProvider (test mode) based on __RUSTFS_SSE_SIMPLE_CMK");
Arc::new(TestSseDekProvider::new())
} else {
debug!("Using KmsSseDekProvider (production mode)");
Arc::new(KmsSseDekProvider::new().await?)
debug!("Using local SSE-S3 provider (KMS not configured)");
Arc::new(TestSseDekProvider::new_for_local_sse())
};
// Store in global cache
@@ -2654,4 +2687,107 @@ mod tests {
let err = validate_ssec_params(params).unwrap_err();
assert_eq!(err.code, S3ErrorCode::InvalidRequest);
}
// ========================================================================
// Unit tests for issue #2041: no mandatory KMS when encryption not used
// ========================================================================
/// When SSE-C params are not present and no managed SSE is requested,
/// encryption should be skipped (Ok(None)). Ensures we do not require KMS
/// when the client sends no encryption headers.
#[tokio::test]
async fn test_sse_encryption_skip_when_no_ssec_and_no_managed_sse_requested() {
let request = EncryptionRequest {
bucket: "test-bucket",
key: "test-key",
server_side_encryption: None,
ssekms_key_id: None,
sse_customer_algorithm: None,
sse_customer_key: None,
sse_customer_key_md5: None,
content_size: 1024,
part_number: None,
part_key: None,
part_nonce: None,
};
let result = sse_encryption(request).await;
match &result {
Ok(None) => {}
Ok(Some(_)) => panic!("expected no encryption material when no SSE params provided"),
Err(e) => {
assert!(
!e.message.contains("No KMS key"),
"must not require KMS when no encryption requested; got: {}",
e.message
);
}
}
}
/// When SSE-C params are partial or invalid, sse_encryption must return an error.
#[tokio::test]
async fn test_sse_encryption_errors_on_invalid_ssec_params() {
let bucket = "test-bucket";
let key = "test-key";
let sse_key = BASE64_STANDARD.encode([42u8; 32]);
let wrong_md5 = BASE64_STANDARD.encode([99u8; 16]);
let request_wrong_md5 = EncryptionRequest {
bucket,
key,
server_side_encryption: None,
ssekms_key_id: None,
sse_customer_algorithm: Some("AES256".to_string()),
sse_customer_key: Some(sse_key.clone()),
sse_customer_key_md5: Some(wrong_md5),
content_size: 1024,
part_number: None,
part_key: None,
part_nonce: None,
};
let err = sse_encryption(request_wrong_md5).await.unwrap_err();
assert_eq!(err.code, S3ErrorCode::InvalidRequest);
let request_unsupported_algorithm = EncryptionRequest {
bucket,
key,
server_side_encryption: None,
ssekms_key_id: None,
sse_customer_algorithm: Some("unsupported-algo".to_string()),
sse_customer_key: Some(sse_key),
sse_customer_key_md5: Some(BASE64_STANDARD.encode(md5::compute([42u8; 32]).0)),
content_size: 1024,
part_number: None,
part_key: None,
part_nonce: None,
};
let err = sse_encryption(request_unsupported_algorithm).await.unwrap_err();
assert!(err.code == S3ErrorCode::InvalidRequest || err.code == S3ErrorCode::InvalidArgument);
}
/// When bucket has no SSE-S3/aws:kms setting and request has no SSE headers,
/// encryption should be skipped (Ok(None)). Ensures no mandatory bucket default SSE.
#[tokio::test]
async fn test_sse_prepare_encryption_skip_when_no_params_and_no_bucket_sse() {
let request = PrepareEncryptionRequest {
bucket: "test-bucket-no-sse-config",
key: "test-key",
server_side_encryption: None,
ssekms_key_id: None,
sse_customer_algorithm: None,
sse_customer_key_md5: None,
};
let result = sse_prepare_encryption(request).await;
match &result {
Ok(None) => {}
Ok(Some(_)) => panic!("expected no encryption when bucket has no SSE config and no request SSE"),
Err(e) => {
assert!(
!e.message.contains("No KMS key"),
"must not require KMS when no bucket SSE and no request SSE; got: {}",
e.message
);
}
}
}
}
+3 -3
View File
@@ -156,8 +156,8 @@ $env:RUSTFS_NS_SCANNER_INTERVAL = "60"
# $env:RUSTFS_REGION = "us-east-1"
$env:RUSTFS_ENABLE_SCANNER = "false"
$env:RUSTFS_ENABLE_HEAL = "false"
$env:RUSTFS_SCANNER_ENABLED = "false"
$env:RUSTFS_HEAL_ENABLED = "false"
# Object cache configuration
$env:RUSTFS_OBJECT_CACHE_ENABLE = "true"
@@ -198,4 +198,4 @@ if (-not $env:MALLOC_CONF) {
# To run in release mode:
# cargo run --profile release --bin rustfs
# To run in debug mode:
cargo run --bin rustfs
cargo run --bin rustfs
+4 -4
View File
@@ -66,7 +66,7 @@ export RUSTFS_CONSOLE_ADDRESS=":9001"
#export RUSTFS_OBS_METER_INTERVAL=1 # Sampling interval in seconds
#export RUSTFS_OBS_SERVICE_NAME=rustfs # Service name
#export RUSTFS_OBS_SERVICE_VERSION=0.1.0 # Service version
export RUSTFS_OBS_ENVIRONMENT=develop # Environment name
export RUSTFS_OBS_ENVIRONMENT=develop # Environment name development, staging, production
export RUSTFS_OBS_LOGGER_LEVEL=info # Log level, supports trace, debug, info, warn, error
export RUSTFS_OBS_LOG_STDOUT_ENABLED=true # Whether to enable local stdout logging
export RUSTFS_OBS_LOG_DIRECTORY="$current_dir/deploy/logs" # Log directory
@@ -178,9 +178,9 @@ export RUSTFS_NS_SCANNER_INTERVAL=60 # Object scanning interval in seconds
#export RUSTFS_REGION="us-east-1"
export RUSTFS_ENABLE_SCANNER=true
export RUSTFS_SCANNER_ENABLED=true
export RUSTFS_ENABLE_HEAL=true
export RUSTFS_HEAL_ENABLED=true
# Object cache configuration
export RUSTFS_OBJECT_CACHE_ENABLE=true
@@ -252,4 +252,4 @@ fi
# To run in release mode, use the following line
#cargo run --profile release --bin rustfs
# To run in debug mode, use the following line
cargo run --bin rustfs
cargo run --bin rustfs
+425
View File
@@ -0,0 +1,425 @@
# Excluded S3 feature tests
# =========================
#
# These tests are intentionally excluded from RustFS compatibility gating.
#
# Exclusion buckets:
# - Vendor-specific behavior not required for RustFS S3 compatibility
# - Intentionally unsupported by product decision (for example ACL authorization)
# Vendor-specific / non-portable tests
test_100_continue_error_retry
test_account_usage
test_atomic_conditional_write_1mb
test_atomic_dual_conditional_write_1mb
test_atomic_write_bucket_gone
test_bucket_get_location
test_bucket_head_extended
test_bucket_header_acl_grants
test_bucket_list_return_data
test_bucket_list_return_data_versioning
test_bucket_logging_bucket_acl_required
test_bucket_logging_bucket_auth_type
test_bucket_logging_cleanup_bucket_concurrent_deletion_j
test_bucket_logging_cleanup_bucket_concurrent_deletion_j_single
test_bucket_logging_cleanup_bucket_concurrent_deletion_s
test_bucket_logging_cleanup_bucket_concurrent_deletion_s_single
test_bucket_logging_cleanup_bucket_deletion_j
test_bucket_logging_cleanup_bucket_deletion_j_single
test_bucket_logging_cleanup_bucket_deletion_s
test_bucket_logging_cleanup_bucket_deletion_s_single
test_bucket_logging_cleanup_concurrent_disabling_j
test_bucket_logging_cleanup_concurrent_disabling_j_single
test_bucket_logging_cleanup_concurrent_disabling_s
test_bucket_logging_cleanup_concurrent_disabling_s_single
test_bucket_logging_cleanup_concurrent_updating_j
test_bucket_logging_cleanup_concurrent_updating_j_single
test_bucket_logging_cleanup_concurrent_updating_s
test_bucket_logging_cleanup_concurrent_updating_s_single
test_bucket_logging_cleanup_disabling_j
test_bucket_logging_cleanup_disabling_j_single
test_bucket_logging_cleanup_disabling_s
test_bucket_logging_cleanup_disabling_s_single
test_bucket_logging_cleanup_updating_j
test_bucket_logging_cleanup_updating_j_single
test_bucket_logging_cleanup_updating_s
test_bucket_logging_cleanup_updating_s_single
test_bucket_logging_concurrent_flush_j
test_bucket_logging_concurrent_flush_j_single
test_bucket_logging_concurrent_flush_s
test_bucket_logging_concurrent_flush_s_single
test_bucket_logging_conf_concurrent_updating_pfx_j
test_bucket_logging_conf_concurrent_updating_pfx_s
test_bucket_logging_conf_concurrent_updating_roll_j
test_bucket_logging_conf_concurrent_updating_roll_s
test_bucket_logging_conf_updating_pfx_j
test_bucket_logging_conf_updating_pfx_s
test_bucket_logging_conf_updating_roll_j
test_bucket_logging_conf_updating_roll_s
test_bucket_logging_copy_objects
test_bucket_logging_copy_objects_bucket
test_bucket_logging_copy_objects_bucket_versioned
test_bucket_logging_copy_objects_versioned
test_bucket_logging_delete_objects
test_bucket_logging_delete_objects_versioned
test_bucket_logging_event_type_j
test_bucket_logging_event_type_s
test_bucket_logging_flush_empty
test_bucket_logging_flush_j
test_bucket_logging_flush_j_single
test_bucket_logging_flush_s
test_bucket_logging_flush_s_single
test_bucket_logging_get_objects
test_bucket_logging_get_objects_versioned
test_bucket_logging_head_objects
test_bucket_logging_head_objects_versioned
test_bucket_logging_key_filter_j
test_bucket_logging_key_filter_s
test_bucket_logging_mpu_copy
test_bucket_logging_mpu_copy_versioned
test_bucket_logging_mpu_j
test_bucket_logging_mpu_s
test_bucket_logging_mpu_versioned_j
test_bucket_logging_mpu_versioned_s
test_bucket_logging_mtime
test_bucket_logging_multi_delete
test_bucket_logging_multi_delete_versioned
test_bucket_logging_multiple_prefixes
test_bucket_logging_notupdating_j
test_bucket_logging_notupdating_j_single
test_bucket_logging_notupdating_s
test_bucket_logging_notupdating_s_single
test_bucket_logging_object_acl_required
test_bucket_logging_object_meta
test_bucket_logging_part_cleanup_concurrent_deletion_j
test_bucket_logging_part_cleanup_concurrent_deletion_s
test_bucket_logging_part_cleanup_concurrent_disabling_j
test_bucket_logging_part_cleanup_concurrent_disabling_s
test_bucket_logging_part_cleanup_concurrent_updating_j
test_bucket_logging_part_cleanup_concurrent_updating_s
test_bucket_logging_part_cleanup_deletion_j
test_bucket_logging_part_cleanup_deletion_s
test_bucket_logging_part_cleanup_disabling_j
test_bucket_logging_part_cleanup_disabling_s
test_bucket_logging_part_cleanup_updating_j
test_bucket_logging_part_cleanup_updating_s
test_bucket_logging_partitioned_key
test_bucket_logging_permission_change_j
test_bucket_logging_permission_change_s
test_bucket_logging_put_and_flush
test_bucket_logging_put_concurrency
test_bucket_logging_put_objects
test_bucket_logging_put_objects_versioned
test_bucket_logging_roll_time
test_bucket_logging_simple_key
test_bucket_logging_single_prefix
test_bucket_logging_target_cleanup_j
test_bucket_logging_target_cleanup_j_single
test_bucket_logging_target_cleanup_s
test_bucket_logging_target_cleanup_s_single
test_bucket_policy_get_obj_acl_existing_tag
test_bucket_policy_get_obj_existing_tag
test_bucket_policy_get_obj_tagging_existing_tag
test_bucket_policy_put_obj_copy_source
test_bucket_policy_put_obj_copy_source_meta
test_bucket_policy_put_obj_kms_noenc
test_bucket_policy_put_obj_request_obj_tag
test_bucket_policy_put_obj_s3_incorrect_algo_sse_s3
test_bucket_policy_put_obj_s3_noenc
test_bucket_policy_put_obj_tagging_existing_tag
test_bucket_policy_set_condition_operator_end_with_IfExists
test_bucket_policy_upload_part_copy
test_bucket_recreate_new_acl
test_bucket_recreate_overwrite_acl
test_copy_object_ifmatch_failed
test_copy_object_ifmatch_good
test_copy_object_ifnonematch_failed
test_copy_object_ifnonematch_good
test_cors_presigned_get_object_tenant_v2
test_cors_presigned_get_object_v2
test_cors_presigned_put_object_tenant_v2
test_cors_presigned_put_object_v2
test_create_bucket_bucket_owner_enforced
test_create_bucket_bucket_owner_preferred
test_create_bucket_object_writer
test_delete_marker_expiration
test_delete_marker_nonversioned
test_delete_marker_suspended
test_delete_marker_versioned
test_delete_object_current_if_match
test_delete_object_current_if_match_last_modified_time
test_delete_object_current_if_match_size
test_delete_object_if_match
test_delete_object_if_match_last_modified_time
test_delete_object_if_match_size
test_delete_object_version_if_match
test_delete_object_version_if_match_last_modified_time
test_delete_object_version_if_match_size
test_delete_objects_current_if_match
test_delete_objects_current_if_match_last_modified_time
test_delete_objects_current_if_match_size
test_delete_objects_if_match
test_delete_objects_if_match_last_modified_time
test_delete_objects_if_match_size
test_delete_objects_version_if_match
test_delete_objects_version_if_match_last_modified_time
test_delete_objects_version_if_match_size
test_delete_tags_obj_public
test_encrypted_transfer_13b
test_encrypted_transfer_1MB
test_encrypted_transfer_1b
test_encrypted_transfer_1kb
test_encryption_sse_c_deny_algo_with_bucket_policy
test_encryption_sse_c_enforced_with_bucket_policy
test_encryption_sse_c_multipart_invalid_chunks_1
test_encryption_sse_c_multipart_invalid_chunks_2
test_encryption_sse_c_multipart_upload
test_encryption_sse_c_post_object_authenticated_request
test_encryption_sse_c_unaligned_multipart_upload
test_expected_bucket_owner
test_get_multipart_checksum_object_attributes
test_get_multipart_object_attributes
test_get_obj_tagging
test_get_object_attributes
test_get_paginated_multipart_object_attributes
test_get_single_multipart_object_attributes
test_get_sse_c_encrypted_object_attributes
test_get_tags_acl_public
test_head_bucket_usage
test_lifecycle_cloud_multiple_transition
test_lifecycle_cloud_transition
test_lifecycle_cloud_transition_large_obj
test_lifecycle_deletemarker_expiration
test_lifecycle_deletemarker_expiration_with_days_tag
test_lifecycle_expiration
test_lifecycle_expiration_date
test_lifecycle_expiration_header_and_tags_head
test_lifecycle_expiration_header_head
test_lifecycle_expiration_header_tags_head
test_lifecycle_expiration_newer_noncurrent
test_lifecycle_expiration_noncur_tags1
test_lifecycle_expiration_size_gt
test_lifecycle_expiration_size_lt
test_lifecycle_expiration_tags1
test_lifecycle_expiration_tags2
test_lifecycle_expiration_versioned_tags2
test_lifecycle_expiration_versioning_enabled
test_lifecycle_multipart_expiration
test_lifecycle_noncur_cloud_transition
test_lifecycle_noncur_expiration
test_lifecycle_noncur_transition
test_lifecycle_transition
test_lifecycle_transition_single_rule_multi_trans
test_lifecyclev2_expiration
test_list_buckets_anonymous
test_list_buckets_paginated
test_list_multipart_upload
test_list_multipart_upload_owner
test_multipart_checksum_sha256
test_multipart_copy_multiple_sizes
test_multipart_copy_versioned
test_multipart_get_part
test_multipart_put_current_object_if_match
test_multipart_put_current_object_if_none_match
test_multipart_put_object_if_match
test_multipart_single_get_part
test_multipart_sse_c_get_part
test_multipart_upload
test_multipart_upload_contents
test_multipart_upload_resend_part
test_multipart_upload_small
test_multipart_use_cksum_helper_crc32
test_multipart_use_cksum_helper_crc32c
test_multipart_use_cksum_helper_crc64nvme
test_multipart_use_cksum_helper_sha1
test_multipart_use_cksum_helper_sha256
test_non_multipart_get_part
test_non_multipart_sse_c_get_part
test_object_copy_canned_acl
test_object_header_acl_grants
test_object_lock_changing_mode_from_compliance
test_object_lock_changing_mode_from_governance_with_bypass
test_object_lock_changing_mode_from_governance_without_bypass
test_object_lock_delete_multipart_object_with_legal_hold_on
test_object_lock_delete_multipart_object_with_retention
test_object_lock_delete_object_with_legal_hold_off
test_object_lock_delete_object_with_legal_hold_on
test_object_lock_delete_object_with_retention
test_object_lock_delete_object_with_retention_and_marker
test_object_lock_get_legal_hold
test_object_lock_get_obj_lock
test_object_lock_get_obj_metadata
test_object_lock_get_obj_retention
test_object_lock_get_obj_retention_iso8601
test_object_lock_multi_delete_object_with_retention
test_object_lock_put_legal_hold
test_object_lock_put_legal_hold_invalid_status
test_object_lock_put_obj_lock
test_object_lock_put_obj_lock_invalid_days
test_object_lock_put_obj_lock_invalid_mode
test_object_lock_put_obj_lock_invalid_status
test_object_lock_put_obj_lock_invalid_years
test_object_lock_put_obj_lock_with_days_and_years
test_object_lock_put_obj_retention
test_object_lock_put_obj_retention_increase_period
test_object_lock_put_obj_retention_invalid_mode
test_object_lock_put_obj_retention_override_default_retention
test_object_lock_put_obj_retention_shorten_period
test_object_lock_put_obj_retention_shorten_period_bypass
test_object_lock_put_obj_retention_versionid
test_object_lock_suspend_versioning
test_object_lock_uploading_obj
test_object_raw_get_x_amz_expires_not_expired
test_object_raw_get_x_amz_expires_not_expired_tenant
test_object_raw_get_x_amz_expires_out_max_range
test_object_raw_get_x_amz_expires_out_positive_range
test_object_raw_put_authenticated_expired
test_object_read_unreadable
test_object_requestid_matches_header_on_error
test_object_set_get_unicode_metadata
test_object_write_with_chunked_transfer_encoding
test_post_object_invalid_date_format
test_post_object_invalid_request_field_value
test_post_object_missing_policy_condition
test_post_object_request_missing_policy_specified_field
test_post_object_set_key_from_filename
test_post_object_success_redirect_action
test_post_object_tags_anonymous_request
test_post_object_wrong_bucket
test_put_bucket_logging_account_j
test_put_bucket_logging_account_s
test_put_bucket_logging_extensions
test_put_bucket_logging_policy_wildcard_objects
test_put_bucket_logging_tenant_j
test_put_bucket_logging_tenant_s
test_put_bucket_ownership_bucket_owner_enforced
test_put_bucket_ownership_bucket_owner_preferred
test_put_bucket_ownership_object_writer
test_put_current_object_if_match
test_put_current_object_if_none_match
test_put_delete_tags
test_put_max_tags
test_put_modify_tags
test_put_obj_with_tags
test_put_object_current_if_match
test_put_object_if_match
test_put_tags_acl_public
test_ranged_big_request_response_code
test_ranged_request_response_code
test_ranged_request_return_trailing_bytes_response_code
test_ranged_request_skip_leading_bytes_response_code
test_read_through
test_restore_noncur_obj
test_restore_object_permanent
test_restore_object_temporary
test_sse_kms_default_post_object_authenticated_request
test_sse_kms_default_upload_1b
test_sse_kms_default_upload_1kb
test_sse_kms_default_upload_1mb
test_sse_kms_default_upload_8mb
test_sse_kms_method_head
test_sse_kms_multipart_invalid_chunks_1
test_sse_kms_multipart_invalid_chunks_2
test_sse_kms_multipart_upload
test_sse_kms_post_object_authenticated_request
test_sse_kms_present
test_sse_kms_transfer_13b
test_sse_kms_transfer_1MB
test_sse_kms_transfer_1b
test_sse_kms_transfer_1kb
test_sse_s3_default_method_head
test_sse_s3_default_multipart_upload
test_sse_s3_default_post_object_authenticated_request
test_sse_s3_default_upload_1b
test_sse_s3_default_upload_1kb
test_sse_s3_default_upload_1mb
test_sse_s3_default_upload_8mb
test_sse_s3_encrypted_upload_1b
test_sse_s3_encrypted_upload_1kb
test_sse_s3_encrypted_upload_1mb
test_sse_s3_encrypted_upload_8mb
test_versioned_object_acl_no_version_specified
test_versioning_copy_obj_version
test_versioning_multi_object_delete_with_marker_create
test_versioning_obj_create_overwrite_multipart
test_versioning_obj_suspended_copy
test_versioning_stack_delete_merkers
# Intentionally unsupported by design: ACL-related tests
test_object_raw_get
test_object_raw_get_bucket_gone
test_object_raw_get_object_gone
test_object_copy_not_owned_object_bucket
test_100_continue
test_cors_origin_response
test_cors_origin_wildcard
test_bucket_policy_put_obj_grant
test_bucket_list_objects_anonymous
test_bucket_listv2_objects_anonymous
test_post_object_anonymous_request
test_post_object_set_success_code
test_post_object_set_invalid_success_code
test_access_bucket_private_object_private
test_access_bucket_private_object_publicread
test_access_bucket_private_object_publicreadwrite
test_access_bucket_private_objectv2_private
test_access_bucket_private_objectv2_publicread
test_access_bucket_private_objectv2_publicreadwrite
test_access_bucket_publicread_object_private
test_access_bucket_publicread_object_publicread
test_access_bucket_publicread_object_publicreadwrite
test_access_bucket_publicreadwrite_object_private
test_access_bucket_publicreadwrite_object_publicread
test_access_bucket_publicreadwrite_object_publicreadwrite
test_object_anon_put_write_access
test_get_public_acl_bucket_policy_status
test_get_authpublic_acl_bucket_policy_status
test_get_publicpolicy_acl_bucket_policy_status
test_get_nonpublicpolicy_acl_bucket_policy_status
test_block_public_put_bucket_acls
test_block_public_object_canned_acls
test_ignore_public_acls
test_bucket_policy_acl
test_bucketv2_policy_acl
test_bucket_policy_put_obj_acl
test_object_presigned_put_object_with_acl
test_object_put_acl_mtime
test_versioned_object_acl
test_object_presigned_put_object_with_acl_tenant
test_bucket_acl_canned
test_bucket_acl_canned_authenticatedread
test_bucket_acl_canned_during_create
test_bucket_acl_canned_private_to_private
test_bucket_acl_canned_publicreadwrite
test_bucket_acl_default
test_bucket_acl_grant_email
test_bucket_acl_grant_email_not_exist
test_bucket_acl_grant_nonexist_user
test_bucket_acl_grant_userid_fullcontrol
test_bucket_acl_grant_userid_read
test_bucket_acl_grant_userid_readacp
test_bucket_acl_grant_userid_write
test_bucket_acl_grant_userid_writeacp
test_bucket_acl_revoke_all
test_bucket_concurrent_set_canned_acl
test_object_acl
test_object_acl_canned
test_object_acl_canned_authenticatedread
test_object_acl_canned_bucketownerfullcontrol
test_object_acl_canned_bucketownerread
test_object_acl_canned_during_create
test_object_acl_canned_publicreadwrite
test_object_acl_default
test_object_acl_full_control_verify_attributes
test_object_acl_full_control_verify_owner
test_object_acl_read
test_object_acl_readacp
test_object_acl_write
test_object_acl_writeacp
test_put_bucket_acl_grant_group_read
test_object_raw_authenticated_bucket_acl
test_object_raw_authenticated_object_acl
test_object_raw_get_bucket_acl
test_object_raw_get_object_acl
test_cors_presigned_put_object_with_acl
test_cors_presigned_put_object_tenant_with_acl
+2 -83
View File
@@ -3,6 +3,7 @@
#
# These tests SHOULD PASS on RustFS for standard S3 API compatibility.
# Run these tests to verify RustFS S3 compatibility.
# Intentionally excluded tests (for example ACL authorization tests) are tracked in excluded_tests.txt.
#
# Covered operations:
# - Bucket: Create, Delete, List, Head, GetLocation
@@ -23,7 +24,6 @@
# - SSE-KMS: KMS-related edge cases
# - Bucket Policy: Multipart upload authorization, SSE condition keys, grant header conditions
# - Versioning: Concurrent multi-object delete
# - ACL: Bucket and Object ACL
# - Object Copy: Same/cross bucket, metadata, versioning
# - POST Object: HTML form upload
# - Raw requests: Authenticated and anonymous
@@ -36,7 +36,6 @@
# - DeleteObject: Proper NoSuchBucket for deleted buckets
# - Multipart Copy: InvalidRange when CopySourceRange exceeds source size
#
# Total: 396 tests
test_basic_key_count
test_bucket_create_naming_bad_short_one
@@ -120,13 +119,6 @@ test_bucketv2_notexist
test_bucketv2_policy_another_bucket
test_get_bucket_policy_status
test_get_nonpublicpolicy_principal_bucket_policy_status
test_get_public_acl_bucket_policy_status
test_get_authpublic_acl_bucket_policy_status
test_get_publicpolicy_acl_bucket_policy_status
test_get_nonpublicpolicy_acl_bucket_policy_status
test_block_public_put_bucket_acls
test_block_public_object_canned_acls
test_ignore_public_acls
test_set_get_del_bucket_policy
test_get_object_ifmatch_good
test_get_object_ifmodifiedsince_good
@@ -146,7 +138,6 @@ test_multipart_upload_overwrite_existing_object
test_multipart_upload_size_too_small
test_object_copy_bucket_not_found
test_object_copy_key_not_found
test_object_copy_not_owned_object_bucket
test_object_head_zero_bytes
test_get_obj_head_tagging
test_object_metadata_replaced_on_put
@@ -217,22 +208,16 @@ test_put_public_block
test_block_public_policy
test_block_public_policy_with_principal
test_block_public_restrict_public_buckets
test_bucket_create_exists_nonowner
test_get_public_block_deny_bucket_policy
test_get_undefined_public_block
# Bucket policy tests
test_bucketv2_policy
test_bucket_policy_acl
test_bucketv2_policy_acl
test_bucket_policy_another_bucket
test_bucket_policy_allow_notprincipal
test_bucket_policy_multipart
test_bucket_policy_put_obj_acl
test_bucket_policy_put_obj_grant
test_bucket_policy_put_obj_kms_s3
test_bucket_policy_put_obj_s3_kms
test_object_presigned_put_object_with_acl
test_object_put_acl_mtime
# Object ownership
test_create_bucket_no_ownership_controls
@@ -275,8 +260,6 @@ test_object_lock_put_obj_retention_invalid_bucket
# Versioning tests
test_get_versioned_object_attributes
test_versioned_concurrent_object_create_and_remove
test_versioned_concurrent_object_create_concurrent_remove
test_versioned_object_acl
test_versioning_bucket_create_suspend
test_versioning_bucket_atomic_upload_return_version_id
test_versioning_bucket_multipart_upload_return_version_id
@@ -294,54 +277,8 @@ test_versioning_obj_plain_null_version_removal
test_versioning_obj_suspend_versions
# Tenant and presigned tests
test_object_presigned_put_object_with_acl_tenant
# ACL tests (bucket and object)
test_access_bucket_private_object_private
test_access_bucket_private_object_publicread
test_access_bucket_private_object_publicreadwrite
test_access_bucket_private_objectv2_private
test_access_bucket_private_objectv2_publicread
test_access_bucket_private_objectv2_publicreadwrite
test_access_bucket_publicread_object_private
test_access_bucket_publicread_object_publicread
test_access_bucket_publicread_object_publicreadwrite
test_access_bucket_publicreadwrite_object_private
test_access_bucket_publicreadwrite_object_publicread
test_access_bucket_publicreadwrite_object_publicreadwrite
test_bucket_acl_canned
test_bucket_acl_canned_authenticatedread
test_bucket_acl_canned_during_create
test_bucket_acl_canned_private_to_private
test_bucket_acl_canned_publicreadwrite
test_bucket_acl_default
test_bucket_acl_grant_email
test_bucket_acl_grant_email_not_exist
test_bucket_acl_grant_nonexist_user
test_bucket_acl_grant_userid_fullcontrol
test_bucket_acl_grant_userid_read
test_bucket_acl_grant_userid_readacp
test_bucket_acl_grant_userid_write
test_bucket_acl_grant_userid_writeacp
test_bucket_acl_revoke_all
test_bucket_concurrent_set_canned_acl
test_object_acl
test_object_acl_canned
test_object_acl_canned_authenticatedread
test_object_acl_canned_bucketownerfullcontrol
test_object_acl_canned_bucketownerread
test_object_acl_canned_during_create
test_object_acl_canned_publicreadwrite
test_object_acl_default
test_object_acl_full_control_verify_attributes
test_object_acl_full_control_verify_owner
test_object_acl_read
test_object_acl_readacp
test_object_acl_write
test_object_acl_writeacp
test_object_anon_put
test_object_anon_put_write_access
test_put_bucket_acl_grant_group_read
# Bucket creation and naming
test_bucket_create_exists
@@ -356,7 +293,6 @@ test_bucket_list_delimiter_prefix
test_bucket_list_delimiter_prefix_underscore
test_bucket_list_many
test_bucket_list_maxkeys_one
test_bucket_list_objects_anonymous
test_bucket_list_objects_anonymous_fail
test_bucket_list_unordered
test_bucket_listv2_both_continuationtoken_startafter
@@ -364,14 +300,12 @@ test_bucket_listv2_delimiter_prefix
test_bucket_listv2_delimiter_prefix_underscore
test_bucket_listv2_many
test_bucket_listv2_maxkeys_one
test_bucket_listv2_objects_anonymous
test_bucket_listv2_objects_anonymous_fail
test_bucket_listv2_unordered
# Object copy tests
test_object_copy_16m
test_object_copy_diff_bucket
test_object_copy_not_owned_bucket
test_object_copy_replacing_metadata
test_object_copy_retaining_metadata
test_object_copy_same_bucket
@@ -388,20 +322,12 @@ test_object_content_encoding_aws_chunked
# Raw request tests
test_object_raw_authenticated
test_object_raw_authenticated_bucket_acl
test_object_raw_authenticated_bucket_gone
test_object_raw_authenticated_object_acl
test_object_raw_authenticated_object_gone
test_object_raw_get
test_object_raw_get_bucket_acl
test_object_raw_get_bucket_gone
test_object_raw_get_object_acl
test_object_raw_get_object_gone
test_object_raw_get_x_amz_expires_out_range_zero
test_object_raw_response_headers
# POST Object tests
test_post_object_anonymous_request
test_post_object_authenticated_no_content_type
test_post_object_authenticated_request
test_post_object_authenticated_request_bad_access_key
@@ -420,8 +346,6 @@ test_post_object_missing_content_length_argument
test_post_object_missing_expires_condition
test_post_object_missing_signature
test_post_object_no_key_specified
test_post_object_set_invalid_success_code
test_post_object_set_success_code
test_post_object_upload_larger_than_chunk
test_post_object_upload_size_below_minimum
test_post_object_upload_size_limit_exceeded
@@ -474,18 +398,13 @@ test_object_checksum_crc64nvme
# CORS tests
test_set_cors
test_cors_origin_response
test_cors_origin_wildcard
test_cors_header_option
test_cors_presigned_get_object
test_cors_presigned_get_object_tenant
test_cors_presigned_put_object
test_cors_presigned_put_object_with_acl
test_cors_presigned_put_object_tenant
test_cors_presigned_put_object_tenant_with_acl
# HTTP and DeleteObject tests
test_100_continue
test_abort_multipart_upload_not_found
test_post_object_tags_authenticated_request
test_object_delete_key_bucket_gone
+4 -4
View File
@@ -1,8 +1,9 @@
# Non-standard S3 tests (Ceph/RGW/MinIO specific)
# ================================================
# Legacy non-standard S3 test list (compatibility fallback)
# =========================================================
#
# This file is kept for backward compatibility.
# Active exclusion classification now uses excluded_tests.txt.
# These tests use vendor-specific extensions not part of AWS S3 API.
# They are PERMANENTLY EXCLUDED from RustFS compatibility testing.
#
# Exclusion reasons:
# - fails_on_aws marker: Ceph-specific features
@@ -19,7 +20,6 @@ test_account_usage
test_atomic_conditional_write_1mb
test_atomic_dual_conditional_write_1mb
test_atomic_write_bucket_gone
test_bucket_create_exists_nonowner
test_bucket_get_location
test_bucket_head_extended
test_bucket_header_acl_grants
+18 -11
View File
@@ -61,9 +61,9 @@ log_error() {
# Test Classification Files
# =============================================================================
# Tests are classified into three categories stored in text files:
# - non_standard_tests.txt: Ceph/RGW specific tests (permanently excluded)
# - unimplemented_tests.txt: Standard S3 features not yet implemented
# - implemented_tests.txt: Tests that should pass on RustFS
# - unimplemented_tests.txt: Standard S3 features planned but not yet implemented
# - excluded_tests.txt: Tests intentionally excluded from RustFS gating
#
# By default, only tests listed in implemented_tests.txt are run.
# Use TESTEXPR env var to override and run custom test selection.
@@ -72,8 +72,9 @@ log_error() {
# Test list files location
TEST_LISTS_DIR="${SCRIPT_DIR}"
IMPLEMENTED_TESTS_FILE="${TEST_LISTS_DIR}/implemented_tests.txt"
NON_STANDARD_TESTS_FILE="${TEST_LISTS_DIR}/non_standard_tests.txt"
UNIMPLEMENTED_TESTS_FILE="${TEST_LISTS_DIR}/unimplemented_tests.txt"
EXCLUDED_TESTS_FILE="${TEST_LISTS_DIR}/excluded_tests.txt"
LEGACY_NON_STANDARD_TESTS_FILE="${TEST_LISTS_DIR}/non_standard_tests.txt"
# =============================================================================
# build_testexpr_from_file: Read test names from file and build pytest -k expr
@@ -110,7 +111,7 @@ build_testexpr_from_file() {
# MARKEXPR: pytest marker expression (safety net for marker-based filtering)
# =============================================================================
# Even though we use file-based test selection, we keep marker exclusions
# as a safety net to ensure no non-standard tests slip through.
# as a safety net to ensure excluded tests do not slip through.
# =============================================================================
if [[ -z "${MARKEXPR:-}" ]]; then
# Minimal marker exclusions as safety net (file-based filtering is primary)
@@ -121,7 +122,7 @@ fi
# TESTEXPR: pytest -k expression to select specific tests
# =============================================================================
# By default, builds an inclusion expression from implemented_tests.txt,
# combined with an exclusion expression from non_standard_tests.txt and
# combined with an exclusion expression from excluded_tests.txt and
# unimplemented_tests.txt to prevent substring-matching collisions.
#
# For example, "test_object_raw_get" in the include list would also match
@@ -144,10 +145,16 @@ if [[ -z "${TESTEXPR:-}" ]]; then
TEST_COUNT=$(grep -v '^#' "${IMPLEMENTED_TESTS_FILE}" | grep -v '^[[:space:]]*$' | wc -l | xargs)
log_info "Loaded ${TEST_COUNT} tests from implemented_tests.txt"
# Build exclusion expression from non-standard and unimplemented lists
# Build exclusion expression from excluded and unimplemented lists
# to guard against pytest -k substring matching false positives
EXCLUDE_EXPR=""
for exclude_file in "${NON_STANDARD_TESTS_FILE}" "${UNIMPLEMENTED_TESTS_FILE}"; do
EXCLUDE_FILES=("${EXCLUDED_TESTS_FILE}" "${UNIMPLEMENTED_TESTS_FILE}")
if [[ ! -f "${EXCLUDED_TESTS_FILE}" && -f "${LEGACY_NON_STANDARD_TESTS_FILE}" ]]; then
log_warn "excluded_tests.txt not found, fallback to legacy non_standard_tests.txt"
EXCLUDE_FILES=("${LEGACY_NON_STANDARD_TESTS_FILE}" "${UNIMPLEMENTED_TESTS_FILE}")
fi
for exclude_file in "${EXCLUDE_FILES[@]}"; do
if [[ -f "${exclude_file}" ]]; then
FILE_EXPR=$(build_testexpr_from_file "${exclude_file}")
if [[ -n "${FILE_EXPR}" ]]; then
@@ -161,7 +168,7 @@ if [[ -z "${TESTEXPR:-}" ]]; then
if [[ -n "${EXCLUDE_EXPR}" ]]; then
TESTEXPR="(${INCLUDE_EXPR}) and not (${EXCLUDE_EXPR})"
log_info "Added exclusion guard from non_standard + unimplemented lists"
log_info "Added exclusion guard from excluded + unimplemented lists"
else
TESTEXPR="${INCLUDE_EXPR}"
fi
@@ -246,9 +253,9 @@ Environment Variables:
Final path: \${DATA_ROOT}/test-data/\${CONTAINER_NAME}
Test Classification Files (in scripts/s3-tests/):
implemented_tests.txt - Tests that should pass (run by default)
unimplemented_tests.txt - Standard S3 features not yet implemented
non_standard_tests.txt - Ceph/RGW specific tests (permanently excluded)
implemented_tests.txt - Implemented tests (run by default)
unimplemented_tests.txt - Standard S3 tests planned but not implemented
excluded_tests.txt - Tests intentionally excluded from RustFS gating
Notes:
- Tests are loaded from implemented_tests.txt by default
+6 -2
View File
@@ -1,8 +1,8 @@
# Unimplemented S3 feature tests
# ==============================
#
# These tests cover STANDARD S3 features not yet implemented in RustFS.
# They are TEMPORARILY EXCLUDED and should be enabled as features are added.
# These tests cover STANDARD S3 features planned but not yet implemented in RustFS.
# They are TEMPORARILY EXCLUDED and should move to implemented_tests.txt when done.
#
# Unimplemented features:
# - Bucket Logging: Access logging
@@ -12,6 +12,8 @@
# Failed tests
test_bucket_create_delete_bucket_ownership
test_bucket_logging_owner
test_object_copy_not_owned_bucket
test_bucket_policy_multipart
test_post_object_upload_checksum
test_put_bucket_logging
test_put_bucket_logging_errors
@@ -29,3 +31,5 @@ test_lifecycle_transition_encrypted
# Tests with known issues (need further investigation)
test_bucket_policy_different_tenant
test_bucket_policy_tenanted_bucket
# Flaky in CI: version count assertion (expects 5, gets 1) - timing/concurrency
test_versioned_concurrent_object_create_concurrent_remove