mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-09 22:59:59 +00:00
e3a8234bc9
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking DecompressReader::poll_read and DecryptReader::poll_read sliced the block body with a fixed `[0..16]` index to read the length varint. The body length comes from an untrusted 24-bit header field, so a corrupted/truncated block shorter than 16 bytes made the slice panic and crash the request task — a read-path DoS on GET of tiered/corrupted data. Pass the whole (arbitrary-length-safe) slice to uvarint and reject a non-positive or out-of-range length prefix with InvalidData. Adds a repro test for each reader; all existing round-trip tests still pass. Refs rustfs/backlog#812 * fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses validate_outbound_ip branched on the IpAddr variant, and the V6 branch's is_loopback/is_unicast_link_local/is_unique_local checks never inspect the embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1, ::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard, letting an attacker reach loopback/private/metadata endpoints. Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which matches only the true mapped form) before classification. Adds reject tests for mapped loopback/private/metadata and an allow test for public IPv6. Refs rustfs/backlog#813 * fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment Four confirmed data-reliability defects: - put_object_multipart_stream: the CompleteMultipartUpload part-collection loop used exclusive `1..total_parts_count`, dropping the final part (and collecting zero parts for a single-part object) — silently truncating the completed object. Extracted collect_complete_parts (1..=total_parts_count) with unit tests. - GCS warm backend get() ignored the requested byte range, returning the whole object for a Range GET; now applies ReadRange::segment like the other backends. - GCS warm backend remove() was an empty stub, so deleting a tiered object left it on GCS forever; now deletes via StorageControl (added a control-plane client), and in_use() actually lists (prefix-scoped) instead of always returning false. - stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a compressed, misaligned error vector; heal_object_dir then zipped it against the full disks array and could make_volume on the WRONG disk. Now returns one index-aligned entry per slot (None -> DiskNotFound), and heal no longer pre-fills the drive report (which would double it). Added an alignment test. Refs rustfs/backlog#807 * fix(kms): stop Vault backend from destroying/reviving keys on failure Two confirmed key-safety defects in the Vault KV2 backend: - get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a fresh random master key and overwriting the stored value. That destroys the original key material, making every DEK ever wrapped by it permanently undecryptable. Decryption must never mutate the stored key: both branches now return a cryptographic_error instead. (The empty-material bootstrap path, which only fills a never-initialized key, is intentionally left intact.) - cancel_key_deletion() reset key_state to Enabled only in the returned response and never persisted it, so the key stayed PendingDeletion in storage and would still be reaped. It now writes the state back via update_key_metadata_in_storage and fails the request if the write fails. Adds ignored (Vault-requiring) integration tests documenting both behaviours. The third item (VaultTransit key state only in memory -> revived as Enabled after restart) is deferred: a fail-closed guard would break restart availability for all transit keys; the correct fix needs a persistent metadata store + Vault integration testing. Tracked in rustfs/backlog#808. Refs rustfs/backlog#808 * fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk Two confirmed admin-API defects: - Standard AssumeRole used the raw client-supplied DurationSeconds with no upper bound, so a caller could mint near-permanent temporary credentials. Clamp it to the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared clamp_assume_role_duration helper, and build the exp claim with saturating_add. This matches the existing AssumeRoleWithWebIdentity path. - ImportBucketMetadata only mutated an in-memory map and returned 200, silently dropping every imported config. It now persists each non-empty config via metadata_sys::update (which merges onto existing on-disk metadata) and returns InternalError if a write fails. Mapping extracted to imported_configs_to_persist with unit tests. Refs rustfs/backlog#809 * fix(heal): enqueue displacing request in release builds push_displacing_lower_priority folded the real enqueue call into debug_assert_eq!(self.push(request), Accepted). In release builds (debug_assertions off) the whole macro — including its argument — is compiled out, so after evicting a lower-priority queued item the new high-priority request was silently dropped and never healed. Hoist self.push(request) out of the assertion so the side effect runs in all builds. Adds a --release regression test. Refs rustfs/backlog#811 * fix(iam): propagate real delete_policy backend errors instead of swallowing them delete_policy's is_from_notify path had its error handling inverted: a real backend failure (disk IO / insufficient quorum) evicted the cache and returned Ok(()), reporting a phantom success while policy.json survived on disk (to be reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent success — returned Err. Propagate real errors and let NoSuchPolicy fall through to the idempotent cache-evict + Ok, matching delete_user / the notification handler in the same file. Adds a backend-error-injection regression test. Refs rustfs/backlog#810 * fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254) still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::) first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests for compatible-form loopback/metadata and confirms ::1 / :: stay rejected. Found by adversarial review of the initial fix. Refs rustfs/backlog#813 * fix(ecstore): fix the same last-part loss in the parallel streaming path put_object_multipart_stream_parallel had the identical off-by-one (1..total_parts_count) that truncated the last part / produced zero parts for a single-part upload — reachable when concurrent stream parts are enabled. Reuse collect_complete_parts, which now returns an error instead of panicking on a gap in the parts map. Adds a missing-part error test. Found by adversarial review of the initial fix. Refs rustfs/backlog#807 * fix(kms): local backend must preserve key material on status change LocalKmsClient (the default KMS backend) regenerated the master key material on enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status change. A single disable+enable cycle therefore destroyed the original key, making every DEK ever wrapped by it permanently undecryptable (silent data loss, no network needed). Preserve the existing material via get_key_material and re-save with only the status changed. Adds a hermetic regression test that wraps a DEK, cycles all four status methods, and asserts the DEK still decrypts. Found by adversarial review of the Vault fix. Refs rustfs/backlog#808 * test(rio): cover the length-prefix guard; correct its comment Add a DecompressReader test that feeds an unterminated length varint so uvarint returns 0 and the new guard (not the downstream codec) produces the InvalidData error, and reword the guard comment which overclaimed that the > len bound prevents a reachable panic (it is belt-and-suspenders). No behavior change. Found by adversarial review. Refs rustfs/backlog#812 * test(rio): build test block headers via vec! to satisfy clippy The new corrupted-block tests built the header with Vec::new() + repeated push, tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed header bytes with vec![] instead. No behavior change. --------- Co-authored-by: houseme <housemecn@gmail.com>
310 lines
11 KiB
Rust
310 lines
11 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
use std::fmt;
|
|
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
|
|
use url::Url;
|
|
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub enum OutboundUrlError {
|
|
MissingHost,
|
|
ForbiddenHost { host: String, reason: &'static str },
|
|
}
|
|
|
|
impl fmt::Display for OutboundUrlError {
|
|
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
match self {
|
|
OutboundUrlError::MissingHost => write!(f, "outbound URL is missing a host"),
|
|
OutboundUrlError::ForbiddenHost { host, reason } => {
|
|
write!(f, "outbound URL host '{host}' is not allowed: {reason}")
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
impl std::error::Error for OutboundUrlError {}
|
|
|
|
pub fn validate_outbound_url(url: &Url) -> Result<(), OutboundUrlError> {
|
|
let Some(raw_host) = url.host_str() else {
|
|
return Err(OutboundUrlError::MissingHost);
|
|
};
|
|
let normalized_host = raw_host.trim_end_matches('.').trim_matches(['[', ']']);
|
|
|
|
if normalized_host.eq_ignore_ascii_case("localhost") {
|
|
return Err(OutboundUrlError::ForbiddenHost {
|
|
host: raw_host.to_string(),
|
|
reason: "loopback host",
|
|
});
|
|
}
|
|
|
|
let Ok(ip) = normalized_host.parse::<IpAddr>() else {
|
|
return Ok(());
|
|
};
|
|
|
|
validate_outbound_ip(ip).map_err(|reason| OutboundUrlError::ForbiddenHost {
|
|
host: raw_host.to_string(),
|
|
reason,
|
|
})
|
|
}
|
|
|
|
fn validate_outbound_ip(ip: IpAddr) -> Result<(), &'static str> {
|
|
// Reject pure-IPv6 special forms first, before any IPv4 normalization. ::1 (loopback) and ::
|
|
// (unspecified) are technically IPv4-compatible forms too, so normalizing first would map
|
|
// them to a harmless-looking 0.0.0.1 / 0.0.0.0 and let them through.
|
|
if let IpAddr::V6(v6) = ip {
|
|
if v6.is_loopback() {
|
|
return Err("loopback address");
|
|
}
|
|
if v6.is_unspecified() {
|
|
return Err("unspecified address");
|
|
}
|
|
if v6.is_unicast_link_local() {
|
|
return Err("link-local address");
|
|
}
|
|
if v6.is_unique_local() {
|
|
return Err("private address");
|
|
}
|
|
}
|
|
|
|
// Normalize IPv4-mapped (::ffff:a.b.c.d) AND IPv4-compatible (::a.b.c.d) IPv6 addresses to
|
|
// their embedded IPv4 so the IPv4 rules below apply. The std is_* checks on the IPv6 variant
|
|
// never inspect the embedded IPv4, so without this an attacker bypasses the guard with e.g.
|
|
// ::ffff:127.0.0.1, ::127.0.0.1 (loopback) or ::169.254.169.254 (cloud metadata service).
|
|
let ip = match ip {
|
|
IpAddr::V6(v6) => match embedded_ipv4(v6) {
|
|
Some(v4) => IpAddr::V4(v4),
|
|
None => IpAddr::V6(v6),
|
|
},
|
|
other => other,
|
|
};
|
|
|
|
if ip.is_unspecified() {
|
|
return Err("unspecified address");
|
|
}
|
|
|
|
if ip == IpAddr::V4(Ipv4Addr::new(169, 254, 169, 254)) {
|
|
return Err("metadata endpoint");
|
|
}
|
|
|
|
match ip {
|
|
IpAddr::V4(ipv4) => {
|
|
if ipv4.is_loopback() {
|
|
return Err("loopback address");
|
|
}
|
|
if ipv4.is_link_local() {
|
|
return Err("link-local address");
|
|
}
|
|
if ipv4.is_private() {
|
|
return Err("private address");
|
|
}
|
|
}
|
|
// Genuine IPv6 (no embedded IPv4) was already classified above.
|
|
IpAddr::V6(_) => {}
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
/// Extract the embedded IPv4 from an IPv4-mapped (`::ffff:a.b.c.d`) or IPv4-compatible
|
|
/// (`::a.b.c.d`) IPv6 address. The pure-IPv6 specials `::` and `::1` are rejected by the caller
|
|
/// before this runs, so returning `None` here means a genuine IPv6 host.
|
|
fn embedded_ipv4(v6: Ipv6Addr) -> Option<Ipv4Addr> {
|
|
if let Some(v4) = v6.to_ipv4_mapped() {
|
|
return Some(v4);
|
|
}
|
|
// IPv4-compatible: the top 96 bits are zero and the low 32 bits carry the IPv4.
|
|
let segs = v6.segments();
|
|
if segs[0..6] == [0, 0, 0, 0, 0, 0] {
|
|
let hi = segs[6].to_be_bytes();
|
|
let lo = segs[7].to_be_bytes();
|
|
let v4 = Ipv4Addr::new(hi[0], hi[1], lo[0], lo[1]);
|
|
// `::` and `::1` are already handled by the caller; anything else is a real embedded v4.
|
|
if !v4.is_unspecified() && v4 != Ipv4Addr::new(0, 0, 0, 1) {
|
|
return Some(v4);
|
|
}
|
|
}
|
|
None
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::{OutboundUrlError, validate_outbound_url};
|
|
use url::Url;
|
|
|
|
#[test]
|
|
fn validate_outbound_url_allows_public_hostname() {
|
|
let url = Url::parse("https://example.com/webhook").expect("public URL should parse");
|
|
assert!(validate_outbound_url(&url).is_ok());
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_localhost() {
|
|
let url = Url::parse("https://localhost/webhook").expect("localhost URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("localhost should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "loopback host",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_loopback_ip() {
|
|
let url = Url::parse("https://127.0.0.1/webhook").expect("loopback URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("loopback IP should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "loopback address",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_private_ip() {
|
|
let url = Url::parse("https://10.0.0.5/webhook").expect("private URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("private IP should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "private address",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_metadata_endpoint() {
|
|
let url = Url::parse("http://169.254.169.254/latest/meta-data").expect("metadata URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("metadata endpoint should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "metadata endpoint",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_link_local_ipv6() {
|
|
let url = Url::parse("https://[fe80::1]/hook").expect("IPv6 URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("link-local IPv6 should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "link-local address",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_ipv4_mapped_loopback() {
|
|
let url = Url::parse("http://[::ffff:127.0.0.1]/webhook").expect("mapped loopback URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("IPv4-mapped loopback should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "loopback address",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_ipv4_mapped_private() {
|
|
let url = Url::parse("http://[::ffff:10.0.0.5]/webhook").expect("mapped private URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("IPv4-mapped private should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "private address",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_ipv4_mapped_metadata_endpoint() {
|
|
let url = Url::parse("http://[::ffff:169.254.169.254]/latest/meta-data").expect("mapped metadata URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("IPv4-mapped metadata endpoint should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "metadata endpoint",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_still_allows_public_ipv6() {
|
|
// Pure public IPv6 (Google DNS) must remain allowed after normalization.
|
|
let url = Url::parse("https://[2001:4860:4860::8888]/webhook").expect("public IPv6 URL should parse");
|
|
assert!(validate_outbound_url(&url).is_ok());
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_ipv4_compatible_loopback() {
|
|
// IPv4-compatible form ::a.b.c.d (deprecated but still routable) must also be caught.
|
|
let url = Url::parse("http://[::127.0.0.1]/webhook").expect("compatible loopback URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("IPv4-compatible loopback should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "loopback address",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_ipv4_compatible_metadata_endpoint() {
|
|
let url = Url::parse("http://[::169.254.169.254]/latest/meta-data").expect("compatible metadata URL should parse");
|
|
let err = validate_outbound_url(&url).expect_err("IPv4-compatible metadata endpoint should be rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "metadata endpoint",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_outbound_url_rejects_ipv6_loopback_and_unspecified() {
|
|
// ::1 / :: must stay rejected even though they look like IPv4-compatible forms.
|
|
let err = validate_outbound_url(&Url::parse("http://[::1]/x").unwrap()).expect_err("::1 rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "loopback address",
|
|
..
|
|
}
|
|
));
|
|
let err = validate_outbound_url(&Url::parse("http://[::]/x").unwrap()).expect_err(":: rejected");
|
|
assert!(matches!(
|
|
err,
|
|
OutboundUrlError::ForbiddenHost {
|
|
reason: "unspecified address",
|
|
..
|
|
}
|
|
));
|
|
}
|
|
}
|