fix(ecstore): tolerate illumos/Solaris EEXIST for non-empty directory removal (#4995)

POSIX lets rmdir report a non-empty directory as either ENOTEMPTY or EEXIST. Linux/macOS/Windows use ENOTEMPTY (ErrorKind::DirectoryNotEmpty); illumos/Solaris return EEXIST (errno 17), which Rust surfaces as ErrorKind::AlreadyExists and which a DirectoryNotEmpty match never catches.

LocalDisk::delete_file removes xl.meta and then recurses to rmdir the object directory, tolerating only NotFound and DirectoryNotEmpty. Since #4300 (transactional delete rollback-staging, new in beta9) the object directory still holds the rollback backup dir when that rmdir runs — the caller removes it only after write quorum is confirmed — so the rmdir legitimately reports "not empty". On Linux that is tolerated; on Solaris it is EEXIST, which fell through to the catch-all arm and became FileAccessDeniedWithContext. That failed the delete commit, rolled the metadata back, and left the object undeletable, so the client retried indefinitely with a spurious FileAccessDenied and no EACCES anywhere (rustfs/rustfs#4978). The same Linux-errno assumption also broke non-force DeleteBucket on a populated bucket on Solaris.

Add a portable is_dir_not_empty_error classifier (DirectoryNotEmpty kind plus raw ENOTEMPTY/EEXIST), mirroring MinIO's isSysErrNotEmpty, and use it at the two directory-removal sites via is_benign_object_rmdir_error (delete_file) and classify_delete_volume_error (delete_volume). The classifier is applied only at rmdir/remove_dir_all sites, where EEXIST unambiguously means "not empty", so EEXIST keeps its normal meaning everywhere else. rmdir never returns EEXIST on Linux/macOS/Windows, so the new raw-errno branch is unreachable there and the change is a strict no-op off illumos/Solaris.

Adds unit tests for the classifier (DirectoryNotEmpty/ENOTEMPTY/EEXIST match, EACCES/ENOENT reject, real non-empty rmdir against the host errno) and call-site decision tests that make a Solaris EEXIST regression detectable on Linux CI.

Fixes #4978

crates/ecstore/src/disk/os.rs

@@ -645,6 +645,34 @@ pub fn file_exists(path: impl AsRef<Path>) -> bool {
     std::fs::metadata(path.as_ref()).map(|_| true).unwrap_or(false)
 }
 
+/// Whether an [`io::Error`] means "the directory is not empty".
+///
+/// POSIX lets `rmdir`/`rename` report a non-empty directory as either
+/// `ENOTEMPTY` or `EEXIST`. Linux uses `ENOTEMPTY` (which Rust surfaces as
+/// [`io::ErrorKind::DirectoryNotEmpty`]); illumos/Solaris return `EEXIST`
+/// (errno 17), which Rust surfaces as [`io::ErrorKind::AlreadyExists`] and
+/// which the `DirectoryNotEmpty` kind therefore never catches. Matching only on
+/// the kind silently misclassifies the Solaris case as a hard failure, so
+/// callers that must treat a still-populated directory as benign (deleting the
+/// object metadata while a rollback-staging dir remains, non-force
+/// `DeleteBucket` on a populated bucket) have to test the raw errno as well.
+/// Mirrors MinIO's `isSysErrNotEmpty`.
+pub fn is_dir_not_empty_error(err: &io::Error) -> bool {
+    // Linux/Windows: ENOTEMPTY / ERROR_DIR_NOT_EMPTY -> DirectoryNotEmpty.
+    if err.kind() == io::ErrorKind::DirectoryNotEmpty {
+        return true;
+    }
+    // illumos/Solaris report a non-empty `rmdir`/`rename` as EEXIST (errno 17),
+    // which Rust surfaces as `AlreadyExists` (so the `DirectoryNotEmpty` kind
+    // never catches it). Confirm against the raw errno directly so the
+    // classification holds regardless of how the platform std maps it.
+    #[cfg(unix)]
+    if matches!(err.raw_os_error(), Some(libc::ENOTEMPTY) | Some(libc::EEXIST)) {
+        return true;
+    }
+    false
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -809,6 +837,62 @@ mod tests {
         assert!(!should_retry_rename(&denied, 1));
     }
 
+    #[test]
+    fn is_dir_not_empty_error_recognizes_directory_not_empty_kind() {
+        let err = io::Error::from(io::ErrorKind::DirectoryNotEmpty);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn is_dir_not_empty_error_recognizes_raw_enotempty() {
+        // Linux/BSD/macOS non-empty rmdir/rename errno.
+        let err = io::Error::from_raw_os_error(libc::ENOTEMPTY);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn is_dir_not_empty_error_recognizes_solaris_eexist() {
+        // illumos/Solaris report a non-empty rmdir/rename as EEXIST, which Rust
+        // surfaces as `AlreadyExists` (never `DirectoryNotEmpty`). This is the
+        // core of rustfs/rustfs#4978: matching only the kind misclassified this
+        // benign condition as a hard failure.
+        let err = io::Error::from_raw_os_error(libc::EEXIST);
+        assert_eq!(err.kind(), io::ErrorKind::AlreadyExists);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[test]
+    fn is_dir_not_empty_error_rejects_unrelated_errors() {
+        assert!(!is_dir_not_empty_error(&io::Error::from(io::ErrorKind::NotFound)));
+        assert!(!is_dir_not_empty_error(&io::Error::from(io::ErrorKind::PermissionDenied)));
+        #[cfg(unix)]
+        {
+            assert!(!is_dir_not_empty_error(&io::Error::from_raw_os_error(libc::EACCES)));
+            assert!(!is_dir_not_empty_error(&io::Error::from_raw_os_error(libc::ENOENT)));
+        }
+    }
+
+    #[tokio::test]
+    async fn is_dir_not_empty_error_matches_real_non_empty_rmdir() {
+        // Validate against the host's actual errno, whatever it is: Linux/macOS
+        // return ENOTEMPTY, illumos/Solaris return EEXIST. The removal must be
+        // classified as "not empty" on every platform.
+        let temp_dir = tempdir().expect("create temp dir");
+        let populated = temp_dir.path().join("populated");
+        std::fs::create_dir(&populated).expect("create dir");
+        std::fs::write(populated.join("child"), b"x").expect("write child");
+
+        let err = std::fs::remove_dir(&populated).expect_err("non-empty rmdir must fail");
+        assert!(
+            is_dir_not_empty_error(&err),
+            "non-empty rmdir must classify as not-empty, got kind {:?} errno {:?}",
+            err.kind(),
+            err.raw_os_error()
+        );
+    }
+
     #[tokio::test]
     async fn rename_all_moves_existing_directory_tree() {
         // Guards the rename_data commit path, which funnels through