Zhengchao An
22741603f5
- common.rs gains an AdminTransport knob (Signed | Awscurl) with admin_execute_at plus three family wrappers: admin_create_user_via, admin_add_canned_policy_via, admin_attach_user_policy_via; the existing admin_create_user now delegates over the Signed transport. - Deleted the four signed admin request clones in admin_mfa_test, admin_auth_test, reliant/tiering, and inline_fast_path_cluster_test; each keeps a thin local wrapper over common::admin_request so call sites keep their Option<&str> body shape. - Deduped the notification_webhook signer onto common::signed_request and the webdav_core signer plus its three admin helpers onto the shared _via helpers. - Consolidated the S3-client-with-credentials builders: admin_auth s3_client_with, existing_object_tag user_client/sts_session_client, bucket_policy_check create_user_client, and the create_user_s3_client copies in group_delete_test and replication_extension_test now delegate to create_s3_client_with_credentials / build_test_s3_config; replication_extension admin_add_canned_policy and admin_attach_policy_to_user route through the _via helpers on the Signed transport. - The awscurl-gated suites (existing_object_tag_policy, bucket_policy_check, policy/policy_variables) keep going through the external awscurl binary via AdminTransport::Awscurl, preserving their wire behavior. Part of rustfs/backlog#1846 (cluster 2).
Protocol E2E Tests
FTPS, WebDAV, and SFTP protocol end-to-end tests for RustFS.
Prerequisites
No external SSH tooling is required. The test framework generates ed25519 host keys in-process via russh::keys under the per-test temp directory before each SFTP server spawn, and russh-sftp drives the protocol from the test process directly.
Running Tests
Use the canonical CI-equivalent protocol command in the parent
e2e_test README.
For targeted debugging of the core suite only:
RUSTFS_BUILD_FEATURES=ftps,webdav,sftp cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
This targeted command does not cover the full e2e-protocols profile.
RUSTFS_BUILD_FEATURES controls which features the test rustfs binary is
built with. When this variable is set, the protocol test runner schedules
only entries whose protocol is present in the requested feature list. Leave
it unset to run every protocol entry.
--test-threads=1 is required because every entry spawns a rustfs server
on fixed bind ports.
Test Coverage
FTPS Tests
- mkdir bucket
- cd to bucket
- put file
- ls list objects
- cd . (stay in current directory)
- cd / (return to root)
- cd nonexistent bucket (should fail)
- delete object
- cdup
- rmdir delete bucket
WebDAV Tests
- PROPFIND at root (list buckets)
- MKCOL (create bucket)
- PUT (upload file)
- GET (download file)
- PROPFIND on bucket (list objects)
- DELETE file
- MOVE file (rename object)
- DELETE bucket
- Authentication failure test
SFTP Tests
The SFTP suite lives in three entries plus a standalone idle-timeout case.
Every assertion runs against a freshly spawned rustfs binary with
RUSTFS_SFTP_ENABLE=true; the test framework also pins
RUSTFS_SFTP_PART_SIZE=5242880 so the multipart boundary is deterministic.
sftp_core (test_sftp_core_operations)
Bind ports 9022 (SFTP) and 9200 (S3). 22 in-suite assertions covering the core protocol surface plus cross-protocol consistency:
- Subsystem canary: SFTPv3 version exchange completes after password auth
- Bucket lifecycle: mkdir, root listing, rmdir, post-delete listing
- Small-file round-trip with SHA256 compare
- Stat on a file (size + file type) and on a bucket (directory)
- SETSTAT on a path returns ok
- Rename within bucket, listing reflects the rename
- Multipart-sized round-trip (just over 2 × part_size) with SHA256 compare
- Negative cases: symlink rejected, open of nonexistent file rejected, read_dir of nonexistent bucket rejected, path traversal rejected
- Spec-letter assertions: APPEND open returns an error, CREATE+EXCLUDE on an existing path returns an error, bad-password authentication is rejected
- Cross-protocol via aws-sdk-s3: SFTP write then S3 read with SHA256 match, S3 write then SFTP read with SHA256 match
- Cross-API directory visibility: SFTP-created sub-directory visible via S3
ListObjectsV2, S3-created
__XLDIR__marker visible via SFTP readdir as a directory entry
sftp_compliance (test_sftp_compliance_suite)
Bind ports 9024 (SFTP) and 9300 (S3). 14 compliance regression cases against one shared server spawn. Each case carries a stable CMPTST-NN identifier:
- CMPTST-01: medium-binary upload then download with SHA256 compare (single-shot PutObject path below the multipart boundary)
- CMPTST-02: zero-byte upload, download, and stat-size match
- CMPTST-03: rm against a bucket path is rejected; the bucket is preserved
- CMPTST-04: rmdir against a non-empty bucket is rejected; the contained object survives
- CMPTST-05: rmdir against a non-empty sub-directory is rejected; the inner object survives
- CMPTST-06: open with a path-traversal pattern cannot leak a host file via SFTP read
- CMPTST-07: read_dir of
/..either errors or returns a listing that contains no host system entries - CMPTST-08: rename across buckets preserves payload and removes the source object
- CMPTST-09: paths with embedded spaces round-trip through the russh-sftp client
- CMPTST-10: read_link is rejected (S3 storage has no symlinks)
- CMPTST-11: SETSTAT on a path and FSETSTAT on a separate open handle both return ok (rsync, WinSCP transfer-success contract)
- CMPTST-12: rename to the same path is a no-op; the file persists with the original payload
- CMPTST-13: implicit-directory round-trip; uploading to a nested key creates the parent directory implicitly and three listing forms surface the inner file
- CMPTST-14: OPEN, WRITE, FSETSTAT, CLOSE on the same write handle all return ok (WinSCP wire shape)
sftp_compliance_readonly (test_sftp_compliance_readonly)
Bind ports 9025 (SFTP) and 9301 (S3). Spawns a second rustfs binary with
RUSTFS_SFTP_READ_ONLY=true; the S3 endpoint stays writable so the suite
can seed a bucket and a fixture object via aws-sdk-s3 before opening the
SFTP session. 7 compliance cases:
- CMPTST-15: put through SFTP is rejected
- CMPTST-16: rm through SFTP is rejected
- CMPTST-17: mkdir through SFTP is rejected
- CMPTST-18: rmdir through SFTP is rejected
- CMPTST-19: rename through SFTP is rejected
- CMPTST-20: ls through SFTP is allowed and lists the seeded bucket
- CMPTST-21: get through SFTP is allowed and returns the seeded payload byte-for-byte
The full case index lives at the top of sftp_compliance.rs; each helper's
log lines name its CMPTST-NN code so a failure in CI points at one named
property without consulting any external doc.
sftp_idle_timeout (test_sftp_idle_timeout_disconnects)
Bind ports 9023 (SFTP) and 9100 (S3). Spawns rustfs with
RUSTFS_SFTP_IDLE_TIMEOUT=5, sleeps 10 s past the timeout, then issues an
SFTP request and asserts the server has closed the session.