mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-25 21:46:50 +00:00
feat(sftp): add SFTPv3 protocol support (#2875)
Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
escapecode
@@ -1,38 +1,26 @@
|
||||
# Protocol E2E Tests
|
||||
|
||||
FTPS and WebDAV protocol end-to-end tests for RustFS.
|
||||
FTPS, WebDAV, and SFTP protocol end-to-end tests for RustFS.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Required Tools
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install sshpass ssh-keygen
|
||||
|
||||
# RHEL/CentOS
|
||||
sudo yum install sshpass openssh-clients
|
||||
|
||||
# macOS
|
||||
brew install sshpass openssh
|
||||
```
|
||||
No external SSH tooling is required. The test framework generates ed25519
|
||||
host keys in-process via russh::keys under the per-test temp directory
|
||||
before each SFTP server spawn, and russh-sftp drives the protocol from the
|
||||
test process directly.
|
||||
|
||||
## Running Tests
|
||||
|
||||
Run all protocol tests (FTPS + WebDAV):
|
||||
```bash
|
||||
RUSTFS_BUILD_FEATURES=ftps,webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
|
||||
RUSTFS_BUILD_FEATURES=ftps,webdav,sftp cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
|
||||
```
|
||||
|
||||
Run FTPS tests only:
|
||||
```bash
|
||||
RUSTFS_BUILD_FEATURES=ftps cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
|
||||
```
|
||||
|
||||
Run WebDAV tests only:
|
||||
```bash
|
||||
RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
|
||||
```
|
||||
`RUSTFS_BUILD_FEATURES` controls which features the test rustfs binary is
|
||||
built with. The protocol test runner schedules every entry (FTPS, WebDAV,
|
||||
SFTP) regardless of the feature set, so the binary must include every
|
||||
protocol the runner spawns or the corresponding entries will fail.
|
||||
`--test-threads=1` is required because every entry spawns a rustfs server
|
||||
on fixed bind ports.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
@@ -59,3 +47,90 @@ RUSTFS_BUILD_FEATURES=webdav cargo test --package e2e_test test_protocol_core_su
|
||||
- DELETE bucket
|
||||
- Authentication failure test
|
||||
|
||||
### SFTP Tests
|
||||
|
||||
The SFTP suite lives in three entries plus a standalone idle-timeout case.
|
||||
Every assertion runs against a freshly spawned rustfs binary with
|
||||
`RUSTFS_SFTP_ENABLE=true`; the test framework also pins
|
||||
`RUSTFS_SFTP_PART_SIZE=5242880` so the multipart boundary is deterministic.
|
||||
|
||||
#### sftp_core (`test_sftp_core_operations`)
|
||||
|
||||
Bind ports 9022 (SFTP) and 9200 (S3). 22 in-suite assertions covering the
|
||||
core protocol surface plus cross-protocol consistency:
|
||||
|
||||
- Subsystem canary: SFTPv3 version exchange completes after password auth
|
||||
- Bucket lifecycle: mkdir, root listing, rmdir, post-delete listing
|
||||
- Small-file round-trip with SHA256 compare
|
||||
- Stat on a file (size + file type) and on a bucket (directory)
|
||||
- SETSTAT on a path returns ok
|
||||
- Rename within bucket, listing reflects the rename
|
||||
- Multipart-sized round-trip (just over 2 × part_size) with SHA256 compare
|
||||
- Negative cases: symlink rejected, open of nonexistent file rejected,
|
||||
read_dir of nonexistent bucket rejected, path traversal rejected
|
||||
- Spec-letter assertions: APPEND open returns an error, CREATE+EXCLUDE on an
|
||||
existing path returns an error, bad-password authentication is rejected
|
||||
- Cross-protocol via aws-sdk-s3: SFTP write then S3 read with SHA256 match,
|
||||
S3 write then SFTP read with SHA256 match
|
||||
- Cross-API directory visibility: SFTP-created sub-directory visible via S3
|
||||
ListObjectsV2, S3-created `__XLDIR__` marker visible via SFTP readdir as a
|
||||
directory entry
|
||||
|
||||
#### sftp_compliance (`test_sftp_compliance_suite`)
|
||||
|
||||
Bind ports 9024 (SFTP) and 9300 (S3). 14 compliance regression cases against
|
||||
one shared server spawn. Each case carries a stable CMPTST-NN identifier:
|
||||
|
||||
- CMPTST-01: medium-binary upload then download with SHA256 compare
|
||||
(single-shot PutObject path below the multipart boundary)
|
||||
- CMPTST-02: zero-byte upload, download, and stat-size match
|
||||
- CMPTST-03: rm against a bucket path is rejected; the bucket is preserved
|
||||
- CMPTST-04: rmdir against a non-empty bucket is rejected; the contained
|
||||
object survives
|
||||
- CMPTST-05: rmdir against a non-empty sub-directory is rejected; the inner
|
||||
object survives
|
||||
- CMPTST-06: open with a path-traversal pattern cannot leak a host file via
|
||||
SFTP read
|
||||
- CMPTST-07: read_dir of `/..` either errors or returns a listing that
|
||||
contains no host system entries
|
||||
- CMPTST-08: rename across buckets preserves payload and removes the source
|
||||
object
|
||||
- CMPTST-09: paths with embedded spaces round-trip through the russh-sftp
|
||||
client
|
||||
- CMPTST-10: read_link is rejected (S3 storage has no symlinks)
|
||||
- CMPTST-11: SETSTAT on a path and FSETSTAT on a separate open handle both
|
||||
return ok (rsync, WinSCP transfer-success contract)
|
||||
- CMPTST-12: rename to the same path is a no-op; the file persists with the
|
||||
original payload
|
||||
- CMPTST-13: implicit-directory round-trip; uploading to a nested key
|
||||
creates the parent directory implicitly and three listing forms surface
|
||||
the inner file
|
||||
- CMPTST-14: OPEN, WRITE, FSETSTAT, CLOSE on the same write handle all
|
||||
return ok (WinSCP wire shape)
|
||||
|
||||
#### sftp_compliance_readonly (`test_sftp_compliance_readonly`)
|
||||
|
||||
Bind ports 9025 (SFTP) and 9301 (S3). Spawns a second rustfs binary with
|
||||
`RUSTFS_SFTP_READ_ONLY=true`; the S3 endpoint stays writable so the suite
|
||||
can seed a bucket and a fixture object via aws-sdk-s3 before opening the
|
||||
SFTP session. 7 compliance cases:
|
||||
|
||||
- CMPTST-15: put through SFTP is rejected
|
||||
- CMPTST-16: rm through SFTP is rejected
|
||||
- CMPTST-17: mkdir through SFTP is rejected
|
||||
- CMPTST-18: rmdir through SFTP is rejected
|
||||
- CMPTST-19: rename through SFTP is rejected
|
||||
- CMPTST-20: ls through SFTP is allowed and lists the seeded bucket
|
||||
- CMPTST-21: get through SFTP is allowed and returns the seeded payload
|
||||
byte-for-byte
|
||||
|
||||
The full case index lives at the top of `sftp_compliance.rs`; each helper's
|
||||
log lines name its CMPTST-NN code so a failure in CI points at one named
|
||||
property without consulting any external doc.
|
||||
|
||||
#### sftp_idle_timeout (`test_sftp_idle_timeout_disconnects`)
|
||||
|
||||
Bind ports 9023 (SFTP) and 9100 (S3). Spawns rustfs with
|
||||
`RUSTFS_SFTP_IDLE_TIMEOUT=5`, sleeps 10 s past the timeout, then issues an
|
||||
SFTP request and asserts the server has closed the session.
|
||||
|
||||
|
||||
@@ -12,9 +12,13 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Protocol tests for FTPS and WebDAV
|
||||
//! Protocol tests for FTPS, WebDAV, and SFTP
|
||||
|
||||
pub mod ftps_core;
|
||||
pub mod sftp_compliance;
|
||||
mod sftp_compliance_tests;
|
||||
pub mod sftp_core;
|
||||
pub mod sftp_helpers;
|
||||
pub mod test_env;
|
||||
pub mod test_runner;
|
||||
pub mod webdav_core;
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Public test entry points for the SFTP compliance suite.
|
||||
//!
|
||||
//! Three suite entries cover every CMPTST-NN identifier:
|
||||
//!
|
||||
//! - test_sftp_compliance_suite CMPTST-01..14 (one shared SFTP session)
|
||||
//! - test_sftp_compliance_readonly CMPTST-15..23 (one shared session, read-only mode)
|
||||
//! - test_sftp_compliance_standalone CMPTST-24..33 (each case spawns its own rustfs)
|
||||
//!
|
||||
//! The first two reuse a single rustfs spawn for the whole bracket
|
||||
//! because every case in the bracket exercises the same protocol
|
||||
//! against the same server. The third aggregates cases that each need
|
||||
//! a different server configuration (idle timeout, read-cache window,
|
||||
//! console disabled) and therefore cannot share a binary.
|
||||
//!
|
||||
//! Each per-case module exposes a single descriptive entry called
|
||||
//! run_<what_it_tests>(). For example cmptst_24 exposes
|
||||
//! cmptst_24::run_concurrent_half_close_no_leak().
|
||||
//!
|
||||
//! The per-case bodies (one cmptst_NN module per case) and the
|
||||
//! cross-case infrastructure (spawn helpers, fixture seeders, the
|
||||
//! half-close / wedge / paused-drain stream wrappers, and the session
|
||||
//! lifecycle counters) live in sftp_compliance_tests.rs. Per-case
|
||||
//! marker comments and the full case-index doc live there too.
|
||||
|
||||
use crate::protocols::sftp_compliance_tests::{
|
||||
cmptst_01, cmptst_02, cmptst_03, cmptst_04, cmptst_05, cmptst_06, cmptst_07, cmptst_08, cmptst_09, cmptst_10, cmptst_11,
|
||||
cmptst_12, cmptst_13, cmptst_14, cmptst_15, cmptst_16, cmptst_17, cmptst_18, cmptst_19, cmptst_20, cmptst_21, cmptst_22,
|
||||
cmptst_23, cmptst_24, cmptst_25, cmptst_26, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, spawn_compliance_rustfs,
|
||||
};
|
||||
use crate::protocols::sftp_helpers::{build_test_s3_client, connect_sftp_to, wait_for_s3_ready};
|
||||
use crate::protocols::test_env::ProtocolTestEnvironment;
|
||||
use anyhow::{Result, anyhow};
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use tracing::info;
|
||||
|
||||
// Read-write compliance suite ports. Distinct from sftp_core (9022/9200)
|
||||
// and from test_sftp_idle_timeout_disconnects (9023/9100) so the SFTP
|
||||
// entries can run sequentially without leftover-listener contention.
|
||||
const COMPLIANCE_RW_SFTP_PORT: u16 = 9024;
|
||||
const COMPLIANCE_RW_SFTP_ADDRESS: &str = "127.0.0.1:9024";
|
||||
const COMPLIANCE_RW_S3_ADDRESS: &str = "127.0.0.1:9300";
|
||||
|
||||
// Read-only compliance suite ports. The SFTP session opened against
|
||||
// this address runs against a server started with
|
||||
// RUSTFS_SFTP_READ_ONLY=true. The S3 endpoint stays writable so the
|
||||
// suite can seed a bucket and a fixture object before running the SFTP
|
||||
// rejection assertions.
|
||||
const COMPLIANCE_RO_SFTP_PORT: u16 = 9025;
|
||||
const COMPLIANCE_RO_SFTP_ADDRESS: &str = "127.0.0.1:9025";
|
||||
const COMPLIANCE_RO_S3_ADDRESS: &str = "127.0.0.1:9301";
|
||||
const COMPLIANCE_RO_S3_ENDPOINT: &str = "http://127.0.0.1:9301";
|
||||
const COMPLIANCE_RO_S3_READY_ATTEMPTS: u32 = 30;
|
||||
|
||||
/// Compliance suite entry: spawn one rustfs server, run every per-case
|
||||
/// helper that closes a coverage gap not exercised by sftp_core. Runs
|
||||
/// CMPTST-01 through CMPTST-14 against the same SFTP session.
|
||||
pub async fn test_sftp_compliance_suite() -> Result<()> {
|
||||
info!("Starting SFTP server for compliance suite on {}", COMPLIANCE_RW_SFTP_ADDRESS);
|
||||
let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RW_SFTP_ADDRESS, COMPLIANCE_RW_S3_ADDRESS, false).await?;
|
||||
|
||||
let result = async {
|
||||
ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RW_SFTP_PORT, 30)
|
||||
.await
|
||||
.map_err(|e| anyhow!("{}", e))?;
|
||||
|
||||
let (session, sftp) = connect_sftp_to(COMPLIANCE_RW_SFTP_ADDRESS).await?;
|
||||
|
||||
cmptst_01::run_medium_binary_round_trip(&sftp).await?;
|
||||
cmptst_02::run_zero_byte_round_trip(&sftp).await?;
|
||||
cmptst_03::run_rm_on_bucket_path_rejected(&sftp).await?;
|
||||
cmptst_04::run_rmdir_nonempty_bucket_rejected(&sftp).await?;
|
||||
cmptst_05::run_rmdir_nonempty_subdir_rejected(&sftp).await?;
|
||||
cmptst_06::run_path_traversal_get_rejected(&sftp).await?;
|
||||
cmptst_07::run_dotdot_collapses_to_root(&sftp).await?;
|
||||
cmptst_08::run_rename_cross_bucket(&sftp).await?;
|
||||
cmptst_09::run_path_with_spaces_round_trip(&sftp).await?;
|
||||
cmptst_10::run_readlink_rejected(&sftp).await?;
|
||||
cmptst_11::run_setstat_after_put_returns_ok(&sftp).await?;
|
||||
cmptst_12::run_rename_same_path_keeps_file(&sftp).await?;
|
||||
cmptst_13::run_implicit_dir_round_trip(&sftp).await?;
|
||||
cmptst_14::run_winscp_setstat_shape_on_handle(&sftp).await?;
|
||||
|
||||
drop(sftp);
|
||||
session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
|
||||
info!("SFTP compliance suite passed");
|
||||
Ok::<(), anyhow::Error>(())
|
||||
}
|
||||
.await;
|
||||
|
||||
// Discard kill/wait errors on the teardown path: the test result
|
||||
// above is the binding outcome, and a server that has already
|
||||
// exited produces an error here that carries no useful signal.
|
||||
server_process.kill_and_wait().await;
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Read-only compliance entry: CMPTST-15 through CMPTST-23. The SFTP
|
||||
/// server runs with RUSTFS_SFTP_READ_ONLY=true. Mutations through SFTP
|
||||
/// must error. Reads through SFTP must succeed. The test seeds a bucket
|
||||
/// and a file through the writable S3 endpoint before opening the SFTP
|
||||
/// session.
|
||||
pub async fn test_sftp_compliance_readonly() -> Result<()> {
|
||||
info!("Starting SFTP server in read-only mode on {}", COMPLIANCE_RO_SFTP_ADDRESS);
|
||||
let (_env, mut server_process) = spawn_compliance_rustfs(COMPLIANCE_RO_SFTP_ADDRESS, COMPLIANCE_RO_S3_ADDRESS, true).await?;
|
||||
|
||||
let result = async {
|
||||
ProtocolTestEnvironment::wait_for_port_ready(COMPLIANCE_RO_SFTP_PORT, 30)
|
||||
.await
|
||||
.map_err(|e| anyhow!("{}", e))?;
|
||||
|
||||
let s3 = build_test_s3_client(COMPLIANCE_RO_S3_ENDPOINT);
|
||||
wait_for_s3_ready(&s3, COMPLIANCE_RO_S3_READY_ATTEMPTS).await?;
|
||||
|
||||
let bucket = "robucket";
|
||||
let seeded_key = "small.txt";
|
||||
let seeded_content = b"read-only seed\n";
|
||||
|
||||
s3.create_bucket()
|
||||
.bucket(bucket)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| anyhow!("S3 CreateBucket {} failed: {:?}", bucket, e))?;
|
||||
s3.put_object()
|
||||
.bucket(bucket)
|
||||
.key(seeded_key)
|
||||
.body(ByteStream::from_static(seeded_content))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, seeded_key, e))?;
|
||||
info!("Seeded read-only fixture via S3: {}/{}", bucket, seeded_key);
|
||||
|
||||
let (session, sftp) = connect_sftp_to(COMPLIANCE_RO_SFTP_ADDRESS).await?;
|
||||
|
||||
cmptst_15::run_ro_put_rejected(&sftp, bucket).await?;
|
||||
cmptst_16::run_ro_rm_rejected(&sftp, bucket, seeded_key).await?;
|
||||
cmptst_17::run_ro_mkdir_rejected(&sftp).await?;
|
||||
cmptst_18::run_ro_rmdir_rejected(&sftp, bucket).await?;
|
||||
cmptst_19::run_ro_rename_rejected(&sftp, bucket, seeded_key).await?;
|
||||
cmptst_20::run_ro_ls_allowed(&sftp, bucket).await?;
|
||||
cmptst_21::run_ro_get_allowed(&sftp, bucket, seeded_key, seeded_content).await?;
|
||||
cmptst_22::run_ro_setstat_rejected(&sftp, bucket, seeded_key).await?;
|
||||
cmptst_23::run_ro_fsetstat_rejected(&sftp, bucket, seeded_key).await?;
|
||||
|
||||
drop(sftp);
|
||||
// Discard the disconnect Result. A read-only session that has
|
||||
// returned errors against every mutation can still be cleanly
|
||||
// torn down, but a transient transport-level error here
|
||||
// carries no useful signal beyond what the assertions above
|
||||
// already pin.
|
||||
let _ = session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
|
||||
info!("SFTP read-only compliance suite passed");
|
||||
Ok::<(), anyhow::Error>(())
|
||||
}
|
||||
.await;
|
||||
|
||||
server_process.kill_and_wait().await;
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Standalone-server compliance entry: runs CMPTST-24..33 in numerical
|
||||
/// order. Each case spawns and tears down its own rustfs because each
|
||||
/// exercises a different server configuration (idle timeout, console
|
||||
/// listener, read-cache window) that cannot share a process with the
|
||||
/// others.
|
||||
///
|
||||
/// CMPTST-30 is omitted by default. Its assertion (per-operation
|
||||
/// wall-clock latency under pipelined metadata ops) is bounded by the
|
||||
/// SSH SFTP subsystem's per-channel serial handler dispatch and is
|
||||
/// structurally infeasible against the production code. The
|
||||
/// test_sftp_handler_latency_regression #[tokio::test] entry remains
|
||||
/// runnable on demand via `--ignored`.
|
||||
///
|
||||
/// CMPTST-31 (paused-drain) is omitted from the default suite for
|
||||
/// runtime cost (200 MiB seed plus a 25 s pause window). The
|
||||
/// test_sftp_paused_drain_regression #[tokio::test] entry covers it
|
||||
/// for direct invocation.
|
||||
///
|
||||
/// CMPTST-24, 25, 26 verify kernel-level state via ss(8) and the
|
||||
/// procfs ESTABLISHED discriminator. They are skipped on non-Linux
|
||||
/// targets where those interfaces are absent.
|
||||
pub async fn test_sftp_compliance_standalone() -> Result<()> {
|
||||
info!("Starting SFTP standalone-server compliance suite");
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
cmptst_24::run_concurrent_half_close_no_leak().await?;
|
||||
cmptst_25::run_wedge_kill_after_silence_in_close_wait().await?;
|
||||
cmptst_26::run_healthy_idle_session_above_fast_threshold().await?;
|
||||
}
|
||||
|
||||
cmptst_27::run_multi_session_mixed_pipelining().await?;
|
||||
cmptst_28::run_5mb_download_with_concurrent_metadata_ops().await?;
|
||||
cmptst_29::run_read_past_eof_volume().await?;
|
||||
cmptst_32::run_read_cache_enabled_round_trip().await?;
|
||||
cmptst_33::run_read_cache_disabled_round_trip().await?;
|
||||
|
||||
info!("SFTP standalone-server compliance suite passed");
|
||||
Ok(())
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,557 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Core SFTP tests
|
||||
|
||||
use crate::common::rustfs_binary_path_with_features;
|
||||
use crate::protocols::sftp_helpers::{
|
||||
AcceptAnyServerKey, ServerProcess, build_test_s3_client, connect_sftp_to, generate_host_key, sftp_read_full,
|
||||
wait_for_s3_ready,
|
||||
};
|
||||
use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, ProtocolTestEnvironment};
|
||||
use anyhow::{Result, anyhow};
|
||||
use aws_sdk_s3::Client as S3Client;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use russh::client::{self, Handle};
|
||||
use russh_sftp::client::SftpSession;
|
||||
use russh_sftp::protocol::{FileAttributes, OpenFlags};
|
||||
use rustfs_config::{
|
||||
ENV_RUSTFS_ADDRESS, ENV_SFTP_ADDRESS, ENV_SFTP_ENABLE, ENV_SFTP_HOST_KEY_DIR, ENV_SFTP_IDLE_TIMEOUT, ENV_SFTP_PART_SIZE,
|
||||
ENV_SFTP_READ_ONLY,
|
||||
};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::process::Command;
|
||||
use tokio::time::sleep;
|
||||
use tracing::info;
|
||||
|
||||
const SFTP_PORT: u16 = 9022;
|
||||
const SFTP_ADDRESS: &str = "127.0.0.1:9022";
|
||||
|
||||
// The cross-protocol assertions reach the same server process the SFTP
|
||||
// session is connected to. The S3 endpoint is bound on a non-default port to
|
||||
// avoid contention with any other rustfs running on port 9000 (for example a
|
||||
// dev-testing harness container).
|
||||
const S3_ADDRESS: &str = "127.0.0.1:9200";
|
||||
const S3_ENDPOINT: &str = "http://127.0.0.1:9200";
|
||||
const S3_READY_ATTEMPTS: u32 = 30;
|
||||
|
||||
// Mirrors GLOBAL_DIR_SUFFIX in rustfs_utils::path. The e2e_test crate does
|
||||
// not depend on rustfs-utils, so the suffix is repeated locally.
|
||||
const XLDIR_SUFFIX: &str = "__XLDIR__";
|
||||
|
||||
// Idle-timeout test uses its own ports so it can run alongside the core suite
|
||||
// without clashing on the default S3 port or on the core SFTP port.
|
||||
const IDLE_SFTP_PORT: u16 = 9023;
|
||||
const IDLE_SFTP_ADDRESS: &str = "127.0.0.1:9023";
|
||||
const IDLE_S3_ADDRESS: &str = "127.0.0.1:9100";
|
||||
const IDLE_TIMEOUT_SECS: u64 = 5;
|
||||
const IDLE_WAIT_SECS: u64 = 10;
|
||||
|
||||
// Pin the server's multipart part_size to the spec minimum (5 MiB) so the
|
||||
// multipart payload in this file is sized relative to a known value and the
|
||||
// Buffering to Streaming transition triggers deterministically regardless of
|
||||
// the server's default.
|
||||
const PART_SIZE_BYTES: usize = 5 * 1024 * 1024;
|
||||
const PART_SIZE_ENV: &str = "5242880";
|
||||
// Just over two part_size worth so the upload issues CreateMultipartUpload,
|
||||
// at least one UploadPart mid-stream, and CompleteMultipartUpload.
|
||||
const MULTIPART_SIZE: usize = PART_SIZE_BYTES * 2 + 1024;
|
||||
|
||||
// Fixed deterministic payload for the S3-write, SFTP-read direction. 256 KiB
|
||||
// is well below part_size so the SFTP read returns the object as a single
|
||||
// GetObject response without invoking the streaming multipart path.
|
||||
const S3_WRITTEN_SIZE: usize = 256 * 1024;
|
||||
|
||||
async fn connect_sftp() -> Result<(Handle<AcceptAnyServerKey>, SftpSession)> {
|
||||
connect_sftp_to(SFTP_ADDRESS).await
|
||||
}
|
||||
|
||||
/// Confirm that an object is byte-identical when fetched via S3 and via SFTP.
|
||||
/// Hashes the expected payload once, then compares both fetched payloads
|
||||
/// against that hash. Either mismatch returns an error naming the side that
|
||||
/// disagreed.
|
||||
async fn assert_cross_protocol_sha_match(
|
||||
s3: &S3Client,
|
||||
sftp: &SftpSession,
|
||||
bucket: &str,
|
||||
key: &str,
|
||||
expected: &[u8],
|
||||
) -> Result<()> {
|
||||
let expected_sha = Sha256::digest(expected);
|
||||
|
||||
let s3_get = s3
|
||||
.get_object()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| anyhow!("S3 GetObject {}/{} failed: {:?}", bucket, key, e))?;
|
||||
let s3_bytes = s3_get
|
||||
.body
|
||||
.collect()
|
||||
.await
|
||||
.map_err(|e| anyhow!("S3 body collect failed for {}/{}: {:?}", bucket, key, e))?
|
||||
.into_bytes();
|
||||
if s3_bytes.len() != expected.len() {
|
||||
return Err(anyhow!(
|
||||
"S3 GetObject byte count mismatch for {}/{}: expected {}, got {}",
|
||||
bucket,
|
||||
key,
|
||||
expected.len(),
|
||||
s3_bytes.len()
|
||||
));
|
||||
}
|
||||
let s3_sha = Sha256::digest(&s3_bytes);
|
||||
if s3_sha != expected_sha {
|
||||
return Err(anyhow!("S3 GetObject SHA256 mismatch for {}/{}", bucket, key));
|
||||
}
|
||||
|
||||
let sftp_path = format!("/{bucket}/{key}");
|
||||
let sftp_bytes = sftp_read_full(sftp, &sftp_path).await?;
|
||||
if sftp_bytes.len() != expected.len() {
|
||||
return Err(anyhow!(
|
||||
"SFTP read byte count mismatch for {}: expected {}, got {}",
|
||||
sftp_path,
|
||||
expected.len(),
|
||||
sftp_bytes.len()
|
||||
));
|
||||
}
|
||||
let sftp_sha = Sha256::digest(&sftp_bytes);
|
||||
if sftp_sha != expected_sha {
|
||||
return Err(anyhow!("SFTP read SHA256 mismatch for {}", sftp_path));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// SFTP core protocol round-trip: banner, mkdir, put, get with SHA compare, rename, delete, rmdir.
|
||||
pub async fn test_sftp_core_operations() -> Result<()> {
|
||||
let env = ProtocolTestEnvironment::new().map_err(|e| anyhow!("{}", e))?;
|
||||
let host_key_dir = PathBuf::from(&env.temp_dir).join("sftp_host_keys");
|
||||
generate_host_key(&host_key_dir).await?;
|
||||
|
||||
info!("Starting SFTP server on {}", SFTP_ADDRESS);
|
||||
let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav,sftp"));
|
||||
let host_key_dir_str = host_key_dir
|
||||
.to_str()
|
||||
.ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
|
||||
let mut server_process = ServerProcess::new(
|
||||
Command::new(&binary_path)
|
||||
.env(ENV_SFTP_ENABLE, "true")
|
||||
.env(ENV_SFTP_ADDRESS, SFTP_ADDRESS)
|
||||
.env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)
|
||||
.env(ENV_SFTP_READ_ONLY, "false")
|
||||
.env(ENV_SFTP_PART_SIZE, PART_SIZE_ENV)
|
||||
.env(ENV_RUSTFS_ADDRESS, S3_ADDRESS)
|
||||
.arg(&env.temp_dir)
|
||||
.spawn()?,
|
||||
);
|
||||
|
||||
let result = async {
|
||||
ProtocolTestEnvironment::wait_for_port_ready(SFTP_PORT, 30)
|
||||
.await
|
||||
.map_err(|e| anyhow!("{}", e))?;
|
||||
|
||||
let (session, sftp) = connect_sftp().await?;
|
||||
|
||||
// --- 1. Subsystem canary: SFTP session reachable after password auth ---
|
||||
// SftpSession::new completes the SFTPv3 version exchange. The
|
||||
// canonicalize call below is a cheap round-trip that confirms
|
||||
// the session handles real wire traffic.
|
||||
info!("Testing SFTP: subsystem canary, server resolves '.' to an absolute path");
|
||||
let pwd = sftp.canonicalize(".").await?;
|
||||
assert!(!pwd.is_empty(), "server must resolve '.' to a non-empty absolute path");
|
||||
info!("PASS: subsystem canary: server resolved '.' to {}", pwd);
|
||||
|
||||
// --- 2. Bucket lifecycle: mkdir then root listing ---
|
||||
let bucket = "coretestbucket";
|
||||
let bucket_path = format!("/{bucket}");
|
||||
info!("Testing SFTP: mkdir bucket {}", bucket_path);
|
||||
sftp.create_dir(&bucket_path).await?;
|
||||
info!("PASS: mkdir bucket {}", bucket_path);
|
||||
|
||||
info!("Testing SFTP: root listing includes the new bucket");
|
||||
let root_entries: Vec<String> = sftp.read_dir("/").await?.map(|e| e.file_name()).collect();
|
||||
assert!(root_entries.iter().any(|n| n == bucket), "root listing should contain the new bucket");
|
||||
info!("PASS: bucket {} appeared in read_dir(\"/\")", bucket);
|
||||
|
||||
// --- 3. Small-file round-trip with SHA256 compare ---
|
||||
info!("Testing SFTP: small-file round-trip with SHA256 compare");
|
||||
let small_path = format!("/{bucket}/small.txt");
|
||||
let small_content = b"hello rustfs sftp\n";
|
||||
let mut wf = sftp
|
||||
.open_with_flags(&small_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
|
||||
.await?;
|
||||
wf.write_all(small_content).await?;
|
||||
wf.flush().await?;
|
||||
wf.shutdown().await?;
|
||||
|
||||
let mut rf = sftp.open_with_flags(&small_path, OpenFlags::READ).await?;
|
||||
let mut buf = Vec::new();
|
||||
rf.read_to_end(&mut buf).await?;
|
||||
rf.shutdown().await?;
|
||||
assert_eq!(buf.as_slice(), small_content, "small-file round-trip content mismatch");
|
||||
let sha_in = Sha256::digest(small_content);
|
||||
let sha_out = Sha256::digest(&buf);
|
||||
assert_eq!(sha_in, sha_out, "small-file SHA256 mismatch");
|
||||
info!("PASS: small-file round-trip SHA256 match");
|
||||
|
||||
// --- 4. Path STAT on a file and on a bucket ---
|
||||
info!("Testing SFTP: stat on file returns size and file type");
|
||||
let file_meta = sftp.metadata(&small_path).await?;
|
||||
assert_eq!(file_meta.size, Some(small_content.len() as u64), "stat size mismatch");
|
||||
assert!(file_meta.file_type().is_file(), "stat on a file must report regular file");
|
||||
info!("PASS: stat on file reports size {} and file type", small_content.len());
|
||||
|
||||
info!("Testing SFTP: stat on bucket reports directory");
|
||||
let bucket_meta = sftp.metadata(&bucket_path).await?;
|
||||
assert!(bucket_meta.file_type().is_dir(), "stat on a bucket must report directory");
|
||||
info!("PASS: stat on bucket reports directory");
|
||||
|
||||
// --- 5. SETSTAT on a file path returns ok ---
|
||||
// SETSTAT is a no-op on the server because S3 has no POSIX mtime or permission
|
||||
// semantics, but it must still return ok. Clients that send SETSTAT after every
|
||||
// transfer (rsync, WinSCP) treat a non-ok status as a transfer failure.
|
||||
info!("Testing SFTP: setstat on a path returns ok");
|
||||
let attrs = FileAttributes {
|
||||
permissions: Some(0o644),
|
||||
..FileAttributes::default()
|
||||
};
|
||||
sftp.set_metadata(&small_path, attrs).await?;
|
||||
info!("PASS: setstat returned ok");
|
||||
|
||||
// --- 6. Rename within bucket and listing reflects it ---
|
||||
info!("Testing SFTP: rename within bucket");
|
||||
let renamed = format!("/{bucket}/renamed.txt");
|
||||
sftp.rename(&small_path, &renamed).await?;
|
||||
info!("PASS: rename {} -> {}", small_path, renamed);
|
||||
|
||||
info!("Testing SFTP: listing reflects rename");
|
||||
let bucket_entries: Vec<String> = sftp.read_dir(&bucket_path).await?.map(|e| e.file_name()).collect();
|
||||
assert!(bucket_entries.iter().any(|n| n == "renamed.txt"), "renamed file must be listed");
|
||||
assert!(!bucket_entries.iter().any(|n| n == "small.txt"), "pre-rename name must be gone");
|
||||
info!("PASS: directory listing reflects the rename");
|
||||
|
||||
// --- 7. Multipart round-trip across the part-size boundary ---
|
||||
// MULTIPART_SIZE is paired with RUSTFS_SFTP_PART_SIZE above so the upload crosses the
|
||||
// multipart threshold regardless of server defaults.
|
||||
info!("Testing SFTP: multipart-sized round-trip with SHA256 compare");
|
||||
let big_path = format!("/{bucket}/big.bin");
|
||||
let big_content: Vec<u8> = (0..MULTIPART_SIZE).map(|i| (i as u8).wrapping_mul(31)).collect();
|
||||
let mut bwf = sftp
|
||||
.open_with_flags(&big_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
|
||||
.await?;
|
||||
bwf.write_all(&big_content).await?;
|
||||
bwf.flush().await?;
|
||||
bwf.shutdown().await?;
|
||||
|
||||
let mut brf = sftp.open_with_flags(&big_path, OpenFlags::READ).await?;
|
||||
let mut big_buf = Vec::with_capacity(MULTIPART_SIZE);
|
||||
brf.read_to_end(&mut big_buf).await?;
|
||||
brf.shutdown().await?;
|
||||
assert_eq!(big_buf.len(), MULTIPART_SIZE, "multipart round-trip length mismatch");
|
||||
let big_in = Sha256::digest(&big_content);
|
||||
let big_out = Sha256::digest(&big_buf);
|
||||
assert_eq!(big_in, big_out, "multipart SHA256 mismatch");
|
||||
info!("PASS: multipart round-trip SHA256 match ({} bytes)", MULTIPART_SIZE);
|
||||
|
||||
// --- 8. Negative cases: symlink, open nonexistent, read_dir nonexistent, path escape ---
|
||||
info!("Testing SFTP: symlink returns an error");
|
||||
let symlink_err = sftp.symlink(&big_path, &format!("/{bucket}/shortcut")).await;
|
||||
assert!(symlink_err.is_err(), "symlink must be rejected by the server");
|
||||
info!("PASS: symlink rejected");
|
||||
|
||||
info!("Testing SFTP: open of nonexistent file returns an error");
|
||||
let missing_path = format!("/{bucket}/not_here.txt");
|
||||
let missing_err = sftp.open_with_flags(&missing_path, OpenFlags::READ).await;
|
||||
assert!(missing_err.is_err(), "open of a nonexistent path must error");
|
||||
info!("PASS: open of nonexistent file rejected");
|
||||
|
||||
info!("Testing SFTP: read_dir of nonexistent bucket returns an error");
|
||||
let missing_bucket = sftp.read_dir("/nosuchbucket").await;
|
||||
assert!(missing_bucket.is_err(), "read_dir of a nonexistent bucket must error");
|
||||
info!("PASS: read_dir of nonexistent bucket rejected");
|
||||
|
||||
info!("Testing SFTP: path traversal cannot escape the storage root");
|
||||
let traversal = sftp.read_dir("/../../../etc").await;
|
||||
assert!(traversal.is_err(), "path traversal must be rejected or resolve to a nonexistent bucket");
|
||||
info!("PASS: path traversal rejected");
|
||||
|
||||
// --- Spec-letter assertion: APPEND open-flag returns an error ---
|
||||
// The driver maps APPEND to OpUnsupported because S3 has no append
|
||||
// primitive. Open requests with APPEND must return a failure rather
|
||||
// than allow a silently mistruncated upload.
|
||||
info!("Testing SFTP: open with APPEND returns an error");
|
||||
let append_err = sftp.open_with_flags(&renamed, OpenFlags::APPEND | OpenFlags::WRITE).await;
|
||||
assert!(append_err.is_err(), "open with APPEND must error");
|
||||
info!("PASS: open with APPEND rejected");
|
||||
|
||||
// --- Spec-letter assertion: O_EXCL on existing path returns an error ---
|
||||
// CREATE | EXCLUDE on a key that already exists must fail. The
|
||||
// existing renamed.txt is the target. EXCLUDE without WRITE is
|
||||
// rejected by the russh-sftp client itself, so WRITE is included.
|
||||
// TRUNCATE is included because the driver requires WRITE | CREATE
|
||||
// | TRUNCATE on every accepted write OPEN.
|
||||
info!("Testing SFTP: open with CREATE + EXCLUDE on existing path returns an error");
|
||||
let excl_err = sftp
|
||||
.open_with_flags(&renamed, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::EXCLUDE | OpenFlags::WRITE)
|
||||
.await;
|
||||
assert!(excl_err.is_err(), "CREATE+EXCLUDE on existing path must error");
|
||||
info!("PASS: CREATE+EXCLUDE on existing path rejected");
|
||||
|
||||
// --- WRITE without CREATE or TRUNCATE is rejected at OPEN ---
|
||||
// The streaming write path overwrites the entire object at
|
||||
// close. A WRITE-only OPEN asks for partial-write semantics
|
||||
// the server cannot honour against S3, so the OPEN is
|
||||
// rejected before any handle is allocated.
|
||||
info!("Testing SFTP: open with WRITE only returns an error");
|
||||
let write_only_err = sftp.open_with_flags(&renamed, OpenFlags::WRITE).await;
|
||||
assert!(write_only_err.is_err(), "WRITE without CREATE or TRUNCATE must be rejected at OPEN");
|
||||
info!("PASS: WRITE only rejected");
|
||||
|
||||
// --- WRITE | CREATE without TRUNCATE is rejected at OPEN ---
|
||||
// Without TRUNCATE the client is asking for create-or-modify-
|
||||
// existing semantics. The server cannot deliver that against
|
||||
// S3, so the OPEN is rejected before any handle is allocated.
|
||||
info!("Testing SFTP: open with WRITE | CREATE without TRUNCATE returns an error");
|
||||
let create_no_trunc_err = sftp.open_with_flags(&renamed, OpenFlags::WRITE | OpenFlags::CREATE).await;
|
||||
assert!(create_no_trunc_err.is_err(), "WRITE | CREATE without TRUNCATE must be rejected at OPEN");
|
||||
info!("PASS: WRITE | CREATE without TRUNCATE rejected");
|
||||
|
||||
// --- Spec-letter assertion: bad password is rejected (separate session) ---
|
||||
// Fresh russh session with wrong credentials. The authenticated
|
||||
// handle is left untouched. Bad auth must not succeed.
|
||||
info!("Testing SFTP: second russh session with wrong password is rejected");
|
||||
let bad_config = Arc::new(client::Config::default());
|
||||
let mut bad_session = client::connect(bad_config, SFTP_ADDRESS, AcceptAnyServerKey).await?;
|
||||
let bad_auth = bad_session.authenticate_password(DEFAULT_ACCESS_KEY, "wrong-secret").await?;
|
||||
assert!(!bad_auth.success(), "bad-password authentication must not succeed");
|
||||
// Discard the disconnect Result. A server that already rejected
|
||||
// auth can return an error here, but the assert above already
|
||||
// pins the auth outcome.
|
||||
let _ = bad_session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
|
||||
info!("PASS: bad-password authentication rejected");
|
||||
|
||||
// --- Cross-protocol setup: aws-sdk-s3 client against the same server ---
|
||||
// The rustfs binary spawned for this suite serves both SFTP on port
|
||||
// 9022 and S3 on port 9000. The S3 stack may need a moment to finish
|
||||
// initialising after TCP is listening, so list_buckets is polled
|
||||
// until it succeeds before any cross-protocol assertion runs.
|
||||
info!("Testing SFTP: prepare aws-sdk-s3 client and wait for S3 readiness");
|
||||
let s3 = build_test_s3_client(S3_ENDPOINT);
|
||||
wait_for_s3_ready(&s3, S3_READY_ATTEMPTS).await?;
|
||||
info!("PASS: S3 endpoint reachable from cross-protocol client");
|
||||
|
||||
// --- SFTP write, S3 read: SHA256 round-trip ---
|
||||
// SFTP creates the object, then assert_cross_protocol_sha_match
|
||||
// fetches it via both S3 GetObject and SFTP READ and compares
|
||||
// each result against the SHA256 of the original payload. Both
|
||||
// sides must match byte-exact, which proves the storage layer
|
||||
// returns the same bytes regardless of wire protocol.
|
||||
info!("Testing SFTP: SFTP write then S3 read, SHA256 round-trip");
|
||||
let sftp_to_s3_key = "sftp_written.bin";
|
||||
let sftp_to_s3_path = format!("/{bucket}/{sftp_to_s3_key}");
|
||||
let sftp_to_s3_content: Vec<u8> = (0..S3_WRITTEN_SIZE).map(|i| (i as u8).wrapping_mul(17)).collect();
|
||||
let mut wf = sftp
|
||||
.open_with_flags(&sftp_to_s3_path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE)
|
||||
.await?;
|
||||
wf.write_all(&sftp_to_s3_content).await?;
|
||||
wf.flush().await?;
|
||||
wf.shutdown().await?;
|
||||
assert_cross_protocol_sha_match(&s3, &sftp, bucket, sftp_to_s3_key, &sftp_to_s3_content).await?;
|
||||
info!("PASS: SFTP-written object matches via S3 GetObject and SFTP READ");
|
||||
|
||||
// --- S3 write, SFTP read: SHA256 round-trip ---
|
||||
// aws-sdk-s3 PutObject writes a fixed deterministic payload. Both
|
||||
// sides then read it back and SHA-compare.
|
||||
info!("Testing SFTP: S3 write then SFTP read, SHA256 round-trip");
|
||||
let s3_to_sftp_key = "s3_written.bin";
|
||||
let s3_to_sftp_content: Vec<u8> = (0..S3_WRITTEN_SIZE).map(|i| (i as u8).wrapping_mul(31)).collect();
|
||||
s3.put_object()
|
||||
.bucket(bucket)
|
||||
.key(s3_to_sftp_key)
|
||||
.body(ByteStream::from(s3_to_sftp_content.clone()))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_to_sftp_key, e))?;
|
||||
assert_cross_protocol_sha_match(&s3, &sftp, bucket, s3_to_sftp_key, &s3_to_sftp_content).await?;
|
||||
info!("PASS: S3-written object matches via S3 GetObject and SFTP READ");
|
||||
|
||||
// --- Cross-API directory visibility: SFTP mkdir, S3 ListObjectsV2 ---
|
||||
// SFTP mkdir writes a __XLDIR__ marker. The rustfs S3 listing path
|
||||
// decodes that marker back to a trailing-slash key, so the asserted
|
||||
// pattern is "subdir_sftp/".
|
||||
info!("Testing SFTP: SFTP-created sub-directory visible via S3 ListObjectsV2");
|
||||
let sftp_subdir_name = "subdir_sftp";
|
||||
let sftp_subdir_path = format!("/{bucket}/{sftp_subdir_name}");
|
||||
sftp.create_dir(&sftp_subdir_path).await?;
|
||||
let listed = s3
|
||||
.list_objects_v2()
|
||||
.bucket(bucket)
|
||||
.prefix(sftp_subdir_name)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| anyhow!("S3 ListObjectsV2 {} failed: {:?}", bucket, e))?;
|
||||
let listed_keys: Vec<String> = listed
|
||||
.contents()
|
||||
.iter()
|
||||
.filter_map(|obj| obj.key().map(|s| s.to_string()))
|
||||
.collect();
|
||||
let visible_via_s3 = listed_keys
|
||||
.iter()
|
||||
.any(|k| k == &format!("{sftp_subdir_name}/") || k == &format!("{sftp_subdir_name}{XLDIR_SUFFIX}"));
|
||||
assert!(
|
||||
visible_via_s3,
|
||||
"SFTP-created sub-directory must appear in S3 ListObjectsV2: keys returned were {listed_keys:?}"
|
||||
);
|
||||
info!("PASS: SFTP mkdir visible to S3 ListObjectsV2");
|
||||
|
||||
// --- Cross-API directory visibility: S3 marker, SFTP readdir ---
|
||||
// aws-sdk-s3 PutObject writes a zero-byte marker keyed with the
|
||||
// __XLDIR__ suffix. SFTP readdir must decode the marker back to a
|
||||
// bare directory entry whose file_type reports as a directory.
|
||||
info!("Testing SFTP: S3-created __XLDIR__ marker visible via SFTP readdir");
|
||||
let s3_subdir_name = "subdir_s3";
|
||||
let s3_subdir_marker_key = format!("{s3_subdir_name}{XLDIR_SUFFIX}");
|
||||
s3.put_object()
|
||||
.bucket(bucket)
|
||||
.key(&s3_subdir_marker_key)
|
||||
.body(ByteStream::from_static(b""))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| anyhow!("S3 PutObject {}/{} failed: {:?}", bucket, s3_subdir_marker_key, e))?;
|
||||
let bucket_entries: Vec<(String, bool)> = sftp
|
||||
.read_dir(&bucket_path)
|
||||
.await?
|
||||
.map(|entry| (entry.file_name(), entry.file_type().is_dir()))
|
||||
.collect();
|
||||
let visible_via_sftp = bucket_entries.iter().any(|(name, is_dir)| name == s3_subdir_name && *is_dir);
|
||||
assert!(
|
||||
visible_via_sftp,
|
||||
"S3-created marker must appear as a directory in SFTP readdir: entries were {bucket_entries:?}"
|
||||
);
|
||||
info!("PASS: S3 marker visible to SFTP readdir as a directory");
|
||||
|
||||
// --- Pre-cleanup of cross-protocol fixtures ---
|
||||
// Removes the new files and sub-directories so the existing rmdir
|
||||
// call below operates against an empty bucket.
|
||||
info!("Testing SFTP: pre-cleanup of cross-protocol fixtures");
|
||||
sftp.remove_file(&format!("/{bucket}/{sftp_to_s3_key}")).await?;
|
||||
sftp.remove_file(&format!("/{bucket}/{s3_to_sftp_key}")).await?;
|
||||
sftp.remove_dir(&sftp_subdir_path).await?;
|
||||
sftp.remove_dir(&format!("/{bucket}/{s3_subdir_name}")).await?;
|
||||
info!("PASS: cross-protocol fixtures removed");
|
||||
|
||||
// --- 9. Cleanup: delete objects, rmdir bucket, confirm root empty ---
|
||||
info!("Testing SFTP: delete objects then rmdir bucket");
|
||||
sftp.remove_file(&renamed).await?;
|
||||
sftp.remove_file(&big_path).await?;
|
||||
sftp.remove_dir(&bucket_path).await?;
|
||||
info!("PASS: delete + rmdir leaves the root empty");
|
||||
|
||||
let final_entries: Vec<String> = sftp.read_dir("/").await?.map(|e| e.file_name()).collect();
|
||||
assert!(!final_entries.iter().any(|n| n == bucket), "bucket must be gone after rmdir");
|
||||
info!("PASS: root listing no longer includes the deleted bucket");
|
||||
|
||||
drop(sftp);
|
||||
session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
|
||||
info!("SFTP core tests passed");
|
||||
Ok::<(), anyhow::Error>(())
|
||||
}
|
||||
.await;
|
||||
|
||||
// Discard kill/wait errors on the teardown path: the test result
|
||||
// above is the binding outcome, and a server that has already
|
||||
// exited produces an error here that carries no useful signal.
|
||||
server_process.kill_and_wait().await;
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Idle-timeout regression: the server must close an SFTP session that
|
||||
/// remains inactive past RUSTFS_SFTP_IDLE_TIMEOUT.
|
||||
///
|
||||
/// Spawns its own rustfs binary on dedicated SFTP and S3 ports so it can run
|
||||
/// independently of the core protocol suite. The disconnect check issues a
|
||||
/// cheap SFTP request after the wait window. The same error path runs in any
|
||||
/// client when the server-initiated SSH_MSG_DISCONNECT arrives. The assertion
|
||||
/// does not pin a specific russh error variant because the exact error
|
||||
/// returned on server-initiated disconnect depends on timing.
|
||||
pub async fn test_sftp_idle_timeout_disconnects() -> Result<()> {
|
||||
let env = ProtocolTestEnvironment::new().map_err(|e| anyhow!("{}", e))?;
|
||||
let host_key_dir = PathBuf::from(&env.temp_dir).join("sftp_host_keys");
|
||||
generate_host_key(&host_key_dir).await?;
|
||||
|
||||
info!("Starting SFTP server with idle timeout {} s on {}", IDLE_TIMEOUT_SECS, IDLE_SFTP_ADDRESS);
|
||||
let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav,sftp"));
|
||||
let host_key_dir_str = host_key_dir
|
||||
.to_str()
|
||||
.ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
|
||||
let mut server_process = ServerProcess::new(
|
||||
Command::new(&binary_path)
|
||||
.env(ENV_SFTP_ENABLE, "true")
|
||||
.env(ENV_SFTP_ADDRESS, IDLE_SFTP_ADDRESS)
|
||||
.env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)
|
||||
.env(ENV_SFTP_READ_ONLY, "false")
|
||||
.env(ENV_SFTP_PART_SIZE, PART_SIZE_ENV)
|
||||
.env(ENV_SFTP_IDLE_TIMEOUT, IDLE_TIMEOUT_SECS.to_string())
|
||||
.env(ENV_RUSTFS_ADDRESS, IDLE_S3_ADDRESS)
|
||||
.arg(&env.temp_dir)
|
||||
.spawn()?,
|
||||
);
|
||||
|
||||
let result = async {
|
||||
ProtocolTestEnvironment::wait_for_port_ready(IDLE_SFTP_PORT, 30)
|
||||
.await
|
||||
.map_err(|e| anyhow!("{}", e))?;
|
||||
|
||||
let (session, sftp) = connect_sftp_to(IDLE_SFTP_ADDRESS).await?;
|
||||
|
||||
// Confirm the session is live before the wait so a failure in the
|
||||
// post-wait read can be attributed to the idle timer rather than to
|
||||
// a setup defect.
|
||||
let pwd = sftp.canonicalize(".").await?;
|
||||
assert!(!pwd.is_empty(), "server must resolve '.' to a non-empty absolute path");
|
||||
|
||||
info!("Idle wait: sleeping {} s past idle timeout {} s", IDLE_WAIT_SECS, IDLE_TIMEOUT_SECS);
|
||||
sleep(Duration::from_secs(IDLE_WAIT_SECS)).await;
|
||||
|
||||
let post_idle = sftp.read_dir("/").await;
|
||||
assert!(
|
||||
post_idle.is_err(),
|
||||
"SFTP request after idle wait must error once the server has closed the session"
|
||||
);
|
||||
info!("PASS: SFTP request after idle wait returned an error");
|
||||
|
||||
drop(sftp);
|
||||
// Discard the disconnect Result. The server has already closed the
|
||||
// session via the idle-timeout path the test is probing. A client
|
||||
// disconnect against a half-closed transport may itself return Err
|
||||
// with no useful signal.
|
||||
let _ = session.disconnect(russh::Disconnect::ByApplication, "", "en").await;
|
||||
Ok::<(), anyhow::Error>(())
|
||||
}
|
||||
.await;
|
||||
|
||||
// Discard kill/wait errors on the teardown path: the test result above
|
||||
// is the binding outcome, and a server that has already exited produces
|
||||
// an error here that carries no useful signal.
|
||||
server_process.kill_and_wait().await;
|
||||
|
||||
result
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Shared helpers for SFTP protocol tests
|
||||
//!
|
||||
//! An accept-any host-key handler, an ed25519 host-key generator that
|
||||
//! matches the rustfs config loader permission gates, a russh client
|
||||
//! connector that authenticates with the default access key, and an
|
||||
//! SFTP-read-to-vec helper.
|
||||
|
||||
use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY};
|
||||
use anyhow::{Result, anyhow};
|
||||
use aws_sdk_s3::Client as S3Client;
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
|
||||
use russh::client::{self, Handle};
|
||||
use russh::keys::ssh_key::LineEnding;
|
||||
use russh::keys::{Algorithm, PrivateKey, PublicKey};
|
||||
use russh_sftp::client::SftpSession;
|
||||
use russh_sftp::protocol::OpenFlags;
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::process::Child;
|
||||
use tokio::time::sleep;
|
||||
use tracing::info;
|
||||
|
||||
/// Accept-any server-key client handler. The test server uses a host key
|
||||
/// generated fresh at the start of each run, so strict verification would
|
||||
/// always fail. The suite exercises the auth path, not the host-key-trust
|
||||
/// path, which is out of scope for this suite.
|
||||
pub struct AcceptAnyServerKey;
|
||||
|
||||
impl client::Handler for AcceptAnyServerKey {
|
||||
type Error = anyhow::Error;
|
||||
|
||||
async fn check_server_key(&mut self, _server_public_key: &PublicKey) -> Result<bool, Self::Error> {
|
||||
Ok(true)
|
||||
}
|
||||
}
|
||||
|
||||
/// Generate an ed25519 host key pair in host_key_dir with mode 0600. The
|
||||
/// key is generated in-process via russh::keys so the test suite has no
|
||||
/// host-tooling dependency on ssh-keygen, which is absent on Alpine,
|
||||
/// distroless, scratch, and Windows images. The RustFS config loader
|
||||
/// accepts the OpenSSH private-key format that PrivateKey::to_openssh
|
||||
/// emits. Both the private key and the .pub file need 0600 because the
|
||||
/// loader scans every entry in the directory and rejects the whole
|
||||
/// directory as insecure unless each file is 0600 or 0400.
|
||||
pub async fn generate_host_key(host_key_dir: &Path) -> Result<()> {
|
||||
tokio::fs::create_dir_all(host_key_dir).await?;
|
||||
let key_path = host_key_dir.join("ssh_host_ed25519_key");
|
||||
let pub_path = host_key_dir.join("ssh_host_ed25519_key.pub");
|
||||
|
||||
let private_key =
|
||||
PrivateKey::random(&mut rand::rng(), Algorithm::Ed25519).map_err(|e| anyhow!("ed25519 key generation failed: {e}"))?;
|
||||
let private_pem = private_key
|
||||
.to_openssh(LineEnding::LF)
|
||||
.map_err(|e| anyhow!("OpenSSH private-key encode failed: {e}"))?;
|
||||
let public_text = private_key
|
||||
.public_key()
|
||||
.to_openssh()
|
||||
.map_err(|e| anyhow!("OpenSSH public-key encode failed: {e}"))?;
|
||||
|
||||
tokio::fs::write(&key_path, private_pem.as_bytes()).await?;
|
||||
tokio::fs::write(&pub_path, format!("{public_text}\n")).await?;
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
for path in [&key_path, &pub_path] {
|
||||
let mut perm = std::fs::metadata(path)?.permissions();
|
||||
perm.set_mode(0o600);
|
||||
std::fs::set_permissions(path, perm)?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Owns a spawned rustfs server child process and guarantees the process
|
||||
/// is sent SIGKILL even if the test panics. The wrapper exists because
|
||||
/// tokio::process::Child does not kill on Drop on stable Rust, so a
|
||||
/// panicking test would otherwise leak a running rustfs binary that
|
||||
/// keeps its listener port held until the test runner exits.
|
||||
///
|
||||
/// Use kill_and_wait on the success and Err paths to reap the child
|
||||
/// cleanly; Drop only fires the synchronous SIGKILL when those paths
|
||||
/// were skipped (panic unwind, runtime abort).
|
||||
pub struct ServerProcess {
|
||||
inner: Option<Child>,
|
||||
}
|
||||
|
||||
impl ServerProcess {
|
||||
pub fn new(child: Child) -> Self {
|
||||
Self { inner: Some(child) }
|
||||
}
|
||||
|
||||
/// Borrow the inner Child for callers that need stdout piping or
|
||||
/// other tokio::process APIs.
|
||||
pub fn child_mut(&mut self) -> &mut Child {
|
||||
self.inner.as_mut().expect("ServerProcess: child already taken")
|
||||
}
|
||||
|
||||
/// Async kill plus wait. Idempotent. Use on every success and Err
|
||||
/// path. After this returns, Drop becomes a no-op.
|
||||
pub async fn kill_and_wait(&mut self) {
|
||||
if let Some(mut child) = self.inner.take() {
|
||||
let _ = child.kill().await;
|
||||
let _ = child.wait().await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ServerProcess {
|
||||
fn drop(&mut self) {
|
||||
if let Some(child) = self.inner.as_mut() {
|
||||
// Synchronous SIGKILL via the kernel. Runs even on panic
|
||||
// unwind. wait() is skipped here (Drop cannot await), so
|
||||
// the process becomes a zombie reaped by the runtime.
|
||||
let _ = child.start_kill();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Open a russh client session against the given address, authenticate
|
||||
/// with the default access key, request the SFTP subsystem, and return
|
||||
/// the session handle plus the SFTP wrapper. The handle is returned so
|
||||
/// the caller can keep the underlying SSH transport alive for the full
|
||||
/// session and disconnect cleanly afterwards.
|
||||
pub async fn connect_sftp_to(address: &str) -> Result<(Handle<AcceptAnyServerKey>, SftpSession)> {
|
||||
let config = Arc::new(client::Config::default());
|
||||
let mut session = client::connect(config, address, AcceptAnyServerKey).await?;
|
||||
let auth = session.authenticate_password(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY).await?;
|
||||
if !auth.success() {
|
||||
return Err(anyhow!("SFTP password auth rejected"));
|
||||
}
|
||||
let channel = session.channel_open_session().await?;
|
||||
channel.request_subsystem(true, "sftp").await?;
|
||||
let sftp = SftpSession::new(channel.into_stream()).await?;
|
||||
Ok((session, sftp))
|
||||
}
|
||||
|
||||
/// Read an SFTP object into memory.
|
||||
pub async fn sftp_read_full(sftp: &SftpSession, path: &str) -> Result<Vec<u8>> {
|
||||
let mut file = sftp.open_with_flags(path, OpenFlags::READ).await?;
|
||||
let mut buf = Vec::new();
|
||||
file.read_to_end(&mut buf).await?;
|
||||
file.shutdown().await?;
|
||||
Ok(buf)
|
||||
}
|
||||
|
||||
/// Construct an aws-sdk-s3 client wired against an http rustfs endpoint.
|
||||
/// Uses the same credential constants the SFTP session authenticates with so
|
||||
/// both protocols see the same backend identity.
|
||||
pub fn build_test_s3_client(endpoint_url: &str) -> S3Client {
|
||||
let credentials = Credentials::new(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, None, None, "sftp-helpers");
|
||||
let mut config = aws_sdk_s3::Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(endpoint_url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest();
|
||||
if endpoint_url.starts_with("http://") {
|
||||
config = config.http_client(SmithyHttpClientBuilder::new().build_http());
|
||||
}
|
||||
S3Client::from_conf(config.build())
|
||||
}
|
||||
|
||||
/// Poll the S3 endpoint until ListBuckets returns successfully or the
|
||||
/// attempt budget is exhausted. The TCP-level wait_for_port_ready check is
|
||||
/// not enough on its own because rustfs accepts connections before the S3
|
||||
/// stack has finished initialising.
|
||||
pub async fn wait_for_s3_ready(client: &S3Client, max_attempts: u32) -> Result<()> {
|
||||
for attempt in 0..max_attempts {
|
||||
if client.list_buckets().send().await.is_ok() {
|
||||
info!("S3 endpoint ready after {} attempts", attempt + 1);
|
||||
return Ok(());
|
||||
}
|
||||
sleep(Duration::from_secs(1)).await;
|
||||
}
|
||||
Err(anyhow!("S3 endpoint did not become ready"))
|
||||
}
|
||||
@@ -32,8 +32,13 @@ impl ProtocolTestEnvironment {
|
||||
/// Create a new test environment
|
||||
/// This environment won't stop any server when dropped
|
||||
pub fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let temp_dir = format!("/tmp/rustfs_protocol_test_{}", uuid::Uuid::new_v4());
|
||||
std::fs::create_dir_all(&temp_dir)?;
|
||||
let mut path = std::env::temp_dir();
|
||||
path.push(format!("rustfs_protocol_test_{}", uuid::Uuid::new_v4()));
|
||||
std::fs::create_dir_all(&path)?;
|
||||
let temp_dir = path
|
||||
.to_str()
|
||||
.ok_or_else(|| format!("temp dir path is not utf-8: {}", path.display()))?
|
||||
.to_string();
|
||||
|
||||
Ok(Self { temp_dir })
|
||||
}
|
||||
|
||||
@@ -16,6 +16,10 @@
|
||||
|
||||
use crate::common::init_logging;
|
||||
use crate::protocols::ftps_core::test_ftps_core_operations;
|
||||
use crate::protocols::sftp_compliance::{
|
||||
test_sftp_compliance_readonly, test_sftp_compliance_standalone, test_sftp_compliance_suite,
|
||||
};
|
||||
use crate::protocols::sftp_core::{test_sftp_core_operations, test_sftp_idle_timeout_disconnects};
|
||||
use crate::protocols::webdav_core::test_webdav_core_operations;
|
||||
use serial_test::serial;
|
||||
use std::time::Instant;
|
||||
@@ -68,6 +72,21 @@ impl ProtocolTestSuite {
|
||||
TestDefinition {
|
||||
name: "test_webdav_core_operations".to_string(),
|
||||
},
|
||||
TestDefinition {
|
||||
name: "test_sftp_core_operations".to_string(),
|
||||
},
|
||||
TestDefinition {
|
||||
name: "test_sftp_compliance_suite".to_string(),
|
||||
},
|
||||
TestDefinition {
|
||||
name: "test_sftp_compliance_readonly".to_string(),
|
||||
},
|
||||
TestDefinition {
|
||||
name: "test_sftp_idle_timeout_disconnects".to_string(),
|
||||
},
|
||||
TestDefinition {
|
||||
name: "test_sftp_compliance_standalone".to_string(),
|
||||
},
|
||||
];
|
||||
|
||||
Self { tests }
|
||||
@@ -94,6 +113,26 @@ impl ProtocolTestSuite {
|
||||
info!("=== Starting WebDAV Core Test ===");
|
||||
"WebDAV core operations (MKCOL, PUT, GET, DELETE, PROPFIND)"
|
||||
}
|
||||
"test_sftp_core_operations" => {
|
||||
info!("=== Starting SFTP Core Test ===");
|
||||
"SFTP core operations (banner, mkdir, put, get with SHA compare, rename, delete, rmdir)"
|
||||
}
|
||||
"test_sftp_compliance_suite" => {
|
||||
info!("=== Starting SFTP Compliance Suite ===");
|
||||
"SFTP compliance regression suite (zero-byte, mutation rejection, traversal, rename, implicit dirs, FSETSTAT)"
|
||||
}
|
||||
"test_sftp_compliance_readonly" => {
|
||||
info!("=== Starting SFTP Read-Only Compliance Suite ===");
|
||||
"SFTP read-only mode (RUSTFS_SFTP_READ_ONLY=true rejects mutations and allows reads)"
|
||||
}
|
||||
"test_sftp_idle_timeout_disconnects" => {
|
||||
info!("=== Starting SFTP Idle-Timeout Test ===");
|
||||
"SFTP idle-timeout disconnects (server closes the session past RUSTFS_SFTP_IDLE_TIMEOUT)"
|
||||
}
|
||||
"test_sftp_compliance_standalone" => {
|
||||
info!("=== Starting SFTP Standalone-Server Compliance Suite ===");
|
||||
"SFTP standalone-server compliance suite"
|
||||
}
|
||||
_ => "",
|
||||
};
|
||||
|
||||
@@ -133,6 +172,11 @@ impl ProtocolTestSuite {
|
||||
match test_def.name.as_str() {
|
||||
"test_ftps_core_operations" => test_ftps_core_operations().await.map_err(|e| e.into()),
|
||||
"test_webdav_core_operations" => test_webdav_core_operations().await.map_err(|e| e.into()),
|
||||
"test_sftp_core_operations" => test_sftp_core_operations().await.map_err(|e| e.into()),
|
||||
"test_sftp_compliance_suite" => test_sftp_compliance_suite().await.map_err(|e| e.into()),
|
||||
"test_sftp_compliance_readonly" => test_sftp_compliance_readonly().await.map_err(|e| e.into()),
|
||||
"test_sftp_idle_timeout_disconnects" => test_sftp_idle_timeout_disconnects().await.map_err(|e| e.into()),
|
||||
"test_sftp_compliance_standalone" => test_sftp_compliance_standalone().await.map_err(|e| e.into()),
|
||||
_ => Err(format!("Test {} not implemented", test_def.name).into()),
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user