# S3 Compatibility Tests

This directory contains scripts for running S3 compatibility tests against RustFS.

## Quick Start

Run the local S3 compatibility test script:

```bash
./scripts/s3-tests/run.sh
```

The script will automatically:

1. **Check prerequisites**: Verify port availability (unless in existing mode)
2. **Build/Start RustFS**:
   - Build mode (default): Compile with `cargo build --release` (skips if binary is recent < 30 min)
   - Binary mode: Use specified or default binary path
   - Docker mode: Build Docker image and start container
   - Existing mode: Skip startup, connect to running service
3. **Wait for readiness**: Multi-step health check (process/container → port → log → S3 API)
4. **Prepare environment**:
   - Generate s3tests configuration from template
   - Provision alt user via admin API
   - Fetch s3-tests at the pinned revision (`S3TESTS_REV`)
   - Install missing dependencies (awscurl, tox, gettext)
5. **Run tests**: Execute ceph s3-tests via tox with configured filters
6. **Collect results**: Save logs and test results in `artifacts/s3tests-${TEST_MODE}/`

## Deployment Modes

The script supports four deployment modes, controlled via the `DEPLOY_MODE` environment variable:

### 1. Build Mode (Default)

Compile with `cargo build --release` and run:

```bash
DEPLOY_MODE=build ./scripts/s3-tests/run.sh
# Or simply (build is the default)
./scripts/s3-tests/run.sh

# Force rebuild even if binary exists and is recent
./scripts/s3-tests/run.sh --no-cache
```

**Behavior**:
- Automatically compiles RustFS binary if it doesn't exist
- If binary exists and was compiled less than **30 minutes** ago, compilation is skipped (unless `--no-cache` is specified)
- Automatically starts the service after compilation
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Automatic Cleanup**: Process is automatically stopped when script exits

### 2. Binary File Mode

Use pre-compiled binary file:

```bash
# Use default path (./target/release/rustfs)
DEPLOY_MODE=binary ./scripts/s3-tests/run.sh

# Specify custom binary path
DEPLOY_MODE=binary RUSTFS_BINARY=./target/release/rustfs ./scripts/s3-tests/run.sh
```

**Behavior**:
- Uses existing binary file (must exist, script will not compile)
- Automatically starts the service using the specified binary
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Automatic Cleanup**: Process is automatically stopped when script exits

### 3. Docker Mode

Build Docker image and run in container:

```bash
DEPLOY_MODE=docker ./scripts/s3-tests/run.sh
```

**Behavior**:
- Automatically builds Docker image using `Dockerfile.source`
- Creates Docker network (`rustfs-net`) if it doesn't exist
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Automatic Cleanup**: Container and network are automatically removed when script exits

### 4. Existing Service Mode

Connect to an already running RustFS service:

```bash
DEPLOY_MODE=existing S3_HOST=127.0.0.1 S3_PORT=9000 ./scripts/s3-tests/run.sh

# Connect to remote service
DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.sh
```

**Behavior**:
- Skips service startup and port availability checks
- Connects directly to the specified service endpoint
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Note**: The service must already have the alt user (`rustfsalt`) provisioned, or the script will provision it automatically

**Automatic Cleanup**: The script uses trap handlers to automatically clean up resources when it exits (success or failure):
- Stops RustFS process (build/binary mode)
- Stops and removes Docker container (docker mode)
- Removes Docker network (docker mode)

## Configuration Options

### Command Line Options

- `-h, --help`: Show help message
- `--no-cache`: Force rebuild even if binary exists and is recent (for build mode)

### Deployment Configuration

- `DEPLOY_MODE`: Deployment mode, options:
  - `build`: Compile with `cargo build --release` and run (default)
  - `binary`: Use pre-compiled binary file
  - `docker`: Build Docker image and run in container
  - `existing`: Use already running service
- `RUSTFS_BINARY`: Path to binary file (for binary mode, default: `./target/release/rustfs`)
- `DATA_ROOT`: Root directory for test data storage (default: `target`)
  - Final path: `${DATA_ROOT}/test-data/${CONTAINER_NAME}`
  - Example: `DATA_ROOT=/tmp` stores data in `/tmp/test-data/rustfs-single/`

### Service Configuration

- `S3_ACCESS_KEY`: Main user access key (default: `rustfsadmin`)
- `S3_SECRET_KEY`: Main user secret key (default: `rustfsadmin`)
- `S3_ALT_ACCESS_KEY`: Alt user access key (default: `rustfsalt`)
- `S3_ALT_SECRET_KEY`: Alt user secret key (default: `rustfsalt`)
- `S3_REGION`: S3 region (default: `us-east-1`)
- `S3_HOST`: S3 service host (default: `127.0.0.1`)
- `S3_PORT`: S3 service port (default: `9000`)

### Test Parameters

- `TEST_MODE`: Test mode (default: `single`)
- `TEST_SCOPE`: Test scope (default: `implemented`)
  - `implemented`: run only the `implemented_tests.txt` whitelist (the PR gate)
  - `all`: run the entire upstream suite (used by the weekly full sweep)
- `MAXFAIL`: Stop after N failures, `0` = never stop (default: `1`)
- `XDIST`: Enable parallel execution with N workers (default: `0`, disabled)
- `S3TESTS_REPO`: s3-tests repository URL (default: `https://github.com/ceph/s3-tests.git`)
- `S3TESTS_REV`: Pinned s3-tests commit for reproducible runs
  - Bump deliberately: upstream changes can rename tests or change assertions,
    so a bump usually requires reclassifying the test list files
- `MARKEXPR`: pytest marker expression for filtering tests
  - Default: no marker filtering; file-based test lists control the selected tests
  - Can be customized to test specific marker groups
- `TESTEXPR`: optional pytest `-k` expression for custom runs
  - Default: exact pytest node ids loaded from `implemented_tests.txt`
  - Setting `TESTEXPR` overrides the implemented test list

### Configuration Files

- `S3TESTS_CONF_TEMPLATE`: Path to s3tests config template (default: `.github/s3tests/s3tests.conf`)
  - Relative to project root
  - Uses `envsubst` to substitute variables (e.g., `${S3_HOST}`)
- `S3TESTS_CONF`: Path to generated s3tests config (default: `s3tests.conf`)
  - Relative to project root
  - This file is generated from the template before running tests

## Examples

### Build Mode (Default)

```bash
# Basic usage - compiles and runs automatically
# Skips compilation if binary exists and is less than 30 minutes old
./scripts/s3-tests/run.sh

# Force rebuild (skip cache check, always compile)
./scripts/s3-tests/run.sh --no-cache

# Run all tests, stop after 50 failures
MAXFAIL=50 ./scripts/s3-tests/run.sh

# Enable parallel execution (4 worker processes)
# Automatically installs pytest-xdist if needed
XDIST=4 ./scripts/s3-tests/run.sh

# Use custom data storage location
# Data will be stored in /tmp/test-data/rustfs-single/
DATA_ROOT=/tmp ./scripts/s3-tests/run.sh

# Run specific test markers (e.g., test multipart uploads only)
MARKEXPR="multipart" ./scripts/s3-tests/run.sh
```

### Binary File Mode

```bash
# First compile the binary
cargo build --release

# Run with default path
DEPLOY_MODE=binary ./scripts/s3-tests/run.sh

# Specify custom path
DEPLOY_MODE=binary RUSTFS_BINARY=/path/to/rustfs ./scripts/s3-tests/run.sh

# Use binary with parallel tests
DEPLOY_MODE=binary XDIST=4 ./scripts/s3-tests/run.sh
```

### Docker Mode

```bash
# Build Docker image and run in container
DEPLOY_MODE=docker ./scripts/s3-tests/run.sh

# Run with parallel tests
DEPLOY_MODE=docker XDIST=4 ./scripts/s3-tests/run.sh
```

### Existing Service Mode

```bash
# Connect to locally running service (default: 127.0.0.1:9000)
DEPLOY_MODE=existing ./scripts/s3-tests/run.sh

# Connect to remote service
DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.sh

# Test specific features (custom marker expression)
DEPLOY_MODE=existing MARKEXPR="not lifecycle and not versioning" ./scripts/s3-tests/run.sh

# Use custom credentials
DEPLOY_MODE=existing \
  S3_ACCESS_KEY=myaccesskey \
  S3_SECRET_KEY=mysecretkey \
  ./scripts/s3-tests/run.sh
```

### Custom Configuration Files

```bash
# Use custom config template and output path
S3TESTS_CONF_TEMPLATE=my-configs/s3tests.conf.template \
S3TESTS_CONF=my-s3tests.conf \
./scripts/s3-tests/run.sh
```

## Test Results

Test results are saved in the `artifacts/s3tests-${TEST_MODE}/` directory (default: `artifacts/s3tests-single/`):

- `junit.xml`: Test results in JUnit format (compatible with CI/CD systems)
- `pytest.log`: Detailed pytest logs with full test output
- `all-collected-nodeids.txt`: Exact node IDs in the pinned upstream suite
- `selected-nodeids.txt`: Exact node IDs expected in this run
- `unsharded-selected-nodeids.txt`: Exact node IDs before deterministic sharding
- `compat-report.md`: Classification report generated by `report_compat.py` —
  regressions against `implemented_tests.txt`, promotion candidates (tests
  that pass but are still listed as unimplemented/excluded), and tests missing
  from every list
- `rustfs-${TEST_MODE}/rustfs.log`: RustFS service logs
- `rustfs-${TEST_MODE}/inspect.json`: Service metadata (PID, binary path, mode, etc.)

View results:

```bash
# Check test summary
cat artifacts/s3tests-single/junit.xml | grep -E "testsuite|testcase"

# View test logs
less artifacts/s3tests-single/pytest.log

# View service logs
less artifacts/s3tests-single/rustfs-single/rustfs.log
```

## Prerequisites

### Required System Dependencies

The following dependencies must be installed manually on your system:

#### All Deployment Modes

- **Python 3**: Required for running s3-tests
  - Check: `python3 --version`
  - Install:
    - macOS: Usually pre-installed, or `brew install python3`
    - Linux: `apt-get install python3` or `yum install python3`

- **Git**: Required for cloning s3-tests repository
  - Check: `git --version`
  - Install:
    - macOS: Usually pre-installed, or `brew install git`
    - Linux: `apt-get install git` or `yum install git`

- **Port checking tools**: One of the following for port availability checks
  - `nc` (netcat): `apt-get install netcat` or `brew install netcat`
  - OR `timeout` command: Usually pre-installed on Linux
  - OR bash built-in TCP redirection support

#### Docker Mode Only

- **Docker**: Required only when using `DEPLOY_MODE=docker`
  - Check: `docker --version`
  - Install: [Docker Installation Guide](https://docs.docker.com/get-docker/)

#### Build Mode Only

- **Rust toolchain**: Required when using `DEPLOY_MODE=build` (default)
  - Check: `rustc --version` and `cargo --version`
  - Install: [Rust Installation Guide](https://www.rust-lang.org/tools/install)

### Auto-installed Dependencies

The script will automatically install the following dependencies if missing (no manual action required):

- **awscurl**: For S3 API calls and user provisioning
  - Installed via: `python3 -m pip install --user --upgrade pip awscurl`
  - Location: `$HOME/.local/bin/awscurl`

- **tox**: For running s3-tests in isolated Python environment
  - Installed via: `python3 -m pip install --user --upgrade pip tox`
  - Location: `$HOME/.local/bin/tox`

- **gettext-base**: For `envsubst` command (config file generation)
  - macOS: Automatically installs via `brew install gettext`
  - Linux: Automatically installs via `sudo apt-get install gettext-base`
  - **Note**: macOS installation may require manual intervention if brew fails

- **s3-tests repository**: Automatically fetched if not present
  - Source: `https://github.com/ceph/s3-tests.git`, pinned to the commit in
    `S3TESTS_REV` (see run.sh) for reproducible runs
  - Location: `${PROJECT_ROOT}/s3-tests`

**Note**: The script adds `$HOME/.local/bin` to `PATH` automatically, so auto-installed Python tools are accessible.

### Proxy Configuration

The script automatically disables proxy for localhost requests to avoid interference. All proxy environment variables (`http_proxy`, `https_proxy`, `HTTP_PROXY`, `HTTPS_PROXY`) are unset at script startup. The `NO_PROXY` variable is set to `127.0.0.1,localhost,::1`.

## Cleanup

The test script automatically cleans up processes and containers when it exits. However, if you need to manually clean up:

### Using the Cleanup Script

A dedicated cleanup script is available to clean up test resources:

```bash
# Report port 9000 usage and clean the test data directory
./scripts/s3-tests/cleanup.sh

# Use custom port
S3_PORT=9001 ./scripts/s3-tests/cleanup.sh
```

The cleanup script will:
- Report any process using the specified port (default: 9000)
- Clean test data directory at `target/test-data/rustfs-single/`
- Leave unrelated processes running

### Manual Cleanup

If the cleanup script doesn't work or you need more control:

```bash
# Kill process on port 9000
lsof -ti:9000 | xargs kill -9

# Or use netstat/ss
kill -9 $(netstat -tuln | grep :9000 | awk '{print $7}' | cut -d'/' -f1)

# Remove test data
rm -rf target/test-data/rustfs-single/

# Stop Docker container (if using docker mode)
docker rm -f rustfs-single

# Remove Docker network (if using docker mode)
docker network rm rustfs-net
```

## Troubleshooting

### Port Already in Use

If port 9000 is already in use, change the port:

```bash
S3_PORT=9001 ./scripts/s3-tests/run.sh
```

**Note**: The script automatically checks if the port is available before starting (except in `existing` mode). If the port is in use, the script will exit with an error message.

### Container Start Failure

Check Docker logs:

```bash
docker logs rustfs-single
```

### Binary Not Found

For binary mode, ensure the binary is compiled:

```bash
cargo build --release
```

Or specify the correct path:

```bash
DEPLOY_MODE=binary RUSTFS_BINARY=/path/to/rustfs ./scripts/s3-tests/run.sh
```

### Test Timeout

Increase wait time or check service status:

```bash
curl http://127.0.0.1:9000/health
```

### Existing Service Not Accessible

For existing mode, ensure the service is running and accessible:

```bash
# Check if service is reachable
curl http://192.168.1.100:9000/health

# Verify S3 API is responding
awscurl --service s3 --region us-east-1 \
  --access_key rustfsadmin \
  --secret_key rustfsadmin \
  -X GET "http://192.168.1.100:9000/"
```

## Workflow Integration

This script is the single source of truth for running ceph s3-tests against
RustFS. Two GitHub Actions workflows delegate to it:

- **PR gate** (`.github/workflows/ci.yml`, job `s3-implemented-tests`): runs
  the `implemented_tests.txt` whitelist against a single-node debug binary on
  every pull request (`DEPLOY_MODE=binary`, `MAXFAIL=0`, `XDIST=4`). Any
  failure blocks the PR.
- **Full sweep** (`.github/workflows/e2e-s3tests.yml`): weekly scheduled (and
  manually dispatchable) run of the ENTIRE upstream suite (`TEST_SCOPE=all`)
  against a Docker deployment — single node or a real 4-node distributed
  cluster behind HAProxy. Regressions, unclassified tests, incomplete
  execution, and infrastructure errors fail the sweep; classified unsupported
  behavior remains informational. Scheduled topology runs are split into four
  deterministic exact-node-ID shards, and every case has a five-minute timeout,
  so one stalled case cannot erase the entire sweep's evidence.

Keeping both workflows on this script means local runs, the PR gate, and the
scheduled sweep always execute tests the same way (same pinned s3-tests
revision, same config template, same user provisioning).

A third workflow, **mint** (`.github/workflows/mint.yml`), complements ceph
s3-tests with MinIO Mint: the functional suites of many real client SDKs and
tools (awscli, mc, aws-sdk-java/go/php/ruby, minio-go/java/js/py, s3cmd, ...).
It runs weekly, is report-only for test failures, and publishes a per-suite
pass/fail table in the job summary.

## Companion Tools

- `report_compat.py` — diffs a junit.xml result against the classification
  lists and the exact pytest collection; run before execution to reject stale
  or missing classifications, then after execution to detect regressions and
  incomplete parameterized cases.
- `api_coverage.py` — quantifies S3 API surface coverage by comparing the
  s3s `S3` trait (at the revision pinned in Cargo.toml) against the methods
  RustFS overrides in `impl S3 for FS`:

  ```bash
  python3 scripts/s3-tests/api_coverage.py            # summary + missing ops
  python3 scripts/s3-tests/api_coverage.py --output api-coverage.md
  ```

- `compare_dual_targets.py` — sends one signed S3 request to two endpoints
  (e.g. RustFS and MinIO) and diffs status, headers, and body; useful when
  triaging a behavioral difference found by the suites above.

## See Also

- [GitHub Actions Workflow](../../.github/workflows/e2e-s3tests.yml)
- [S3 Tests Configuration](../../.github/s3tests/s3tests.conf)
- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
