gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,838 Commits 23 Branches 117 Tags
main
Commit Graph

7 Commits

Author SHA1 Message Date
houseme f3a7a4b0da chore(deps): localize workspace dependency features (#4888) 
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.

Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-07-16 03:55:27 +00:00
houseme 750e5d15eb feat(checksums): add native S3 additional checksum support (#4805) 
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2)

Add the AWS 2026-04 additional checksum algorithms as base types in
rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len,
hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path
silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the
single source of truth, allocate the new base-type bits append-only above
bit 9 to preserve the on-disk varint format, and add streaming hashers whose
digest uses the S3 canonical big-endian encoding (seed 0).

The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is
rejected and they are never routed through add_part()/can_merge(). A
round-trip guardrail test asserts every base type survives all dispatch
sites, failing loudly if a future algorithm is added but a match arm or the
mask is forgotten.

Refs rustfs/backlog#1254 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3)

Lock the byte order and seed of the new algorithms against the OFFICIAL
upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128,
SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte
identical to what AWS SDKs (awscrt) compute — the interop correctness this
feature hinges on. Add a non-empty regression lock (official "fox" vectors)
that also asserts the encoded field is the standard-base64 of the raw digest.

Refs rustfs/backlog#1255 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8)

Cover the xl.meta varint (de)serialization for the new algorithms:
to_bytes() -> read_checksums() must recover the value under the Display key
for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node
reading a future, unknown base-type bit degrades safely — skips the entry and
returns without panicking or mis-decoding a length. Combined with the
append-only bit allocation from S2, this protects mixed-version clusters.

Refs rustfs/backlog#1260 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5)

HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128
and SHA-512 checksums that S3 stored, closing the head_object gap in #4800.
s3s HeadObjectOutput has no typed field for these, so they are emitted as raw
response headers via response.headers (the same mechanism RustFS already uses
for tagging-count), keyed by ChecksumType::key(). The existing five typed
algorithms are unchanged. Also carries the Cargo.lock update for the
xxhash-rust dependency introduced in S2.

Refs rustfs/backlog#1257 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(checksums): fail-closed on unknown checksum algorithm (S7)

A. Harden unknown/unsupported checksum algorithms to fail closed instead of
   panicking. ChecksumMode::base() in the outbound S3 client
   (crates/ecstore/src/client/checksum.rs) previously did
   `panic!("enum err.")` for any mode without a concrete base algorithm (e.g.
   a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added
   unit tests proving base() never panics and hasher() returns Err for
   unsupported modes. rustfs-checksums FromStr already returns Err on unknown
   names; added a regression test asserting garbage/unknown names fail closed.

B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional
   algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str,
   into_impl, name constants, the x-amz-checksum-* header constants and the
   HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio
   spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128
   big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each
   digest against a direct library computation. MD5 stays intentionally
   rejected (PR #4513) and is left untouched.

C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8"
   with 7 variants already consuming 7 bits; adding the 4 new algorithms would
   overflow u8 and require a breaking repr change, so ChecksumMode is left
   unchanged. The new algorithms are available through the rustfs-checksums
   ChecksumAlgorithm path.

Refs rustfs/backlog#1259 rustfs/backlog#1252

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4)

Complete the additional-checksum round-trip so AWS SDKs can verify integrity on
download and confirm it on upload:

- GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and
  SHA-512 checksums (the download-side path SDKs auto-verify). The values flow
  from build_get_object_checksums through GetObjectOutputContext into
  finalize_get_object_response and are emitted after wrap_response_with_cors.
- PutObject echoes the server-computed additional checksum on its response,
  captured at the want_checksum set points before opts is moved.

Both reuse a single centralized helper, inject_additional_checksum_headers,
which HeadObject now also uses. This is the ONLY place that emits these headers,
so when s3s gains typed fields for these algorithms the migration is one spot
(fill the typed field, drop the insert) with no risk of duplicate headers.

The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value
lands after the body) is left for e2e coverage in S10.

Refs rustfs/backlog#1257 rustfs/backlog#1256 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9)

Make multipart uploads work end-to-end for the composite-only algorithms
(XXHash3/64/128, SHA-512):

- complete_part_checksum previously returned the outer None for any algorithm
  outside the five typed ones, which failed CompleteMultipartUpload with
  InvalidPart. It now accepts any valid base type with no double-check value
  (Some(None)) — mirroring the missing-value path of the typed algorithms —
  since s3s CompletePart has no field to carry a client-supplied per-part
  value and the part was already verified server-side at UploadPart. Genuinely
  unset/invalid types are still rejected.
- The existing COMPOSITE assembly (Checksum::new_from_data over the
  concatenated per-part raw digests; full_object_requested() is false so
  add_part() is correctly bypassed) already works for these algorithms via the
  S2 wiring. A rio test locks the assembly and that add_part refuses them.
- UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on
  their responses via the shared inject_additional_checksum_headers helper
  (now pub(crate)), since s3s has no typed output field.

Refs rustfs/backlog#1261 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6)

Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from
the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest,
COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector.

Thanks to the single-source-of-truth wiring from S2, every dispatch site
(GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the
COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all
arm — no handler changes needed. Tests are extended to cover MD5 across them.

Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client
reject "md5" so it could never silently fall back to CRC32. This change is on
the server-side rio path and never falls back — it implements MD5 correctly
rather than substituting another algorithm — so the #4513 intent is preserved,
and the outbound client keeps rejecting md5 (S7).

Refs rustfs/backlog#1258 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11)

from_string_with_obj_type ran alg.to_uppercase() on every checksummed request,
allocating a String just to compare against a fixed set of algorithm names.
Replace it with eq_ignore_ascii_case, which is allocation-free and, for the
ASCII algorithm names involved, exactly equivalent. A test locks that
case-insensitivity, the CRC64NVME full-object assumption, composite-only
FULL_OBJECT rejection, and unknown/empty handling are all unchanged.

The other S11 notes are intentionally not acted on: the Phase-0 header scan is
N/A (we chose full support over rejection, so there is no reject guard), and
parallelizing the serialized hash passes is deferred pending a measured need.

Refs rustfs/backlog#1263 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(checksums): collapse 5 duplicated response-checksum loops into one

Review of the accumulated commits found the same "iterate decrypted checksums,
match five typed algorithms, drop the rest" loop copy-pasted across five
response paths (GetObject, HeadObject, GetObjectAttributes object-level and
part-level, CompleteMultipartUpload). That was patch-on-patch duplication.

Collapse it into a single source of truth:
- rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the
  five-typed vs additional-algorithm split.
- object_usecase gains ResponseChecksums + classify_response_checksums(), which
  performs the typed/extra split once. All five call sites now destructure its
  result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a
  hand-rolled five-way comparison.

Behaviour is unchanged (GetObjectAttributes still cannot surface the additional
algorithms — an s3s XML-body limitation, now documented in one spot). One pass
over the map; extra pairs pushed only when a new-algorithm checksum is present.

Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* test(checksums): unit tests for classifier/echo helpers + fix unused import

Add direct unit tests for the refactored single-source-of-truth helpers:
- rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and
  that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification.
- object_usecase classify_response_checksums() — typed fields vs `extra` headers,
  the checksum-type marker, and empty input.
- additional_checksum_echo_pairs() — echo pair only for additional algorithms,
  none for the five typed ones, none for None.
- inject_additional_checksum_headers() — writes all pairs; empty is a no-op.

Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left
by the classifier refactor (would fail the -D warnings gate).

Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length)

The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword
the S8 forward-compat comment; no code change.

Refs rustfs/backlog#1260
Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10)

Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04
additional algorithms. aws_sdk_s3 has no typed builder for these, so the
x-amz-checksum-<algo> header is injected via mutate_request (value from
rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic
checksum calculation disabled (request_checksum_calculation=WhenRequired) so the
injected header is the only checksum on the wire. For each of XXHash3/64/128,
SHA-512 and MD5: a correct value is accepted and the object stored intact; a
mismatched value is rejected with BadDigest and nothing is stored.

Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that
additionally confirms the HEAD/GET header echo (14/14).

Refs rustfs/backlog#1262 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

* style(get): allow too_many_arguments on finalize_get_object_response

The classifier refactor added an extra_checksum_headers parameter, pushing
finalize_get_object_response to 8 args and tripping clippy::too_many_arguments
under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already
carry; no behavior change.

Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-07-14 12:03:59 +00:00
houseme 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012) 
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
 2026-05-19 12:50:25 +00:00
heihutu 3c28f0a0ba feat(metrics): migrate system monitoring from rustfs-obs to rustfs-metrics (#2242) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-20 18:52:33 +08:00
houseme e0c99bced4 chore: add tls log and removing unused crates (#359) 
* chore: add tls log

* improve code for http

* improve code dependencies for `cargo.toml` and removing unused crates

* modify name

* improve code

* fix

* Update crates/config/src/constants/env.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* improve code

* fix

* add `is_enabled` and `is_disabled`

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-08-07 19:02:09 +08:00
houseme c5df1f92c2 refactor: replace lazy_static with LazyLock and notify crate registry create_targets_from_config (#311) 
* improve code for notify

* improve code for logger and fix typo (#272)

* Add GNU to  build.yml (#275)

* fix unzip error

* fix url change error

fix url change error

* Simplify user experience and integrate console and endpoint

Simplify user experience and integrate console and endpoint

* Add gnu to  build.yml

* upgrade version

* feat: add `cargo clippy --fix --allow-dirty` to pre-commit command (#282)

Resolves #277

- Add --fix flag to automatically fix clippy warnings
- Add --allow-dirty flag to run on dirty Git trees
- Improves code quality in pre-commit workflow

* fix: the issue where preview fails when the path length exceeds 255 characters (#280)

* fix

* fix: improve Windows build support and CI/CD workflow (#283)

- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support

Resolves Windows build failures in GitHub Actions.

* fix: update link in README.md leading to a 404 error (#285)

* add rustfs.spec for rustfs (#103)

add support on loongarch64

* improve cargo.lock

* build(deps): bump the dependencies group with 5 updates (#289)

Bumps the dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [hyper-util](https://github.com/hyperium/hyper-util) | `0.1.15` | `0.1.16` |
| [rand](https://github.com/rust-random/rand) | `0.9.1` | `0.9.2` |
| [serde_json](https://github.com/serde-rs/json) | `1.0.140` | `1.0.141` |
| [strum](https://github.com/Peternator7/strum) | `0.27.1` | `0.27.2` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.36.0` | `0.36.1` |


Updates `hyper-util` from 0.1.15 to 0.1.16
- [Release notes](https://github.com/hyperium/hyper-util/releases)
- [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.15...v0.1.16)

Updates `rand` from 0.9.1 to 0.9.2
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.1...rand_core-0.9.2)

Updates `serde_json` from 1.0.140 to 1.0.141
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.140...v1.0.141)

Updates `strum` from 0.27.1 to 0.27.2
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/compare/v0.27.1...v0.27.2)

Updates `sysinfo` from 0.36.0 to 0.36.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.0...v0.36.1)

---
updated-dependencies:
- dependency-name: hyper-util
  dependency-version: 0.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rand
  dependency-version: 0.9.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde_json
  dependency-version: 1.0.141
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: strum
  dependency-version: 0.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* improve code for logger

* improve

* upgrade

* refactor: 优化构建工作流，统一 latest 文件处理和简化制品上传 (#293)

* Refactor: DatabaseManagerSystem as global

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix: fmt

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add e2e_test for s3select

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add test script for e2e

Signed-off-by: junxiang Mu <1948535941@qq.com>

* improve code for registry and intergation

* improve code for registry `create_targets_from_config`

* fix

* Feature up/ilm (#305)

* fix

* fix

* fix

* fix delete-marker expiration. add api_restore.

* fix

* time retry object upload

* lock file

* make fmt

* fix

* restore object

* fix

* fix

* serde-rs-xml -> quick-xml

* fix

* checksum

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* transfer lang to english

* upgrade clap version from 4.5.41 to 4.5.42

* refactor: replace `lazy_static` with `LazyLock`

* add router

* fix: modify comment

* improve code

* fix typos

* fix

* fix: modify name and fmt

* improve code for registry

* fix test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: Marco Orlandin <mipnamic@mipnamic.net>
Co-authored-by: zhangwenlong <zhangwenlong@loongson.cn>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: likewu <likewu@126.com>
 2025-07-30 19:02:10 +08:00
likewu d56cee26db Feature up/ilm (#305) 
* fix

* fix

* fix

* fix delete-marker expiration. add api_restore.

* fix

* time retry object upload

* lock file

* make fmt

* fix

* restore object

* fix

* fix

* serde-rs-xml -> quick-xml

* fix

* checksum

* fix

* fix

* fix

* fix

* fix

* fix

* fix
 2025-07-29 14:21:19 +08:00