gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 16:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,651 Commits 21 Branches 117 Tags
efebcb66be412fa2d11f5e9dcfd7b027aa005b3e
Commit Graph

581 Commits

Author SHA1 Message Date
houseme 0a00d8d500 perf(server): lighten internode data-plane stack (#3735) 
* refactor(server): split internode dispatch scaffold

* test(server): cover internode dispatch prefix split

* refactor(server): name internode stack boundaries

* perf(server): skip internode request logging layer

* perf(server): skip internode trace layer

* perf(server): use lite internode request context

* feat(metrics): track internode rpc duration

* feat(ecstore): add put object stage summary logs

* test(metrics): update internode descriptor expectations

* fix(server): tighten internode path matching

* fix(pr): address review follow-up comments

* style(ecstore): simplify commit tail duration field

* refactor(ecstore): group put stage summary fields

* refactor(ecstore): inline put stage summary log

* fix(s3): return storage class for object attributes

* merge: sync latest main and resolve object attributes conflict

* fmt

* fix(server): remove duplicate rpc imports

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* fix(s3select): align object_store with datafusion

* chore(deps): prune workspace dependencies

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

  Separate fuzz harness compilation from execution to eliminate redundant
  builds across targets. Introduce matrix-based parallel execution for
  PR smoke and nightly fuzz jobs.

  Changes:

  - Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
    (`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
  - Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
  - Add run_single_target.sh for matrix jobs (no build phase)
  - Optimize `local_metadata` fuzz target: reduce prefix iterations from
    8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
    per-iteration cost by ~3-5x
  - Move archive path validation (`validate_extract_relative_path`,
    `normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
    eliminating `rustfs` binary crate dependency from fuzz harness
  - Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
  - Add unit tests for archive path validation in rustfs-utils
  - Update fuzz/README.md with new workflow and script documentation

  Expected CI improvement: PR smoke wall-clock from ~120min (frequent
  timeout) to ~40min; nightly from ~180min to ~60min.

* refactor(fuzz): consolidate scripts and fix prefix test alignment

Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.

Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.

Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.

Net: -118 lines, zero functionality loss.

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.

Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env

Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars

Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
  979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
  calls to 5 prefixes with parser-only (no decompress), aligned
  with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation

Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.

* fix(rpc): resolve internode metrics via app context

* fmt

* ci: speed up fuzz smoke artifact restore

---------

Signed-off-by: houseme <housemecn@gmail.com>
 2026-06-23 21:36:39 +08:00
houseme 46775ae019 feat: harden runtime capability snapshots (#3784) 
* feat(admin): expose runtime capability snapshots

* feat(runtime): refine workload admission snapshots

* test(ci): align architecture migration checks

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* build(deps): refresh cargo deny lockfile
 2026-06-23 18:10:49 +08:00
houseme 2f85ef4455 fix(security): extend outbound egress guard (#3744) 2026-06-22 19:20:31 +08:00
dependabot[bot] 5a601869f6 chore(deps): bump sysinfo from 0.39.3 to 0.39.4 in the dependencies group (#3712) 2026-06-22 09:53:57 +08:00
houseme 9e6d0c7292 chore(deps): refresh workflow actions and crate pins (#3696) 2026-06-21 20:29:17 +08:00
安正超 08b8f58e64 refactor: prune notify ecstore object alias (#3620) 2026-06-19 20:11:56 +08:00
houseme 8cf3c0bfbd fix(ecstore): support MinIO DARE fixture compatibility (#3590) 2026-06-19 10:13:39 +08:00
houseme 53ff19dd2d fix(kms): harden local master key storage (#3589) 2026-06-19 08:50:35 +08:00
安正超 b1c6578df1 refactor: narrow test harness compatibility surfaces (#3592) 2026-06-19 07:10:52 +08:00
安正超 c28fee0013 refactor: continue storage api contract cleanup (#3580) 
* refactor: move delete object contracts to storage api

* refactor: narrow store api compatibility exports

* refactor: route table catalog test through storage compat
 2026-06-18 22:42:02 +08:00
houseme 8cf11af649 fix(security): add shared outbound egress guard (#3567) 
* fix(security): block unsafe outbound webhook targets

* chore: keep issue-3557 plan local only
 2026-06-18 15:30:09 +08:00
安正超 54bbd05b25 refactor: move object list helper contracts (#3546) 2026-06-18 06:09:33 +08:00
houseme 87a3d107d6 test(property): add path validator invariants (#3541) 
test(property): add path validator invariants for #3525
 2026-06-18 02:14:24 +08:00
houseme bc769948ab test(property): add erasure recoverability invariants (#3540) 2026-06-18 01:34:07 +08:00
安正超 53337a6f71 refactor: move HTTP range helper contracts (#3533) 2026-06-18 00:47:23 +08:00
houseme cc84d9065e test(property): add filemeta invariants (#3536) 
* test(property): add filemeta invariants for #3523

* docs: relocate issue-3518 property testing plan

* docs: keep issue-3518 property plan local only
 2026-06-18 00:36:55 +08:00
houseme 841f86fe90 test(property): expand parser invariants (#3534) 
* test(property): harden parser invariants for #3522

* test(property): satisfy clippy for #3522
 2026-06-17 23:52:30 +08:00
安正超 919deeb816 refactor: move object option helper contracts (#3521) 2026-06-17 22:56:10 +08:00
Henry Guo 65c724c786 feat(table-catalog): add manifest reachability cleanup (#3484) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-15 22:29:23 +08:00
houseme adec195486 chore(scanner): enable pulsar telemetry feature (#3481) 2026-06-15 18:43:14 +08:00
dependabot[bot] fbf1f9e5b8 build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group (#3458) 
* build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group

Bumps the dependencies group with 1 update: [brotli](https://github.com/dropbox/rust-brotli).


Updates `brotli` from 8.0.3 to 8.0.4
- [Release notes](https://github.com/dropbox/rust-brotli/releases)
- [Commits](https://github.com/dropbox/rust-brotli/compare/8.0.3...8.0.4)

---
updated-dependencies:
- dependency-name: brotli
  dependency-version: 8.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump s3s from cf4c3346 to c59b62f7ba in the dependencies group

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-15 12:06:01 +08:00
houseme 036741cb1c refactor(request-id): align contracts and lock field names (#3454) 
* refactor(request-id): align header log and trace contracts

* test(contract): lock external request-id field names

* chore(deps): drop unused rustfs-ecstore links

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-15 01:59:11 +08:00
cxymds 046d5386ba feat: stream object zip downloads (#3380) 
* feat: stream object zip downloads

* fix: stream zip downloads page by page

* fix: prepare zip downloads before streaming

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-14 14:26:44 +08:00
houseme e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
houseme 0d68851f7a refactor(logging): standardize rustfs runtime events (#3396) 
* fix(rpc): adjust log levels and reduce noise in storage RPC layer

Issue #682: pool rebalance normal flow logs were using warn! instead
of info!. Additionally, several high-frequency RPC entry points (ping,
write_stream, read_at, walk_dir) were logging at info! level, causing
unnecessary log noise in production.

Changes:
- load_rebalance_meta: warn! → info! for normal flow (3 occurrences)
- ping body decode: info! → debug! (fires on every health check)
- write_stream/read_at/walk_dir entry: info! → debug!
- metrics.rs: lowercase error messages, add structured error field
- http_service.rs: add structured error field to walk_dir failure
- Add tracing::instrument fields for start_rebalance context

* refactor(logging): standardize rustfs runtime events

* build(deps): bump workspace dependency versions

* build(deps): pin time to 0.3.47

* build(deps): update postgres client versions
 2026-06-13 08:39:47 +08:00
安正超 434b71e569 feat(extension): add admin views (#3386) 2026-06-12 15:54:42 +08:00
安正超 1e7c376586 feat(targets): add extension schema adapter (#3385) 2026-06-12 15:27:55 +08:00
安正超 7d6d56a547 feat(extension): add schema contracts (#3384) 2026-06-12 15:07:17 +08:00
安正超 7146c893cb refactor(storage-api): remove legacy bucket DTO re-export (#3364) 2026-06-11 21:24:56 +08:00
安正超 2205991180 refactor(config): extract server config model (#3351) 2026-06-11 16:11:17 +08:00
houseme 7d38b0cf90 refactor(obs): unify local and otlp log output (#3352) 
wip(obs): start lg-002 output unification
 2026-06-11 16:10:54 +08:00
houseme 8d337c4e87 feat(obs): add logging redaction guard rails (#3349) 
* feat(obs): add logging redaction guard rails

* chore(docs): keep logging plans out of lg-001 pr

* test(obs): guard against unmasked access key logs

* test(obs): enforce masked access key log patterns
 2026-06-11 15:24:32 +08:00
houseme ee96e194a3 build(deps): bump s3s for header parsing fix (#3344) 2026-06-11 04:28:36 +00:00
安正超 87968275a7 refactor(storage): route maintenance inventory reads (#3337) 2026-06-11 10:32:48 +08:00
安正超 94c53af264 refactor(storage): use admin API for observability reads (#3335) 2026-06-11 08:13:15 +08:00
安正超 8a90bda16a refactor(admin): use storage admin drive counts (#3332) 2026-06-11 06:36:18 +08:00
安正超 d9ddd1bedc feat(storage-api): add disk inventory contract (#3330) 2026-06-10 16:33:31 +00:00
安正超 bb5d9565a6 feat(storage-api): add bucket DTO contract (#3314) 
* feat(storage-api): add bucket DTO contract

* ci(build): increase workflow timeout to 90 minutes

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-10 07:16:14 +00:00
houseme b9c924a6ed build(ci): update macOS runner and bump regex/uuid (#3318) 
* ci(build): use macos-26-intel for x86_64 job

* build(deps): bump regex and uuid

* ci(build): disable cross for macos x86_64 job

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
 2026-06-10 04:42:10 +00:00
houseme 68400933b5 chore(release): prepare 1.0.0-beta.8 (#3317) 
* chore(release): prepare 1.0.0-beta.8

* chore(release): align release assets for 1.0.0-beta.8
 2026-06-10 04:34:39 +00:00
安正超 3795d44b86 feat(storage-api): add error code contract (#3313) 2026-06-10 09:03:29 +08:00
安正超 45391ac776 feat(storage-api): add contract crate (#3310) 2026-06-09 20:06:15 +00:00
安正超 03eb10b07f feat: add KMS config redaction safeguards (#3303) 2026-06-09 14:08:58 +00:00
安正超 b9a3a2245a feat: add admin route policy inventory (#3286) 
* feat: add admin route policy inventory

* fix: align table route policy actions

* fix: align load table route policy action
 2026-06-09 02:17:50 +00:00
houseme 9354b939ca refactor(zip): simplify archive extraction path (#3290) 
* refactor(zip): simplify archive extraction path

* build(deps): align datafusion and http updates

* fix(test): restore pre-commit compatibility
 2026-06-08 20:03:02 +00:00
唐小鸭 f7724d223b feat(rio): rio_v2 is compatible with minio for storing data. (#3115) 
* Set up a compatibility layer for replacing old Rio components with new ones.

* fix(rio). compress range

* feat(rio). Add the experimental feature rio_v2 to support minio data at the binary level.

* feat(rio_v2): add sse-c test

* test compression component

* simple fix

* fix minlz encode

* fix metadata

* fix kms key cache error

* Update launch.json

* ci: set nix crate download user agent

* fix: gate obs pyroscope backend

* ignore minio test

* fix encrypt check

* fix

* fix

* fix

* Update object_usecase.rs

* Update ci.yml

* fix

* ci add rio-v2 test

* fix

* ci fix

* fix

* Reconstructed into a more reasonable compatibility mode

* fix

* fix

---------

Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-06-08 11:59:14 +00:00
houseme c479f3d0cb fix(ecstore): gate rustix fs diagnostics on windows (#3267) 
* fix(ecstore): gate rustix fs diagnostics on windows

* fix(ecstore): improve windows disk validation diagnostics

* build(deps): bump workspace dependency versions
 2026-06-08 00:05:55 +00:00
安正超 7a9bf707ee feat: add security governance contract types (#3270) 2026-06-07 23:14:11 +00:00
安正超 3bd89944c2 perf(erasure): remove UUID from clone + increase encode inflight budget (#3212) 
* perf(erasure): remove UUID from clone + increase encode inflight budget

Two targeted optimizations for the erasure encoding hot path:

1. Erasure::clone() no longer generates Uuid::new_v4() per clone.
   The _id field is unused in the hot path; reusing the original ID
   eliminates a CSPRNG call per block encode (100 calls for a 100MB
   object with 1MB blocks).

2. Default RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES raised from 8MB
   to 32MB. This increases the encode pipeline depth from ~5 to ~20
   blocks, allowing more read-ahead between the encoder and disk
   writer stages. The per-request memory bound is still controlled
   by the 8-block hard cap and the env var override.

3. Added encode_data_owned() utility method for zero-copy encoding
   when the caller already owns a heap buffer (Vec<u8> → BytesMut
   via Bytes::try_into_mut). Not used in the hot path yet but
   available for future callers.

All 1157 ecstore tests pass. Criterion micro-benchmarks show no
regression (< 2% variance). Single-machine warp E2E tests were
inconclusive due to high variance; a dedicated multi-disk test
environment is needed for reliable E2E comparison.

Ref: https://github.com/rustfs/backlog/issues/659

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: update Cargo.lock

* fix(erasure): align encode inflight cap and tests

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-04 13:49:36 +00:00
houseme 66b8699927 chore(release): prepare 1.0.0-beta.7 (#3184) 
* chore(release): prepare 1.0.0-beta.7

* chore(release): align release assets for 1.0.0-beta.7
 2026-06-03 04:18:50 +00:00