mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-18 18:46:17 +00:00
1cf0f7af15
* refactor(replication): split four oversized hot-path functions into focused helpers Pure-move decomposition of the four oversized functions flagged by the replication compatibility review (P1-18), unblocking migration milestone M2 which requires resyncer moves to stay mechanical: - resync_bucket (522 lines -> 61-line step sequence): leader lock, target resolution, walk/collector/worker spawning, and dispatch loop extracted into focused helpers; pure decision helpers (DTO builders, HEAD-result classification) separated from IO orchestration. - replicate_all (411 lines -> 113-line main body): initial target-info seeding, read/stat option builders, skip-path notes, target HEAD action resolution, and the multipart/single-put payload transport extracted as private free functions. - start_mrf_processor (306 lines -> 46-line spawn body): recovery guard, ledger load, per-entry replay (delete/object/metadata), and retained entry resolution extracted; retry bookkeeping semantics preserved exactly (inner continue-paths push inside helpers, outer Missed push stays in the loop). - apply_iam_item (255 lines -> match dispatch skeleton): one helper per IAM item type. No behavior change: log texts, error paths, event emissions, and metric counts are byte-identical; existing tests unchanged and green (238 ecstore replication/mrf/resync + 232 rustfs site-replication). * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172) * feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets Implements the MinIO active-active read-proxy protocol (P1-5 of the replication compatibility review): when a GET/HEAD/GetObjectTagging/ PutObjectTagging/DeleteObjectTagging request fails locally with not-found and the bucket has replication targets, the request is proxied to the targets in rule order, mirroring bucket-replication.go proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget. Protocol surface: - Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed into ObjectOptions (proxy_request + proxy_header_set, matching MinIO ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY value is never re-proxied. Outbound client proxy calls send the marker as "true"; replication worker convergence HEADs send it as "false" so a peer's proxy layer cannot answer a convergence check by proxying back to the source (which would fake Completed without a PUT). - Target selection: new replication_proxy.rs get_proxy_targets — empty when the marker is set, versioning is suspended, or no replication config; otherwise filter_target_arns -> TargetClient lookup, skipping targets with proxying disabled. - TargetClient gains head_object_for_proxy/get_object (streaming) and the three tagging calls. Proxy calls never send the replication-check SSE-C exemption header; customer SSE-C keys are forwarded verbatim so the target performs real decryption. Conditional (If-*) headers are not forwarded (MinIO parity); Range and part_number are, with parts_count/tag_count/storage_class/expiration passed through. - Metrics: proxy counters now count only real client proxy traffic, MinIO-aligned (one total per proxied request, one failed when no target served it). The previous misattributed counters — replication worker HEAD/PUT (#2672) and local tagging operations (#2682) — are removed; ReplProxyMetric now maps the tagging counters instead of dropping them. e2e (fake_s3_target extended with tagging + header journaling): proxied GET body + outbound header contract (marker present, no replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero outbound requests, GetObjectTagging, and metric mapping unit tests. Rolling note: proxying only activates for buckets with replication targets; requests carrying the marker keep pre-upgrade behavior. Refs rustfs/backlog#1675 (P1-5) * fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178) SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport headers. A MinIO/generic-S3 target silently discards them, storing bare ciphertext with no decryption material — yet the PUT succeeded, so the object reported COMPLETED with a silently unreadable replica (backlog#1675 N2). Fail-closed design: - SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in BucketTargetSys per target ARN with a recording timestamp. Entries reset whenever the target is rebuilt, edited, or removed (arn_remotes_map lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes): an expired verdict in either direction is re-earned through the audit, so an Unsupported target recovers automatically after an upgrade (at most one wasted PUT+HEAD audit per bad target per TTL window) and a Supported verdict cannot outlive a backend swapped behind the same endpoint. - Replication worker (replicate_object and replicate_all): fresh Unsupported targets never receive the PUT — the attempt fails immediately into the normal MRF retry channel with a "run ?replication-check to re-probe" hint. Unknown or expired verdicts are audited: after the PUT the worker HEADs the replica back through the replication-check channel (source version id mapped through resolve_read_api_version_id, so null-version objects audit correctly) and requires SSE-C evidence (the echoed customer-algorithm header); missing evidence records Unsupported and fails the attempt. Convergence HEADs are audited the same way, so a broken ciphertext replica from an earlier attempt can never launder itself into COMPLETED via an ETag match. The gate/evidence policy is pure (replication_target_boundary, staleness folded in as an input) for the M2 worker migration. - replication-check grows an SsecPassthrough probe phase: a probe PUT carrying the live transport-header shape, HEAD-back for evidence, and a machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure. The probe verdict is synced into the runtime capability cache. Unlike VersionFidelity, a failed SsecPassthrough phase does NOT fail the target overall — it is a capability limit, not a broken replication contract, and a plaintext-only deployment against such a target must not turn red. - fake_s3_target: default mode now models a RustFS target (stores the transport headers, echoes SSE-C evidence); the new drop_unlisted_replication_headers mode models MinIO. The journal records whether a request carried transport headers. Receiver-echo verification: the replication-check HEAD exemption only skips SSE-C key validation; the response has always built sse-customer-algorithm from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver change was needed — pinned end to end by the replication-check e2e against a real RustFS target. Rolling-upgrade constraint: RustFS targets older than the replication-check HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail it outright), so SSE-C replication to such targets reports FAILED. This is deliberate — FAILED-and-retryable beats a silently undecryptable replica — and self-heals: once the target is upgraded, the next TTL expiry (or a manual ?replication-check re-probe) re-audits and records Supported. Plaintext and managed-SSE replication are unaffected. The capability cache is per-node; each node audits independently. Known limitations: - The audit judges evidence from the echoed customer-algorithm header only. A hypothetical target that preserves that one header while dropping other transport headers (partial-drop) would pass the audit; no known target behaves this way — observed targets drop the whole unknown-header family. - A mixed-version target cluster can flap the verdict between audits routed to different target nodes until the rollout completes; the TTL bounds how long each stale verdict persists. New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a header-dropping fake (FAILED + no second PUT via the capability cache, journal-asserted; red run showed the old COMPLETED), replication-check reports the SsecPassthrough phase Code while the target stays OK overall, SSE-C heal convergence after a real target outage, and SSE-C existing-object resync landing a REPLICA readable with the customer key. TTL expiry in both directions is pinned at the cache and gate seams. * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180) * refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) Pure-move milestone M2 of the ECStore replication split (backlog#1675 P1-17): relocate the resyncer's IO-free decision helpers, with their unit tests, into the crates they already belong to by type ownership. No behavior change. Moved into crates/replication: - resync.rs: resync_status_duration - delete.rs: resync_existing_delete_replication_info, replicate_delete_outcome, target_delete_version_id, delete_marker_purge_version_id, delete_marker_purge_mrf_entry - object.rs: version_identity_drifted, is_replication_target_offline_error, SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate, ssec_passthrough_evidence_present (param-demoted to the echoed customer-algorithm string; ECStore keeps the HeadObjectOutput adapter) - filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the filemeta-independence contract) ECStore rewiring (Rule #14: imports stay in *_boundary.rs): - resync/object-decision/target boundaries re-export the moved symbols; resyncer call sites are unchanged - bucket_target_sys keeps only the verdict cache + TTL and re-exports the capability enum so existing consumer paths keep compiling Not moved (signatures carry ECStore or aws-sdk types): verify_resync_head_result, resync_target_error_detail, the SdkError classifiers, the replicate_all_* option/info builders, and the env-coupled bounded_resync_max_jobs admission clamp. README milestone table updated. * chore(replication): retire the datatypes.rs relay early README sanctions retiring datatypes.rs ahead of M4. The module was a pure relay (resync boundary -> datatypes -> mod.rs facade) with no external consumer importing it directly, so the facade now re-exports ResyncStatusType from replication_resync_boundary and the relay file is deleted. Consumers stay behind the ECStore facade, keeping Migration Rule #15 intact — the original retirement wording ("consumers import through rustfs-replication directly") conflicted with that rule and is corrected in the README. * chore(arch): extend migration guards to the M2-moved decision contracts The adversarial review of the M2 move found the per-symbol ratchet in check_architecture_migration_rules.sh was not extended for the moved symbols, leaving them free to be redefined in ECStore or imported past their boundary without CI noticing: - resync definition pin + boundary fences gain resync_status_duration; - the object-decision boundary fences gain the five delete-family helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id, replicate_delete_outcome, resync_existing_delete_replication_info, target_delete_version_id); - the target-boundary fence gains the SSE-C gate family, the offline classifier, and version_identity_drifted; - a new definition pin rejects ECStore redefinitions of the M2-moved fns/enums (ssec_passthrough_evidence_present deliberately excluded: ECStore keeps a thin HeadObjectOutput adapter under that name). Mutation-verified: a probe fn ssec_passthrough_gate under crates/ecstore/src/bucket/replication trips the new pin. Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal from the filemeta side and tightens the M2 README note on bounded_resync_max_jobs.
546 lines
27 KiB
Rust
546 lines
27 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! Explicit ECStore public facades for outer crate compatibility boundaries.
|
|
|
|
pub mod admin {
|
|
pub use crate::diagnostics::admin_server_info::{get_local_server_property, get_server_info};
|
|
}
|
|
|
|
pub mod bitrot {
|
|
pub use crate::io_support::bitrot::{create_bitrot_reader, create_bitrot_writer};
|
|
}
|
|
|
|
pub mod bucket {
|
|
pub mod bandwidth {
|
|
pub mod monitor {
|
|
pub use crate::bucket::bandwidth::monitor::{BandwidthDetails, Monitor};
|
|
}
|
|
}
|
|
|
|
pub mod bucket_target_sys {
|
|
pub use crate::bucket::bucket_target_sys::{
|
|
AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
|
|
SsecPassthroughCapability, TargetClient, append_version_id_query,
|
|
};
|
|
}
|
|
|
|
pub mod lifecycle {
|
|
pub mod bucket_lifecycle_audit {
|
|
pub use crate::bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc;
|
|
}
|
|
|
|
pub mod bucket_lifecycle_ops {
|
|
pub use crate::bucket::lifecycle::bucket_lifecycle_ops::{
|
|
ExpiryState, LifecycleOps, ManualTransitionCancelCheck, ManualTransitionProgressSink,
|
|
ManualTransitionQueueSnapshot, ManualTransitionRunExecution, ManualTransitionRunOptions,
|
|
ManualTransitionRunReport, RestoreRequestOps, TransitionState, TransitionedObject, apply_expiry_rule,
|
|
apply_transition_rule, enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects,
|
|
enqueue_transition_for_existing_objects_scoped, enqueue_transition_for_existing_objects_scoped_with_cancel,
|
|
enqueue_transition_immediate, expire_transitioned_object, get_global_expiry_state, get_global_transition_state,
|
|
init_background_expiry, manual_transition_queue_snapshot, post_restore_opts,
|
|
run_stale_multipart_upload_cleanup_once, validate_transition_tier,
|
|
};
|
|
}
|
|
|
|
pub mod manual_transition_job {
|
|
pub use crate::bucket::lifecycle::manual_transition_job::{
|
|
ManualTransitionJobRecord, ManualTransitionJobState, ManualTransitionScopeAdmission,
|
|
ManualTransitionScopeAdmissionClaim, claim_manual_transition_scope_admission,
|
|
delete_manual_transition_scope_admission_if_current, load_manual_transition_job_record,
|
|
load_manual_transition_job_record_with_etag, load_manual_transition_scope_admission,
|
|
manual_transition_job_lease_expired, manual_transition_scope_admission_lease_expired,
|
|
manual_transition_scope_key, persist_manual_transition_job_progress,
|
|
persist_manual_transition_job_progress_if_owned, renew_manual_transition_job_lease,
|
|
renew_manual_transition_job_lease_if_owned, request_manual_transition_job_cancel,
|
|
save_manual_transition_job_record, save_manual_transition_job_record_if_current,
|
|
save_manual_transition_scope_admission_if_absent, update_manual_transition_job_record,
|
|
};
|
|
}
|
|
|
|
pub mod transition_transaction {
|
|
pub use crate::bucket::lifecycle::transition_transaction::{
|
|
TransitionOperatorDeleteResult, TransitionOperatorError, TransitionOperatorProbe, TransitionOperatorStatus,
|
|
delete_transition_candidate_for_operator, finalize_missing_transition_transaction_for_operator,
|
|
inspect_transition_transaction_for_operator,
|
|
};
|
|
}
|
|
|
|
pub mod evaluator {
|
|
pub use crate::bucket::lifecycle::evaluator::Evaluator;
|
|
}
|
|
|
|
#[allow(clippy::module_inception)]
|
|
pub mod lifecycle {
|
|
pub use crate::bucket::lifecycle::lifecycle::{
|
|
Event, ExpirationOptions, IlmAction, Lifecycle, LifecycleCalculate, ObjectOpts, RuleValidate,
|
|
TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, expected_expiry_time, object_opts_from_object_info,
|
|
};
|
|
}
|
|
|
|
pub mod rule {
|
|
pub use crate::bucket::lifecycle::rule::{Filter, NoncurrentVersionTransitionOps, TransitionOps};
|
|
}
|
|
|
|
pub mod tier_delete_journal {
|
|
#[cfg(feature = "test-util")]
|
|
pub use crate::bucket::lifecycle::tier_delete_journal::recover_tier_delete_journal_entries;
|
|
pub use crate::bucket::lifecycle::tier_delete_journal::{
|
|
persist_tier_delete_journal_entry, record_tier_delete_journal_backend_identity,
|
|
};
|
|
}
|
|
|
|
pub mod tier_last_day_stats {
|
|
pub use crate::bucket::lifecycle::tier_last_day_stats::{DailyAllTierStats, LastDayTierStats};
|
|
}
|
|
|
|
pub mod tier_sweeper {
|
|
pub use crate::bucket::lifecycle::tier_sweeper::{
|
|
Jentry, transitioned_delete_journal_entry, transitioned_force_delete_journal_entry,
|
|
};
|
|
}
|
|
}
|
|
|
|
pub mod metadata {
|
|
pub use crate::bucket::metadata::BUCKET_DURABILITY_CONFIG;
|
|
pub use crate::bucket::metadata::{
|
|
BUCKET_ACCELERATE_CONFIG, BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_LOGGING_CONFIG,
|
|
BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, BUCKET_QUOTA_CONFIG_FILE,
|
|
BUCKET_REPLICATION_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG, BUCKET_SSECONFIG, BUCKET_TABLE_CATALOG_META_PREFIX,
|
|
BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX, BUCKET_TABLE_CONFIG, BUCKET_TABLE_RESERVED_PREFIX, BUCKET_TAGGING_CONFIG,
|
|
BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, BUCKET_WEBSITE_CONFIG, BucketMetadata, OBJECT_LOCK_CONFIG,
|
|
load_bucket_metadata, table_catalog_path_hash,
|
|
};
|
|
}
|
|
|
|
pub mod durability {
|
|
pub use crate::bucket::durability::{
|
|
BUCKET_DURABILITY_MODE_NONE, BUCKET_DURABILITY_MODE_RELAXED, BUCKET_DURABILITY_MODE_STRICT, BucketDurabilityConfig,
|
|
};
|
|
}
|
|
|
|
pub mod metadata_sys {
|
|
#[cfg(feature = "test-util")]
|
|
pub use crate::bucket::metadata_sys::ConfigWriteLockProbe;
|
|
pub use crate::bucket::metadata_sys::{
|
|
BucketMetadataMutationGuard, BucketMetadataSys, ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
|
|
acquire_bucket_metadata_transaction_lock_for_incarnation, capture_bucket_metadata_incarnation, delete,
|
|
delete_if_incarnation, delete_under_transaction_lock, get, get_accelerate_config, get_bucket_policy,
|
|
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
|
|
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
|
|
get_object_lock_config, get_object_lock_config_state, get_public_access_block_config, get_quota_config,
|
|
get_replication_config, get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config,
|
|
get_website_config, init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
|
|
set_bucket_metadata, update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
|
|
update_quota_if_incarnation, update_under_transaction_lock,
|
|
};
|
|
}
|
|
|
|
pub mod migration {
|
|
pub use crate::bucket::migration::{LegacyBlobDecryptFn, try_migrate_bucket_metadata, try_migrate_iam_config};
|
|
}
|
|
|
|
pub mod object_lock {
|
|
pub use crate::bucket::object_lock::{ObjectLockApi, ObjectLockStatusExt};
|
|
|
|
pub mod objectlock {
|
|
pub use crate::bucket::object_lock::objectlock::{get_object_legalhold_meta, get_object_retention_meta};
|
|
}
|
|
|
|
pub mod objectlock_sys {
|
|
pub use crate::bucket::object_lock::objectlock_sys::{
|
|
BucketObjectLockSys, ObjectLockBlockReason, add_years, check_object_lock_for_deletion,
|
|
check_retention_for_modification, is_retention_active,
|
|
};
|
|
}
|
|
}
|
|
|
|
pub mod policy_sys {
|
|
pub use crate::bucket::policy_sys::PolicySys;
|
|
}
|
|
|
|
pub mod quota {
|
|
pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
|
|
|
|
pub mod checker {
|
|
pub use crate::bucket::quota::checker::QuotaChecker;
|
|
}
|
|
}
|
|
|
|
pub mod replication {
|
|
pub use crate::bucket::replication::replication_pool::{
|
|
DurableMrfBacklogSummary, DurableMrfBucketBacklog, DurableMrfTargetBacklog, MrfBacklogObservabilitySummary,
|
|
MrfBucketBacklogObservability, durable_mrf_backlog_summary_snapshot, durable_mrf_target_backlog_snapshot,
|
|
mrf_backlog_observability_snapshot,
|
|
};
|
|
pub use crate::bucket::replication::{
|
|
BucketReplicationResyncStatus, BucketReplicationStat, BucketReplicationStats, BucketStats,
|
|
DeleteReplicationConfigSnapshot, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, InQueueMetric,
|
|
MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION,
|
|
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE,
|
|
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
|
|
ReplicateDecision, ReplicateObjectInfo, ReplicationBatchAdmission, ReplicationConfig,
|
|
ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
|
|
ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
|
|
ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
|
|
ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
|
|
ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
|
|
VersionPurgeStatusType, XferStats, commit_force_delete_intent, complete_force_delete_intent,
|
|
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
|
|
get_global_replication_stats, get_proxy_targets, init_background_replication,
|
|
invalid_replication_config_status_field, persist_force_delete_intent, read_durable_mrf_backlog,
|
|
replication_state_to_filemeta, replication_status_to_filemeta, replication_statuses_map, replication_target_arns,
|
|
resync_start_conflict_id, should_remove_replication_target, should_schedule_delete_replication,
|
|
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
|
|
unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
|
|
version_purge_status_to_filemeta,
|
|
};
|
|
}
|
|
|
|
pub mod tagging {
|
|
pub use crate::bucket::tagging::{decode_tags, decode_tags_to_map, encode_tags};
|
|
}
|
|
|
|
pub mod target {
|
|
pub use crate::bucket::target::{
|
|
ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials, LatencyStat, duration_from_secs_or_nanos,
|
|
};
|
|
}
|
|
|
|
pub mod utils {
|
|
pub use crate::bucket::utils::{
|
|
check_bucket_and_object_names, check_list_objs_args, check_object_name_for_length_and_slash,
|
|
check_valid_bucket_name_strict, deserialize, has_bad_path_component, is_meta_bucketname, is_valid_object_prefix,
|
|
serialize,
|
|
};
|
|
}
|
|
|
|
pub mod versioning {
|
|
pub use crate::bucket::versioning::VersioningApi;
|
|
}
|
|
|
|
pub mod versioning_sys {
|
|
pub use crate::bucket::versioning_sys::BucketVersioningSys;
|
|
}
|
|
}
|
|
|
|
pub mod cache {
|
|
pub use crate::cache_value::metacache_set::{ListPathRawOptions, list_path_raw};
|
|
}
|
|
|
|
pub mod capacity {
|
|
pub use crate::core::pools::{
|
|
PoolDecommissionInfo, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free, path2_bucket_object,
|
|
path2_bucket_object_with_base_path,
|
|
};
|
|
pub use crate::store::utils::is_reserved_or_invalid_bucket;
|
|
}
|
|
|
|
pub mod client {
|
|
pub mod admin_handler_utils {
|
|
pub use crate::client::admin_handler_utils::AdminError;
|
|
}
|
|
|
|
pub mod api_put_object {
|
|
pub use crate::client::api_put_object::{AdvancedPutOptions, PutObjectOptions};
|
|
}
|
|
|
|
pub mod object_api_utils {
|
|
pub use crate::client::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader, to_s3s_etag};
|
|
}
|
|
|
|
pub mod transition_api {
|
|
pub use crate::client::transition_api::{
|
|
BucketLookupType, CreateBucketConfiguration, LocationConstraint, ObjectInfo, ObjectMultipartInfo, Options,
|
|
PutObjectPartOptions, ReadCloser, ReaderImpl, RequestMetadata, RestoreInfo, SendRequest, TransitionClient,
|
|
TransitionCore, UploadInfo, to_object_info,
|
|
};
|
|
}
|
|
}
|
|
|
|
pub mod cluster {
|
|
pub use crate::cluster::{
|
|
ClusterControlPlane, ClusterControlPlaneSnapshot, ClusterDriveMembership, ClusterEndpointType, ClusterLocalNodeStorage,
|
|
ClusterLocalNodeStorageSnapshot, ClusterMembershipSnapshot, ClusterNodeMembership, ClusterPeerHealth,
|
|
ClusterPeerHealthSnapshot, ClusterPoolState, ClusterPoolStateSnapshot, ClusterRpcBoundarySnapshot,
|
|
ClusterRpcChannelSnapshot, ClusterRpcPlane, ClusterRpcTransport, local_node_storage_snapshot_from_membership,
|
|
membership_snapshot_from_endpoint_pools, peer_health_snapshot_from_membership, pool_state_snapshot_from_endpoint_pools,
|
|
rpc_boundary_snapshot, topology_snapshot_from_endpoint_pools, topology_snapshot_from_endpoint_pools_with_capabilities,
|
|
};
|
|
}
|
|
|
|
pub mod compression {
|
|
pub use crate::io_support::compress::{
|
|
MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible, is_disk_compression_enabled, is_multipart_disk_compression_enabled,
|
|
};
|
|
}
|
|
|
|
pub mod config {
|
|
pub mod com {
|
|
pub use crate::config::com::{
|
|
COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
|
|
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
|
|
is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
|
|
read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
|
|
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
|
|
save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
|
|
server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
|
|
with_server_config_read_lock, with_server_config_write_lock,
|
|
};
|
|
}
|
|
|
|
pub mod storageclass {
|
|
pub use crate::config::storageclass::{
|
|
CAPABILITY_CONTRACT_VERSION, CLASS_RRS, CLASS_STANDARD, Config, DEEP_ARCHIVE, DEFAULT_INLINE_BLOCK, DEFAULT_KVS,
|
|
DEFAULT_RRS_PARITY, EXPRESS_ONEZONE, GLACIER, GLACIER_IR, INLINE_BLOCK, INLINE_BLOCK_ENV, INTELLIGENT_TIERING,
|
|
LEGACY_LABEL_BEHAVIOR, MIN_PARITY_DRIVES, ONEZONE_IA, OPTIMIZE, OPTIMIZE_ENV, OUTPOSTS, RRS, RRS_ENV, SCHEME_PREFIX,
|
|
SNOW, STANDARD, STANDARD_ENV, STANDARD_IA, SUPPORTED_WRITE_CLASSES, StorageClass, UNSUPPORTED_WRITE_ERROR,
|
|
default_parity_count, effective_class, is_supported_write_class, lookup_config, lookup_config_for_pools,
|
|
parse_storage_class, validate_parity, validate_parity_inner,
|
|
};
|
|
}
|
|
|
|
pub use crate::config::{
|
|
RUSTFS_CONFIG_PREFIX, init, init_global_config_sys, set_global_storage_class, try_migrate_server_config,
|
|
};
|
|
}
|
|
|
|
pub mod data_usage {
|
|
#[cfg(feature = "test-util")]
|
|
pub use crate::data_usage::seed_bucket_usage_memory_for_test;
|
|
pub use crate::data_usage::{
|
|
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
|
|
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
|
|
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
|
|
load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size,
|
|
record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
|
|
record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
|
|
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
|
|
remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
|
|
store_data_usage_in_backend,
|
|
};
|
|
}
|
|
|
|
pub mod disk {
|
|
pub use crate::disk::disk_store::get_object_disk_read_timeout;
|
|
pub use crate::disk::local::ScanGuard;
|
|
pub use crate::disk::{
|
|
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
|
|
CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
|
|
DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
|
|
PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
|
|
STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
|
|
validate_batch_read_version_item_count,
|
|
};
|
|
pub use bytes::Bytes;
|
|
pub use endpoint::Endpoint;
|
|
pub use error::DiskError;
|
|
pub use error_reduce::is_all_buckets_not_found;
|
|
|
|
pub mod endpoint {
|
|
pub use crate::layout::endpoint::{Endpoint, EndpointType};
|
|
}
|
|
|
|
pub mod error {
|
|
pub use crate::disk::error::{DiskError, Error, FileAccessDeniedWithContext, Result};
|
|
}
|
|
|
|
pub mod error_reduce {
|
|
pub use crate::disk::error_reduce::{
|
|
BASE_IGNORED_ERRS, BUCKET_OP_IGNORED_ERRS, OBJECT_OP_IGNORED_ERRS, WriteQuorumFailureSummary,
|
|
build_write_quorum_failure_summary, count_errs, count_retryable_failures, is_all_buckets_not_found, is_ignored_err,
|
|
reduce_errs, reduce_quorum_errs, reduce_read_quorum_errs, reduce_write_quorum_errs,
|
|
};
|
|
}
|
|
}
|
|
|
|
pub mod error {
|
|
pub use crate::error::{
|
|
Error, Result, StorageError, classify_system_path_failure_reason, is_err_bucket_not_found, is_err_object_not_found,
|
|
is_err_version_not_found,
|
|
};
|
|
}
|
|
|
|
pub mod erasure {
|
|
pub use crate::erasure::coding::{
|
|
BitrotReader, BitrotSelfTestError, BitrotWriter, BitrotWriterWrapper, CustomWriter, Erasure, ErasureConstructionError,
|
|
ReedSolomonEncoder, bitrot_self_test, calc_shard_size, calc_shard_size_legacy,
|
|
};
|
|
}
|
|
|
|
pub mod event {
|
|
pub use crate::event::name::EventName;
|
|
pub use crate::services::event_notification::{EventArgs, register_event_dispatch_hook, send_event};
|
|
}
|
|
|
|
pub mod global {
|
|
pub use crate::runtime::global::{
|
|
set_global_endpoints, set_global_region, set_global_rustfs_port, set_object_store_resolver, shutdown_background_services,
|
|
update_erasure_type,
|
|
};
|
|
}
|
|
|
|
pub mod runtime {
|
|
pub use crate::runtime::instance::{InstanceContext, bootstrap_ctx};
|
|
pub use crate::runtime::sources::{
|
|
boot_time, bucket_monitor, deployment_id, endpoint_pools, expiry_state_handle, first_cluster_node_is_local,
|
|
global_lock_client, global_lock_clients, global_tier_config_mgr, local_disk_map_read, object_store_handle, region,
|
|
rustfs_port, setup_is_dist_erasure, setup_is_erasure, setup_is_erasure_sd, transition_state_handle,
|
|
};
|
|
}
|
|
|
|
pub mod layout {
|
|
pub use crate::layout::disks_layout::DisksLayout;
|
|
pub use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints, SetupType};
|
|
}
|
|
|
|
pub mod metrics {
|
|
pub use crate::services::metrics_realtime::{CollectMetricsOpts, MetricType, collect_local_metrics};
|
|
}
|
|
|
|
pub mod notification {
|
|
#[cfg(any(test, feature = "test-util"))]
|
|
pub use crate::services::notification_sys::rotate_cross_pool_fence_fleet_proof_for_test;
|
|
pub use crate::services::notification_sys::{
|
|
CrossPoolFenceFleetProofToken, NotificationPeerErr, NotificationSys, acquire_cross_pool_fence_fleet_proof,
|
|
cross_pool_fence_fleet_proof_matches, get_global_notification_sys, new_global_notification_sys,
|
|
start_remote_version_state_fleet_probe,
|
|
};
|
|
}
|
|
|
|
pub mod object {
|
|
pub use crate::object_api::{
|
|
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
|
|
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
|
|
ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
|
|
RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, StreamConsumer,
|
|
get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
|
|
register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
|
|
};
|
|
pub use crate::store::{
|
|
PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
|
|
SnapshotConsistencyError,
|
|
};
|
|
}
|
|
|
|
pub mod rebalance {
|
|
pub use crate::services::rebalance::{
|
|
DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo,
|
|
RebalanceMeta, RebalanceStats, RebalanceStopPropagationRecord, decode_rebalance_stop_propagation_record,
|
|
encode_rebalance_stop_propagation_record,
|
|
};
|
|
}
|
|
|
|
pub mod rio {
|
|
pub use crate::io_support::rio::{
|
|
DecryptReader, DynReader, EncryptReader, HardLimitReader, HashReader, ReadStream, Reader, WriteEncryption, WritePlan,
|
|
boxed_reader, compression_metadata_value, wrap_reader,
|
|
};
|
|
}
|
|
|
|
pub mod rpc {
|
|
pub use crate::cluster::rpc::{
|
|
AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS,
|
|
PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
|
|
ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
|
|
check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
|
|
gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
|
|
normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_put_file_capability,
|
|
sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
|
|
tonic_rpc_auth_failure_reason, verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature,
|
|
verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest,
|
|
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
|
|
};
|
|
}
|
|
|
|
pub mod set_disk {
|
|
pub use crate::set_disk::{DEFAULT_READ_BUFFER_SIZE, SetDisks, get_lock_acquire_timeout, is_valid_storage_class};
|
|
|
|
/// Return the canonical object-metadata identity used for read-quorum grouping.
|
|
pub fn file_info_quorum_hash(meta: &rustfs_filemeta::FileInfo) -> [u8; 32] {
|
|
crate::set_disk::SetDisks::file_info_quorum_hash(meta)
|
|
}
|
|
|
|
#[cfg(feature = "test-util")]
|
|
pub mod test_util {
|
|
pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
|
|
pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
|
|
}
|
|
}
|
|
|
|
pub mod store_list {
|
|
pub use crate::store::list_objects::{ListPathOptions, max_keys_plus_one};
|
|
}
|
|
|
|
pub mod storage {
|
|
pub use crate::core::pools::HealLifecycleExpiryContext;
|
|
pub use crate::store::HealWalkVersion;
|
|
pub use crate::store::{
|
|
ECStore, all_local_disk, all_local_disk_path, find_local_disk_by_ref, init_local_disks,
|
|
init_local_disks_with_instance_ctx, init_lock_clients, prewarm_local_disk_id_map,
|
|
prewarm_local_disk_id_map_with_instance_ctx,
|
|
};
|
|
}
|
|
|
|
pub mod tier {
|
|
#[allow(clippy::module_inception)]
|
|
pub mod tier {
|
|
pub use crate::services::tier::tier::{
|
|
ERR_TIER_BACKEND_IN_USE, ERR_TIER_BACKEND_NOT_EMPTY, ERR_TIER_INVALID_CONFIG, ERR_TIER_MISSING_CREDENTIALS,
|
|
ERR_TIER_TYPE_UNSUPPORTED, TIER_CONFIG_FILE, TIER_CONFIG_FORMAT, TIER_CONFIG_V1, TIER_CONFIG_VERSION, TierConfigMgr,
|
|
TierConfigUpdateError, is_err_config_not_found, try_migrate_tiering_config,
|
|
};
|
|
}
|
|
|
|
pub mod tier_admin {
|
|
pub use crate::services::tier::tier_admin::TierCreds;
|
|
}
|
|
|
|
pub mod tier_config {
|
|
pub use crate::services::tier::tier_config::{
|
|
ServicePrincipalAuth, TierAliyun, TierAzure, TierConfig, TierGCS, TierHuaweicloud, TierMinIO, TierR2, TierRustFS,
|
|
TierS3, TierTencent, TierType, TierWasabi,
|
|
};
|
|
}
|
|
|
|
pub mod tier_handlers {
|
|
pub use crate::services::tier::tier_handlers::{
|
|
ERR_TIER_ALREADY_EXISTS, ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS,
|
|
ERR_TIER_NAME_NOT_UPPERCASE, ERR_TIER_NOT_FOUND, ERR_TIER_PERM_ERR, ERR_TIER_RESERVED_NAME,
|
|
};
|
|
}
|
|
|
|
pub mod tier_mutation_peer {
|
|
pub use crate::services::tier::tier_mutation_peer::{
|
|
MAX_TIER_MUTATION_PEER_COMMIT_ETAG_SIZE, TierMutationPeerError, TierMutationPeerOutcome, TierMutationPeerResult,
|
|
TierMutationPeerState, handle_tier_mutation_peer_request,
|
|
};
|
|
}
|
|
|
|
pub mod warm_backend {
|
|
pub use crate::services::tier::warm_backend::{
|
|
WarmBackend, WarmBackendGetOpts, WarmBackendImpl, build_transition_put_options, check_warm_backend, new_warm_backend,
|
|
};
|
|
}
|
|
|
|
#[cfg(feature = "test-util")]
|
|
pub mod test_util {
|
|
pub use crate::services::tier::test_util::{
|
|
FaultConfig, MockStoredObject, MockWarmBackend, MockWarmOp, TransitionCleanupStoreBarrier, TransitionMeta,
|
|
assert_transition_meta_consistent, free_version_count, read_transition_meta, register_mock_tier,
|
|
register_mock_tier_backend, wait_for_free_version_absence,
|
|
};
|
|
}
|
|
}
|