rustfs/crates/filemeta

RustFS FileMeta - File Metadata Management

High-performance file metadata management for RustFS distributed object storage

📖 Documentation · 🐛 Bug Reports · 💬 Discussions

---

📖 Overview

RustFS FileMeta is the metadata management module for the RustFS distributed object storage system. It provides efficient storage, retrieval, and management of file metadata, supporting features like versioning, tagging, and extended attributes with high performance and reliability.

Note: This is a core submodule of RustFS that provides essential metadata management capabilities for the distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.

✨ Features

📝 Metadata Management

• File Information: Complete file metadata including size, timestamps, and checksums
• Object Versioning: Version-aware metadata management
• Extended Attributes: Custom metadata and tagging support
• Inline Metadata: Optimized storage for small metadata

🚀 Performance Features

• FlatBuffers Serialization: Zero-copy metadata serialization
• Efficient Storage: Optimized metadata storage layout
• Fast Lookups: High-performance metadata queries
• Batch Operations: Bulk metadata operations

🔧 Advanced Capabilities

• Schema Evolution: Forward and backward compatible metadata schemas
• Compression: Metadata compression for space efficiency
• Validation: Metadata integrity verification
• Migration: Seamless metadata format migration

📦 Installation

Add this to your Cargo.toml:

[dependencies]
rustfs-filemeta = "0.1.0"

🔧 Usage

Basic Metadata Operations

use rustfs_filemeta::{FileInfo, XLMeta};
use std::collections::HashMap;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create file metadata
    let mut file_info = FileInfo::new();
    file_info.name = "example.txt".to_string();
    file_info.size = 1024;
    file_info.mod_time = chrono::Utc::now();

    // Add custom metadata
    let mut user_defined = HashMap::new();
    user_defined.insert("author".to_string(), "john@example.com".to_string());
    user_defined.insert("department".to_string(), "engineering".to_string());
    file_info.user_defined = user_defined;

    // Create XL metadata
    let xl_meta = XLMeta::new(file_info);

    // Serialize metadata
    let serialized = xl_meta.serialize()?;

    // Deserialize metadata
    let deserialized = XLMeta::deserialize(&serialized)?;

    println!("File: {}, Size: {}", deserialized.file_info.name, deserialized.file_info.size);

    Ok(())
}

Advanced Metadata Management

use rustfs_filemeta::{XLMeta, FileInfo, VersionInfo};

async fn advanced_metadata_example() -> Result<(), Box<dyn std::error::Error>> {
    // Create versioned metadata
    let mut xl_meta = XLMeta::new(FileInfo::default());

    // Set version information
    xl_meta.set_version_info(VersionInfo {
        version_id: "v1.0.0".to_string(),
        is_latest: true,
        delete_marker: false,
        restore_ongoing: false,
    });

    // Add checksums
    xl_meta.add_checksum("md5", "d41d8cd98f00b204e9800998ecf8427e");
    xl_meta.add_checksum("sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855");

    // Set object tags
    xl_meta.set_tags(vec![
        ("Environment".to_string(), "Production".to_string()),
        ("Owner".to_string(), "DataTeam".to_string()),
    ]);

    // Set retention information
    xl_meta.set_retention_info(
        chrono::Utc::now() + chrono::Duration::days(365),
        "GOVERNANCE".to_string(),
    );

    // Validate metadata
    xl_meta.validate()?;

    Ok(())
}

Inline Metadata Operations

use rustfs_filemeta::{InlineMetadata, MetadataSize};

fn inline_metadata_example() -> Result<(), Box<dyn std::error::Error>> {
    // Create inline metadata for small files
    let mut inline_meta = InlineMetadata::new();

    // Set basic properties
    inline_meta.set_content_type("text/plain");
    inline_meta.set_content_encoding("gzip");
    inline_meta.set_cache_control("max-age=3600");

    // Add custom headers
    inline_meta.add_header("x-custom-field", "custom-value");
    inline_meta.add_header("x-app-version", "1.2.3");

    // Check if metadata fits inline storage
    if inline_meta.size() <= MetadataSize::INLINE_THRESHOLD {
        println!("Metadata can be stored inline");
    } else {
        println!("Metadata requires separate storage");
    }

    // Serialize for storage
    let bytes = inline_meta.to_bytes()?;

    // Deserialize from storage
    let restored = InlineMetadata::from_bytes(&bytes)?;

    Ok(())
}

🏗️ Architecture

Metadata Storage Layout

FileMeta Architecture:
┌─────────────────────────────────────────────────────────────┐
│                    Metadata API Layer                       │
├─────────────────────────────────────────────────────────────┤
│    XL Metadata    │   Inline Metadata   │   Version Info    │
├─────────────────────────────────────────────────────────────┤
│              FlatBuffers Serialization                      │
├─────────────────────────────────────────────────────────────┤
│    Compression    │    Validation       │    Migration      │
├─────────────────────────────────────────────────────────────┤
│              Storage Backend Integration                     │
└─────────────────────────────────────────────────────────────┘

Metadata Types

Type	Use Case	Storage	Performance
XLMeta	Large objects with rich metadata	Separate file	High durability
InlineMeta	Small objects with minimal metadata	Embedded	Fastest access
VersionMeta	Object versioning information	Version-specific	Version-aware

🧪 Testing

Run the test suite:

# Run all tests
cargo test

# Run serialization benchmarks
cargo bench

# Test metadata validation
cargo test validation

# Test schema migration
cargo test migration

🚀 Performance

FileMeta is optimized for high-performance metadata operations:

• Serialization: Zero-copy FlatBuffers serialization
• Storage: Compact binary format reduces I/O
• Caching: Intelligent metadata caching
• Batch Operations: Efficient bulk metadata processing

📋 Requirements

• Rust: 1.70.0 or later
• Platforms: Linux, macOS, Windows
• Memory: Minimal memory footprint
• Storage: Compatible with RustFS storage backend

🌍 Related Projects

This module is part of the RustFS ecosystem:

• RustFS Main - Core distributed storage system
• RustFS ECStore - Erasure coding storage engine
• RustFS Utils - Utility functions
• RustFS Proto - Protocol definitions

📚 Documentation

For comprehensive documentation, visit:

• RustFS Documentation
• FileMeta API Reference

🔗 Links

• Documentation - Complete RustFS manual
• Changelog - Release notes and updates
• GitHub Discussions - Community support

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

📄 License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

---

RustFS is a trademark of RustFS, Inc.
All other trademarks are the property of their respective owners.

Made with ❤️ by the RustFS Team