Merge pull request 'MyFSIO v0.4.2 Release' (#35) from next into main

Reviewed-on: #35

.dockerignore

@@ -11,7 +11,3 @@ htmlcov
 logs
 data
 tmp
-tests
-myfsio_core/target
-Dockerfile
-.dockerignore

.gitignore

@@ -27,11 +27,8 @@ dist/
 .eggs/
 
 # Rust / maturin build artifacts
-python/myfsio_core/target/
-python/myfsio_core/Cargo.lock
-
-# Rust engine build artifacts
-rust/myfsio-engine/target/
+myfsio_core/target/
+myfsio_core/Cargo.lock
 
 # Local runtime artifacts
 logs/

Dockerfile

@@ -1,9 +1,9 @@
-FROM python:3.14.3-slim AS builder
+FROM python:3.14.3-slim
 
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1
 
-WORKDIR /build
+WORKDIR /app
 
 RUN apt-get update \
     && apt-get install -y --no-install-recommends build-essential curl \
@@ -12,34 +12,23 @@ RUN apt-get update \
 
 ENV PATH="/root/.cargo/bin:${PATH}"
 
-RUN pip install --no-cache-dir maturin
-
-COPY myfsio_core ./myfsio_core
-RUN cd myfsio_core \
-    && maturin build --release --out /wheels
-
-
-FROM python:3.14.3-slim
-
-ENV PYTHONDONTWRITEBYTECODE=1 \
-    PYTHONUNBUFFERED=1
-
-WORKDIR /app
-
 COPY requirements.txt ./
 RUN pip install --no-cache-dir -r requirements.txt
 
-COPY --from=builder /wheels/*.whl /tmp/
-RUN pip install --no-cache-dir /tmp/*.whl && rm /tmp/*.whl
+COPY . .
 
-COPY app ./app
-COPY templates ./templates
-COPY static ./static
-COPY run.py ./
-COPY docker-entrypoint.sh ./
+RUN pip install --no-cache-dir maturin \
+    && cd myfsio_core \
+    && maturin build --release \
+    && pip install target/wheels/*.whl \
+    && cd .. \
+    && rm -rf myfsio_core/target \
+    && pip uninstall -y maturin \
+    && rustup self uninstall -y
 
-RUN chmod +x docker-entrypoint.sh \
-    && mkdir -p /app/data \
+RUN chmod +x docker-entrypoint.sh
+
+RUN mkdir -p /app/data \
     && useradd -m -u 1000 myfsio \ 
     && chown -R myfsio:myfsio /app
 

README.md

@@ -1,212 +1,255 @@
 # MyFSIO
 
-MyFSIO is an S3-compatible object storage server with a Rust runtime and a filesystem-backed storage engine. The active server lives under `rust/myfsio-engine` and serves both the S3 API and the built-in web UI from a single process.
-
-The repository still contains a `python/` tree, but you do not need Python to run the current server.
+A lightweight, S3-compatible object storage system built with Flask. MyFSIO implements core AWS S3 REST API operations with filesystem-backed storage, making it ideal for local development, testing, and self-hosted storage scenarios.
 
 ## Features
 
-- S3-compatible REST API with Signature Version 4 authentication
-- Browser UI for buckets, objects, IAM users, policies, replication, metrics, and site administration
-- Filesystem-backed storage rooted at `data/`
-- Bucket versioning, multipart uploads, presigned URLs, CORS, object and bucket tagging
-- Server-side encryption and built-in KMS support
-- Optional background services for lifecycle, garbage collection, integrity scanning, operation metrics, and system metrics history
-- Replication, site sync, and static website hosting support
+**Core Storage**
+- S3-compatible REST API with AWS Signature Version 4 authentication
+- Bucket and object CRUD operations
+- Object versioning with version history
+- Multipart uploads for large files
+- Presigned URLs (1 second to 7 days validity)
 
-## Runtime Model
+**Security & Access Control**
+- IAM users with access key management and rotation
+- Bucket policies (AWS Policy Version 2012-10-17)
+- Server-side encryption (SSE-S3 and SSE-KMS)
+- Built-in Key Management Service (KMS)
+- Rate limiting per endpoint
 
-MyFSIO now runs as one Rust process:
+**Advanced Features**
+- Cross-bucket replication to remote S3-compatible endpoints
+- Hot-reload for bucket policies (no restart required)
+- CORS configuration per bucket
 
-- API listener on `HOST` + `PORT` (default `127.0.0.1:5000`)
-- UI listener on `HOST` + `UI_PORT` (default `127.0.0.1:5100`)
-- Shared state for storage, IAM, policies, sessions, metrics, and background workers
+**Management UI**
+- Web console for bucket and object management
+- IAM dashboard for user administration
+- Inline JSON policy editor with presets
+- Object browser with folder navigation and bulk operations
+- Dark mode support
 
-If you want API-only mode, set `UI_ENABLED=false`. There is no separate "UI-only" runtime anymore.
+## Architecture
+
+```
++------------------+         +------------------+
+|   API Server     |         |   UI Server      |
+|   (port 5000)    |         |   (port 5100)    |
+|                  |         |                  |
+|  - S3 REST API   |<------->|  - Web Console   |
+|  - SigV4 Auth    |         |  - IAM Dashboard |
+|  - Presign URLs  |         |  - Bucket Editor |
++--------+---------+         +------------------+
+         |
+         v
++------------------+         +------------------+
+| Object Storage   |         | System Metadata  |
+| (filesystem)     |         | (.myfsio.sys/)   |
+|                  |         |                  |
+| data/<bucket>/   |         | - IAM config     |
+|   <objects>      |         | - Bucket policies|
+|                  |         | - Encryption keys|
++------------------+         +------------------+
+```
 
 ## Quick Start
 
-From the repository root:
-
 ```bash
-cd rust/myfsio-engine
-cargo run -p myfsio-server --
+# Clone and setup
+git clone https://gitea.jzwsite.com/kqjy/MyFSIO
+cd s3
+python -m venv .venv
+
+# Activate virtual environment
+# Windows PowerShell:
+.\.venv\Scripts\Activate.ps1
+# Windows CMD:
+.venv\Scripts\activate.bat
+# Linux/macOS:
+source .venv/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+
+# (Optional) Build Rust native extension for better performance
+# Requires Rust toolchain: https://rustup.rs
+pip install maturin
+cd myfsio_core && maturin develop --release && cd ..
+
+# Start both servers
+python run.py
+
+# Or start individually
+python run.py --mode api   # API only (port 5000)
+python run.py --mode ui    # UI only (port 5100)
 ```
 
-Useful URLs:
+**Credentials:** Generated automatically on first run and printed to the console. If missed, check the IAM config file at `<STORAGE_ROOT>/.myfsio.sys/config/iam.json`.
 
-- UI: `http://127.0.0.1:5100/ui`
-- API: `http://127.0.0.1:5000/`
-- Health: `http://127.0.0.1:5000/myfsio/health`
-
-On first boot, MyFSIO creates `data/.myfsio.sys/config/iam.json` and prints the generated admin access key and secret key to the console.
-
-### Common CLI commands
-
-```bash
-# Show resolved configuration
-cargo run -p myfsio-server -- --show-config
-
-# Validate configuration and exit non-zero on critical issues
-cargo run -p myfsio-server -- --check-config
-
-# Reset admin credentials
-cargo run -p myfsio-server -- --reset-cred
-
-# API only
-UI_ENABLED=false cargo run -p myfsio-server --
-```
-
-## Building a Binary
-
-```bash
-cd rust/myfsio-engine
-cargo build --release -p myfsio-server
-```
-
-Binary locations:
-
-- Linux/macOS: `rust/myfsio-engine/target/release/myfsio-server`
-- Windows: `rust/myfsio-engine/target/release/myfsio-server.exe`
-
-Run the built binary directly:
-
-```bash
-./target/release/myfsio-server
-```
+- **Web Console:** http://127.0.0.1:5100/ui
+- **API Endpoint:** http://127.0.0.1:5000
 
 ## Configuration
 
-The server reads environment variables from the process environment and also loads, when present:
-
-- `/opt/myfsio/myfsio.env`
-- `.env`
-- `myfsio.env`
-
-Core settings:
-
 | Variable | Default | Description |
-| --- | --- | --- |
-| `HOST` | `127.0.0.1` | Bind address for API and UI listeners |
-| `PORT` | `5000` | API port |
-| `UI_PORT` | `5100` | UI port |
-| `UI_ENABLED` | `true` | Disable to run API-only |
-| `STORAGE_ROOT` | `./data` | Root directory for buckets and system metadata |
-| `IAM_CONFIG` | `<STORAGE_ROOT>/.myfsio.sys/config/iam.json` | IAM config path |
-| `API_BASE_URL` | unset | Public API base used by the UI and presigned URL generation |
-| `AWS_REGION` | `us-east-1` | Region used in SigV4 scope |
-| `SIGV4_TIMESTAMP_TOLERANCE_SECONDS` | `900` | Allowed request time skew |
-| `PRESIGNED_URL_MIN_EXPIRY_SECONDS` | `1` | Minimum presigned URL expiry |
-| `PRESIGNED_URL_MAX_EXPIRY_SECONDS` | `604800` | Maximum presigned URL expiry |
-| `SECRET_KEY` | loaded from `.myfsio.sys/config/.secret` if present | Session signing key and IAM-at-rest encryption key |
-| `ADMIN_ACCESS_KEY` | unset | Optional first-run or reset access key |
-| `ADMIN_SECRET_KEY` | unset | Optional first-run or reset secret key |
-
-Feature toggles:
-
-| Variable | Default |
-| --- | --- |
-| `ENCRYPTION_ENABLED` | `false` |
-| `KMS_ENABLED` | `false` |
-| `GC_ENABLED` | `false` |
-| `INTEGRITY_ENABLED` | `false` |
-| `LIFECYCLE_ENABLED` | `false` |
-| `METRICS_HISTORY_ENABLED` | `false` |
-| `OPERATION_METRICS_ENABLED` | `false` |
-| `WEBSITE_HOSTING_ENABLED` | `false` |
-| `SITE_SYNC_ENABLED` | `false` |
-
-Metrics and replication tuning:
-
-| Variable | Default |
-| --- | --- |
-| `OPERATION_METRICS_INTERVAL_MINUTES` | `5` |
-| `OPERATION_METRICS_RETENTION_HOURS` | `24` |
-| `METRICS_HISTORY_INTERVAL_MINUTES` | `5` |
-| `METRICS_HISTORY_RETENTION_HOURS` | `24` |
-| `REPLICATION_CONNECT_TIMEOUT_SECONDS` | `5` |
-| `REPLICATION_READ_TIMEOUT_SECONDS` | `30` |
-| `REPLICATION_MAX_RETRIES` | `2` |
-| `REPLICATION_STREAMING_THRESHOLD_BYTES` | `10485760` |
-| `REPLICATION_MAX_FAILURES_PER_BUCKET` | `50` |
-| `SITE_SYNC_INTERVAL_SECONDS` | `60` |
-| `SITE_SYNC_BATCH_SIZE` | `100` |
-| `SITE_SYNC_CONNECT_TIMEOUT_SECONDS` | `10` |
-| `SITE_SYNC_READ_TIMEOUT_SECONDS` | `120` |
-| `SITE_SYNC_MAX_RETRIES` | `2` |
-| `SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS` | `1.0` |
-
-UI asset overrides:
-
-| Variable | Default |
-| --- | --- |
-| `TEMPLATES_DIR` | built-in crate templates directory |
-| `STATIC_DIR` | built-in crate static directory |
-
-See [docs.md](./docs.md) for the full Rust-side operations guide.
+|----------|---------|-------------|
+| `STORAGE_ROOT` | `./data` | Filesystem root for bucket storage |
+| `IAM_CONFIG` | `.myfsio.sys/config/iam.json` | IAM user and policy store |
+| `BUCKET_POLICY_PATH` | `.myfsio.sys/config/bucket_policies.json` | Bucket policy store |
+| `API_BASE_URL` | `http://127.0.0.1:5000` | API endpoint for UI calls |
+| `MAX_UPLOAD_SIZE` | `1073741824` | Maximum upload size in bytes (1 GB) |
+| `MULTIPART_MIN_PART_SIZE` | `5242880` | Minimum multipart part size (5 MB) |
+| `UI_PAGE_SIZE` | `100` | Default page size for listings |
+| `SECRET_KEY` | `dev-secret-key` | Flask session secret |
+| `AWS_REGION` | `us-east-1` | Region for SigV4 signing |
+| `AWS_SERVICE` | `s3` | Service name for SigV4 signing |
+| `ENCRYPTION_ENABLED` | `false` | Enable server-side encryption |
+| `KMS_ENABLED` | `false` | Enable Key Management Service |
+| `LOG_LEVEL` | `INFO` | Logging verbosity |
+| `SIGV4_TIMESTAMP_TOLERANCE_SECONDS` | `900` | Max time skew for SigV4 requests |
+| `PRESIGNED_URL_MAX_EXPIRY_SECONDS` | `604800` | Max presigned URL expiry (7 days) |
+| `REPLICATION_CONNECT_TIMEOUT_SECONDS` | `5` | Replication connection timeout |
+| `SITE_SYNC_ENABLED` | `false` | Enable bi-directional site sync |
+| `OBJECT_TAG_LIMIT` | `50` | Maximum tags per object |
 
 ## Data Layout
 
-```text
-data/
-  <bucket>/
-  .myfsio.sys/
-    config/
-      iam.json
-      bucket_policies.json
-      connections.json
-      operation_metrics.json
-      metrics_history.json
-    buckets/<bucket>/
-      meta/
-      versions/
-    multipart/
-    keys/
 ```
+data/
+├── <bucket>/                    # User buckets with objects
+└── .myfsio.sys/                 # System metadata
+    ├── config/
+    │   ├── iam.json             # IAM users and policies
+    │   ├── bucket_policies.json # Bucket policies
+    │   ├── replication_rules.json
+    │   └── connections.json     # Remote S3 connections
+    ├── buckets/<bucket>/
+    │   ├── meta/                # Object metadata (.meta.json)
+    │   ├── versions/            # Archived object versions
+    │   └── .bucket.json         # Bucket config (versioning, CORS)
+    ├── multipart/               # Active multipart uploads
+    └── keys/                    # Encryption keys (SSE-S3/KMS)
+```
+
+## API Reference
+
+All endpoints require AWS Signature Version 4 authentication unless using presigned URLs or public bucket policies.
+
+### Bucket Operations
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| `GET` | `/` | List all buckets |
+| `PUT` | `/<bucket>` | Create bucket |
+| `DELETE` | `/<bucket>` | Delete bucket (must be empty) |
+| `HEAD` | `/<bucket>` | Check bucket exists |
+
+### Object Operations
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| `GET` | `/<bucket>` | List objects (supports `list-type=2`) |
+| `PUT` | `/<bucket>/<key>` | Upload object |
+| `GET` | `/<bucket>/<key>` | Download object |
+| `DELETE` | `/<bucket>/<key>` | Delete object |
+| `HEAD` | `/<bucket>/<key>` | Get object metadata |
+| `POST` | `/<bucket>/<key>?uploads` | Initiate multipart upload |
+| `PUT` | `/<bucket>/<key>?partNumber=N&uploadId=X` | Upload part |
+| `POST` | `/<bucket>/<key>?uploadId=X` | Complete multipart upload |
+| `DELETE` | `/<bucket>/<key>?uploadId=X` | Abort multipart upload |
+
+### Bucket Policies (S3-compatible)
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| `GET` | `/<bucket>?policy` | Get bucket policy |
+| `PUT` | `/<bucket>?policy` | Set bucket policy |
+| `DELETE` | `/<bucket>?policy` | Delete bucket policy |
+
+### Versioning
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| `GET` | `/<bucket>/<key>?versionId=X` | Get specific version |
+| `DELETE` | `/<bucket>/<key>?versionId=X` | Delete specific version |
+| `GET` | `/<bucket>?versions` | List object versions |
+
+### Health Check
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| `GET` | `/myfsio/health` | Health check endpoint |
+
+## IAM & Access Control
+
+### Users and Access Keys
+
+On first run, MyFSIO creates a default admin user (`localadmin`/`localadmin`). Use the IAM dashboard to:
+
+- Create and delete users
+- Generate and rotate access keys
+- Attach inline policies to users
+- Control IAM management permissions
+
+### Bucket Policies
+
+Bucket policies follow AWS policy grammar (Version `2012-10-17`) with support for:
+
+- Principal-based access (`*` for anonymous, specific users)
+- Action-based permissions (`s3:GetObject`, `s3:PutObject`, etc.)
+- Resource patterns (`arn:aws:s3:::bucket/*`)
+- Condition keys
+
+**Policy Presets:**
+- **Public:** Grants anonymous read access (`s3:GetObject`, `s3:ListBucket`)
+- **Private:** Removes bucket policy (IAM-only access)
+- **Custom:** Manual policy editing with draft preservation
+
+Policies hot-reload when the JSON file changes.
+
+## Server-Side Encryption
+
+MyFSIO supports two encryption modes:
+
+- **SSE-S3:** Server-managed keys with automatic key rotation
+- **SSE-KMS:** Customer-managed keys via built-in KMS
+
+Enable encryption with:
+```bash
+ENCRYPTION_ENABLED=true python run.py
+```
+
+## Cross-Bucket Replication
+
+Replicate objects to remote S3-compatible endpoints:
+
+1. Configure remote connections in the UI
+2. Create replication rules specifying source/destination
+3. Objects are automatically replicated on upload
 
 ## Docker
 
-Build the Rust image from the `rust/` directory:
-
 ```bash
-docker build -t myfsio ./rust
-docker run --rm -p 5000:5000 -p 5100:5100 -v "${PWD}/data:/app/data" myfsio
+docker build -t myfsio .
+docker run -p 5000:5000 -p 5100:5100 -v ./data:/app/data myfsio
 ```
 
-If the instance sits behind a reverse proxy, set `API_BASE_URL` to the public S3 endpoint.
-
-## Linux Installation
-
-The repository includes `scripts/install.sh` for systemd-style Linux installs. Build the Rust binary first, then pass it to the installer:
-
-```bash
-cd rust/myfsio-engine
-cargo build --release -p myfsio-server
-
-cd ../..
-sudo ./scripts/install.sh --binary ./rust/myfsio-engine/target/release/myfsio-server
-```
-
-The installer copies the binary into `/opt/myfsio/myfsio`, writes `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` unit.
-
 ## Testing
 
-Run the Rust test suite from the workspace:
-
 ```bash
-cd rust/myfsio-engine
-cargo test
+# Run all tests
+pytest tests/ -v
+
+# Run specific test file
+pytest tests/test_api.py -v
+
+# Run with coverage
+pytest tests/ --cov=app --cov-report=html
 ```
 
-## Health Check
+## References
 
-`GET /myfsio/health` returns:
-
-```json
-{
-  "status": "ok",
-  "version": "0.5.0"
-}
-```
-
-The `version` field comes from the Rust crate version in `rust/myfsio-engine/crates/myfsio-server/Cargo.toml`.
+- [Amazon S3 Documentation](https://docs.aws.amazon.com/s3/)
+- [AWS Signature Version 4](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)
+- [S3 Bucket Policy Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html)

app/ui.py

@@ -225,10 +225,10 @@ def _policy_allows_public_read(policy: dict[str, Any]) -> bool:
 
 def _bucket_access_descriptor(policy: dict[str, Any] | None) -> tuple[str, str]:
     if not policy:
-        return ("IAM only", "bg-secondary-subtle text-secondary-emphasis")
+        return ("IAM only", "text-bg-secondary")
     if _policy_allows_public_read(policy):
-        return ("Public read", "bg-warning-subtle text-warning-emphasis")
-    return ("Custom policy", "bg-info-subtle text-info-emphasis")
+        return ("Public read", "text-bg-warning")
+    return ("Custom policy", "text-bg-info")
 
 
 def _current_principal():

app/version.py

@@ -1,6 +1,6 @@
 from __future__ import annotations
 
-APP_VERSION = "0.4.3"
+APP_VERSION = "0.4.2"
 
 
 def get_version() -> str:

docker-entrypoint.sh

@@ -0,0 +1,5 @@
+#!/bin/sh
+set -e
+
+# Run both services using the python runner in production mode
+exec python run.py --prod

myfsio_core/src/storage.rs

@@ -125,7 +125,7 @@ pub fn delete_index_entry(py: Python<'_>, path: &str, entry_name: &str) -> PyRes
         fs::write(&path_owned, serialized)
             .map_err(|e| PyIOError::new_err(format!("Failed to write index: {}", e)))?;
 
-        Ok(true)
+        Ok(false)
     })
 }
 

python/docker-entrypoint.sh

@@ -1,4 +0,0 @@
-#!/bin/sh
-set -e
-
-exec python run.py --prod

rust/.dockerignore

@@ -1,9 +0,0 @@
-.git
-.gitignore
-logs
-data
-tmp
-myfsio-engine/target
-myfsio-engine/tests
-Dockerfile
-.dockerignore

rust/Dockerfile

@@ -1,45 +0,0 @@
-FROM rust:1-slim-bookworm AS builder
-
-WORKDIR /build
-
-RUN apt-get update \
-    && apt-get install -y --no-install-recommends pkg-config libssl-dev \
-    && rm -rf /var/lib/apt/lists/*
-
-COPY myfsio-engine ./myfsio-engine
-
-RUN cd myfsio-engine \
-    && cargo build --release --bin myfsio-server \
-    && strip target/release/myfsio-server
-
-
-FROM debian:bookworm-slim
-
-WORKDIR /app
-
-RUN apt-get update \
-    && apt-get install -y --no-install-recommends ca-certificates curl \
-    && rm -rf /var/lib/apt/lists/* \
-    && mkdir -p /app/data \
-    && useradd -m -u 1000 myfsio \
-    && chown -R myfsio:myfsio /app
-
-COPY --from=builder /build/myfsio-engine/target/release/myfsio-server /usr/local/bin/myfsio-server
-COPY --from=builder /build/myfsio-engine/templates /app/templates
-COPY docker-entrypoint.sh /app/docker-entrypoint.sh
-
-RUN chmod +x /app/docker-entrypoint.sh \
-    && chown -R myfsio:myfsio /app
-
-USER myfsio
-
-EXPOSE 5000
-ENV HOST=0.0.0.0 \
-    PORT=5000 \
-    STORAGE_ROOT=/app/data \
-    RUST_LOG=info
-
-HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
-    CMD curl -fsS "http://localhost:${PORT}/myfsio/health" || exit 1
-
-CMD ["/app/docker-entrypoint.sh"]

rust/docker-entrypoint.sh

@@ -1,4 +0,0 @@
-#!/bin/sh
-set -e
-
-exec /usr/local/bin/myfsio-server

rust/myfsio-engine/Cargo.toml

@@ -1,61 +0,0 @@
-[workspace]
-resolver = "2"
-members = [
-    "crates/myfsio-common",
-    "crates/myfsio-auth",
-    "crates/myfsio-crypto",
-    "crates/myfsio-storage",
-    "crates/myfsio-xml",
-    "crates/myfsio-server",
-]
-
-[workspace.package]
-version = "0.4.3"
-edition = "2021"
-
-[workspace.dependencies]
-tokio = { version = "1", features = ["full"] }
-axum = { version = "0.8" }
-tower = { version = "0.5" }
-tower-http = { version = "0.6", features = ["cors", "trace", "fs", "compression-gzip"] }
-hyper = { version = "1" }
-bytes = "1"
-serde = { version = "1", features = ["derive"] }
-serde_json = "1"
-quick-xml = { version = "0.37", features = ["serialize"] }
-hmac = "0.12"
-sha2 = "0.10"
-md-5 = "0.10"
-hex = "0.4"
-aes = "0.8"
-aes-gcm = "0.10"
-cbc = { version = "0.1", features = ["alloc"] }
-hkdf = "0.12"
-uuid = { version = "1", features = ["v4"] }
-parking_lot = "0.12"
-lru = "0.14"
-percent-encoding = "2"
-regex = "1"
-unicode-normalization = "0.1"
-tracing = "0.1"
-tracing-subscriber = "0.3"
-thiserror = "2"
-chrono = { version = "0.4", features = ["serde"] }
-base64 = "0.22"
-tokio-util = { version = "0.7", features = ["io"] }
-futures = "0.3"
-dashmap = "6"
-crc32fast = "1"
-duckdb = { version = "1", features = ["bundled"] }
-reqwest = { version = "0.12", default-features = false, features = ["stream", "rustls-tls", "json"] }
-aws-sdk-s3 = { version = "1", features = ["behavior-version-latest", "rt-tokio"] }
-aws-config = { version = "1", features = ["behavior-version-latest"] }
-aws-credential-types = "1"
-aws-smithy-runtime-api = "1"
-aws-smithy-types = "1"
-async-trait = "0.1"
-tera = "1"
-cookie = "0.18"
-subtle = "2"
-clap = { version = "4", features = ["derive"] }
-dotenvy = "0.15"

rust/myfsio-engine/crates/myfsio-auth/Cargo.toml

@@ -1,27 +0,0 @@
-[package]
-name = "myfsio-auth"
-version.workspace = true
-edition.workspace = true
-
-[dependencies]
-myfsio-common = { path = "../myfsio-common" }
-hmac = { workspace = true }
-sha2 = { workspace = true }
-hex = { workspace = true }
-aes = { workspace = true }
-cbc = { workspace = true }
-base64 = { workspace = true }
-pbkdf2 = "0.12"
-rand = "0.8"
-lru = { workspace = true }
-parking_lot = { workspace = true }
-percent-encoding = { workspace = true }
-serde = { workspace = true }
-serde_json = { workspace = true }
-thiserror = { workspace = true }
-chrono = { workspace = true }
-tracing = { workspace = true }
-uuid = { workspace = true }
-
-[dev-dependencies]
-tempfile = "3"

rust/myfsio-engine/crates/myfsio-auth/src/fernet.rs

@@ -1,118 +0,0 @@
-use aes::cipher::{block_padding::Pkcs7, BlockDecryptMut, BlockEncryptMut, KeyIvInit};
-use base64::{engine::general_purpose::URL_SAFE, Engine};
-use hmac::{Hmac, Mac};
-use rand::RngCore;
-use sha2::Sha256;
-
-type Aes128CbcDec = cbc::Decryptor<aes::Aes128>;
-type Aes128CbcEnc = cbc::Encryptor<aes::Aes128>;
-type HmacSha256 = Hmac<Sha256>;
-
-pub fn derive_fernet_key(secret: &str) -> String {
-    let mut derived = [0u8; 32];
-    pbkdf2::pbkdf2_hmac::<Sha256>(
-        secret.as_bytes(),
-        b"myfsio-iam-encryption",
-        100_000,
-        &mut derived,
-    );
-    URL_SAFE.encode(derived)
-}
-
-pub fn decrypt(key_b64: &str, token: &str) -> Result<Vec<u8>, &'static str> {
-    let key_bytes = URL_SAFE
-        .decode(key_b64)
-        .map_err(|_| "invalid fernet key base64")?;
-    if key_bytes.len() != 32 {
-        return Err("fernet key must be 32 bytes");
-    }
-
-    let signing_key = &key_bytes[..16];
-    let encryption_key = &key_bytes[16..];
-
-    let token_bytes = URL_SAFE
-        .decode(token)
-        .map_err(|_| "invalid fernet token base64")?;
-
-    if token_bytes.len() < 57 {
-        return Err("fernet token too short");
-    }
-
-    if token_bytes[0] != 0x80 {
-        return Err("invalid fernet version");
-    }
-
-    let hmac_offset = token_bytes.len() - 32;
-    let payload = &token_bytes[..hmac_offset];
-    let expected_hmac = &token_bytes[hmac_offset..];
-
-    let mut mac = HmacSha256::new_from_slice(signing_key).map_err(|_| "hmac key error")?;
-    mac.update(payload);
-    mac.verify_slice(expected_hmac)
-        .map_err(|_| "HMAC verification failed")?;
-
-    let iv = &token_bytes[9..25];
-    let ciphertext = &token_bytes[25..hmac_offset];
-
-    let plaintext = Aes128CbcDec::new(encryption_key.into(), iv.into())
-        .decrypt_padded_vec_mut::<Pkcs7>(ciphertext)
-        .map_err(|_| "AES-CBC decryption failed")?;
-
-    Ok(plaintext)
-}
-
-pub fn encrypt(key_b64: &str, plaintext: &[u8]) -> Result<String, &'static str> {
-    let key_bytes = URL_SAFE
-        .decode(key_b64)
-        .map_err(|_| "invalid fernet key base64")?;
-    if key_bytes.len() != 32 {
-        return Err("fernet key must be 32 bytes");
-    }
-
-    let signing_key = &key_bytes[..16];
-    let encryption_key = &key_bytes[16..];
-
-    let mut iv = [0u8; 16];
-    rand::thread_rng().fill_bytes(&mut iv);
-
-    let timestamp = std::time::SystemTime::now()
-        .duration_since(std::time::UNIX_EPOCH)
-        .map_err(|_| "system time error")?
-        .as_secs();
-
-    let ciphertext = Aes128CbcEnc::new(encryption_key.into(), (&iv).into())
-        .encrypt_padded_vec_mut::<Pkcs7>(plaintext);
-
-    let mut payload = Vec::with_capacity(1 + 8 + 16 + ciphertext.len());
-    payload.push(0x80);
-    payload.extend_from_slice(&timestamp.to_be_bytes());
-    payload.extend_from_slice(&iv);
-    payload.extend_from_slice(&ciphertext);
-
-    let mut mac = HmacSha256::new_from_slice(signing_key).map_err(|_| "hmac key error")?;
-    mac.update(&payload);
-    let tag = mac.finalize().into_bytes();
-
-    let mut token_bytes = payload;
-    token_bytes.extend_from_slice(&tag);
-    Ok(URL_SAFE.encode(&token_bytes))
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_derive_fernet_key_format() {
-        let key = derive_fernet_key("test-secret");
-        let decoded = URL_SAFE.decode(&key).unwrap();
-        assert_eq!(decoded.len(), 32);
-    }
-
-    #[test]
-    fn test_roundtrip_with_python_compat() {
-        let key = derive_fernet_key("dev-secret-key");
-        let decoded = URL_SAFE.decode(&key).unwrap();
-        assert_eq!(decoded.len(), 32);
-    }
-}

rust/myfsio-engine/crates/myfsio-auth/src/lib.rs

@@ -1,4 +0,0 @@
-mod fernet;
-pub mod iam;
-pub mod principal;
-pub mod sigv4;

rust/myfsio-engine/crates/myfsio-auth/src/principal.rs

@@ -1 +0,0 @@
-pub use myfsio_common::types::Principal;

rust/myfsio-engine/crates/myfsio-auth/src/sigv4.rs

@@ -1,287 +0,0 @@
-use hmac::{Hmac, Mac};
-use lru::LruCache;
-use parking_lot::Mutex;
-use percent_encoding::{percent_encode, AsciiSet, NON_ALPHANUMERIC};
-use sha2::{Digest, Sha256};
-use std::num::NonZeroUsize;
-use std::sync::LazyLock;
-use std::time::Instant;
-
-type HmacSha256 = Hmac<Sha256>;
-
-struct CacheEntry {
-    key: Vec<u8>,
-    created: Instant,
-}
-
-static SIGNING_KEY_CACHE: LazyLock<Mutex<LruCache<(String, String, String, String), CacheEntry>>> =
-    LazyLock::new(|| Mutex::new(LruCache::new(NonZeroUsize::new(256).unwrap())));
-
-const CACHE_TTL_SECS: u64 = 60;
-
-const AWS_ENCODE_SET: &AsciiSet = &NON_ALPHANUMERIC
-    .remove(b'-')
-    .remove(b'_')
-    .remove(b'.')
-    .remove(b'~');
-
-fn hmac_sha256(key: &[u8], msg: &[u8]) -> Vec<u8> {
-    let mut mac = HmacSha256::new_from_slice(key).expect("HMAC key length is always valid");
-    mac.update(msg);
-    mac.finalize().into_bytes().to_vec()
-}
-
-fn sha256_hex(data: &[u8]) -> String {
-    let mut hasher = Sha256::new();
-    hasher.update(data);
-    hex::encode(hasher.finalize())
-}
-
-fn aws_uri_encode(input: &str) -> String {
-    percent_encode(input.as_bytes(), AWS_ENCODE_SET).to_string()
-}
-
-pub fn derive_signing_key_cached(
-    secret_key: &str,
-    date_stamp: &str,
-    region: &str,
-    service: &str,
-) -> Vec<u8> {
-    let cache_key = (
-        secret_key.to_owned(),
-        date_stamp.to_owned(),
-        region.to_owned(),
-        service.to_owned(),
-    );
-
-    {
-        let mut cache = SIGNING_KEY_CACHE.lock();
-        if let Some(entry) = cache.get(&cache_key) {
-            if entry.created.elapsed().as_secs() < CACHE_TTL_SECS {
-                return entry.key.clone();
-            }
-            cache.pop(&cache_key);
-        }
-    }
-
-    let k_date = hmac_sha256(
-        format!("AWS4{}", secret_key).as_bytes(),
-        date_stamp.as_bytes(),
-    );
-    let k_region = hmac_sha256(&k_date, region.as_bytes());
-    let k_service = hmac_sha256(&k_region, service.as_bytes());
-    let k_signing = hmac_sha256(&k_service, b"aws4_request");
-
-    {
-        let mut cache = SIGNING_KEY_CACHE.lock();
-        cache.put(
-            cache_key,
-            CacheEntry {
-                key: k_signing.clone(),
-                created: Instant::now(),
-            },
-        );
-    }
-
-    k_signing
-}
-
-fn constant_time_compare_inner(a: &[u8], b: &[u8]) -> bool {
-    if a.len() != b.len() {
-        return false;
-    }
-    let mut result: u8 = 0;
-    for (x, y) in a.iter().zip(b.iter()) {
-        result |= x ^ y;
-    }
-    result == 0
-}
-
-pub fn verify_sigv4_signature(
-    method: &str,
-    canonical_uri: &str,
-    query_params: &[(String, String)],
-    signed_headers_str: &str,
-    header_values: &[(String, String)],
-    payload_hash: &str,
-    amz_date: &str,
-    date_stamp: &str,
-    region: &str,
-    service: &str,
-    secret_key: &str,
-    provided_signature: &str,
-) -> bool {
-    let mut sorted_params = query_params.to_vec();
-    sorted_params.sort_by(|a, b| a.0.cmp(&b.0).then_with(|| a.1.cmp(&b.1)));
-
-    let canonical_query_string = sorted_params
-        .iter()
-        .map(|(k, v)| format!("{}={}", aws_uri_encode(k), aws_uri_encode(v)))
-        .collect::<Vec<_>>()
-        .join("&");
-
-    let mut canonical_headers = String::new();
-    for (name, value) in header_values {
-        let lower_name = name.to_lowercase();
-        let normalized = value.split_whitespace().collect::<Vec<_>>().join(" ");
-        let final_value = if lower_name == "expect" && normalized.is_empty() {
-            "100-continue"
-        } else {
-            &normalized
-        };
-        canonical_headers.push_str(&lower_name);
-        canonical_headers.push(':');
-        canonical_headers.push_str(final_value);
-        canonical_headers.push('\n');
-    }
-
-    let canonical_request = format!(
-        "{}\n{}\n{}\n{}\n{}\n{}",
-        method,
-        canonical_uri,
-        canonical_query_string,
-        canonical_headers,
-        signed_headers_str,
-        payload_hash
-    );
-
-    let credential_scope = format!("{}/{}/{}/aws4_request", date_stamp, region, service);
-    let cr_hash = sha256_hex(canonical_request.as_bytes());
-    let string_to_sign = format!(
-        "AWS4-HMAC-SHA256\n{}\n{}\n{}",
-        amz_date, credential_scope, cr_hash
-    );
-
-    let signing_key = derive_signing_key_cached(secret_key, date_stamp, region, service);
-    let calculated = hmac_sha256(&signing_key, string_to_sign.as_bytes());
-    let calculated_hex = hex::encode(&calculated);
-
-    constant_time_compare_inner(calculated_hex.as_bytes(), provided_signature.as_bytes())
-}
-
-pub fn derive_signing_key(
-    secret_key: &str,
-    date_stamp: &str,
-    region: &str,
-    service: &str,
-) -> Vec<u8> {
-    derive_signing_key_cached(secret_key, date_stamp, region, service)
-}
-
-pub fn compute_signature(signing_key: &[u8], string_to_sign: &str) -> String {
-    let sig = hmac_sha256(signing_key, string_to_sign.as_bytes());
-    hex::encode(sig)
-}
-
-pub fn compute_post_policy_signature(signing_key: &[u8], policy_b64: &str) -> String {
-    let sig = hmac_sha256(signing_key, policy_b64.as_bytes());
-    hex::encode(sig)
-}
-
-pub fn build_string_to_sign(
-    amz_date: &str,
-    credential_scope: &str,
-    canonical_request: &str,
-) -> String {
-    let cr_hash = sha256_hex(canonical_request.as_bytes());
-    format!(
-        "AWS4-HMAC-SHA256\n{}\n{}\n{}",
-        amz_date, credential_scope, cr_hash
-    )
-}
-
-pub fn constant_time_compare(a: &str, b: &str) -> bool {
-    constant_time_compare_inner(a.as_bytes(), b.as_bytes())
-}
-
-pub fn clear_signing_key_cache() {
-    SIGNING_KEY_CACHE.lock().clear();
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_derive_signing_key() {
-        let key = derive_signing_key(
-            "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
-            "20130524",
-            "us-east-1",
-            "s3",
-        );
-        assert_eq!(key.len(), 32);
-    }
-
-    #[test]
-    fn test_derive_signing_key_cached() {
-        let key1 = derive_signing_key("secret", "20240101", "us-east-1", "s3");
-        let key2 = derive_signing_key("secret", "20240101", "us-east-1", "s3");
-        assert_eq!(key1, key2);
-    }
-
-    #[test]
-    fn test_constant_time_compare() {
-        assert!(constant_time_compare("abc", "abc"));
-        assert!(!constant_time_compare("abc", "abd"));
-        assert!(!constant_time_compare("abc", "abcd"));
-    }
-
-    #[test]
-    fn test_build_string_to_sign() {
-        let result = build_string_to_sign(
-            "20130524T000000Z",
-            "20130524/us-east-1/s3/aws4_request",
-            "GET\n/\n\nhost:example.com\n\nhost\nUNSIGNED-PAYLOAD",
-        );
-        assert!(result.starts_with("AWS4-HMAC-SHA256\n"));
-        assert!(result.contains("20130524T000000Z"));
-    }
-
-    #[test]
-    fn test_aws_uri_encode() {
-        assert_eq!(aws_uri_encode("hello world"), "hello%20world");
-        assert_eq!(aws_uri_encode("test-file_name.txt"), "test-file_name.txt");
-        assert_eq!(aws_uri_encode("a/b"), "a%2Fb");
-    }
-
-    #[test]
-    fn test_verify_sigv4_roundtrip() {
-        let secret = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";
-        let date_stamp = "20130524";
-        let region = "us-east-1";
-        let service = "s3";
-        let amz_date = "20130524T000000Z";
-
-        let signing_key = derive_signing_key(secret, date_stamp, region, service);
-
-        let canonical_request =
-            "GET\n/\n\nhost:examplebucket.s3.amazonaws.com\n\nhost\nUNSIGNED-PAYLOAD";
-        let string_to_sign = build_string_to_sign(
-            amz_date,
-            &format!("{}/{}/{}/aws4_request", date_stamp, region, service),
-            canonical_request,
-        );
-
-        let signature = compute_signature(&signing_key, &string_to_sign);
-
-        let result = verify_sigv4_signature(
-            "GET",
-            "/",
-            &[],
-            "host",
-            &[(
-                "host".to_string(),
-                "examplebucket.s3.amazonaws.com".to_string(),
-            )],
-            "UNSIGNED-PAYLOAD",
-            amz_date,
-            date_stamp,
-            region,
-            service,
-            secret,
-            &signature,
-        );
-        assert!(result);
-    }
-}

rust/myfsio-engine/crates/myfsio-common/Cargo.toml

@@ -1,11 +0,0 @@
-[package]
-name = "myfsio-common"
-version.workspace = true
-edition.workspace = true
-
-[dependencies]
-thiserror = { workspace = true }
-serde = { workspace = true }
-serde_json = { workspace = true }
-chrono = { workspace = true }
-uuid = { workspace = true }

rust/myfsio-engine/crates/myfsio-common/src/constants.rs

@@ -1,20 +0,0 @@
-pub const SYSTEM_ROOT: &str = ".myfsio.sys";
-pub const SYSTEM_BUCKETS_DIR: &str = "buckets";
-pub const SYSTEM_MULTIPART_DIR: &str = "multipart";
-pub const BUCKET_META_DIR: &str = "meta";
-pub const BUCKET_VERSIONS_DIR: &str = "versions";
-pub const BUCKET_CONFIG_FILE: &str = ".bucket.json";
-pub const STATS_FILE: &str = "stats.json";
-pub const ETAG_INDEX_FILE: &str = "etag_index.json";
-pub const INDEX_FILE: &str = "_index.json";
-pub const MANIFEST_FILE: &str = "manifest.json";
-
-pub const INTERNAL_FOLDERS: &[&str] = &[".meta", ".versions", ".multipart"];
-
-pub const DEFAULT_REGION: &str = "us-east-1";
-pub const AWS_SERVICE: &str = "s3";
-
-pub const DEFAULT_MAX_KEYS: usize = 1000;
-pub const DEFAULT_OBJECT_KEY_MAX_BYTES: usize = 1024;
-pub const DEFAULT_CHUNK_SIZE: usize = 65536;
-pub const STREAM_CHUNK_SIZE: usize = 1_048_576;

rust/myfsio-engine/crates/myfsio-common/src/error.rs

@@ -1,225 +0,0 @@
-use std::fmt;
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum S3ErrorCode {
-    AccessDenied,
-    BucketAlreadyExists,
-    BucketNotEmpty,
-    EntityTooLarge,
-    InternalError,
-    InvalidAccessKeyId,
-    InvalidArgument,
-    InvalidBucketName,
-    InvalidKey,
-    InvalidPolicyDocument,
-    InvalidRange,
-    InvalidRequest,
-    MalformedXML,
-    MethodNotAllowed,
-    NoSuchBucket,
-    NoSuchKey,
-    NoSuchUpload,
-    NoSuchVersion,
-    NoSuchTagSet,
-    PreconditionFailed,
-    NotModified,
-    QuotaExceeded,
-    SignatureDoesNotMatch,
-    SlowDown,
-}
-
-impl S3ErrorCode {
-    pub fn http_status(&self) -> u16 {
-        match self {
-            Self::AccessDenied => 403,
-            Self::BucketAlreadyExists => 409,
-            Self::BucketNotEmpty => 409,
-            Self::EntityTooLarge => 413,
-            Self::InternalError => 500,
-            Self::InvalidAccessKeyId => 403,
-            Self::InvalidArgument => 400,
-            Self::InvalidBucketName => 400,
-            Self::InvalidKey => 400,
-            Self::InvalidPolicyDocument => 400,
-            Self::InvalidRange => 416,
-            Self::InvalidRequest => 400,
-            Self::MalformedXML => 400,
-            Self::MethodNotAllowed => 405,
-            Self::NoSuchBucket => 404,
-            Self::NoSuchKey => 404,
-            Self::NoSuchUpload => 404,
-            Self::NoSuchVersion => 404,
-            Self::NoSuchTagSet => 404,
-            Self::PreconditionFailed => 412,
-            Self::NotModified => 304,
-            Self::QuotaExceeded => 403,
-            Self::SignatureDoesNotMatch => 403,
-            Self::SlowDown => 429,
-        }
-    }
-
-    pub fn as_str(&self) -> &'static str {
-        match self {
-            Self::AccessDenied => "AccessDenied",
-            Self::BucketAlreadyExists => "BucketAlreadyExists",
-            Self::BucketNotEmpty => "BucketNotEmpty",
-            Self::EntityTooLarge => "EntityTooLarge",
-            Self::InternalError => "InternalError",
-            Self::InvalidAccessKeyId => "InvalidAccessKeyId",
-            Self::InvalidArgument => "InvalidArgument",
-            Self::InvalidBucketName => "InvalidBucketName",
-            Self::InvalidKey => "InvalidKey",
-            Self::InvalidPolicyDocument => "InvalidPolicyDocument",
-            Self::InvalidRange => "InvalidRange",
-            Self::InvalidRequest => "InvalidRequest",
-            Self::MalformedXML => "MalformedXML",
-            Self::MethodNotAllowed => "MethodNotAllowed",
-            Self::NoSuchBucket => "NoSuchBucket",
-            Self::NoSuchKey => "NoSuchKey",
-            Self::NoSuchUpload => "NoSuchUpload",
-            Self::NoSuchVersion => "NoSuchVersion",
-            Self::NoSuchTagSet => "NoSuchTagSet",
-            Self::PreconditionFailed => "PreconditionFailed",
-            Self::NotModified => "NotModified",
-            Self::QuotaExceeded => "QuotaExceeded",
-            Self::SignatureDoesNotMatch => "SignatureDoesNotMatch",
-            Self::SlowDown => "SlowDown",
-        }
-    }
-
-    pub fn default_message(&self) -> &'static str {
-        match self {
-            Self::AccessDenied => "Access Denied",
-            Self::BucketAlreadyExists => "The requested bucket name is not available",
-            Self::BucketNotEmpty => "The bucket you tried to delete is not empty",
-            Self::EntityTooLarge => "Your proposed upload exceeds the maximum allowed size",
-            Self::InternalError => "We encountered an internal error. Please try again.",
-            Self::InvalidAccessKeyId => "The access key ID you provided does not exist",
-            Self::InvalidArgument => "Invalid argument",
-            Self::InvalidBucketName => "The specified bucket is not valid",
-            Self::InvalidKey => "The specified key is not valid",
-            Self::InvalidPolicyDocument => "The content of the form does not meet the conditions specified in the policy document",
-            Self::InvalidRange => "The requested range is not satisfiable",
-            Self::InvalidRequest => "Invalid request",
-            Self::MalformedXML => "The XML you provided was not well-formed",
-            Self::MethodNotAllowed => "The specified method is not allowed against this resource",
-            Self::NoSuchBucket => "The specified bucket does not exist",
-            Self::NoSuchKey => "The specified key does not exist",
-            Self::NoSuchUpload => "The specified multipart upload does not exist",
-            Self::NoSuchVersion => "The specified version does not exist",
-            Self::NoSuchTagSet => "The TagSet does not exist",
-            Self::PreconditionFailed => "At least one of the preconditions you specified did not hold",
-            Self::NotModified => "Not Modified",
-            Self::QuotaExceeded => "The bucket quota has been exceeded",
-            Self::SignatureDoesNotMatch => "The request signature we calculated does not match the signature you provided",
-            Self::SlowDown => "Please reduce your request rate",
-        }
-    }
-}
-
-impl fmt::Display for S3ErrorCode {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        f.write_str(self.as_str())
-    }
-}
-
-#[derive(Debug, Clone)]
-pub struct S3Error {
-    pub code: S3ErrorCode,
-    pub message: String,
-    pub resource: String,
-    pub request_id: String,
-}
-
-impl S3Error {
-    pub fn new(code: S3ErrorCode, message: impl Into<String>) -> Self {
-        Self {
-            code,
-            message: message.into(),
-            resource: String::new(),
-            request_id: String::new(),
-        }
-    }
-
-    pub fn from_code(code: S3ErrorCode) -> Self {
-        Self::new(code, code.default_message())
-    }
-
-    pub fn with_resource(mut self, resource: impl Into<String>) -> Self {
-        self.resource = resource.into();
-        self
-    }
-
-    pub fn with_request_id(mut self, request_id: impl Into<String>) -> Self {
-        self.request_id = request_id.into();
-        self
-    }
-
-    pub fn http_status(&self) -> u16 {
-        self.code.http_status()
-    }
-
-    pub fn to_xml(&self) -> String {
-        format!(
-            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
-            <Error>\
-            <Code>{}</Code>\
-            <Message>{}</Message>\
-            <Resource>{}</Resource>\
-            <RequestId>{}</RequestId>\
-            </Error>",
-            self.code.as_str(),
-            xml_escape(&self.message),
-            xml_escape(&self.resource),
-            xml_escape(&self.request_id),
-        )
-    }
-}
-
-impl fmt::Display for S3Error {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "{}: {}", self.code, self.message)
-    }
-}
-
-impl std::error::Error for S3Error {}
-
-fn xml_escape(s: &str) -> String {
-    s.replace('&', "&amp;")
-        .replace('<', "&lt;")
-        .replace('>', "&gt;")
-        .replace('"', "&quot;")
-        .replace('\'', "&apos;")
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_error_codes() {
-        assert_eq!(S3ErrorCode::NoSuchKey.http_status(), 404);
-        assert_eq!(S3ErrorCode::AccessDenied.http_status(), 403);
-        assert_eq!(S3ErrorCode::NoSuchBucket.as_str(), "NoSuchBucket");
-    }
-
-    #[test]
-    fn test_error_to_xml() {
-        let err = S3Error::from_code(S3ErrorCode::NoSuchKey)
-            .with_resource("/test-bucket/test-key")
-            .with_request_id("abc123");
-        let xml = err.to_xml();
-        assert!(xml.contains("<Code>NoSuchKey</Code>"));
-        assert!(xml.contains("<Resource>/test-bucket/test-key</Resource>"));
-        assert!(xml.contains("<RequestId>abc123</RequestId>"));
-    }
-
-    #[test]
-    fn test_xml_escape() {
-        let err = S3Error::new(S3ErrorCode::InvalidArgument, "key <test> & \"value\"")
-            .with_resource("/bucket/key&amp");
-        let xml = err.to_xml();
-        assert!(xml.contains("&lt;test&gt;"));
-        assert!(xml.contains("&amp;"));
-    }
-}

rust/myfsio-engine/crates/myfsio-common/src/lib.rs

@@ -1,3 +0,0 @@
-pub mod constants;
-pub mod error;
-pub mod types;

rust/myfsio-engine/crates/myfsio-common/src/types.rs

@@ -1,176 +0,0 @@
-use std::collections::HashMap;
-
-use chrono::{DateTime, Utc};
-use serde::{Deserialize, Serialize};
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct ObjectMeta {
-    pub key: String,
-    pub size: u64,
-    pub last_modified: DateTime<Utc>,
-    pub etag: Option<String>,
-    pub content_type: Option<String>,
-    pub storage_class: Option<String>,
-    pub metadata: HashMap<String, String>,
-}
-
-impl ObjectMeta {
-    pub fn new(key: String, size: u64, last_modified: DateTime<Utc>) -> Self {
-        Self {
-            key,
-            size,
-            last_modified,
-            etag: None,
-            content_type: None,
-            storage_class: Some("STANDARD".to_string()),
-            metadata: HashMap::new(),
-        }
-    }
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct BucketMeta {
-    pub name: String,
-    pub creation_date: DateTime<Utc>,
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct BucketStats {
-    pub objects: u64,
-    pub bytes: u64,
-    pub version_count: u64,
-    pub version_bytes: u64,
-}
-
-impl BucketStats {
-    pub fn total_objects(&self) -> u64 {
-        self.objects + self.version_count
-    }
-
-    pub fn total_bytes(&self) -> u64 {
-        self.bytes + self.version_bytes
-    }
-}
-
-#[derive(Debug, Clone)]
-pub struct ListObjectsResult {
-    pub objects: Vec<ObjectMeta>,
-    pub is_truncated: bool,
-    pub next_continuation_token: Option<String>,
-}
-
-#[derive(Debug, Clone)]
-pub struct ShallowListResult {
-    pub objects: Vec<ObjectMeta>,
-    pub common_prefixes: Vec<String>,
-    pub is_truncated: bool,
-    pub next_continuation_token: Option<String>,
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct ListParams {
-    pub max_keys: usize,
-    pub continuation_token: Option<String>,
-    pub prefix: Option<String>,
-    pub start_after: Option<String>,
-}
-
-#[derive(Debug, Clone, Default)]
-pub struct ShallowListParams {
-    pub prefix: String,
-    pub delimiter: String,
-    pub max_keys: usize,
-    pub continuation_token: Option<String>,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct PartMeta {
-    pub part_number: u32,
-    pub etag: String,
-    pub size: u64,
-    pub last_modified: Option<DateTime<Utc>>,
-}
-
-#[derive(Debug, Clone)]
-pub struct PartInfo {
-    pub part_number: u32,
-    pub etag: String,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct MultipartUploadInfo {
-    pub upload_id: String,
-    pub key: String,
-    pub initiated: DateTime<Utc>,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct VersionInfo {
-    pub version_id: String,
-    pub key: String,
-    pub size: u64,
-    pub last_modified: DateTime<Utc>,
-    pub etag: Option<String>,
-    pub is_latest: bool,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct Tag {
-    pub key: String,
-    pub value: String,
-}
-
-#[derive(Debug, Clone, Default, Serialize, Deserialize)]
-pub struct BucketConfig {
-    #[serde(default)]
-    pub versioning_enabled: bool,
-    #[serde(default)]
-    pub tags: Vec<Tag>,
-    #[serde(default)]
-    pub cors: Option<serde_json::Value>,
-    #[serde(default)]
-    pub encryption: Option<serde_json::Value>,
-    #[serde(default)]
-    pub lifecycle: Option<serde_json::Value>,
-    #[serde(default)]
-    pub website: Option<serde_json::Value>,
-    #[serde(default)]
-    pub quota: Option<QuotaConfig>,
-    #[serde(default)]
-    pub acl: Option<serde_json::Value>,
-    #[serde(default)]
-    pub notification: Option<serde_json::Value>,
-    #[serde(default)]
-    pub logging: Option<serde_json::Value>,
-    #[serde(default)]
-    pub object_lock: Option<serde_json::Value>,
-    #[serde(default)]
-    pub policy: Option<serde_json::Value>,
-    #[serde(default)]
-    pub replication: Option<serde_json::Value>,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct QuotaConfig {
-    pub max_bytes: Option<u64>,
-    pub max_objects: Option<u64>,
-}
-
-#[derive(Debug, Clone)]
-pub struct Principal {
-    pub access_key: String,
-    pub user_id: String,
-    pub display_name: String,
-    pub is_admin: bool,
-}
-
-impl Principal {
-    pub fn new(access_key: String, user_id: String, display_name: String, is_admin: bool) -> Self {
-        Self {
-            access_key,
-            user_id,
-            display_name,
-            is_admin,
-        }
-    }
-}

rust/myfsio-engine/crates/myfsio-crypto/Cargo.toml

@@ -1,24 +0,0 @@
-[package]
-name = "myfsio-crypto"
-version.workspace = true
-edition.workspace = true
-
-[dependencies]
-myfsio-common = { path = "../myfsio-common" }
-md-5 = { workspace = true }
-sha2 = { workspace = true }
-hex = { workspace = true }
-aes-gcm = { workspace = true }
-hkdf = { workspace = true }
-thiserror = { workspace = true }
-tokio = { workspace = true }
-serde = { workspace = true }
-serde_json = { workspace = true }
-uuid = { workspace = true }
-chrono = { workspace = true }
-base64 = { workspace = true }
-rand = "0.8"
-
-[dev-dependencies]
-tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
-tempfile = "3"

rust/myfsio-engine/crates/myfsio-crypto/src/aes_gcm.rs

@@ -1,253 +0,0 @@
-use aes_gcm::aead::Aead;
-use aes_gcm::{Aes256Gcm, KeyInit, Nonce};
-use hkdf::Hkdf;
-use sha2::Sha256;
-use std::fs::File;
-use std::io::{Read, Seek, SeekFrom, Write};
-use std::path::Path;
-use thiserror::Error;
-
-const DEFAULT_CHUNK_SIZE: usize = 65536;
-const HEADER_SIZE: usize = 4;
-
-#[derive(Debug, Error)]
-pub enum CryptoError {
-    #[error("IO error: {0}")]
-    Io(#[from] std::io::Error),
-    #[error("Invalid key size: expected 32 bytes, got {0}")]
-    InvalidKeySize(usize),
-    #[error("Invalid nonce size: expected 12 bytes, got {0}")]
-    InvalidNonceSize(usize),
-    #[error("Encryption failed: {0}")]
-    EncryptionFailed(String),
-    #[error("Decryption failed at chunk {0}")]
-    DecryptionFailed(u32),
-    #[error("HKDF expand failed: {0}")]
-    HkdfFailed(String),
-}
-
-fn read_exact_chunk(reader: &mut impl Read, buf: &mut [u8]) -> std::io::Result<usize> {
-    let mut filled = 0;
-    while filled < buf.len() {
-        match reader.read(&mut buf[filled..]) {
-            Ok(0) => break,
-            Ok(n) => filled += n,
-            Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => continue,
-            Err(e) => return Err(e),
-        }
-    }
-    Ok(filled)
-}
-
-fn derive_chunk_nonce(base_nonce: &[u8], chunk_index: u32) -> Result<[u8; 12], CryptoError> {
-    let hkdf = Hkdf::<Sha256>::new(Some(base_nonce), b"chunk_nonce");
-    let mut okm = [0u8; 12];
-    hkdf.expand(&chunk_index.to_be_bytes(), &mut okm)
-        .map_err(|e| CryptoError::HkdfFailed(e.to_string()))?;
-    Ok(okm)
-}
-
-pub fn encrypt_stream_chunked(
-    input_path: &Path,
-    output_path: &Path,
-    key: &[u8],
-    base_nonce: &[u8],
-    chunk_size: Option<usize>,
-) -> Result<u32, CryptoError> {
-    if key.len() != 32 {
-        return Err(CryptoError::InvalidKeySize(key.len()));
-    }
-    if base_nonce.len() != 12 {
-        return Err(CryptoError::InvalidNonceSize(base_nonce.len()));
-    }
-
-    let chunk_size = chunk_size.unwrap_or(DEFAULT_CHUNK_SIZE);
-    let key_arr: [u8; 32] = key.try_into().unwrap();
-    let nonce_arr: [u8; 12] = base_nonce.try_into().unwrap();
-    let cipher = Aes256Gcm::new(&key_arr.into());
-
-    let mut infile = File::open(input_path)?;
-    let mut outfile = File::create(output_path)?;
-
-    outfile.write_all(&[0u8; 4])?;
-
-    let mut buf = vec![0u8; chunk_size];
-    let mut chunk_index: u32 = 0;
-
-    loop {
-        let n = read_exact_chunk(&mut infile, &mut buf)?;
-        if n == 0 {
-            break;
-        }
-
-        let nonce_bytes = derive_chunk_nonce(&nonce_arr, chunk_index)?;
-        let nonce = Nonce::from_slice(&nonce_bytes);
-
-        let encrypted = cipher
-            .encrypt(nonce, &buf[..n])
-            .map_err(|e| CryptoError::EncryptionFailed(e.to_string()))?;
-
-        let size = encrypted.len() as u32;
-        outfile.write_all(&size.to_be_bytes())?;
-        outfile.write_all(&encrypted)?;
-
-        chunk_index += 1;
-    }
-
-    outfile.seek(SeekFrom::Start(0))?;
-    outfile.write_all(&chunk_index.to_be_bytes())?;
-
-    Ok(chunk_index)
-}
-
-pub fn decrypt_stream_chunked(
-    input_path: &Path,
-    output_path: &Path,
-    key: &[u8],
-    base_nonce: &[u8],
-) -> Result<u32, CryptoError> {
-    if key.len() != 32 {
-        return Err(CryptoError::InvalidKeySize(key.len()));
-    }
-    if base_nonce.len() != 12 {
-        return Err(CryptoError::InvalidNonceSize(base_nonce.len()));
-    }
-
-    let key_arr: [u8; 32] = key.try_into().unwrap();
-    let nonce_arr: [u8; 12] = base_nonce.try_into().unwrap();
-    let cipher = Aes256Gcm::new(&key_arr.into());
-
-    let mut infile = File::open(input_path)?;
-    let mut outfile = File::create(output_path)?;
-
-    let mut header = [0u8; HEADER_SIZE];
-    infile.read_exact(&mut header)?;
-    let chunk_count = u32::from_be_bytes(header);
-
-    let mut size_buf = [0u8; HEADER_SIZE];
-    for chunk_index in 0..chunk_count {
-        infile.read_exact(&mut size_buf)?;
-        let chunk_size = u32::from_be_bytes(size_buf) as usize;
-
-        let mut encrypted = vec![0u8; chunk_size];
-        infile.read_exact(&mut encrypted)?;
-
-        let nonce_bytes = derive_chunk_nonce(&nonce_arr, chunk_index)?;
-        let nonce = Nonce::from_slice(&nonce_bytes);
-
-        let decrypted = cipher
-            .decrypt(nonce, encrypted.as_ref())
-            .map_err(|_| CryptoError::DecryptionFailed(chunk_index))?;
-
-        outfile.write_all(&decrypted)?;
-    }
-
-    Ok(chunk_count)
-}
-
-pub async fn encrypt_stream_chunked_async(
-    input_path: &Path,
-    output_path: &Path,
-    key: &[u8],
-    base_nonce: &[u8],
-    chunk_size: Option<usize>,
-) -> Result<u32, CryptoError> {
-    let input_path = input_path.to_owned();
-    let output_path = output_path.to_owned();
-    let key = key.to_vec();
-    let base_nonce = base_nonce.to_vec();
-    tokio::task::spawn_blocking(move || {
-        encrypt_stream_chunked(&input_path, &output_path, &key, &base_nonce, chunk_size)
-    })
-    .await
-    .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))?
-}
-
-pub async fn decrypt_stream_chunked_async(
-    input_path: &Path,
-    output_path: &Path,
-    key: &[u8],
-    base_nonce: &[u8],
-) -> Result<u32, CryptoError> {
-    let input_path = input_path.to_owned();
-    let output_path = output_path.to_owned();
-    let key = key.to_vec();
-    let base_nonce = base_nonce.to_vec();
-    tokio::task::spawn_blocking(move || {
-        decrypt_stream_chunked(&input_path, &output_path, &key, &base_nonce)
-    })
-    .await
-    .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))?
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::io::Write as IoWrite;
-
-    #[test]
-    fn test_encrypt_decrypt_roundtrip() {
-        let dir = tempfile::tempdir().unwrap();
-        let input = dir.path().join("input.bin");
-        let encrypted = dir.path().join("encrypted.bin");
-        let decrypted = dir.path().join("decrypted.bin");
-
-        let data = b"Hello, this is a test of AES-256-GCM chunked encryption!";
-        std::fs::File::create(&input)
-            .unwrap()
-            .write_all(data)
-            .unwrap();
-
-        let key = [0x42u8; 32];
-        let nonce = [0x01u8; 12];
-
-        let chunks = encrypt_stream_chunked(&input, &encrypted, &key, &nonce, Some(16)).unwrap();
-        assert!(chunks > 0);
-
-        let chunks2 = decrypt_stream_chunked(&encrypted, &decrypted, &key, &nonce).unwrap();
-        assert_eq!(chunks, chunks2);
-
-        let result = std::fs::read(&decrypted).unwrap();
-        assert_eq!(result, data);
-    }
-
-    #[test]
-    fn test_invalid_key_size() {
-        let dir = tempfile::tempdir().unwrap();
-        let input = dir.path().join("input.bin");
-        std::fs::File::create(&input)
-            .unwrap()
-            .write_all(b"test")
-            .unwrap();
-
-        let result = encrypt_stream_chunked(
-            &input,
-            &dir.path().join("out"),
-            &[0u8; 16],
-            &[0u8; 12],
-            None,
-        );
-        assert!(matches!(result, Err(CryptoError::InvalidKeySize(16))));
-    }
-
-    #[test]
-    fn test_wrong_key_fails_decrypt() {
-        let dir = tempfile::tempdir().unwrap();
-        let input = dir.path().join("input.bin");
-        let encrypted = dir.path().join("encrypted.bin");
-        let decrypted = dir.path().join("decrypted.bin");
-
-        std::fs::File::create(&input)
-            .unwrap()
-            .write_all(b"secret data")
-            .unwrap();
-
-        let key = [0x42u8; 32];
-        let nonce = [0x01u8; 12];
-        encrypt_stream_chunked(&input, &encrypted, &key, &nonce, None).unwrap();
-
-        let wrong_key = [0x43u8; 32];
-        let result = decrypt_stream_chunked(&encrypted, &decrypted, &wrong_key, &nonce);
-        assert!(matches!(result, Err(CryptoError::DecryptionFailed(_))));
-    }
-}

rust/myfsio-engine/crates/myfsio-crypto/src/encryption.rs

@@ -1,377 +0,0 @@
-use base64::engine::general_purpose::STANDARD as B64;
-use base64::Engine;
-use rand::RngCore;
-use std::collections::HashMap;
-use std::path::Path;
-
-use crate::aes_gcm::{decrypt_stream_chunked, encrypt_stream_chunked, CryptoError};
-use crate::kms::KmsService;
-
-#[derive(Debug, Clone, PartialEq)]
-pub enum SseAlgorithm {
-    Aes256,
-    AwsKms,
-    CustomerProvided,
-}
-
-impl SseAlgorithm {
-    pub fn as_str(&self) -> &'static str {
-        match self {
-            SseAlgorithm::Aes256 => "AES256",
-            SseAlgorithm::AwsKms => "aws:kms",
-            SseAlgorithm::CustomerProvided => "AES256",
-        }
-    }
-}
-
-#[derive(Debug, Clone)]
-pub struct EncryptionContext {
-    pub algorithm: SseAlgorithm,
-    pub kms_key_id: Option<String>,
-    pub customer_key: Option<Vec<u8>>,
-}
-
-#[derive(Debug, Clone)]
-pub struct EncryptionMetadata {
-    pub algorithm: String,
-    pub nonce: String,
-    pub encrypted_data_key: Option<String>,
-    pub kms_key_id: Option<String>,
-}
-
-impl EncryptionMetadata {
-    pub fn to_metadata_map(&self) -> HashMap<String, String> {
-        let mut map = HashMap::new();
-        map.insert(
-            "x-amz-server-side-encryption".to_string(),
-            self.algorithm.clone(),
-        );
-        map.insert("x-amz-encryption-nonce".to_string(), self.nonce.clone());
-        if let Some(ref dk) = self.encrypted_data_key {
-            map.insert("x-amz-encrypted-data-key".to_string(), dk.clone());
-        }
-        if let Some(ref kid) = self.kms_key_id {
-            map.insert("x-amz-encryption-key-id".to_string(), kid.clone());
-        }
-        map
-    }
-
-    pub fn from_metadata(meta: &HashMap<String, String>) -> Option<Self> {
-        let algorithm = meta.get("x-amz-server-side-encryption")?;
-        let nonce = meta.get("x-amz-encryption-nonce")?;
-        Some(Self {
-            algorithm: algorithm.clone(),
-            nonce: nonce.clone(),
-            encrypted_data_key: meta.get("x-amz-encrypted-data-key").cloned(),
-            kms_key_id: meta.get("x-amz-encryption-key-id").cloned(),
-        })
-    }
-
-    pub fn is_encrypted(meta: &HashMap<String, String>) -> bool {
-        meta.contains_key("x-amz-server-side-encryption")
-    }
-
-    pub fn clean_metadata(meta: &mut HashMap<String, String>) {
-        meta.remove("x-amz-server-side-encryption");
-        meta.remove("x-amz-encryption-nonce");
-        meta.remove("x-amz-encrypted-data-key");
-        meta.remove("x-amz-encryption-key-id");
-    }
-}
-
-pub struct EncryptionService {
-    master_key: [u8; 32],
-    kms: Option<std::sync::Arc<KmsService>>,
-}
-
-impl EncryptionService {
-    pub fn new(master_key: [u8; 32], kms: Option<std::sync::Arc<KmsService>>) -> Self {
-        Self { master_key, kms }
-    }
-
-    pub fn generate_data_key(&self) -> ([u8; 32], [u8; 12]) {
-        let mut data_key = [0u8; 32];
-        let mut nonce = [0u8; 12];
-        rand::thread_rng().fill_bytes(&mut data_key);
-        rand::thread_rng().fill_bytes(&mut nonce);
-        (data_key, nonce)
-    }
-
-    pub fn wrap_data_key(&self, data_key: &[u8; 32]) -> Result<String, CryptoError> {
-        use aes_gcm::aead::Aead;
-        use aes_gcm::{Aes256Gcm, KeyInit, Nonce};
-
-        let cipher = Aes256Gcm::new((&self.master_key).into());
-        let mut nonce_bytes = [0u8; 12];
-        rand::thread_rng().fill_bytes(&mut nonce_bytes);
-        let nonce = Nonce::from_slice(&nonce_bytes);
-
-        let encrypted = cipher
-            .encrypt(nonce, data_key.as_slice())
-            .map_err(|e| CryptoError::EncryptionFailed(e.to_string()))?;
-
-        let mut combined = Vec::with_capacity(12 + encrypted.len());
-        combined.extend_from_slice(&nonce_bytes);
-        combined.extend_from_slice(&encrypted);
-        Ok(B64.encode(&combined))
-    }
-
-    pub fn unwrap_data_key(&self, wrapped_b64: &str) -> Result<[u8; 32], CryptoError> {
-        use aes_gcm::aead::Aead;
-        use aes_gcm::{Aes256Gcm, KeyInit, Nonce};
-
-        let combined = B64.decode(wrapped_b64).map_err(|e| {
-            CryptoError::EncryptionFailed(format!("Bad wrapped key encoding: {}", e))
-        })?;
-        if combined.len() < 12 {
-            return Err(CryptoError::EncryptionFailed(
-                "Wrapped key too short".to_string(),
-            ));
-        }
-
-        let (nonce_bytes, ciphertext) = combined.split_at(12);
-        let cipher = Aes256Gcm::new((&self.master_key).into());
-        let nonce = Nonce::from_slice(nonce_bytes);
-
-        let plaintext = cipher
-            .decrypt(nonce, ciphertext)
-            .map_err(|_| CryptoError::DecryptionFailed(0))?;
-
-        if plaintext.len() != 32 {
-            return Err(CryptoError::InvalidKeySize(plaintext.len()));
-        }
-        let mut key = [0u8; 32];
-        key.copy_from_slice(&plaintext);
-        Ok(key)
-    }
-
-    pub async fn encrypt_object(
-        &self,
-        input_path: &Path,
-        output_path: &Path,
-        ctx: &EncryptionContext,
-    ) -> Result<EncryptionMetadata, CryptoError> {
-        let (data_key, nonce) = self.generate_data_key();
-
-        let (encrypted_data_key, kms_key_id) = match ctx.algorithm {
-            SseAlgorithm::Aes256 => {
-                let wrapped = self.wrap_data_key(&data_key)?;
-                (Some(wrapped), None)
-            }
-            SseAlgorithm::AwsKms => {
-                let kms = self
-                    .kms
-                    .as_ref()
-                    .ok_or_else(|| CryptoError::EncryptionFailed("KMS not available".into()))?;
-                let kid = ctx
-                    .kms_key_id
-                    .as_ref()
-                    .ok_or_else(|| CryptoError::EncryptionFailed("No KMS key ID".into()))?;
-                let ciphertext = kms.encrypt_data(kid, &data_key).await?;
-                (Some(B64.encode(&ciphertext)), Some(kid.clone()))
-            }
-            SseAlgorithm::CustomerProvided => (None, None),
-        };
-
-        let actual_key = if ctx.algorithm == SseAlgorithm::CustomerProvided {
-            let ck = ctx
-                .customer_key
-                .as_ref()
-                .ok_or_else(|| CryptoError::EncryptionFailed("No customer key provided".into()))?;
-            if ck.len() != 32 {
-                return Err(CryptoError::InvalidKeySize(ck.len()));
-            }
-            let mut k = [0u8; 32];
-            k.copy_from_slice(ck);
-            k
-        } else {
-            data_key
-        };
-
-        let ip = input_path.to_owned();
-        let op = output_path.to_owned();
-        let ak = actual_key;
-        let n = nonce;
-        tokio::task::spawn_blocking(move || encrypt_stream_chunked(&ip, &op, &ak, &n, None))
-            .await
-            .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??;
-
-        Ok(EncryptionMetadata {
-            algorithm: ctx.algorithm.as_str().to_string(),
-            nonce: B64.encode(nonce),
-            encrypted_data_key,
-            kms_key_id,
-        })
-    }
-
-    pub async fn decrypt_object(
-        &self,
-        input_path: &Path,
-        output_path: &Path,
-        enc_meta: &EncryptionMetadata,
-        customer_key: Option<&[u8]>,
-    ) -> Result<(), CryptoError> {
-        let nonce_bytes = B64
-            .decode(&enc_meta.nonce)
-            .map_err(|e| CryptoError::EncryptionFailed(format!("Bad nonce encoding: {}", e)))?;
-        if nonce_bytes.len() != 12 {
-            return Err(CryptoError::InvalidNonceSize(nonce_bytes.len()));
-        }
-
-        let data_key: [u8; 32] = if let Some(ck) = customer_key {
-            if ck.len() != 32 {
-                return Err(CryptoError::InvalidKeySize(ck.len()));
-            }
-            let mut k = [0u8; 32];
-            k.copy_from_slice(ck);
-            k
-        } else if enc_meta.algorithm == "aws:kms" {
-            let kms = self
-                .kms
-                .as_ref()
-                .ok_or_else(|| CryptoError::EncryptionFailed("KMS not available".into()))?;
-            let kid = enc_meta
-                .kms_key_id
-                .as_ref()
-                .ok_or_else(|| CryptoError::EncryptionFailed("No KMS key ID in metadata".into()))?;
-            let encrypted_dk = enc_meta.encrypted_data_key.as_ref().ok_or_else(|| {
-                CryptoError::EncryptionFailed("No encrypted data key in metadata".into())
-            })?;
-            let ct = B64.decode(encrypted_dk).map_err(|e| {
-                CryptoError::EncryptionFailed(format!("Bad data key encoding: {}", e))
-            })?;
-            let dk = kms.decrypt_data(kid, &ct).await?;
-            if dk.len() != 32 {
-                return Err(CryptoError::InvalidKeySize(dk.len()));
-            }
-            let mut k = [0u8; 32];
-            k.copy_from_slice(&dk);
-            k
-        } else {
-            let wrapped = enc_meta.encrypted_data_key.as_ref().ok_or_else(|| {
-                CryptoError::EncryptionFailed("No encrypted data key in metadata".into())
-            })?;
-            self.unwrap_data_key(wrapped)?
-        };
-
-        let ip = input_path.to_owned();
-        let op = output_path.to_owned();
-        let nb: [u8; 12] = nonce_bytes.try_into().unwrap();
-        tokio::task::spawn_blocking(move || decrypt_stream_chunked(&ip, &op, &data_key, &nb))
-            .await
-            .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??;
-
-        Ok(())
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::io::Write;
-
-    fn test_master_key() -> [u8; 32] {
-        [0x42u8; 32]
-    }
-
-    #[test]
-    fn test_wrap_unwrap_data_key() {
-        let svc = EncryptionService::new(test_master_key(), None);
-        let dk = [0xAAu8; 32];
-        let wrapped = svc.wrap_data_key(&dk).unwrap();
-        let unwrapped = svc.unwrap_data_key(&wrapped).unwrap();
-        assert_eq!(dk, unwrapped);
-    }
-
-    #[tokio::test]
-    async fn test_encrypt_decrypt_object_sse_s3() {
-        let dir = tempfile::tempdir().unwrap();
-        let input = dir.path().join("plain.bin");
-        let encrypted = dir.path().join("enc.bin");
-        let decrypted = dir.path().join("dec.bin");
-
-        let data = b"SSE-S3 encrypted content for testing!";
-        std::fs::File::create(&input)
-            .unwrap()
-            .write_all(data)
-            .unwrap();
-
-        let svc = EncryptionService::new(test_master_key(), None);
-
-        let ctx = EncryptionContext {
-            algorithm: SseAlgorithm::Aes256,
-            kms_key_id: None,
-            customer_key: None,
-        };
-
-        let meta = svc.encrypt_object(&input, &encrypted, &ctx).await.unwrap();
-        assert_eq!(meta.algorithm, "AES256");
-        assert!(meta.encrypted_data_key.is_some());
-
-        svc.decrypt_object(&encrypted, &decrypted, &meta, None)
-            .await
-            .unwrap();
-
-        let result = std::fs::read(&decrypted).unwrap();
-        assert_eq!(result, data);
-    }
-
-    #[tokio::test]
-    async fn test_encrypt_decrypt_object_sse_c() {
-        let dir = tempfile::tempdir().unwrap();
-        let input = dir.path().join("plain.bin");
-        let encrypted = dir.path().join("enc.bin");
-        let decrypted = dir.path().join("dec.bin");
-
-        let data = b"SSE-C encrypted content!";
-        std::fs::File::create(&input)
-            .unwrap()
-            .write_all(data)
-            .unwrap();
-
-        let customer_key = [0xBBu8; 32];
-        let svc = EncryptionService::new(test_master_key(), None);
-
-        let ctx = EncryptionContext {
-            algorithm: SseAlgorithm::CustomerProvided,
-            kms_key_id: None,
-            customer_key: Some(customer_key.to_vec()),
-        };
-
-        let meta = svc.encrypt_object(&input, &encrypted, &ctx).await.unwrap();
-        assert!(meta.encrypted_data_key.is_none());
-
-        svc.decrypt_object(&encrypted, &decrypted, &meta, Some(&customer_key))
-            .await
-            .unwrap();
-
-        let result = std::fs::read(&decrypted).unwrap();
-        assert_eq!(result, data);
-    }
-
-    #[test]
-    fn test_encryption_metadata_roundtrip() {
-        let meta = EncryptionMetadata {
-            algorithm: "AES256".to_string(),
-            nonce: "dGVzdG5vbmNlMTI=".to_string(),
-            encrypted_data_key: Some("c29tZWtleQ==".to_string()),
-            kms_key_id: None,
-        };
-        let map = meta.to_metadata_map();
-        let restored = EncryptionMetadata::from_metadata(&map).unwrap();
-        assert_eq!(restored.algorithm, "AES256");
-        assert_eq!(restored.nonce, meta.nonce);
-        assert_eq!(restored.encrypted_data_key, meta.encrypted_data_key);
-    }
-
-    #[test]
-    fn test_is_encrypted() {
-        let mut meta = HashMap::new();
-        assert!(!EncryptionMetadata::is_encrypted(&meta));
-        meta.insert(
-            "x-amz-server-side-encryption".to_string(),
-            "AES256".to_string(),
-        );
-        assert!(EncryptionMetadata::is_encrypted(&meta));
-    }
-}

rust/myfsio-engine/crates/myfsio-crypto/src/hashing.rs

@@ -1,138 +0,0 @@
-use md5::{Digest, Md5};
-use sha2::Sha256;
-use std::io::Read;
-use std::path::Path;
-
-const CHUNK_SIZE: usize = 65536;
-
-pub fn md5_file(path: &Path) -> std::io::Result<String> {
-    let mut file = std::fs::File::open(path)?;
-    let mut hasher = Md5::new();
-    let mut buf = vec![0u8; CHUNK_SIZE];
-    loop {
-        let n = file.read(&mut buf)?;
-        if n == 0 {
-            break;
-        }
-        hasher.update(&buf[..n]);
-    }
-    Ok(format!("{:x}", hasher.finalize()))
-}
-
-pub fn md5_bytes(data: &[u8]) -> String {
-    let mut hasher = Md5::new();
-    hasher.update(data);
-    format!("{:x}", hasher.finalize())
-}
-
-pub fn sha256_file(path: &Path) -> std::io::Result<String> {
-    let mut file = std::fs::File::open(path)?;
-    let mut hasher = Sha256::new();
-    let mut buf = vec![0u8; CHUNK_SIZE];
-    loop {
-        let n = file.read(&mut buf)?;
-        if n == 0 {
-            break;
-        }
-        hasher.update(&buf[..n]);
-    }
-    Ok(format!("{:x}", hasher.finalize()))
-}
-
-pub fn sha256_bytes(data: &[u8]) -> String {
-    let mut hasher = Sha256::new();
-    hasher.update(data);
-    format!("{:x}", hasher.finalize())
-}
-
-pub fn md5_sha256_file(path: &Path) -> std::io::Result<(String, String)> {
-    let mut file = std::fs::File::open(path)?;
-    let mut md5_hasher = Md5::new();
-    let mut sha_hasher = Sha256::new();
-    let mut buf = vec![0u8; CHUNK_SIZE];
-    loop {
-        let n = file.read(&mut buf)?;
-        if n == 0 {
-            break;
-        }
-        md5_hasher.update(&buf[..n]);
-        sha_hasher.update(&buf[..n]);
-    }
-    Ok((
-        format!("{:x}", md5_hasher.finalize()),
-        format!("{:x}", sha_hasher.finalize()),
-    ))
-}
-
-pub async fn md5_file_async(path: &Path) -> std::io::Result<String> {
-    let path = path.to_owned();
-    tokio::task::spawn_blocking(move || md5_file(&path))
-        .await
-        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?
-}
-
-pub async fn sha256_file_async(path: &Path) -> std::io::Result<String> {
-    let path = path.to_owned();
-    tokio::task::spawn_blocking(move || sha256_file(&path))
-        .await
-        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?
-}
-
-pub async fn md5_sha256_file_async(path: &Path) -> std::io::Result<(String, String)> {
-    let path = path.to_owned();
-    tokio::task::spawn_blocking(move || md5_sha256_file(&path))
-        .await
-        .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::io::Write;
-
-    #[test]
-    fn test_md5_bytes() {
-        assert_eq!(md5_bytes(b""), "d41d8cd98f00b204e9800998ecf8427e");
-        assert_eq!(md5_bytes(b"hello"), "5d41402abc4b2a76b9719d911017c592");
-    }
-
-    #[test]
-    fn test_sha256_bytes() {
-        let hash = sha256_bytes(b"hello");
-        assert_eq!(
-            hash,
-            "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
-        );
-    }
-
-    #[test]
-    fn test_md5_file() {
-        let mut tmp = tempfile::NamedTempFile::new().unwrap();
-        tmp.write_all(b"hello").unwrap();
-        tmp.flush().unwrap();
-        let hash = md5_file(tmp.path()).unwrap();
-        assert_eq!(hash, "5d41402abc4b2a76b9719d911017c592");
-    }
-
-    #[test]
-    fn test_md5_sha256_file() {
-        let mut tmp = tempfile::NamedTempFile::new().unwrap();
-        tmp.write_all(b"hello").unwrap();
-        tmp.flush().unwrap();
-        let (md5, sha) = md5_sha256_file(tmp.path()).unwrap();
-        assert_eq!(md5, "5d41402abc4b2a76b9719d911017c592");
-        assert_eq!(
-            sha,
-            "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
-        );
-    }
-
-    #[tokio::test]
-    async fn test_md5_file_async() {
-        let mut tmp = tempfile::NamedTempFile::new().unwrap();
-        tmp.write_all(b"hello").unwrap();
-        tmp.flush().unwrap();
-        let hash = md5_file_async(tmp.path()).await.unwrap();
-        assert_eq!(hash, "5d41402abc4b2a76b9719d911017c592");
-    }
-}

rust/myfsio-engine/crates/myfsio-crypto/src/kms.rs

@@ -1,451 +0,0 @@
-use aes_gcm::aead::Aead;
-use aes_gcm::{Aes256Gcm, KeyInit, Nonce};
-use base64::engine::general_purpose::STANDARD as B64;
-use base64::Engine;
-use chrono::{DateTime, Utc};
-use rand::RngCore;
-use serde::{Deserialize, Serialize};
-use std::path::{Path, PathBuf};
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-use crate::aes_gcm::CryptoError;
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct KmsKey {
-    #[serde(rename = "KeyId")]
-    pub key_id: String,
-    #[serde(rename = "Arn")]
-    pub arn: String,
-    #[serde(rename = "Description")]
-    pub description: String,
-    #[serde(rename = "CreationDate")]
-    pub creation_date: DateTime<Utc>,
-    #[serde(rename = "Enabled")]
-    pub enabled: bool,
-    #[serde(rename = "KeyState")]
-    pub key_state: String,
-    #[serde(rename = "KeyUsage")]
-    pub key_usage: String,
-    #[serde(rename = "KeySpec")]
-    pub key_spec: String,
-    #[serde(rename = "EncryptedKeyMaterial")]
-    pub encrypted_key_material: String,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-struct KmsStore {
-    keys: Vec<KmsKey>,
-}
-
-pub struct KmsService {
-    keys_path: PathBuf,
-    master_key: Arc<RwLock<[u8; 32]>>,
-    keys: Arc<RwLock<Vec<KmsKey>>>,
-}
-
-impl KmsService {
-    pub async fn new(keys_dir: &Path) -> Result<Self, CryptoError> {
-        std::fs::create_dir_all(keys_dir).map_err(CryptoError::Io)?;
-
-        let keys_path = keys_dir.join("kms_keys.json");
-
-        let master_key = Self::load_or_create_master_key(&keys_dir.join("kms_master.key"))?;
-
-        let keys = if keys_path.exists() {
-            let data = std::fs::read_to_string(&keys_path).map_err(CryptoError::Io)?;
-            let store: KmsStore = serde_json::from_str(&data)
-                .map_err(|e| CryptoError::EncryptionFailed(format!("Bad KMS store: {}", e)))?;
-            store.keys
-        } else {
-            Vec::new()
-        };
-
-        Ok(Self {
-            keys_path,
-            master_key: Arc::new(RwLock::new(master_key)),
-            keys: Arc::new(RwLock::new(keys)),
-        })
-    }
-
-    fn load_or_create_master_key(path: &Path) -> Result<[u8; 32], CryptoError> {
-        if path.exists() {
-            let encoded = std::fs::read_to_string(path).map_err(CryptoError::Io)?;
-            let decoded = B64.decode(encoded.trim()).map_err(|e| {
-                CryptoError::EncryptionFailed(format!("Bad master key encoding: {}", e))
-            })?;
-            if decoded.len() != 32 {
-                return Err(CryptoError::InvalidKeySize(decoded.len()));
-            }
-            let mut key = [0u8; 32];
-            key.copy_from_slice(&decoded);
-            Ok(key)
-        } else {
-            let mut key = [0u8; 32];
-            rand::thread_rng().fill_bytes(&mut key);
-            let encoded = B64.encode(key);
-            std::fs::write(path, &encoded).map_err(CryptoError::Io)?;
-            Ok(key)
-        }
-    }
-
-    fn encrypt_key_material(
-        master_key: &[u8; 32],
-        plaintext_key: &[u8],
-    ) -> Result<String, CryptoError> {
-        let cipher = Aes256Gcm::new(master_key.into());
-        let mut nonce_bytes = [0u8; 12];
-        rand::thread_rng().fill_bytes(&mut nonce_bytes);
-        let nonce = Nonce::from_slice(&nonce_bytes);
-
-        let ciphertext = cipher
-            .encrypt(nonce, plaintext_key)
-            .map_err(|e| CryptoError::EncryptionFailed(e.to_string()))?;
-
-        let mut combined = Vec::with_capacity(12 + ciphertext.len());
-        combined.extend_from_slice(&nonce_bytes);
-        combined.extend_from_slice(&ciphertext);
-        Ok(B64.encode(&combined))
-    }
-
-    fn decrypt_key_material(
-        master_key: &[u8; 32],
-        encrypted_b64: &str,
-    ) -> Result<Vec<u8>, CryptoError> {
-        let combined = B64.decode(encrypted_b64).map_err(|e| {
-            CryptoError::EncryptionFailed(format!("Bad key material encoding: {}", e))
-        })?;
-        if combined.len() < 12 {
-            return Err(CryptoError::EncryptionFailed(
-                "Encrypted key material too short".to_string(),
-            ));
-        }
-
-        let (nonce_bytes, ciphertext) = combined.split_at(12);
-        let cipher = Aes256Gcm::new(master_key.into());
-        let nonce = Nonce::from_slice(nonce_bytes);
-
-        cipher
-            .decrypt(nonce, ciphertext)
-            .map_err(|_| CryptoError::DecryptionFailed(0))
-    }
-
-    async fn save(&self) -> Result<(), CryptoError> {
-        let keys = self.keys.read().await;
-        let store = KmsStore { keys: keys.clone() };
-        let json = serde_json::to_string_pretty(&store)
-            .map_err(|e| CryptoError::EncryptionFailed(e.to_string()))?;
-        std::fs::write(&self.keys_path, json).map_err(CryptoError::Io)?;
-        Ok(())
-    }
-
-    pub async fn create_key(&self, description: &str) -> Result<KmsKey, CryptoError> {
-        let key_id = uuid::Uuid::new_v4().to_string();
-        let arn = format!("arn:aws:kms:local:000000000000:key/{}", key_id);
-
-        let mut plaintext_key = [0u8; 32];
-        rand::thread_rng().fill_bytes(&mut plaintext_key);
-
-        let master = self.master_key.read().await;
-        let encrypted = Self::encrypt_key_material(&master, &plaintext_key)?;
-
-        let kms_key = KmsKey {
-            key_id: key_id.clone(),
-            arn,
-            description: description.to_string(),
-            creation_date: Utc::now(),
-            enabled: true,
-            key_state: "Enabled".to_string(),
-            key_usage: "ENCRYPT_DECRYPT".to_string(),
-            key_spec: "SYMMETRIC_DEFAULT".to_string(),
-            encrypted_key_material: encrypted,
-        };
-
-        self.keys.write().await.push(kms_key.clone());
-        self.save().await?;
-        Ok(kms_key)
-    }
-
-    pub async fn list_keys(&self) -> Vec<KmsKey> {
-        self.keys.read().await.clone()
-    }
-
-    pub async fn get_key(&self, key_id: &str) -> Option<KmsKey> {
-        let keys = self.keys.read().await;
-        keys.iter()
-            .find(|k| k.key_id == key_id || k.arn == key_id)
-            .cloned()
-    }
-
-    pub async fn delete_key(&self, key_id: &str) -> Result<bool, CryptoError> {
-        let mut keys = self.keys.write().await;
-        let len_before = keys.len();
-        keys.retain(|k| k.key_id != key_id && k.arn != key_id);
-        let removed = keys.len() < len_before;
-        drop(keys);
-        if removed {
-            self.save().await?;
-        }
-        Ok(removed)
-    }
-
-    pub async fn enable_key(&self, key_id: &str) -> Result<bool, CryptoError> {
-        let mut keys = self.keys.write().await;
-        if let Some(key) = keys.iter_mut().find(|k| k.key_id == key_id) {
-            key.enabled = true;
-            key.key_state = "Enabled".to_string();
-            drop(keys);
-            self.save().await?;
-            Ok(true)
-        } else {
-            Ok(false)
-        }
-    }
-
-    pub async fn disable_key(&self, key_id: &str) -> Result<bool, CryptoError> {
-        let mut keys = self.keys.write().await;
-        if let Some(key) = keys.iter_mut().find(|k| k.key_id == key_id) {
-            key.enabled = false;
-            key.key_state = "Disabled".to_string();
-            drop(keys);
-            self.save().await?;
-            Ok(true)
-        } else {
-            Ok(false)
-        }
-    }
-
-    pub async fn decrypt_data_key(&self, key_id: &str) -> Result<Vec<u8>, CryptoError> {
-        let keys = self.keys.read().await;
-        let key = keys
-            .iter()
-            .find(|k| k.key_id == key_id || k.arn == key_id)
-            .ok_or_else(|| CryptoError::EncryptionFailed("KMS key not found".to_string()))?;
-
-        if !key.enabled {
-            return Err(CryptoError::EncryptionFailed(
-                "KMS key is disabled".to_string(),
-            ));
-        }
-
-        let master = self.master_key.read().await;
-        Self::decrypt_key_material(&master, &key.encrypted_key_material)
-    }
-
-    pub async fn encrypt_data(
-        &self,
-        key_id: &str,
-        plaintext: &[u8],
-    ) -> Result<Vec<u8>, CryptoError> {
-        let data_key = self.decrypt_data_key(key_id).await?;
-        if data_key.len() != 32 {
-            return Err(CryptoError::InvalidKeySize(data_key.len()));
-        }
-
-        let key_arr: [u8; 32] = data_key.try_into().unwrap();
-        let cipher = Aes256Gcm::new(&key_arr.into());
-        let mut nonce_bytes = [0u8; 12];
-        rand::thread_rng().fill_bytes(&mut nonce_bytes);
-        let nonce = Nonce::from_slice(&nonce_bytes);
-
-        let ciphertext = cipher
-            .encrypt(nonce, plaintext)
-            .map_err(|e| CryptoError::EncryptionFailed(e.to_string()))?;
-
-        let mut result = Vec::with_capacity(12 + ciphertext.len());
-        result.extend_from_slice(&nonce_bytes);
-        result.extend_from_slice(&ciphertext);
-        Ok(result)
-    }
-
-    pub async fn decrypt_data(
-        &self,
-        key_id: &str,
-        ciphertext: &[u8],
-    ) -> Result<Vec<u8>, CryptoError> {
-        if ciphertext.len() < 12 {
-            return Err(CryptoError::EncryptionFailed(
-                "Ciphertext too short".to_string(),
-            ));
-        }
-
-        let data_key = self.decrypt_data_key(key_id).await?;
-        if data_key.len() != 32 {
-            return Err(CryptoError::InvalidKeySize(data_key.len()));
-        }
-
-        let key_arr: [u8; 32] = data_key.try_into().unwrap();
-        let (nonce_bytes, ct) = ciphertext.split_at(12);
-        let cipher = Aes256Gcm::new(&key_arr.into());
-        let nonce = Nonce::from_slice(nonce_bytes);
-
-        cipher
-            .decrypt(nonce, ct)
-            .map_err(|_| CryptoError::DecryptionFailed(0))
-    }
-
-    pub async fn generate_data_key(
-        &self,
-        key_id: &str,
-        num_bytes: usize,
-    ) -> Result<(Vec<u8>, Vec<u8>), CryptoError> {
-        let kms_key = self.decrypt_data_key(key_id).await?;
-        if kms_key.len() != 32 {
-            return Err(CryptoError::InvalidKeySize(kms_key.len()));
-        }
-
-        let mut plaintext_key = vec![0u8; num_bytes];
-        rand::thread_rng().fill_bytes(&mut plaintext_key);
-
-        let key_arr: [u8; 32] = kms_key.try_into().unwrap();
-        let cipher = Aes256Gcm::new(&key_arr.into());
-        let mut nonce_bytes = [0u8; 12];
-        rand::thread_rng().fill_bytes(&mut nonce_bytes);
-        let nonce = Nonce::from_slice(&nonce_bytes);
-
-        let encrypted = cipher
-            .encrypt(nonce, plaintext_key.as_slice())
-            .map_err(|e| CryptoError::EncryptionFailed(e.to_string()))?;
-
-        let mut wrapped = Vec::with_capacity(12 + encrypted.len());
-        wrapped.extend_from_slice(&nonce_bytes);
-        wrapped.extend_from_slice(&encrypted);
-
-        Ok((plaintext_key, wrapped))
-    }
-}
-
-pub async fn load_or_create_master_key(keys_dir: &Path) -> Result<[u8; 32], CryptoError> {
-    std::fs::create_dir_all(keys_dir).map_err(CryptoError::Io)?;
-    let path = keys_dir.join("master.key");
-
-    if path.exists() {
-        let encoded = std::fs::read_to_string(&path).map_err(CryptoError::Io)?;
-        let decoded = B64.decode(encoded.trim()).map_err(|e| {
-            CryptoError::EncryptionFailed(format!("Bad master key encoding: {}", e))
-        })?;
-        if decoded.len() != 32 {
-            return Err(CryptoError::InvalidKeySize(decoded.len()));
-        }
-        let mut key = [0u8; 32];
-        key.copy_from_slice(&decoded);
-        Ok(key)
-    } else {
-        let mut key = [0u8; 32];
-        rand::thread_rng().fill_bytes(&mut key);
-        let encoded = B64.encode(key);
-        std::fs::write(&path, &encoded).map_err(CryptoError::Io)?;
-        Ok(key)
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[tokio::test]
-    async fn test_create_and_list_keys() {
-        let dir = tempfile::tempdir().unwrap();
-        let kms = KmsService::new(dir.path()).await.unwrap();
-
-        let key = kms.create_key("test key").await.unwrap();
-        assert!(key.enabled);
-        assert_eq!(key.description, "test key");
-        assert!(key.key_id.len() > 0);
-
-        let keys = kms.list_keys().await;
-        assert_eq!(keys.len(), 1);
-        assert_eq!(keys[0].key_id, key.key_id);
-    }
-
-    #[tokio::test]
-    async fn test_enable_disable_key() {
-        let dir = tempfile::tempdir().unwrap();
-        let kms = KmsService::new(dir.path()).await.unwrap();
-
-        let key = kms.create_key("toggle").await.unwrap();
-        assert!(key.enabled);
-
-        kms.disable_key(&key.key_id).await.unwrap();
-        let k = kms.get_key(&key.key_id).await.unwrap();
-        assert!(!k.enabled);
-
-        kms.enable_key(&key.key_id).await.unwrap();
-        let k = kms.get_key(&key.key_id).await.unwrap();
-        assert!(k.enabled);
-    }
-
-    #[tokio::test]
-    async fn test_delete_key() {
-        let dir = tempfile::tempdir().unwrap();
-        let kms = KmsService::new(dir.path()).await.unwrap();
-
-        let key = kms.create_key("doomed").await.unwrap();
-        assert!(kms.delete_key(&key.key_id).await.unwrap());
-        assert!(kms.get_key(&key.key_id).await.is_none());
-        assert_eq!(kms.list_keys().await.len(), 0);
-    }
-
-    #[tokio::test]
-    async fn test_encrypt_decrypt_data() {
-        let dir = tempfile::tempdir().unwrap();
-        let kms = KmsService::new(dir.path()).await.unwrap();
-
-        let key = kms.create_key("enc-key").await.unwrap();
-        let plaintext = b"Hello, KMS!";
-
-        let ciphertext = kms.encrypt_data(&key.key_id, plaintext).await.unwrap();
-        assert_ne!(&ciphertext, plaintext);
-
-        let decrypted = kms.decrypt_data(&key.key_id, &ciphertext).await.unwrap();
-        assert_eq!(decrypted, plaintext);
-    }
-
-    #[tokio::test]
-    async fn test_generate_data_key() {
-        let dir = tempfile::tempdir().unwrap();
-        let kms = KmsService::new(dir.path()).await.unwrap();
-
-        let key = kms.create_key("data-key-gen").await.unwrap();
-        let (plaintext, wrapped) = kms.generate_data_key(&key.key_id, 32).await.unwrap();
-
-        assert_eq!(plaintext.len(), 32);
-        assert!(wrapped.len() > 32);
-    }
-
-    #[tokio::test]
-    async fn test_disabled_key_cannot_encrypt() {
-        let dir = tempfile::tempdir().unwrap();
-        let kms = KmsService::new(dir.path()).await.unwrap();
-
-        let key = kms.create_key("disabled").await.unwrap();
-        kms.disable_key(&key.key_id).await.unwrap();
-
-        let result = kms.encrypt_data(&key.key_id, b"test").await;
-        assert!(result.is_err());
-    }
-
-    #[tokio::test]
-    async fn test_persistence_across_reload() {
-        let dir = tempfile::tempdir().unwrap();
-
-        let key_id = {
-            let kms = KmsService::new(dir.path()).await.unwrap();
-            let key = kms.create_key("persistent").await.unwrap();
-            key.key_id
-        };
-
-        let kms2 = KmsService::new(dir.path()).await.unwrap();
-        let key = kms2.get_key(&key_id).await;
-        assert!(key.is_some());
-        assert_eq!(key.unwrap().description, "persistent");
-    }
-
-    #[tokio::test]
-    async fn test_master_key_roundtrip() {
-        let dir = tempfile::tempdir().unwrap();
-        let key1 = load_or_create_master_key(dir.path()).await.unwrap();
-        let key2 = load_or_create_master_key(dir.path()).await.unwrap();
-        assert_eq!(key1, key2);
-    }
-}

rust/myfsio-engine/crates/myfsio-crypto/src/lib.rs

@@ -1,4 +0,0 @@
-pub mod aes_gcm;
-pub mod encryption;
-pub mod hashing;
-pub mod kms;

rust/myfsio-engine/crates/myfsio-server/Cargo.toml

@@ -1,54 +0,0 @@
-[package]
-name = "myfsio-server"
-version.workspace = true
-edition.workspace = true
-
-[dependencies]
-myfsio-common = { path = "../myfsio-common" }
-myfsio-auth = { path = "../myfsio-auth" }
-myfsio-crypto = { path = "../myfsio-crypto" }
-myfsio-storage = { path = "../myfsio-storage" }
-myfsio-xml = { path = "../myfsio-xml" }
-base64 = { workspace = true }
-axum = { workspace = true }
-tokio = { workspace = true }
-tower = { workspace = true }
-tower-http = { workspace = true }
-hyper = { workspace = true }
-bytes = { workspace = true }
-serde = { workspace = true }
-serde_json = { workspace = true }
-tracing = { workspace = true }
-tracing-subscriber = { workspace = true }
-tokio-util = { workspace = true }
-chrono = { workspace = true }
-uuid = { workspace = true }
-futures = { workspace = true }
-http-body-util = "0.1"
-percent-encoding = { workspace = true }
-quick-xml = { workspace = true }
-mime_guess = "2"
-crc32fast = { workspace = true }
-duckdb = { workspace = true }
-roxmltree = "0.20"
-parking_lot = { workspace = true }
-regex = "1"
-multer = "3"
-reqwest = { workspace = true }
-aws-sdk-s3 = { workspace = true }
-aws-config = { workspace = true }
-aws-credential-types = { workspace = true }
-aws-smithy-types = { workspace = true }
-async-trait = { workspace = true }
-rand = "0.8"
-tera = { workspace = true }
-cookie = { workspace = true }
-subtle = { workspace = true }
-clap = { workspace = true }
-dotenvy = { workspace = true }
-sysinfo = "0.32"
-aes-gcm = { workspace = true }
-
-[dev-dependencies]
-tempfile = "3"
-tower = { workspace = true, features = ["util"] }

rust/myfsio-engine/crates/myfsio-server/src/config.rs

@@ -1,245 +0,0 @@
-use std::net::SocketAddr;
-use std::path::PathBuf;
-
-#[derive(Debug, Clone)]
-pub struct ServerConfig {
-    pub bind_addr: SocketAddr,
-    pub ui_bind_addr: SocketAddr,
-    pub storage_root: PathBuf,
-    pub region: String,
-    pub iam_config_path: PathBuf,
-    pub sigv4_timestamp_tolerance_secs: u64,
-    pub presigned_url_min_expiry: u64,
-    pub presigned_url_max_expiry: u64,
-    pub secret_key: Option<String>,
-    pub encryption_enabled: bool,
-    pub kms_enabled: bool,
-    pub gc_enabled: bool,
-    pub integrity_enabled: bool,
-    pub metrics_enabled: bool,
-    pub metrics_history_enabled: bool,
-    pub metrics_interval_minutes: u64,
-    pub metrics_retention_hours: u64,
-    pub metrics_history_interval_minutes: u64,
-    pub metrics_history_retention_hours: u64,
-    pub lifecycle_enabled: bool,
-    pub website_hosting_enabled: bool,
-    pub replication_connect_timeout_secs: u64,
-    pub replication_read_timeout_secs: u64,
-    pub replication_max_retries: u32,
-    pub replication_streaming_threshold_bytes: u64,
-    pub replication_max_failures_per_bucket: usize,
-    pub site_sync_enabled: bool,
-    pub site_sync_interval_secs: u64,
-    pub site_sync_batch_size: usize,
-    pub site_sync_connect_timeout_secs: u64,
-    pub site_sync_read_timeout_secs: u64,
-    pub site_sync_max_retries: u32,
-    pub site_sync_clock_skew_tolerance: f64,
-    pub ui_enabled: bool,
-    pub templates_dir: PathBuf,
-    pub static_dir: PathBuf,
-}
-
-impl ServerConfig {
-    pub fn from_env() -> Self {
-        let host = std::env::var("HOST").unwrap_or_else(|_| "127.0.0.1".to_string());
-        let port: u16 = std::env::var("PORT")
-            .unwrap_or_else(|_| "5000".to_string())
-            .parse()
-            .unwrap_or(5000);
-        let ui_port: u16 = std::env::var("UI_PORT")
-            .unwrap_or_else(|_| "5100".to_string())
-            .parse()
-            .unwrap_or(5100);
-        let storage_root = std::env::var("STORAGE_ROOT").unwrap_or_else(|_| "./data".to_string());
-        let region = std::env::var("AWS_REGION").unwrap_or_else(|_| "us-east-1".to_string());
-
-        let storage_path = PathBuf::from(&storage_root);
-        let iam_config_path = std::env::var("IAM_CONFIG")
-            .map(PathBuf::from)
-            .unwrap_or_else(|_| {
-                storage_path
-                    .join(".myfsio.sys")
-                    .join("config")
-                    .join("iam.json")
-            });
-
-        let sigv4_timestamp_tolerance_secs: u64 =
-            std::env::var("SIGV4_TIMESTAMP_TOLERANCE_SECONDS")
-                .unwrap_or_else(|_| "900".to_string())
-                .parse()
-                .unwrap_or(900);
-
-        let presigned_url_min_expiry: u64 = std::env::var("PRESIGNED_URL_MIN_EXPIRY_SECONDS")
-            .unwrap_or_else(|_| "1".to_string())
-            .parse()
-            .unwrap_or(1);
-
-        let presigned_url_max_expiry: u64 = std::env::var("PRESIGNED_URL_MAX_EXPIRY_SECONDS")
-            .unwrap_or_else(|_| "604800".to_string())
-            .parse()
-            .unwrap_or(604800);
-
-        let secret_key = {
-            let env_key = std::env::var("SECRET_KEY").ok();
-            match env_key {
-                Some(k) if !k.is_empty() && k != "dev-secret-key" => Some(k),
-                _ => {
-                    let secret_file = storage_path
-                        .join(".myfsio.sys")
-                        .join("config")
-                        .join(".secret");
-                    std::fs::read_to_string(&secret_file)
-                        .ok()
-                        .map(|s| s.trim().to_string())
-                }
-            }
-        };
-
-        let encryption_enabled = std::env::var("ENCRYPTION_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let kms_enabled = std::env::var("KMS_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let gc_enabled = std::env::var("GC_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let integrity_enabled = std::env::var("INTEGRITY_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let metrics_enabled = std::env::var("OPERATION_METRICS_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let metrics_history_enabled = std::env::var("METRICS_HISTORY_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let metrics_interval_minutes = parse_u64_env("OPERATION_METRICS_INTERVAL_MINUTES", 5);
-        let metrics_retention_hours = parse_u64_env("OPERATION_METRICS_RETENTION_HOURS", 24);
-        let metrics_history_interval_minutes = parse_u64_env("METRICS_HISTORY_INTERVAL_MINUTES", 5);
-        let metrics_history_retention_hours = parse_u64_env("METRICS_HISTORY_RETENTION_HOURS", 24);
-
-        let lifecycle_enabled = std::env::var("LIFECYCLE_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let website_hosting_enabled = std::env::var("WEBSITE_HOSTING_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-
-        let replication_connect_timeout_secs =
-            parse_u64_env("REPLICATION_CONNECT_TIMEOUT_SECONDS", 5);
-        let replication_read_timeout_secs = parse_u64_env("REPLICATION_READ_TIMEOUT_SECONDS", 30);
-        let replication_max_retries = parse_u64_env("REPLICATION_MAX_RETRIES", 2) as u32;
-        let replication_streaming_threshold_bytes =
-            parse_u64_env("REPLICATION_STREAMING_THRESHOLD_BYTES", 10_485_760);
-        let replication_max_failures_per_bucket =
-            parse_u64_env("REPLICATION_MAX_FAILURES_PER_BUCKET", 50) as usize;
-
-        let site_sync_enabled = std::env::var("SITE_SYNC_ENABLED")
-            .unwrap_or_else(|_| "false".to_string())
-            .to_lowercase()
-            == "true";
-        let site_sync_interval_secs = parse_u64_env("SITE_SYNC_INTERVAL_SECONDS", 60);
-        let site_sync_batch_size = parse_u64_env("SITE_SYNC_BATCH_SIZE", 100) as usize;
-        let site_sync_connect_timeout_secs = parse_u64_env("SITE_SYNC_CONNECT_TIMEOUT_SECONDS", 10);
-        let site_sync_read_timeout_secs = parse_u64_env("SITE_SYNC_READ_TIMEOUT_SECONDS", 120);
-        let site_sync_max_retries = parse_u64_env("SITE_SYNC_MAX_RETRIES", 2) as u32;
-        let site_sync_clock_skew_tolerance: f64 =
-            std::env::var("SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS")
-                .ok()
-                .and_then(|s| s.parse().ok())
-                .unwrap_or(1.0);
-
-        let ui_enabled = std::env::var("UI_ENABLED")
-            .unwrap_or_else(|_| "true".to_string())
-            .to_lowercase()
-            == "true";
-        let templates_dir = std::env::var("TEMPLATES_DIR")
-            .map(PathBuf::from)
-            .unwrap_or_else(|_| default_templates_dir());
-        let static_dir = std::env::var("STATIC_DIR")
-            .map(PathBuf::from)
-            .unwrap_or_else(|_| default_static_dir());
-
-        let host_ip: std::net::IpAddr = host.parse().unwrap();
-        Self {
-            bind_addr: SocketAddr::new(host_ip, port),
-            ui_bind_addr: SocketAddr::new(host_ip, ui_port),
-            storage_root: storage_path,
-            region,
-            iam_config_path,
-            sigv4_timestamp_tolerance_secs,
-            presigned_url_min_expiry,
-            presigned_url_max_expiry,
-            secret_key,
-            encryption_enabled,
-            kms_enabled,
-            gc_enabled,
-            integrity_enabled,
-            metrics_enabled,
-            metrics_history_enabled,
-            metrics_interval_minutes,
-            metrics_retention_hours,
-            metrics_history_interval_minutes,
-            metrics_history_retention_hours,
-            lifecycle_enabled,
-            website_hosting_enabled,
-            replication_connect_timeout_secs,
-            replication_read_timeout_secs,
-            replication_max_retries,
-            replication_streaming_threshold_bytes,
-            replication_max_failures_per_bucket,
-            site_sync_enabled,
-            site_sync_interval_secs,
-            site_sync_batch_size,
-            site_sync_connect_timeout_secs,
-            site_sync_read_timeout_secs,
-            site_sync_max_retries,
-            site_sync_clock_skew_tolerance,
-            ui_enabled,
-            templates_dir,
-            static_dir,
-        }
-    }
-}
-
-fn default_templates_dir() -> PathBuf {
-    let manifest_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
-    manifest_dir.join("templates")
-}
-
-fn default_static_dir() -> PathBuf {
-    let manifest_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
-    for candidate in [
-        manifest_dir.join("static"),
-        manifest_dir.join("..").join("..").join("..").join("static"),
-    ] {
-        if candidate.exists() {
-            return candidate;
-        }
-    }
-    manifest_dir.join("static")
-}
-
-fn parse_u64_env(key: &str, default: u64) -> u64 {
-    std::env::var(key)
-        .ok()
-        .and_then(|s| s.parse().ok())
-        .unwrap_or(default)
-}

rust/myfsio-engine/crates/myfsio-server/src/handlers/chunked.rs

@@ -1,184 +0,0 @@
-use std::pin::Pin;
-use std::task::{Context, Poll};
-
-use bytes::{Buf, BytesMut};
-use tokio::io::{AsyncRead, ReadBuf};
-
-enum State {
-    ReadSize,
-    ReadData(u64),
-    ReadTrailer,
-    Finished,
-}
-
-pub struct AwsChunkedStream<S> {
-    inner: S,
-    buffer: BytesMut,
-    state: State,
-    pending: BytesMut,
-    eof: bool,
-}
-
-impl<S> AwsChunkedStream<S> {
-    pub fn new(inner: S) -> Self {
-        Self {
-            inner,
-            buffer: BytesMut::with_capacity(8192),
-            state: State::ReadSize,
-            pending: BytesMut::new(),
-            eof: false,
-        }
-    }
-
-    fn find_crlf(&self) -> Option<usize> {
-        for i in 0..self.buffer.len().saturating_sub(1) {
-            if self.buffer[i] == b'\r' && self.buffer[i + 1] == b'\n' {
-                return Some(i);
-            }
-        }
-        None
-    }
-
-    fn parse_chunk_size(line: &[u8]) -> std::io::Result<u64> {
-        let text = std::str::from_utf8(line).map_err(|_| {
-            std::io::Error::new(
-                std::io::ErrorKind::InvalidData,
-                "invalid chunk size encoding",
-            )
-        })?;
-        let head = text.split(';').next().unwrap_or("").trim();
-        u64::from_str_radix(head, 16).map_err(|_| {
-            std::io::Error::new(
-                std::io::ErrorKind::InvalidData,
-                format!("invalid chunk size: {}", head),
-            )
-        })
-    }
-
-    fn try_advance(&mut self, out: &mut ReadBuf<'_>) -> std::io::Result<bool> {
-        loop {
-            if out.remaining() == 0 {
-                return Ok(true);
-            }
-
-            if !self.pending.is_empty() {
-                let take = std::cmp::min(self.pending.len(), out.remaining());
-                out.put_slice(&self.pending[..take]);
-                self.pending.advance(take);
-                continue;
-            }
-
-            match self.state {
-                State::Finished => return Ok(true),
-                State::ReadSize => {
-                    let idx = match self.find_crlf() {
-                        Some(i) => i,
-                        None => return Ok(false),
-                    };
-                    let line = self.buffer.split_to(idx);
-                    self.buffer.advance(2);
-                    let size = Self::parse_chunk_size(&line)?;
-                    if size == 0 {
-                        self.state = State::ReadTrailer;
-                    } else {
-                        self.state = State::ReadData(size);
-                    }
-                }
-                State::ReadData(remaining) => {
-                    if self.buffer.is_empty() {
-                        return Ok(false);
-                    }
-                    let avail = std::cmp::min(self.buffer.len() as u64, remaining) as usize;
-                    let take = std::cmp::min(avail, out.remaining());
-                    out.put_slice(&self.buffer[..take]);
-                    self.buffer.advance(take);
-                    let new_remaining = remaining - take as u64;
-                    if new_remaining == 0 {
-                        if self.buffer.len() < 2 {
-                            self.state = State::ReadData(0);
-                            return Ok(false);
-                        }
-                        if &self.buffer[..2] != b"\r\n" {
-                            return Err(std::io::Error::new(
-                                std::io::ErrorKind::InvalidData,
-                                "malformed chunk terminator",
-                            ));
-                        }
-                        self.buffer.advance(2);
-                        self.state = State::ReadSize;
-                    } else {
-                        self.state = State::ReadData(new_remaining);
-                    }
-                }
-                State::ReadTrailer => {
-                    let idx = match self.find_crlf() {
-                        Some(i) => i,
-                        None => return Ok(false),
-                    };
-                    if idx == 0 {
-                        self.buffer.advance(2);
-                        self.state = State::Finished;
-                    } else {
-                        self.buffer.advance(idx + 2);
-                    }
-                }
-            }
-        }
-    }
-}
-
-impl<S> AsyncRead for AwsChunkedStream<S>
-where
-    S: AsyncRead + Unpin,
-{
-    fn poll_read(
-        mut self: Pin<&mut Self>,
-        cx: &mut Context<'_>,
-        buf: &mut ReadBuf<'_>,
-    ) -> Poll<std::io::Result<()>> {
-        loop {
-            let before = buf.filled().len();
-            let done = match self.try_advance(buf) {
-                Ok(v) => v,
-                Err(e) => return Poll::Ready(Err(e)),
-            };
-            if buf.filled().len() > before {
-                return Poll::Ready(Ok(()));
-            }
-            if done {
-                return Poll::Ready(Ok(()));
-            }
-            if self.eof {
-                return Poll::Ready(Err(std::io::Error::new(
-                    std::io::ErrorKind::UnexpectedEof,
-                    "unexpected EOF in aws-chunked stream",
-                )));
-            }
-
-            let mut tmp = [0u8; 8192];
-            let mut rb = ReadBuf::new(&mut tmp);
-            match Pin::new(&mut self.inner).poll_read(cx, &mut rb) {
-                Poll::Ready(Ok(())) => {
-                    let n = rb.filled().len();
-                    if n == 0 {
-                        self.eof = true;
-                        continue;
-                    }
-                    self.buffer.extend_from_slice(rb.filled());
-                }
-                Poll::Ready(Err(e)) => return Poll::Ready(Err(e)),
-                Poll::Pending => return Poll::Pending,
-            }
-        }
-    }
-}
-
-pub fn decode_body(body: axum::body::Body) -> impl AsyncRead + Send + Unpin {
-    use futures::TryStreamExt;
-    let stream = tokio_util::io::StreamReader::new(
-        http_body_util::BodyStream::new(body)
-            .map_ok(|frame| frame.into_data().unwrap_or_default())
-            .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e)),
-    );
-    AwsChunkedStream::new(stream)
-}

rust/myfsio-engine/crates/myfsio-server/src/handlers/kms.rs

@@ -1,541 +0,0 @@
-use aes_gcm::aead::Aead;
-use aes_gcm::{Aes256Gcm, KeyInit, Nonce};
-use axum::body::Body;
-use axum::extract::State;
-use axum::http::StatusCode;
-use axum::response::{IntoResponse, Response};
-use base64::engine::general_purpose::STANDARD as B64;
-use base64::Engine;
-use rand::RngCore;
-use serde_json::{json, Value};
-
-use crate::state::AppState;
-
-fn json_ok(value: Value) -> Response {
-    (
-        StatusCode::OK,
-        [("content-type", "application/json")],
-        value.to_string(),
-    )
-        .into_response()
-}
-
-fn json_err(status: StatusCode, msg: &str) -> Response {
-    (
-        status,
-        [("content-type", "application/json")],
-        json!({"error": msg}).to_string(),
-    )
-        .into_response()
-}
-
-async fn read_json(body: Body) -> Result<Value, Response> {
-    let body_bytes = http_body_util::BodyExt::collect(body)
-        .await
-        .map_err(|_| json_err(StatusCode::BAD_REQUEST, "Invalid request body"))?
-        .to_bytes();
-    if body_bytes.is_empty() {
-        Ok(json!({}))
-    } else {
-        serde_json::from_slice(&body_bytes)
-            .map_err(|_| json_err(StatusCode::BAD_REQUEST, "Invalid JSON"))
-    }
-}
-
-fn require_kms(
-    state: &AppState,
-) -> Result<&std::sync::Arc<myfsio_crypto::kms::KmsService>, Response> {
-    state
-        .kms
-        .as_ref()
-        .ok_or_else(|| json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"))
-}
-
-fn decode_b64(value: &str, field: &str) -> Result<Vec<u8>, Response> {
-    B64.decode(value).map_err(|_| {
-        json_err(
-            StatusCode::BAD_REQUEST,
-            &format!("Invalid base64 {}", field),
-        )
-    })
-}
-
-fn require_str<'a>(value: &'a Value, names: &[&str], message: &str) -> Result<&'a str, Response> {
-    for name in names {
-        if let Some(found) = value.get(*name).and_then(|v| v.as_str()) {
-            return Ok(found);
-        }
-    }
-    Err(json_err(StatusCode::BAD_REQUEST, message))
-}
-
-pub async fn list_keys(State(state): State<AppState>) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-
-    let keys = kms.list_keys().await;
-    let keys_json: Vec<Value> = keys
-        .iter()
-        .map(|k| {
-            json!({
-                "KeyId": k.key_id,
-                "Arn": k.arn,
-                "Description": k.description,
-                "CreationDate": k.creation_date.to_rfc3339(),
-                "Enabled": k.enabled,
-                "KeyState": k.key_state,
-                "KeyUsage": k.key_usage,
-                "KeySpec": k.key_spec,
-            })
-        })
-        .collect();
-
-    json_ok(json!({"keys": keys_json}))
-}
-
-pub async fn create_key(State(state): State<AppState>, body: Body) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-
-    let description = req
-        .get("Description")
-        .or_else(|| req.get("description"))
-        .and_then(|d| d.as_str())
-        .unwrap_or("");
-
-    match kms.create_key(description).await {
-        Ok(key) => json_ok(json!({
-            "KeyId": key.key_id,
-            "Arn": key.arn,
-            "Description": key.description,
-            "CreationDate": key.creation_date.to_rfc3339(),
-            "Enabled": key.enabled,
-            "KeyState": key.key_state,
-        })),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn get_key(
-    State(state): State<AppState>,
-    axum::extract::Path(key_id): axum::extract::Path<String>,
-) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-
-    match kms.get_key(&key_id).await {
-        Some(key) => json_ok(json!({
-            "KeyId": key.key_id,
-            "Arn": key.arn,
-            "Description": key.description,
-            "CreationDate": key.creation_date.to_rfc3339(),
-            "Enabled": key.enabled,
-            "KeyState": key.key_state,
-            "KeyUsage": key.key_usage,
-            "KeySpec": key.key_spec,
-        })),
-        None => json_err(StatusCode::NOT_FOUND, "Key not found"),
-    }
-}
-
-pub async fn delete_key(
-    State(state): State<AppState>,
-    axum::extract::Path(key_id): axum::extract::Path<String>,
-) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-
-    match kms.delete_key(&key_id).await {
-        Ok(true) => StatusCode::NO_CONTENT.into_response(),
-        Ok(false) => json_err(StatusCode::NOT_FOUND, "Key not found"),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn enable_key(
-    State(state): State<AppState>,
-    axum::extract::Path(key_id): axum::extract::Path<String>,
-) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-
-    match kms.enable_key(&key_id).await {
-        Ok(true) => json_ok(json!({"status": "enabled"})),
-        Ok(false) => json_err(StatusCode::NOT_FOUND, "Key not found"),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn disable_key(
-    State(state): State<AppState>,
-    axum::extract::Path(key_id): axum::extract::Path<String>,
-) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-
-    match kms.disable_key(&key_id).await {
-        Ok(true) => json_ok(json!({"status": "disabled"})),
-        Ok(false) => json_err(StatusCode::NOT_FOUND, "Key not found"),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn encrypt(State(state): State<AppState>, body: Body) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-
-    let key_id = match require_str(&req, &["KeyId", "key_id"], "Missing KeyId") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let plaintext_b64 = match require_str(&req, &["Plaintext", "plaintext"], "Missing Plaintext") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let plaintext = match decode_b64(plaintext_b64, "Plaintext") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-
-    match kms.encrypt_data(key_id, &plaintext).await {
-        Ok(ct) => json_ok(json!({
-            "KeyId": key_id,
-            "CiphertextBlob": B64.encode(&ct),
-        })),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn decrypt(State(state): State<AppState>, body: Body) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-
-    let key_id = match require_str(&req, &["KeyId", "key_id"], "Missing KeyId") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let ciphertext_b64 = match require_str(
-        &req,
-        &["CiphertextBlob", "ciphertext_blob"],
-        "Missing CiphertextBlob",
-    ) {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let ciphertext = match decode_b64(ciphertext_b64, "CiphertextBlob") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-
-    match kms.decrypt_data(key_id, &ciphertext).await {
-        Ok(pt) => json_ok(json!({
-            "KeyId": key_id,
-            "Plaintext": B64.encode(&pt),
-        })),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn generate_data_key(State(state): State<AppState>, body: Body) -> Response {
-    generate_data_key_inner(state, body, true).await
-}
-
-pub async fn generate_data_key_without_plaintext(
-    State(state): State<AppState>,
-    body: Body,
-) -> Response {
-    generate_data_key_inner(state, body, false).await
-}
-
-async fn generate_data_key_inner(state: AppState, body: Body, include_plaintext: bool) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-
-    let key_id = match require_str(&req, &["KeyId", "key_id"], "Missing KeyId") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let num_bytes = req
-        .get("NumberOfBytes")
-        .and_then(|v| v.as_u64())
-        .unwrap_or(32) as usize;
-
-    if !(1..=1024).contains(&num_bytes) {
-        return json_err(StatusCode::BAD_REQUEST, "NumberOfBytes must be 1-1024");
-    }
-
-    match kms.generate_data_key(key_id, num_bytes).await {
-        Ok((plaintext, wrapped)) => {
-            let mut value = json!({
-                "KeyId": key_id,
-                "CiphertextBlob": B64.encode(&wrapped),
-            });
-            if include_plaintext {
-                value["Plaintext"] = json!(B64.encode(&plaintext));
-            }
-            json_ok(value)
-        }
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn re_encrypt(State(state): State<AppState>, body: Body) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-
-    let ciphertext_b64 = match require_str(
-        &req,
-        &["CiphertextBlob", "ciphertext_blob"],
-        "CiphertextBlob is required",
-    ) {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let destination_key_id = match require_str(
-        &req,
-        &["DestinationKeyId", "destination_key_id"],
-        "DestinationKeyId is required",
-    ) {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let ciphertext = match decode_b64(ciphertext_b64, "CiphertextBlob") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-
-    let keys = kms.list_keys().await;
-    let mut source_key_id: Option<String> = None;
-    let mut plaintext: Option<Vec<u8>> = None;
-    for key in keys {
-        if !key.enabled {
-            continue;
-        }
-        if let Ok(value) = kms.decrypt_data(&key.key_id, &ciphertext).await {
-            source_key_id = Some(key.key_id);
-            plaintext = Some(value);
-            break;
-        }
-    }
-
-    let Some(source_key_id) = source_key_id else {
-        return json_err(
-            StatusCode::BAD_REQUEST,
-            "Could not determine source key for CiphertextBlob",
-        );
-    };
-    let plaintext = plaintext.unwrap_or_default();
-
-    match kms.encrypt_data(destination_key_id, &plaintext).await {
-        Ok(new_ciphertext) => json_ok(json!({
-            "CiphertextBlob": B64.encode(&new_ciphertext),
-            "SourceKeyId": source_key_id,
-            "KeyId": destination_key_id,
-        })),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}
-
-pub async fn generate_random(State(state): State<AppState>, body: Body) -> Response {
-    if let Err(response) = require_kms(&state) {
-        return response;
-    }
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-    let num_bytes = req
-        .get("NumberOfBytes")
-        .and_then(|v| v.as_u64())
-        .unwrap_or(32) as usize;
-
-    if !(1..=1024).contains(&num_bytes) {
-        return json_err(StatusCode::BAD_REQUEST, "NumberOfBytes must be 1-1024");
-    }
-
-    let mut bytes = vec![0u8; num_bytes];
-    rand::thread_rng().fill_bytes(&mut bytes);
-    json_ok(json!({
-        "Plaintext": B64.encode(bytes),
-    }))
-}
-
-pub async fn client_generate_key(State(state): State<AppState>) -> Response {
-    let _ = state;
-
-    let mut key = [0u8; 32];
-    rand::thread_rng().fill_bytes(&mut key);
-    json_ok(json!({
-        "Key": B64.encode(key),
-        "Algorithm": "AES-256-GCM",
-        "KeySize": 32,
-    }))
-}
-
-pub async fn client_encrypt(State(state): State<AppState>, body: Body) -> Response {
-    let _ = state;
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-    let plaintext_b64 =
-        match require_str(&req, &["Plaintext", "plaintext"], "Plaintext is required") {
-            Ok(value) => value,
-            Err(response) => return response,
-        };
-    let key_b64 = match require_str(&req, &["Key", "key"], "Key is required") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-
-    let plaintext = match decode_b64(plaintext_b64, "Plaintext") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let key_bytes = match decode_b64(key_b64, "Key") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    if key_bytes.len() != 32 {
-        return json_err(StatusCode::BAD_REQUEST, "Key must decode to 32 bytes");
-    }
-
-    let cipher = match Aes256Gcm::new_from_slice(&key_bytes) {
-        Ok(cipher) => cipher,
-        Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid encryption key"),
-    };
-    let mut nonce_bytes = [0u8; 12];
-    rand::thread_rng().fill_bytes(&mut nonce_bytes);
-    let nonce = Nonce::from_slice(&nonce_bytes);
-
-    match cipher.encrypt(nonce, plaintext.as_ref()) {
-        Ok(ciphertext) => json_ok(json!({
-            "Ciphertext": B64.encode(ciphertext),
-            "Nonce": B64.encode(nonce_bytes),
-            "Algorithm": "AES-256-GCM",
-        })),
-        Err(e) => json_err(StatusCode::BAD_REQUEST, &e.to_string()),
-    }
-}
-
-pub async fn client_decrypt(State(state): State<AppState>, body: Body) -> Response {
-    let _ = state;
-    let req = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-    let ciphertext_b64 = match require_str(
-        &req,
-        &["Ciphertext", "ciphertext"],
-        "Ciphertext is required",
-    ) {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let nonce_b64 = match require_str(&req, &["Nonce", "nonce"], "Nonce is required") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let key_b64 = match require_str(&req, &["Key", "key"], "Key is required") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-
-    let ciphertext = match decode_b64(ciphertext_b64, "Ciphertext") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let nonce_bytes = match decode_b64(nonce_b64, "Nonce") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    let key_bytes = match decode_b64(key_b64, "Key") {
-        Ok(value) => value,
-        Err(response) => return response,
-    };
-    if key_bytes.len() != 32 {
-        return json_err(StatusCode::BAD_REQUEST, "Key must decode to 32 bytes");
-    }
-    if nonce_bytes.len() != 12 {
-        return json_err(StatusCode::BAD_REQUEST, "Nonce must decode to 12 bytes");
-    }
-
-    let cipher = match Aes256Gcm::new_from_slice(&key_bytes) {
-        Ok(cipher) => cipher,
-        Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid encryption key"),
-    };
-    let nonce = Nonce::from_slice(&nonce_bytes);
-
-    match cipher.decrypt(nonce, ciphertext.as_ref()) {
-        Ok(plaintext) => json_ok(json!({
-            "Plaintext": B64.encode(plaintext),
-        })),
-        Err(e) => json_err(StatusCode::BAD_REQUEST, &e.to_string()),
-    }
-}
-
-pub async fn materials(
-    State(state): State<AppState>,
-    axum::extract::Path(key_id): axum::extract::Path<String>,
-    body: Body,
-) -> Response {
-    let kms = match require_kms(&state) {
-        Ok(kms) => kms,
-        Err(response) => return response,
-    };
-    let _ = match read_json(body).await {
-        Ok(req) => req,
-        Err(response) => return response,
-    };
-
-    match kms.generate_data_key(&key_id, 32).await {
-        Ok((plaintext, wrapped)) => json_ok(json!({
-            "PlaintextKey": B64.encode(plaintext),
-            "EncryptedKey": B64.encode(wrapped),
-            "KeyId": key_id,
-            "Algorithm": "AES-256-GCM",
-            "KeyWrapAlgorithm": "kms",
-        })),
-        Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()),
-    }
-}

rust/myfsio-engine/crates/myfsio-server/src/handlers/select.rs

@@ -1,578 +0,0 @@
-use std::collections::HashMap;
-use std::path::{Path, PathBuf};
-
-use axum::body::Body;
-use axum::http::{HeaderMap, HeaderName, StatusCode};
-use axum::response::{IntoResponse, Response};
-use base64::Engine;
-use bytes::Bytes;
-use crc32fast::Hasher;
-use duckdb::types::ValueRef;
-use duckdb::Connection;
-use futures::stream;
-use http_body_util::BodyExt;
-use myfsio_common::error::{S3Error, S3ErrorCode};
-use myfsio_storage::traits::StorageEngine;
-
-use crate::state::AppState;
-
-#[cfg(target_os = "windows")]
-#[link(name = "Rstrtmgr")]
-extern "system" {}
-
-const CHUNK_SIZE: usize = 65_536;
-
-pub async fn post_select_object_content(
-    state: &AppState,
-    bucket: &str,
-    key: &str,
-    headers: &HeaderMap,
-    body: Body,
-) -> Response {
-    if let Some(resp) = require_xml_content_type(headers) {
-        return resp;
-    }
-
-    let body_bytes = match body.collect().await {
-        Ok(collected) => collected.to_bytes(),
-        Err(_) => {
-            return s3_error_response(S3Error::new(
-                S3ErrorCode::MalformedXML,
-                "Unable to parse XML document",
-            ));
-        }
-    };
-
-    let request = match parse_select_request(&body_bytes) {
-        Ok(r) => r,
-        Err(err) => return s3_error_response(err),
-    };
-
-    let object_path = match state.storage.get_object_path(bucket, key).await {
-        Ok(path) => path,
-        Err(_) => {
-            return s3_error_response(S3Error::new(S3ErrorCode::NoSuchKey, "Object not found"));
-        }
-    };
-
-    let join_res =
-        tokio::task::spawn_blocking(move || execute_select_query(object_path, request)).await;
-    let chunks = match join_res {
-        Ok(Ok(chunks)) => chunks,
-        Ok(Err(message)) => {
-            return s3_error_response(S3Error::new(S3ErrorCode::InvalidRequest, message));
-        }
-        Err(_) => {
-            return s3_error_response(S3Error::new(
-                S3ErrorCode::InternalError,
-                "SelectObjectContent execution failed",
-            ));
-        }
-    };
-
-    let bytes_returned: usize = chunks.iter().map(|c| c.len()).sum();
-    let mut events: Vec<Bytes> = Vec::with_capacity(chunks.len() + 2);
-    for chunk in chunks {
-        events.push(Bytes::from(encode_select_event("Records", &chunk)));
-    }
-
-    let stats_payload = build_stats_xml(0, bytes_returned);
-    events.push(Bytes::from(encode_select_event(
-        "Stats",
-        stats_payload.as_bytes(),
-    )));
-    events.push(Bytes::from(encode_select_event("End", b"")));
-
-    let stream = stream::iter(events.into_iter().map(Ok::<Bytes, std::io::Error>));
-    let body = Body::from_stream(stream);
-
-    let mut response = (StatusCode::OK, body).into_response();
-    response.headers_mut().insert(
-        HeaderName::from_static("content-type"),
-        "application/octet-stream".parse().unwrap(),
-    );
-    response.headers_mut().insert(
-        HeaderName::from_static("x-amz-request-charged"),
-        "requester".parse().unwrap(),
-    );
-    response
-}
-
-#[derive(Clone)]
-struct SelectRequest {
-    expression: String,
-    input_format: InputFormat,
-    output_format: OutputFormat,
-}
-
-#[derive(Clone)]
-enum InputFormat {
-    Csv(CsvInputConfig),
-    Json(JsonInputConfig),
-    Parquet,
-}
-
-#[derive(Clone)]
-struct CsvInputConfig {
-    file_header_info: String,
-    field_delimiter: String,
-    quote_character: String,
-}
-
-#[derive(Clone)]
-struct JsonInputConfig {
-    json_type: String,
-}
-
-#[derive(Clone)]
-enum OutputFormat {
-    Csv(CsvOutputConfig),
-    Json(JsonOutputConfig),
-}
-
-#[derive(Clone)]
-struct CsvOutputConfig {
-    field_delimiter: String,
-    record_delimiter: String,
-    quote_character: String,
-}
-
-#[derive(Clone)]
-struct JsonOutputConfig {
-    record_delimiter: String,
-}
-
-fn parse_select_request(payload: &[u8]) -> Result<SelectRequest, S3Error> {
-    let xml = String::from_utf8_lossy(payload);
-    let doc = roxmltree::Document::parse(&xml)
-        .map_err(|_| S3Error::new(S3ErrorCode::MalformedXML, "Unable to parse XML document"))?;
-
-    let root = doc.root_element();
-    if root.tag_name().name() != "SelectObjectContentRequest" {
-        return Err(S3Error::new(
-            S3ErrorCode::MalformedXML,
-            "Root element must be SelectObjectContentRequest",
-        ));
-    }
-
-    let expression = child_text(&root, "Expression")
-        .filter(|v| !v.is_empty())
-        .ok_or_else(|| S3Error::new(S3ErrorCode::InvalidRequest, "Expression is required"))?;
-
-    let expression_type = child_text(&root, "ExpressionType").unwrap_or_else(|| "SQL".to_string());
-    if !expression_type.eq_ignore_ascii_case("SQL") {
-        return Err(S3Error::new(
-            S3ErrorCode::InvalidRequest,
-            "Only SQL expression type is supported",
-        ));
-    }
-
-    let input_node = child(&root, "InputSerialization").ok_or_else(|| {
-        S3Error::new(
-            S3ErrorCode::InvalidRequest,
-            "InputSerialization is required",
-        )
-    })?;
-    let output_node = child(&root, "OutputSerialization").ok_or_else(|| {
-        S3Error::new(
-            S3ErrorCode::InvalidRequest,
-            "OutputSerialization is required",
-        )
-    })?;
-
-    let input_format = parse_input_format(&input_node)?;
-    let output_format = parse_output_format(&output_node)?;
-
-    Ok(SelectRequest {
-        expression,
-        input_format,
-        output_format,
-    })
-}
-
-fn parse_input_format(node: &roxmltree::Node<'_, '_>) -> Result<InputFormat, S3Error> {
-    if let Some(csv_node) = child(node, "CSV") {
-        return Ok(InputFormat::Csv(CsvInputConfig {
-            file_header_info: child_text(&csv_node, "FileHeaderInfo")
-                .unwrap_or_else(|| "NONE".to_string())
-                .to_ascii_uppercase(),
-            field_delimiter: child_text(&csv_node, "FieldDelimiter")
-                .unwrap_or_else(|| ",".to_string()),
-            quote_character: child_text(&csv_node, "QuoteCharacter")
-                .unwrap_or_else(|| "\"".to_string()),
-        }));
-    }
-
-    if let Some(json_node) = child(node, "JSON") {
-        return Ok(InputFormat::Json(JsonInputConfig {
-            json_type: child_text(&json_node, "Type")
-                .unwrap_or_else(|| "DOCUMENT".to_string())
-                .to_ascii_uppercase(),
-        }));
-    }
-
-    if child(node, "Parquet").is_some() {
-        return Ok(InputFormat::Parquet);
-    }
-
-    Err(S3Error::new(
-        S3ErrorCode::InvalidRequest,
-        "InputSerialization must specify CSV, JSON, or Parquet",
-    ))
-}
-
-fn parse_output_format(node: &roxmltree::Node<'_, '_>) -> Result<OutputFormat, S3Error> {
-    if let Some(csv_node) = child(node, "CSV") {
-        return Ok(OutputFormat::Csv(CsvOutputConfig {
-            field_delimiter: child_text(&csv_node, "FieldDelimiter")
-                .unwrap_or_else(|| ",".to_string()),
-            record_delimiter: child_text(&csv_node, "RecordDelimiter")
-                .unwrap_or_else(|| "\n".to_string()),
-            quote_character: child_text(&csv_node, "QuoteCharacter")
-                .unwrap_or_else(|| "\"".to_string()),
-        }));
-    }
-
-    if let Some(json_node) = child(node, "JSON") {
-        return Ok(OutputFormat::Json(JsonOutputConfig {
-            record_delimiter: child_text(&json_node, "RecordDelimiter")
-                .unwrap_or_else(|| "\n".to_string()),
-        }));
-    }
-
-    Err(S3Error::new(
-        S3ErrorCode::InvalidRequest,
-        "OutputSerialization must specify CSV or JSON",
-    ))
-}
-
-fn child<'a, 'input>(
-    node: &'a roxmltree::Node<'a, 'input>,
-    name: &str,
-) -> Option<roxmltree::Node<'a, 'input>> {
-    node.children()
-        .find(|n| n.is_element() && n.tag_name().name() == name)
-}
-
-fn child_text(node: &roxmltree::Node<'_, '_>, name: &str) -> Option<String> {
-    child(node, name)
-        .and_then(|n| n.text())
-        .map(|s| s.to_string())
-}
-
-fn execute_select_query(path: PathBuf, request: SelectRequest) -> Result<Vec<Vec<u8>>, String> {
-    let conn =
-        Connection::open_in_memory().map_err(|e| format!("DuckDB connection error: {}", e))?;
-
-    load_input_table(&conn, &path, &request.input_format)?;
-
-    let expression = request
-        .expression
-        .replace("s3object", "data")
-        .replace("S3Object", "data");
-
-    let mut stmt = conn
-        .prepare(&expression)
-        .map_err(|e| format!("SQL execution error: {}", e))?;
-    let mut rows = stmt
-        .query([])
-        .map_err(|e| format!("SQL execution error: {}", e))?;
-    let stmt_ref = rows
-        .as_ref()
-        .ok_or_else(|| "SQL execution error: statement metadata unavailable".to_string())?;
-    let col_count = stmt_ref.column_count();
-    let mut columns: Vec<String> = Vec::with_capacity(col_count);
-    for i in 0..col_count {
-        let name = stmt_ref
-            .column_name(i)
-            .map(|s| s.to_string())
-            .unwrap_or_else(|_| format!("_{}", i));
-        columns.push(name);
-    }
-
-    match request.output_format {
-        OutputFormat::Csv(cfg) => collect_csv_chunks(&mut rows, col_count, cfg),
-        OutputFormat::Json(cfg) => collect_json_chunks(&mut rows, col_count, &columns, cfg),
-    }
-}
-
-fn load_input_table(conn: &Connection, path: &Path, input: &InputFormat) -> Result<(), String> {
-    let path_str = path.to_string_lossy().replace('\\', "/");
-    match input {
-        InputFormat::Csv(cfg) => {
-            let header = cfg.file_header_info == "USE" || cfg.file_header_info == "IGNORE";
-            let delimiter = normalize_single_char(&cfg.field_delimiter, ',');
-            let quote = normalize_single_char(&cfg.quote_character, '"');
-
-            let sql = format!(
-                "CREATE TABLE data AS SELECT * FROM read_csv('{}', header={}, delim='{}', quote='{}')",
-                sql_escape(&path_str),
-                if header { "true" } else { "false" },
-                sql_escape(&delimiter),
-                sql_escape(&quote)
-            );
-            conn.execute_batch(&sql)
-                .map_err(|e| format!("Failed loading CSV data: {}", e))?;
-        }
-        InputFormat::Json(cfg) => {
-            let format = if cfg.json_type == "LINES" {
-                "newline_delimited"
-            } else {
-                "array"
-            };
-            let sql = format!(
-                "CREATE TABLE data AS SELECT * FROM read_json_auto('{}', format='{}')",
-                sql_escape(&path_str),
-                format
-            );
-            conn.execute_batch(&sql)
-                .map_err(|e| format!("Failed loading JSON data: {}", e))?;
-        }
-        InputFormat::Parquet => {
-            let sql = format!(
-                "CREATE TABLE data AS SELECT * FROM read_parquet('{}')",
-                sql_escape(&path_str)
-            );
-            conn.execute_batch(&sql)
-                .map_err(|e| format!("Failed loading Parquet data: {}", e))?;
-        }
-    }
-    Ok(())
-}
-
-fn sql_escape(value: &str) -> String {
-    value.replace('\'', "''")
-}
-
-fn normalize_single_char(value: &str, default_char: char) -> String {
-    value.chars().next().unwrap_or(default_char).to_string()
-}
-
-fn collect_csv_chunks(
-    rows: &mut duckdb::Rows<'_>,
-    col_count: usize,
-    cfg: CsvOutputConfig,
-) -> Result<Vec<Vec<u8>>, String> {
-    let delimiter = cfg.field_delimiter;
-    let record_delimiter = cfg.record_delimiter;
-    let quote = cfg.quote_character;
-
-    let mut chunks: Vec<Vec<u8>> = Vec::new();
-    let mut buffer = String::new();
-
-    while let Some(row) = rows
-        .next()
-        .map_err(|e| format!("SQL execution error: {}", e))?
-    {
-        let mut fields: Vec<String> = Vec::with_capacity(col_count);
-        for i in 0..col_count {
-            let value = row
-                .get_ref(i)
-                .map_err(|e| format!("SQL execution error: {}", e))?;
-            if matches!(value, ValueRef::Null) {
-                fields.push(String::new());
-                continue;
-            }
-
-            let mut text = value_ref_to_string(value);
-            if text.contains(&delimiter)
-                || text.contains(&quote)
-                || text.contains(&record_delimiter)
-            {
-                text = text.replace(&quote, &(quote.clone() + &quote));
-                text = format!("{}{}{}", quote, text, quote);
-            }
-            fields.push(text);
-        }
-        buffer.push_str(&fields.join(&delimiter));
-        buffer.push_str(&record_delimiter);
-
-        while buffer.len() >= CHUNK_SIZE {
-            let rest = buffer.split_off(CHUNK_SIZE);
-            chunks.push(buffer.into_bytes());
-            buffer = rest;
-        }
-    }
-
-    if !buffer.is_empty() {
-        chunks.push(buffer.into_bytes());
-    }
-    Ok(chunks)
-}
-
-fn collect_json_chunks(
-    rows: &mut duckdb::Rows<'_>,
-    col_count: usize,
-    columns: &[String],
-    cfg: JsonOutputConfig,
-) -> Result<Vec<Vec<u8>>, String> {
-    let record_delimiter = cfg.record_delimiter;
-    let mut chunks: Vec<Vec<u8>> = Vec::new();
-    let mut buffer = String::new();
-
-    while let Some(row) = rows
-        .next()
-        .map_err(|e| format!("SQL execution error: {}", e))?
-    {
-        let mut record: HashMap<String, serde_json::Value> = HashMap::with_capacity(col_count);
-        for i in 0..col_count {
-            let value = row
-                .get_ref(i)
-                .map_err(|e| format!("SQL execution error: {}", e))?;
-            let key = columns.get(i).cloned().unwrap_or_else(|| format!("_{}", i));
-            record.insert(key, value_ref_to_json(value));
-        }
-        let line = serde_json::to_string(&record)
-            .map_err(|e| format!("JSON output encoding failed: {}", e))?;
-        buffer.push_str(&line);
-        buffer.push_str(&record_delimiter);
-
-        while buffer.len() >= CHUNK_SIZE {
-            let rest = buffer.split_off(CHUNK_SIZE);
-            chunks.push(buffer.into_bytes());
-            buffer = rest;
-        }
-    }
-
-    if !buffer.is_empty() {
-        chunks.push(buffer.into_bytes());
-    }
-    Ok(chunks)
-}
-
-fn value_ref_to_string(value: ValueRef<'_>) -> String {
-    match value {
-        ValueRef::Null => String::new(),
-        ValueRef::Boolean(v) => v.to_string(),
-        ValueRef::TinyInt(v) => v.to_string(),
-        ValueRef::SmallInt(v) => v.to_string(),
-        ValueRef::Int(v) => v.to_string(),
-        ValueRef::BigInt(v) => v.to_string(),
-        ValueRef::UTinyInt(v) => v.to_string(),
-        ValueRef::USmallInt(v) => v.to_string(),
-        ValueRef::UInt(v) => v.to_string(),
-        ValueRef::UBigInt(v) => v.to_string(),
-        ValueRef::Float(v) => v.to_string(),
-        ValueRef::Double(v) => v.to_string(),
-        ValueRef::Decimal(v) => v.to_string(),
-        ValueRef::Text(v) => String::from_utf8_lossy(v).into_owned(),
-        ValueRef::Blob(v) => base64::engine::general_purpose::STANDARD.encode(v),
-        _ => format!("{:?}", value),
-    }
-}
-
-fn value_ref_to_json(value: ValueRef<'_>) -> serde_json::Value {
-    match value {
-        ValueRef::Null => serde_json::Value::Null,
-        ValueRef::Boolean(v) => serde_json::Value::Bool(v),
-        ValueRef::TinyInt(v) => serde_json::json!(v),
-        ValueRef::SmallInt(v) => serde_json::json!(v),
-        ValueRef::Int(v) => serde_json::json!(v),
-        ValueRef::BigInt(v) => serde_json::json!(v),
-        ValueRef::UTinyInt(v) => serde_json::json!(v),
-        ValueRef::USmallInt(v) => serde_json::json!(v),
-        ValueRef::UInt(v) => serde_json::json!(v),
-        ValueRef::UBigInt(v) => serde_json::json!(v),
-        ValueRef::Float(v) => serde_json::json!(v),
-        ValueRef::Double(v) => serde_json::json!(v),
-        ValueRef::Decimal(v) => serde_json::Value::String(v.to_string()),
-        ValueRef::Text(v) => serde_json::Value::String(String::from_utf8_lossy(v).into_owned()),
-        ValueRef::Blob(v) => {
-            serde_json::Value::String(base64::engine::general_purpose::STANDARD.encode(v))
-        }
-        _ => serde_json::Value::String(format!("{:?}", value)),
-    }
-}
-
-fn require_xml_content_type(headers: &HeaderMap) -> Option<Response> {
-    let value = headers
-        .get("content-type")
-        .and_then(|v| v.to_str().ok())
-        .unwrap_or("")
-        .trim();
-    if value.is_empty() {
-        return None;
-    }
-    let lowered = value.to_ascii_lowercase();
-    if lowered.starts_with("application/xml") || lowered.starts_with("text/xml") {
-        return None;
-    }
-    Some(s3_error_response(S3Error::new(
-        S3ErrorCode::InvalidRequest,
-        "Content-Type must be application/xml or text/xml",
-    )))
-}
-
-fn s3_error_response(err: S3Error) -> Response {
-    let status =
-        StatusCode::from_u16(err.http_status()).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR);
-    let resource = if err.resource.is_empty() {
-        "/".to_string()
-    } else {
-        err.resource.clone()
-    };
-    let body = err
-        .with_resource(resource)
-        .with_request_id(uuid::Uuid::new_v4().simple().to_string())
-        .to_xml();
-    (status, [("content-type", "application/xml")], body).into_response()
-}
-
-fn build_stats_xml(bytes_scanned: usize, bytes_returned: usize) -> String {
-    format!(
-        "<Stats><BytesScanned>{}</BytesScanned><BytesProcessed>{}</BytesProcessed><BytesReturned>{}</BytesReturned></Stats>",
-        bytes_scanned,
-        bytes_scanned,
-        bytes_returned
-    )
-}
-
-fn encode_select_event(event_type: &str, payload: &[u8]) -> Vec<u8> {
-    let mut headers = Vec::new();
-    headers.extend(encode_select_header(":event-type", event_type));
-    if event_type == "Records" {
-        headers.extend(encode_select_header(
-            ":content-type",
-            "application/octet-stream",
-        ));
-    } else if event_type == "Stats" {
-        headers.extend(encode_select_header(":content-type", "text/xml"));
-    }
-    headers.extend(encode_select_header(":message-type", "event"));
-
-    let headers_len = headers.len() as u32;
-    let total_len = 4 + 4 + 4 + headers.len() + payload.len() + 4;
-
-    let mut message = Vec::with_capacity(total_len);
-    let mut prelude = Vec::with_capacity(8);
-    prelude.extend((total_len as u32).to_be_bytes());
-    prelude.extend(headers_len.to_be_bytes());
-
-    let prelude_crc = crc32(&prelude);
-    message.extend(prelude);
-    message.extend(prelude_crc.to_be_bytes());
-    message.extend(headers);
-    message.extend(payload);
-
-    let msg_crc = crc32(&message);
-    message.extend(msg_crc.to_be_bytes());
-    message
-}
-
-fn encode_select_header(name: &str, value: &str) -> Vec<u8> {
-    let name_bytes = name.as_bytes();
-    let value_bytes = value.as_bytes();
-    let mut header = Vec::with_capacity(1 + name_bytes.len() + 1 + 2 + value_bytes.len());
-    header.push(name_bytes.len() as u8);
-    header.extend(name_bytes);
-    header.push(7);
-    header.extend((value_bytes.len() as u16).to_be_bytes());
-    header.extend(value_bytes);
-    header
-}
-
-fn crc32(data: &[u8]) -> u32 {
-    let mut hasher = Hasher::new();
-    hasher.update(data);
-    hasher.finalize()
-}

rust/myfsio-engine/crates/myfsio-server/src/handlers/ui.rs

@@ -1,210 +0,0 @@
-use std::collections::HashMap;
-use std::error::Error as StdError;
-
-use axum::extract::{Extension, Form, State};
-use axum::http::{header, HeaderMap, StatusCode};
-use axum::response::{IntoResponse, Redirect, Response};
-use tera::Context;
-
-use crate::middleware::session::SessionHandle;
-use crate::session::FlashMessage;
-use crate::state::AppState;
-
-pub async fn login_page(
-    State(state): State<AppState>,
-    Extension(session): Extension<SessionHandle>,
-) -> Response {
-    if session.read(|s| s.is_authenticated()) {
-        return Redirect::to("/ui/buckets").into_response();
-    }
-
-    let mut ctx = base_context(&session, None);
-    let flashed = session.write(|s| s.take_flash());
-    inject_flash(&mut ctx, flashed);
-
-    render(&state, "login.html", &ctx)
-}
-
-#[derive(serde::Deserialize)]
-pub struct LoginForm {
-    pub access_key: String,
-    pub secret_key: String,
-    #[serde(default)]
-    pub csrf_token: String,
-    #[serde(default)]
-    pub next: Option<String>,
-}
-
-pub async fn login_submit(
-    State(state): State<AppState>,
-    Extension(session): Extension<SessionHandle>,
-    Form(form): Form<LoginForm>,
-) -> Response {
-    let access_key = form.access_key.trim();
-    let secret_key = form.secret_key.trim();
-
-    match state.iam.get_secret_key(access_key) {
-        Some(expected) if constant_time_eq_str(&expected, secret_key) => {
-            let display = state
-                .iam
-                .get_user(access_key)
-                .await
-                .and_then(|v| {
-                    v.get("display_name")
-                        .and_then(|d| d.as_str())
-                        .map(|s| s.to_string())
-                })
-                .unwrap_or_else(|| access_key.to_string());
-
-            session.write(|s| {
-                s.user_id = Some(access_key.to_string());
-                s.display_name = Some(display);
-                s.rotate_csrf();
-                s.push_flash("success", "Signed in successfully.");
-            });
-
-            let next = form
-                .next
-                .as_deref()
-                .filter(|n| n.starts_with("/ui/") || *n == "/ui")
-                .unwrap_or("/ui/buckets")
-                .to_string();
-            Redirect::to(&next).into_response()
-        }
-        _ => {
-            session.write(|s| {
-                s.push_flash("danger", "Invalid access key or secret key.");
-            });
-            Redirect::to("/login").into_response()
-        }
-    }
-}
-
-pub async fn logout(Extension(session): Extension<SessionHandle>) -> Response {
-    session.write(|s| {
-        s.user_id = None;
-        s.display_name = None;
-        s.flash.clear();
-        s.rotate_csrf();
-        s.push_flash("info", "Signed out.");
-    });
-    Redirect::to("/login").into_response()
-}
-
-pub async fn csrf_error_page(
-    State(state): State<AppState>,
-    Extension(session): Extension<SessionHandle>,
-) -> Response {
-    let ctx = base_context(&session, None);
-    let mut resp = render(&state, "csrf_error.html", &ctx);
-    *resp.status_mut() = StatusCode::FORBIDDEN;
-    resp
-}
-
-pub async fn root_redirect() -> Response {
-    Redirect::to("/ui/buckets").into_response()
-}
-
-pub async fn not_found_page(
-    State(state): State<AppState>,
-    Extension(session): Extension<SessionHandle>,
-) -> Response {
-    let ctx = base_context(&session, None);
-    let mut resp = render(&state, "404.html", &ctx);
-    *resp.status_mut() = StatusCode::NOT_FOUND;
-    resp
-}
-
-pub async fn require_login(
-    Extension(session): Extension<SessionHandle>,
-    req: axum::extract::Request,
-    next: axum::middleware::Next,
-) -> Response {
-    if session.read(|s| s.is_authenticated()) {
-        return next.run(req).await;
-    }
-    let path = req.uri().path().to_string();
-    let query = req
-        .uri()
-        .query()
-        .map(|q| format!("?{}", q))
-        .unwrap_or_default();
-    let next_url = format!("{}{}", path, query);
-    let encoded =
-        percent_encoding::utf8_percent_encode(&next_url, percent_encoding::NON_ALPHANUMERIC)
-            .to_string();
-    let target = format!("/login?next={}", encoded);
-    Redirect::to(&target).into_response()
-}
-
-pub fn render(state: &AppState, template: &str, ctx: &Context) -> Response {
-    let engine = match &state.templates {
-        Some(e) => e,
-        None => {
-            return (
-                StatusCode::INTERNAL_SERVER_ERROR,
-                "Templates not configured",
-            )
-                .into_response();
-        }
-    };
-    match engine.render(template, ctx) {
-        Ok(html) => {
-            let mut headers = HeaderMap::new();
-            headers.insert(
-                header::CONTENT_TYPE,
-                "text/html; charset=utf-8".parse().unwrap(),
-            );
-            (StatusCode::OK, headers, html).into_response()
-        }
-        Err(e) => {
-            let mut detail = format!("{}", e);
-            let mut src = StdError::source(&e);
-            while let Some(s) = src {
-                detail.push_str(" | ");
-                detail.push_str(&s.to_string());
-                src = s.source();
-            }
-            tracing::error!("Template render failed ({}): {}", template, detail);
-            let fallback_ctx = Context::new();
-            let body = if template != "500.html" {
-                engine
-                    .render("500.html", &fallback_ctx)
-                    .unwrap_or_else(|_| "Internal Server Error".to_string())
-            } else {
-                "Internal Server Error".to_string()
-            };
-            let mut headers = HeaderMap::new();
-            headers.insert(
-                header::CONTENT_TYPE,
-                "text/html; charset=utf-8".parse().unwrap(),
-            );
-            (StatusCode::INTERNAL_SERVER_ERROR, headers, body).into_response()
-        }
-    }
-}
-
-pub fn base_context(session: &SessionHandle, endpoint: Option<&str>) -> Context {
-    let mut ctx = Context::new();
-    let snapshot = session.snapshot();
-    ctx.insert("csrf_token_value", &snapshot.csrf_token);
-    ctx.insert("is_authenticated", &snapshot.user_id.is_some());
-    ctx.insert("current_user", &snapshot.user_id);
-    ctx.insert("current_user_display_name", &snapshot.display_name);
-    ctx.insert("current_endpoint", &endpoint.unwrap_or(""));
-    ctx.insert("request_args", &HashMap::<String, String>::new());
-    ctx.insert("null", &serde_json::Value::Null);
-    ctx.insert("none", &serde_json::Value::Null);
-    ctx
-}
-
-pub fn inject_flash(ctx: &mut Context, flashed: Vec<FlashMessage>) {
-    ctx.insert("flashed_messages", &flashed);
-}
-
-fn constant_time_eq_str(a: &str, b: &str) -> bool {
-    if a.len() != b.len() {
-        return false;
-    }
-    subtle::ConstantTimeEq::ct_eq(a.as_bytes(), b.as_bytes()).into()
-}

rust/myfsio-engine/crates/myfsio-server/src/lib.rs

@@ -1,504 +0,0 @@
-pub mod config;
-pub mod handlers;
-pub mod middleware;
-pub mod services;
-pub mod session;
-pub mod state;
-pub mod stores;
-pub mod templates;
-
-use axum::Router;
-
-pub const SERVER_HEADER: &str = "MyFSIO";
-
-pub fn create_ui_router(state: state::AppState) -> Router {
-    use axum::routing::{delete, get, post, put};
-    use handlers::ui;
-    use handlers::ui_api;
-    use handlers::ui_pages;
-
-    let protected = Router::new()
-        .route("/", get(ui::root_redirect))
-        .route("/ui", get(ui::root_redirect))
-        .route("/ui/", get(ui::root_redirect))
-        .route("/ui/buckets", get(ui_pages::buckets_overview))
-        .route("/ui/buckets/create", post(ui_pages::create_bucket))
-        .route("/ui/buckets/{bucket_name}", get(ui_pages::bucket_detail))
-        .route(
-            "/ui/buckets/{bucket_name}/delete",
-            post(ui_pages::delete_bucket),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/versioning",
-            post(ui_pages::update_bucket_versioning),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/quota",
-            post(ui_pages::update_bucket_quota),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/encryption",
-            post(ui_pages::update_bucket_encryption),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/policy",
-            post(ui_pages::update_bucket_policy),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/replication",
-            post(ui_pages::update_bucket_replication),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/website",
-            post(ui_pages::update_bucket_website),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/upload",
-            post(ui_api::upload_object),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/multipart/initiate",
-            post(ui_api::initiate_multipart_upload),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/multipart/{upload_id}/part",
-            put(ui_api::upload_multipart_part),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/multipart/{upload_id}/complete",
-            post(ui_api::complete_multipart_upload),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/multipart/{upload_id}/abort",
-            delete(ui_api::abort_multipart_upload),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/objects",
-            get(ui_api::list_bucket_objects),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/objects/stream",
-            get(ui_api::stream_bucket_objects),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/folders",
-            get(ui_api::list_bucket_folders),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/copy-targets",
-            get(ui_api::list_copy_targets),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/objects/{*rest}",
-            get(ui_api::object_get_dispatch).post(ui_api::object_post_dispatch),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/acl",
-            get(ui_api::bucket_acl).post(ui_api::update_bucket_acl),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/cors",
-            get(ui_api::bucket_cors).post(ui_api::update_bucket_cors),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/lifecycle",
-            get(ui_api::bucket_lifecycle).post(ui_api::update_bucket_lifecycle),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/lifecycle/history",
-            get(ui_api::lifecycle_history_stub),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/replication/status",
-            get(ui_api::replication_status),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/replication/failures",
-            get(ui_api::replication_failures).delete(ui_api::clear_replication_failures),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/replication/failures/retry",
-            post(ui_api::retry_replication_failure),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/replication/failures/retry-all",
-            post(ui_api::retry_all_replication_failures),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/replication/failures/dismiss",
-            delete(ui_api::dismiss_replication_failure),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/replication/failures/clear",
-            delete(ui_api::clear_replication_failures),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/bulk-delete",
-            post(ui_api::bulk_delete_objects),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/bulk-download",
-            post(ui_api::bulk_download_objects),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/archived",
-            get(ui_api::archived_objects),
-        )
-        .route(
-            "/ui/buckets/{bucket_name}/archived/{*rest}",
-            post(ui_api::archived_post_dispatch),
-        )
-        .route("/ui/iam", get(ui_pages::iam_dashboard))
-        .route("/ui/iam/users", post(ui_pages::create_iam_user))
-        .route("/ui/iam/users/{user_id}", post(ui_pages::update_iam_user))
-        .route(
-            "/ui/iam/users/{user_id}/delete",
-            post(ui_pages::delete_iam_user),
-        )
-        .route(
-            "/ui/iam/users/{user_id}/policies",
-            post(ui_pages::update_iam_policies),
-        )
-        .route(
-            "/ui/iam/users/{user_id}/expiry",
-            post(ui_pages::update_iam_expiry),
-        )
-        .route(
-            "/ui/iam/users/{user_id}/rotate-secret",
-            post(ui_pages::rotate_iam_secret),
-        )
-        .route("/ui/connections/create", post(ui_pages::create_connection))
-        .route("/ui/connections/test", post(ui_api::test_connection))
-        .route(
-            "/ui/connections/{connection_id}",
-            post(ui_pages::update_connection),
-        )
-        .route(
-            "/ui/connections/{connection_id}/delete",
-            post(ui_pages::delete_connection),
-        )
-        .route(
-            "/ui/connections/{connection_id}/health",
-            get(ui_api::connection_health),
-        )
-        .route("/ui/sites", get(ui_pages::sites_dashboard))
-        .route("/ui/sites/local", post(ui_pages::update_local_site))
-        .route("/ui/sites/peers", post(ui_pages::add_peer_site))
-        .route(
-            "/ui/sites/peers/{site_id}/update",
-            post(ui_pages::update_peer_site),
-        )
-        .route(
-            "/ui/sites/peers/{site_id}/delete",
-            post(ui_pages::delete_peer_site),
-        )
-        .route("/ui/sites/peers/{site_id}/health", get(ui_api::peer_health))
-        .route(
-            "/ui/sites/peers/{site_id}/sync-stats",
-            get(ui_api::peer_sync_stats),
-        )
-        .route(
-            "/ui/sites/peers/{site_id}/bidirectional-status",
-            get(ui_api::peer_bidirectional_status),
-        )
-        .route("/ui/connections", get(ui_pages::connections_dashboard))
-        .route("/ui/metrics", get(ui_pages::metrics_dashboard))
-        .route(
-            "/ui/metrics/settings",
-            get(ui_api::metrics_settings).put(ui_api::update_metrics_settings),
-        )
-        .route("/ui/metrics/api", get(ui_api::metrics_api))
-        .route("/ui/metrics/history", get(ui_api::metrics_history))
-        .route("/ui/metrics/operations", get(ui_api::metrics_operations))
-        .route(
-            "/ui/metrics/operations/history",
-            get(ui_api::metrics_operations_history),
-        )
-        .route("/ui/system", get(ui_pages::system_dashboard))
-        .route("/ui/system/gc/status", get(ui_api::gc_status_ui))
-        .route("/ui/system/gc/run", post(ui_api::gc_run_ui))
-        .route("/ui/system/gc/history", get(ui_api::gc_history_ui))
-        .route(
-            "/ui/system/integrity/status",
-            get(ui_api::integrity_status_ui),
-        )
-        .route("/ui/system/integrity/run", post(ui_api::integrity_run_ui))
-        .route(
-            "/ui/system/integrity/history",
-            get(ui_api::integrity_history_ui),
-        )
-        .route(
-            "/ui/website-domains",
-            get(ui_pages::website_domains_dashboard),
-        )
-        .route(
-            "/ui/website-domains/create",
-            post(ui_pages::create_website_domain),
-        )
-        .route(
-            "/ui/website-domains/{domain}",
-            post(ui_pages::update_website_domain),
-        )
-        .route(
-            "/ui/website-domains/{domain}/delete",
-            post(ui_pages::delete_website_domain),
-        )
-        .route("/ui/replication/new", get(ui_pages::replication_wizard))
-        .route(
-            "/ui/replication/create",
-            post(ui_pages::create_peer_replication_rules_from_query),
-        )
-        .route(
-            "/ui/sites/peers/{site_id}/replication-rules",
-            post(ui_pages::create_peer_replication_rules),
-        )
-        .route("/ui/docs", get(ui_pages::docs_page))
-        .layer(axum::middleware::from_fn(ui::require_login));
-
-    let public = Router::new()
-        .route("/login", get(ui::login_page).post(ui::login_submit))
-        .route("/logout", post(ui::logout).get(ui::logout))
-        .route("/csrf-error", get(ui::csrf_error_page));
-
-    let session_state = middleware::SessionLayerState {
-        store: state.sessions.clone(),
-        secure: false,
-    };
-
-    let static_service = tower_http::services::ServeDir::new(&state.config.static_dir);
-
-    protected
-        .merge(public)
-        .fallback(ui::not_found_page)
-        .layer(axum::middleware::from_fn(middleware::csrf_layer))
-        .layer(axum::middleware::from_fn_with_state(
-            session_state,
-            middleware::session_layer,
-        ))
-        .layer(axum::middleware::from_fn_with_state(
-            state.clone(),
-            middleware::ui_metrics_layer,
-        ))
-        .with_state(state)
-        .nest_service("/static", static_service)
-        .layer(axum::middleware::from_fn(middleware::server_header))
-        .layer(tower_http::compression::CompressionLayer::new())
-}
-
-pub fn create_router(state: state::AppState) -> Router {
-    let mut router = Router::new()
-        .route("/myfsio/health", axum::routing::get(handlers::health_check))
-        .route("/", axum::routing::get(handlers::list_buckets))
-        .route(
-            "/{bucket}",
-            axum::routing::put(handlers::create_bucket)
-                .get(handlers::get_bucket)
-                .delete(handlers::delete_bucket)
-                .head(handlers::head_bucket)
-                .post(handlers::post_bucket),
-        )
-        .route(
-            "/{bucket}/",
-            axum::routing::put(handlers::create_bucket)
-                .get(handlers::get_bucket)
-                .delete(handlers::delete_bucket)
-                .head(handlers::head_bucket)
-                .post(handlers::post_bucket),
-        )
-        .route(
-            "/{bucket}/{*key}",
-            axum::routing::put(handlers::put_object)
-                .get(handlers::get_object)
-                .delete(handlers::delete_object)
-                .head(handlers::head_object)
-                .post(handlers::post_object),
-        );
-
-    if state.config.kms_enabled {
-        router = router
-            .route(
-                "/kms/keys",
-                axum::routing::get(handlers::kms::list_keys).post(handlers::kms::create_key),
-            )
-            .route(
-                "/kms/keys/{key_id}",
-                axum::routing::get(handlers::kms::get_key).delete(handlers::kms::delete_key),
-            )
-            .route(
-                "/kms/keys/{key_id}/enable",
-                axum::routing::post(handlers::kms::enable_key),
-            )
-            .route(
-                "/kms/keys/{key_id}/disable",
-                axum::routing::post(handlers::kms::disable_key),
-            )
-            .route("/kms/encrypt", axum::routing::post(handlers::kms::encrypt))
-            .route("/kms/decrypt", axum::routing::post(handlers::kms::decrypt))
-            .route(
-                "/kms/generate-data-key",
-                axum::routing::post(handlers::kms::generate_data_key),
-            )
-            .route(
-                "/kms/generate-data-key-without-plaintext",
-                axum::routing::post(handlers::kms::generate_data_key_without_plaintext),
-            )
-            .route(
-                "/kms/re-encrypt",
-                axum::routing::post(handlers::kms::re_encrypt),
-            )
-            .route(
-                "/kms/generate-random",
-                axum::routing::post(handlers::kms::generate_random),
-            )
-            .route(
-                "/kms/client/generate-key",
-                axum::routing::post(handlers::kms::client_generate_key),
-            )
-            .route(
-                "/kms/client/encrypt",
-                axum::routing::post(handlers::kms::client_encrypt),
-            )
-            .route(
-                "/kms/client/decrypt",
-                axum::routing::post(handlers::kms::client_decrypt),
-            )
-            .route(
-                "/kms/materials/{key_id}",
-                axum::routing::post(handlers::kms::materials),
-            );
-    }
-
-    router = router
-        .route(
-            "/admin/site",
-            axum::routing::get(handlers::admin::get_local_site)
-                .put(handlers::admin::update_local_site),
-        )
-        .route(
-            "/admin/sites",
-            axum::routing::get(handlers::admin::list_all_sites)
-                .post(handlers::admin::register_peer_site),
-        )
-        .route(
-            "/admin/sites/{site_id}",
-            axum::routing::get(handlers::admin::get_peer_site)
-                .put(handlers::admin::update_peer_site)
-                .delete(handlers::admin::delete_peer_site),
-        )
-        .route(
-            "/admin/sites/{site_id}/health",
-            axum::routing::get(handlers::admin::check_peer_health)
-                .post(handlers::admin::check_peer_health),
-        )
-        .route(
-            "/admin/sites/{site_id}/bidirectional-status",
-            axum::routing::get(handlers::admin::check_bidirectional_status),
-        )
-        .route(
-            "/admin/topology",
-            axum::routing::get(handlers::admin::get_topology),
-        )
-        .route(
-            "/admin/site/local",
-            axum::routing::get(handlers::admin::get_local_site)
-                .put(handlers::admin::update_local_site),
-        )
-        .route(
-            "/admin/site/all",
-            axum::routing::get(handlers::admin::list_all_sites),
-        )
-        .route(
-            "/admin/site/peers",
-            axum::routing::post(handlers::admin::register_peer_site),
-        )
-        .route(
-            "/admin/site/peers/{site_id}",
-            axum::routing::get(handlers::admin::get_peer_site)
-                .put(handlers::admin::update_peer_site)
-                .delete(handlers::admin::delete_peer_site),
-        )
-        .route(
-            "/admin/site/peers/{site_id}/health",
-            axum::routing::post(handlers::admin::check_peer_health),
-        )
-        .route(
-            "/admin/site/topology",
-            axum::routing::get(handlers::admin::get_topology),
-        )
-        .route(
-            "/admin/site/peers/{site_id}/bidirectional-status",
-            axum::routing::get(handlers::admin::check_bidirectional_status),
-        )
-        .route(
-            "/admin/iam/users",
-            axum::routing::get(handlers::admin::iam_list_users),
-        )
-        .route(
-            "/admin/iam/users/{identifier}",
-            axum::routing::get(handlers::admin::iam_get_user),
-        )
-        .route(
-            "/admin/iam/users/{identifier}/policies",
-            axum::routing::get(handlers::admin::iam_get_user_policies),
-        )
-        .route(
-            "/admin/iam/users/{identifier}/access-keys",
-            axum::routing::post(handlers::admin::iam_create_access_key),
-        )
-        .route(
-            "/admin/iam/users/{identifier}/access-keys/{access_key}",
-            axum::routing::delete(handlers::admin::iam_delete_access_key),
-        )
-        .route(
-            "/admin/iam/users/{identifier}/disable",
-            axum::routing::post(handlers::admin::iam_disable_user),
-        )
-        .route(
-            "/admin/iam/users/{identifier}/enable",
-            axum::routing::post(handlers::admin::iam_enable_user),
-        )
-        .route(
-            "/admin/website-domains",
-            axum::routing::get(handlers::admin::list_website_domains)
-                .post(handlers::admin::create_website_domain),
-        )
-        .route(
-            "/admin/website-domains/{domain}",
-            axum::routing::get(handlers::admin::get_website_domain)
-                .put(handlers::admin::update_website_domain)
-                .delete(handlers::admin::delete_website_domain),
-        )
-        .route(
-            "/admin/gc/status",
-            axum::routing::get(handlers::admin::gc_status),
-        )
-        .route(
-            "/admin/gc/run",
-            axum::routing::post(handlers::admin::gc_run),
-        )
-        .route(
-            "/admin/gc/history",
-            axum::routing::get(handlers::admin::gc_history),
-        )
-        .route(
-            "/admin/integrity/status",
-            axum::routing::get(handlers::admin::integrity_status),
-        )
-        .route(
-            "/admin/integrity/run",
-            axum::routing::post(handlers::admin::integrity_run),
-        )
-        .route(
-            "/admin/integrity/history",
-            axum::routing::get(handlers::admin::integrity_history),
-        );
-
-    router
-        .layer(axum::middleware::from_fn_with_state(
-            state.clone(),
-            middleware::auth_layer,
-        ))
-        .layer(axum::middleware::from_fn(middleware::server_header))
-        .layer(tower_http::compression::CompressionLayer::new())
-        .with_state(state)
-}

rust/myfsio-engine/crates/myfsio-server/src/main.rs

@@ -1,425 +0,0 @@
-use clap::{Parser, Subcommand};
-use myfsio_server::config::ServerConfig;
-use myfsio_server::state::AppState;
-
-#[derive(Parser)]
-#[command(
-    name = "myfsio",
-    version,
-    about = "MyFSIO S3-compatible storage engine"
-)]
-struct Cli {
-    #[arg(long, help = "Validate configuration and exit")]
-    check_config: bool,
-    #[arg(long, help = "Show configuration summary and exit")]
-    show_config: bool,
-    #[arg(long, help = "Reset admin credentials and exit")]
-    reset_cred: bool,
-    #[command(subcommand)]
-    command: Option<Command>,
-}
-
-#[derive(Subcommand)]
-enum Command {
-    Serve,
-    Version,
-}
-
-#[tokio::main]
-async fn main() {
-    load_env_files();
-    tracing_subscriber::fmt::init();
-
-    let cli = Cli::parse();
-    let config = ServerConfig::from_env();
-
-    if cli.reset_cred {
-        reset_admin_credentials(&config);
-        return;
-    }
-    if cli.check_config || cli.show_config {
-        print_config_summary(&config);
-        if cli.check_config {
-            let issues = validate_config(&config);
-            for issue in &issues {
-                println!("{issue}");
-            }
-            if issues.iter().any(|issue| issue.starts_with("CRITICAL:")) {
-                std::process::exit(1);
-            }
-        }
-        return;
-    }
-
-    match cli.command.unwrap_or(Command::Serve) {
-        Command::Version => {
-            println!("myfsio {}", env!("CARGO_PKG_VERSION"));
-            return;
-        }
-        Command::Serve => {}
-    }
-
-    ensure_iam_bootstrap(&config);
-    let bind_addr = config.bind_addr;
-    let ui_bind_addr = config.ui_bind_addr;
-
-    tracing::info!("MyFSIO Rust Engine starting — API on {}", bind_addr);
-    if config.ui_enabled {
-        tracing::info!("UI will bind on {}", ui_bind_addr);
-    }
-    tracing::info!("Storage root: {}", config.storage_root.display());
-    tracing::info!("Region: {}", config.region);
-    tracing::info!(
-        "Encryption: {}, KMS: {}, GC: {}, Lifecycle: {}, Integrity: {}, Metrics History: {}, Operation Metrics: {}, UI: {}",
-        config.encryption_enabled,
-        config.kms_enabled,
-        config.gc_enabled,
-        config.lifecycle_enabled,
-        config.integrity_enabled,
-        config.metrics_history_enabled,
-        config.metrics_enabled,
-        config.ui_enabled
-    );
-
-    let state = if config.encryption_enabled || config.kms_enabled {
-        AppState::new_with_encryption(config.clone()).await
-    } else {
-        AppState::new(config.clone())
-    };
-
-    let mut bg_handles: Vec<tokio::task::JoinHandle<()>> = Vec::new();
-
-    if let Some(ref gc) = state.gc {
-        bg_handles.push(gc.clone().start_background());
-        tracing::info!("GC background service started");
-    }
-
-    if let Some(ref integrity) = state.integrity {
-        bg_handles.push(integrity.clone().start_background());
-        tracing::info!("Integrity checker background service started");
-    }
-
-    if let Some(ref metrics) = state.metrics {
-        bg_handles.push(metrics.clone().start_background());
-        tracing::info!("Metrics collector background service started");
-    }
-
-    if let Some(ref system_metrics) = state.system_metrics {
-        bg_handles.push(system_metrics.clone().start_background());
-        tracing::info!("System metrics history collector started");
-    }
-
-    if config.lifecycle_enabled {
-        let lifecycle =
-            std::sync::Arc::new(myfsio_server::services::lifecycle::LifecycleService::new(
-                state.storage.clone(),
-                myfsio_server::services::lifecycle::LifecycleConfig::default(),
-            ));
-        bg_handles.push(lifecycle.start_background());
-        tracing::info!("Lifecycle manager background service started");
-    }
-
-    if let Some(ref site_sync) = state.site_sync {
-        let worker = site_sync.clone();
-        bg_handles.push(tokio::spawn(async move {
-            worker.run().await;
-        }));
-        tracing::info!("Site sync worker started");
-    }
-
-    let ui_enabled = config.ui_enabled;
-    let api_app = myfsio_server::create_router(state.clone());
-    let ui_app = if ui_enabled {
-        Some(myfsio_server::create_ui_router(state.clone()))
-    } else {
-        None
-    };
-
-    let api_listener = match tokio::net::TcpListener::bind(bind_addr).await {
-        Ok(listener) => listener,
-        Err(err) => {
-            if err.kind() == std::io::ErrorKind::AddrInUse {
-                tracing::error!("API port already in use: {}", bind_addr);
-            } else {
-                tracing::error!("Failed to bind API {}: {}", bind_addr, err);
-            }
-            for handle in bg_handles {
-                handle.abort();
-            }
-            std::process::exit(1);
-        }
-    };
-    tracing::info!("API listening on {}", bind_addr);
-
-    let ui_listener = if let Some(ref app) = ui_app {
-        let _ = app;
-        match tokio::net::TcpListener::bind(ui_bind_addr).await {
-            Ok(listener) => {
-                tracing::info!("UI listening on {}", ui_bind_addr);
-                Some(listener)
-            }
-            Err(err) => {
-                if err.kind() == std::io::ErrorKind::AddrInUse {
-                    tracing::error!("UI port already in use: {}", ui_bind_addr);
-                } else {
-                    tracing::error!("Failed to bind UI {}: {}", ui_bind_addr, err);
-                }
-                for handle in bg_handles {
-                    handle.abort();
-                }
-                std::process::exit(1);
-            }
-        }
-    } else {
-        None
-    };
-
-    let shutdown = shutdown_signal_shared();
-    let api_shutdown = shutdown.clone();
-    let api_task = tokio::spawn(async move {
-        axum::serve(api_listener, api_app)
-            .with_graceful_shutdown(async move {
-                api_shutdown.notified().await;
-            })
-            .await
-    });
-
-    let ui_task = if let (Some(listener), Some(app)) = (ui_listener, ui_app) {
-        let ui_shutdown = shutdown.clone();
-        Some(tokio::spawn(async move {
-            axum::serve(listener, app)
-                .with_graceful_shutdown(async move {
-                    ui_shutdown.notified().await;
-                })
-                .await
-        }))
-    } else {
-        None
-    };
-
-    tokio::signal::ctrl_c()
-        .await
-        .expect("Failed to listen for Ctrl+C");
-    tracing::info!("Shutdown signal received");
-    shutdown.notify_waiters();
-
-    if let Err(err) = api_task.await.unwrap_or(Ok(())) {
-        tracing::error!("API server exited with error: {}", err);
-    }
-    if let Some(task) = ui_task {
-        if let Err(err) = task.await.unwrap_or(Ok(())) {
-            tracing::error!("UI server exited with error: {}", err);
-        }
-    }
-
-    for handle in bg_handles {
-        handle.abort();
-    }
-}
-
-fn print_config_summary(config: &ServerConfig) {
-    println!("MyFSIO Rust Configuration");
-    println!("Version: {}", env!("CARGO_PKG_VERSION"));
-    println!("API bind: {}", config.bind_addr);
-    println!("UI bind: {}", config.ui_bind_addr);
-    println!("UI enabled: {}", config.ui_enabled);
-    println!("Storage root: {}", config.storage_root.display());
-    println!("IAM config: {}", config.iam_config_path.display());
-    println!("Region: {}", config.region);
-    println!("Encryption enabled: {}", config.encryption_enabled);
-    println!("KMS enabled: {}", config.kms_enabled);
-    println!("GC enabled: {}", config.gc_enabled);
-    println!("Integrity enabled: {}", config.integrity_enabled);
-    println!("Lifecycle enabled: {}", config.lifecycle_enabled);
-    println!(
-        "Website hosting enabled: {}",
-        config.website_hosting_enabled
-    );
-    println!("Site sync enabled: {}", config.site_sync_enabled);
-    println!(
-        "Metrics history enabled: {}",
-        config.metrics_history_enabled
-    );
-    println!("Operation metrics enabled: {}", config.metrics_enabled);
-}
-
-fn validate_config(config: &ServerConfig) -> Vec<String> {
-    let mut issues = Vec::new();
-
-    if config.ui_enabled && config.bind_addr == config.ui_bind_addr {
-        issues.push(
-            "CRITICAL: API and UI bind addresses cannot be identical when UI is enabled."
-                .to_string(),
-        );
-    }
-    if config.presigned_url_min_expiry > config.presigned_url_max_expiry {
-        issues.push("CRITICAL: PRESIGNED_URL_MIN_EXPIRY_SECONDS cannot exceed PRESIGNED_URL_MAX_EXPIRY_SECONDS.".to_string());
-    }
-    if let Err(err) = std::fs::create_dir_all(&config.storage_root) {
-        issues.push(format!(
-            "CRITICAL: Cannot create storage root {}: {}",
-            config.storage_root.display(),
-            err
-        ));
-    }
-    if let Some(parent) = config.iam_config_path.parent() {
-        if let Err(err) = std::fs::create_dir_all(parent) {
-            issues.push(format!(
-                "CRITICAL: Cannot create IAM config directory {}: {}",
-                parent.display(),
-                err
-            ));
-        }
-    }
-    if config.encryption_enabled && config.secret_key.is_none() {
-        issues.push(
-            "WARNING: ENCRYPTION_ENABLED=true but SECRET_KEY is not configured; secure-at-rest config encryption is unavailable.".to_string(),
-        );
-    }
-    if config.site_sync_enabled && !config.website_hosting_enabled {
-        issues.push(
-            "INFO: SITE_SYNC_ENABLED=true without WEBSITE_HOSTING_ENABLED; this is valid but unrelated.".to_string(),
-        );
-    }
-
-    issues
-}
-
-fn shutdown_signal_shared() -> std::sync::Arc<tokio::sync::Notify> {
-    std::sync::Arc::new(tokio::sync::Notify::new())
-}
-
-fn load_env_files() {
-    let cwd = std::env::current_dir().ok();
-    let mut candidates: Vec<std::path::PathBuf> = Vec::new();
-    candidates.push(std::path::PathBuf::from("/opt/myfsio/myfsio.env"));
-    if let Some(ref dir) = cwd {
-        candidates.push(dir.join(".env"));
-        candidates.push(dir.join("myfsio.env"));
-        for ancestor in dir.ancestors().skip(1).take(4) {
-            candidates.push(ancestor.join(".env"));
-            candidates.push(ancestor.join("myfsio.env"));
-        }
-    }
-
-    let mut seen = std::collections::HashSet::new();
-    for path in candidates {
-        if !seen.insert(path.clone()) {
-            continue;
-        }
-        if path.is_file() {
-            match dotenvy::from_path_override(&path) {
-                Ok(()) => eprintln!("Loaded env file: {}", path.display()),
-                Err(e) => eprintln!("Failed to load env file {}: {}", path.display(), e),
-            }
-        }
-    }
-}
-
-fn ensure_iam_bootstrap(config: &ServerConfig) {
-    let iam_path = &config.iam_config_path;
-    if iam_path.exists() {
-        return;
-    }
-
-    let access_key = std::env::var("ADMIN_ACCESS_KEY")
-        .ok()
-        .map(|s| s.trim().to_string())
-        .filter(|s| !s.is_empty())
-        .unwrap_or_else(|| format!("AK{}", uuid::Uuid::new_v4().simple()));
-    let secret_key = std::env::var("ADMIN_SECRET_KEY")
-        .ok()
-        .map(|s| s.trim().to_string())
-        .filter(|s| !s.is_empty())
-        .unwrap_or_else(|| format!("SK{}", uuid::Uuid::new_v4().simple()));
-
-    let user_id = format!("u-{}", &uuid::Uuid::new_v4().simple().to_string()[..16]);
-    let created_at = chrono::Utc::now().to_rfc3339();
-
-    let body = serde_json::json!({
-        "version": 2,
-        "users": [{
-            "user_id": user_id,
-            "display_name": "Local Admin",
-            "enabled": true,
-            "access_keys": [{
-                "access_key": access_key,
-                "secret_key": secret_key,
-                "status": "active",
-                "created_at": created_at,
-            }],
-            "policies": [{
-                "bucket": "*",
-                "actions": ["*"],
-                "prefix": "*",
-            }]
-        }]
-    });
-
-    let json = match serde_json::to_string_pretty(&body) {
-        Ok(s) => s,
-        Err(e) => {
-            tracing::error!("Failed to serialize IAM bootstrap config: {}", e);
-            return;
-        }
-    };
-
-    if let Some(parent) = iam_path.parent() {
-        if let Err(e) = std::fs::create_dir_all(parent) {
-            tracing::error!(
-                "Failed to create IAM config dir {}: {}",
-                parent.display(),
-                e
-            );
-            return;
-        }
-    }
-
-    if let Err(e) = std::fs::write(iam_path, json) {
-        tracing::error!(
-            "Failed to write IAM bootstrap config {}: {}",
-            iam_path.display(),
-            e
-        );
-        return;
-    }
-
-    tracing::info!("============================================================");
-    tracing::info!("MYFSIO - ADMIN CREDENTIALS INITIALIZED");
-    tracing::info!("============================================================");
-    tracing::info!("Access Key: {}", access_key);
-    tracing::info!("Secret Key: {}", secret_key);
-    tracing::info!("Saved to: {}", iam_path.display());
-    tracing::info!("============================================================");
-}
-
-fn reset_admin_credentials(config: &ServerConfig) {
-    if let Some(parent) = config.iam_config_path.parent() {
-        if let Err(err) = std::fs::create_dir_all(parent) {
-            eprintln!(
-                "Failed to create IAM config directory {}: {}",
-                parent.display(),
-                err
-            );
-            std::process::exit(1);
-        }
-    }
-
-    if config.iam_config_path.exists() {
-        let backup = config
-            .iam_config_path
-            .with_extension(format!("bak-{}", chrono::Utc::now().timestamp()));
-        if let Err(err) = std::fs::rename(&config.iam_config_path, &backup) {
-            eprintln!(
-                "Failed to back up existing IAM config {}: {}",
-                config.iam_config_path.display(),
-                err
-            );
-            std::process::exit(1);
-        }
-        println!("Backed up existing IAM config to {}", backup.display());
-    }
-
-    ensure_iam_bootstrap(config);
-    println!("Admin credentials reset.");
-}

rust/myfsio-engine/crates/myfsio-server/src/middleware/mod.rs

@@ -1,85 +0,0 @@
-mod auth;
-pub mod session;
-
-pub use auth::auth_layer;
-pub use session::{csrf_layer, session_layer, SessionHandle, SessionLayerState};
-
-use axum::extract::{Request, State};
-use axum::middleware::Next;
-use axum::response::Response;
-use std::time::Instant;
-
-use crate::state::AppState;
-
-pub async fn server_header(req: Request, next: Next) -> Response {
-    let mut resp = next.run(req).await;
-    resp.headers_mut()
-        .insert("server", crate::SERVER_HEADER.parse().unwrap());
-    resp
-}
-
-pub async fn ui_metrics_layer(State(state): State<AppState>, req: Request, next: Next) -> Response {
-    let metrics = match state.metrics.clone() {
-        Some(m) => m,
-        None => return next.run(req).await,
-    };
-    let start = Instant::now();
-    let method = req.method().clone();
-    let path = req.uri().path().to_string();
-    let endpoint_type = classify_ui_endpoint(&path);
-    let bytes_in = req
-        .headers()
-        .get(axum::http::header::CONTENT_LENGTH)
-        .and_then(|v| v.to_str().ok())
-        .and_then(|v| v.parse::<u64>().ok())
-        .unwrap_or(0);
-
-    let response = next.run(req).await;
-
-    let latency_ms = start.elapsed().as_secs_f64() * 1000.0;
-    let status = response.status().as_u16();
-    let bytes_out = response
-        .headers()
-        .get(axum::http::header::CONTENT_LENGTH)
-        .and_then(|v| v.to_str().ok())
-        .and_then(|v| v.parse::<u64>().ok())
-        .unwrap_or(0);
-    let error_code = if status >= 400 { Some("UIError") } else { None };
-    metrics.record_request(
-        method.as_str(),
-        endpoint_type,
-        status,
-        latency_ms,
-        bytes_in,
-        bytes_out,
-        error_code,
-    );
-
-    response
-}
-
-fn classify_ui_endpoint(path: &str) -> &'static str {
-    if path.contains("/upload") {
-        "ui_upload"
-    } else if path.starts_with("/ui/buckets/") {
-        "ui_bucket"
-    } else if path.starts_with("/ui/iam") {
-        "ui_iam"
-    } else if path.starts_with("/ui/sites") {
-        "ui_sites"
-    } else if path.starts_with("/ui/connections") {
-        "ui_connections"
-    } else if path.starts_with("/ui/metrics") {
-        "ui_metrics"
-    } else if path.starts_with("/ui/system") {
-        "ui_system"
-    } else if path.starts_with("/ui/website-domains") {
-        "ui_website_domains"
-    } else if path.starts_with("/ui/replication") {
-        "ui_replication"
-    } else if path.starts_with("/login") || path.starts_with("/logout") {
-        "ui_auth"
-    } else {
-        "ui_other"
-    }
-}

rust/myfsio-engine/crates/myfsio-server/src/middleware/session.rs

@@ -1,228 +0,0 @@
-use std::sync::Arc;
-
-use axum::extract::{Request, State};
-use axum::http::{header, HeaderValue, StatusCode};
-use axum::middleware::Next;
-use axum::response::{IntoResponse, Response};
-use cookie::{Cookie, SameSite};
-use parking_lot::Mutex;
-
-use crate::session::{
-    csrf_tokens_match, SessionData, SessionStore, CSRF_FIELD_NAME, CSRF_HEADER_NAME,
-    SESSION_COOKIE_NAME,
-};
-
-#[derive(Clone)]
-pub struct SessionLayerState {
-    pub store: Arc<SessionStore>,
-    pub secure: bool,
-}
-
-#[derive(Clone)]
-pub struct SessionHandle {
-    pub id: String,
-    inner: Arc<Mutex<SessionData>>,
-    dirty: Arc<Mutex<bool>>,
-}
-
-impl SessionHandle {
-    pub fn new(id: String, data: SessionData) -> Self {
-        Self {
-            id,
-            inner: Arc::new(Mutex::new(data)),
-            dirty: Arc::new(Mutex::new(false)),
-        }
-    }
-
-    pub fn read<R>(&self, f: impl FnOnce(&SessionData) -> R) -> R {
-        let guard = self.inner.lock();
-        f(&guard)
-    }
-
-    pub fn write<R>(&self, f: impl FnOnce(&mut SessionData) -> R) -> R {
-        let mut guard = self.inner.lock();
-        let out = f(&mut guard);
-        *self.dirty.lock() = true;
-        out
-    }
-
-    pub fn snapshot(&self) -> SessionData {
-        self.inner.lock().clone()
-    }
-
-    pub fn is_dirty(&self) -> bool {
-        *self.dirty.lock()
-    }
-}
-
-pub async fn session_layer(
-    State(state): State<SessionLayerState>,
-    mut req: Request,
-    next: Next,
-) -> Response {
-    let cookie_id = extract_session_cookie(&req);
-
-    let (session_id, session_data, is_new) =
-        match cookie_id.and_then(|id| state.store.get(&id).map(|data| (id.clone(), data))) {
-            Some((id, data)) => (id, data, false),
-            None => {
-                let (id, data) = state.store.create();
-                (id, data, true)
-            }
-        };
-
-    let handle = SessionHandle::new(session_id.clone(), session_data);
-    req.extensions_mut().insert(handle.clone());
-
-    let mut resp = next.run(req).await;
-
-    if handle.is_dirty() {
-        state.store.save(&handle.id, handle.snapshot());
-    }
-
-    if is_new {
-        let cookie = build_session_cookie(&session_id, state.secure);
-        if let Ok(value) = HeaderValue::from_str(&cookie.to_string()) {
-            resp.headers_mut().append(header::SET_COOKIE, value);
-        }
-    }
-
-    resp
-}
-
-pub async fn csrf_layer(req: Request, next: Next) -> Response {
-    const CSRF_HEADER_ALIAS: &str = "x-csrftoken";
-
-    let method = req.method().clone();
-    let needs_check = matches!(
-        method,
-        axum::http::Method::POST
-            | axum::http::Method::PUT
-            | axum::http::Method::PATCH
-            | axum::http::Method::DELETE
-    );
-
-    if !needs_check {
-        return next.run(req).await;
-    }
-
-    let is_ui = req.uri().path().starts_with("/ui/")
-        || req.uri().path() == "/ui"
-        || req.uri().path() == "/login"
-        || req.uri().path() == "/logout";
-    if !is_ui {
-        return next.run(req).await;
-    }
-
-    let handle = match req.extensions().get::<SessionHandle>() {
-        Some(h) => h.clone(),
-        None => return (StatusCode::FORBIDDEN, "Missing session").into_response(),
-    };
-
-    let expected = handle.read(|s| s.csrf_token.clone());
-
-    let header_token = req
-        .headers()
-        .get(CSRF_HEADER_NAME)
-        .or_else(|| req.headers().get(CSRF_HEADER_ALIAS))
-        .and_then(|v| v.to_str().ok())
-        .map(|s| s.to_string());
-
-    if let Some(token) = header_token.as_deref() {
-        if csrf_tokens_match(&expected, token) {
-            return next.run(req).await;
-        }
-    }
-
-    let content_type = req
-        .headers()
-        .get(header::CONTENT_TYPE)
-        .and_then(|v| v.to_str().ok())
-        .unwrap_or("")
-        .to_string();
-
-    let (parts, body) = req.into_parts();
-    let bytes = match axum::body::to_bytes(body, usize::MAX).await {
-        Ok(b) => b,
-        Err(_) => return (StatusCode::BAD_REQUEST, "Body read failed").into_response(),
-    };
-
-    let form_token = if content_type.starts_with("application/x-www-form-urlencoded") {
-        extract_form_token(&bytes)
-    } else if content_type.starts_with("multipart/form-data") {
-        extract_multipart_token(&content_type, &bytes)
-    } else {
-        None
-    };
-
-    if let Some(token) = form_token {
-        if csrf_tokens_match(&expected, &token) {
-            let req = Request::from_parts(parts, axum::body::Body::from(bytes));
-            return next.run(req).await;
-        }
-    }
-
-    tracing::warn!(
-        path = %parts.uri.path(),
-        content_type = %content_type,
-        expected_len = expected.len(),
-        header_present = header_token.is_some(),
-        "CSRF token mismatch"
-    );
-    (StatusCode::FORBIDDEN, "Invalid CSRF token").into_response()
-}
-
-fn extract_multipart_token(content_type: &str, body: &[u8]) -> Option<String> {
-    let boundary = multer::parse_boundary(content_type).ok()?;
-    let prefix = format!("--{}", boundary);
-    let text = std::str::from_utf8(body).ok()?;
-    let needle = "name=\"csrf_token\"";
-    let idx = text.find(needle)?;
-    let after = &text[idx + needle.len()..];
-    let body_start = after.find("\r\n\r\n")? + 4;
-    let tail = &after[body_start..];
-    let end = tail
-        .find(&format!("\r\n--{}", prefix.trim_start_matches("--")))
-        .or_else(|| tail.find("\r\n--"))
-        .unwrap_or(tail.len());
-    Some(tail[..end].trim().to_string())
-}
-
-fn extract_session_cookie(req: &Request) -> Option<String> {
-    let raw = req.headers().get(header::COOKIE)?.to_str().ok()?;
-    for pair in raw.split(';') {
-        if let Ok(cookie) = Cookie::parse(pair.trim().to_string()) {
-            if cookie.name() == SESSION_COOKIE_NAME {
-                return Some(cookie.value().to_string());
-            }
-        }
-    }
-    None
-}
-
-fn build_session_cookie(id: &str, secure: bool) -> Cookie<'static> {
-    let mut cookie = Cookie::new(SESSION_COOKIE_NAME, id.to_string());
-    cookie.set_http_only(true);
-    cookie.set_same_site(SameSite::Lax);
-    cookie.set_secure(secure);
-    cookie.set_path("/");
-    cookie
-}
-
-fn extract_form_token(body: &[u8]) -> Option<String> {
-    let text = std::str::from_utf8(body).ok()?;
-    let prefix = format!("{}=", CSRF_FIELD_NAME);
-    for pair in text.split('&') {
-        if let Some(rest) = pair.strip_prefix(&prefix) {
-            return urldecode(rest);
-        }
-    }
-    None
-}
-
-fn urldecode(s: &str) -> Option<String> {
-    percent_encoding::percent_decode_str(&s.replace('+', " "))
-        .decode_utf8()
-        .ok()
-        .map(|c| c.into_owned())
-}

rust/myfsio-engine/crates/myfsio-server/src/services/access_logging.rs

@@ -1,105 +0,0 @@
-use parking_lot::RwLock;
-use serde::{Deserialize, Serialize};
-use std::collections::HashMap;
-use std::path::{Path, PathBuf};
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct LoggingConfiguration {
-    pub target_bucket: String,
-    #[serde(default)]
-    pub target_prefix: String,
-    #[serde(default = "default_enabled")]
-    pub enabled: bool,
-}
-
-fn default_enabled() -> bool {
-    true
-}
-
-#[derive(Serialize, Deserialize)]
-struct StoredLoggingFile {
-    #[serde(rename = "LoggingEnabled")]
-    logging_enabled: Option<StoredLoggingEnabled>,
-}
-
-#[derive(Serialize, Deserialize)]
-struct StoredLoggingEnabled {
-    #[serde(rename = "TargetBucket")]
-    target_bucket: String,
-    #[serde(rename = "TargetPrefix", default)]
-    target_prefix: String,
-}
-
-pub struct AccessLoggingService {
-    storage_root: PathBuf,
-    cache: RwLock<HashMap<String, Option<LoggingConfiguration>>>,
-}
-
-impl AccessLoggingService {
-    pub fn new(storage_root: &Path) -> Self {
-        Self {
-            storage_root: storage_root.to_path_buf(),
-            cache: RwLock::new(HashMap::new()),
-        }
-    }
-
-    fn config_path(&self, bucket: &str) -> PathBuf {
-        self.storage_root
-            .join(".myfsio.sys")
-            .join("buckets")
-            .join(bucket)
-            .join("logging.json")
-    }
-
-    pub fn get(&self, bucket: &str) -> Option<LoggingConfiguration> {
-        if let Some(cached) = self.cache.read().get(bucket).cloned() {
-            return cached;
-        }
-
-        let path = self.config_path(bucket);
-        let config = if path.exists() {
-            std::fs::read_to_string(&path)
-                .ok()
-                .and_then(|s| serde_json::from_str::<StoredLoggingFile>(&s).ok())
-                .and_then(|f| f.logging_enabled)
-                .map(|e| LoggingConfiguration {
-                    target_bucket: e.target_bucket,
-                    target_prefix: e.target_prefix,
-                    enabled: true,
-                })
-        } else {
-            None
-        };
-
-        self.cache
-            .write()
-            .insert(bucket.to_string(), config.clone());
-        config
-    }
-
-    pub fn set(&self, bucket: &str, config: LoggingConfiguration) -> std::io::Result<()> {
-        let path = self.config_path(bucket);
-        if let Some(parent) = path.parent() {
-            std::fs::create_dir_all(parent)?;
-        }
-        let stored = StoredLoggingFile {
-            logging_enabled: Some(StoredLoggingEnabled {
-                target_bucket: config.target_bucket.clone(),
-                target_prefix: config.target_prefix.clone(),
-            }),
-        };
-        let json = serde_json::to_string_pretty(&stored)
-            .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
-        std::fs::write(&path, json)?;
-        self.cache.write().insert(bucket.to_string(), Some(config));
-        Ok(())
-    }
-
-    pub fn delete(&self, bucket: &str) {
-        let path = self.config_path(bucket);
-        if path.exists() {
-            let _ = std::fs::remove_file(&path);
-        }
-        self.cache.write().insert(bucket.to_string(), None);
-    }
-}

rust/myfsio-engine/crates/myfsio-server/src/services/gc.rs

@@ -1,286 +0,0 @@
-use serde_json::{json, Value};
-use std::path::PathBuf;
-use std::sync::Arc;
-use std::time::Instant;
-use tokio::sync::RwLock;
-
-pub struct GcConfig {
-    pub interval_hours: f64,
-    pub temp_file_max_age_hours: f64,
-    pub multipart_max_age_days: u64,
-    pub lock_file_max_age_hours: f64,
-    pub dry_run: bool,
-}
-
-impl Default for GcConfig {
-    fn default() -> Self {
-        Self {
-            interval_hours: 6.0,
-            temp_file_max_age_hours: 24.0,
-            multipart_max_age_days: 7,
-            lock_file_max_age_hours: 1.0,
-            dry_run: false,
-        }
-    }
-}
-
-pub struct GcService {
-    storage_root: PathBuf,
-    config: GcConfig,
-    running: Arc<RwLock<bool>>,
-    started_at: Arc<RwLock<Option<Instant>>>,
-    history: Arc<RwLock<Vec<Value>>>,
-    history_path: PathBuf,
-}
-
-impl GcService {
-    pub fn new(storage_root: PathBuf, config: GcConfig) -> Self {
-        let history_path = storage_root
-            .join(".myfsio.sys")
-            .join("config")
-            .join("gc_history.json");
-
-        let history = if history_path.exists() {
-            std::fs::read_to_string(&history_path)
-                .ok()
-                .and_then(|s| serde_json::from_str::<Value>(&s).ok())
-                .and_then(|v| v.get("executions").and_then(|e| e.as_array().cloned()))
-                .unwrap_or_default()
-        } else {
-            Vec::new()
-        };
-
-        Self {
-            storage_root,
-            config,
-            running: Arc::new(RwLock::new(false)),
-            started_at: Arc::new(RwLock::new(None)),
-            history: Arc::new(RwLock::new(history)),
-            history_path,
-        }
-    }
-
-    pub async fn status(&self) -> Value {
-        let running = *self.running.read().await;
-        let scan_elapsed_seconds = self
-            .started_at
-            .read()
-            .await
-            .as_ref()
-            .map(|started| started.elapsed().as_secs_f64());
-        json!({
-            "enabled": true,
-            "running": running,
-            "scanning": running,
-            "scan_elapsed_seconds": scan_elapsed_seconds,
-            "interval_hours": self.config.interval_hours,
-            "temp_file_max_age_hours": self.config.temp_file_max_age_hours,
-            "multipart_max_age_days": self.config.multipart_max_age_days,
-            "lock_file_max_age_hours": self.config.lock_file_max_age_hours,
-            "dry_run": self.config.dry_run,
-        })
-    }
-
-    pub async fn history(&self) -> Value {
-        let history = self.history.read().await;
-        let mut executions: Vec<Value> = history.iter().cloned().collect();
-        executions.reverse();
-        json!({ "executions": executions })
-    }
-
-    pub async fn run_now(&self, dry_run: bool) -> Result<Value, String> {
-        {
-            let mut running = self.running.write().await;
-            if *running {
-                return Err("GC already running".to_string());
-            }
-            *running = true;
-        }
-        *self.started_at.write().await = Some(Instant::now());
-
-        let start = Instant::now();
-        let result = self.execute_gc(dry_run || self.config.dry_run).await;
-        let elapsed = start.elapsed().as_secs_f64();
-
-        *self.running.write().await = false;
-        *self.started_at.write().await = None;
-
-        let mut result_json = result.clone();
-        if let Some(obj) = result_json.as_object_mut() {
-            obj.insert("execution_time_seconds".to_string(), json!(elapsed));
-        }
-
-        let record = json!({
-            "timestamp": chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
-            "dry_run": dry_run || self.config.dry_run,
-            "result": result_json,
-        });
-
-        {
-            let mut history = self.history.write().await;
-            history.push(record);
-            if history.len() > 50 {
-                let excess = history.len() - 50;
-                history.drain(..excess);
-            }
-        }
-        self.save_history().await;
-
-        Ok(result)
-    }
-
-    async fn execute_gc(&self, dry_run: bool) -> Value {
-        let mut temp_files_deleted = 0u64;
-        let mut temp_bytes_freed = 0u64;
-        let mut multipart_uploads_deleted = 0u64;
-        let mut lock_files_deleted = 0u64;
-        let mut empty_dirs_removed = 0u64;
-        let mut errors: Vec<String> = Vec::new();
-
-        let now = std::time::SystemTime::now();
-        let temp_max_age =
-            std::time::Duration::from_secs_f64(self.config.temp_file_max_age_hours * 3600.0);
-        let multipart_max_age =
-            std::time::Duration::from_secs(self.config.multipart_max_age_days * 86400);
-        let lock_max_age =
-            std::time::Duration::from_secs_f64(self.config.lock_file_max_age_hours * 3600.0);
-
-        let tmp_dir = self.storage_root.join(".myfsio.sys").join("tmp");
-        if tmp_dir.exists() {
-            match std::fs::read_dir(&tmp_dir) {
-                Ok(entries) => {
-                    for entry in entries.flatten() {
-                        if let Ok(metadata) = entry.metadata() {
-                            if let Ok(modified) = metadata.modified() {
-                                if let Ok(age) = now.duration_since(modified) {
-                                    if age > temp_max_age {
-                                        let size = metadata.len();
-                                        if !dry_run {
-                                            if let Err(e) = std::fs::remove_file(entry.path()) {
-                                                errors.push(format!(
-                                                    "Failed to remove temp file: {}",
-                                                    e
-                                                ));
-                                                continue;
-                                            }
-                                        }
-                                        temp_files_deleted += 1;
-                                        temp_bytes_freed += size;
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-                Err(e) => errors.push(format!("Failed to read tmp dir: {}", e)),
-            }
-        }
-
-        let multipart_dir = self.storage_root.join(".myfsio.sys").join("multipart");
-        if multipart_dir.exists() {
-            if let Ok(bucket_dirs) = std::fs::read_dir(&multipart_dir) {
-                for bucket_entry in bucket_dirs.flatten() {
-                    if let Ok(uploads) = std::fs::read_dir(bucket_entry.path()) {
-                        for upload in uploads.flatten() {
-                            if let Ok(metadata) = upload.metadata() {
-                                if let Ok(modified) = metadata.modified() {
-                                    if let Ok(age) = now.duration_since(modified) {
-                                        if age > multipart_max_age {
-                                            if !dry_run {
-                                                let _ = std::fs::remove_dir_all(upload.path());
-                                            }
-                                            multipart_uploads_deleted += 1;
-                                        }
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        }
-
-        let buckets_dir = self.storage_root.join(".myfsio.sys").join("buckets");
-        if buckets_dir.exists() {
-            if let Ok(bucket_dirs) = std::fs::read_dir(&buckets_dir) {
-                for bucket_entry in bucket_dirs.flatten() {
-                    let locks_dir = bucket_entry.path().join("locks");
-                    if locks_dir.exists() {
-                        if let Ok(locks) = std::fs::read_dir(&locks_dir) {
-                            for lock in locks.flatten() {
-                                if let Ok(metadata) = lock.metadata() {
-                                    if let Ok(modified) = metadata.modified() {
-                                        if let Ok(age) = now.duration_since(modified) {
-                                            if age > lock_max_age {
-                                                if !dry_run {
-                                                    let _ = std::fs::remove_file(lock.path());
-                                                }
-                                                lock_files_deleted += 1;
-                                            }
-                                        }
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        }
-
-        if !dry_run {
-            for dir in [&tmp_dir, &multipart_dir] {
-                if dir.exists() {
-                    if let Ok(entries) = std::fs::read_dir(dir) {
-                        for entry in entries.flatten() {
-                            if entry.path().is_dir() {
-                                if let Ok(mut contents) = std::fs::read_dir(entry.path()) {
-                                    if contents.next().is_none() {
-                                        let _ = std::fs::remove_dir(entry.path());
-                                        empty_dirs_removed += 1;
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        }
-
-        json!({
-            "temp_files_deleted": temp_files_deleted,
-            "temp_bytes_freed": temp_bytes_freed,
-            "multipart_uploads_deleted": multipart_uploads_deleted,
-            "lock_files_deleted": lock_files_deleted,
-            "empty_dirs_removed": empty_dirs_removed,
-            "errors": errors,
-        })
-    }
-
-    async fn save_history(&self) {
-        let history = self.history.read().await;
-        let data = json!({ "executions": *history });
-        if let Some(parent) = self.history_path.parent() {
-            let _ = std::fs::create_dir_all(parent);
-        }
-        let _ = std::fs::write(
-            &self.history_path,
-            serde_json::to_string_pretty(&data).unwrap_or_default(),
-        );
-    }
-
-    pub fn start_background(self: Arc<Self>) -> tokio::task::JoinHandle<()> {
-        let interval = std::time::Duration::from_secs_f64(self.config.interval_hours * 3600.0);
-        tokio::spawn(async move {
-            let mut timer = tokio::time::interval(interval);
-            timer.tick().await;
-            loop {
-                timer.tick().await;
-                tracing::info!("GC cycle starting");
-                match self.run_now(false).await {
-                    Ok(result) => tracing::info!("GC cycle complete: {:?}", result),
-                    Err(e) => tracing::warn!("GC cycle failed: {}", e),
-                }
-            }
-        })
-    }
-}

rust/myfsio-engine/crates/myfsio-server/src/services/integrity.rs

@@ -1,732 +0,0 @@
-use myfsio_common::constants::{
-    BUCKET_META_DIR, BUCKET_VERSIONS_DIR, INDEX_FILE, SYSTEM_BUCKETS_DIR, SYSTEM_ROOT,
-};
-use myfsio_storage::fs_backend::FsStorageBackend;
-use serde_json::{json, Map, Value};
-use std::collections::{HashMap, HashSet};
-use std::path::{Path, PathBuf};
-use std::sync::Arc;
-use std::time::Instant;
-use tokio::sync::RwLock;
-
-const MAX_ISSUES: usize = 500;
-const INTERNAL_FOLDERS: &[&str] = &[".meta", ".versions", ".multipart"];
-
-pub struct IntegrityConfig {
-    pub interval_hours: f64,
-    pub batch_size: usize,
-    pub auto_heal: bool,
-    pub dry_run: bool,
-}
-
-impl Default for IntegrityConfig {
-    fn default() -> Self {
-        Self {
-            interval_hours: 24.0,
-            batch_size: 10_000,
-            auto_heal: false,
-            dry_run: false,
-        }
-    }
-}
-
-pub struct IntegrityService {
-    #[allow(dead_code)]
-    storage: Arc<FsStorageBackend>,
-    storage_root: PathBuf,
-    config: IntegrityConfig,
-    running: Arc<RwLock<bool>>,
-    started_at: Arc<RwLock<Option<Instant>>>,
-    history: Arc<RwLock<Vec<Value>>>,
-    history_path: PathBuf,
-}
-
-#[derive(Default)]
-struct ScanState {
-    objects_scanned: u64,
-    buckets_scanned: u64,
-    corrupted_objects: u64,
-    orphaned_objects: u64,
-    phantom_metadata: u64,
-    stale_versions: u64,
-    etag_cache_inconsistencies: u64,
-    issues: Vec<Value>,
-    errors: Vec<String>,
-}
-
-impl ScanState {
-    fn batch_exhausted(&self, batch_size: usize) -> bool {
-        self.objects_scanned >= batch_size as u64
-    }
-
-    fn push_issue(&mut self, issue_type: &str, bucket: &str, key: &str, detail: String) {
-        if self.issues.len() < MAX_ISSUES {
-            self.issues.push(json!({
-                "issue_type": issue_type,
-                "bucket": bucket,
-                "key": key,
-                "detail": detail,
-            }));
-        }
-    }
-
-    fn into_json(self, elapsed: f64) -> Value {
-        json!({
-            "objects_scanned": self.objects_scanned,
-            "buckets_scanned": self.buckets_scanned,
-            "corrupted_objects": self.corrupted_objects,
-            "orphaned_objects": self.orphaned_objects,
-            "phantom_metadata": self.phantom_metadata,
-            "stale_versions": self.stale_versions,
-            "etag_cache_inconsistencies": self.etag_cache_inconsistencies,
-            "issues_healed": 0,
-            "issues": self.issues,
-            "errors": self.errors,
-            "execution_time_seconds": elapsed,
-        })
-    }
-}
-
-impl IntegrityService {
-    pub fn new(
-        storage: Arc<FsStorageBackend>,
-        storage_root: &Path,
-        config: IntegrityConfig,
-    ) -> Self {
-        let history_path = storage_root
-            .join(SYSTEM_ROOT)
-            .join("config")
-            .join("integrity_history.json");
-
-        let history = if history_path.exists() {
-            std::fs::read_to_string(&history_path)
-                .ok()
-                .and_then(|s| serde_json::from_str::<Value>(&s).ok())
-                .and_then(|v| v.get("executions").and_then(|e| e.as_array().cloned()))
-                .unwrap_or_default()
-        } else {
-            Vec::new()
-        };
-
-        Self {
-            storage,
-            storage_root: storage_root.to_path_buf(),
-            config,
-            running: Arc::new(RwLock::new(false)),
-            started_at: Arc::new(RwLock::new(None)),
-            history: Arc::new(RwLock::new(history)),
-            history_path,
-        }
-    }
-
-    pub async fn status(&self) -> Value {
-        let running = *self.running.read().await;
-        let scan_elapsed_seconds = self
-            .started_at
-            .read()
-            .await
-            .as_ref()
-            .map(|started| started.elapsed().as_secs_f64());
-        json!({
-            "enabled": true,
-            "running": running,
-            "scanning": running,
-            "scan_elapsed_seconds": scan_elapsed_seconds,
-            "interval_hours": self.config.interval_hours,
-            "batch_size": self.config.batch_size,
-            "auto_heal": self.config.auto_heal,
-            "dry_run": self.config.dry_run,
-        })
-    }
-
-    pub async fn history(&self) -> Value {
-        let history = self.history.read().await;
-        let mut executions: Vec<Value> = history.iter().cloned().collect();
-        executions.reverse();
-        json!({ "executions": executions })
-    }
-
-    pub async fn run_now(&self, dry_run: bool, auto_heal: bool) -> Result<Value, String> {
-        {
-            let mut running = self.running.write().await;
-            if *running {
-                return Err("Integrity check already running".to_string());
-            }
-            *running = true;
-        }
-        *self.started_at.write().await = Some(Instant::now());
-
-        let start = Instant::now();
-        let storage_root = self.storage_root.clone();
-        let batch_size = self.config.batch_size;
-        let result =
-            tokio::task::spawn_blocking(move || scan_all_buckets(&storage_root, batch_size))
-                .await
-                .unwrap_or_else(|e| {
-                    let mut st = ScanState::default();
-                    st.errors.push(format!("scan task failed: {}", e));
-                    st
-                });
-        let elapsed = start.elapsed().as_secs_f64();
-
-        *self.running.write().await = false;
-        *self.started_at.write().await = None;
-
-        let result_json = result.into_json(elapsed);
-
-        let record = json!({
-            "timestamp": chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
-            "dry_run": dry_run,
-            "auto_heal": auto_heal,
-            "result": result_json.clone(),
-        });
-
-        {
-            let mut history = self.history.write().await;
-            history.push(record);
-            if history.len() > 50 {
-                let excess = history.len() - 50;
-                history.drain(..excess);
-            }
-        }
-        self.save_history().await;
-
-        Ok(result_json)
-    }
-
-    async fn save_history(&self) {
-        let history = self.history.read().await;
-        let data = json!({ "executions": *history });
-        if let Some(parent) = self.history_path.parent() {
-            let _ = std::fs::create_dir_all(parent);
-        }
-        let _ = std::fs::write(
-            &self.history_path,
-            serde_json::to_string_pretty(&data).unwrap_or_default(),
-        );
-    }
-
-    pub fn start_background(self: Arc<Self>) -> tokio::task::JoinHandle<()> {
-        let interval = std::time::Duration::from_secs_f64(self.config.interval_hours * 3600.0);
-        tokio::spawn(async move {
-            let mut timer = tokio::time::interval(interval);
-            timer.tick().await;
-            loop {
-                timer.tick().await;
-                tracing::info!("Integrity check starting");
-                match self.run_now(false, false).await {
-                    Ok(result) => tracing::info!("Integrity check complete: {:?}", result),
-                    Err(e) => tracing::warn!("Integrity check failed: {}", e),
-                }
-            }
-        })
-    }
-}
-
-fn scan_all_buckets(storage_root: &Path, batch_size: usize) -> ScanState {
-    let mut state = ScanState::default();
-    let buckets = match list_bucket_names(storage_root) {
-        Ok(b) => b,
-        Err(e) => {
-            state.errors.push(format!("list buckets: {}", e));
-            return state;
-        }
-    };
-
-    for bucket in &buckets {
-        if state.batch_exhausted(batch_size) {
-            break;
-        }
-        state.buckets_scanned += 1;
-
-        let bucket_path = storage_root.join(bucket);
-        let meta_root = storage_root
-            .join(SYSTEM_ROOT)
-            .join(SYSTEM_BUCKETS_DIR)
-            .join(bucket)
-            .join(BUCKET_META_DIR);
-
-        let index_entries = collect_index_entries(&meta_root);
-
-        check_corrupted(&mut state, bucket, &bucket_path, &index_entries, batch_size);
-        check_phantom(&mut state, bucket, &bucket_path, &index_entries, batch_size);
-        check_orphaned(&mut state, bucket, &bucket_path, &index_entries, batch_size);
-        check_stale_versions(&mut state, storage_root, bucket, batch_size);
-        check_etag_cache(&mut state, storage_root, bucket, &index_entries, batch_size);
-    }
-
-    state
-}
-
-fn list_bucket_names(storage_root: &Path) -> std::io::Result<Vec<String>> {
-    let mut names = Vec::new();
-    if !storage_root.exists() {
-        return Ok(names);
-    }
-    for entry in std::fs::read_dir(storage_root)? {
-        let entry = entry?;
-        let name = entry.file_name().to_string_lossy().to_string();
-        if name == SYSTEM_ROOT {
-            continue;
-        }
-        if entry.file_type().map(|t| t.is_dir()).unwrap_or(false) {
-            names.push(name);
-        }
-    }
-    Ok(names)
-}
-
-#[allow(dead_code)]
-struct IndexEntryInfo {
-    entry: Value,
-    index_file: PathBuf,
-    key_name: String,
-}
-
-fn collect_index_entries(meta_root: &Path) -> HashMap<String, IndexEntryInfo> {
-    let mut out: HashMap<String, IndexEntryInfo> = HashMap::new();
-    if !meta_root.exists() {
-        return out;
-    }
-
-    let mut stack: Vec<PathBuf> = vec![meta_root.to_path_buf()];
-    while let Some(dir) = stack.pop() {
-        let rd = match std::fs::read_dir(&dir) {
-            Ok(r) => r,
-            Err(_) => continue,
-        };
-        for entry in rd.flatten() {
-            let path = entry.path();
-            let ft = match entry.file_type() {
-                Ok(t) => t,
-                Err(_) => continue,
-            };
-            if ft.is_dir() {
-                stack.push(path);
-                continue;
-            }
-            if entry.file_name().to_string_lossy() != INDEX_FILE {
-                continue;
-            }
-            let rel_dir = match path.parent().and_then(|p| p.strip_prefix(meta_root).ok()) {
-                Some(p) => p.to_path_buf(),
-                None => continue,
-            };
-            let dir_prefix = if rel_dir.as_os_str().is_empty() {
-                String::new()
-            } else {
-                rel_dir
-                    .components()
-                    .map(|c| c.as_os_str().to_string_lossy().to_string())
-                    .collect::<Vec<_>>()
-                    .join("/")
-            };
-
-            let content = match std::fs::read_to_string(&path) {
-                Ok(c) => c,
-                Err(_) => continue,
-            };
-            let index_data: Map<String, Value> = match serde_json::from_str(&content) {
-                Ok(Value::Object(m)) => m,
-                _ => continue,
-            };
-
-            for (key_name, entry_val) in index_data {
-                let full_key = if dir_prefix.is_empty() {
-                    key_name.clone()
-                } else {
-                    format!("{}/{}", dir_prefix, key_name)
-                };
-                out.insert(
-                    full_key,
-                    IndexEntryInfo {
-                        entry: entry_val,
-                        index_file: path.clone(),
-                        key_name,
-                    },
-                );
-            }
-        }
-    }
-    out
-}
-
-fn stored_etag(entry: &Value) -> Option<String> {
-    entry
-        .get("metadata")
-        .and_then(|m| m.get("__etag__"))
-        .and_then(|v| v.as_str())
-        .map(|s| s.to_string())
-}
-
-fn check_corrupted(
-    state: &mut ScanState,
-    bucket: &str,
-    bucket_path: &Path,
-    entries: &HashMap<String, IndexEntryInfo>,
-    batch_size: usize,
-) {
-    let mut keys: Vec<&String> = entries.keys().collect();
-    keys.sort();
-
-    for full_key in keys {
-        if state.batch_exhausted(batch_size) {
-            return;
-        }
-        let info = &entries[full_key];
-        let object_path = bucket_path.join(full_key);
-        if !object_path.exists() {
-            continue;
-        }
-        state.objects_scanned += 1;
-
-        let Some(stored) = stored_etag(&info.entry) else {
-            continue;
-        };
-
-        match myfsio_crypto::hashing::md5_file(&object_path) {
-            Ok(actual) => {
-                if actual != stored {
-                    state.corrupted_objects += 1;
-                    state.push_issue(
-                        "corrupted_object",
-                        bucket,
-                        full_key,
-                        format!("stored_etag={} actual_etag={}", stored, actual),
-                    );
-                }
-            }
-            Err(e) => state
-                .errors
-                .push(format!("hash {}/{}: {}", bucket, full_key, e)),
-        }
-    }
-}
-
-fn check_phantom(
-    state: &mut ScanState,
-    bucket: &str,
-    bucket_path: &Path,
-    entries: &HashMap<String, IndexEntryInfo>,
-    batch_size: usize,
-) {
-    let mut keys: Vec<&String> = entries.keys().collect();
-    keys.sort();
-
-    for full_key in keys {
-        if state.batch_exhausted(batch_size) {
-            return;
-        }
-        state.objects_scanned += 1;
-        let object_path = bucket_path.join(full_key);
-        if !object_path.exists() {
-            state.phantom_metadata += 1;
-            state.push_issue(
-                "phantom_metadata",
-                bucket,
-                full_key,
-                "metadata entry without file on disk".to_string(),
-            );
-        }
-    }
-}
-
-fn check_orphaned(
-    state: &mut ScanState,
-    bucket: &str,
-    bucket_path: &Path,
-    entries: &HashMap<String, IndexEntryInfo>,
-    batch_size: usize,
-) {
-    let indexed: HashSet<&String> = entries.keys().collect();
-    let mut stack: Vec<(PathBuf, String)> = vec![(bucket_path.to_path_buf(), String::new())];
-
-    while let Some((dir, prefix)) = stack.pop() {
-        if state.batch_exhausted(batch_size) {
-            return;
-        }
-        let rd = match std::fs::read_dir(&dir) {
-            Ok(r) => r,
-            Err(_) => continue,
-        };
-        for entry in rd.flatten() {
-            if state.batch_exhausted(batch_size) {
-                return;
-            }
-            let name = entry.file_name().to_string_lossy().to_string();
-            let ft = match entry.file_type() {
-                Ok(t) => t,
-                Err(_) => continue,
-            };
-            if ft.is_dir() {
-                if prefix.is_empty() && INTERNAL_FOLDERS.contains(&name.as_str()) {
-                    continue;
-                }
-                let new_prefix = if prefix.is_empty() {
-                    name
-                } else {
-                    format!("{}/{}", prefix, name)
-                };
-                stack.push((entry.path(), new_prefix));
-            } else if ft.is_file() {
-                let full_key = if prefix.is_empty() {
-                    name
-                } else {
-                    format!("{}/{}", prefix, name)
-                };
-                state.objects_scanned += 1;
-                if !indexed.contains(&full_key) {
-                    state.orphaned_objects += 1;
-                    state.push_issue(
-                        "orphaned_object",
-                        bucket,
-                        &full_key,
-                        "file exists without metadata entry".to_string(),
-                    );
-                }
-            }
-        }
-    }
-}
-
-fn check_stale_versions(
-    state: &mut ScanState,
-    storage_root: &Path,
-    bucket: &str,
-    batch_size: usize,
-) {
-    let versions_root = storage_root
-        .join(SYSTEM_ROOT)
-        .join(SYSTEM_BUCKETS_DIR)
-        .join(bucket)
-        .join(BUCKET_VERSIONS_DIR);
-    if !versions_root.exists() {
-        return;
-    }
-
-    let mut stack: Vec<PathBuf> = vec![versions_root.clone()];
-    while let Some(dir) = stack.pop() {
-        if state.batch_exhausted(batch_size) {
-            return;
-        }
-        let rd = match std::fs::read_dir(&dir) {
-            Ok(r) => r,
-            Err(_) => continue,
-        };
-
-        let mut bin_stems: HashMap<String, PathBuf> = HashMap::new();
-        let mut json_stems: HashMap<String, PathBuf> = HashMap::new();
-        let mut subdirs: Vec<PathBuf> = Vec::new();
-
-        for entry in rd.flatten() {
-            let ft = match entry.file_type() {
-                Ok(t) => t,
-                Err(_) => continue,
-            };
-            let path = entry.path();
-            if ft.is_dir() {
-                subdirs.push(path);
-                continue;
-            }
-            let name = entry.file_name().to_string_lossy().to_string();
-            if let Some(stem) = name.strip_suffix(".bin") {
-                bin_stems.insert(stem.to_string(), path);
-            } else if let Some(stem) = name.strip_suffix(".json") {
-                json_stems.insert(stem.to_string(), path);
-            }
-        }
-
-        for (stem, path) in &bin_stems {
-            if state.batch_exhausted(batch_size) {
-                return;
-            }
-            state.objects_scanned += 1;
-            if !json_stems.contains_key(stem) {
-                state.stale_versions += 1;
-                let key = path
-                    .strip_prefix(&versions_root)
-                    .map(|p| p.to_string_lossy().replace('\\', "/"))
-                    .unwrap_or_else(|_| path.display().to_string());
-                state.push_issue(
-                    "stale_version",
-                    bucket,
-                    &key,
-                    "version data without manifest".to_string(),
-                );
-            }
-        }
-
-        for (stem, path) in &json_stems {
-            if state.batch_exhausted(batch_size) {
-                return;
-            }
-            state.objects_scanned += 1;
-            if !bin_stems.contains_key(stem) {
-                state.stale_versions += 1;
-                let key = path
-                    .strip_prefix(&versions_root)
-                    .map(|p| p.to_string_lossy().replace('\\', "/"))
-                    .unwrap_or_else(|_| path.display().to_string());
-                state.push_issue(
-                    "stale_version",
-                    bucket,
-                    &key,
-                    "version manifest without data".to_string(),
-                );
-            }
-        }
-
-        stack.extend(subdirs);
-    }
-}
-
-fn check_etag_cache(
-    state: &mut ScanState,
-    storage_root: &Path,
-    bucket: &str,
-    entries: &HashMap<String, IndexEntryInfo>,
-    batch_size: usize,
-) {
-    let etag_index_path = storage_root
-        .join(SYSTEM_ROOT)
-        .join(SYSTEM_BUCKETS_DIR)
-        .join(bucket)
-        .join("etag_index.json");
-    if !etag_index_path.exists() {
-        return;
-    }
-
-    let cache: HashMap<String, Value> = match std::fs::read_to_string(&etag_index_path)
-        .ok()
-        .and_then(|s| serde_json::from_str(&s).ok())
-    {
-        Some(Value::Object(m)) => m.into_iter().collect(),
-        _ => return,
-    };
-
-    for (full_key, cached_val) in cache {
-        if state.batch_exhausted(batch_size) {
-            return;
-        }
-        state.objects_scanned += 1;
-        let Some(cached_etag) = cached_val.as_str() else {
-            continue;
-        };
-        let Some(info) = entries.get(&full_key) else {
-            continue;
-        };
-        let Some(stored) = stored_etag(&info.entry) else {
-            continue;
-        };
-        if cached_etag != stored {
-            state.etag_cache_inconsistencies += 1;
-            state.push_issue(
-                "etag_cache_inconsistency",
-                bucket,
-                &full_key,
-                format!("cached_etag={} index_etag={}", cached_etag, stored),
-            );
-        }
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::fs;
-
-    fn md5_hex(bytes: &[u8]) -> String {
-        myfsio_crypto::hashing::md5_bytes(bytes)
-    }
-
-    fn write_index(meta_dir: &Path, entries: &[(&str, &str)]) {
-        fs::create_dir_all(meta_dir).unwrap();
-        let mut map = Map::new();
-        for (name, etag) in entries {
-            map.insert(
-                name.to_string(),
-                json!({ "metadata": { "__etag__": etag } }),
-            );
-        }
-        fs::write(
-            meta_dir.join(INDEX_FILE),
-            serde_json::to_string(&Value::Object(map)).unwrap(),
-        )
-        .unwrap();
-    }
-
-    #[test]
-    fn scan_detects_each_issue_type() {
-        let tmp = tempfile::tempdir().unwrap();
-        let root = tmp.path();
-        let bucket = "testbucket";
-        let bucket_path = root.join(bucket);
-        let meta_root = root
-            .join(SYSTEM_ROOT)
-            .join(SYSTEM_BUCKETS_DIR)
-            .join(bucket)
-            .join(BUCKET_META_DIR);
-        fs::create_dir_all(&bucket_path).unwrap();
-
-        let clean_bytes = b"clean file contents";
-        let clean_etag = md5_hex(clean_bytes);
-        fs::write(bucket_path.join("clean.txt"), clean_bytes).unwrap();
-
-        let corrupted_bytes = b"actual content";
-        fs::write(bucket_path.join("corrupted.txt"), corrupted_bytes).unwrap();
-
-        fs::write(bucket_path.join("orphan.txt"), b"no metadata").unwrap();
-
-        write_index(
-            &meta_root,
-            &[
-                ("clean.txt", &clean_etag),
-                ("corrupted.txt", "00000000000000000000000000000000"),
-                ("phantom.txt", "deadbeefdeadbeefdeadbeefdeadbeef"),
-            ],
-        );
-
-        let versions_root = root
-            .join(SYSTEM_ROOT)
-            .join(SYSTEM_BUCKETS_DIR)
-            .join(bucket)
-            .join(BUCKET_VERSIONS_DIR)
-            .join("someobject");
-        fs::create_dir_all(&versions_root).unwrap();
-        fs::write(versions_root.join("v1.bin"), b"orphan bin").unwrap();
-        fs::write(versions_root.join("v2.json"), b"{}").unwrap();
-
-        let etag_index = root
-            .join(SYSTEM_ROOT)
-            .join(SYSTEM_BUCKETS_DIR)
-            .join(bucket)
-            .join("etag_index.json");
-        fs::write(
-            &etag_index,
-            serde_json::to_string(&json!({ "clean.txt": "stale-cached-etag" })).unwrap(),
-        )
-        .unwrap();
-
-        let state = scan_all_buckets(root, 10_000);
-
-        assert_eq!(state.corrupted_objects, 1, "corrupted");
-        assert_eq!(state.phantom_metadata, 1, "phantom");
-        assert_eq!(state.orphaned_objects, 1, "orphaned");
-        assert_eq!(state.stale_versions, 2, "stale versions");
-        assert_eq!(state.etag_cache_inconsistencies, 1, "etag cache");
-        assert_eq!(state.buckets_scanned, 1);
-        assert!(
-            state.errors.is_empty(),
-            "unexpected errors: {:?}",
-            state.errors
-        );
-    }
-
-    #[test]
-    fn skips_system_root_as_bucket() {
-        let tmp = tempfile::tempdir().unwrap();
-        fs::create_dir_all(tmp.path().join(SYSTEM_ROOT).join("config")).unwrap();
-        let state = scan_all_buckets(tmp.path(), 100);
-        assert_eq!(state.buckets_scanned, 0);
-    }
-}

rust/myfsio-engine/crates/myfsio-server/src/services/lifecycle.rs

@@ -1,167 +0,0 @@
-use myfsio_storage::fs_backend::FsStorageBackend;
-use myfsio_storage::traits::StorageEngine;
-use serde_json::{json, Value};
-use std::sync::Arc;
-use tokio::sync::RwLock;
-
-pub struct LifecycleConfig {
-    pub interval_seconds: u64,
-}
-
-impl Default for LifecycleConfig {
-    fn default() -> Self {
-        Self {
-            interval_seconds: 3600,
-        }
-    }
-}
-
-pub struct LifecycleService {
-    storage: Arc<FsStorageBackend>,
-    config: LifecycleConfig,
-    running: Arc<RwLock<bool>>,
-}
-
-impl LifecycleService {
-    pub fn new(storage: Arc<FsStorageBackend>, config: LifecycleConfig) -> Self {
-        Self {
-            storage,
-            config,
-            running: Arc::new(RwLock::new(false)),
-        }
-    }
-
-    pub async fn run_cycle(&self) -> Result<Value, String> {
-        {
-            let mut running = self.running.write().await;
-            if *running {
-                return Err("Lifecycle already running".to_string());
-            }
-            *running = true;
-        }
-
-        let result = self.evaluate_rules().await;
-        *self.running.write().await = false;
-        Ok(result)
-    }
-
-    async fn evaluate_rules(&self) -> Value {
-        let buckets = match self.storage.list_buckets().await {
-            Ok(b) => b,
-            Err(e) => return json!({"error": e.to_string()}),
-        };
-
-        let mut total_expired = 0u64;
-        let mut total_multipart_aborted = 0u64;
-        let mut errors: Vec<String> = Vec::new();
-
-        for bucket in &buckets {
-            let config = match self.storage.get_bucket_config(&bucket.name).await {
-                Ok(c) => c,
-                Err(_) => continue,
-            };
-
-            let lifecycle = match &config.lifecycle {
-                Some(lc) => lc,
-                None => continue,
-            };
-
-            let rules = match lifecycle
-                .as_str()
-                .and_then(|s| serde_json::from_str::<Value>(s).ok())
-            {
-                Some(v) => v,
-                None => continue,
-            };
-
-            let rules_arr = match rules.get("Rules").and_then(|r| r.as_array()) {
-                Some(a) => a.clone(),
-                None => continue,
-            };
-
-            for rule in &rules_arr {
-                if rule.get("Status").and_then(|s| s.as_str()) != Some("Enabled") {
-                    continue;
-                }
-
-                let prefix = rule
-                    .get("Filter")
-                    .and_then(|f| f.get("Prefix"))
-                    .and_then(|p| p.as_str())
-                    .or_else(|| rule.get("Prefix").and_then(|p| p.as_str()))
-                    .unwrap_or("");
-
-                if let Some(exp) = rule.get("Expiration") {
-                    if let Some(days) = exp.get("Days").and_then(|d| d.as_u64()) {
-                        let cutoff = chrono::Utc::now() - chrono::Duration::days(days as i64);
-                        let params = myfsio_common::types::ListParams {
-                            max_keys: 1000,
-                            prefix: if prefix.is_empty() {
-                                None
-                            } else {
-                                Some(prefix.to_string())
-                            },
-                            ..Default::default()
-                        };
-                        if let Ok(result) = self.storage.list_objects(&bucket.name, &params).await {
-                            for obj in &result.objects {
-                                if obj.last_modified < cutoff {
-                                    match self.storage.delete_object(&bucket.name, &obj.key).await {
-                                        Ok(()) => total_expired += 1,
-                                        Err(e) => errors
-                                            .push(format!("{}:{}: {}", bucket.name, obj.key, e)),
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-
-                if let Some(abort) = rule.get("AbortIncompleteMultipartUpload") {
-                    if let Some(days) = abort.get("DaysAfterInitiation").and_then(|d| d.as_u64()) {
-                        let cutoff = chrono::Utc::now() - chrono::Duration::days(days as i64);
-                        if let Ok(uploads) = self.storage.list_multipart_uploads(&bucket.name).await
-                        {
-                            for upload in &uploads {
-                                if upload.initiated < cutoff {
-                                    match self
-                                        .storage
-                                        .abort_multipart(&bucket.name, &upload.upload_id)
-                                        .await
-                                    {
-                                        Ok(()) => total_multipart_aborted += 1,
-                                        Err(e) => errors
-                                            .push(format!("abort {}: {}", upload.upload_id, e)),
-                                    }
-                                }
-                            }
-                        }
-                    }
-                }
-            }
-        }
-
-        json!({
-            "objects_expired": total_expired,
-            "multipart_aborted": total_multipart_aborted,
-            "buckets_evaluated": buckets.len(),
-            "errors": errors,
-        })
-    }
-
-    pub fn start_background(self: Arc<Self>) -> tokio::task::JoinHandle<()> {
-        let interval = std::time::Duration::from_secs(self.config.interval_seconds);
-        tokio::spawn(async move {
-            let mut timer = tokio::time::interval(interval);
-            timer.tick().await;
-            loop {
-                timer.tick().await;
-                tracing::info!("Lifecycle evaluation starting");
-                match self.run_cycle().await {
-                    Ok(result) => tracing::info!("Lifecycle cycle complete: {:?}", result),
-                    Err(e) => tracing::warn!("Lifecycle cycle failed: {}", e),
-                }
-            }
-        })
-    }
-}

rust/myfsio-engine/crates/myfsio-server/src/services/metrics.rs

@@ -1,368 +0,0 @@
-use chrono::{DateTime, Utc};
-use parking_lot::Mutex;
-use rand::Rng;
-use serde::{Deserialize, Serialize};
-use serde_json::{json, Value};
-use std::collections::HashMap;
-use std::path::{Path, PathBuf};
-use std::sync::Arc;
-use std::time::{SystemTime, UNIX_EPOCH};
-
-const MAX_LATENCY_SAMPLES: usize = 5000;
-
-pub struct MetricsConfig {
-    pub interval_minutes: u64,
-    pub retention_hours: u64,
-}
-
-impl Default for MetricsConfig {
-    fn default() -> Self {
-        Self {
-            interval_minutes: 5,
-            retention_hours: 24,
-        }
-    }
-}
-
-#[derive(Debug, Clone)]
-struct OperationStats {
-    count: u64,
-    success_count: u64,
-    error_count: u64,
-    latency_sum_ms: f64,
-    latency_min_ms: f64,
-    latency_max_ms: f64,
-    bytes_in: u64,
-    bytes_out: u64,
-    latency_samples: Vec<f64>,
-}
-
-impl Default for OperationStats {
-    fn default() -> Self {
-        Self {
-            count: 0,
-            success_count: 0,
-            error_count: 0,
-            latency_sum_ms: 0.0,
-            latency_min_ms: f64::INFINITY,
-            latency_max_ms: 0.0,
-            bytes_in: 0,
-            bytes_out: 0,
-            latency_samples: Vec::new(),
-        }
-    }
-}
-
-impl OperationStats {
-    fn record(&mut self, latency_ms: f64, success: bool, bytes_in: u64, bytes_out: u64) {
-        self.count += 1;
-        if success {
-            self.success_count += 1;
-        } else {
-            self.error_count += 1;
-        }
-        self.latency_sum_ms += latency_ms;
-        if latency_ms < self.latency_min_ms {
-            self.latency_min_ms = latency_ms;
-        }
-        if latency_ms > self.latency_max_ms {
-            self.latency_max_ms = latency_ms;
-        }
-        self.bytes_in += bytes_in;
-        self.bytes_out += bytes_out;
-
-        if self.latency_samples.len() < MAX_LATENCY_SAMPLES {
-            self.latency_samples.push(latency_ms);
-        } else {
-            let mut rng = rand::thread_rng();
-            let j = rng.gen_range(0..self.count as usize);
-            if j < MAX_LATENCY_SAMPLES {
-                self.latency_samples[j] = latency_ms;
-            }
-        }
-    }
-
-    fn compute_percentile(sorted: &[f64], p: f64) -> f64 {
-        if sorted.is_empty() {
-            return 0.0;
-        }
-        let k = (sorted.len() - 1) as f64 * (p / 100.0);
-        let f = k.floor() as usize;
-        let c = (f + 1).min(sorted.len() - 1);
-        let d = k - f as f64;
-        sorted[f] + d * (sorted[c] - sorted[f])
-    }
-
-    fn to_json(&self) -> Value {
-        let avg = if self.count > 0 {
-            self.latency_sum_ms / self.count as f64
-        } else {
-            0.0
-        };
-        let min = if self.latency_min_ms.is_infinite() {
-            0.0
-        } else {
-            self.latency_min_ms
-        };
-        let mut sorted = self.latency_samples.clone();
-        sorted.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
-        json!({
-            "count": self.count,
-            "success_count": self.success_count,
-            "error_count": self.error_count,
-            "latency_avg_ms": round2(avg),
-            "latency_min_ms": round2(min),
-            "latency_max_ms": round2(self.latency_max_ms),
-            "latency_p50_ms": round2(Self::compute_percentile(&sorted, 50.0)),
-            "latency_p95_ms": round2(Self::compute_percentile(&sorted, 95.0)),
-            "latency_p99_ms": round2(Self::compute_percentile(&sorted, 99.0)),
-            "bytes_in": self.bytes_in,
-            "bytes_out": self.bytes_out,
-        })
-    }
-}
-
-fn round2(v: f64) -> f64 {
-    (v * 100.0).round() / 100.0
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct MetricsSnapshot {
-    pub timestamp: DateTime<Utc>,
-    pub window_seconds: u64,
-    pub by_method: HashMap<String, Value>,
-    pub by_endpoint: HashMap<String, Value>,
-    pub by_status_class: HashMap<String, u64>,
-    pub error_codes: HashMap<String, u64>,
-    pub totals: Value,
-}
-
-struct Inner {
-    by_method: HashMap<String, OperationStats>,
-    by_endpoint: HashMap<String, OperationStats>,
-    by_status_class: HashMap<String, u64>,
-    error_codes: HashMap<String, u64>,
-    totals: OperationStats,
-    window_start: f64,
-    snapshots: Vec<MetricsSnapshot>,
-}
-
-pub struct MetricsService {
-    config: MetricsConfig,
-    inner: Arc<Mutex<Inner>>,
-    snapshots_path: PathBuf,
-}
-
-impl MetricsService {
-    pub fn new(storage_root: &Path, config: MetricsConfig) -> Self {
-        let snapshots_path = storage_root
-            .join(".myfsio.sys")
-            .join("config")
-            .join("operation_metrics.json");
-
-        let mut snapshots: Vec<MetricsSnapshot> = if snapshots_path.exists() {
-            std::fs::read_to_string(&snapshots_path)
-                .ok()
-                .and_then(|s| serde_json::from_str::<Value>(&s).ok())
-                .and_then(|v| {
-                    v.get("snapshots").and_then(|s| {
-                        serde_json::from_value::<Vec<MetricsSnapshot>>(s.clone()).ok()
-                    })
-                })
-                .unwrap_or_default()
-        } else {
-            Vec::new()
-        };
-        let cutoff = now_secs() - (config.retention_hours * 3600) as f64;
-        snapshots.retain(|s| s.timestamp.timestamp() as f64 > cutoff);
-
-        Self {
-            config,
-            inner: Arc::new(Mutex::new(Inner {
-                by_method: HashMap::new(),
-                by_endpoint: HashMap::new(),
-                by_status_class: HashMap::new(),
-                error_codes: HashMap::new(),
-                totals: OperationStats::default(),
-                window_start: now_secs(),
-                snapshots,
-            })),
-            snapshots_path,
-        }
-    }
-
-    pub fn record_request(
-        &self,
-        method: &str,
-        endpoint_type: &str,
-        status_code: u16,
-        latency_ms: f64,
-        bytes_in: u64,
-        bytes_out: u64,
-        error_code: Option<&str>,
-    ) {
-        let success = (200..400).contains(&status_code);
-        let status_class = format!("{}xx", status_code / 100);
-
-        let mut inner = self.inner.lock();
-        inner
-            .by_method
-            .entry(method.to_string())
-            .or_default()
-            .record(latency_ms, success, bytes_in, bytes_out);
-        inner
-            .by_endpoint
-            .entry(endpoint_type.to_string())
-            .or_default()
-            .record(latency_ms, success, bytes_in, bytes_out);
-        *inner.by_status_class.entry(status_class).or_insert(0) += 1;
-        if let Some(code) = error_code {
-            *inner.error_codes.entry(code.to_string()).or_insert(0) += 1;
-        }
-        inner
-            .totals
-            .record(latency_ms, success, bytes_in, bytes_out);
-    }
-
-    pub fn get_current_stats(&self) -> Value {
-        let inner = self.inner.lock();
-        let window_seconds = (now_secs() - inner.window_start).max(0.0) as u64;
-        let by_method: HashMap<String, Value> = inner
-            .by_method
-            .iter()
-            .map(|(k, v)| (k.clone(), v.to_json()))
-            .collect();
-        let by_endpoint: HashMap<String, Value> = inner
-            .by_endpoint
-            .iter()
-            .map(|(k, v)| (k.clone(), v.to_json()))
-            .collect();
-        json!({
-            "timestamp": Utc::now().to_rfc3339(),
-            "window_seconds": window_seconds,
-            "by_method": by_method,
-            "by_endpoint": by_endpoint,
-            "by_status_class": inner.by_status_class,
-            "error_codes": inner.error_codes,
-            "totals": inner.totals.to_json(),
-        })
-    }
-
-    pub fn get_history(&self, hours: Option<u64>) -> Vec<MetricsSnapshot> {
-        let inner = self.inner.lock();
-        let mut snapshots = inner.snapshots.clone();
-        if let Some(h) = hours {
-            let cutoff = now_secs() - (h * 3600) as f64;
-            snapshots.retain(|s| s.timestamp.timestamp() as f64 > cutoff);
-        }
-        snapshots
-    }
-
-    pub fn snapshot(&self) -> Value {
-        let current = self.get_current_stats();
-        let history = self.get_history(None);
-        json!({
-            "enabled": true,
-            "current": current,
-            "snapshots": history,
-        })
-    }
-
-    fn take_snapshot(&self) {
-        let snapshot = {
-            let mut inner = self.inner.lock();
-            let window_seconds = (now_secs() - inner.window_start).max(0.0) as u64;
-
-            let by_method: HashMap<String, Value> = inner
-                .by_method
-                .iter()
-                .map(|(k, v)| (k.clone(), v.to_json()))
-                .collect();
-            let by_endpoint: HashMap<String, Value> = inner
-                .by_endpoint
-                .iter()
-                .map(|(k, v)| (k.clone(), v.to_json()))
-                .collect();
-
-            let snap = MetricsSnapshot {
-                timestamp: Utc::now(),
-                window_seconds,
-                by_method,
-                by_endpoint,
-                by_status_class: inner.by_status_class.clone(),
-                error_codes: inner.error_codes.clone(),
-                totals: inner.totals.to_json(),
-            };
-
-            inner.snapshots.push(snap.clone());
-            let cutoff = now_secs() - (self.config.retention_hours * 3600) as f64;
-            inner
-                .snapshots
-                .retain(|s| s.timestamp.timestamp() as f64 > cutoff);
-
-            inner.by_method.clear();
-            inner.by_endpoint.clear();
-            inner.by_status_class.clear();
-            inner.error_codes.clear();
-            inner.totals = OperationStats::default();
-            inner.window_start = now_secs();
-
-            snap
-        };
-        let _ = snapshot;
-        self.save_snapshots();
-    }
-
-    fn save_snapshots(&self) {
-        let snapshots = { self.inner.lock().snapshots.clone() };
-        if let Some(parent) = self.snapshots_path.parent() {
-            let _ = std::fs::create_dir_all(parent);
-        }
-        let data = json!({ "snapshots": snapshots });
-        let _ = std::fs::write(
-            &self.snapshots_path,
-            serde_json::to_string_pretty(&data).unwrap_or_default(),
-        );
-    }
-
-    pub fn start_background(self: Arc<Self>) -> tokio::task::JoinHandle<()> {
-        let interval = std::time::Duration::from_secs(self.config.interval_minutes * 60);
-        tokio::spawn(async move {
-            let mut timer = tokio::time::interval(interval);
-            timer.tick().await;
-            loop {
-                timer.tick().await;
-                self.take_snapshot();
-            }
-        })
-    }
-}
-
-pub fn classify_endpoint(path: &str) -> &'static str {
-    if path.is_empty() || path == "/" {
-        return "service";
-    }
-    let trimmed = path.trim_end_matches('/');
-    if trimmed.starts_with("/ui") {
-        return "ui";
-    }
-    if trimmed.starts_with("/kms") {
-        return "kms";
-    }
-    if trimmed.starts_with("/myfsio") {
-        return "service";
-    }
-    let parts: Vec<&str> = trimmed.trim_start_matches('/').split('/').collect();
-    match parts.len() {
-        0 => "service",
-        1 => "bucket",
-        _ => "object",
-    }
-}
-
-fn now_secs() -> f64 {
-    SystemTime::now()
-        .duration_since(UNIX_EPOCH)
-        .map(|d| d.as_secs_f64())
-        .unwrap_or(0.0)
-}

rust/myfsio-engine/crates/myfsio-server/src/services/mod.rs

@@ -1,11 +0,0 @@
-pub mod access_logging;
-pub mod gc;
-pub mod integrity;
-pub mod lifecycle;
-pub mod metrics;
-pub mod replication;
-pub mod s3_client;
-pub mod site_registry;
-pub mod site_sync;
-pub mod system_metrics;
-pub mod website_domains;

rust/myfsio-engine/crates/myfsio-server/src/services/replication.rs

@@ -1,713 +0,0 @@
-use std::collections::HashMap;
-use std::path::{Path, PathBuf};
-use std::sync::Arc;
-use std::time::{Duration, SystemTime, UNIX_EPOCH};
-
-use aws_sdk_s3::primitives::ByteStream;
-use parking_lot::Mutex;
-use serde::{Deserialize, Serialize};
-use tokio::sync::Semaphore;
-
-use myfsio_common::types::ListParams;
-use myfsio_storage::fs_backend::FsStorageBackend;
-use myfsio_storage::traits::StorageEngine;
-
-use crate::services::s3_client::{build_client, check_endpoint_health, ClientOptions};
-use crate::stores::connections::{ConnectionStore, RemoteConnection};
-
-pub const MODE_NEW_ONLY: &str = "new_only";
-pub const MODE_ALL: &str = "all";
-pub const MODE_BIDIRECTIONAL: &str = "bidirectional";
-
-#[derive(Debug, Clone, Default, Serialize, Deserialize)]
-pub struct ReplicationStats {
-    #[serde(default)]
-    pub objects_synced: u64,
-    #[serde(default)]
-    pub objects_pending: u64,
-    #[serde(default)]
-    pub objects_orphaned: u64,
-    #[serde(default)]
-    pub bytes_synced: u64,
-    #[serde(default)]
-    pub last_sync_at: Option<f64>,
-    #[serde(default)]
-    pub last_sync_key: Option<String>,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct ReplicationRule {
-    pub bucket_name: String,
-    pub target_connection_id: String,
-    pub target_bucket: String,
-    #[serde(default = "default_true")]
-    pub enabled: bool,
-    #[serde(default = "default_mode")]
-    pub mode: String,
-    #[serde(default)]
-    pub created_at: Option<f64>,
-    #[serde(default)]
-    pub stats: ReplicationStats,
-    #[serde(default = "default_true")]
-    pub sync_deletions: bool,
-    #[serde(default)]
-    pub last_pull_at: Option<f64>,
-    #[serde(default)]
-    pub filter_prefix: Option<String>,
-}
-
-fn default_true() -> bool {
-    true
-}
-fn default_mode() -> String {
-    MODE_NEW_ONLY.to_string()
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct ReplicationFailure {
-    pub object_key: String,
-    pub error_message: String,
-    pub timestamp: f64,
-    pub failure_count: u32,
-    pub bucket_name: String,
-    pub action: String,
-    #[serde(default)]
-    pub last_error_code: Option<String>,
-}
-
-pub struct ReplicationFailureStore {
-    storage_root: PathBuf,
-    max_failures_per_bucket: usize,
-    cache: Mutex<HashMap<String, Vec<ReplicationFailure>>>,
-}
-
-impl ReplicationFailureStore {
-    pub fn new(storage_root: PathBuf, max_failures_per_bucket: usize) -> Self {
-        Self {
-            storage_root,
-            max_failures_per_bucket,
-            cache: Mutex::new(HashMap::new()),
-        }
-    }
-
-    fn path(&self, bucket: &str) -> PathBuf {
-        self.storage_root
-            .join(".myfsio.sys")
-            .join("buckets")
-            .join(bucket)
-            .join("replication_failures.json")
-    }
-
-    fn load_from_disk(&self, bucket: &str) -> Vec<ReplicationFailure> {
-        let path = self.path(bucket);
-        if !path.exists() {
-            return Vec::new();
-        }
-        match std::fs::read_to_string(&path) {
-            Ok(text) => {
-                let parsed: serde_json::Value = match serde_json::from_str(&text) {
-                    Ok(v) => v,
-                    Err(_) => return Vec::new(),
-                };
-                parsed
-                    .get("failures")
-                    .and_then(|v| serde_json::from_value(v.clone()).ok())
-                    .unwrap_or_default()
-            }
-            Err(_) => Vec::new(),
-        }
-    }
-
-    fn save_to_disk(&self, bucket: &str, failures: &[ReplicationFailure]) {
-        let path = self.path(bucket);
-        if let Some(parent) = path.parent() {
-            let _ = std::fs::create_dir_all(parent);
-        }
-        let trimmed = &failures[..failures.len().min(self.max_failures_per_bucket)];
-        let data = serde_json::json!({ "failures": trimmed });
-        let _ = std::fs::write(
-            &path,
-            serde_json::to_string_pretty(&data).unwrap_or_default(),
-        );
-    }
-
-    pub fn load(&self, bucket: &str) -> Vec<ReplicationFailure> {
-        let mut cache = self.cache.lock();
-        if let Some(existing) = cache.get(bucket) {
-            return existing.clone();
-        }
-        let loaded = self.load_from_disk(bucket);
-        cache.insert(bucket.to_string(), loaded.clone());
-        loaded
-    }
-
-    pub fn save(&self, bucket: &str, failures: Vec<ReplicationFailure>) {
-        let trimmed: Vec<ReplicationFailure> = failures
-            .into_iter()
-            .take(self.max_failures_per_bucket)
-            .collect();
-        self.save_to_disk(bucket, &trimmed);
-        self.cache.lock().insert(bucket.to_string(), trimmed);
-    }
-
-    pub fn add(&self, bucket: &str, failure: ReplicationFailure) {
-        let mut failures = self.load(bucket);
-        if let Some(existing) = failures
-            .iter_mut()
-            .find(|f| f.object_key == failure.object_key)
-        {
-            existing.failure_count += 1;
-            existing.timestamp = failure.timestamp;
-            existing.error_message = failure.error_message.clone();
-            existing.last_error_code = failure.last_error_code.clone();
-        } else {
-            failures.insert(0, failure);
-        }
-        self.save(bucket, failures);
-    }
-
-    pub fn remove(&self, bucket: &str, object_key: &str) -> bool {
-        let failures = self.load(bucket);
-        let before = failures.len();
-        let after: Vec<_> = failures
-            .into_iter()
-            .filter(|f| f.object_key != object_key)
-            .collect();
-        if after.len() != before {
-            self.save(bucket, after);
-            true
-        } else {
-            false
-        }
-    }
-
-    pub fn clear(&self, bucket: &str) {
-        self.cache.lock().remove(bucket);
-        let path = self.path(bucket);
-        let _ = std::fs::remove_file(path);
-    }
-
-    pub fn get(&self, bucket: &str, object_key: &str) -> Option<ReplicationFailure> {
-        self.load(bucket)
-            .into_iter()
-            .find(|f| f.object_key == object_key)
-    }
-
-    pub fn count(&self, bucket: &str) -> usize {
-        self.load(bucket).len()
-    }
-}
-
-pub struct ReplicationManager {
-    storage: Arc<FsStorageBackend>,
-    connections: Arc<ConnectionStore>,
-    rules_path: PathBuf,
-    rules: Mutex<HashMap<String, ReplicationRule>>,
-    client_options: ClientOptions,
-    streaming_threshold_bytes: u64,
-    pub failures: Arc<ReplicationFailureStore>,
-    semaphore: Arc<Semaphore>,
-}
-
-impl ReplicationManager {
-    pub fn new(
-        storage: Arc<FsStorageBackend>,
-        connections: Arc<ConnectionStore>,
-        storage_root: &Path,
-        connect_timeout: Duration,
-        read_timeout: Duration,
-        max_retries: u32,
-        streaming_threshold_bytes: u64,
-        max_failures_per_bucket: usize,
-    ) -> Self {
-        let rules_path = storage_root
-            .join(".myfsio.sys")
-            .join("config")
-            .join("replication_rules.json");
-        let rules = load_rules(&rules_path);
-        let failures = Arc::new(ReplicationFailureStore::new(
-            storage_root.to_path_buf(),
-            max_failures_per_bucket,
-        ));
-        let client_options = ClientOptions {
-            connect_timeout,
-            read_timeout,
-            max_attempts: max_retries,
-        };
-        Self {
-            storage,
-            connections,
-            rules_path,
-            rules: Mutex::new(rules),
-            client_options,
-            streaming_threshold_bytes,
-            failures,
-            semaphore: Arc::new(Semaphore::new(4)),
-        }
-    }
-
-    pub fn reload_rules(&self) {
-        *self.rules.lock() = load_rules(&self.rules_path);
-    }
-
-    pub fn list_rules(&self) -> Vec<ReplicationRule> {
-        self.rules.lock().values().cloned().collect()
-    }
-
-    pub fn get_rule(&self, bucket: &str) -> Option<ReplicationRule> {
-        self.rules.lock().get(bucket).cloned()
-    }
-
-    pub fn set_rule(&self, rule: ReplicationRule) {
-        {
-            let mut guard = self.rules.lock();
-            guard.insert(rule.bucket_name.clone(), rule);
-        }
-        self.save_rules();
-    }
-
-    pub fn delete_rule(&self, bucket: &str) {
-        {
-            let mut guard = self.rules.lock();
-            guard.remove(bucket);
-        }
-        self.save_rules();
-    }
-
-    pub fn save_rules(&self) {
-        let snapshot: HashMap<String, ReplicationRule> = self.rules.lock().clone();
-        if let Some(parent) = self.rules_path.parent() {
-            let _ = std::fs::create_dir_all(parent);
-        }
-        if let Ok(text) = serde_json::to_string_pretty(&snapshot) {
-            let _ = std::fs::write(&self.rules_path, text);
-        }
-    }
-
-    fn update_last_sync(&self, bucket: &str, key: &str) {
-        {
-            let mut guard = self.rules.lock();
-            if let Some(rule) = guard.get_mut(bucket) {
-                rule.stats.last_sync_at = Some(now_secs());
-                rule.stats.last_sync_key = Some(key.to_string());
-            }
-        }
-        self.save_rules();
-    }
-
-    pub async fn trigger(self: Arc<Self>, bucket: String, key: String, action: String) {
-        let rule = match self.get_rule(&bucket) {
-            Some(r) if r.enabled => r,
-            _ => return,
-        };
-        let connection = match self.connections.get(&rule.target_connection_id) {
-            Some(c) => c,
-            None => {
-                tracing::warn!(
-                    "Replication skipped for {}/{}: connection {} not found",
-                    bucket,
-                    key,
-                    rule.target_connection_id
-                );
-                return;
-            }
-        };
-        let permit = match self.semaphore.clone().try_acquire_owned() {
-            Ok(p) => p,
-            Err(_) => {
-                let sem = self.semaphore.clone();
-                match sem.acquire_owned().await {
-                    Ok(p) => p,
-                    Err(_) => return,
-                }
-            }
-        };
-        let manager = self.clone();
-        tokio::spawn(async move {
-            let _permit = permit;
-            manager
-                .replicate_task(&bucket, &key, &rule, &connection, &action)
-                .await;
-        });
-    }
-
-    pub async fn replicate_existing_objects(self: Arc<Self>, bucket: String) -> usize {
-        let rule = match self.get_rule(&bucket) {
-            Some(r) if r.enabled => r,
-            _ => return 0,
-        };
-        let connection = match self.connections.get(&rule.target_connection_id) {
-            Some(c) => c,
-            None => {
-                tracing::warn!(
-                    "Cannot replicate existing objects for {}: connection {} not found",
-                    bucket,
-                    rule.target_connection_id
-                );
-                return 0;
-            }
-        };
-        if !self.check_endpoint(&connection).await {
-            tracing::warn!(
-                "Cannot replicate existing objects for {}: endpoint {} is unreachable",
-                bucket,
-                connection.endpoint_url
-            );
-            return 0;
-        }
-
-        let mut continuation_token: Option<String> = None;
-        let mut submitted = 0usize;
-
-        loop {
-            let page = match self
-                .storage
-                .list_objects(
-                    &bucket,
-                    &ListParams {
-                        max_keys: 1000,
-                        continuation_token: continuation_token.clone(),
-                        prefix: rule.filter_prefix.clone(),
-                        start_after: None,
-                    },
-                )
-                .await
-            {
-                Ok(page) => page,
-                Err(err) => {
-                    tracing::error!(
-                        "Failed to list existing objects for replication in {}: {}",
-                        bucket,
-                        err
-                    );
-                    break;
-                }
-            };
-
-            let next_token = page.next_continuation_token.clone();
-            let is_truncated = page.is_truncated;
-
-            for object in page.objects {
-                submitted += 1;
-                self.clone()
-                    .trigger(bucket.clone(), object.key, "write".to_string())
-                    .await;
-            }
-
-            if !is_truncated {
-                break;
-            }
-
-            continuation_token = next_token;
-            if continuation_token.is_none() {
-                break;
-            }
-        }
-
-        submitted
-    }
-
-    pub fn schedule_existing_objects_sync(self: Arc<Self>, bucket: String) {
-        tokio::spawn(async move {
-            let submitted = self
-                .clone()
-                .replicate_existing_objects(bucket.clone())
-                .await;
-            if submitted > 0 {
-                tracing::info!(
-                    "Scheduled {} existing object(s) for replication in {}",
-                    submitted,
-                    bucket
-                );
-            }
-        });
-    }
-
-    async fn replicate_task(
-        &self,
-        bucket: &str,
-        object_key: &str,
-        rule: &ReplicationRule,
-        conn: &RemoteConnection,
-        action: &str,
-    ) {
-        if object_key.contains("..") || object_key.starts_with('/') || object_key.starts_with('\\')
-        {
-            tracing::error!("Invalid object key (path traversal): {}", object_key);
-            return;
-        }
-
-        let client = build_client(conn, &self.client_options);
-
-        if action == "delete" {
-            match client
-                .delete_object()
-                .bucket(&rule.target_bucket)
-                .key(object_key)
-                .send()
-                .await
-            {
-                Ok(_) => {
-                    tracing::info!(
-                        "Replicated DELETE {}/{} to {} ({})",
-                        bucket,
-                        object_key,
-                        conn.name,
-                        rule.target_bucket
-                    );
-                    self.update_last_sync(bucket, object_key);
-                    self.failures.remove(bucket, object_key);
-                }
-                Err(err) => {
-                    let msg = format!("{:?}", err);
-                    tracing::error!(
-                        "Replication DELETE failed {}/{}: {}",
-                        bucket,
-                        object_key,
-                        msg
-                    );
-                    self.failures.add(
-                        bucket,
-                        ReplicationFailure {
-                            object_key: object_key.to_string(),
-                            error_message: msg,
-                            timestamp: now_secs(),
-                            failure_count: 1,
-                            bucket_name: bucket.to_string(),
-                            action: "delete".to_string(),
-                            last_error_code: None,
-                        },
-                    );
-                }
-            }
-            return;
-        }
-
-        let src_path = match self.storage.get_object_path(bucket, object_key).await {
-            Ok(p) => p,
-            Err(_) => {
-                tracing::error!("Source object not found: {}/{}", bucket, object_key);
-                return;
-            }
-        };
-        let file_size = match tokio::fs::metadata(&src_path).await {
-            Ok(m) => m.len(),
-            Err(_) => 0,
-        };
-        let content_type = mime_guess::from_path(&src_path)
-            .first_raw()
-            .map(|s| s.to_string());
-
-        let upload_result = upload_object(
-            &client,
-            &rule.target_bucket,
-            object_key,
-            &src_path,
-            file_size,
-            self.streaming_threshold_bytes,
-            content_type.as_deref(),
-        )
-        .await;
-
-        let final_result = match upload_result {
-            Err(err) if is_no_such_bucket(&err) => {
-                tracing::info!(
-                    "Target bucket {} not found, creating it",
-                    rule.target_bucket
-                );
-                match client
-                    .create_bucket()
-                    .bucket(&rule.target_bucket)
-                    .send()
-                    .await
-                {
-                    Ok(_) | Err(_) => {
-                        upload_object(
-                            &client,
-                            &rule.target_bucket,
-                            object_key,
-                            &src_path,
-                            file_size,
-                            self.streaming_threshold_bytes,
-                            content_type.as_deref(),
-                        )
-                        .await
-                    }
-                }
-            }
-            other => other,
-        };
-
-        match final_result {
-            Ok(()) => {
-                tracing::info!(
-                    "Replicated {}/{} to {} ({})",
-                    bucket,
-                    object_key,
-                    conn.name,
-                    rule.target_bucket
-                );
-                self.update_last_sync(bucket, object_key);
-                self.failures.remove(bucket, object_key);
-            }
-            Err(err) => {
-                let msg = err.to_string();
-                tracing::error!("Replication failed {}/{}: {}", bucket, object_key, msg);
-                self.failures.add(
-                    bucket,
-                    ReplicationFailure {
-                        object_key: object_key.to_string(),
-                        error_message: msg,
-                        timestamp: now_secs(),
-                        failure_count: 1,
-                        bucket_name: bucket.to_string(),
-                        action: action.to_string(),
-                        last_error_code: None,
-                    },
-                );
-            }
-        }
-    }
-
-    pub async fn check_endpoint(&self, conn: &RemoteConnection) -> bool {
-        let client = build_client(conn, &self.client_options);
-        check_endpoint_health(&client).await
-    }
-
-    pub async fn retry_failed(&self, bucket: &str, object_key: &str) -> bool {
-        let failure = match self.failures.get(bucket, object_key) {
-            Some(f) => f,
-            None => return false,
-        };
-        let rule = match self.get_rule(bucket) {
-            Some(r) if r.enabled => r,
-            _ => return false,
-        };
-        let conn = match self.connections.get(&rule.target_connection_id) {
-            Some(c) => c,
-            None => return false,
-        };
-        self.replicate_task(bucket, object_key, &rule, &conn, &failure.action)
-            .await;
-        true
-    }
-
-    pub async fn retry_all(&self, bucket: &str) -> (usize, usize) {
-        let failures = self.failures.load(bucket);
-        if failures.is_empty() {
-            return (0, 0);
-        }
-        let rule = match self.get_rule(bucket) {
-            Some(r) if r.enabled => r,
-            _ => return (0, failures.len()),
-        };
-        let conn = match self.connections.get(&rule.target_connection_id) {
-            Some(c) => c,
-            None => return (0, failures.len()),
-        };
-        let mut submitted = 0;
-        for failure in failures {
-            self.replicate_task(bucket, &failure.object_key, &rule, &conn, &failure.action)
-                .await;
-            submitted += 1;
-        }
-        (submitted, 0)
-    }
-
-    pub fn get_failure_count(&self, bucket: &str) -> usize {
-        self.failures.count(bucket)
-    }
-
-    pub fn get_failed_items(
-        &self,
-        bucket: &str,
-        limit: usize,
-        offset: usize,
-    ) -> Vec<ReplicationFailure> {
-        self.failures
-            .load(bucket)
-            .into_iter()
-            .skip(offset)
-            .take(limit)
-            .collect()
-    }
-
-    pub fn dismiss_failure(&self, bucket: &str, key: &str) -> bool {
-        self.failures.remove(bucket, key)
-    }
-
-    pub fn clear_failures(&self, bucket: &str) {
-        self.failures.clear(bucket);
-    }
-
-    pub fn rules_snapshot(&self) -> HashMap<String, ReplicationRule> {
-        self.rules.lock().clone()
-    }
-
-    pub fn update_last_pull(&self, bucket: &str, at: f64) {
-        {
-            let mut guard = self.rules.lock();
-            if let Some(rule) = guard.get_mut(bucket) {
-                rule.last_pull_at = Some(at);
-            }
-        }
-        self.save_rules();
-    }
-
-    pub fn client_options(&self) -> &ClientOptions {
-        &self.client_options
-    }
-}
-
-fn is_no_such_bucket<E: std::fmt::Debug>(err: &E) -> bool {
-    let text = format!("{:?}", err);
-    text.contains("NoSuchBucket")
-}
-
-async fn upload_object(
-    client: &aws_sdk_s3::Client,
-    bucket: &str,
-    key: &str,
-    path: &Path,
-    file_size: u64,
-    streaming_threshold: u64,
-    content_type: Option<&str>,
-) -> Result<(), aws_sdk_s3::error::SdkError<aws_sdk_s3::operation::put_object::PutObjectError>> {
-    let mut req = client.put_object().bucket(bucket).key(key);
-    if let Some(ct) = content_type {
-        req = req.content_type(ct);
-    }
-
-    let body = if file_size >= streaming_threshold {
-        ByteStream::from_path(path).await.map_err(|e| {
-            aws_sdk_s3::error::SdkError::construction_failure(Box::new(std::io::Error::new(
-                std::io::ErrorKind::Other,
-                e,
-            )))
-        })?
-    } else {
-        let bytes = tokio::fs::read(path)
-            .await
-            .map_err(|e| aws_sdk_s3::error::SdkError::construction_failure(Box::new(e)))?;
-        ByteStream::from(bytes)
-    };
-
-    req.body(body).send().await.map(|_| ())
-}
-
-fn load_rules(path: &Path) -> HashMap<String, ReplicationRule> {
-    if !path.exists() {
-        return HashMap::new();
-    }
-    match std::fs::read_to_string(path) {
-        Ok(text) => serde_json::from_str(&text).unwrap_or_default(),
-        Err(_) => HashMap::new(),
-    }
-}
-
-fn now_secs() -> f64 {
-    SystemTime::now()
-        .duration_since(UNIX_EPOCH)
-        .map(|d| d.as_secs_f64())
-        .unwrap_or(0.0)
-}

rust/myfsio-engine/crates/myfsio-server/src/services/s3_client.rs

@@ -1,64 +0,0 @@
-use std::time::Duration;
-
-use aws_config::BehaviorVersion;
-use aws_credential_types::Credentials;
-use aws_sdk_s3::config::{Region, SharedCredentialsProvider};
-use aws_sdk_s3::Client;
-
-use crate::stores::connections::RemoteConnection;
-
-pub struct ClientOptions {
-    pub connect_timeout: Duration,
-    pub read_timeout: Duration,
-    pub max_attempts: u32,
-}
-
-impl Default for ClientOptions {
-    fn default() -> Self {
-        Self {
-            connect_timeout: Duration::from_secs(5),
-            read_timeout: Duration::from_secs(30),
-            max_attempts: 2,
-        }
-    }
-}
-
-pub fn build_client(connection: &RemoteConnection, options: &ClientOptions) -> Client {
-    let credentials = Credentials::new(
-        connection.access_key.clone(),
-        connection.secret_key.clone(),
-        None,
-        None,
-        "myfsio-replication",
-    );
-
-    let timeout_config = aws_smithy_types::timeout::TimeoutConfig::builder()
-        .connect_timeout(options.connect_timeout)
-        .read_timeout(options.read_timeout)
-        .build();
-
-    let retry_config =
-        aws_smithy_types::retry::RetryConfig::standard().with_max_attempts(options.max_attempts);
-
-    let config = aws_sdk_s3::config::Builder::new()
-        .behavior_version(BehaviorVersion::latest())
-        .credentials_provider(SharedCredentialsProvider::new(credentials))
-        .region(Region::new(connection.region.clone()))
-        .endpoint_url(connection.endpoint_url.clone())
-        .force_path_style(true)
-        .timeout_config(timeout_config)
-        .retry_config(retry_config)
-        .build();
-
-    Client::from_conf(config)
-}
-
-pub async fn check_endpoint_health(client: &Client) -> bool {
-    match client.list_buckets().send().await {
-        Ok(_) => true,
-        Err(err) => {
-            tracing::warn!("Endpoint health check failed: {:?}", err);
-            false
-        }
-    }
-}

rust/myfsio-engine/crates/myfsio-server/src/services/site_registry.rs

@@ -1,148 +0,0 @@
-use chrono::Utc;
-use parking_lot::RwLock;
-use serde::{Deserialize, Serialize};
-use std::path::PathBuf;
-use std::sync::Arc;
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct SiteInfo {
-    pub site_id: String,
-    pub endpoint: String,
-    #[serde(default = "default_region")]
-    pub region: String,
-    #[serde(default = "default_priority")]
-    pub priority: i32,
-    #[serde(default)]
-    pub display_name: String,
-    #[serde(default)]
-    pub created_at: Option<String>,
-}
-
-fn default_region() -> String {
-    "us-east-1".to_string()
-}
-fn default_priority() -> i32 {
-    100
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct PeerSite {
-    pub site_id: String,
-    pub endpoint: String,
-    #[serde(default = "default_region")]
-    pub region: String,
-    #[serde(default = "default_priority")]
-    pub priority: i32,
-    #[serde(default)]
-    pub display_name: String,
-    #[serde(default)]
-    pub connection_id: Option<String>,
-    #[serde(default)]
-    pub created_at: Option<String>,
-    #[serde(default)]
-    pub is_healthy: bool,
-    #[serde(default)]
-    pub last_health_check: Option<String>,
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize, Default)]
-struct RegistryData {
-    #[serde(default)]
-    local: Option<SiteInfo>,
-    #[serde(default)]
-    peers: Vec<PeerSite>,
-}
-
-pub struct SiteRegistry {
-    path: PathBuf,
-    data: Arc<RwLock<RegistryData>>,
-}
-
-impl SiteRegistry {
-    pub fn new(storage_root: &std::path::Path) -> Self {
-        let path = storage_root
-            .join(".myfsio.sys")
-            .join("config")
-            .join("site_registry.json");
-        let data = if path.exists() {
-            std::fs::read_to_string(&path)
-                .ok()
-                .and_then(|s| serde_json::from_str(&s).ok())
-                .unwrap_or_default()
-        } else {
-            RegistryData::default()
-        };
-        Self {
-            path,
-            data: Arc::new(RwLock::new(data)),
-        }
-    }
-
-    fn save(&self) {
-        let data = self.data.read();
-        if let Some(parent) = self.path.parent() {
-            let _ = std::fs::create_dir_all(parent);
-        }
-        if let Ok(json) = serde_json::to_string_pretty(&*data) {
-            let _ = std::fs::write(&self.path, json);
-        }
-    }
-
-    pub fn get_local_site(&self) -> Option<SiteInfo> {
-        self.data.read().local.clone()
-    }
-
-    pub fn set_local_site(&self, site: SiteInfo) {
-        self.data.write().local = Some(site);
-        self.save();
-    }
-
-    pub fn list_peers(&self) -> Vec<PeerSite> {
-        self.data.read().peers.clone()
-    }
-
-    pub fn get_peer(&self, site_id: &str) -> Option<PeerSite> {
-        self.data
-            .read()
-            .peers
-            .iter()
-            .find(|p| p.site_id == site_id)
-            .cloned()
-    }
-
-    pub fn add_peer(&self, peer: PeerSite) {
-        self.data.write().peers.push(peer);
-        self.save();
-    }
-
-    pub fn update_peer(&self, peer: PeerSite) {
-        let mut data = self.data.write();
-        if let Some(existing) = data.peers.iter_mut().find(|p| p.site_id == peer.site_id) {
-            *existing = peer;
-        }
-        drop(data);
-        self.save();
-    }
-
-    pub fn delete_peer(&self, site_id: &str) -> bool {
-        let mut data = self.data.write();
-        let len_before = data.peers.len();
-        data.peers.retain(|p| p.site_id != site_id);
-        let removed = data.peers.len() < len_before;
-        drop(data);
-        if removed {
-            self.save();
-        }
-        removed
-    }
-
-    pub fn update_health(&self, site_id: &str, is_healthy: bool) {
-        let mut data = self.data.write();
-        if let Some(peer) = data.peers.iter_mut().find(|p| p.site_id == site_id) {
-            peer.is_healthy = is_healthy;
-            peer.last_health_check = Some(Utc::now().to_rfc3339());
-        }
-        drop(data);
-        self.save();
-    }
-}