First porting of Python to Rust - update docs and bug fixes

.gitignore

@@ -27,8 +27,11 @@ dist/
 .eggs/
 
 # Rust / maturin build artifacts
-myfsio_core/target/
+python/myfsio_core/target/
-myfsio_core/Cargo.lock
+python/myfsio_core/Cargo.lock
+
+# Rust engine build artifacts
+rust/myfsio-engine/target/
 
 # Local runtime artifacts
 logs/

README.md

@@ -1,255 +1,212 @@
 # MyFSIO
 
-A lightweight, S3-compatible object storage system built with Flask. MyFSIO implements core AWS S3 REST API operations with filesystem-backed storage, making it ideal for local development, testing, and self-hosted storage scenarios.
+MyFSIO is an S3-compatible object storage server with a Rust runtime and a filesystem-backed storage engine. The active server lives under `rust/myfsio-engine` and serves both the S3 API and the built-in web UI from a single process.
+
+The repository still contains a `python/` tree, but you do not need Python to run the current server.
 
 ## Features
 
-**Core Storage**
+- S3-compatible REST API with Signature Version 4 authentication
-- S3-compatible REST API with AWS Signature Version 4 authentication
+- Browser UI for buckets, objects, IAM users, policies, replication, metrics, and site administration
-- Bucket and object CRUD operations
+- Filesystem-backed storage rooted at `data/`
-- Object versioning with version history
+- Bucket versioning, multipart uploads, presigned URLs, CORS, object and bucket tagging
-- Multipart uploads for large files
+- Server-side encryption and built-in KMS support
-- Presigned URLs (1 second to 7 days validity)
+- Optional background services for lifecycle, garbage collection, integrity scanning, operation metrics, and system metrics history
+- Replication, site sync, and static website hosting support
 
-**Security & Access Control**
+## Runtime Model
-- IAM users with access key management and rotation
-- Bucket policies (AWS Policy Version 2012-10-17)
-- Server-side encryption (SSE-S3 and SSE-KMS)
-- Built-in Key Management Service (KMS)
-- Rate limiting per endpoint
 
-**Advanced Features**
+MyFSIO now runs as one Rust process:
-- Cross-bucket replication to remote S3-compatible endpoints
-- Hot-reload for bucket policies (no restart required)
-- CORS configuration per bucket
 
-**Management UI**
+- API listener on `HOST` + `PORT` (default `127.0.0.1:5000`)
-- Web console for bucket and object management
+- UI listener on `HOST` + `UI_PORT` (default `127.0.0.1:5100`)
-- IAM dashboard for user administration
+- Shared state for storage, IAM, policies, sessions, metrics, and background workers
-- Inline JSON policy editor with presets
-- Object browser with folder navigation and bulk operations
-- Dark mode support
 
-## Architecture
+If you want API-only mode, set `UI_ENABLED=false`. There is no separate "UI-only" runtime anymore.
-
-```
-+------------------+         +------------------+
-|   API Server     |         |   UI Server      |
-|   (port 5000)    |         |   (port 5100)    |
-|                  |         |                  |
-|  - S3 REST API   |<------->|  - Web Console   |
-|  - SigV4 Auth    |         |  - IAM Dashboard |
-|  - Presign URLs  |         |  - Bucket Editor |
-+--------+---------+         +------------------+
-         |
-         v
-+------------------+         +------------------+
-| Object Storage   |         | System Metadata  |
-| (filesystem)     |         | (.myfsio.sys/)   |
-|                  |         |                  |
-| data/<bucket>/   |         | - IAM config     |
-|   <objects>      |         | - Bucket policies|
-|                  |         | - Encryption keys|
-+------------------+         +------------------+
-```
 
 ## Quick Start
 
+From the repository root:
+
 ```bash
-# Clone and setup
+cd rust/myfsio-engine
-git clone https://gitea.jzwsite.com/kqjy/MyFSIO
+cargo run -p myfsio-server --
-cd s3
-python -m venv .venv
-
-# Activate virtual environment
-# Windows PowerShell:
-.\.venv\Scripts\Activate.ps1
-# Windows CMD:
-.venv\Scripts\activate.bat
-# Linux/macOS:
-source .venv/bin/activate
-
-# Install dependencies
-pip install -r requirements.txt
-
-# (Optional) Build Rust native extension for better performance
-# Requires Rust toolchain: https://rustup.rs
-pip install maturin
-cd myfsio_core && maturin develop --release && cd ..
-
-# Start both servers
-python run.py
-
-# Or start individually
-python run.py --mode api   # API only (port 5000)
-python run.py --mode ui    # UI only (port 5100)
 ```
 
-**Credentials:** Generated automatically on first run and printed to the console. If missed, check the IAM config file at `<STORAGE_ROOT>/.myfsio.sys/config/iam.json`.
+Useful URLs:
 
-- **Web Console:** http://127.0.0.1:5100/ui
+- UI: `http://127.0.0.1:5100/ui`
-- **API Endpoint:** http://127.0.0.1:5000
+- API: `http://127.0.0.1:5000/`
+- Health: `http://127.0.0.1:5000/myfsio/health`
+
+On first boot, MyFSIO creates `data/.myfsio.sys/config/iam.json` and prints the generated admin access key and secret key to the console.
+
+### Common CLI commands
+
+```bash
+# Show resolved configuration
+cargo run -p myfsio-server -- --show-config
+
+# Validate configuration and exit non-zero on critical issues
+cargo run -p myfsio-server -- --check-config
+
+# Reset admin credentials
+cargo run -p myfsio-server -- --reset-cred
+
+# API only
+UI_ENABLED=false cargo run -p myfsio-server --
+```
+
+## Building a Binary
+
+```bash
+cd rust/myfsio-engine
+cargo build --release -p myfsio-server
+```
+
+Binary locations:
+
+- Linux/macOS: `rust/myfsio-engine/target/release/myfsio-server`
+- Windows: `rust/myfsio-engine/target/release/myfsio-server.exe`
+
+Run the built binary directly:
+
+```bash
+./target/release/myfsio-server
+```
 
 ## Configuration
 
+The server reads environment variables from the process environment and also loads, when present:
+
+- `/opt/myfsio/myfsio.env`
+- `.env`
+- `myfsio.env`
+
+Core settings:
+
 | Variable | Default | Description |
-|----------|---------|-------------|
+| --- | --- | --- |
-| `STORAGE_ROOT` | `./data` | Filesystem root for bucket storage |
+| `HOST` | `127.0.0.1` | Bind address for API and UI listeners |
-| `IAM_CONFIG` | `.myfsio.sys/config/iam.json` | IAM user and policy store |
+| `PORT` | `5000` | API port |
-| `BUCKET_POLICY_PATH` | `.myfsio.sys/config/bucket_policies.json` | Bucket policy store |
+| `UI_PORT` | `5100` | UI port |
-| `API_BASE_URL` | `http://127.0.0.1:5000` | API endpoint for UI calls |
+| `UI_ENABLED` | `true` | Disable to run API-only |
-| `MAX_UPLOAD_SIZE` | `1073741824` | Maximum upload size in bytes (1 GB) |
+| `STORAGE_ROOT` | `./data` | Root directory for buckets and system metadata |
-| `MULTIPART_MIN_PART_SIZE` | `5242880` | Minimum multipart part size (5 MB) |
+| `IAM_CONFIG` | `<STORAGE_ROOT>/.myfsio.sys/config/iam.json` | IAM config path |
-| `UI_PAGE_SIZE` | `100` | Default page size for listings |
+| `API_BASE_URL` | unset | Public API base used by the UI and presigned URL generation |
-| `SECRET_KEY` | `dev-secret-key` | Flask session secret |
+| `AWS_REGION` | `us-east-1` | Region used in SigV4 scope |
-| `AWS_REGION` | `us-east-1` | Region for SigV4 signing |
+| `SIGV4_TIMESTAMP_TOLERANCE_SECONDS` | `900` | Allowed request time skew |
-| `AWS_SERVICE` | `s3` | Service name for SigV4 signing |
+| `PRESIGNED_URL_MIN_EXPIRY_SECONDS` | `1` | Minimum presigned URL expiry |
-| `ENCRYPTION_ENABLED` | `false` | Enable server-side encryption |
+| `PRESIGNED_URL_MAX_EXPIRY_SECONDS` | `604800` | Maximum presigned URL expiry |
-| `KMS_ENABLED` | `false` | Enable Key Management Service |
+| `SECRET_KEY` | loaded from `.myfsio.sys/config/.secret` if present | Session signing key and IAM-at-rest encryption key |
-| `LOG_LEVEL` | `INFO` | Logging verbosity |
+| `ADMIN_ACCESS_KEY` | unset | Optional first-run or reset access key |
-| `SIGV4_TIMESTAMP_TOLERANCE_SECONDS` | `900` | Max time skew for SigV4 requests |
+| `ADMIN_SECRET_KEY` | unset | Optional first-run or reset secret key |
-| `PRESIGNED_URL_MAX_EXPIRY_SECONDS` | `604800` | Max presigned URL expiry (7 days) |
+
-| `REPLICATION_CONNECT_TIMEOUT_SECONDS` | `5` | Replication connection timeout |
+Feature toggles:
-| `SITE_SYNC_ENABLED` | `false` | Enable bi-directional site sync |
+
-| `OBJECT_TAG_LIMIT` | `50` | Maximum tags per object |
+| Variable | Default |
+| --- | --- |
+| `ENCRYPTION_ENABLED` | `false` |
+| `KMS_ENABLED` | `false` |
+| `GC_ENABLED` | `false` |
+| `INTEGRITY_ENABLED` | `false` |
+| `LIFECYCLE_ENABLED` | `false` |
+| `METRICS_HISTORY_ENABLED` | `false` |
+| `OPERATION_METRICS_ENABLED` | `false` |
+| `WEBSITE_HOSTING_ENABLED` | `false` |
+| `SITE_SYNC_ENABLED` | `false` |
+
+Metrics and replication tuning:
+
+| Variable | Default |
+| --- | --- |
+| `OPERATION_METRICS_INTERVAL_MINUTES` | `5` |
+| `OPERATION_METRICS_RETENTION_HOURS` | `24` |
+| `METRICS_HISTORY_INTERVAL_MINUTES` | `5` |
+| `METRICS_HISTORY_RETENTION_HOURS` | `24` |
+| `REPLICATION_CONNECT_TIMEOUT_SECONDS` | `5` |
+| `REPLICATION_READ_TIMEOUT_SECONDS` | `30` |
+| `REPLICATION_MAX_RETRIES` | `2` |
+| `REPLICATION_STREAMING_THRESHOLD_BYTES` | `10485760` |
+| `REPLICATION_MAX_FAILURES_PER_BUCKET` | `50` |
+| `SITE_SYNC_INTERVAL_SECONDS` | `60` |
+| `SITE_SYNC_BATCH_SIZE` | `100` |
+| `SITE_SYNC_CONNECT_TIMEOUT_SECONDS` | `10` |
+| `SITE_SYNC_READ_TIMEOUT_SECONDS` | `120` |
+| `SITE_SYNC_MAX_RETRIES` | `2` |
+| `SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS` | `1.0` |
+
+UI asset overrides:
+
+| Variable | Default |
+| --- | --- |
+| `TEMPLATES_DIR` | built-in crate templates directory |
+| `STATIC_DIR` | built-in crate static directory |
+
+See [docs.md](./docs.md) for the full Rust-side operations guide.
 
 ## Data Layout
 
-```
+```text
 data/
-├── <bucket>/                    # User buckets with objects
+  <bucket>/
-└── .myfsio.sys/                 # System metadata
+  .myfsio.sys/
-    ├── config/
+    config/
-    │   ├── iam.json             # IAM users and policies
+      iam.json
-    │   ├── bucket_policies.json # Bucket policies
+      bucket_policies.json
-    │   ├── replication_rules.json
+      connections.json
-    │   └── connections.json     # Remote S3 connections
+      operation_metrics.json
-    ├── buckets/<bucket>/
+      metrics_history.json
-    │   ├── meta/                # Object metadata (.meta.json)
+    buckets/<bucket>/
-    │   ├── versions/            # Archived object versions
+      meta/
-    │   └── .bucket.json         # Bucket config (versioning, CORS)
+      versions/
-    ├── multipart/               # Active multipart uploads
+    multipart/
-    └── keys/                    # Encryption keys (SSE-S3/KMS)
+    keys/
 ```
 
-## API Reference
-
-All endpoints require AWS Signature Version 4 authentication unless using presigned URLs or public bucket policies.
-
-### Bucket Operations
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/` | List all buckets |
-| `PUT` | `/<bucket>` | Create bucket |
-| `DELETE` | `/<bucket>` | Delete bucket (must be empty) |
-| `HEAD` | `/<bucket>` | Check bucket exists |
-
-### Object Operations
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/<bucket>` | List objects (supports `list-type=2`) |
-| `PUT` | `/<bucket>/<key>` | Upload object |
-| `GET` | `/<bucket>/<key>` | Download object |
-| `DELETE` | `/<bucket>/<key>` | Delete object |
-| `HEAD` | `/<bucket>/<key>` | Get object metadata |
-| `POST` | `/<bucket>/<key>?uploads` | Initiate multipart upload |
-| `PUT` | `/<bucket>/<key>?partNumber=N&uploadId=X` | Upload part |
-| `POST` | `/<bucket>/<key>?uploadId=X` | Complete multipart upload |
-| `DELETE` | `/<bucket>/<key>?uploadId=X` | Abort multipart upload |
-
-### Bucket Policies (S3-compatible)
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/<bucket>?policy` | Get bucket policy |
-| `PUT` | `/<bucket>?policy` | Set bucket policy |
-| `DELETE` | `/<bucket>?policy` | Delete bucket policy |
-
-### Versioning
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/<bucket>/<key>?versionId=X` | Get specific version |
-| `DELETE` | `/<bucket>/<key>?versionId=X` | Delete specific version |
-| `GET` | `/<bucket>?versions` | List object versions |
-
-### Health Check
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/myfsio/health` | Health check endpoint |
-
-## IAM & Access Control
-
-### Users and Access Keys
-
-On first run, MyFSIO creates a default admin user (`localadmin`/`localadmin`). Use the IAM dashboard to:
-
-- Create and delete users
-- Generate and rotate access keys
-- Attach inline policies to users
-- Control IAM management permissions
-
-### Bucket Policies
-
-Bucket policies follow AWS policy grammar (Version `2012-10-17`) with support for:
-
-- Principal-based access (`*` for anonymous, specific users)
-- Action-based permissions (`s3:GetObject`, `s3:PutObject`, etc.)
-- Resource patterns (`arn:aws:s3:::bucket/*`)
-- Condition keys
-
-**Policy Presets:**
-- **Public:** Grants anonymous read access (`s3:GetObject`, `s3:ListBucket`)
-- **Private:** Removes bucket policy (IAM-only access)
-- **Custom:** Manual policy editing with draft preservation
-
-Policies hot-reload when the JSON file changes.
-
-## Server-Side Encryption
-
-MyFSIO supports two encryption modes:
-
-- **SSE-S3:** Server-managed keys with automatic key rotation
-- **SSE-KMS:** Customer-managed keys via built-in KMS
-
-Enable encryption with:
-```bash
-ENCRYPTION_ENABLED=true python run.py
-```
-
-## Cross-Bucket Replication
-
-Replicate objects to remote S3-compatible endpoints:
-
-1. Configure remote connections in the UI
-2. Create replication rules specifying source/destination
-3. Objects are automatically replicated on upload
-
 ## Docker
 
+Build the Rust image from the `rust/` directory:
+
 ```bash
-docker build -t myfsio .
+docker build -t myfsio ./rust
-docker run -p 5000:5000 -p 5100:5100 -v ./data:/app/data myfsio
+docker run --rm -p 5000:5000 -p 5100:5100 -v "${PWD}/data:/app/data" myfsio
 ```
 
+If the instance sits behind a reverse proxy, set `API_BASE_URL` to the public S3 endpoint.
+
+## Linux Installation
+
+The repository includes `scripts/install.sh` for systemd-style Linux installs. Build the Rust binary first, then pass it to the installer:
+
+```bash
+cd rust/myfsio-engine
+cargo build --release -p myfsio-server
+
+cd ../..
+sudo ./scripts/install.sh --binary ./rust/myfsio-engine/target/release/myfsio-server
+```
+
+The installer copies the binary into `/opt/myfsio/myfsio`, writes `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` unit.
+
 ## Testing
 
+Run the Rust test suite from the workspace:
+
 ```bash
-# Run all tests
+cd rust/myfsio-engine
-pytest tests/ -v
+cargo test
-
-# Run specific test file
-pytest tests/test_api.py -v
-
-# Run with coverage
-pytest tests/ --cov=app --cov-report=html
 ```
 
-## References
+## Health Check
 
-- [Amazon S3 Documentation](https://docs.aws.amazon.com/s3/)
+`GET /myfsio/health` returns:
-- [AWS Signature Version 4](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)
+
-- [S3 Bucket Policy Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html)
+```json
+{
+  "status": "ok",
+  "version": "0.5.0"
+}
+```
+
+The `version` field comes from the Rust crate version in `rust/myfsio-engine/crates/myfsio-server/Cargo.toml`.

docker-entrypoint.sh

@@ -1,5 +0,0 @@
-#!/bin/sh
-set -e
-
-# Run both services using the python runner in production mode
-exec python run.py --prod

python/.dockerignore

@@ -11,3 +11,7 @@ htmlcov
 logs
 data
 tmp
+tests
+myfsio_core/target
+Dockerfile
+.dockerignore

python/Dockerfile

@@ -1,9 +1,9 @@
-FROM python:3.14.3-slim
+FROM python:3.14.3-slim AS builder
 
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1
 
-WORKDIR /app
+WORKDIR /build
 
 RUN apt-get update \
     && apt-get install -y --no-install-recommends build-essential curl \
@@ -12,24 +12,35 @@ RUN apt-get update \
 
 ENV PATH="/root/.cargo/bin:${PATH}"
 
+RUN pip install --no-cache-dir maturin
+
+COPY myfsio_core ./myfsio_core
+RUN cd myfsio_core \
+    && maturin build --release --out /wheels
+
+
+FROM python:3.14.3-slim
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1
+
+WORKDIR /app
+
 COPY requirements.txt ./
 RUN pip install --no-cache-dir -r requirements.txt
 
-COPY . .
+COPY --from=builder /wheels/*.whl /tmp/
+RUN pip install --no-cache-dir /tmp/*.whl && rm /tmp/*.whl
 
-RUN pip install --no-cache-dir maturin \
+COPY app ./app
-    && cd myfsio_core \
+COPY templates ./templates
-    && maturin build --release \
+COPY static ./static
-    && pip install target/wheels/*.whl \
+COPY run.py ./
-    && cd .. \
+COPY docker-entrypoint.sh ./
-    && rm -rf myfsio_core/target \
-    && pip uninstall -y maturin \
-    && rustup self uninstall -y
 
-RUN chmod +x docker-entrypoint.sh
+RUN chmod +x docker-entrypoint.sh \
-
+    && mkdir -p /app/data \
-RUN mkdir -p /app/data \
+    && useradd -m -u 1000 myfsio \
-    && useradd -m -u 1000 myfsio \ 
     && chown -R myfsio:myfsio /app
 
 USER myfsio

python/app/ui.py

@@ -225,10 +225,10 @@ def _policy_allows_public_read(policy: dict[str, Any]) -> bool:
 
 def _bucket_access_descriptor(policy: dict[str, Any] | None) -> tuple[str, str]:
     if not policy:
-        return ("IAM only", "text-bg-secondary")
+        return ("IAM only", "bg-secondary-subtle text-secondary-emphasis")
     if _policy_allows_public_read(policy):
-        return ("Public read", "text-bg-warning")
+        return ("Public read", "bg-warning-subtle text-warning-emphasis")
-    return ("Custom policy", "text-bg-info")
+    return ("Custom policy", "bg-info-subtle text-info-emphasis")
 
 
 def _current_principal():

python/app/version.py

@@ -1,6 +1,6 @@
 from __future__ import annotations
 
-APP_VERSION = "0.4.2"
+APP_VERSION = "0.4.3"
 
 
 def get_version() -> str:

python/docker-entrypoint.sh

@@ -0,0 +1,4 @@
+#!/bin/sh
+set -e
+
+exec python run.py --prod

python/myfsio_core/src/storage.rs

@@ -125,7 +125,7 @@ pub fn delete_index_entry(py: Python<'_>, path: &str, entry_name: &str) -> PyRes
         fs::write(&path_owned, serialized)
             .map_err(|e| PyIOError::new_err(format!("Failed to write index: {}", e)))?;
 
-        Ok(false)
+        Ok(true)
     })
 }
 

python/templates/bucket_detail.html

@@ -921,14 +921,14 @@
                   <path d="M8 16A8 8 0 1 0 8 0a8 8 0 0 0 0 16zm.93-9.412-1 4.705c-.07.34.029.533.304.533.194 0 .487-.07.686-.246l-.088.416c-.287.346-.92.598-1.465.598-.703 0-1.002-.422-.808-1.319l.738-3.468c.064-.293.006-.399-.287-.47l-.451-.081.082-.381 2.29-.287zM8 5.5a1 1 0 1 1 0-2 1 1 0 0 1 0 2z"/>
                 </svg>
                 <div>
-                  <strong>Storage quota enabled</strong>
+                  <strong>Storage quota active</strong>
                   <p class="mb-0 small">
                     {% if max_bytes is not none and max_objects is not none %}
-                    Limited to {{ max_bytes | filesizeformat }} and {{ max_objects }} objects.
+                    This bucket is limited to {{ max_bytes | filesizeformat }} storage and {{ max_objects }} objects.
                     {% elif max_bytes is not none %}
-                    Limited to {{ max_bytes | filesizeformat }} storage.
+                    This bucket is limited to {{ max_bytes | filesizeformat }} storage.
                     {% else %}
-                    Limited to {{ max_objects }} objects.
+                    This bucket is limited to {{ max_objects }} objects.
                     {% endif %}
                   </p>
                 </div>