diff --git a/README.md b/README.md index 772178d..b9a6318 100644 --- a/README.md +++ b/README.md @@ -1,255 +1,212 @@ # MyFSIO -A lightweight, S3-compatible object storage system built with Flask. MyFSIO implements core AWS S3 REST API operations with filesystem-backed storage, making it ideal for local development, testing, and self-hosted storage scenarios. +MyFSIO is an S3-compatible object storage server with a Rust runtime and a filesystem-backed storage engine. The active server lives under `rust/myfsio-engine` and serves both the S3 API and the built-in web UI from a single process. + +The repository still contains a `python/` tree, but you do not need Python to run the current server. ## Features -**Core Storage** -- S3-compatible REST API with AWS Signature Version 4 authentication -- Bucket and object CRUD operations -- Object versioning with version history -- Multipart uploads for large files -- Presigned URLs (1 second to 7 days validity) +- S3-compatible REST API with Signature Version 4 authentication +- Browser UI for buckets, objects, IAM users, policies, replication, metrics, and site administration +- Filesystem-backed storage rooted at `data/` +- Bucket versioning, multipart uploads, presigned URLs, CORS, object and bucket tagging +- Server-side encryption and built-in KMS support +- Optional background services for lifecycle, garbage collection, integrity scanning, operation metrics, and system metrics history +- Replication, site sync, and static website hosting support -**Security & Access Control** -- IAM users with access key management and rotation -- Bucket policies (AWS Policy Version 2012-10-17) -- Server-side encryption (SSE-S3 and SSE-KMS) -- Built-in Key Management Service (KMS) -- Rate limiting per endpoint +## Runtime Model -**Advanced Features** -- Cross-bucket replication to remote S3-compatible endpoints -- Hot-reload for bucket policies (no restart required) -- CORS configuration per bucket +MyFSIO now runs as one Rust process: -**Management UI** -- Web console for bucket and object management -- IAM dashboard for user administration -- Inline JSON policy editor with presets -- Object browser with folder navigation and bulk operations -- Dark mode support +- API listener on `HOST` + `PORT` (default `127.0.0.1:5000`) +- UI listener on `HOST` + `UI_PORT` (default `127.0.0.1:5100`) +- Shared state for storage, IAM, policies, sessions, metrics, and background workers -## Architecture - -``` -+------------------+ +------------------+ -| API Server | | UI Server | -| (port 5000) | | (port 5100) | -| | | | -| - S3 REST API |<------->| - Web Console | -| - SigV4 Auth | | - IAM Dashboard | -| - Presign URLs | | - Bucket Editor | -+--------+---------+ +------------------+ - | - v -+------------------+ +------------------+ -| Object Storage | | System Metadata | -| (filesystem) | | (.myfsio.sys/) | -| | | | -| data// | | - IAM config | -| | | - Bucket policies| -| | | - Encryption keys| -+------------------+ +------------------+ -``` +If you want API-only mode, set `UI_ENABLED=false`. There is no separate "UI-only" runtime anymore. ## Quick Start +From the repository root: + ```bash -# Clone and setup -git clone https://gitea.jzwsite.com/kqjy/MyFSIO -cd s3 -python -m venv .venv - -# Activate virtual environment -# Windows PowerShell: -.\.venv\Scripts\Activate.ps1 -# Windows CMD: -.venv\Scripts\activate.bat -# Linux/macOS: -source .venv/bin/activate - -# Install dependencies -pip install -r requirements.txt - -# (Optional) Build Rust native extension for better performance -# Requires Rust toolchain: https://rustup.rs -pip install maturin -cd myfsio_core && maturin develop --release && cd .. - -# Start both servers -python run.py - -# Or start individually -python run.py --mode api # API only (port 5000) -python run.py --mode ui # UI only (port 5100) +cd rust/myfsio-engine +cargo run -p myfsio-server -- ``` -**Credentials:** Generated automatically on first run and printed to the console. If missed, check the IAM config file at `/.myfsio.sys/config/iam.json`. +Useful URLs: -- **Web Console:** http://127.0.0.1:5100/ui -- **API Endpoint:** http://127.0.0.1:5000 +- UI: `http://127.0.0.1:5100/ui` +- API: `http://127.0.0.1:5000/` +- Health: `http://127.0.0.1:5000/myfsio/health` + +On first boot, MyFSIO creates `data/.myfsio.sys/config/iam.json` and prints the generated admin access key and secret key to the console. + +### Common CLI commands + +```bash +# Show resolved configuration +cargo run -p myfsio-server -- --show-config + +# Validate configuration and exit non-zero on critical issues +cargo run -p myfsio-server -- --check-config + +# Reset admin credentials +cargo run -p myfsio-server -- --reset-cred + +# API only +UI_ENABLED=false cargo run -p myfsio-server -- +``` + +## Building a Binary + +```bash +cd rust/myfsio-engine +cargo build --release -p myfsio-server +``` + +Binary locations: + +- Linux/macOS: `rust/myfsio-engine/target/release/myfsio-server` +- Windows: `rust/myfsio-engine/target/release/myfsio-server.exe` + +Run the built binary directly: + +```bash +./target/release/myfsio-server +``` ## Configuration +The server reads environment variables from the process environment and also loads, when present: + +- `/opt/myfsio/myfsio.env` +- `.env` +- `myfsio.env` + +Core settings: + | Variable | Default | Description | -|----------|---------|-------------| -| `STORAGE_ROOT` | `./data` | Filesystem root for bucket storage | -| `IAM_CONFIG` | `.myfsio.sys/config/iam.json` | IAM user and policy store | -| `BUCKET_POLICY_PATH` | `.myfsio.sys/config/bucket_policies.json` | Bucket policy store | -| `API_BASE_URL` | `http://127.0.0.1:5000` | API endpoint for UI calls | -| `MAX_UPLOAD_SIZE` | `1073741824` | Maximum upload size in bytes (1 GB) | -| `MULTIPART_MIN_PART_SIZE` | `5242880` | Minimum multipart part size (5 MB) | -| `UI_PAGE_SIZE` | `100` | Default page size for listings | -| `SECRET_KEY` | `dev-secret-key` | Flask session secret | -| `AWS_REGION` | `us-east-1` | Region for SigV4 signing | -| `AWS_SERVICE` | `s3` | Service name for SigV4 signing | -| `ENCRYPTION_ENABLED` | `false` | Enable server-side encryption | -| `KMS_ENABLED` | `false` | Enable Key Management Service | -| `LOG_LEVEL` | `INFO` | Logging verbosity | -| `SIGV4_TIMESTAMP_TOLERANCE_SECONDS` | `900` | Max time skew for SigV4 requests | -| `PRESIGNED_URL_MAX_EXPIRY_SECONDS` | `604800` | Max presigned URL expiry (7 days) | -| `REPLICATION_CONNECT_TIMEOUT_SECONDS` | `5` | Replication connection timeout | -| `SITE_SYNC_ENABLED` | `false` | Enable bi-directional site sync | -| `OBJECT_TAG_LIMIT` | `50` | Maximum tags per object | +| --- | --- | --- | +| `HOST` | `127.0.0.1` | Bind address for API and UI listeners | +| `PORT` | `5000` | API port | +| `UI_PORT` | `5100` | UI port | +| `UI_ENABLED` | `true` | Disable to run API-only | +| `STORAGE_ROOT` | `./data` | Root directory for buckets and system metadata | +| `IAM_CONFIG` | `/.myfsio.sys/config/iam.json` | IAM config path | +| `API_BASE_URL` | unset | Public API base used by the UI and presigned URL generation | +| `AWS_REGION` | `us-east-1` | Region used in SigV4 scope | +| `SIGV4_TIMESTAMP_TOLERANCE_SECONDS` | `900` | Allowed request time skew | +| `PRESIGNED_URL_MIN_EXPIRY_SECONDS` | `1` | Minimum presigned URL expiry | +| `PRESIGNED_URL_MAX_EXPIRY_SECONDS` | `604800` | Maximum presigned URL expiry | +| `SECRET_KEY` | loaded from `.myfsio.sys/config/.secret` if present | Session signing key and IAM-at-rest encryption key | +| `ADMIN_ACCESS_KEY` | unset | Optional first-run or reset access key | +| `ADMIN_SECRET_KEY` | unset | Optional first-run or reset secret key | + +Feature toggles: + +| Variable | Default | +| --- | --- | +| `ENCRYPTION_ENABLED` | `false` | +| `KMS_ENABLED` | `false` | +| `GC_ENABLED` | `false` | +| `INTEGRITY_ENABLED` | `false` | +| `LIFECYCLE_ENABLED` | `false` | +| `METRICS_HISTORY_ENABLED` | `false` | +| `OPERATION_METRICS_ENABLED` | `false` | +| `WEBSITE_HOSTING_ENABLED` | `false` | +| `SITE_SYNC_ENABLED` | `false` | + +Metrics and replication tuning: + +| Variable | Default | +| --- | --- | +| `OPERATION_METRICS_INTERVAL_MINUTES` | `5` | +| `OPERATION_METRICS_RETENTION_HOURS` | `24` | +| `METRICS_HISTORY_INTERVAL_MINUTES` | `5` | +| `METRICS_HISTORY_RETENTION_HOURS` | `24` | +| `REPLICATION_CONNECT_TIMEOUT_SECONDS` | `5` | +| `REPLICATION_READ_TIMEOUT_SECONDS` | `30` | +| `REPLICATION_MAX_RETRIES` | `2` | +| `REPLICATION_STREAMING_THRESHOLD_BYTES` | `10485760` | +| `REPLICATION_MAX_FAILURES_PER_BUCKET` | `50` | +| `SITE_SYNC_INTERVAL_SECONDS` | `60` | +| `SITE_SYNC_BATCH_SIZE` | `100` | +| `SITE_SYNC_CONNECT_TIMEOUT_SECONDS` | `10` | +| `SITE_SYNC_READ_TIMEOUT_SECONDS` | `120` | +| `SITE_SYNC_MAX_RETRIES` | `2` | +| `SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS` | `1.0` | + +UI asset overrides: + +| Variable | Default | +| --- | --- | +| `TEMPLATES_DIR` | built-in crate templates directory | +| `STATIC_DIR` | built-in crate static directory | + +See [docs.md](./docs.md) for the full Rust-side operations guide. ## Data Layout -``` +```text data/ -├── / # User buckets with objects -└── .myfsio.sys/ # System metadata - ├── config/ - │ ├── iam.json # IAM users and policies - │ ├── bucket_policies.json # Bucket policies - │ ├── replication_rules.json - │ └── connections.json # Remote S3 connections - ├── buckets// - │ ├── meta/ # Object metadata (.meta.json) - │ ├── versions/ # Archived object versions - │ └── .bucket.json # Bucket config (versioning, CORS) - ├── multipart/ # Active multipart uploads - └── keys/ # Encryption keys (SSE-S3/KMS) + / + .myfsio.sys/ + config/ + iam.json + bucket_policies.json + connections.json + operation_metrics.json + metrics_history.json + buckets// + meta/ + versions/ + multipart/ + keys/ ``` -## API Reference - -All endpoints require AWS Signature Version 4 authentication unless using presigned URLs or public bucket policies. - -### Bucket Operations - -| Method | Endpoint | Description | -|--------|----------|-------------| -| `GET` | `/` | List all buckets | -| `PUT` | `/` | Create bucket | -| `DELETE` | `/` | Delete bucket (must be empty) | -| `HEAD` | `/` | Check bucket exists | - -### Object Operations - -| Method | Endpoint | Description | -|--------|----------|-------------| -| `GET` | `/` | List objects (supports `list-type=2`) | -| `PUT` | `//` | Upload object | -| `GET` | `//` | Download object | -| `DELETE` | `//` | Delete object | -| `HEAD` | `//` | Get object metadata | -| `POST` | `//?uploads` | Initiate multipart upload | -| `PUT` | `//?partNumber=N&uploadId=X` | Upload part | -| `POST` | `//?uploadId=X` | Complete multipart upload | -| `DELETE` | `//?uploadId=X` | Abort multipart upload | - -### Bucket Policies (S3-compatible) - -| Method | Endpoint | Description | -|--------|----------|-------------| -| `GET` | `/?policy` | Get bucket policy | -| `PUT` | `/?policy` | Set bucket policy | -| `DELETE` | `/?policy` | Delete bucket policy | - -### Versioning - -| Method | Endpoint | Description | -|--------|----------|-------------| -| `GET` | `//?versionId=X` | Get specific version | -| `DELETE` | `//?versionId=X` | Delete specific version | -| `GET` | `/?versions` | List object versions | - -### Health Check - -| Method | Endpoint | Description | -|--------|----------|-------------| -| `GET` | `/myfsio/health` | Health check endpoint | - -## IAM & Access Control - -### Users and Access Keys - -On first run, MyFSIO creates a default admin user (`localadmin`/`localadmin`). Use the IAM dashboard to: - -- Create and delete users -- Generate and rotate access keys -- Attach inline policies to users -- Control IAM management permissions - -### Bucket Policies - -Bucket policies follow AWS policy grammar (Version `2012-10-17`) with support for: - -- Principal-based access (`*` for anonymous, specific users) -- Action-based permissions (`s3:GetObject`, `s3:PutObject`, etc.) -- Resource patterns (`arn:aws:s3:::bucket/*`) -- Condition keys - -**Policy Presets:** -- **Public:** Grants anonymous read access (`s3:GetObject`, `s3:ListBucket`) -- **Private:** Removes bucket policy (IAM-only access) -- **Custom:** Manual policy editing with draft preservation - -Policies hot-reload when the JSON file changes. - -## Server-Side Encryption - -MyFSIO supports two encryption modes: - -- **SSE-S3:** Server-managed keys with automatic key rotation -- **SSE-KMS:** Customer-managed keys via built-in KMS - -Enable encryption with: -```bash -ENCRYPTION_ENABLED=true python run.py -``` - -## Cross-Bucket Replication - -Replicate objects to remote S3-compatible endpoints: - -1. Configure remote connections in the UI -2. Create replication rules specifying source/destination -3. Objects are automatically replicated on upload - ## Docker +Build the Rust image from the `rust/` directory: + ```bash -docker build -t myfsio . -docker run -p 5000:5000 -p 5100:5100 -v ./data:/app/data myfsio +docker build -t myfsio ./rust +docker run --rm -p 5000:5000 -p 5100:5100 -v "${PWD}/data:/app/data" myfsio ``` +If the instance sits behind a reverse proxy, set `API_BASE_URL` to the public S3 endpoint. + +## Linux Installation + +The repository includes `scripts/install.sh` for systemd-style Linux installs. Build the Rust binary first, then pass it to the installer: + +```bash +cd rust/myfsio-engine +cargo build --release -p myfsio-server + +cd ../.. +sudo ./scripts/install.sh --binary ./rust/myfsio-engine/target/release/myfsio-server +``` + +The installer copies the binary into `/opt/myfsio/myfsio`, writes `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` unit. + ## Testing +Run the Rust test suite from the workspace: + ```bash -# Run all tests -pytest tests/ -v - -# Run specific test file -pytest tests/test_api.py -v - -# Run with coverage -pytest tests/ --cov=app --cov-report=html +cd rust/myfsio-engine +cargo test ``` -## References +## Health Check -- [Amazon S3 Documentation](https://docs.aws.amazon.com/s3/) -- [AWS Signature Version 4](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) -- [S3 Bucket Policy Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html) +`GET /myfsio/health` returns: + +```json +{ + "status": "ok", + "version": "0.5.0" +} +``` + +The `version` field comes from the Rust crate version in `rust/myfsio-engine/crates/myfsio-server/Cargo.toml`. diff --git a/docs.md b/docs.md index ba36f68..b2438b7 100644 --- a/docs.md +++ b/docs.md @@ -1,2605 +1,419 @@ -# MyFSIO Documentation +# MyFSIO Rust Operations Guide -This document expands on the README to describe the full workflow for running, configuring, and extending MyFSIO. Use it as a playbook for local S3-style experimentation. +This document describes the current Rust server in `rust/myfsio-engine`. It replaces the older Python-oriented runbook. -## 1. System Overview +## 1. What Changed -MyFSIO ships two Flask entrypoints that share the same storage, IAM, and bucket-policy state: +The active runtime is now Rust: -- **API server** – Implements the S3-compatible REST API, policy evaluation, and Signature Version 4 presign service. -- **UI server** – Provides the browser console for buckets, IAM, and policies. It proxies all storage operations through the S3 API via boto3 (SigV4-signed), mirroring the architecture used by MinIO and Garage. +- One process serves both the S3 API and the web UI. +- The server entrypoint is `myfsio-server`. +- The main development workflow is `cargo run -p myfsio-server --`. +- API-only mode is controlled with `UI_ENABLED=false`. -Both servers read `AppConfig`, so editing JSON stores on disk instantly affects both surfaces. +The `python/` directory may still contain older implementation code, templates, and tests, but it is not required to run the current server. -## 2. Quickstart +## 2. Quick Start + +From the repository root: ```bash -python -m venv .venv -. .venv/Scripts/activate # PowerShell: .\.venv\Scripts\Activate.ps1 -pip install -r requirements.txt - -# Run both API and UI -python run.py +cd rust/myfsio-engine +cargo run -p myfsio-server -- ``` -Visit `http://127.0.0.1:5100/ui` to use the console and `http://127.0.0.1:5000/` (with IAM headers) for raw API calls. +Default endpoints: -### Run modes +- UI: `http://127.0.0.1:5100/ui` +- API: `http://127.0.0.1:5000/` +- Health: `http://127.0.0.1:5000/myfsio/health` -You can run services individually if needed: +On first startup, MyFSIO bootstraps an admin user in `data/.myfsio.sys/config/iam.json` and prints the generated access key and secret key to stdout. -```bash -python run.py --mode api # API only (port 5000) -python run.py --mode ui # UI only (port 5100) -``` +### Windows -### Configuration validation - -Validate your configuration before deploying: - -```bash -# Show configuration summary -python run.py --show-config -./myfsio --show-config - -# Validate and check for issues (exits with code 1 if critical issues found) -python run.py --check-config -./myfsio --check-config -``` - -### Linux Installation (Recommended for Production) - -For production deployments on Linux, use the provided installation script: - -```bash -# Download the binary and install script -# Then run the installer with sudo: -sudo ./scripts/install.sh --binary ./myfsio - -# Or with custom paths: -sudo ./scripts/install.sh \ - --binary ./myfsio \ - --install-dir /opt/myfsio \ - --data-dir /mnt/storage/myfsio \ - --log-dir /var/log/myfsio \ - --api-url https://s3.example.com \ - --user myfsio - -# Non-interactive mode (for automation): -sudo ./scripts/install.sh --binary ./myfsio -y -``` - -The installer will: -1. Create a dedicated system user -2. Set up directories with proper permissions -3. Generate a secure `SECRET_KEY` -4. Create an environment file at `/opt/myfsio/myfsio.env` -5. Install and configure a systemd service - -After installation: -```bash -sudo systemctl start myfsio # Start the service -sudo systemctl enable myfsio # Enable on boot -sudo systemctl status myfsio # Check status -sudo journalctl -u myfsio -f # View logs -``` - -To uninstall: -```bash -sudo ./scripts/uninstall.sh # Full removal -sudo ./scripts/uninstall.sh --keep-data # Keep data directory -``` - -### Docker quickstart - -The repo now ships a `Dockerfile` so you can run both services in one container: - -```bash -docker build -t myfsio . -docker run --rm -p 5000:5000 -p 5100:5100 \ - -v "$PWD/data:/app/data" \ - -v "$PWD/logs:/app/logs" \ - -e SECRET_KEY="change-me" \ - --name myfsio myfsio -``` - -PowerShell (Windows) example: +From PowerShell: ```powershell -docker run --rm -p 5000:5000 -p 5100:5100 ` - -v ${PWD}\data:/app/data ` - -v ${PWD}\logs:/app/logs ` - -e SECRET_KEY="change-me" ` - --name myfsio myfsio +cd rust\myfsio-engine +cargo run -p myfsio-server -- ``` -Key mount points: -- `/app/data` → persists buckets directly under `/app/data/` while system metadata (IAM config, bucket policies, versions, multipart uploads, etc.) lives under `/app/data/.myfsio.sys` (for example, `/app/data/.myfsio.sys/config/iam.json`). -- `/app/logs` → captures the rotating app log. -- `/app/tmp-storage` (optional) if you rely on the demo upload staging folders. - -With these volumes attached you can rebuild/restart the container without losing stored objects or credentials. - -### Versioning - -The repo now tracks a human-friendly release string inside `app/version.py` (see the `APP_VERSION` constant). Edit that value whenever you cut a release. The constant flows into Flask as `APP_VERSION` and is exposed via `GET /myfsio/health`, so you can monitor deployments or surface it in UIs. - -## 3. Configuration Reference - -All configuration is done via environment variables. The table below lists every supported variable. - -### Core Settings - -| Variable | Default | Notes | -| --- | --- | --- | -| `STORAGE_ROOT` | `/data` | Filesystem home for all buckets/objects. | -| `MAX_UPLOAD_SIZE` | `1073741824` (1 GiB) | Bytes. Caps incoming uploads in both API + UI. | -| `UI_PAGE_SIZE` | `100` | `MaxKeys` hint shown in listings. | -| `SECRET_KEY` | Auto-generated | Flask session key. Auto-generates and persists if not set. **Set explicitly in production.** | -| `API_BASE_URL` | `http://127.0.0.1:5000` | Internal S3 API URL used by the web UI proxy. Also used for presigned URL generation. Set to your public URL if running behind a reverse proxy. | -| `AWS_REGION` | `us-east-1` | Region embedded in SigV4 credential scope. | -| `AWS_SERVICE` | `s3` | Service string for SigV4. | -| `DISPLAY_TIMEZONE` | `UTC` | Timezone for timestamps in the web UI (e.g., `US/Eastern`, `Asia/Tokyo`). | - -### IAM & Security - -| Variable | Default | Notes | -| --- | --- | --- | -| `IAM_CONFIG` | `data/.myfsio.sys/config/iam.json` | Stores users, secrets, and inline policies. Encrypted at rest when `SECRET_KEY` is set. | -| `BUCKET_POLICY_PATH` | `data/.myfsio.sys/config/bucket_policies.json` | Bucket policy store (auto hot-reload). | -| `AUTH_MAX_ATTEMPTS` | `5` | Failed login attempts before lockout. | -| `AUTH_LOCKOUT_MINUTES` | `15` | Lockout duration after max failed attempts. | -| `SESSION_LIFETIME_DAYS` | `30` | How long UI sessions remain valid. | -| `SECRET_TTL_SECONDS` | `300` | TTL for ephemeral secrets (presigned URLs). | -| `UI_ENFORCE_BUCKET_POLICIES` | `false` | Whether the UI should enforce bucket policies. | -| `ADMIN_ACCESS_KEY` | (none) | Custom access key for the admin user on first run or credential reset. If unset, a random key is generated. | -| `ADMIN_SECRET_KEY` | (none) | Custom secret key for the admin user on first run or credential reset. If unset, a random key is generated. | - -### CORS (Cross-Origin Resource Sharing) - -| Variable | Default | Notes | -| --- | --- | --- | -| `CORS_ORIGINS` | `*` | Comma-separated allowed origins. Use specific domains in production. | -| `CORS_METHODS` | `GET,PUT,POST,DELETE,OPTIONS,HEAD` | Allowed HTTP methods. | -| `CORS_ALLOW_HEADERS` | `*` | Allowed request headers. | -| `CORS_EXPOSE_HEADERS` | `*` | Response headers visible to browsers (e.g., `ETag`). | - -### Rate Limiting - -| Variable | Default | Notes | -| --- | --- | --- | -| `RATE_LIMIT_DEFAULT` | `200 per minute` | Default rate limit for API endpoints. | -| `RATE_LIMIT_LIST_BUCKETS` | `60 per minute` | Rate limit for listing buckets (`GET /`). | -| `RATE_LIMIT_BUCKET_OPS` | `120 per minute` | Rate limit for bucket operations (PUT/DELETE/GET/POST on `/`). | -| `RATE_LIMIT_OBJECT_OPS` | `240 per minute` | Rate limit for object operations (PUT/GET/DELETE/POST on `//`). | -| `RATE_LIMIT_HEAD_OPS` | `100 per minute` | Rate limit for HEAD requests (bucket and object). | -| `RATE_LIMIT_ADMIN` | `60 per minute` | Rate limit for admin API endpoints (`/admin/*`). | -| `RATE_LIMIT_STORAGE_URI` | `memory://` | Storage backend for rate limits. Use `redis://host:port` for distributed setups. | - -### Server Configuration - -| Variable | Default | Notes | -| --- | --- | --- | -| `SERVER_THREADS` | `0` (auto) | Granian blocking threads (1-64). Set to `0` for auto-calculation based on CPU cores (×2). | -| `SERVER_CONNECTION_LIMIT` | `0` (auto) | Maximum concurrent requests per worker (10-1000). Set to `0` for auto-calculation based on available RAM. | -| `SERVER_BACKLOG` | `0` (auto) | TCP listen backlog (128-4096). Set to `0` for auto-calculation (connection_limit × 2). | -| `SERVER_CHANNEL_TIMEOUT` | `120` | Seconds before idle connections are closed (10-300). | - -### Logging - -| Variable | Default | Notes | -| --- | --- | --- | -| `LOG_LEVEL` | `INFO` | Log verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR`. | -| `LOG_TO_FILE` | `true` | Enable file logging. | -| `LOG_DIR` | `/logs` | Directory for log files. | -| `LOG_FILE` | `app.log` | Log filename. | -| `LOG_MAX_BYTES` | `5242880` (5 MB) | Max log file size before rotation. | -| `LOG_BACKUP_COUNT` | `3` | Number of rotated log files to keep. | - -### Encryption - -| Variable | Default | Notes | -| --- | --- | --- | -| `ENCRYPTION_ENABLED` | `false` | Enable server-side encryption support. | -| `ENCRYPTION_MASTER_KEY_PATH` | `data/.myfsio.sys/keys/master.key` | Path to the master encryption key file. | -| `DEFAULT_ENCRYPTION_ALGORITHM` | `AES256` | Default algorithm for new encrypted objects. | -| `KMS_ENABLED` | `false` | Enable KMS key management for encryption. | -| `KMS_KEYS_PATH` | `data/.myfsio.sys/keys/kms_keys.json` | Path to store KMS key metadata. | - - -## Lifecycle Rules - -Lifecycle rules automate object management by scheduling deletions based on object age. - -### Enabling Lifecycle Enforcement - -By default, lifecycle enforcement is disabled. Enable it by setting the environment variable: +### API-only mode ```bash -LIFECYCLE_ENABLED=true python run.py +UI_ENABLED=false cargo run -p myfsio-server -- ``` -Or in your `myfsio.env` file: -``` -LIFECYCLE_ENABLED=true -LIFECYCLE_INTERVAL_SECONDS=3600 # Check interval (default: 1 hour) -``` +There is no separate UI-only mode in the Rust server. -### Configuring Rules - -Once enabled, configure lifecycle rules via: -- **Web UI:** Bucket Details → Lifecycle tab → Add Rule -- **S3 API:** `PUT /?lifecycle` with XML configuration - -### Available Actions - -| Action | Description | -|--------|-------------| -| **Expiration** | Delete current version objects after N days | -| **NoncurrentVersionExpiration** | Delete old versions N days after becoming noncurrent (requires versioning) | -| **AbortIncompleteMultipartUpload** | Clean up incomplete multipart uploads after N days | - -### Example Configuration (XML) - -```xml - - - DeleteOldLogs - Enabled - logs/ - 30 - - -``` - -## Garbage Collection - -The garbage collector (GC) automatically cleans up orphaned data that accumulates over time: stale temporary files from failed uploads, abandoned multipart uploads, stale lock files, orphaned metadata entries, orphaned version files, and empty directories. - -### Enabling GC - -By default, GC is disabled. Enable it by setting: +## 3. Build and Run a Binary ```bash -GC_ENABLED=true python run.py +cd rust/myfsio-engine +cargo build --release -p myfsio-server ``` -Or in your `myfsio.env` file: -``` -GC_ENABLED=true -GC_INTERVAL_HOURS=6 # Run every 6 hours (default) -GC_TEMP_FILE_MAX_AGE_HOURS=24 # Delete temp files older than 24h -GC_MULTIPART_MAX_AGE_DAYS=7 # Delete orphaned multipart uploads older than 7 days -GC_LOCK_FILE_MAX_AGE_HOURS=1 # Delete stale lock files older than 1h -GC_DRY_RUN=false # Set to true to log without deleting -``` - -### What Gets Cleaned - -| Type | Location | Condition | -|------|----------|-----------| -| **Temp files** | `.myfsio.sys/tmp/` | Older than `GC_TEMP_FILE_MAX_AGE_HOURS` | -| **Orphaned multipart uploads** | `.myfsio.sys/multipart/` and `/.multipart/` | Older than `GC_MULTIPART_MAX_AGE_DAYS` | -| **Stale lock files** | `.myfsio.sys/buckets//locks/` | Older than `GC_LOCK_FILE_MAX_AGE_HOURS` | -| **Orphaned metadata** | `.myfsio.sys/buckets//meta/` and `/.meta/` | Object file no longer exists | -| **Orphaned versions** | `.myfsio.sys/buckets//versions/` and `/.versions/` | Main object no longer exists | -| **Empty directories** | Various internal directories | Directory is empty after cleanup | - -### Admin API - -All GC endpoints require admin (`iam:*`) permissions. - -| Method | Route | Description | -|--------|-------|-------------| -| `GET` | `/admin/gc/status` | Get GC status and configuration | -| `POST` | `/admin/gc/run` | Trigger a manual GC run (body: `{"dry_run": true}` for preview) | -| `GET` | `/admin/gc/history` | Get GC execution history (query: `?limit=50&offset=0`) | - -### Dry Run Mode - -Set `GC_DRY_RUN=true` to log what would be deleted without actually removing anything. You can also trigger a one-time dry run via the admin API: +Run it directly: ```bash -curl -X POST "http://localhost:5000/admin/gc/run" \ - -H "X-Access-Key: " -H "X-Secret-Key: " \ - -H "Content-Type: application/json" \ - -d '{"dry_run": true}' +./target/release/myfsio-server ``` -### Performance Tuning +On Windows: -| Variable | Default | Notes | -| --- | --- | --- | -| `STREAM_CHUNK_SIZE` | `65536` (64 KB) | Chunk size for streaming large files. | -| `MULTIPART_MIN_PART_SIZE` | `5242880` (5 MB) | Minimum part size for multipart uploads. | -| `BUCKET_STATS_CACHE_TTL` | `60` | Seconds to cache bucket statistics. | -| `BULK_DELETE_MAX_KEYS` | `500` | Maximum keys per bulk delete request. | -| `BULK_DOWNLOAD_MAX_BYTES` | `1073741824` (1 GiB) | Maximum total size for bulk ZIP downloads. | -| `OBJECT_CACHE_TTL` | `60` | Seconds to cache object metadata. | +```powershell +.\target\release\myfsio-server.exe +``` -#### Gzip Compression +## 4. CLI Commands -API responses for JSON, XML, HTML, CSS, and JavaScript are automatically gzip-compressed when the client sends `Accept-Encoding: gzip`. Compression activates for responses larger than 500 bytes and is handled by a WSGI middleware (`app/compression.py`). Binary object downloads and streaming responses are never compressed. No configuration is needed. - -### Server Settings - -| Variable | Default | Notes | -| --- | --- | --- | -| `APP_HOST` | `0.0.0.0` | Network interface to bind to. | -| `APP_PORT` | `5000` | API server port (UI uses 5100). | -| `FLASK_DEBUG` | `0` | Enable Flask debug mode. **Never enable in production.** | - -### Production Checklist - -Before deploying to production, ensure you: - -1. **Set `SECRET_KEY`** - Use a strong, unique value (e.g., `openssl rand -base64 32`). This also enables IAM config encryption at rest. -2. **Restrict CORS** - Set `CORS_ORIGINS` to your specific domains instead of `*` -3. **Configure `API_BASE_URL`** - Required for correct presigned URLs behind proxies -4. **Enable HTTPS** - Use a reverse proxy (nginx, Cloudflare) with TLS termination -5. **Review rate limits** - Adjust `RATE_LIMIT_DEFAULT` based on your needs -6. **Secure master keys** - Back up `ENCRYPTION_MASTER_KEY_PATH` if using encryption -7. **Use `--prod` flag** - Runs with Granian instead of Flask dev server -8. **Set credential expiry** - Assign `expires_at` to non-admin users for time-limited access - -### Proxy Configuration - -If running behind a reverse proxy (e.g., Nginx, Cloudflare, or a tunnel), ensure the proxy sets the standard forwarding headers: -- `X-Forwarded-Host` -- `X-Forwarded-Proto` - -The application automatically trusts these headers to generate correct presigned URLs (e.g., `https://s3.example.com/...` instead of `http://127.0.0.1:5000/...`). Alternatively, you can explicitly set `API_BASE_URL` to your public endpoint. - -| Variable | Default | Notes | -| --- | --- | --- | -| `NUM_TRUSTED_PROXIES` | `1` | Number of trusted reverse proxies for `X-Forwarded-*` header processing. | -| `ALLOWED_REDIRECT_HOSTS` | `""` | Comma-separated whitelist of safe redirect targets. Empty allows only same-host redirects. | -| `ALLOW_INTERNAL_ENDPOINTS` | `false` | Allow connections to internal/private IPs for webhooks and replication targets. **Keep disabled in production unless needed.** | - -## Integrity Scanner - -The integrity scanner detects and optionally auto-repairs data inconsistencies: corrupted objects (ETag mismatch), orphaned files without metadata, phantom metadata without files, stale version archives, ETag cache drift, and unmigrated legacy `.meta.json` files. - -### Enabling Integrity Scanner - -By default, the integrity scanner is disabled. Enable it by setting: +The Rust CLI supports these operational commands: ```bash -INTEGRITY_ENABLED=true python run.py +# Start serving (default command) +cargo run -p myfsio-server -- + +# Print version +cargo run -p myfsio-server -- version + +# Show resolved configuration +cargo run -p myfsio-server -- --show-config + +# Validate configuration and exit with code 1 on critical issues +cargo run -p myfsio-server -- --check-config + +# Back up the current IAM file and generate fresh admin credentials +cargo run -p myfsio-server -- --reset-cred ``` -Or in your `myfsio.env` file: -``` -INTEGRITY_ENABLED=true -INTEGRITY_INTERVAL_HOURS=24 # Run every 24 hours (default) -INTEGRITY_BATCH_SIZE=1000 # Max objects to scan per cycle -INTEGRITY_AUTO_HEAL=false # Automatically repair detected issues -INTEGRITY_DRY_RUN=false # Set to true to log without healing -``` +If you are running a release build instead of `cargo run`, replace the `cargo run ... --` prefix with the binary path. -### What Gets Checked +## 5. Environment Files -| Check | Detection | Heal Action | -|-------|-----------|-------------| -| **Corrupted objects** | File MD5 does not match stored `__etag__` | Update `__etag__` in index (disk data is authoritative) | -| **Orphaned objects** | File exists on disk without metadata entry | Create index entry with computed MD5/size/mtime | -| **Phantom metadata** | Index entry exists but file is missing from disk | Remove stale entry from `_index.json` | -| **Stale versions** | `.json` manifest without `.bin` data or vice versa | Remove orphaned version file | -| **ETag cache inconsistency** | `etag_index.json` entry differs from metadata `__etag__` | Delete `etag_index.json` (auto-rebuilt on next list) | -| **Legacy metadata drift** | Legacy `.meta.json` differs from index or is unmigrated | Migrate to index and delete legacy file | +At startup, the server tries to load environment files from these locations when they exist: -### Admin API +1. `/opt/myfsio/myfsio.env` +2. `.env` in the current directory +3. `myfsio.env` in the current directory +4. `.env` and `myfsio.env` in a few parent directories -All integrity endpoints require admin (`iam:*`) permissions. +That makes local development and systemd installs behave consistently. -| Method | Route | Description | -|--------|-------|-------------| -| `GET` | `/admin/integrity/status` | Get scanner status and configuration | -| `POST` | `/admin/integrity/run` | Trigger a manual scan (body: `{"dry_run": true, "auto_heal": true}`) | -| `GET` | `/admin/integrity/history` | Get scan history (query: `?limit=50&offset=0`) | +## 6. Verified Configuration Reference -### Dry Run Mode +These values are taken from `rust/myfsio-engine/crates/myfsio-server/src/config.rs`. -Set `INTEGRITY_DRY_RUN=true` to log detected issues without making any changes. You can also trigger a one-time dry run via the admin API: - -```bash -curl -X POST "http://localhost:5000/admin/integrity/run" \ - -H "X-Access-Key: " -H "X-Secret-Key: " \ - -H "Content-Type: application/json" \ - -d '{"dry_run": true, "auto_heal": true}' -``` - -### Configuration Reference +### Network and runtime | Variable | Default | Description | -|----------|---------|-------------| -| `INTEGRITY_ENABLED` | `false` | Enable background integrity scanning | -| `INTEGRITY_INTERVAL_HOURS` | `24` | Hours between scan cycles | -| `INTEGRITY_BATCH_SIZE` | `1000` | Max objects to scan per cycle | -| `INTEGRITY_AUTO_HEAL` | `false` | Automatically repair detected issues | -| `INTEGRITY_DRY_RUN` | `false` | Log issues without healing | - -## 4. Upgrading and Updates - -### Version Checking - -The application version is tracked in `app/version.py` and exposed via: -- **Health endpoint:** `GET /myfsio/health` returns JSON with `version` field -- **Metrics dashboard:** Navigate to `/ui/metrics` to see the running version in the System Status card - -To check your current version: - -```bash -# API health endpoint -curl http://localhost:5000/myfsio/health - -# Or inspect version.py directly -cat app/version.py | grep APP_VERSION -``` - -### Pre-Update Backup Procedures - -**Always backup before upgrading to prevent data loss:** - -```bash -# 1. Stop the application -# Ctrl+C if running in terminal, or: -docker stop myfsio # if using Docker - -# 2. Backup configuration files (CRITICAL) -mkdir -p backups/$(date +%Y%m%d_%H%M%S) -cp -r data/.myfsio.sys/config backups/$(date +%Y%m%d_%H%M%S)/ - -# 3. Backup all data (optional but recommended) -tar -czf backups/data_$(date +%Y%m%d_%H%M%S).tar.gz data/ - -# 4. Backup logs for audit trail -cp -r logs backups/$(date +%Y%m%d_%H%M%S)/ -``` - -**Windows PowerShell:** - -```powershell -# Create timestamped backup -$timestamp = Get-Date -Format "yyyyMMdd_HHmmss" -New-Item -ItemType Directory -Path "backups\$timestamp" -Force - -# Backup configs -Copy-Item -Recurse "data\.myfsio.sys\config" "backups\$timestamp\" - -# Backup entire data directory -Compress-Archive -Path "data\" -DestinationPath "backups\data_$timestamp.zip" -``` - -**Critical files to backup:** -- `data/.myfsio.sys/config/iam.json` – User accounts and access keys -- `data/.myfsio.sys/config/bucket_policies.json` – Bucket access policies -- `data/.myfsio.sys/config/kms_keys.json` – Encryption keys (if using KMS) -- `data/.myfsio.sys/config/secret_store.json` – Application secrets - -### Update Procedures - -#### Source Installation Updates - -```bash -# 1. Backup (see above) -# 2. Pull latest code -git fetch origin -git checkout main # or your target branch/tag -git pull - -# 3. Check for dependency changes -pip install -r requirements.txt - -# 4. Review CHANGELOG/release notes for breaking changes -cat CHANGELOG.md # if available - -# 5. Run migration scripts (if any) -# python scripts/migrate_vX_to_vY.py # example - -# 6. Restart application -python run.py -``` - -#### Docker Updates - -```bash -# 1. Backup (see above) -# 2. Pull/rebuild image -docker pull yourregistry/myfsio:latest -# OR rebuild from source: -docker build -t myfsio:latest . - -# 3. Stop and remove old container -docker stop myfsio -docker rm myfsio - -# 4. Start new container with same volumes -docker run -d \ - --name myfsio \ - -p 5000:5000 -p 5100:5100 \ - -v "$(pwd)/data:/app/data" \ - -v "$(pwd)/logs:/app/logs" \ - -e SECRET_KEY="your-secret" \ - myfsio:latest - -# 5. Verify health -curl http://localhost:5000/myfsio/health -``` - -### Version Compatibility Checks - -Before upgrading across major versions, verify compatibility: - -| From Version | To Version | Breaking Changes | Migration Required | -|--------------|------------|------------------|-------------------| -| 0.1.x | 0.2.x | None expected | No | -| 0.1.6 | 0.1.7 | None | No | -| < 0.1.0 | >= 0.1.0 | New IAM config format | Yes - run migration script | - -**Automatic compatibility detection:** - -The application will log warnings on startup if config files need migration: - -``` -WARNING: IAM config format is outdated (v1). Please run: python scripts/migrate_iam.py -``` - -**Manual compatibility check:** - -```bash -# Compare version schemas -python -c "from app.version import APP_VERSION; print(f'Running: {APP_VERSION}')" -python scripts/check_compatibility.py data/.myfsio.sys/config/ -``` - -### Migration Steps for Breaking Changes - -When release notes indicate breaking changes, follow these steps: - -#### Config Format Migrations - -```bash -# 1. Backup first (critical!) -cp data/.myfsio.sys/config/iam.json data/.myfsio.sys/config/iam.json.backup - -# 2. Run provided migration script -python scripts/migrate_iam_v1_to_v2.py - -# 3. Validate migration -python scripts/validate_config.py - -# 4. Test with read-only mode first (if available) -# python run.py --read-only - -# 5. Restart normally -python run.py -``` - -#### Database/Storage Schema Changes - -If object metadata format changes: - -```bash -# 1. Run storage migration script -python scripts/migrate_storage.py --dry-run # preview changes - -# 2. Apply migration -python scripts/migrate_storage.py --apply - -# 3. Verify integrity -python scripts/verify_storage.py -``` - -#### IAM Policy Updates - -If IAM action names change (e.g., `s3:Get` → `s3:GetObject`): - -```bash -# Migration script will update all policies -python scripts/migrate_policies.py \ - --input data/.myfsio.sys/config/iam.json \ - --backup data/.myfsio.sys/config/iam.json.v1 - -# Review changes before committing -python scripts/diff_policies.py \ - data/.myfsio.sys/config/iam.json.v1 \ - data/.myfsio.sys/config/iam.json -``` - -### Rollback Procedures - -If an update causes issues, rollback to the previous version: - -#### Quick Rollback (Source) - -```bash -# 1. Stop application -# Ctrl+C or kill process - -# 2. Revert code -git checkout -# OR -git reset --hard HEAD~1 - -# 3. Restore configs from backup -cp backups/20241213_103000/config/* data/.myfsio.sys/config/ - -# 4. Downgrade dependencies if needed -pip install -r requirements.txt - -# 5. Restart -python run.py -``` - -#### Docker Rollback - -```bash -# 1. Stop current container -docker stop myfsio -docker rm myfsio - -# 2. Start previous version -docker run -d \ - --name myfsio \ - -p 5000:5000 -p 5100:5100 \ - -v "$(pwd)/data:/app/data" \ - -v "$(pwd)/logs:/app/logs" \ - -e SECRET_KEY="your-secret" \ - myfsio:0.1.3 # specify previous version tag - -# 3. Verify -curl http://localhost:5000/myfsio/health -``` - -#### Emergency Config Restore - -If only config is corrupted but code is fine: - -```bash -# Stop app -# Restore from latest backup -cp backups/20241213_103000/config/iam.json data/.myfsio.sys/config/ -cp backups/20241213_103000/config/bucket_policies.json data/.myfsio.sys/config/ - -# Restart app -python run.py -``` - -### Blue-Green Deployment (Zero Downtime) - -For production environments requiring zero downtime: - -```bash -# 1. Run new version on different port (e.g., 5001/5101) -APP_PORT=5001 UI_PORT=5101 python run.py & - -# 2. Health check new instance -curl http://localhost:5001/myfsio/health - -# 3. Update load balancer to route to new ports - -# 4. Monitor for issues - -# 5. Gracefully stop old instance -kill -SIGTERM -``` - -### Post-Update Verification - -After any update, verify functionality: - -```bash -# 1. Health check -curl http://localhost:5000/myfsio/health - -# 2. Login to UI -open http://localhost:5100/ui - -# 3. Test IAM authentication -curl -H "X-Amz-Security-Token: :" \ - http://localhost:5000/ - -# 4. Test presigned URL generation -# Via UI or API - -# 5. Check logs for errors -tail -n 100 logs/myfsio.log -``` - -### Automated Update Scripts - -Create a custom update script for your environment: - -```bash -#!/bin/bash -# update.sh - Automated update with rollback capability - -set -e # Exit on error - -VERSION_NEW="$1" -BACKUP_DIR="backups/$(date +%Y%m%d_%H%M%S)" - -echo "Creating backup..." -mkdir -p "$BACKUP_DIR" -cp -r data/.myfsio.sys/config "$BACKUP_DIR/" - -echo "Updating to version $VERSION_NEW..." -git fetch origin -git checkout "v$VERSION_NEW" -pip install -r requirements.txt - -echo "Starting application..." -python run.py & -APP_PID=$! - -# Wait and health check -sleep 5 -if curl -f http://localhost:5000/myfsio/health; then - echo "Update successful!" -else - echo "Health check failed, rolling back..." - kill $APP_PID - git checkout - - cp -r "$BACKUP_DIR/config/*" data/.myfsio.sys/config/ - python run.py & - exit 1 -fi -``` - -## 4. Authentication & IAM - -MyFSIO implements a comprehensive Identity and Access Management (IAM) system that controls who can access your buckets and what operations they can perform. The system supports both simple action-based permissions and AWS-compatible policy syntax. - -### Getting Started - -1. On first boot, `data/.myfsio.sys/config/iam.json` is created with a randomly generated admin user. The access key and secret key are printed to the console during first startup. You can set `ADMIN_ACCESS_KEY` and `ADMIN_SECRET_KEY` environment variables to use custom credentials instead of random ones. If `SECRET_KEY` is configured, the IAM config file is encrypted at rest using AES (Fernet). To reset admin credentials later, run `python run.py --reset-cred`. -2. Sign into the UI using the generated credentials, then open **IAM**: - - **Create user**: supply a display name, optional JSON inline policy array, and optional credential expiry date. - - **Set expiry**: assign an expiration date to any user's credentials. Expired credentials are rejected at authentication time. The UI shows expiry badges and preset durations (1h, 24h, 7d, 30d, 90d). - - **Rotate secret**: generates a new secret key; the UI surfaces it once. - - **Policy editor**: select a user, paste an array of objects (`{"bucket": "*", "actions": ["list", "read"]}`), and submit. An optional `"prefix"` field restricts object-level actions to a key prefix (e.g., `"uploads/"`). Alias support includes AWS-style verbs (e.g., `s3:GetObject`). -3. Wildcard action `iam:*` is supported for admin user definitions. - -> **Breaking Change (v0.2.0+):** Previous versions used fixed default credentials (`localadmin/localadmin`). If upgrading from an older version, your existing credentials remain unchanged, but new installations will generate random credentials. - -### Authentication - -The API expects every request to include authentication headers. The UI persists them in the Flask session after login. - -| Header | Description | +| --- | --- | --- | +| `HOST` | `127.0.0.1` | Bind address for both listeners | +| `PORT` | `5000` | S3 API port | +| `UI_PORT` | `5100` | Web UI port | +| `UI_ENABLED` | `true` | Disable to run API-only | +| `API_BASE_URL` | unset | Public-facing API base used by the UI and presigned URL generation | +| `TEMPLATES_DIR` | built-in templates dir | Optional override for UI templates | +| `STATIC_DIR` | built-in static dir | Optional override for static assets | + +### Storage and auth + +| Variable | Default | Description | +| --- | --- | --- | +| `STORAGE_ROOT` | `./data` | Root for buckets and internal state | +| `IAM_CONFIG` | `/.myfsio.sys/config/iam.json` | IAM config path | +| `AWS_REGION` | `us-east-1` | SigV4 region | +| `SIGV4_TIMESTAMP_TOLERANCE_SECONDS` | `900` | Allowed request time skew | +| `PRESIGNED_URL_MIN_EXPIRY_SECONDS` | `1` | Minimum presigned URL lifetime | +| `PRESIGNED_URL_MAX_EXPIRY_SECONDS` | `604800` | Maximum presigned URL lifetime | +| `SECRET_KEY` | unset, then fallback to `.myfsio.sys/config/.secret` if present | Session signing and IAM config encryption key | +| `ADMIN_ACCESS_KEY` | unset | Optional deterministic first-run/reset access key | +| `ADMIN_SECRET_KEY` | unset | Optional deterministic first-run/reset secret key | + +### Feature toggles + +| Variable | Default | Description | +| --- | --- | --- | +| `ENCRYPTION_ENABLED` | `false` | Enable object encryption support | +| `KMS_ENABLED` | `false` | Enable built-in KMS support | +| `GC_ENABLED` | `false` | Start the garbage collector worker | +| `INTEGRITY_ENABLED` | `false` | Start the integrity worker | +| `LIFECYCLE_ENABLED` | `false` | Start the lifecycle worker | +| `METRICS_HISTORY_ENABLED` | `false` | Persist system metrics snapshots | +| `OPERATION_METRICS_ENABLED` | `false` | Persist API operation metrics | +| `WEBSITE_HOSTING_ENABLED` | `false` | Enable website domain and hosting features | +| `SITE_SYNC_ENABLED` | `false` | Start the site sync worker | + +### Metrics tuning + +| Variable | Default | Description | +| --- | --- | --- | +| `OPERATION_METRICS_INTERVAL_MINUTES` | `5` | Snapshot interval for operation metrics | +| `OPERATION_METRICS_RETENTION_HOURS` | `24` | Retention window for operation metrics | +| `METRICS_HISTORY_INTERVAL_MINUTES` | `5` | Snapshot interval for system metrics | +| `METRICS_HISTORY_RETENTION_HOURS` | `24` | Retention window for system metrics | + +### Replication and site sync + +| Variable | Default | Description | +| --- | --- | --- | +| `REPLICATION_CONNECT_TIMEOUT_SECONDS` | `5` | Replication connect timeout | +| `REPLICATION_READ_TIMEOUT_SECONDS` | `30` | Replication read timeout | +| `REPLICATION_MAX_RETRIES` | `2` | Replication retry count | +| `REPLICATION_STREAMING_THRESHOLD_BYTES` | `10485760` | Switch to streaming for large copies | +| `REPLICATION_MAX_FAILURES_PER_BUCKET` | `50` | Failure budget before a bucket is skipped | +| `SITE_SYNC_INTERVAL_SECONDS` | `60` | Poll interval for the site sync worker | +| `SITE_SYNC_BATCH_SIZE` | `100` | Max objects processed per site sync batch | +| `SITE_SYNC_CONNECT_TIMEOUT_SECONDS` | `10` | Site sync connect timeout | +| `SITE_SYNC_READ_TIMEOUT_SECONDS` | `120` | Site sync read timeout | +| `SITE_SYNC_MAX_RETRIES` | `2` | Site sync retry count | +| `SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS` | `1.0` | Allowed skew between peers | + +### Site identity values used by the UI + +These are read directly by UI pages: + +| Variable | Description | | --- | --- | -| `X-Access-Key` | The user's access key identifier | -| `X-Secret-Key` | The user's secret key for signing | +| `SITE_ID` | Local site identifier shown in the UI | +| `SITE_ENDPOINT` | Public endpoint for this site | +| `SITE_REGION` | Display region for the local site | -**Security Features:** -- **Lockout Protection**: After `AUTH_MAX_ATTEMPTS` (default: 5) failed login attempts, the account is locked for `AUTH_LOCKOUT_MINUTES` (default: 15 minutes). -- **Credential Expiry**: Each user can have an optional `expires_at` timestamp (ISO 8601). Once expired, all API requests using those credentials are rejected. Set or clear expiry via the UI or API. -- **IAM Config Encryption**: When `SECRET_KEY` is set, the IAM config file (`iam.json`) is encrypted at rest using Fernet (AES-256-CBC with HMAC). Existing plaintext configs are automatically encrypted on next load. -- **Session Management**: UI sessions remain valid for `SESSION_LIFETIME_DAYS` (default: 30 days). -- **Hot Reload**: IAM configuration changes take effect immediately without restart. -- **Credential Reset**: Run `python run.py --reset-cred` to reset admin credentials. Supports `ADMIN_ACCESS_KEY` and `ADMIN_SECRET_KEY` env vars for deterministic keys. +## 7. Data Layout -### Permission Model +With the default `STORAGE_ROOT=./data`, the Rust server writes: -MyFSIO uses a two-layer permission model: - -1. **IAM User Policies** – Define what a user can do across the system (stored in `iam.json`) -2. **Bucket Policies** – Define who can access a specific bucket (stored in `bucket_policies.json`) - -Both layers are evaluated for each request. A user must have permission in their IAM policy AND the bucket policy must allow the action (or have no explicit deny). - -### Available IAM Actions - -#### S3 Actions (Bucket/Object Operations) - -| Action | Description | AWS Aliases | -| --- | --- | --- | -| `list` | List buckets and objects | `s3:ListBucket`, `s3:ListAllMyBuckets`, `s3:ListBucketVersions`, `s3:ListMultipartUploads`, `s3:ListParts` | -| `read` | Download objects, get metadata | `s3:GetObject`, `s3:GetObjectVersion`, `s3:GetObjectTagging`, `s3:GetObjectVersionTagging`, `s3:GetObjectAcl`, `s3:GetBucketVersioning`, `s3:HeadObject`, `s3:HeadBucket` | -| `write` | Upload objects, manage object tags | `s3:PutObject`, `s3:PutObjectTagging`, `s3:CreateMultipartUpload`, `s3:UploadPart`, `s3:CompleteMultipartUpload`, `s3:AbortMultipartUpload`, `s3:CopyObject` | -| `delete` | Remove objects and versions | `s3:DeleteObject`, `s3:DeleteObjectVersion`, `s3:DeleteObjectTagging` | -| `create_bucket` | Create new buckets | `s3:CreateBucket` | -| `delete_bucket` | Delete buckets | `s3:DeleteBucket` | -| `share` | Manage Access Control Lists (ACLs) | `s3:PutObjectAcl`, `s3:PutBucketAcl`, `s3:GetBucketAcl` | -| `policy` | Manage bucket policies | `s3:PutBucketPolicy`, `s3:GetBucketPolicy`, `s3:DeleteBucketPolicy` | -| `versioning` | Manage bucket versioning configuration | `s3:GetBucketVersioning`, `s3:PutBucketVersioning` | -| `tagging` | Manage bucket-level tags | `s3:GetBucketTagging`, `s3:PutBucketTagging`, `s3:DeleteBucketTagging` | -| `encryption` | Manage bucket encryption configuration | `s3:GetEncryptionConfiguration`, `s3:PutEncryptionConfiguration`, `s3:DeleteEncryptionConfiguration` | -| `lifecycle` | Manage lifecycle rules | `s3:GetLifecycleConfiguration`, `s3:PutLifecycleConfiguration`, `s3:DeleteLifecycleConfiguration`, `s3:GetBucketLifecycle`, `s3:PutBucketLifecycle` | -| `cors` | Manage CORS configuration | `s3:GetBucketCors`, `s3:PutBucketCors`, `s3:DeleteBucketCors` | -| `replication` | Configure and manage replication | `s3:GetReplicationConfiguration`, `s3:PutReplicationConfiguration`, `s3:DeleteReplicationConfiguration`, `s3:ReplicateObject`, `s3:ReplicateTags`, `s3:ReplicateDelete` | -| `quota` | Manage bucket storage quotas | `s3:GetBucketQuota`, `s3:PutBucketQuota`, `s3:DeleteBucketQuota` | -| `object_lock` | Manage object lock, retention, and legal holds | `s3:GetObjectLockConfiguration`, `s3:PutObjectLockConfiguration`, `s3:PutObjectRetention`, `s3:GetObjectRetention`, `s3:PutObjectLegalHold`, `s3:GetObjectLegalHold` | -| `notification` | Manage bucket event notifications | `s3:GetBucketNotificationConfiguration`, `s3:PutBucketNotificationConfiguration`, `s3:DeleteBucketNotificationConfiguration` | -| `logging` | Manage bucket access logging | `s3:GetBucketLogging`, `s3:PutBucketLogging`, `s3:DeleteBucketLogging` | -| `website` | Manage static website hosting configuration | `s3:GetBucketWebsite`, `s3:PutBucketWebsite`, `s3:DeleteBucketWebsite` | - -#### IAM Actions (User Management) - -| Action | Description | AWS Aliases | -| --- | --- | --- | -| `iam:list_users` | View all IAM users and their policies | `iam:ListUsers` | -| `iam:create_user` | Create new IAM users | `iam:CreateUser` | -| `iam:delete_user` | Delete IAM users | `iam:DeleteUser` | -| `iam:rotate_key` | Rotate user secret keys | `iam:RotateAccessKey` | -| `iam:update_policy` | Modify user policies | `iam:PutUserPolicy` | -| `iam:create_key` | Create additional access keys for a user | `iam:CreateAccessKey` | -| `iam:delete_key` | Delete an access key from a user | `iam:DeleteAccessKey` | -| `iam:get_user` | View user details and access keys | `iam:GetUser` | -| `iam:get_policy` | View user policy configuration | `iam:GetPolicy` | -| `iam:disable_user` | Temporarily disable/enable a user account | `iam:DisableUser` | -| `iam:*` | **Admin wildcard** – grants all IAM actions | — | - -#### Wildcards - -| Wildcard | Scope | Description | -| --- | --- | --- | -| `*` (in actions) | All S3 actions | Grants all 19 S3 actions including `list`, `read`, `write`, `delete`, `create_bucket`, `delete_bucket`, `share`, `policy`, `versioning`, `tagging`, `encryption`, `lifecycle`, `cors`, `replication`, `quota`, `object_lock`, `notification`, `logging`, `website` | -| `iam:*` | All IAM actions | Grants all `iam:*` actions for user management | -| `*` (in bucket) | All buckets | Policy applies to every bucket | - -### IAM Policy Structure - -User policies are stored as a JSON array of policy objects. Each object specifies a bucket, the allowed actions, and an optional prefix for object-level scoping: - -```json -[ - { - "bucket": "", - "actions": ["", "", ...], - "prefix": "" - } -] +```text +data/ + / + .myfsio.sys/ + config/ + iam.json + bucket_policies.json + connections.json + gc_history.json + integrity_history.json + metrics_history.json + operation_metrics.json + buckets// + meta/ + versions/ + multipart/ + keys/ ``` -**Fields:** -- `bucket`: The bucket name (case-insensitive) or `*` for all buckets -- `actions`: Array of action strings (simple names or AWS aliases) -- `prefix`: *(optional)* Restrict object-level actions to keys starting with this prefix. Defaults to `*` (all objects). Example: `"uploads/"` restricts to keys under `uploads/` +Important files: -### Example User Policies +- `data/.myfsio.sys/config/iam.json`: IAM users, access keys, and inline policies +- `data/.myfsio.sys/config/bucket_policies.json`: bucket policies +- `data/.myfsio.sys/config/connections.json`: replication connection settings +- `data/.myfsio.sys/config/.secret`: persisted secret key when one has been generated for the install -**Full Administrator (complete system access):** -```json -[{"bucket": "*", "actions": ["list", "read", "write", "delete", "share", "policy", "create_bucket", "delete_bucket", "versioning", "tagging", "encryption", "lifecycle", "cors", "replication", "quota", "object_lock", "notification", "logging", "website", "iam:*"]}] -``` +## 8. Background Services -**Read-Only User (browse and download only):** -```json -[{"bucket": "*", "actions": ["list", "read"]}] -``` +The Rust server can start several workers from the same process. -**Single Bucket Full Access (no access to other buckets):** -```json -[{"bucket": "user-bucket", "actions": ["list", "read", "write", "delete"]}] -``` +### Lifecycle -**Operator (data operations + bucket management, no config):** -```json -[{"bucket": "*", "actions": ["list", "read", "write", "delete", "create_bucket", "delete_bucket"]}] -``` - -**Multiple Bucket Access (different permissions per bucket):** -```json -[ - {"bucket": "public-data", "actions": ["list", "read"]}, - {"bucket": "my-uploads", "actions": ["list", "read", "write", "delete"]}, - {"bucket": "team-shared", "actions": ["list", "read", "write"]} -] -``` - -**Prefix-Scoped Access (restrict to a folder inside a shared bucket):** -```json -[{"bucket": "shared-data", "actions": ["list", "read", "write", "delete"], "prefix": "team-a/"}] -``` - -**IAM Manager (manage users but no data access):** -```json -[{"bucket": "*", "actions": ["iam:list_users", "iam:create_user", "iam:delete_user", "iam:rotate_key", "iam:update_policy", "iam:create_key", "iam:delete_key", "iam:get_user", "iam:get_policy", "iam:disable_user"]}] -``` - -**Replication Operator (manage replication only):** -```json -[{"bucket": "*", "actions": ["list", "read", "replication"]}] -``` - -**Lifecycle Manager (configure object expiration):** -```json -[{"bucket": "*", "actions": ["list", "lifecycle"]}] -``` - -**CORS Administrator (configure cross-origin access):** -```json -[{"bucket": "*", "actions": ["cors"]}] -``` - -**Bucket Administrator (full bucket config, no IAM access):** -```json -[{"bucket": "my-bucket", "actions": ["list", "read", "write", "delete", "create_bucket", "delete_bucket", "share", "policy", "versioning", "tagging", "encryption", "lifecycle", "cors", "replication", "quota", "object_lock", "notification", "logging", "website"]}] -``` - -**Upload-Only User (write but cannot create/delete buckets):** -```json -[{"bucket": "drop-box", "actions": ["write"]}] -``` - -**Backup Operator (read, list, and replicate):** -```json -[{"bucket": "*", "actions": ["list", "read", "replication"]}] -``` - -### Using AWS-Style Action Names - -You can use AWS S3 action names instead of simple names. They are automatically normalized: - -```json -[ - { - "bucket": "my-bucket", - "actions": [ - "s3:ListBucket", - "s3:GetObject", - "s3:PutObject", - "s3:DeleteObject" - ] - } -] -``` - -This is equivalent to: -```json -[{"bucket": "my-bucket", "actions": ["list", "read", "write", "delete"]}] -``` - -### Managing Users via API +Enable with: ```bash -# List all users (requires iam:list_users) -curl http://localhost:5000/iam/users \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Create a new user (requires iam:create_user) -curl -X POST http://localhost:5000/iam/users \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "display_name": "New User", - "policies": [{"bucket": "*", "actions": ["list", "read"]}], - "expires_at": "2026-12-31T23:59:59Z" - }' - -# Rotate user secret (requires iam:rotate_key) -curl -X POST http://localhost:5000/iam/users//rotate \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Update user policies (requires iam:update_policy) -curl -X PUT http://localhost:5000/iam/users//policies \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '[{"bucket": "*", "actions": ["list", "read", "write"]}]' - -# Update credential expiry (requires iam:update_policy) -curl -X POST http://localhost:5000/iam/users//expiry \ - -H "Content-Type: application/x-www-form-urlencoded" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d 'expires_at=2026-12-31T23:59:59Z' - -# Remove credential expiry (never expires) -curl -X POST http://localhost:5000/iam/users//expiry \ - -H "Content-Type: application/x-www-form-urlencoded" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d 'expires_at=' - -# Delete a user (requires iam:delete_user) -curl -X DELETE http://localhost:5000/iam/users/ \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Get user details (requires iam:get_user) — via Admin API -curl http://localhost:5000/admin/iam/users/ \ - -H "Authorization: AWS4-HMAC-SHA256 ..." - -# Get user policies (requires iam:get_policy) — via Admin API -curl http://localhost:5000/admin/iam/users//policies \ - -H "Authorization: AWS4-HMAC-SHA256 ..." - -# Create additional access key for a user (requires iam:create_key) -curl -X POST http://localhost:5000/admin/iam/users//keys \ - -H "Authorization: AWS4-HMAC-SHA256 ..." - -# Delete an access key (requires iam:delete_key) -curl -X DELETE http://localhost:5000/admin/iam/users//keys/ \ - -H "Authorization: AWS4-HMAC-SHA256 ..." - -# Disable a user account (requires iam:disable_user) -curl -X POST http://localhost:5000/admin/iam/users//disable \ - -H "Authorization: AWS4-HMAC-SHA256 ..." - -# Re-enable a user account (requires iam:disable_user) -curl -X POST http://localhost:5000/admin/iam/users//enable \ - -H "Authorization: AWS4-HMAC-SHA256 ..." +LIFECYCLE_ENABLED=true cargo run -p myfsio-server -- ``` -### Permission Precedence +Current Rust behavior: -When a request is made, permissions are evaluated in this order: +- Runs as a Tokio background task, not a cron job +- Default interval is 3600 seconds +- Evaluates bucket lifecycle configuration and applies expiration and multipart abort rules -1. **Authentication** – Verify the access key and secret key are valid -2. **Lockout Check** – Ensure the account is not locked due to failed attempts -3. **Expiry Check** – Reject requests if the user's credentials have expired (`expires_at`) -4. **IAM Policy Check** – Verify the user has the required action for the target bucket -5. **Bucket Policy Check** – If a bucket policy exists, verify it allows the action +At the moment, the interval is hardcoded through `LifecycleConfig::default()` rather than exposed as an environment variable. -A request is allowed only if: -- The IAM policy grants the action, AND -- The bucket policy allows the action (or no bucket policy exists) +### Garbage collection -### Common Permission Scenarios - -| Scenario | Required Actions | -| --- | --- | -| Browse bucket contents | `list` | -| Download a file | `read` | -| Upload a file | `write` | -| Delete a file | `delete` | -| Generate presigned URL (GET) | `read` | -| Generate presigned URL (PUT) | `write` | -| Generate presigned URL (DELETE) | `delete` | -| Enable versioning | `write` (includes `s3:PutBucketVersioning`) | -| View bucket policy | `policy` | -| Modify bucket policy | `policy` | -| Configure lifecycle rules | `lifecycle` | -| View lifecycle rules | `lifecycle` | -| Configure CORS | `cors` | -| View CORS rules | `cors` | -| Configure replication | `replication` (admin-only for creation) | -| Pause/resume replication | `replication` | -| Manage other users | `iam:*` or specific `iam:` actions | -| Set bucket quotas | `iam:*` or `iam:list_users` (admin feature) | - -### Security Best Practices - -1. **Principle of Least Privilege** – Grant only the permissions users need -2. **Avoid Wildcards** – Use specific bucket names instead of `*` when possible -3. **Rotate Secrets Regularly** – Use the rotate key feature periodically -4. **Separate Admin Accounts** – Don't use admin accounts for daily operations -5. **Monitor Failed Logins** – Check logs for repeated authentication failures -6. **Use Bucket Policies for Fine-Grained Control** – Combine with IAM for defense in depth - -## 5. Bucket Policies & Presets - -- **Storage**: Policies are persisted in `data/.myfsio.sys/config/bucket_policies.json` under `{"policies": {"bucket": {...}}}`. -- **Hot reload**: Both API and UI call `maybe_reload()` before evaluating policies. Editing the JSON on disk is immediately reflected—no restarts required. -- **UI editor**: Each bucket detail page includes: - - A preset selector: **Private** detaches the policy (delete mode), **Public** injects an allow policy granting anonymous `s3:ListBucket` + `s3:GetObject`, and **Custom** restores your draft. - - A read-only preview of the attached policy. - - Autosave behavior for custom drafts while you type. - -### Editing via CLI +Enable with: ```bash -curl -X PUT "http://127.0.0.1:5000/test?policy" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": "*", - "Action": ["s3:ListBucket"], - "Resource": ["arn:aws:s3:::test"] - } - ] - }' +GC_ENABLED=true cargo run -p myfsio-server -- ``` -The UI will reflect this change as soon as the request completes thanks to the hot reload. +Current Rust defaults from `GcConfig::default()`: -### UI Object Browser +- Run every 6 hours +- Temp files older than 24 hours are eligible for cleanup +- Multipart uploads older than 7 days are eligible for cleanup +- Lock files older than 1 hour are eligible for cleanup -The bucket detail page includes a powerful object browser with the following features: +Those GC timings are currently hardcoded defaults, not environment-driven configuration. -#### Folder Navigation +### Integrity scanning -Objects with forward slashes (`/`) in their keys are displayed as a folder hierarchy. Click a folder row to navigate into it. A breadcrumb navigation bar shows your current path and allows quick navigation back to parent folders or the root. - -#### Pagination & Infinite Scroll - -- Objects load in configurable batches (50, 100, 150, 200, or 250 per page) -- Scroll to the bottom to automatically load more objects (infinite scroll) -- A **Load more** button is available as a fallback for touch devices or when infinite scroll doesn't trigger -- The footer shows the current load status (e.g., "Showing 100 of 500 objects") - -#### Bulk Operations - -- Select multiple objects using checkboxes -- **Bulk Delete**: Delete multiple objects at once -- **Bulk Download**: Download selected objects as a single ZIP archive (up to `BULK_DOWNLOAD_MAX_BYTES`, default 1 GiB) - -#### Search & Filter - -Use the search box to filter objects by name in real-time. The filter applies to the currently loaded objects. - -#### Error Handling - -If object loading fails (e.g., network error), a friendly error message is displayed with a **Retry** button to attempt loading again. - -#### Object Preview - -Click any object row to view its details in the preview sidebar: -- File size and last modified date -- ETag (content hash) -- Custom metadata (if present) -- Download and presign (share link) buttons -- Version history (when versioning is enabled) - -#### Drag & Drop Upload - -Drag files directly onto the objects table to upload them to the current bucket and folder path. - -## 6. Presigned URLs - -- Trigger from the UI using the **Presign** button after selecting an object. -- Supported methods: `GET`, `PUT`, `DELETE`; expiration must be `1..604800` seconds. -- The service signs requests using the caller's IAM credentials and enforces bucket policies both when issuing and when the presigned URL is used. -- Legacy share links have been removed; presigned URLs now handle both private and public workflows. - -### Multipart Upload Example - -```python -import boto3 - -s3 = boto3.client('s3', endpoint_url='http://localhost:5000') - -# Initiate -response = s3.create_multipart_upload(Bucket='mybucket', Key='large.bin') -upload_id = response['UploadId'] - -# Upload parts -parts = [] -chunks = [b'chunk1', b'chunk2'] # Example data chunks -for part_number, chunk in enumerate(chunks, start=1): - response = s3.upload_part( - Bucket='mybucket', - Key='large.bin', - PartNumber=part_number, - UploadId=upload_id, - Body=chunk - ) - parts.append({'PartNumber': part_number, 'ETag': response['ETag']}) - -# Complete -s3.complete_multipart_upload( - Bucket='mybucket', - Key='large.bin', - UploadId=upload_id, - MultipartUpload={'Parts': parts} -) -``` - -## 7. Encryption - -MyFSIO supports **server-side encryption at rest** to protect your data. When enabled, objects are encrypted using AES-256-GCM before being written to disk. - -### Encryption Types - -| Type | Description | -|------|-------------| -| **AES-256 (SSE-S3)** | Server-managed encryption using a local master key | -| **KMS (SSE-KMS)** | Encryption using customer-managed keys via the built-in KMS | -| **SSE-C** | Server-side encryption with customer-provided keys (per-request) | - -### Enabling Encryption - -#### 1. Set Environment Variables - -```powershell -# PowerShell -$env:ENCRYPTION_ENABLED = "true" -$env:KMS_ENABLED = "true" # Optional, for KMS key management -python run.py -``` +Enable with: ```bash -# Bash -export ENCRYPTION_ENABLED=true -export KMS_ENABLED=true -python run.py +INTEGRITY_ENABLED=true cargo run -p myfsio-server -- ``` -#### 2. Configure Bucket Default Encryption (UI) +Current Rust defaults from `IntegrityConfig::default()`: -1. Navigate to your bucket in the UI -2. Click the **Properties** tab -3. Find the **Default Encryption** card -4. Click **Enable Encryption** -5. Choose algorithm: - - **AES-256**: Uses the server's master key - - **aws:kms**: Uses a KMS-managed key (select from dropdown) -6. Save changes +- Run every 24 hours +- Batch size 1000 +- Auto-heal disabled -Once enabled, all **new objects** uploaded to the bucket will be automatically encrypted. +### Metrics history -### KMS Key Management - -When `KMS_ENABLED=true`, you can manage encryption keys via the KMS API: +Enable with: ```bash -# Create a new KMS key -curl -X POST http://localhost:5000/kms/keys \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{"alias": "my-key", "description": "Production encryption key"}' - -# List all keys -curl http://localhost:5000/kms/keys \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Get key details -curl http://localhost:5000/kms/keys/{key-id} \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Rotate a key (creates new key material) -curl -X POST http://localhost:5000/kms/keys/{key-id}/rotate \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Disable/Enable a key -curl -X POST http://localhost:5000/kms/keys/{key-id}/disable \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -curl -X POST http://localhost:5000/kms/keys/{key-id}/enable \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Schedule key deletion (30-day waiting period) -curl -X DELETE http://localhost:5000/kms/keys/{key-id}?waiting_period_days=30 \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +METRICS_HISTORY_ENABLED=true cargo run -p myfsio-server -- ``` -### How It Works - -1. **Envelope Encryption**: Each object is encrypted with a unique Data Encryption Key (DEK) -2. **Key Wrapping**: The DEK is encrypted (wrapped) by the master key or KMS key -3. **Storage**: The encrypted DEK is stored alongside the encrypted object -4. **Decryption**: On read, the DEK is unwrapped and used to decrypt the object - -### Client-Side Encryption - -For additional security, you can use client-side encryption. The `ClientEncryptionHelper` class provides utilities: - -```python -from app.encryption import ClientEncryptionHelper - -# Generate a client-side key -key = ClientEncryptionHelper.generate_key() -key_b64 = ClientEncryptionHelper.key_to_base64(key) - -# Encrypt before upload -plaintext = b"sensitive data" -encrypted, metadata = ClientEncryptionHelper.encrypt_for_upload(plaintext, key) - -# Upload with metadata headers -# x-amz-meta-x-amz-key: -# x-amz-meta-x-amz-iv: -# x-amz-meta-x-amz-matdesc: - -# Decrypt after download -decrypted = ClientEncryptionHelper.decrypt_from_download(encrypted, metadata, key) -``` - -### SSE-C (Customer-Provided Keys) - -With SSE-C, you provide your own 256-bit AES encryption key with each request. The server encrypts/decrypts using your key but never stores it. You must supply the same key for both upload and download. - -**Required headers:** - -| Header | Value | -|--------|-------| -| `x-amz-server-side-encryption-customer-algorithm` | `AES256` | -| `x-amz-server-side-encryption-customer-key` | Base64-encoded 256-bit key | -| `x-amz-server-side-encryption-customer-key-MD5` | Base64-encoded MD5 of the key | +Tune it with: ```bash -# Generate a 256-bit key -KEY=$(openssl rand -base64 32) -KEY_MD5=$(echo -n "$KEY" | base64 -d | openssl dgst -md5 -binary | base64) - -# Upload with SSE-C -curl -X PUT "http://localhost:5000/my-bucket/secret.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-server-side-encryption-customer-algorithm: AES256" \ - -H "x-amz-server-side-encryption-customer-key: $KEY" \ - -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" \ - --data-binary @secret.txt - -# Download with SSE-C (same key required) -curl "http://localhost:5000/my-bucket/secret.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-server-side-encryption-customer-algorithm: AES256" \ - -H "x-amz-server-side-encryption-customer-key: $KEY" \ - -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" +METRICS_HISTORY_INTERVAL_MINUTES=10 +METRICS_HISTORY_RETENTION_HOURS=72 ``` -**Key points:** -- SSE-C does not require `ENCRYPTION_ENABLED` or `KMS_ENABLED` — the key is provided per-request -- If you lose your key, the data is irrecoverable -- The MD5 header is optional but recommended for integrity verification +Snapshots are stored in `data/.myfsio.sys/config/metrics_history.json`. -### Important Notes +### Operation metrics -- **Existing objects are NOT encrypted** - Only new uploads after enabling encryption are encrypted -- **Master key security** - The master key file (`master.key`) should be backed up securely and protected -- **Key rotation** - Rotating a KMS key creates new key material; existing objects remain encrypted with the old material -- **Disabled keys** - Objects encrypted with a disabled key cannot be decrypted until the key is re-enabled -- **Deleted keys** - Once a key is deleted (after the waiting period), objects encrypted with it are permanently inaccessible - -### Verifying Encryption - -To verify an object is encrypted: -1. Check the raw file in `data//` - it should be unreadable binary -2. Look for `.meta` files containing encryption metadata -3. Download via the API/UI - the object should be automatically decrypted - -## 8. Bucket Quotas - -MyFSIO supports **storage quotas** to limit how much data a bucket can hold. Quotas are enforced on uploads and multipart completions. - -### Quota Types - -| Limit | Description | -|-------|-------------| -| **Max Size (MB)** | Maximum total storage in megabytes (includes current objects + archived versions) | -| **Max Objects** | Maximum number of objects (includes current objects + archived versions) | - -### Managing Quotas (Admin Only) - -Quota management is restricted to administrators (users with `iam:*` or `iam:list_users` permissions). - -#### Via UI - -1. Navigate to your bucket in the UI -2. Click the **Properties** tab -3. Find the **Storage Quota** card -4. Enter limits: - - **Max Size (MB)**: Leave empty for unlimited - - **Max Objects**: Leave empty for unlimited -5. Click **Update Quota** - -To remove a quota, click **Remove Quota**. - -#### Via API +Enable with: ```bash -# Set quota (max 100MB, max 1000 objects) -curl -X PUT "http://localhost:5000/bucket/?quota" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{"max_bytes": 104857600, "max_objects": 1000}' - -# Get current quota -curl "http://localhost:5000/bucket/?quota" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Remove quota -curl -X PUT "http://localhost:5000/bucket/?quota" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{"max_bytes": null, "max_objects": null}' +OPERATION_METRICS_ENABLED=true cargo run -p myfsio-server -- ``` -### Quota Behavior - -- **Version Counting**: When versioning is enabled, archived versions count toward the quota -- **Enforcement Points**: Quotas are checked during `PUT` object and `CompleteMultipartUpload` operations -- **Error Response**: When quota is exceeded, the API returns `HTTP 400` with error code `QuotaExceeded` -- **Visibility**: All users can view quota usage in the bucket detail page, but only admins can modify quotas - -### Example Error - -```xml - - QuotaExceeded - Bucket quota exceeded: storage limit reached - my-bucket - -``` - -## 9. Operation Metrics - -Operation metrics provide real-time visibility into API request statistics, including request counts, latency, error rates, and bandwidth usage. - -### Enabling Operation Metrics - -By default, operation metrics are disabled. Enable by setting the environment variable: +Tune it with: ```bash -OPERATION_METRICS_ENABLED=true python run.py -``` - -Or in your `myfsio.env` file: -``` -OPERATION_METRICS_ENABLED=true OPERATION_METRICS_INTERVAL_MINUTES=5 OPERATION_METRICS_RETENTION_HOURS=24 ``` -### Configuration Options +Snapshots are stored in `data/.myfsio.sys/config/operation_metrics.json`. -| Variable | Default | Description | -|----------|---------|-------------| -| `OPERATION_METRICS_ENABLED` | `false` | Enable/disable operation metrics | -| `OPERATION_METRICS_INTERVAL_MINUTES` | `5` | Snapshot interval (minutes) | -| `OPERATION_METRICS_RETENTION_HOURS` | `24` | History retention period (hours) | +## 9. Encryption and KMS -### What's Tracked - -**Request Statistics:** -- Request counts by HTTP method (GET, PUT, POST, DELETE, HEAD, OPTIONS) -- Response status codes grouped by class (2xx, 3xx, 4xx, 5xx) -- Latency statistics (min, max, average) -- Bytes transferred in/out - -**Endpoint Breakdown:** -- `object` - Object operations (GET/PUT/DELETE objects) -- `bucket` - Bucket operations (list, create, delete buckets) -- `ui` - Web UI requests -- `service` - Health checks, internal endpoints -- `kms` - KMS API operations - -**S3 Error Codes:** -Tracks API-specific error codes like `NoSuchKey`, `AccessDenied`, `BucketNotFound`. Note: These are separate from HTTP status codes - a 404 from the UI won't appear here, only S3 API errors. - -### API Endpoints +Object encryption and built-in KMS are both optional. ```bash -# Get current operation metrics -curl http://localhost:5100/ui/metrics/operations \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Get operation metrics history -curl http://localhost:5100/ui/metrics/operations/history \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Filter history by time range -curl "http://localhost:5100/ui/metrics/operations/history?hours=6" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +ENCRYPTION_ENABLED=true KMS_ENABLED=true cargo run -p myfsio-server -- ``` -### Storage Location +Notes: -Operation metrics data is stored at: -``` -data/.myfsio.sys/config/operation_metrics.json -``` +- If `ENCRYPTION_ENABLED=true` and `SECRET_KEY` is not configured, the server still starts, but `--check-config` warns that secure-at-rest config encryption is unavailable. +- KMS and the object encryption master key live under `data/.myfsio.sys/keys/`. -### UI Dashboard +## 10. Docker -When enabled, the Metrics page (`/ui/metrics`) shows an "API Operations" section with: -- Summary cards: Requests, Success Rate, Errors, Latency, Bytes In, Bytes Out -- Charts: Requests by Method (doughnut), Requests by Status (bar), Requests by Endpoint (horizontal bar) -- S3 Error Codes table with distribution - -Data refreshes every 5 seconds. - -## 10. Site Replication - -### Permission Model - -Replication uses a two-tier permission system: - -| Role | Capabilities | -|------|--------------| -| **Admin** (users with `iam:*` permissions) | Create/delete replication rules, configure connections and target buckets | -| **Users** (with `replication` permission) | Enable/disable (pause/resume) existing replication rules | - -> **Note:** The Replication tab is hidden for users without the `replication` permission on the bucket. - -This separation allows administrators to pre-configure where data should replicate, while allowing authorized users to toggle replication on/off without accessing connection credentials. - -### Replication Modes - -| Mode | Behavior | -|------|----------| -| `new_only` | Only replicate new/modified objects (default) | -| `all` | Sync all existing objects when rule is enabled | -| `bidirectional` | Two-way sync with Last-Write-Wins conflict resolution | - -### Architecture - -- **Source Instance**: The MyFSIO instance where you upload files. It runs the replication worker. -- **Target Instance**: Another MyFSIO instance (or any S3-compatible service like AWS S3, MinIO) that receives the copies. - -For `new_only` and `all` modes, replication is **asynchronous** (happens in the background) and **one-way** (Source -> Target). - -For `bidirectional` mode, replication is **two-way** with automatic conflict resolution. - -### Setup Guide - -#### 1. Prepare the Target Instance - -If your target is another MyFSIO server (e.g., running on a different machine or port), you need to create a destination bucket and a user with write permissions. - -**Option A: Using the UI (Easiest)** -If you have access to the UI of the target instance: -1. Log in to the Target UI. -2. Create a new bucket (e.g., `backup-bucket`). -3. Go to **IAM**, create a new user (e.g., `replication-user`), and copy the Access/Secret keys. - -**Option B: Headless Setup (API Only)** -If the target server is only running the API (`run_api.py`) and has no UI access, you can bootstrap the credentials and bucket by running a Python script on the server itself. - -Run this script on the **Target Server**: - -```python -# setup_target.py -from pathlib import Path -from app.iam import IamService -from app.storage import ObjectStorage - -# Initialize services (paths match default config) -data_dir = Path("data") -iam = IamService(data_dir / ".myfsio.sys" / "config" / "iam.json") -storage = ObjectStorage(data_dir) - -# 1. Create the bucket -bucket_name = "backup-bucket" -try: - storage.create_bucket(bucket_name) - print(f"Bucket '{bucket_name}' created.") -except Exception as e: - print(f"Bucket creation skipped: {e}") - -# 2. Create the user -try: - # Create user with full access (or restrict policy as needed) - creds = iam.create_user( - display_name="Replication User", - policies=[{"bucket": bucket_name, "actions": ["write", "read", "list"]}] - ) - print("\n--- CREDENTIALS GENERATED ---") - print(f"Access Key: {creds['access_key']}") - print(f"Secret Key: {creds['secret_key']}") - print("-----------------------------") -except Exception as e: - print(f"User creation failed: {e}") -``` - -Save and run: `python setup_target.py` - -#### 2. Configure the Source Instance - -Now, configure the primary instance to replicate to the target. - -1. **Access the Console**: - Log in to the UI of your Source Instance. - -2. **Add a Connection**: - - Navigate to **Connections** in the top menu. - - Click **Add Connection**. - - **Name**: `Secondary Site`. - - **Endpoint URL**: The URL of your Target Instance's API (e.g., `http://target-server:5002`). - - **Access Key**: The key you generated on the Target. - - **Secret Key**: The secret you generated on the Target. - - Click **Add Connection**. - -3. **Enable Replication** (Admin): - - Navigate to **Buckets** and select the source bucket. - - Switch to the **Replication** tab. - - Select the `Secondary Site` connection. - - Enter the target bucket name (`backup-bucket`). - - Click **Enable Replication**. - - Once configured, users with `replication` permission on this bucket can pause/resume replication without needing access to connection details. - -### Verification - -1. Upload a file to the source bucket. -2. Check the target bucket (via UI, CLI, or API). The file should appear shortly. +Build the Rust image from the `rust/` directory: ```bash -# Verify on target using AWS CLI -aws --endpoint-url http://target-server:5002 s3 ls s3://backup-bucket +docker build -t myfsio ./rust +docker run --rm \ + -p 5000:5000 \ + -p 5100:5100 \ + -v "$PWD/data:/app/data" \ + myfsio ``` -### Pausing and Resuming Replication +The container entrypoint runs `/usr/local/bin/myfsio-server`. -Users with the `replication` permission (but not admin rights) can pause and resume existing replication rules: +Inside the image: -1. Navigate to the bucket's **Replication** tab. -2. If replication is **Active**, click **Pause Replication** to temporarily stop syncing. -3. If replication is **Paused**, click **Resume Replication** to continue syncing. +- `HOST=0.0.0.0` +- `PORT=5000` +- `STORAGE_ROOT=/app/data` -When paused, new objects uploaded to the source will not replicate until replication is resumed. Objects uploaded while paused will be replicated once resumed. +If you want generated links and presigned URLs to use an external hostname, set `API_BASE_URL`. -> **Note:** Only admins can create new replication rules, change the target connection/bucket, or delete rules entirely. +## 11. Linux Installer -### Bidirectional Site Replication - -For true two-way synchronization with automatic conflict resolution, use the `bidirectional` replication mode. This enables a background sync worker that periodically pulls changes from the remote site. - -> **Important:** Both sites must be configured to sync with each other. Each site pushes its changes and pulls from the other. You must set up connections and replication rules on both ends. - -#### Step 1: Enable Site Sync on Both Sites - -Set these environment variables on **both** Site A and Site B: +The repository includes `scripts/install.sh`. For the Rust server, build the binary first and pass the path explicitly: ```bash -SITE_SYNC_ENABLED=true -SITE_SYNC_INTERVAL_SECONDS=60 # How often to pull changes (default: 60) -SITE_SYNC_BATCH_SIZE=100 # Max objects per sync cycle (default: 100) +cd rust/myfsio-engine +cargo build --release -p myfsio-server + +cd ../.. +sudo ./scripts/install.sh --binary ./rust/myfsio-engine/target/release/myfsio-server ``` -#### Step 2: Create IAM Users for Cross-Site Access +The installer copies that binary to `/opt/myfsio/myfsio`, creates `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` systemd unit. -On each site, create an IAM user that the other site will use to connect: +## 12. Updating and Rollback -| Site | Create User For | Required Permissions | -|------|-----------------|---------------------| -| Site A | Site B to connect | `read`, `write`, `list`, `delete` on target bucket | -| Site B | Site A to connect | `read`, `write`, `list`, `delete` on target bucket | +Recommended update flow: -Example policy for the replication user: -```json -[{"bucket": "my-bucket", "actions": ["read", "write", "list", "delete"]}] +1. Stop the running service. +2. Back up `data/.myfsio.sys/config/`. +3. Build or download the new Rust binary. +4. Run `myfsio-server --check-config` against the target environment. +5. Start the service and verify `/myfsio/health`. + +Example backup: + +```bash +cp -r data/.myfsio.sys/config config-backup ``` -#### Step 3: Create Connections +Health check: -On each site, add a connection pointing to the other: +```bash +curl http://127.0.0.1:5000/myfsio/health +``` -**On Site A:** -- Go to **Connections** and add a connection to Site B -- Endpoint: `https://site-b.example.com` -- Credentials: Site B's IAM user (created in Step 2) - -**On Site B:** -- Go to **Connections** and add a connection to Site A -- Endpoint: `https://site-a.example.com` -- Credentials: Site A's IAM user (created in Step 2) - -#### Step 4: Enable Bidirectional Replication - -On each site, go to the bucket's **Replication** tab and enable with mode `bidirectional`: - -**On Site A:** -- Source bucket: `my-bucket` -- Target connection: Site B connection -- Target bucket: `my-bucket` -- Mode: **Bidirectional sync** - -**On Site B:** -- Source bucket: `my-bucket` -- Target connection: Site A connection -- Target bucket: `my-bucket` -- Mode: **Bidirectional sync** - -#### How It Works - -- **PUSH**: Local changes replicate to remote immediately on write/delete -- **PULL**: Background worker fetches remote changes every `SITE_SYNC_INTERVAL_SECONDS` -- **Loop Prevention**: `S3ReplicationAgent` and `SiteSyncAgent` User-Agents prevent infinite sync loops - -#### Conflict Resolution (Last-Write-Wins) - -When the same object exists on both sites, the system uses Last-Write-Wins (LWW) based on `last_modified` timestamps: - -- **Remote newer**: Pull the remote version -- **Local newer**: Keep the local version -- **Same timestamp**: Use ETag as tiebreaker (higher ETag wins) - -A 1-second clock skew tolerance prevents false conflicts from minor time differences. - -#### Deletion Synchronization - -When `sync_deletions=true` (default), remote deletions propagate locally only if: -1. The object was previously synced FROM remote (tracked in sync state) -2. The local version hasn't been modified since last sync - -This prevents accidental deletion of local-only objects. - -#### Sync State Storage - -Sync state is stored at: `data/.myfsio.sys/buckets//site_sync_state.json` +The response includes the active Rust crate version: ```json { - "synced_objects": { - "path/to/file.txt": { - "last_synced_at": 1706100000.0, - "remote_etag": "abc123", - "source": "remote" - } - }, - "last_full_sync": 1706100000.0 + "status": "ok", + "version": "0.5.0" } ``` -### Legacy Bidirectional Setup (Manual) +## 13. Credential Reset -For simpler use cases without the site sync worker, you can manually configure two one-way rules: - -1. Follow the steps above to replicate **A → B**. -2. Repeat the process on Server B to replicate **B → A**: - - Create a connection on Server B pointing to Server A. - - Enable replication on the target bucket on Server B. - -**Loop Prevention**: The system automatically detects replication traffic using custom User-Agents (`S3ReplicationAgent` and `SiteSyncAgent`). This prevents infinite loops where an object replicated from A to B is immediately replicated back to A. - -**Deletes**: Deleting an object on one server will propagate the deletion to the other server. - -**Note**: Deleting a bucket will automatically remove its associated replication configuration. - -## 12. Running Tests +To rotate the bootstrap admin credentials: ```bash -pytest -q +cargo run -p myfsio-server -- --reset-cred ``` -The suite now includes a boto3 integration test that spins up a live HTTP server and drives the API through the official AWS SDK. If you want to skip it (for faster unit-only loops), run `pytest -m "not integration"`. +The command: -The suite covers bucket CRUD, presigned downloads, bucket policy enforcement, and regression tests for anonymous reads when a Public policy is attached. +- backs up the existing IAM file with a timestamped `.bak-...` suffix +- writes a fresh admin config +- respects `ADMIN_ACCESS_KEY` and `ADMIN_SECRET_KEY` if you set them -## 13. Troubleshooting +## 14. Testing -| Symptom | Likely Cause | Fix | -| --- | --- | --- | -| 403 from API despite Public preset | Policy didn’t save or bucket key path mismatch | Reapply Public preset, confirm bucket name in `Resource` matches `arn:aws:s3:::bucket/*`. | -| UI still shows old policy text | Browser cached view before hot reload | Refresh; JSON is already reloaded on server. | -| Presign modal errors with 403 | IAM user lacks `read/write/delete` for target bucket or bucket policy denies | Update IAM inline policies or remove conflicting deny statements. | -| Large upload rejected immediately | File exceeds `MAX_UPLOAD_SIZE` | Increase env var or shrink object. | - -## 14. API Matrix - -``` -# Service Endpoints -GET /myfsio/health # Health check - -# Bucket Operations -GET / # List buckets -PUT / # Create bucket -DELETE / # Remove bucket -GET / # List objects (supports ?list-type=2) -HEAD / # Check bucket exists -POST / # POST object upload (HTML form) -POST /?delete # Bulk delete objects - -# Bucket Configuration -GET /?policy # Fetch bucket policy -PUT /?policy # Upsert bucket policy -DELETE /?policy # Delete bucket policy -GET /?quota # Get bucket quota -PUT /?quota # Set bucket quota (admin only) -GET /?versioning # Get versioning status -PUT /?versioning # Enable/disable versioning -GET /?lifecycle # Get lifecycle rules -PUT /?lifecycle # Set lifecycle rules -DELETE /?lifecycle # Delete lifecycle rules -GET /?cors # Get CORS configuration -PUT /?cors # Set CORS configuration -DELETE /?cors # Delete CORS configuration -GET /?encryption # Get encryption configuration -PUT /?encryption # Set default encryption -DELETE /?encryption # Delete encryption configuration -GET /?acl # Get bucket ACL -PUT /?acl # Set bucket ACL -GET /?tagging # Get bucket tags -PUT /?tagging # Set bucket tags -DELETE /?tagging # Delete bucket tags -GET /?replication # Get replication configuration -PUT /?replication # Set replication rules -DELETE /?replication # Delete replication configuration -GET /?logging # Get access logging configuration -PUT /?logging # Set access logging -GET /?notification # Get event notifications -PUT /?notification # Set event notifications (webhooks) -GET /?object-lock # Get object lock configuration -PUT /?object-lock # Set object lock configuration -GET /?website # Get website configuration -PUT /?website # Set website configuration -DELETE /?website # Delete website configuration -GET /?uploads # List active multipart uploads -GET /?versions # List object versions -GET /?location # Get bucket location/region - -# Object Operations -PUT // # Upload object -GET // # Download object (supports Range header) -DELETE // # Delete object -HEAD // # Get object metadata -POST // # POST upload with policy -POST //?select # SelectObjectContent (SQL query) - -# Object Configuration -GET //?tagging # Get object tags -PUT //?tagging # Set object tags -DELETE //?tagging # Delete object tags -GET //?acl # Get object ACL -PUT //?acl # Set object ACL -PUT //?retention # Set object retention -GET //?retention # Get object retention -PUT //?legal-hold # Set legal hold -GET //?legal-hold # Get legal hold status - -# Multipart Upload -POST //?uploads # Initiate multipart upload -PUT //?uploadId=X&partNumber=N # Upload part -PUT //?uploadId=X&partNumber=N (with x-amz-copy-source) # UploadPartCopy -POST //?uploadId=X # Complete multipart upload -DELETE //?uploadId=X # Abort multipart upload -GET //?uploadId=X # List parts - -# Copy Operations -PUT // (with x-amz-copy-source header) # CopyObject - -# Admin API -GET /admin/site # Get local site info -PUT /admin/site # Update local site -GET /admin/sites # List peer sites -POST /admin/sites # Register peer site -GET /admin/sites/ # Get peer site -PUT /admin/sites/ # Update peer site -DELETE /admin/sites/ # Unregister peer site -GET /admin/sites//health # Check peer health -GET /admin/topology # Get cluster topology -GET /admin/website-domains # List domain mappings -POST /admin/website-domains # Create domain mapping -GET /admin/website-domains/ # Get domain mapping -PUT /admin/website-domains/ # Update domain mapping -DELETE /admin/website-domains/ # Delete domain mapping - -# KMS API -GET /kms/keys # List KMS keys -POST /kms/keys # Create KMS key -GET /kms/keys/ # Get key details -DELETE /kms/keys/ # Schedule key deletion -POST /kms/keys//enable # Enable key -POST /kms/keys//disable # Disable key -POST /kms/keys//rotate # Rotate key material -POST /kms/encrypt # Encrypt data -POST /kms/decrypt # Decrypt data -POST /kms/generate-data-key # Generate data key -POST /kms/generate-random # Generate random bytes -``` - -## 15. Health Check Endpoint - -The API exposes a simple health check endpoint for monitoring and load balancer integration: +Run the Rust test suite: ```bash -# Check API health -curl http://localhost:5000/myfsio/health - -# Response -{"status": "ok", "version": "0.1.7"} +cd rust/myfsio-engine +cargo test ``` -The response includes: -- `status`: Always `"ok"` when the server is running -- `version`: Current application version from `app/version.py` +If you are validating documentation changes for the UI, the most relevant coverage lives under: -Use this endpoint for: -- Load balancer health checks -- Kubernetes liveness/readiness probes -- Monitoring system integration (Prometheus, Datadog, etc.) +- `rust/myfsio-engine/crates/myfsio-server/tests` +- `rust/myfsio-engine/crates/myfsio-storage/src` -## 16. Object Lock & Retention +## 15. API Notes -Object Lock prevents objects from being deleted or overwritten for a specified retention period. MyFSIO supports both GOVERNANCE and COMPLIANCE modes. +The Rust server exposes: -### Retention Modes +- `GET /myfsio/health` +- S3 bucket and object operations on `/` and `//` +- UI routes under `/ui/...` +- admin routes under `/admin/...` +- KMS routes under `/kms/...` -| Mode | Description | -|------|-------------| -| **GOVERNANCE** | Objects can't be deleted by normal users, but users with `s3:BypassGovernanceRetention` permission can override | -| **COMPLIANCE** | Objects can't be deleted or overwritten by anyone, including root, until the retention period expires | +For a route-level view, inspect: -### Enabling Object Lock - -Object Lock must be enabled when creating a bucket: - -```bash -# Create bucket with Object Lock enabled -curl -X PUT "http://localhost:5000/my-bucket" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-bucket-object-lock-enabled: true" - -# Set default retention configuration -curl -X PUT "http://localhost:5000/my-bucket?object-lock" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "ObjectLockEnabled": "Enabled", - "Rule": { - "DefaultRetention": { - "Mode": "GOVERNANCE", - "Days": 30 - } - } - }' -``` - -### Per-Object Retention - -Set retention on individual objects: - -```bash -# Set object retention -curl -X PUT "http://localhost:5000/my-bucket/important.pdf?retention" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "Mode": "COMPLIANCE", - "RetainUntilDate": "2025-12-31T23:59:59Z" - }' - -# Get object retention -curl "http://localhost:5000/my-bucket/important.pdf?retention" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." -``` - -### Legal Hold - -Legal hold provides indefinite protection independent of retention settings: - -```bash -# Enable legal hold -curl -X PUT "http://localhost:5000/my-bucket/document.pdf?legal-hold" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{"Status": "ON"}' - -# Disable legal hold -curl -X PUT "http://localhost:5000/my-bucket/document.pdf?legal-hold" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{"Status": "OFF"}' - -# Check legal hold status -curl "http://localhost:5000/my-bucket/document.pdf?legal-hold" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." -``` - -## 17. Access Logging - -Enable S3-style access logging to track all requests to your buckets. - -### Configuration - -```bash -# Enable access logging -curl -X PUT "http://localhost:5000/my-bucket?logging" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "LoggingEnabled": { - "TargetBucket": "log-bucket", - "TargetPrefix": "logs/my-bucket/" - } - }' - -# Get logging configuration -curl "http://localhost:5000/my-bucket?logging" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Disable logging (empty configuration) -curl -X PUT "http://localhost:5000/my-bucket?logging" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{}' -``` - -### Log Format - -Access logs are written in S3-compatible format with fields including: -- Timestamp, bucket, key -- Operation (REST.GET.OBJECT, REST.PUT.OBJECT, etc.) -- Request ID, requester, source IP -- HTTP status, error code, bytes sent -- Total time, turn-around time -- Referrer, User-Agent - -## 18. Bucket Notifications & Webhooks - -Configure event notifications to trigger webhooks when objects are created or deleted. - -### Supported Events - -| Event Type | Description | -|-----------|-------------| -| `s3:ObjectCreated:*` | Any object creation (PUT, POST, COPY, multipart) | -| `s3:ObjectCreated:Put` | Object created via PUT | -| `s3:ObjectCreated:Post` | Object created via POST | -| `s3:ObjectCreated:Copy` | Object created via COPY | -| `s3:ObjectCreated:CompleteMultipartUpload` | Multipart upload completed | -| `s3:ObjectRemoved:*` | Any object deletion | -| `s3:ObjectRemoved:Delete` | Object deleted | -| `s3:ObjectRemoved:DeleteMarkerCreated` | Delete marker created (versioned bucket) | - -### Configuration - -```bash -# Set notification configuration -curl -X PUT "http://localhost:5000/my-bucket?notification" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "TopicConfigurations": [ - { - "Id": "upload-notify", - "TopicArn": "https://webhook.example.com/s3-events", - "Events": ["s3:ObjectCreated:*"], - "Filter": { - "Key": { - "FilterRules": [ - {"Name": "prefix", "Value": "uploads/"}, - {"Name": "suffix", "Value": ".jpg"} - ] - } - } - } - ] - }' - -# Get notification configuration -curl "http://localhost:5000/my-bucket?notification" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." -``` - -### Webhook Payload - -The webhook receives a JSON payload similar to AWS S3 event notifications: - -```json -{ - "Records": [ - { - "eventVersion": "2.1", - "eventSource": "myfsio:s3", - "eventTime": "2024-01-15T10:30:00.000Z", - "eventName": "ObjectCreated:Put", - "s3": { - "bucket": {"name": "my-bucket"}, - "object": { - "key": "uploads/photo.jpg", - "size": 102400, - "eTag": "abc123..." - } - } - } - ] -} -``` - -### Security Notes - -- Webhook URLs are validated to prevent SSRF attacks -- Internal/private IP ranges are blocked by default -- Use HTTPS endpoints in production - -## 19. SelectObjectContent (SQL Queries) - -Query CSV, JSON, or Parquet files directly using SQL without downloading the entire object. Requires DuckDB to be installed. - -### Prerequisites - -```bash -pip install duckdb -``` - -### Usage - -```bash -# Query a CSV file -curl -X POST "http://localhost:5000/my-bucket/data.csv?select" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "Expression": "SELECT name, age FROM s3object WHERE age > 25", - "ExpressionType": "SQL", - "InputSerialization": { - "CSV": { - "FileHeaderInfo": "USE", - "FieldDelimiter": "," - } - }, - "OutputSerialization": { - "JSON": {} - } - }' - -# Query a JSON file -curl -X POST "http://localhost:5000/my-bucket/data.json?select" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "Expression": "SELECT * FROM s3object s WHERE s.status = '\"active'\"", - "ExpressionType": "SQL", - "InputSerialization": {"JSON": {"Type": "LINES"}}, - "OutputSerialization": {"JSON": {}} - }' -``` - -### Supported Input Formats - -| Format | Options | -|--------|---------| -| **CSV** | `FileHeaderInfo` (USE, IGNORE, NONE), `FieldDelimiter`, `QuoteCharacter`, `RecordDelimiter` | -| **JSON** | `Type` (DOCUMENT, LINES) | -| **Parquet** | Automatic schema detection | - -### Output Formats - -- **JSON**: Returns results as JSON records -- **CSV**: Returns results as CSV - -## 20. PostObject (HTML Form Upload) - -Upload objects using HTML forms with policy-based authorization. Useful for browser-based direct uploads. - -### Form Fields - -| Field | Required | Description | -|-------|----------|-------------| -| `key` | Yes | Object key (can include `${filename}` placeholder) | -| `file` | Yes | The file to upload | -| `policy` | No | Base64-encoded policy document | -| `x-amz-signature` | No | Policy signature | -| `x-amz-credential` | No | Credential scope | -| `x-amz-algorithm` | No | Signing algorithm (AWS4-HMAC-SHA256) | -| `x-amz-date` | No | Request timestamp | -| `Content-Type` | No | MIME type of the file | -| `x-amz-meta-*` | No | Custom metadata | - -### Example HTML Form - -```html -
- - - - - -
-``` - -### With Policy (Signed Upload) - -For authenticated uploads, include a policy document: - -```bash -# Generate policy and signature using boto3 or similar -# Then include in form: -# - policy: base64(policy_document) -# - x-amz-signature: HMAC-SHA256(policy, signing_key) -# - x-amz-credential: access_key/date/region/s3/aws4_request -# - x-amz-algorithm: AWS4-HMAC-SHA256 -# - x-amz-date: YYYYMMDDTHHMMSSZ -``` - -## 21. Advanced S3 Operations - -### CopyObject - -Copy objects within or between buckets: - -```bash -# Copy within same bucket -curl -X PUT "http://localhost:5000/my-bucket/copy-of-file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-copy-source: /my-bucket/original-file.txt" - -# Copy to different bucket -curl -X PUT "http://localhost:5000/other-bucket/file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-copy-source: /my-bucket/original-file.txt" - -# Copy with metadata replacement -curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-copy-source: /my-bucket/file.txt" \ - -H "x-amz-metadata-directive: REPLACE" \ - -H "x-amz-meta-newkey: newvalue" -``` - -### MoveObject (UI) - -Move an object to a different key or bucket. This is a UI-only convenience operation that performs a copy followed by a delete of the source. Requires `read` and `delete` on the source, and `write` on the destination. - -```bash -# Move via UI API -curl -X POST "http://localhost:5100/ui/buckets/my-bucket/objects/old-path/file.txt/move" \ - -H "Content-Type: application/json" \ - --cookie "session=..." \ - -d '{"dest_bucket": "other-bucket", "dest_key": "new-path/file.txt"}' -``` - -The move is atomic from the caller's perspective: if the copy succeeds but the delete fails, the object exists in both locations (no data loss). - -### UploadPartCopy - -Copy data from an existing object into a multipart upload part: - -```bash -# Initiate multipart upload -UPLOAD_ID=$(curl -X POST "http://localhost:5000/my-bucket/large-file.bin?uploads" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." | jq -r '.UploadId') - -# Copy bytes 0-10485759 from source as part 1 -curl -X PUT "http://localhost:5000/my-bucket/large-file.bin?uploadId=$UPLOAD_ID&partNumber=1" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-copy-source: /source-bucket/source-file.bin" \ - -H "x-amz-copy-source-range: bytes=0-10485759" - -# Copy bytes 10485760-20971519 as part 2 -curl -X PUT "http://localhost:5000/my-bucket/large-file.bin?uploadId=$UPLOAD_ID&partNumber=2" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-copy-source: /source-bucket/source-file.bin" \ - -H "x-amz-copy-source-range: bytes=10485760-20971519" -``` - -### Range Requests - -Download partial content using the Range header: - -```bash -# Get first 1000 bytes -curl "http://localhost:5000/my-bucket/large-file.bin" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "Range: bytes=0-999" - -# Get bytes 1000-1999 -curl "http://localhost:5000/my-bucket/large-file.bin" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "Range: bytes=1000-1999" - -# Get last 500 bytes -curl "http://localhost:5000/my-bucket/large-file.bin" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "Range: bytes=-500" - -# Get from byte 5000 to end -curl "http://localhost:5000/my-bucket/large-file.bin" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "Range: bytes=5000-" -``` - -Range responses include: -- HTTP 206 Partial Content status -- `Content-Range` header showing the byte range -- `Accept-Ranges: bytes` header - -### Conditional Requests - -Use conditional headers for cache validation: - -```bash -# Only download if modified since -curl "http://localhost:5000/my-bucket/file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "If-Modified-Since: Wed, 15 Jan 2025 10:00:00 GMT" - -# Only download if ETag doesn't match (changed) -curl "http://localhost:5000/my-bucket/file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "If-None-Match: \"abc123...\"" - -# Only download if ETag matches -curl "http://localhost:5000/my-bucket/file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "If-Match: \"abc123...\"" -``` - -## 22. Access Control Lists (ACLs) - -ACLs provide legacy-style permission management for buckets and objects. - -### Canned ACLs - -| ACL | Description | -|-----|-------------| -| `private` | Owner gets FULL_CONTROL (default) | -| `public-read` | Owner FULL_CONTROL, public READ | -| `public-read-write` | Owner FULL_CONTROL, public READ and WRITE | -| `authenticated-read` | Owner FULL_CONTROL, authenticated users READ | - -### Setting ACLs - -```bash -# Set bucket ACL using canned ACL -curl -X PUT "http://localhost:5000/my-bucket?acl" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-acl: public-read" - -# Set object ACL -curl -X PUT "http://localhost:5000/my-bucket/file.txt?acl" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-acl: private" - -# Set ACL during upload -curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-acl: public-read" \ - --data-binary @file.txt - -# Get bucket ACL -curl "http://localhost:5000/my-bucket?acl" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Get object ACL -curl "http://localhost:5000/my-bucket/file.txt?acl" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." -``` - -### ACL vs Bucket Policies - -- **ACLs**: Simple, limited options, legacy approach -- **Bucket Policies**: Powerful, flexible, recommended for new deployments - -For most use cases, prefer bucket policies over ACLs. - -## 23. Object & Bucket Tagging - -Add metadata tags to buckets and objects for organization, cost allocation, or lifecycle rule filtering. - -### Bucket Tagging - -```bash -# Set bucket tags -curl -X PUT "http://localhost:5000/my-bucket?tagging" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "TagSet": [ - {"Key": "Environment", "Value": "Production"}, - {"Key": "Team", "Value": "Engineering"} - ] - }' - -# Get bucket tags -curl "http://localhost:5000/my-bucket?tagging" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Delete bucket tags -curl -X DELETE "http://localhost:5000/my-bucket?tagging" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." -``` - -### Object Tagging - -```bash -# Set object tags -curl -X PUT "http://localhost:5000/my-bucket/file.txt?tagging" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "TagSet": [ - {"Key": "Classification", "Value": "Confidential"}, - {"Key": "Owner", "Value": "john@example.com"} - ] - }' - -# Get object tags -curl "http://localhost:5000/my-bucket/file.txt?tagging" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Delete object tags -curl -X DELETE "http://localhost:5000/my-bucket/file.txt?tagging" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Set tags during upload -curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -H "x-amz-tagging: Environment=Staging&Team=QA" \ - --data-binary @file.txt -``` - -### Tagging Limits - -- Maximum 50 tags per object (configurable via `OBJECT_TAG_LIMIT`) -- Tag key: 1-128 Unicode characters -- Tag value: 0-256 Unicode characters - -### Use Cases - -- **Lifecycle Rules**: Filter objects for expiration by tag -- **Access Control**: Use tag conditions in bucket policies -- **Cost Tracking**: Group objects by project or department -- **Automation**: Trigger actions based on object tags - -## 24. CORS Configuration - -Configure Cross-Origin Resource Sharing for browser-based applications. - -### Setting CORS Rules - -```bash -# Set CORS configuration -curl -X PUT "http://localhost:5000/my-bucket?cors" \ - -H "Content-Type: application/json" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ - -d '{ - "CORSRules": [ - { - "AllowedOrigins": ["https://example.com", "https://app.example.com"], - "AllowedMethods": ["GET", "PUT", "POST", "DELETE"], - "AllowedHeaders": ["*"], - "ExposeHeaders": ["ETag", "x-amz-meta-*"], - "MaxAgeSeconds": 3600 - } - ] - }' - -# Get CORS configuration -curl "http://localhost:5000/my-bucket?cors" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Delete CORS configuration -curl -X DELETE "http://localhost:5000/my-bucket?cors" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." -``` - -### CORS Rule Fields - -| Field | Description | -|-------|-------------| -| `AllowedOrigins` | Origins allowed to access the bucket (required) | -| `AllowedMethods` | HTTP methods allowed (GET, PUT, POST, DELETE, HEAD) | -| `AllowedHeaders` | Request headers allowed in preflight | -| `ExposeHeaders` | Response headers visible to browser | -| `MaxAgeSeconds` | How long browser can cache preflight response | - -## 25. List Objects API v2 - -MyFSIO supports both ListBucketResult v1 and v2 APIs. - -### Using v2 API - -```bash -# List with v2 (supports continuation tokens) -curl "http://localhost:5000/my-bucket?list-type=2" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# With prefix and delimiter (folder-like listing) -curl "http://localhost:5000/my-bucket?list-type=2&prefix=photos/&delimiter=/" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Pagination with continuation token -curl "http://localhost:5000/my-bucket?list-type=2&max-keys=100&continuation-token=TOKEN" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." - -# Start after specific key -curl "http://localhost:5000/my-bucket?list-type=2&start-after=photos/2024/" \ - -H "X-Access-Key: ..." -H "X-Secret-Key: ..." -``` - -### v1 vs v2 Differences - -| Feature | v1 | v2 | -|---------|----|----| -| Pagination | `marker` | `continuation-token` | -| Start position | `marker` | `start-after` | -| Fetch owner info | Always included | Use `fetch-owner=true` | -| Max keys | 1000 | 1000 | - -### Query Parameters - -| Parameter | Description | -|-----------|-------------| -| `list-type` | Set to `2` for v2 API | -| `prefix` | Filter objects by key prefix | -| `delimiter` | Group objects (typically `/`) | -| `max-keys` | Maximum results (1-1000, default 1000) | -| `continuation-token` | Token from previous response | -| `start-after` | Start listing after this key | -| `fetch-owner` | Include owner info in response | -| `encoding-type` | Set to `url` for URL-encoded keys - -## 26. Static Website Hosting - -MyFSIO can serve S3 buckets as static websites via custom domain mappings. When a request arrives with a `Host` header matching a mapped domain, MyFSIO resolves the bucket and serves objects directly. - -### Enabling - -Set the environment variable: - -```bash -WEBSITE_HOSTING_ENABLED=true -``` - -When disabled, all website hosting endpoints return 400 and domain-based serving is skipped. - -### Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `WEBSITE_HOSTING_ENABLED` | `false` | Master switch for website hosting | - -### Setting Up a Website - -**Step 1: Configure the bucket website settings** - -```bash -curl -X PUT "http://localhost:5000/my-site?website" \ - -H "Authorization: ..." \ - -d ' - - index.html - 404.html -' -``` - -- `IndexDocument` with `Suffix` is required (must not contain `/`) -- `ErrorDocument` is optional - -**Step 2: Map a domain to the bucket** - -```bash -curl -X POST "http://localhost:5000/admin/website-domains" \ - -H "Authorization: ..." \ - -H "Content-Type: application/json" \ - -d '{"domain": "example.com", "bucket": "my-site"}' -``` - -**Step 3: Point your domain to MyFSIO** - -For HTTP-only (direct access), point DNS to the MyFSIO host on port 5000. - -For HTTPS (recommended), use a reverse proxy. The critical requirement is passing the original `Host` header so MyFSIO can match the domain to a bucket. - -**nginx example:** - -```nginx -server { - server_name example.com; - listen 443 ssl; - - ssl_certificate /etc/ssl/certs/example.com.pem; - ssl_certificate_key /etc/ssl/private/example.com.key; - - location / { - proxy_pass http://127.0.0.1:5000; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - } -} -``` - -`proxy_set_header Host $host;` is required — without it, MyFSIO cannot match the incoming domain to a bucket. You do not need any path-based routing rules; MyFSIO handles all object resolution internally. - -### How Domain Routing Works - -1. A request arrives with `Host: example.com` -2. MyFSIO's `before_request` hook strips the port and looks up the domain in the `WebsiteDomainStore` -3. If a match is found, it loads the bucket's website config (index/error documents) -4. Object key resolution: - - `/` or trailing `/` → append `index_document` (e.g., `index.html`) - - `/path` → try exact match, then try `path/index_document` - - Not found → serve `error_document` with 404 status -5. If no domain match is found, the request falls through to normal S3 API / UI routing - -### Domain Mapping Admin API - -All endpoints require admin (`iam:*`) permissions. - -| Method | Route | Body | Description | -|--------|-------|------|-------------| -| `GET` | `/admin/website-domains` | — | List all mappings | -| `POST` | `/admin/website-domains` | `{"domain": "...", "bucket": "..."}` | Create mapping | -| `GET` | `/admin/website-domains/` | — | Get single mapping | -| `PUT` | `/admin/website-domains/` | `{"bucket": "..."}` | Update mapping | -| `DELETE` | `/admin/website-domains/` | — | Delete mapping | - -### Bucket Website API - -| Method | Route | Description | -|--------|-------|-------------| -| `PUT` | `/?website` | Set website config (XML body) | -| `GET` | `/?website` | Get website config (XML response) | -| `DELETE` | `/?website` | Remove website config | - -### Web UI - -- **Per-bucket config:** Bucket Details → Properties tab → "Static Website Hosting" card -- **Domain management:** Sidebar → "Domains" (visible when hosting is enabled and user is admin) +- `rust/myfsio-engine/crates/myfsio-server/src/lib.rs` +- `rust/myfsio-engine/crates/myfsio-server/src/handlers/` diff --git a/python/app/ui.py b/python/app/ui.py index 29ffd05..f43ddb2 100644 --- a/python/app/ui.py +++ b/python/app/ui.py @@ -225,10 +225,10 @@ def _policy_allows_public_read(policy: dict[str, Any]) -> bool: def _bucket_access_descriptor(policy: dict[str, Any] | None) -> tuple[str, str]: if not policy: - return ("IAM only", "text-bg-secondary") + return ("IAM only", "bg-secondary-subtle text-secondary-emphasis") if _policy_allows_public_read(policy): - return ("Public read", "text-bg-warning") - return ("Custom policy", "text-bg-info") + return ("Public read", "bg-warning-subtle text-warning-emphasis") + return ("Custom policy", "bg-info-subtle text-info-emphasis") def _current_principal(): diff --git a/python/templates/bucket_detail.html b/python/templates/bucket_detail.html index f913880..c3a168f 100644 --- a/python/templates/bucket_detail.html +++ b/python/templates/bucket_detail.html @@ -921,14 +921,14 @@
- Storage quota enabled + Storage quota active

{% if max_bytes is not none and max_objects is not none %} - Limited to {{ max_bytes | filesizeformat }} and {{ max_objects }} objects. + This bucket is limited to {{ max_bytes | filesizeformat }} storage and {{ max_objects }} objects. {% elif max_bytes is not none %} - Limited to {{ max_bytes | filesizeformat }} storage. + This bucket is limited to {{ max_bytes | filesizeformat }} storage. {% else %} - Limited to {{ max_objects }} objects. + This bucket is limited to {{ max_objects }} objects. {% endif %}

diff --git a/rust/myfsio-engine/Cargo.lock b/rust/myfsio-engine/Cargo.lock index c470a57..dd29eae 100644 --- a/rust/myfsio-engine/Cargo.lock +++ b/rust/myfsio-engine/Cargo.lock @@ -326,6 +326,18 @@ dependencies = [ "regex-syntax", ] +[[package]] +name = "async-compression" +version = "0.4.41" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0f9ee0f6e02ffd7ad5816e9464499fba7b3effd01123b515c41d1697c43dad1" +dependencies = [ + "compression-codecs", + "compression-core", + "pin-project-lite", + "tokio", +] + [[package]] name = "async-trait" version = "0.1.89" @@ -1165,6 +1177,23 @@ dependencies = [ "unicode-width", ] +[[package]] +name = "compression-codecs" +version = "0.4.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb7b51a7d9c967fc26773061ba86150f19c50c0d65c887cb1fbe295fd16619b7" +dependencies = [ + "compression-core", + "flate2", + "memchr", +] + +[[package]] +name = "compression-core" +version = "0.4.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "75984efb6ed102a0d42db99afb6c1948f0380d1d91808d5529916e6c08b49d8d" + [[package]] name = "const-oid" version = "0.9.6" @@ -1442,6 +1471,12 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "dotenvy" +version = "0.15.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aaf95b3e5c8f23aa320147307562d361db0ae0d51242340f558153b4eb2439b" + [[package]] name = "duckdb" version = "1.10501.0" @@ -2137,7 +2172,7 @@ dependencies = [ "js-sys", "log", "wasm-bindgen", - "windows-core", + "windows-core 0.62.2", ] [[package]] @@ -2595,7 +2630,7 @@ dependencies = [ [[package]] name = "myfsio-auth" -version = "0.1.0" +version = "0.5.0" dependencies = [ "aes", "base64", @@ -2608,6 +2643,7 @@ dependencies = [ "parking_lot", "pbkdf2", "percent-encoding", + "rand 0.8.5", "serde", "serde_json", "sha2 0.10.9", @@ -2619,7 +2655,7 @@ dependencies = [ [[package]] name = "myfsio-common" -version = "0.1.0" +version = "0.5.0" dependencies = [ "chrono", "serde", @@ -2630,7 +2666,7 @@ dependencies = [ [[package]] name = "myfsio-crypto" -version = "0.1.0" +version = "0.5.0" dependencies = [ "aes-gcm", "base64", @@ -2651,8 +2687,9 @@ dependencies = [ [[package]] name = "myfsio-server" -version = "0.1.0" +version = "0.5.0" dependencies = [ + "aes-gcm", "async-trait", "aws-config", "aws-credential-types", @@ -2665,6 +2702,7 @@ dependencies = [ "clap", "cookie", "crc32fast", + "dotenvy", "duckdb", "futures", "http-body-util", @@ -2686,6 +2724,7 @@ dependencies = [ "serde", "serde_json", "subtle", + "sysinfo", "tempfile", "tera", "tokio", @@ -2699,7 +2738,7 @@ dependencies = [ [[package]] name = "myfsio-storage" -version = "0.1.0" +version = "0.5.0" dependencies = [ "chrono", "dashmap", @@ -2722,7 +2761,7 @@ dependencies = [ [[package]] name = "myfsio-xml" -version = "0.1.0" +version = "0.5.0" dependencies = [ "chrono", "myfsio-common", @@ -2730,6 +2769,15 @@ dependencies = [ "serde", ] +[[package]] +name = "ntapi" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3b335231dfd352ffb0f8017f3b6027a4917f7df785ea2143d8af2adc66980ae" +dependencies = [ + "winapi", +] + [[package]] name = "nu-ansi-term" version = "0.50.3" @@ -3222,6 +3270,26 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rayon" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d" +dependencies = [ + "either", + "rayon-core", +] + +[[package]] +name = "rayon-core" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" +dependencies = [ + "crossbeam-deque", + "crossbeam-utils", +] + [[package]] name = "redox_syscall" version = "0.5.18" @@ -3905,6 +3973,20 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "sysinfo" +version = "0.32.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c33cd241af0f2e9e3b5c32163b873b29956890b5342e6745b917ce9d490f4af" +dependencies = [ + "core-foundation-sys", + "libc", + "memchr", + "ntapi", + "rayon", + "windows", +] + [[package]] name = "tap" version = "1.0.1" @@ -4164,6 +4246,7 @@ version = "0.6.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8" dependencies = [ + "async-compression", "bitflags", "bytes", "futures-core", @@ -4569,6 +4652,22 @@ dependencies = [ "rustls-pki-types", ] +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + [[package]] name = "winapi-util" version = "0.1.11" @@ -4578,19 +4677,58 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + +[[package]] +name = "windows" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12342cb4d8e3b046f3d80effd474a7a02447231330ef77d71daa6fbc40681143" +dependencies = [ + "windows-core 0.57.0", + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-core" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2ed2439a290666cd67ecce2b0ffaad89c2a56b976b736e6ece670297897832d" +dependencies = [ + "windows-implement 0.57.0", + "windows-interface 0.57.0", + "windows-result 0.1.2", + "windows-targets 0.52.6", +] + [[package]] name = "windows-core" version = "0.62.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" dependencies = [ - "windows-implement", - "windows-interface", + "windows-implement 0.60.2", + "windows-interface 0.59.3", "windows-link", - "windows-result", + "windows-result 0.4.1", "windows-strings", ] +[[package]] +name = "windows-implement" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9107ddc059d5b6fbfbffdfa7a7fe3e22a226def0b2608f72e9d552763d3e1ad7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "windows-implement" version = "0.60.2" @@ -4602,6 +4740,17 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "windows-interface" +version = "0.57.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29bee4b38ea3cde66011baa44dba677c432a78593e202392d1e9070cf2a7fca7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "windows-interface" version = "0.59.3" @@ -4619,6 +4768,15 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" +[[package]] +name = "windows-result" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5e383302e8ec8515204254685643de10811af0ed97ea37210dc26fb0032647f8" +dependencies = [ + "windows-targets 0.52.6", +] + [[package]] name = "windows-result" version = "0.4.1" diff --git a/rust/myfsio-engine/Cargo.toml b/rust/myfsio-engine/Cargo.toml index 5e049b7..6f3bde5 100644 --- a/rust/myfsio-engine/Cargo.toml +++ b/rust/myfsio-engine/Cargo.toml @@ -9,11 +9,15 @@ members = [ "crates/myfsio-server", ] +[workspace.package] +version = "0.4.3" +edition = "2021" + [workspace.dependencies] tokio = { version = "1", features = ["full"] } axum = { version = "0.8" } tower = { version = "0.5" } -tower-http = { version = "0.6", features = ["cors", "trace", "fs"] } +tower-http = { version = "0.6", features = ["cors", "trace", "fs", "compression-gzip"] } hyper = { version = "1" } bytes = "1" serde = { version = "1", features = ["derive"] } @@ -54,3 +58,4 @@ tera = "1" cookie = "0.18" subtle = "2" clap = { version = "4", features = ["derive"] } +dotenvy = "0.15" diff --git a/rust/myfsio-engine/crates/myfsio-auth/Cargo.toml b/rust/myfsio-engine/crates/myfsio-auth/Cargo.toml index 384427a..7622bf9 100644 --- a/rust/myfsio-engine/crates/myfsio-auth/Cargo.toml +++ b/rust/myfsio-engine/crates/myfsio-auth/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "myfsio-auth" -version = "0.1.0" -edition = "2021" +version.workspace = true +edition.workspace = true [dependencies] myfsio-common = { path = "../myfsio-common" } @@ -12,6 +12,7 @@ aes = { workspace = true } cbc = { workspace = true } base64 = { workspace = true } pbkdf2 = "0.12" +rand = "0.8" lru = { workspace = true } parking_lot = { workspace = true } percent-encoding = { workspace = true } diff --git a/rust/myfsio-engine/crates/myfsio-auth/src/fernet.rs b/rust/myfsio-engine/crates/myfsio-auth/src/fernet.rs index ba7fb64..1b2e129 100644 --- a/rust/myfsio-engine/crates/myfsio-auth/src/fernet.rs +++ b/rust/myfsio-engine/crates/myfsio-auth/src/fernet.rs @@ -1,9 +1,11 @@ -use aes::cipher::{block_padding::Pkcs7, BlockDecryptMut, KeyIvInit}; +use aes::cipher::{block_padding::Pkcs7, BlockDecryptMut, BlockEncryptMut, KeyIvInit}; use base64::{engine::general_purpose::URL_SAFE, Engine}; use hmac::{Hmac, Mac}; +use rand::RngCore; use sha2::Sha256; type Aes128CbcDec = cbc::Decryptor; +type Aes128CbcEnc = cbc::Encryptor; type HmacSha256 = Hmac; pub fn derive_fernet_key(secret: &str) -> String { @@ -44,8 +46,7 @@ pub fn decrypt(key_b64: &str, token: &str) -> Result, &'static str> { let payload = &token_bytes[..hmac_offset]; let expected_hmac = &token_bytes[hmac_offset..]; - let mut mac = - HmacSha256::new_from_slice(signing_key).map_err(|_| "hmac key error")?; + let mut mac = HmacSha256::new_from_slice(signing_key).map_err(|_| "hmac key error")?; mac.update(payload); mac.verify_slice(expected_hmac) .map_err(|_| "HMAC verification failed")?; @@ -60,6 +61,43 @@ pub fn decrypt(key_b64: &str, token: &str) -> Result, &'static str> { Ok(plaintext) } +pub fn encrypt(key_b64: &str, plaintext: &[u8]) -> Result { + let key_bytes = URL_SAFE + .decode(key_b64) + .map_err(|_| "invalid fernet key base64")?; + if key_bytes.len() != 32 { + return Err("fernet key must be 32 bytes"); + } + + let signing_key = &key_bytes[..16]; + let encryption_key = &key_bytes[16..]; + + let mut iv = [0u8; 16]; + rand::thread_rng().fill_bytes(&mut iv); + + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_err(|_| "system time error")? + .as_secs(); + + let ciphertext = Aes128CbcEnc::new(encryption_key.into(), (&iv).into()) + .encrypt_padded_vec_mut::(plaintext); + + let mut payload = Vec::with_capacity(1 + 8 + 16 + ciphertext.len()); + payload.push(0x80); + payload.extend_from_slice(×tamp.to_be_bytes()); + payload.extend_from_slice(&iv); + payload.extend_from_slice(&ciphertext); + + let mut mac = HmacSha256::new_from_slice(signing_key).map_err(|_| "hmac key error")?; + mac.update(&payload); + let tag = mac.finalize().into_bytes(); + + let mut token_bytes = payload; + token_bytes.extend_from_slice(&tag); + Ok(URL_SAFE.encode(&token_bytes)) +} + #[cfg(test)] mod tests { use super::*; diff --git a/rust/myfsio-engine/crates/myfsio-auth/src/iam.rs b/rust/myfsio-engine/crates/myfsio-auth/src/iam.rs index 34f41bd..a7f45b9 100644 --- a/rust/myfsio-engine/crates/myfsio-auth/src/iam.rs +++ b/rust/myfsio-engine/crates/myfsio-auth/src/iam.rs @@ -69,7 +69,10 @@ impl RawIamUser { } } let display_name = self.display_name.unwrap_or_else(|| { - access_keys.first().map(|k| k.access_key.clone()).unwrap_or_else(|| "unknown".to_string()) + access_keys + .first() + .map(|k| k.access_key.clone()) + .unwrap_or_else(|| "unknown".to_string()) }); let user_id = self.user_id.unwrap_or_else(|| { format!("u-{}", display_name.to_ascii_lowercase().replace(' ', "-")) @@ -173,7 +176,7 @@ impl IamService { (None, Some(_)) => true, (Some(old), Some(new)) => old != new, (Some(_), None) => true, - (None, None) => state.key_secrets.is_empty(), + (None, None) => false, } }; @@ -188,7 +191,11 @@ impl IamService { let content = match std::fs::read_to_string(&self.config_path) { Ok(c) => c, Err(e) => { - tracing::warn!("Failed to read IAM config {}: {}", self.config_path.display(), e); + tracing::warn!( + "Failed to read IAM config {}: {}", + self.config_path.display(), + e + ); return; } }; @@ -205,7 +212,10 @@ impl IamService { } }, Err(e) => { - tracing::error!("Failed to decrypt IAM config: {}. SECRET_KEY may have changed.", e); + tracing::error!( + "Failed to decrypt IAM config: {}. SECRET_KEY may have changed.", + e + ); return; } }, @@ -226,7 +236,11 @@ impl IamService { } }; - let users: Vec = raw_config.users.into_iter().map(|u| u.normalize()).collect(); + let users: Vec = raw_config + .users + .into_iter() + .map(|u| u.normalize()) + .collect(); let mut key_secrets = HashMap::new(); let mut key_index = HashMap::new(); @@ -254,9 +268,11 @@ impl IamService { state.file_mtime = file_mtime; state.last_check = Instant::now(); - tracing::info!("IAM config reloaded: {} users, {} keys", + tracing::info!( + "IAM config reloaded: {} users, {} keys", users.len(), - state.key_secrets.len()); + state.key_secrets.len() + ); } pub fn get_secret_key(&self, access_key: &str) -> Option { @@ -308,9 +324,10 @@ impl IamService { } } - let is_admin = user.policies.iter().any(|p| { - p.bucket == "*" && p.actions.iter().any(|a| a == "*") - }); + let is_admin = user + .policies + .iter() + .any(|p| p.bucket == "*" && p.actions.iter().any(|a| a == "*")); Some(Principal::new( access_key.to_string(), @@ -341,10 +358,7 @@ impl IamService { return true; } - let normalized_bucket = bucket_name - .unwrap_or("*") - .trim() - .to_ascii_lowercase(); + let normalized_bucket = bucket_name.unwrap_or("*").trim().to_ascii_lowercase(); let normalized_action = action.trim().to_ascii_lowercase(); let state = self.state.read(); @@ -383,6 +397,46 @@ impl IamService { false } + pub fn export_config(&self, mask_secrets: bool) -> serde_json::Value { + self.reload_if_needed(); + let state = self.state.read(); + let users: Vec = state + .user_records + .values() + .map(|u| { + let access_keys: Vec = u + .access_keys + .iter() + .map(|k| { + let secret = if mask_secrets { + "***".to_string() + } else { + k.secret_key.clone() + }; + serde_json::json!({ + "access_key": k.access_key, + "secret_key": secret, + "status": k.status, + "created_at": k.created_at, + }) + }) + .collect(); + serde_json::json!({ + "user_id": u.user_id, + "display_name": u.display_name, + "enabled": u.enabled, + "expires_at": u.expires_at, + "access_keys": access_keys, + "policies": u.policies, + }) + }) + .collect(); + serde_json::json!({ + "version": 2, + "users": users, + }) + } + pub async fn list_users(&self) -> Vec { self.reload_if_needed(); let state = self.state.read(); @@ -411,12 +465,12 @@ impl IamService { self.reload_if_needed(); let state = self.state.read(); - let user = state - .user_records - .get(identifier) - .or_else(|| { - state.key_index.get(identifier).and_then(|uid| state.user_records.get(uid)) - })?; + let user = state.user_records.get(identifier).or_else(|| { + state + .key_index + .get(identifier) + .and_then(|uid| state.user_records.get(uid)) + })?; Some(serde_json::json!({ "user_id": user.user_id, @@ -449,8 +503,7 @@ impl IamService { .users .iter_mut() .find(|u| { - u.user_id == identifier - || u.access_keys.iter().any(|k| k.access_key == identifier) + u.user_id == identifier || u.access_keys.iter().any(|k| k.access_key == identifier) }) .ok_or_else(|| "User not found".to_string())?; @@ -468,12 +521,12 @@ impl IamService { pub fn get_user_policies(&self, identifier: &str) -> Option> { self.reload_if_needed(); let state = self.state.read(); - let user = state - .user_records - .get(identifier) - .or_else(|| { - state.key_index.get(identifier).and_then(|uid| state.user_records.get(uid)) - })?; + let user = state.user_records.get(identifier).or_else(|| { + state + .key_index + .get(identifier) + .and_then(|uid| state.user_records.get(uid)) + })?; Some( user.policies .iter() @@ -496,8 +549,7 @@ impl IamService { .users .iter_mut() .find(|u| { - u.user_id == identifier - || u.access_keys.iter().any(|k| k.access_key == identifier) + u.user_id == identifier || u.access_keys.iter().any(|k| k.access_key == identifier) }) .ok_or_else(|| format!("User '{}' not found", identifier))?; @@ -557,6 +609,178 @@ impl IamService { self.reload(); Ok(()) } + + fn load_config(&self) -> Result { + let content = std::fs::read_to_string(&self.config_path) + .map_err(|e| format!("Failed to read IAM config: {}", e))?; + let raw_text = if content.starts_with("MYFSIO_IAM_ENC:") { + let encrypted_token = &content["MYFSIO_IAM_ENC:".len()..]; + let key = self.fernet_key.as_ref().ok_or_else(|| { + "IAM config is encrypted but no SECRET_KEY configured".to_string() + })?; + let plaintext = crate::fernet::decrypt(key, encrypted_token.trim()) + .map_err(|e| format!("Failed to decrypt IAM config: {}", e))?; + String::from_utf8(plaintext) + .map_err(|e| format!("Decrypted IAM config not UTF-8: {}", e))? + } else { + content + }; + let raw: RawIamConfig = serde_json::from_str(&raw_text) + .map_err(|e| format!("Failed to parse IAM config: {}", e))?; + Ok(IamConfig { + version: 2, + users: raw.users.into_iter().map(|u| u.normalize()).collect(), + }) + } + + fn save_config(&self, config: &IamConfig) -> Result<(), String> { + let json = serde_json::to_string_pretty(config) + .map_err(|e| format!("Failed to serialize IAM config: {}", e))?; + let payload = if let Some(key) = &self.fernet_key { + let token = crate::fernet::encrypt(key, json.as_bytes()) + .map_err(|e| format!("Failed to encrypt IAM config: {}", e))?; + format!("MYFSIO_IAM_ENC:{}", token) + } else { + json + }; + std::fs::write(&self.config_path, payload) + .map_err(|e| format!("Failed to write IAM config: {}", e))?; + self.reload(); + Ok(()) + } + + pub fn create_user( + &self, + display_name: &str, + policies: Option>, + access_key: Option, + secret_key: Option, + expires_at: Option, + ) -> Result { + let mut config = self.load_config()?; + + let new_ak = access_key + .filter(|s| !s.trim().is_empty()) + .unwrap_or_else(|| format!("AK{}", uuid::Uuid::new_v4().simple())); + let new_sk = secret_key + .filter(|s| !s.trim().is_empty()) + .unwrap_or_else(|| format!("SK{}", uuid::Uuid::new_v4().simple())); + + if config + .users + .iter() + .any(|u| u.access_keys.iter().any(|k| k.access_key == new_ak)) + { + return Err(format!("Access key '{}' already exists", new_ak)); + } + + let user_id = format!("u-{}", uuid::Uuid::new_v4().simple()); + let resolved_policies = policies.unwrap_or_else(|| { + vec![IamPolicy { + bucket: "*".to_string(), + actions: vec!["*".to_string()], + prefix: "*".to_string(), + }] + }); + + let user = IamUser { + user_id: user_id.clone(), + display_name: display_name.to_string(), + enabled: true, + expires_at, + access_keys: vec![AccessKey { + access_key: new_ak.clone(), + secret_key: new_sk.clone(), + status: "active".to_string(), + created_at: Some(chrono::Utc::now().to_rfc3339()), + }], + policies: resolved_policies, + }; + config.users.push(user); + + self.save_config(&config)?; + Ok(serde_json::json!({ + "user_id": user_id, + "access_key": new_ak, + "secret_key": new_sk, + "display_name": display_name, + })) + } + + pub fn delete_user(&self, identifier: &str) -> Result<(), String> { + let mut config = self.load_config()?; + let before = config.users.len(); + config.users.retain(|u| { + u.user_id != identifier && !u.access_keys.iter().any(|k| k.access_key == identifier) + }); + if config.users.len() == before { + return Err(format!("User '{}' not found", identifier)); + } + self.save_config(&config) + } + + pub fn update_user( + &self, + identifier: &str, + display_name: Option, + expires_at: Option>, + ) -> Result<(), String> { + let mut config = self.load_config()?; + let user = config + .users + .iter_mut() + .find(|u| { + u.user_id == identifier || u.access_keys.iter().any(|k| k.access_key == identifier) + }) + .ok_or_else(|| format!("User '{}' not found", identifier))?; + if let Some(name) = display_name { + user.display_name = name; + } + if let Some(exp) = expires_at { + user.expires_at = exp; + } + self.save_config(&config) + } + + pub fn update_user_policies( + &self, + identifier: &str, + policies: Vec, + ) -> Result<(), String> { + let mut config = self.load_config()?; + let user = config + .users + .iter_mut() + .find(|u| { + u.user_id == identifier || u.access_keys.iter().any(|k| k.access_key == identifier) + }) + .ok_or_else(|| format!("User '{}' not found", identifier))?; + user.policies = policies; + self.save_config(&config) + } + + pub fn rotate_secret(&self, identifier: &str) -> Result { + let mut config = self.load_config()?; + let user = config + .users + .iter_mut() + .find(|u| { + u.user_id == identifier || u.access_keys.iter().any(|k| k.access_key == identifier) + }) + .ok_or_else(|| format!("User '{}' not found", identifier))?; + let key = user + .access_keys + .first_mut() + .ok_or_else(|| "User has no access keys".to_string())?; + let new_sk = format!("SK{}", uuid::Uuid::new_v4().simple()); + key.secret_key = new_sk.clone(); + let ak = key.access_key.clone(); + self.save_config(&config)?; + Ok(serde_json::json!({ + "access_key": ak, + "secret_key": new_sk, + })) + } } fn bucket_matches(policy_bucket: &str, bucket: &str) -> bool { @@ -622,10 +846,7 @@ mod tests { let svc = IamService::new(tmp.path().to_path_buf()); let secret = svc.get_secret_key("AKIAIOSFODNN7EXAMPLE"); - assert_eq!( - secret.unwrap(), - "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" - ); + assert_eq!(secret.unwrap(), "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"); } #[test] @@ -664,7 +885,9 @@ mod tests { tmp.flush().unwrap(); let svc = IamService::new(tmp.path().to_path_buf()); - assert!(svc.authenticate("AKIAIOSFODNN7EXAMPLE", "wrongsecret").is_none()); + assert!(svc + .authenticate("AKIAIOSFODNN7EXAMPLE", "wrongsecret") + .is_none()); } #[test] @@ -784,29 +1007,9 @@ mod tests { let svc = IamService::new(tmp.path().to_path_buf()); let principal = svc.get_principal("READER_KEY").unwrap(); - assert!(svc.authorize( - &principal, - Some("docs"), - "read", - Some("reports/2026.csv"), - )); - assert!(!svc.authorize( - &principal, - Some("docs"), - "write", - Some("reports/2026.csv"), - )); - assert!(!svc.authorize( - &principal, - Some("docs"), - "read", - Some("private/2026.csv"), - )); - assert!(!svc.authorize( - &principal, - Some("other"), - "read", - Some("reports/2026.csv"), - )); + assert!(svc.authorize(&principal, Some("docs"), "read", Some("reports/2026.csv"),)); + assert!(!svc.authorize(&principal, Some("docs"), "write", Some("reports/2026.csv"),)); + assert!(!svc.authorize(&principal, Some("docs"), "read", Some("private/2026.csv"),)); + assert!(!svc.authorize(&principal, Some("other"), "read", Some("reports/2026.csv"),)); } } diff --git a/rust/myfsio-engine/crates/myfsio-auth/src/lib.rs b/rust/myfsio-engine/crates/myfsio-auth/src/lib.rs index 083404f..0d15c37 100644 --- a/rust/myfsio-engine/crates/myfsio-auth/src/lib.rs +++ b/rust/myfsio-engine/crates/myfsio-auth/src/lib.rs @@ -1,4 +1,4 @@ -pub mod sigv4; -pub mod principal; -pub mod iam; mod fernet; +pub mod iam; +pub mod principal; +pub mod sigv4; diff --git a/rust/myfsio-engine/crates/myfsio-auth/src/sigv4.rs b/rust/myfsio-engine/crates/myfsio-auth/src/sigv4.rs index b515ee0..5ef6997 100644 --- a/rust/myfsio-engine/crates/myfsio-auth/src/sigv4.rs +++ b/rust/myfsio-engine/crates/myfsio-auth/src/sigv4.rs @@ -64,7 +64,10 @@ pub fn derive_signing_key_cached( } } - let k_date = hmac_sha256(format!("AWS4{}", secret_key).as_bytes(), date_stamp.as_bytes()); + let k_date = hmac_sha256( + format!("AWS4{}", secret_key).as_bytes(), + date_stamp.as_bytes(), + ); let k_region = hmac_sha256(&k_date, region.as_bytes()); let k_service = hmac_sha256(&k_region, service.as_bytes()); let k_signing = hmac_sha256(&k_service, b"aws4_request"); @@ -134,7 +137,11 @@ pub fn verify_sigv4_signature( let canonical_request = format!( "{}\n{}\n{}\n{}\n{}\n{}", - method, canonical_uri, canonical_query_string, canonical_headers, signed_headers_str, + method, + canonical_uri, + canonical_query_string, + canonical_headers, + signed_headers_str, payload_hash ); @@ -197,7 +204,12 @@ mod tests { #[test] fn test_derive_signing_key() { - let key = derive_signing_key("wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "20130524", "us-east-1", "s3"); + let key = derive_signing_key( + "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", + "20130524", + "us-east-1", + "s3", + ); assert_eq!(key.len(), 32); } @@ -217,7 +229,11 @@ mod tests { #[test] fn test_build_string_to_sign() { - let result = build_string_to_sign("20130524T000000Z", "20130524/us-east-1/s3/aws4_request", "GET\n/\n\nhost:example.com\n\nhost\nUNSIGNED-PAYLOAD"); + let result = build_string_to_sign( + "20130524T000000Z", + "20130524/us-east-1/s3/aws4_request", + "GET\n/\n\nhost:example.com\n\nhost\nUNSIGNED-PAYLOAD", + ); assert!(result.starts_with("AWS4-HMAC-SHA256\n")); assert!(result.contains("20130524T000000Z")); } @@ -239,8 +255,13 @@ mod tests { let signing_key = derive_signing_key(secret, date_stamp, region, service); - let canonical_request = "GET\n/\n\nhost:examplebucket.s3.amazonaws.com\n\nhost\nUNSIGNED-PAYLOAD"; - let string_to_sign = build_string_to_sign(amz_date, &format!("{}/{}/{}/aws4_request", date_stamp, region, service), canonical_request); + let canonical_request = + "GET\n/\n\nhost:examplebucket.s3.amazonaws.com\n\nhost\nUNSIGNED-PAYLOAD"; + let string_to_sign = build_string_to_sign( + amz_date, + &format!("{}/{}/{}/aws4_request", date_stamp, region, service), + canonical_request, + ); let signature = compute_signature(&signing_key, &string_to_sign); @@ -249,7 +270,10 @@ mod tests { "/", &[], "host", - &[("host".to_string(), "examplebucket.s3.amazonaws.com".to_string())], + &[( + "host".to_string(), + "examplebucket.s3.amazonaws.com".to_string(), + )], "UNSIGNED-PAYLOAD", amz_date, date_stamp, diff --git a/rust/myfsio-engine/crates/myfsio-common/Cargo.toml b/rust/myfsio-engine/crates/myfsio-common/Cargo.toml index e140021..29774bc 100644 --- a/rust/myfsio-engine/crates/myfsio-common/Cargo.toml +++ b/rust/myfsio-engine/crates/myfsio-common/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "myfsio-common" -version = "0.1.0" -edition = "2021" +version.workspace = true +edition.workspace = true [dependencies] thiserror = { workspace = true } diff --git a/rust/myfsio-engine/crates/myfsio-crypto/Cargo.toml b/rust/myfsio-engine/crates/myfsio-crypto/Cargo.toml index 45cecc7..09be255 100644 --- a/rust/myfsio-engine/crates/myfsio-crypto/Cargo.toml +++ b/rust/myfsio-engine/crates/myfsio-crypto/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "myfsio-crypto" -version = "0.1.0" -edition = "2021" +version.workspace = true +edition.workspace = true [dependencies] myfsio-common = { path = "../myfsio-common" } diff --git a/rust/myfsio-engine/crates/myfsio-crypto/src/aes_gcm.rs b/rust/myfsio-engine/crates/myfsio-crypto/src/aes_gcm.rs index dc7a346..b349de3 100644 --- a/rust/myfsio-engine/crates/myfsio-crypto/src/aes_gcm.rs +++ b/rust/myfsio-engine/crates/myfsio-crypto/src/aes_gcm.rs @@ -193,7 +193,10 @@ mod tests { let decrypted = dir.path().join("decrypted.bin"); let data = b"Hello, this is a test of AES-256-GCM chunked encryption!"; - std::fs::File::create(&input).unwrap().write_all(data).unwrap(); + std::fs::File::create(&input) + .unwrap() + .write_all(data) + .unwrap(); let key = [0x42u8; 32]; let nonce = [0x01u8; 12]; @@ -212,9 +215,18 @@ mod tests { fn test_invalid_key_size() { let dir = tempfile::tempdir().unwrap(); let input = dir.path().join("input.bin"); - std::fs::File::create(&input).unwrap().write_all(b"test").unwrap(); + std::fs::File::create(&input) + .unwrap() + .write_all(b"test") + .unwrap(); - let result = encrypt_stream_chunked(&input, &dir.path().join("out"), &[0u8; 16], &[0u8; 12], None); + let result = encrypt_stream_chunked( + &input, + &dir.path().join("out"), + &[0u8; 16], + &[0u8; 12], + None, + ); assert!(matches!(result, Err(CryptoError::InvalidKeySize(16)))); } @@ -225,7 +237,10 @@ mod tests { let encrypted = dir.path().join("encrypted.bin"); let decrypted = dir.path().join("decrypted.bin"); - std::fs::File::create(&input).unwrap().write_all(b"secret data").unwrap(); + std::fs::File::create(&input) + .unwrap() + .write_all(b"secret data") + .unwrap(); let key = [0x42u8; 32]; let nonce = [0x01u8; 12]; diff --git a/rust/myfsio-engine/crates/myfsio-crypto/src/encryption.rs b/rust/myfsio-engine/crates/myfsio-crypto/src/encryption.rs index 2a78f3c..37a5193 100644 --- a/rust/myfsio-engine/crates/myfsio-crypto/src/encryption.rs +++ b/rust/myfsio-engine/crates/myfsio-crypto/src/encryption.rs @@ -4,9 +4,7 @@ use rand::RngCore; use std::collections::HashMap; use std::path::Path; -use crate::aes_gcm::{ - encrypt_stream_chunked, decrypt_stream_chunked, CryptoError, -}; +use crate::aes_gcm::{decrypt_stream_chunked, encrypt_stream_chunked, CryptoError}; use crate::kms::KmsService; #[derive(Debug, Clone, PartialEq)] @@ -172,15 +170,14 @@ impl EncryptionService { let ciphertext = kms.encrypt_data(kid, &data_key).await?; (Some(B64.encode(&ciphertext)), Some(kid.clone())) } - SseAlgorithm::CustomerProvided => { - (None, None) - } + SseAlgorithm::CustomerProvided => (None, None), }; let actual_key = if ctx.algorithm == SseAlgorithm::CustomerProvided { - let ck = ctx.customer_key.as_ref().ok_or_else(|| { - CryptoError::EncryptionFailed("No customer key provided".into()) - })?; + let ck = ctx + .customer_key + .as_ref() + .ok_or_else(|| CryptoError::EncryptionFailed("No customer key provided".into()))?; if ck.len() != 32 { return Err(CryptoError::InvalidKeySize(ck.len())); } @@ -195,11 +192,9 @@ impl EncryptionService { let op = output_path.to_owned(); let ak = actual_key; let n = nonce; - tokio::task::spawn_blocking(move || { - encrypt_stream_chunked(&ip, &op, &ak, &n, None) - }) - .await - .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??; + tokio::task::spawn_blocking(move || encrypt_stream_chunked(&ip, &op, &ak, &n, None)) + .await + .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??; Ok(EncryptionMetadata { algorithm: ctx.algorithm.as_str().to_string(), @@ -216,9 +211,9 @@ impl EncryptionService { enc_meta: &EncryptionMetadata, customer_key: Option<&[u8]>, ) -> Result<(), CryptoError> { - let nonce_bytes = B64.decode(&enc_meta.nonce).map_err(|e| { - CryptoError::EncryptionFailed(format!("Bad nonce encoding: {}", e)) - })?; + let nonce_bytes = B64 + .decode(&enc_meta.nonce) + .map_err(|e| CryptoError::EncryptionFailed(format!("Bad nonce encoding: {}", e)))?; if nonce_bytes.len() != 12 { return Err(CryptoError::InvalidNonceSize(nonce_bytes.len())); } @@ -262,11 +257,9 @@ impl EncryptionService { let ip = input_path.to_owned(); let op = output_path.to_owned(); let nb: [u8; 12] = nonce_bytes.try_into().unwrap(); - tokio::task::spawn_blocking(move || { - decrypt_stream_chunked(&ip, &op, &data_key, &nb) - }) - .await - .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??; + tokio::task::spawn_blocking(move || decrypt_stream_chunked(&ip, &op, &data_key, &nb)) + .await + .map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??; Ok(()) } @@ -298,7 +291,10 @@ mod tests { let decrypted = dir.path().join("dec.bin"); let data = b"SSE-S3 encrypted content for testing!"; - std::fs::File::create(&input).unwrap().write_all(data).unwrap(); + std::fs::File::create(&input) + .unwrap() + .write_all(data) + .unwrap(); let svc = EncryptionService::new(test_master_key(), None); @@ -328,7 +324,10 @@ mod tests { let decrypted = dir.path().join("dec.bin"); let data = b"SSE-C encrypted content!"; - std::fs::File::create(&input).unwrap().write_all(data).unwrap(); + std::fs::File::create(&input) + .unwrap() + .write_all(data) + .unwrap(); let customer_key = [0xBBu8; 32]; let svc = EncryptionService::new(test_master_key(), None); @@ -369,7 +368,10 @@ mod tests { fn test_is_encrypted() { let mut meta = HashMap::new(); assert!(!EncryptionMetadata::is_encrypted(&meta)); - meta.insert("x-amz-server-side-encryption".to_string(), "AES256".to_string()); + meta.insert( + "x-amz-server-side-encryption".to_string(), + "AES256".to_string(), + ); assert!(EncryptionMetadata::is_encrypted(&meta)); } } diff --git a/rust/myfsio-engine/crates/myfsio-crypto/src/hashing.rs b/rust/myfsio-engine/crates/myfsio-crypto/src/hashing.rs index 88d4f54..31e2277 100644 --- a/rust/myfsio-engine/crates/myfsio-crypto/src/hashing.rs +++ b/rust/myfsio-engine/crates/myfsio-crypto/src/hashing.rs @@ -99,7 +99,10 @@ mod tests { #[test] fn test_sha256_bytes() { let hash = sha256_bytes(b"hello"); - assert_eq!(hash, "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"); + assert_eq!( + hash, + "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824" + ); } #[test] @@ -118,7 +121,10 @@ mod tests { tmp.flush().unwrap(); let (md5, sha) = md5_sha256_file(tmp.path()).unwrap(); assert_eq!(md5, "5d41402abc4b2a76b9719d911017c592"); - assert_eq!(sha, "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"); + assert_eq!( + sha, + "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824" + ); } #[tokio::test] diff --git a/rust/myfsio-engine/crates/myfsio-crypto/src/kms.rs b/rust/myfsio-engine/crates/myfsio-crypto/src/kms.rs index 790c9fe..afdd8d3 100644 --- a/rust/myfsio-engine/crates/myfsio-crypto/src/kms.rs +++ b/rust/myfsio-engine/crates/myfsio-crypto/src/kms.rs @@ -132,9 +132,7 @@ impl KmsService { async fn save(&self) -> Result<(), CryptoError> { let keys = self.keys.read().await; - let store = KmsStore { - keys: keys.clone(), - }; + let store = KmsStore { keys: keys.clone() }; let json = serde_json::to_string_pretty(&store) .map_err(|e| CryptoError::EncryptionFailed(e.to_string()))?; std::fs::write(&self.keys_path, json).map_err(CryptoError::Io)?; diff --git a/rust/myfsio-engine/crates/myfsio-crypto/src/lib.rs b/rust/myfsio-engine/crates/myfsio-crypto/src/lib.rs index 402bb7a..7f57a35 100644 --- a/rust/myfsio-engine/crates/myfsio-crypto/src/lib.rs +++ b/rust/myfsio-engine/crates/myfsio-crypto/src/lib.rs @@ -1,4 +1,4 @@ -pub mod hashing; pub mod aes_gcm; -pub mod kms; pub mod encryption; +pub mod hashing; +pub mod kms; diff --git a/rust/myfsio-engine/crates/myfsio-server/Cargo.toml b/rust/myfsio-engine/crates/myfsio-server/Cargo.toml index 58aa363..6a1092b 100644 --- a/rust/myfsio-engine/crates/myfsio-server/Cargo.toml +++ b/rust/myfsio-engine/crates/myfsio-server/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "myfsio-server" -version = "0.1.0" -edition = "2021" +version.workspace = true +edition.workspace = true [dependencies] myfsio-common = { path = "../myfsio-common" } @@ -45,6 +45,9 @@ tera = { workspace = true } cookie = { workspace = true } subtle = { workspace = true } clap = { workspace = true } +dotenvy = { workspace = true } +sysinfo = "0.32" +aes-gcm = { workspace = true } [dev-dependencies] tempfile = "3" diff --git a/rust/myfsio-engine/crates/myfsio-server/src/config.rs b/rust/myfsio-engine/crates/myfsio-server/src/config.rs index 009fa1a..6fffe74 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/config.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/config.rs @@ -4,6 +4,7 @@ use std::path::PathBuf; #[derive(Debug, Clone)] pub struct ServerConfig { pub bind_addr: SocketAddr, + pub ui_bind_addr: SocketAddr, pub storage_root: PathBuf, pub region: String, pub iam_config_path: PathBuf, @@ -16,6 +17,11 @@ pub struct ServerConfig { pub gc_enabled: bool, pub integrity_enabled: bool, pub metrics_enabled: bool, + pub metrics_history_enabled: bool, + pub metrics_interval_minutes: u64, + pub metrics_retention_hours: u64, + pub metrics_history_interval_minutes: u64, + pub metrics_history_retention_hours: u64, pub lifecycle_enabled: bool, pub website_hosting_enabled: bool, pub replication_connect_timeout_secs: u64, @@ -42,22 +48,28 @@ impl ServerConfig { .unwrap_or_else(|_| "5000".to_string()) .parse() .unwrap_or(5000); - let storage_root = std::env::var("STORAGE_ROOT") - .unwrap_or_else(|_| "./data".to_string()); - let region = std::env::var("AWS_REGION") - .unwrap_or_else(|_| "us-east-1".to_string()); + let ui_port: u16 = std::env::var("UI_PORT") + .unwrap_or_else(|_| "5100".to_string()) + .parse() + .unwrap_or(5100); + let storage_root = std::env::var("STORAGE_ROOT").unwrap_or_else(|_| "./data".to_string()); + let region = std::env::var("AWS_REGION").unwrap_or_else(|_| "us-east-1".to_string()); let storage_path = PathBuf::from(&storage_root); let iam_config_path = std::env::var("IAM_CONFIG") .map(PathBuf::from) .unwrap_or_else(|_| { - storage_path.join(".myfsio.sys").join("config").join("iam.json") + storage_path + .join(".myfsio.sys") + .join("config") + .join("iam.json") }); - let sigv4_timestamp_tolerance_secs: u64 = std::env::var("SIGV4_TIMESTAMP_TOLERANCE_SECONDS") - .unwrap_or_else(|_| "900".to_string()) - .parse() - .unwrap_or(900); + let sigv4_timestamp_tolerance_secs: u64 = + std::env::var("SIGV4_TIMESTAMP_TOLERANCE_SECONDS") + .unwrap_or_else(|_| "900".to_string()) + .parse() + .unwrap_or(900); let presigned_url_min_expiry: u64 = std::env::var("PRESIGNED_URL_MIN_EXPIRY_SECONDS") .unwrap_or_else(|_| "1".to_string()) @@ -78,40 +90,60 @@ impl ServerConfig { .join(".myfsio.sys") .join("config") .join(".secret"); - std::fs::read_to_string(&secret_file).ok().map(|s| s.trim().to_string()) + std::fs::read_to_string(&secret_file) + .ok() + .map(|s| s.trim().to_string()) } } }; let encryption_enabled = std::env::var("ENCRYPTION_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; let kms_enabled = std::env::var("KMS_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; let gc_enabled = std::env::var("GC_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; let integrity_enabled = std::env::var("INTEGRITY_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; let metrics_enabled = std::env::var("OPERATION_METRICS_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; + + let metrics_history_enabled = std::env::var("METRICS_HISTORY_ENABLED") + .unwrap_or_else(|_| "false".to_string()) + .to_lowercase() + == "true"; + + let metrics_interval_minutes = parse_u64_env("OPERATION_METRICS_INTERVAL_MINUTES", 5); + let metrics_retention_hours = parse_u64_env("OPERATION_METRICS_RETENTION_HOURS", 24); + let metrics_history_interval_minutes = parse_u64_env("METRICS_HISTORY_INTERVAL_MINUTES", 5); + let metrics_history_retention_hours = parse_u64_env("METRICS_HISTORY_RETENTION_HOURS", 24); let lifecycle_enabled = std::env::var("LIFECYCLE_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; let website_hosting_enabled = std::env::var("WEBSITE_HOSTING_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; - let replication_connect_timeout_secs = parse_u64_env("REPLICATION_CONNECT_TIMEOUT_SECONDS", 5); + let replication_connect_timeout_secs = + parse_u64_env("REPLICATION_CONNECT_TIMEOUT_SECONDS", 5); let replication_read_timeout_secs = parse_u64_env("REPLICATION_READ_TIMEOUT_SECONDS", 30); let replication_max_retries = parse_u64_env("REPLICATION_MAX_RETRIES", 2) as u32; let replication_streaming_threshold_bytes = @@ -121,20 +153,23 @@ impl ServerConfig { let site_sync_enabled = std::env::var("SITE_SYNC_ENABLED") .unwrap_or_else(|_| "false".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; let site_sync_interval_secs = parse_u64_env("SITE_SYNC_INTERVAL_SECONDS", 60); let site_sync_batch_size = parse_u64_env("SITE_SYNC_BATCH_SIZE", 100) as usize; let site_sync_connect_timeout_secs = parse_u64_env("SITE_SYNC_CONNECT_TIMEOUT_SECONDS", 10); let site_sync_read_timeout_secs = parse_u64_env("SITE_SYNC_READ_TIMEOUT_SECONDS", 120); let site_sync_max_retries = parse_u64_env("SITE_SYNC_MAX_RETRIES", 2) as u32; - let site_sync_clock_skew_tolerance: f64 = std::env::var("SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS") - .ok() - .and_then(|s| s.parse().ok()) - .unwrap_or(1.0); + let site_sync_clock_skew_tolerance: f64 = + std::env::var("SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS") + .ok() + .and_then(|s| s.parse().ok()) + .unwrap_or(1.0); let ui_enabled = std::env::var("UI_ENABLED") .unwrap_or_else(|_| "true".to_string()) - .to_lowercase() == "true"; + .to_lowercase() + == "true"; let templates_dir = std::env::var("TEMPLATES_DIR") .map(PathBuf::from) .unwrap_or_else(|_| default_templates_dir()); @@ -142,8 +177,10 @@ impl ServerConfig { .map(PathBuf::from) .unwrap_or_else(|_| default_static_dir()); + let host_ip: std::net::IpAddr = host.parse().unwrap(); Self { - bind_addr: SocketAddr::new(host.parse().unwrap(), port), + bind_addr: SocketAddr::new(host_ip, port), + ui_bind_addr: SocketAddr::new(host_ip, ui_port), storage_root: storage_path, region, iam_config_path, @@ -156,6 +193,11 @@ impl ServerConfig { gc_enabled, integrity_enabled, metrics_enabled, + metrics_history_enabled, + metrics_interval_minutes, + metrics_retention_hours, + metrics_history_interval_minutes, + metrics_history_retention_hours, lifecycle_enabled, website_hosting_enabled, replication_connect_timeout_secs, diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/admin.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/admin.rs index e09f83f..8a08e2b 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/admin.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/admin.rs @@ -26,15 +26,31 @@ fn json_error(code: &str, message: &str, status: StatusCode) -> Response { ) } +fn push_issue(result: &mut serde_json::Value, issue: serde_json::Value) { + if let Some(items) = result + .get_mut("issues") + .and_then(|value| value.as_array_mut()) + { + items.push(issue); + } +} + fn require_admin(principal: &Principal) -> Option { if !principal.is_admin { - return Some(json_error("AccessDenied", "Admin access required", StatusCode::FORBIDDEN)); + return Some(json_error( + "AccessDenied", + "Admin access required", + StatusCode::FORBIDDEN, + )); } None } async fn read_json_body(body: Body) -> Option { - let bytes = http_body_util::BodyExt::collect(body).await.ok()?.to_bytes(); + let bytes = http_body_util::BodyExt::collect(body) + .await + .ok()? + .to_bytes(); serde_json::from_slice(&bytes).ok() } @@ -46,7 +62,10 @@ fn validate_site_id(site_id: &str) -> Option { if !first.is_ascii_alphanumeric() { return Some("site_id must start with alphanumeric".to_string()); } - if !site_id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') { + if !site_id + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') + { return Some("site_id must contain only alphanumeric, hyphens, underscores".to_string()); } None @@ -78,7 +97,9 @@ pub async fn get_local_site( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } if let Some(ref registry) = state.site_registry { if let Some(local) = registry.get_local_site() { @@ -86,7 +107,11 @@ pub async fn get_local_site( } } - json_error("NotFound", "Local site not configured", StatusCode::NOT_FOUND) + json_error( + "NotFound", + "Local site not configured", + StatusCode::NOT_FOUND, + ) } pub async fn update_local_site( @@ -94,27 +119,51 @@ pub async fn update_local_site( Extension(principal): Extension, body: Body, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_error("InvalidRequest", "Site registry not available", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Site registry not available", + StatusCode::BAD_REQUEST, + ) + } }; let payload = match read_json_body(body).await { Some(v) => v, - None => return json_error("MalformedJSON", "Invalid JSON body", StatusCode::BAD_REQUEST), + None => { + return json_error( + "MalformedJSON", + "Invalid JSON body", + StatusCode::BAD_REQUEST, + ) + } }; let site_id = match payload.get("site_id").and_then(|v| v.as_str()) { Some(s) => s.to_string(), - None => return json_error("ValidationError", "site_id is required", StatusCode::BAD_REQUEST), + None => { + return json_error( + "ValidationError", + "site_id is required", + StatusCode::BAD_REQUEST, + ) + } }; if let Some(err) = validate_site_id(&site_id) { return json_error("ValidationError", &err, StatusCode::BAD_REQUEST); } - let endpoint = payload.get("endpoint").and_then(|v| v.as_str()).unwrap_or("").to_string(); + let endpoint = payload + .get("endpoint") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); if !endpoint.is_empty() { if let Some(err) = validate_endpoint(&endpoint) { return json_error("ValidationError", &err, StatusCode::BAD_REQUEST); @@ -137,9 +186,20 @@ pub async fn update_local_site( let site = SiteInfo { site_id: site_id.clone(), endpoint, - region: payload.get("region").and_then(|v| v.as_str()).unwrap_or("us-east-1").to_string(), - priority: payload.get("priority").and_then(|v| v.as_i64()).unwrap_or(100) as i32, - display_name: payload.get("display_name").and_then(|v| v.as_str()).unwrap_or(&site_id).to_string(), + region: payload + .get("region") + .and_then(|v| v.as_str()) + .unwrap_or("us-east-1") + .to_string(), + priority: payload + .get("priority") + .and_then(|v| v.as_i64()) + .unwrap_or(100) as i32, + display_name: payload + .get("display_name") + .and_then(|v| v.as_str()) + .unwrap_or(&site_id) + .to_string(), created_at: existing.and_then(|e| e.created_at), }; @@ -151,20 +211,30 @@ pub async fn list_all_sites( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_response(StatusCode::OK, serde_json::json!({"local": null, "peers": [], "total_peers": 0})), + None => { + return json_response( + StatusCode::OK, + serde_json::json!({"local": null, "peers": [], "total_peers": 0}), + ) + } }; let local = registry.get_local_site(); let peers = registry.list_peers(); - json_response(StatusCode::OK, serde_json::json!({ - "local": local, - "peers": peers, - "total_peers": peers.len(), - })) + json_response( + StatusCode::OK, + serde_json::json!({ + "local": local, + "peers": peers, + "total_peers": peers.len(), + }), + ) } pub async fn register_peer_site( @@ -172,20 +242,40 @@ pub async fn register_peer_site( Extension(principal): Extension, body: Body, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_error("InvalidRequest", "Site registry not available", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Site registry not available", + StatusCode::BAD_REQUEST, + ) + } }; let payload = match read_json_body(body).await { Some(v) => v, - None => return json_error("MalformedJSON", "Invalid JSON body", StatusCode::BAD_REQUEST), + None => { + return json_error( + "MalformedJSON", + "Invalid JSON body", + StatusCode::BAD_REQUEST, + ) + } }; let site_id = match payload.get("site_id").and_then(|v| v.as_str()) { Some(s) => s.to_string(), - None => return json_error("ValidationError", "site_id is required", StatusCode::BAD_REQUEST), + None => { + return json_error( + "ValidationError", + "site_id is required", + StatusCode::BAD_REQUEST, + ) + } }; if let Some(err) = validate_site_id(&site_id) { return json_error("ValidationError", &err, StatusCode::BAD_REQUEST); @@ -193,24 +283,41 @@ pub async fn register_peer_site( let endpoint = match payload.get("endpoint").and_then(|v| v.as_str()) { Some(e) => e.to_string(), - None => return json_error("ValidationError", "endpoint is required", StatusCode::BAD_REQUEST), + None => { + return json_error( + "ValidationError", + "endpoint is required", + StatusCode::BAD_REQUEST, + ) + } }; if let Some(err) = validate_endpoint(&endpoint) { return json_error("ValidationError", &err, StatusCode::BAD_REQUEST); } - let region = payload.get("region").and_then(|v| v.as_str()).unwrap_or("us-east-1").to_string(); + let region = payload + .get("region") + .and_then(|v| v.as_str()) + .unwrap_or("us-east-1") + .to_string(); if let Some(err) = validate_region(®ion) { return json_error("ValidationError", &err, StatusCode::BAD_REQUEST); } - let priority = payload.get("priority").and_then(|v| v.as_i64()).unwrap_or(100); + let priority = payload + .get("priority") + .and_then(|v| v.as_i64()) + .unwrap_or(100); if let Some(err) = validate_priority(priority) { return json_error("ValidationError", &err, StatusCode::BAD_REQUEST); } if registry.get_peer(&site_id).is_some() { - return json_error("AlreadyExists", &format!("Peer site '{}' already exists", site_id), StatusCode::CONFLICT); + return json_error( + "AlreadyExists", + &format!("Peer site '{}' already exists", site_id), + StatusCode::CONFLICT, + ); } let peer = PeerSite { @@ -218,8 +325,15 @@ pub async fn register_peer_site( endpoint, region, priority: priority as i32, - display_name: payload.get("display_name").and_then(|v| v.as_str()).unwrap_or(&site_id).to_string(), - connection_id: payload.get("connection_id").and_then(|v| v.as_str()).map(|s| s.to_string()), + display_name: payload + .get("display_name") + .and_then(|v| v.as_str()) + .unwrap_or(&site_id) + .to_string(), + connection_id: payload + .get("connection_id") + .and_then(|v| v.as_str()) + .map(|s| s.to_string()), created_at: Some(chrono::Utc::now().to_rfc3339()), is_healthy: false, last_health_check: None, @@ -234,15 +348,27 @@ pub async fn get_peer_site( Extension(principal): Extension, Path(site_id): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_error("NotFound", "Site registry not available", StatusCode::NOT_FOUND), + None => { + return json_error( + "NotFound", + "Site registry not available", + StatusCode::NOT_FOUND, + ) + } }; match registry.get_peer(&site_id) { Some(peer) => json_response(StatusCode::OK, serde_json::to_value(&peer).unwrap()), - None => json_error("NotFound", &format!("Peer site '{}' not found", site_id), StatusCode::NOT_FOUND), + None => json_error( + "NotFound", + &format!("Peer site '{}' not found", site_id), + StatusCode::NOT_FOUND, + ), } } @@ -252,20 +378,40 @@ pub async fn update_peer_site( Path(site_id): Path, body: Body, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_error("NotFound", "Site registry not available", StatusCode::NOT_FOUND), + None => { + return json_error( + "NotFound", + "Site registry not available", + StatusCode::NOT_FOUND, + ) + } }; let existing = match registry.get_peer(&site_id) { Some(p) => p, - None => return json_error("NotFound", &format!("Peer site '{}' not found", site_id), StatusCode::NOT_FOUND), + None => { + return json_error( + "NotFound", + &format!("Peer site '{}' not found", site_id), + StatusCode::NOT_FOUND, + ) + } }; let payload = match read_json_body(body).await { Some(v) => v, - None => return json_error("MalformedJSON", "Invalid JSON body", StatusCode::BAD_REQUEST), + None => { + return json_error( + "MalformedJSON", + "Invalid JSON body", + StatusCode::BAD_REQUEST, + ) + } }; if let Some(ep) = payload.get("endpoint").and_then(|v| v.as_str()) { @@ -286,11 +432,30 @@ pub async fn update_peer_site( let peer = PeerSite { site_id: site_id.clone(), - endpoint: payload.get("endpoint").and_then(|v| v.as_str()).unwrap_or(&existing.endpoint).to_string(), - region: payload.get("region").and_then(|v| v.as_str()).unwrap_or(&existing.region).to_string(), - priority: payload.get("priority").and_then(|v| v.as_i64()).unwrap_or(existing.priority as i64) as i32, - display_name: payload.get("display_name").and_then(|v| v.as_str()).unwrap_or(&existing.display_name).to_string(), - connection_id: payload.get("connection_id").and_then(|v| v.as_str()).map(|s| s.to_string()).or(existing.connection_id), + endpoint: payload + .get("endpoint") + .and_then(|v| v.as_str()) + .unwrap_or(&existing.endpoint) + .to_string(), + region: payload + .get("region") + .and_then(|v| v.as_str()) + .unwrap_or(&existing.region) + .to_string(), + priority: payload + .get("priority") + .and_then(|v| v.as_i64()) + .unwrap_or(existing.priority as i64) as i32, + display_name: payload + .get("display_name") + .and_then(|v| v.as_str()) + .unwrap_or(&existing.display_name) + .to_string(), + connection_id: payload + .get("connection_id") + .and_then(|v| v.as_str()) + .map(|s| s.to_string()) + .or(existing.connection_id), created_at: existing.created_at, is_healthy: existing.is_healthy, last_health_check: existing.last_health_check, @@ -305,14 +470,26 @@ pub async fn delete_peer_site( Extension(principal): Extension, Path(site_id): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_error("NotFound", "Site registry not available", StatusCode::NOT_FOUND), + None => { + return json_error( + "NotFound", + "Site registry not available", + StatusCode::NOT_FOUND, + ) + } }; if !registry.delete_peer(&site_id) { - return json_error("NotFound", &format!("Peer site '{}' not found", site_id), StatusCode::NOT_FOUND); + return json_error( + "NotFound", + &format!("Peer site '{}' not found", site_id), + StatusCode::NOT_FOUND, + ); } StatusCode::NO_CONTENT.into_response() } @@ -322,32 +499,77 @@ pub async fn check_peer_health( Extension(principal): Extension, Path(site_id): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_error("NotFound", "Site registry not available", StatusCode::NOT_FOUND), + None => { + return json_error( + "NotFound", + "Site registry not available", + StatusCode::NOT_FOUND, + ) + } }; if registry.get_peer(&site_id).is_none() { - return json_error("NotFound", &format!("Peer site '{}' not found", site_id), StatusCode::NOT_FOUND); + return json_error( + "NotFound", + &format!("Peer site '{}' not found", site_id), + StatusCode::NOT_FOUND, + ); } - json_response(StatusCode::OK, serde_json::json!({ - "site_id": site_id, - "is_healthy": false, - "error": "Health check not implemented in standalone mode", - "checked_at": chrono::Utc::now().timestamp_millis() as f64 / 1000.0, - })) + let peer = registry.get_peer(&site_id).unwrap(); + let checked_at = chrono::Utc::now().timestamp_millis() as f64 / 1000.0; + let mut is_healthy = false; + let mut error: Option = None; + + if let Some(connection_id) = peer.connection_id.as_deref() { + if let Some(connection) = state.connections.get(connection_id) { + is_healthy = state.replication.check_endpoint(&connection).await; + if !is_healthy { + error = Some(format!( + "Cannot reach endpoint: {}", + connection.endpoint_url + )); + } + } else { + error = Some(format!("Connection '{}' not found", connection_id)); + } + } else { + error = Some("No connection configured for this peer".to_string()); + } + + registry.update_health(&site_id, is_healthy); + + json_response( + StatusCode::OK, + serde_json::json!({ + "site_id": site_id, + "is_healthy": is_healthy, + "error": error, + "checked_at": checked_at, + }), + ) } pub async fn get_topology( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_response(StatusCode::OK, serde_json::json!({"sites": [], "total": 0, "healthy_count": 0})), + None => { + return json_response( + StatusCode::OK, + serde_json::json!({"sites": [], "total": 0, "healthy_count": 0}), + ) + } }; let local = registry.get_local_site(); @@ -356,25 +578,41 @@ pub async fn get_topology( let mut sites: Vec = Vec::new(); if let Some(l) = local { let mut v = serde_json::to_value(&l).unwrap(); - v.as_object_mut().unwrap().insert("is_local".to_string(), serde_json::json!(true)); - v.as_object_mut().unwrap().insert("is_healthy".to_string(), serde_json::json!(true)); + v.as_object_mut() + .unwrap() + .insert("is_local".to_string(), serde_json::json!(true)); + v.as_object_mut() + .unwrap() + .insert("is_healthy".to_string(), serde_json::json!(true)); sites.push(v); } for p in &peers { let mut v = serde_json::to_value(p).unwrap(); - v.as_object_mut().unwrap().insert("is_local".to_string(), serde_json::json!(false)); + v.as_object_mut() + .unwrap() + .insert("is_local".to_string(), serde_json::json!(false)); sites.push(v); } sites.sort_by_key(|s| s.get("priority").and_then(|v| v.as_i64()).unwrap_or(100)); - let healthy_count = sites.iter().filter(|s| s.get("is_healthy").and_then(|v| v.as_bool()).unwrap_or(false)).count(); + let healthy_count = sites + .iter() + .filter(|s| { + s.get("is_healthy") + .and_then(|v| v.as_bool()) + .unwrap_or(false) + }) + .count(); - json_response(StatusCode::OK, serde_json::json!({ - "sites": sites, - "total": sites.len(), - "healthy_count": healthy_count, - })) + json_response( + StatusCode::OK, + serde_json::json!({ + "sites": sites, + "total": sites.len(), + "healthy_count": healthy_count, + }), + ) } pub async fn check_bidirectional_status( @@ -382,34 +620,315 @@ pub async fn check_bidirectional_status( Extension(principal): Extension, Path(site_id): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let registry = match &state.site_registry { Some(r) => r, - None => return json_error("NotFound", "Site registry not available", StatusCode::NOT_FOUND), + None => { + return json_error( + "NotFound", + "Site registry not available", + StatusCode::NOT_FOUND, + ) + } }; if registry.get_peer(&site_id).is_none() { - return json_error("NotFound", &format!("Peer site '{}' not found", site_id), StatusCode::NOT_FOUND); + return json_error( + "NotFound", + &format!("Peer site '{}' not found", site_id), + StatusCode::NOT_FOUND, + ); } let local = registry.get_local_site(); - json_response(StatusCode::OK, serde_json::json!({ + let peer = registry.get_peer(&site_id).unwrap(); + let local_bidirectional_rules: Vec = state + .replication + .list_rules() + .into_iter() + .filter(|rule| { + peer.connection_id + .as_deref() + .map(|connection_id| rule.target_connection_id == connection_id) + .unwrap_or(false) + && rule.mode == crate::services::replication::MODE_BIDIRECTIONAL + }) + .map(|rule| { + serde_json::json!({ + "bucket_name": rule.bucket_name, + "target_bucket": rule.target_bucket, + "enabled": rule.enabled, + }) + }) + .collect(); + + let mut result = serde_json::json!({ "site_id": site_id, - "local_site_id": local.as_ref().map(|l| &l.site_id), - "local_endpoint": local.as_ref().map(|l| &l.endpoint), - "local_bidirectional_rules": [], - "local_site_sync_enabled": false, + "local_site_id": local.as_ref().map(|l| l.site_id.clone()), + "local_endpoint": local.as_ref().map(|l| l.endpoint.clone()), + "local_bidirectional_rules": local_bidirectional_rules, + "local_site_sync_enabled": state.config.site_sync_enabled, "remote_status": null, - "issues": [{"code": "NOT_IMPLEMENTED", "message": "Bidirectional status check not implemented in standalone mode", "severity": "warning"}], + "issues": Vec::::new(), "is_fully_configured": false, - })) + }); + + if local + .as_ref() + .map(|site| site.site_id.trim().is_empty()) + .unwrap_or(true) + { + push_issue( + &mut result, + serde_json::json!({ + "code": "NO_LOCAL_SITE_ID", + "message": "Local site identity not configured", + "severity": "error", + }), + ); + } + if local + .as_ref() + .map(|site| site.endpoint.trim().is_empty()) + .unwrap_or(true) + { + push_issue( + &mut result, + serde_json::json!({ + "code": "NO_LOCAL_ENDPOINT", + "message": "Local site endpoint not configured (remote site cannot reach back)", + "severity": "error", + }), + ); + } + + let Some(connection_id) = peer.connection_id.as_deref() else { + push_issue( + &mut result, + serde_json::json!({ + "code": "NO_CONNECTION", + "message": "No connection configured for this peer", + "severity": "error", + }), + ); + return json_response(StatusCode::OK, result); + }; + + let Some(connection) = state.connections.get(connection_id) else { + push_issue( + &mut result, + serde_json::json!({ + "code": "CONNECTION_NOT_FOUND", + "message": format!("Connection '{}' not found", connection_id), + "severity": "error", + }), + ); + return json_response(StatusCode::OK, result); + }; + + if result["local_bidirectional_rules"] + .as_array() + .map(|rules| rules.is_empty()) + .unwrap_or(true) + { + push_issue( + &mut result, + serde_json::json!({ + "code": "NO_LOCAL_BIDIRECTIONAL_RULES", + "message": "No bidirectional replication rules configured on this site", + "severity": "warning", + }), + ); + } + if !state.config.site_sync_enabled { + push_issue( + &mut result, + serde_json::json!({ + "code": "SITE_SYNC_DISABLED", + "message": "Site sync worker is disabled (SITE_SYNC_ENABLED=false). Pull operations will not work.", + "severity": "warning", + }), + ); + } + if !state.replication.check_endpoint(&connection).await { + push_issue( + &mut result, + serde_json::json!({ + "code": "REMOTE_UNREACHABLE", + "message": "Remote endpoint is not reachable", + "severity": "error", + }), + ); + return json_response(StatusCode::OK, result); + } + + let admin_url = format!( + "{}/admin/sites", + connection.endpoint_url.trim_end_matches('/') + ); + match reqwest::Client::new() + .get(&admin_url) + .header("accept", "application/json") + .header("x-access-key", &connection.access_key) + .header("x-secret-key", &connection.secret_key) + .timeout(std::time::Duration::from_secs(10)) + .send() + .await + { + Ok(resp) if resp.status().is_success() => match resp.json::().await { + Ok(remote_data) => { + let remote_local = remote_data + .get("local") + .cloned() + .unwrap_or(serde_json::Value::Null); + let remote_peers = remote_data + .get("peers") + .and_then(|value| value.as_array()) + .cloned() + .unwrap_or_default(); + let mut has_peer_for_us = false; + let mut peer_connection_configured = false; + + for remote_peer in &remote_peers { + let matches_site = local + .as_ref() + .map(|site| { + remote_peer.get("site_id").and_then(|v| v.as_str()) + == Some(site.site_id.as_str()) + || remote_peer.get("endpoint").and_then(|v| v.as_str()) + == Some(site.endpoint.as_str()) + }) + .unwrap_or(false); + if matches_site { + has_peer_for_us = true; + peer_connection_configured = remote_peer + .get("connection_id") + .and_then(|v| v.as_str()) + .map(|v| !v.trim().is_empty()) + .unwrap_or(false); + break; + } + } + + result["remote_status"] = serde_json::json!({ + "reachable": true, + "local_site": remote_local, + "site_sync_enabled": serde_json::Value::Null, + "has_peer_for_us": has_peer_for_us, + "peer_connection_configured": peer_connection_configured, + "has_bidirectional_rules_for_us": serde_json::Value::Null, + }); + + if !has_peer_for_us { + push_issue( + &mut result, + serde_json::json!({ + "code": "REMOTE_NO_PEER_FOR_US", + "message": "Remote site does not have this site registered as a peer", + "severity": "error", + }), + ); + } else if !peer_connection_configured { + push_issue( + &mut result, + serde_json::json!({ + "code": "REMOTE_NO_CONNECTION_FOR_US", + "message": "Remote site has us as peer but no connection configured (cannot push back)", + "severity": "error", + }), + ); + } + } + Err(_) => { + result["remote_status"] = serde_json::json!({ + "reachable": true, + "invalid_response": true, + }); + push_issue( + &mut result, + serde_json::json!({ + "code": "REMOTE_INVALID_RESPONSE", + "message": "Remote admin API returned invalid JSON", + "severity": "warning", + }), + ); + } + }, + Ok(resp) + if resp.status() == StatusCode::UNAUTHORIZED + || resp.status() == StatusCode::FORBIDDEN => + { + result["remote_status"] = serde_json::json!({ + "reachable": true, + "admin_access_denied": true, + }); + push_issue( + &mut result, + serde_json::json!({ + "code": "REMOTE_ADMIN_ACCESS_DENIED", + "message": "Cannot verify remote configuration (admin access denied)", + "severity": "warning", + }), + ); + } + Ok(resp) => { + result["remote_status"] = serde_json::json!({ + "reachable": true, + "admin_api_error": resp.status().as_u16(), + }); + push_issue( + &mut result, + serde_json::json!({ + "code": "REMOTE_ADMIN_API_ERROR", + "message": format!("Remote admin API returned status {}", resp.status().as_u16()), + "severity": "warning", + }), + ); + } + Err(_) => { + result["remote_status"] = serde_json::json!({ + "reachable": false, + "error": "Connection failed", + }); + push_issue( + &mut result, + serde_json::json!({ + "code": "REMOTE_ADMIN_UNREACHABLE", + "message": "Could not reach remote admin API", + "severity": "warning", + }), + ); + } + } + + let has_errors = result["issues"] + .as_array() + .map(|items| { + items.iter().any(|issue| { + issue.get("severity").and_then(|value| value.as_str()) == Some("error") + }) + }) + .unwrap_or(true); + result["is_fully_configured"] = serde_json::json!( + !has_errors + && result["local_bidirectional_rules"] + .as_array() + .map(|rules| !rules.is_empty()) + .unwrap_or(false) + ); + + json_response(StatusCode::OK, result) } pub async fn iam_list_users( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let users = state.iam.list_users().await; json_response(StatusCode::OK, serde_json::json!({"users": users})) } @@ -419,10 +938,16 @@ pub async fn iam_get_user( Extension(principal): Extension, Path(identifier): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match state.iam.get_user(&identifier).await { Some(user) => json_response(StatusCode::OK, user), - None => json_error("NotFound", &format!("User '{}' not found", identifier), StatusCode::NOT_FOUND), + None => json_error( + "NotFound", + &format!("User '{}' not found", identifier), + StatusCode::NOT_FOUND, + ), } } @@ -431,10 +956,16 @@ pub async fn iam_get_user_policies( Extension(principal): Extension, Path(identifier): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match state.iam.get_user_policies(&identifier) { Some(policies) => json_response(StatusCode::OK, serde_json::json!({"policies": policies})), - None => json_error("NotFound", &format!("User '{}' not found", identifier), StatusCode::NOT_FOUND), + None => json_error( + "NotFound", + &format!("User '{}' not found", identifier), + StatusCode::NOT_FOUND, + ), } } @@ -443,7 +974,9 @@ pub async fn iam_create_access_key( Extension(principal): Extension, Path(identifier): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match state.iam.create_access_key(&identifier) { Ok(result) => json_response(StatusCode::CREATED, result), Err(e) => json_error("InvalidRequest", &e, StatusCode::BAD_REQUEST), @@ -455,7 +988,9 @@ pub async fn iam_delete_access_key( Extension(principal): Extension, Path((_identifier, access_key)): Path<(String, String)>, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match state.iam.delete_access_key(&access_key) { Ok(()) => StatusCode::NO_CONTENT.into_response(), Err(e) => json_error("InvalidRequest", &e, StatusCode::BAD_REQUEST), @@ -467,7 +1002,9 @@ pub async fn iam_disable_user( Extension(principal): Extension, Path(identifier): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match state.iam.set_user_enabled(&identifier, false).await { Ok(()) => json_response(StatusCode::OK, serde_json::json!({"status": "disabled"})), Err(e) => json_error("InvalidRequest", &e, StatusCode::BAD_REQUEST), @@ -479,7 +1016,9 @@ pub async fn iam_enable_user( Extension(principal): Extension, Path(identifier): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match state.iam.set_user_enabled(&identifier, true).await { Ok(()) => json_response(StatusCode::OK, serde_json::json!({"status": "enabled"})), Err(e) => json_error("InvalidRequest", &e, StatusCode::BAD_REQUEST), @@ -490,10 +1029,18 @@ pub async fn list_website_domains( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let store = match &state.website_domains { Some(s) => s, - None => return json_error("InvalidRequest", "Website hosting is not enabled", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Website hosting is not enabled", + StatusCode::BAD_REQUEST, + ) + } }; json_response(StatusCode::OK, serde_json::json!(store.list_all())) } @@ -503,41 +1050,85 @@ pub async fn create_website_domain( Extension(principal): Extension, body: Body, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let store = match &state.website_domains { Some(s) => s, - None => return json_error("InvalidRequest", "Website hosting is not enabled", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Website hosting is not enabled", + StatusCode::BAD_REQUEST, + ) + } }; let payload = match read_json_body(body).await { Some(v) => v, - None => return json_error("MalformedJSON", "Invalid JSON body", StatusCode::BAD_REQUEST), + None => { + return json_error( + "MalformedJSON", + "Invalid JSON body", + StatusCode::BAD_REQUEST, + ) + } }; let domain = normalize_domain(payload.get("domain").and_then(|v| v.as_str()).unwrap_or("")); if domain.is_empty() { - return json_error("ValidationError", "domain is required", StatusCode::BAD_REQUEST); + return json_error( + "ValidationError", + "domain is required", + StatusCode::BAD_REQUEST, + ); } if !is_valid_domain(&domain) { - return json_error("ValidationError", &format!("Invalid domain: '{}'", domain), StatusCode::BAD_REQUEST); + return json_error( + "ValidationError", + &format!("Invalid domain: '{}'", domain), + StatusCode::BAD_REQUEST, + ); } - let bucket = payload.get("bucket").and_then(|v| v.as_str()).unwrap_or("").trim().to_string(); + let bucket = payload + .get("bucket") + .and_then(|v| v.as_str()) + .unwrap_or("") + .trim() + .to_string(); if bucket.is_empty() { - return json_error("ValidationError", "bucket is required", StatusCode::BAD_REQUEST); + return json_error( + "ValidationError", + "bucket is required", + StatusCode::BAD_REQUEST, + ); } match state.storage.bucket_exists(&bucket).await { Ok(true) => {} - _ => return json_error("NoSuchBucket", &format!("Bucket '{}' does not exist", bucket), StatusCode::NOT_FOUND), + _ => { + return json_error( + "NoSuchBucket", + &format!("Bucket '{}' does not exist", bucket), + StatusCode::NOT_FOUND, + ) + } } if store.get_bucket(&domain).is_some() { - return json_error("Conflict", &format!("Domain '{}' is already mapped", domain), StatusCode::CONFLICT); + return json_error( + "Conflict", + &format!("Domain '{}' is already mapped", domain), + StatusCode::CONFLICT, + ); } store.set_mapping(&domain, &bucket); - json_response(StatusCode::CREATED, serde_json::json!({"domain": domain, "bucket": bucket})) + json_response( + StatusCode::CREATED, + serde_json::json!({"domain": domain, "bucket": bucket}), + ) } pub async fn get_website_domain( @@ -545,16 +1136,31 @@ pub async fn get_website_domain( Extension(principal): Extension, Path(domain): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let store = match &state.website_domains { Some(s) => s, - None => return json_error("InvalidRequest", "Website hosting is not enabled", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Website hosting is not enabled", + StatusCode::BAD_REQUEST, + ) + } }; let domain = normalize_domain(&domain); match store.get_bucket(&domain) { - Some(bucket) => json_response(StatusCode::OK, serde_json::json!({"domain": domain, "bucket": bucket})), - None => json_error("NotFound", &format!("No mapping found for domain '{}'", domain), StatusCode::NOT_FOUND), + Some(bucket) => json_response( + StatusCode::OK, + serde_json::json!({"domain": domain, "bucket": bucket}), + ), + None => json_error( + "NotFound", + &format!("No mapping found for domain '{}'", domain), + StatusCode::NOT_FOUND, + ), } } @@ -564,34 +1170,70 @@ pub async fn update_website_domain( Path(domain): Path, body: Body, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let store = match &state.website_domains { Some(s) => s, - None => return json_error("InvalidRequest", "Website hosting is not enabled", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Website hosting is not enabled", + StatusCode::BAD_REQUEST, + ) + } }; let domain = normalize_domain(&domain); let payload = match read_json_body(body).await { Some(v) => v, - None => return json_error("MalformedJSON", "Invalid JSON body", StatusCode::BAD_REQUEST), + None => { + return json_error( + "MalformedJSON", + "Invalid JSON body", + StatusCode::BAD_REQUEST, + ) + } }; - let bucket = payload.get("bucket").and_then(|v| v.as_str()).unwrap_or("").trim().to_string(); + let bucket = payload + .get("bucket") + .and_then(|v| v.as_str()) + .unwrap_or("") + .trim() + .to_string(); if bucket.is_empty() { - return json_error("ValidationError", "bucket is required", StatusCode::BAD_REQUEST); + return json_error( + "ValidationError", + "bucket is required", + StatusCode::BAD_REQUEST, + ); } match state.storage.bucket_exists(&bucket).await { Ok(true) => {} - _ => return json_error("NoSuchBucket", &format!("Bucket '{}' does not exist", bucket), StatusCode::NOT_FOUND), + _ => { + return json_error( + "NoSuchBucket", + &format!("Bucket '{}' does not exist", bucket), + StatusCode::NOT_FOUND, + ) + } } if store.get_bucket(&domain).is_none() { - return json_error("NotFound", &format!("No mapping found for domain '{}'", domain), StatusCode::NOT_FOUND); + return json_error( + "NotFound", + &format!("No mapping found for domain '{}'", domain), + StatusCode::NOT_FOUND, + ); } store.set_mapping(&domain, &bucket); - json_response(StatusCode::OK, serde_json::json!({"domain": domain, "bucket": bucket})) + json_response( + StatusCode::OK, + serde_json::json!({"domain": domain, "bucket": bucket}), + ) } pub async fn delete_website_domain( @@ -599,15 +1241,27 @@ pub async fn delete_website_domain( Extension(principal): Extension, Path(domain): Path, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let store = match &state.website_domains { Some(s) => s, - None => return json_error("InvalidRequest", "Website hosting is not enabled", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Website hosting is not enabled", + StatusCode::BAD_REQUEST, + ) + } }; let domain = normalize_domain(&domain); if !store.delete_mapping(&domain) { - return json_error("NotFound", &format!("No mapping found for domain '{}'", domain), StatusCode::NOT_FOUND); + return json_error( + "NotFound", + &format!("No mapping found for domain '{}'", domain), + StatusCode::NOT_FOUND, + ); } StatusCode::NO_CONTENT.into_response() } @@ -622,10 +1276,15 @@ pub async fn gc_status( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match &state.gc { Some(gc) => json_response(StatusCode::OK, gc.status().await), - None => json_response(StatusCode::OK, serde_json::json!({"enabled": false, "message": "GC is not enabled. Set GC_ENABLED=true to enable."})), + None => json_response( + StatusCode::OK, + serde_json::json!({"enabled": false, "message": "GC is not enabled. Set GC_ENABLED=true to enable."}), + ), } } @@ -634,14 +1293,25 @@ pub async fn gc_run( Extension(principal): Extension, body: Body, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let gc = match &state.gc { Some(gc) => gc, - None => return json_error("InvalidRequest", "GC is not enabled", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "GC is not enabled", + StatusCode::BAD_REQUEST, + ) + } }; let payload = read_json_body(body).await.unwrap_or(serde_json::json!({})); - let dry_run = payload.get("dry_run").and_then(|v| v.as_bool()).unwrap_or(false); + let dry_run = payload + .get("dry_run") + .and_then(|v| v.as_bool()) + .unwrap_or(false); match gc.run_now(dry_run).await { Ok(result) => json_response(StatusCode::OK, result), @@ -653,9 +1323,14 @@ pub async fn gc_history( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match &state.gc { - Some(gc) => json_response(StatusCode::OK, serde_json::json!({"executions": gc.history().await})), + Some(gc) => json_response( + StatusCode::OK, + serde_json::json!({"executions": gc.history().await}), + ), None => json_response(StatusCode::OK, serde_json::json!({"executions": []})), } } @@ -664,10 +1339,15 @@ pub async fn integrity_status( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match &state.integrity { Some(checker) => json_response(StatusCode::OK, checker.status().await), - None => json_response(StatusCode::OK, serde_json::json!({"enabled": false, "message": "Integrity checker is not enabled. Set INTEGRITY_ENABLED=true to enable."})), + None => json_response( + StatusCode::OK, + serde_json::json!({"enabled": false, "message": "Integrity checker is not enabled. Set INTEGRITY_ENABLED=true to enable."}), + ), } } @@ -676,15 +1356,29 @@ pub async fn integrity_run( Extension(principal): Extension, body: Body, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } let checker = match &state.integrity { Some(c) => c, - None => return json_error("InvalidRequest", "Integrity checker is not enabled", StatusCode::BAD_REQUEST), + None => { + return json_error( + "InvalidRequest", + "Integrity checker is not enabled", + StatusCode::BAD_REQUEST, + ) + } }; let payload = read_json_body(body).await.unwrap_or(serde_json::json!({})); - let dry_run = payload.get("dry_run").and_then(|v| v.as_bool()).unwrap_or(false); - let auto_heal = payload.get("auto_heal").and_then(|v| v.as_bool()).unwrap_or(false); + let dry_run = payload + .get("dry_run") + .and_then(|v| v.as_bool()) + .unwrap_or(false); + let auto_heal = payload + .get("auto_heal") + .and_then(|v| v.as_bool()) + .unwrap_or(false); match checker.run_now(dry_run, auto_heal).await { Ok(result) => json_response(StatusCode::OK, result), @@ -696,9 +1390,14 @@ pub async fn integrity_history( State(state): State, Extension(principal): Extension, ) -> Response { - if let Some(err) = require_admin(&principal) { return err; } + if let Some(err) = require_admin(&principal) { + return err; + } match &state.integrity { - Some(checker) => json_response(StatusCode::OK, serde_json::json!({"executions": checker.history().await})), + Some(checker) => json_response( + StatusCode::OK, + serde_json::json!({"executions": checker.history().await}), + ), None => json_response(StatusCode::OK, serde_json::json!({"executions": []})), } } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/chunked.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/chunked.rs index c061781..316dad1 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/chunked.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/chunked.rs @@ -41,7 +41,10 @@ impl AwsChunkedStream { fn parse_chunk_size(line: &[u8]) -> std::io::Result { let text = std::str::from_utf8(line).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidData, "invalid chunk size encoding") + std::io::Error::new( + std::io::ErrorKind::InvalidData, + "invalid chunk size encoding", + ) })?; let head = text.split(';').next().unwrap_or("").trim(); u64::from_str_radix(head, 16).map_err(|_| { @@ -179,4 +182,3 @@ pub fn decode_body(body: axum::body::Body) -> impl AsyncRead + Send + Unpin { ); AwsChunkedStream::new(stream) } - diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/config.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/config.rs index a28b3b5..971e55c 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/config.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/config.rs @@ -13,8 +13,14 @@ fn xml_response(status: StatusCode, xml: String) -> Response { fn storage_err(err: myfsio_storage::error::StorageError) -> Response { let s3err = S3Error::from(err); - let status = StatusCode::from_u16(s3err.http_status()).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); - (status, [("content-type", "application/xml")], s3err.to_xml()).into_response() + let status = + StatusCode::from_u16(s3err.http_status()).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); + ( + status, + [("content-type", "application/xml")], + s3err.to_xml(), + ) + .into_response() } fn json_response(status: StatusCode, value: serde_json::Value) -> Response { @@ -68,7 +74,7 @@ pub async fn get_tagging(state: &AppState, bucket: &str) -> Response { Ok(config) => { let mut xml = String::from( "\ - " + ", ); for tag in &config.tags { xml.push_str(&format!( @@ -130,7 +136,11 @@ pub async fn get_cors(state: &AppState, bucket: &str) -> Response { } else { xml_response( StatusCode::NOT_FOUND, - S3Error::new(S3ErrorCode::NoSuchKey, "The CORS configuration does not exist").to_xml(), + S3Error::new( + S3ErrorCode::NoSuchKey, + "The CORS configuration does not exist", + ) + .to_xml(), ) } } @@ -192,7 +202,8 @@ pub async fn get_encryption(state: &AppState, bucket: &str) -> Response { S3Error::new( S3ErrorCode::InvalidRequest, "The server side encryption configuration was not found", - ).to_xml(), + ) + .to_xml(), ) } } @@ -240,7 +251,11 @@ pub async fn get_lifecycle(state: &AppState, bucket: &str) -> Response { } else { xml_response( StatusCode::NOT_FOUND, - S3Error::new(S3ErrorCode::NoSuchKey, "The lifecycle configuration does not exist").to_xml(), + S3Error::new( + S3ErrorCode::NoSuchKey, + "The lifecycle configuration does not exist", + ) + .to_xml(), ) } } @@ -328,17 +343,17 @@ pub async fn put_quota(state: &AppState, bucket: &str, body: Body) -> Response { Err(_) => { return xml_response( StatusCode::BAD_REQUEST, - S3Error::new(S3ErrorCode::InvalidArgument, "Request body must be valid JSON").to_xml(), + S3Error::new( + S3ErrorCode::InvalidArgument, + "Request body must be valid JSON", + ) + .to_xml(), ); } }; - let max_size = payload - .get("max_size_bytes") - .and_then(|v| v.as_u64()); - let max_objects = payload - .get("max_objects") - .and_then(|v| v.as_u64()); + let max_size = payload.get("max_size_bytes").and_then(|v| v.as_u64()); + let max_objects = payload.get("max_objects").and_then(|v| v.as_u64()); if max_size.is_none() && max_objects.is_none() { return xml_response( @@ -603,7 +618,11 @@ pub async fn get_website(state: &AppState, bucket: &str) -> Response { } else { xml_response( StatusCode::NOT_FOUND, - S3Error::new(S3ErrorCode::NoSuchKey, "The website configuration does not exist").to_xml(), + S3Error::new( + S3ErrorCode::NoSuchKey, + "The website configuration does not exist", + ) + .to_xml(), ) } } @@ -677,19 +696,120 @@ pub async fn get_notification(state: &AppState, bucket: &str) -> Response { } pub async fn get_logging(state: &AppState, bucket: &str) -> Response { - match state.storage.get_bucket_config(bucket).await { - Ok(config) => { - if let Some(l) = &config.logging { - xml_response(StatusCode::OK, l.to_string()) - } else { - let xml = "\ - \ - "; - xml_response(StatusCode::OK, xml.to_string()) - } + match state.storage.bucket_exists(bucket).await { + Ok(true) => {} + Ok(false) => { + return storage_err(myfsio_storage::error::StorageError::BucketNotFound( + bucket.to_string(), + )) } - Err(e) => storage_err(e), + Err(e) => return storage_err(e), } + + let logging_config = if let Some(cfg) = state.access_logging.get(bucket) { + Some(cfg) + } else { + match state.storage.get_bucket_config(bucket).await { + Ok(config) => { + let legacy = legacy_logging_config(&config); + if let Some(cfg) = legacy.as_ref() { + if let Err(err) = state.access_logging.set(bucket, cfg.clone()) { + tracing::warn!( + "Failed to migrate legacy bucket logging config for {}: {}", + bucket, + err + ); + } + } + legacy + } + Err(e) => return storage_err(e), + } + }; + + let body = match logging_config { + Some(cfg) if cfg.enabled => format!( + "\ + \ + {}{}\ + ", + xml_escape(&cfg.target_bucket), + xml_escape(&cfg.target_prefix), + ), + _ => "\ + " + .to_string(), + }; + xml_response(StatusCode::OK, body) +} + +fn xml_escape(s: &str) -> String { + s.replace('&', "&") + .replace('<', "<") + .replace('>', ">") + .replace('"', """) + .replace('\'', "'") +} + +fn legacy_logging_config( + config: &myfsio_common::types::BucketConfig, +) -> Option { + let value = config.logging.as_ref()?; + match value { + serde_json::Value::String(xml) => parse_logging_config_xml(xml), + serde_json::Value::Object(_) => parse_logging_config_value(value.clone()), + _ => None, + } +} + +fn parse_logging_config_value( + value: serde_json::Value, +) -> Option { + let logging_enabled = value.get("LoggingEnabled")?; + let target_bucket = logging_enabled + .get("TargetBucket") + .and_then(|value| value.as_str()) + .map(str::trim) + .filter(|value| !value.is_empty())? + .to_string(); + let target_prefix = logging_enabled + .get("TargetPrefix") + .and_then(|value| value.as_str()) + .unwrap_or_default() + .to_string(); + Some(crate::services::access_logging::LoggingConfiguration { + target_bucket, + target_prefix, + enabled: true, + }) +} + +fn parse_logging_config_xml( + xml: &str, +) -> Option { + let doc = roxmltree::Document::parse(xml).ok()?; + let root = doc.root_element(); + let logging_enabled = root + .children() + .find(|n| n.is_element() && n.tag_name().name() == "LoggingEnabled")?; + let target_bucket = logging_enabled + .children() + .find(|n| n.is_element() && n.tag_name().name() == "TargetBucket") + .and_then(|n| n.text()) + .map(str::trim) + .filter(|value| !value.is_empty())? + .to_string(); + let target_prefix = logging_enabled + .children() + .find(|n| n.is_element() && n.tag_name().name() == "TargetPrefix") + .and_then(|n| n.text()) + .unwrap_or_default() + .to_string(); + Some(crate::services::access_logging::LoggingConfiguration { + target_bucket, + target_prefix, + enabled: true, + }) } pub async fn put_object_lock(state: &AppState, bucket: &str, body: Body) -> Response { @@ -757,35 +877,125 @@ pub async fn delete_notification(state: &AppState, bucket: &str) -> Response { } pub async fn put_logging(state: &AppState, bucket: &str, body: Body) -> Response { + match state.storage.bucket_exists(bucket).await { + Ok(true) => {} + Ok(false) => { + return storage_err(myfsio_storage::error::StorageError::BucketNotFound( + bucket.to_string(), + )) + } + Err(e) => return storage_err(e), + } + let body_bytes = match http_body_util::BodyExt::collect(body).await { Ok(collected) => collected.to_bytes(), Err(_) => return StatusCode::BAD_REQUEST.into_response(), }; - let value = serde_json::Value::String(String::from_utf8_lossy(&body_bytes).to_string()); - match state.storage.get_bucket_config(bucket).await { - Ok(mut config) => { - config.logging = Some(value); - match state.storage.set_bucket_config(bucket, &config).await { - Ok(()) => StatusCode::OK.into_response(), - Err(e) => storage_err(e), - } - } - Err(e) => storage_err(e), + if body_bytes.iter().all(u8::is_ascii_whitespace) { + state.access_logging.delete(bucket); + return StatusCode::OK.into_response(); } + + let xml = match std::str::from_utf8(&body_bytes) { + Ok(s) => s, + Err(_) => { + return s3_error_response( + S3ErrorCode::MalformedXML, + "Unable to parse XML document", + StatusCode::BAD_REQUEST, + ) + } + }; + + let doc = match roxmltree::Document::parse(xml) { + Ok(d) => d, + Err(_) => { + return s3_error_response( + S3ErrorCode::MalformedXML, + "Unable to parse XML document", + StatusCode::BAD_REQUEST, + ) + } + }; + + let root = doc.root_element(); + let logging_enabled = root + .children() + .find(|n| n.is_element() && n.tag_name().name() == "LoggingEnabled"); + + let Some(le) = logging_enabled else { + state.access_logging.delete(bucket); + return StatusCode::OK.into_response(); + }; + + let target_bucket = le + .children() + .find(|n| n.is_element() && n.tag_name().name() == "TargetBucket") + .and_then(|n| n.text()) + .map(str::trim) + .unwrap_or_default(); + + if target_bucket.is_empty() { + return s3_error_response( + S3ErrorCode::InvalidArgument, + "TargetBucket is required", + StatusCode::BAD_REQUEST, + ); + } + + let cfg = crate::services::access_logging::LoggingConfiguration { + target_bucket: target_bucket.to_string(), + target_prefix: le + .children() + .find(|n| n.is_element() && n.tag_name().name() == "TargetPrefix") + .and_then(|n| n.text()) + .unwrap_or_default() + .to_string(), + enabled: true, + }; + + match state.storage.bucket_exists(&cfg.target_bucket).await { + Ok(true) => {} + Ok(false) => { + return s3_error_response( + S3ErrorCode::InvalidArgument, + "Target bucket does not exist", + StatusCode::BAD_REQUEST, + ) + } + Err(e) => return storage_err(e), + } + + if let Err(e) = state.access_logging.set(bucket, cfg) { + tracing::error!( + "Failed to persist bucket logging config for {}: {}", + bucket, + e + ); + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + } + + StatusCode::OK.into_response() } pub async fn delete_logging(state: &AppState, bucket: &str) -> Response { - match state.storage.get_bucket_config(bucket).await { - Ok(mut config) => { - config.logging = None; - match state.storage.set_bucket_config(bucket, &config).await { - Ok(()) => StatusCode::NO_CONTENT.into_response(), - Err(e) => storage_err(e), - } + match state.storage.bucket_exists(bucket).await { + Ok(true) => {} + Ok(false) => { + return storage_err(myfsio_storage::error::StorageError::BucketNotFound( + bucket.to_string(), + )) } - Err(e) => storage_err(e), + Err(e) => return storage_err(e), } + state.access_logging.delete(bucket); + StatusCode::NO_CONTENT.into_response() +} + +fn s3_error_response(code: S3ErrorCode, message: &str, status: StatusCode) -> Response { + let err = S3Error::new(code, message.to_string()); + (status, [("content-type", "application/xml")], err.to_xml()).into_response() } pub async fn list_object_versions(state: &AppState, bucket: &str) -> Response { @@ -812,7 +1022,7 @@ pub async fn list_object_versions(state: &AppState, bucket: &str) -> Response { let mut xml = String::from( "\ - " + ", ); xml.push_str(&format!("{}", bucket)); @@ -842,7 +1052,7 @@ pub async fn get_object_tagging(state: &AppState, bucket: &str, key: &str) -> Re Ok(tags) => { let mut xml = String::from( "\ - " + ", ); for tag in &tags { xml.push_str(&format!( @@ -910,20 +1120,24 @@ pub async fn put_object_acl(state: &AppState, bucket: &str, key: &str, _body: Bo pub async fn get_object_retention(state: &AppState, bucket: &str, key: &str) -> Response { match state.storage.head_object(bucket, key).await { - Ok(_) => { - xml_response( - StatusCode::NOT_FOUND, - S3Error::new( - S3ErrorCode::InvalidRequest, - "No retention policy configured", - ).to_xml(), + Ok(_) => xml_response( + StatusCode::NOT_FOUND, + S3Error::new( + S3ErrorCode::InvalidRequest, + "No retention policy configured", ) - } + .to_xml(), + ), Err(e) => storage_err(e), } } -pub async fn put_object_retention(state: &AppState, bucket: &str, key: &str, _body: Body) -> Response { +pub async fn put_object_retention( + state: &AppState, + bucket: &str, + key: &str, + _body: Body, +) -> Response { match state.storage.head_object(bucket, key).await { Ok(_) => StatusCode::OK.into_response(), Err(e) => storage_err(e), @@ -942,13 +1156,68 @@ pub async fn get_object_legal_hold(state: &AppState, bucket: &str, key: &str) -> } } -pub async fn put_object_legal_hold(state: &AppState, bucket: &str, key: &str, _body: Body) -> Response { +pub async fn put_object_legal_hold( + state: &AppState, + bucket: &str, + key: &str, + _body: Body, +) -> Response { match state.storage.head_object(bucket, key).await { Ok(_) => StatusCode::OK.into_response(), Err(e) => storage_err(e), } } +#[cfg(test)] +mod tests { + use super::{legacy_logging_config, parse_logging_config_xml}; + use myfsio_common::types::BucketConfig; + + #[test] + fn parses_legacy_logging_xml_string() { + let mut config = BucketConfig::default(); + config.logging = Some(serde_json::Value::String( + "\ + \ + logsaudit/\ + " + .to_string(), + )); + + let parsed = legacy_logging_config(&config).expect("expected legacy logging config"); + assert_eq!(parsed.target_bucket, "logs"); + assert_eq!(parsed.target_prefix, "audit/"); + assert!(parsed.enabled); + } + + #[test] + fn parses_legacy_logging_json_object() { + let mut config = BucketConfig::default(); + config.logging = Some(serde_json::json!({ + "LoggingEnabled": { + "TargetBucket": "logs", + "TargetPrefix": "archive/" + } + })); + + let parsed = legacy_logging_config(&config).expect("expected legacy logging config"); + assert_eq!(parsed.target_bucket, "logs"); + assert_eq!(parsed.target_prefix, "archive/"); + assert!(parsed.enabled); + } + + #[test] + fn ignores_logging_xml_without_enabled_block() { + let parsed = parse_logging_config_xml( + "\ + \ + ", + ); + + assert!(parsed.is_none()); + } +} + fn parse_tagging_xml(xml: &str) -> Vec { let mut tags = Vec::new(); let mut in_tag = false; diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/kms.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/kms.rs index 0a4fba0..857382e 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/kms.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/kms.rs @@ -1,14 +1,17 @@ +use aes_gcm::aead::Aead; +use aes_gcm::{Aes256Gcm, KeyInit, Nonce}; use axum::body::Body; use axum::extract::State; use axum::http::StatusCode; use axum::response::{IntoResponse, Response}; use base64::engine::general_purpose::STANDARD as B64; use base64::Engine; -use serde_json::json; +use rand::RngCore; +use serde_json::{json, Value}; use crate::state::AppState; -fn json_ok(value: serde_json::Value) -> Response { +fn json_ok(value: Value) -> Response { ( StatusCode::OK, [("content-type", "application/json")], @@ -26,14 +29,54 @@ fn json_err(status: StatusCode, msg: &str) -> Response { .into_response() } +async fn read_json(body: Body) -> Result { + let body_bytes = http_body_util::BodyExt::collect(body) + .await + .map_err(|_| json_err(StatusCode::BAD_REQUEST, "Invalid request body"))? + .to_bytes(); + if body_bytes.is_empty() { + Ok(json!({})) + } else { + serde_json::from_slice(&body_bytes) + .map_err(|_| json_err(StatusCode::BAD_REQUEST, "Invalid JSON")) + } +} + +fn require_kms( + state: &AppState, +) -> Result<&std::sync::Arc, Response> { + state + .kms + .as_ref() + .ok_or_else(|| json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled")) +} + +fn decode_b64(value: &str, field: &str) -> Result, Response> { + B64.decode(value).map_err(|_| { + json_err( + StatusCode::BAD_REQUEST, + &format!("Invalid base64 {}", field), + ) + }) +} + +fn require_str<'a>(value: &'a Value, names: &[&str], message: &str) -> Result<&'a str, Response> { + for name in names { + if let Some(found) = value.get(*name).and_then(|v| v.as_str()) { + return Ok(found); + } + } + Err(json_err(StatusCode::BAD_REQUEST, message)) +} + pub async fn list_keys(State(state): State) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, }; let keys = kms.list_keys().await; - let keys_json: Vec = keys + let keys_json: Vec = keys .iter() .map(|k| { json!({ @@ -53,31 +96,22 @@ pub async fn list_keys(State(state): State) -> Response { } pub async fn create_key(State(state): State, body: Body) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, + }; + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, }; - let body_bytes = match http_body_util::BodyExt::collect(body).await { - Ok(c) => c.to_bytes(), - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid request body"), - }; + let description = req + .get("Description") + .or_else(|| req.get("description")) + .and_then(|d| d.as_str()) + .unwrap_or(""); - let description = if body_bytes.is_empty() { - String::new() - } else { - match serde_json::from_slice::(&body_bytes) { - Ok(v) => v - .get("Description") - .or_else(|| v.get("description")) - .and_then(|d| d.as_str()) - .unwrap_or("") - .to_string(), - Err(_) => String::new(), - } - }; - - match kms.create_key(&description).await { + match kms.create_key(description).await { Ok(key) => json_ok(json!({ "KeyId": key.key_id, "Arn": key.arn, @@ -94,9 +128,9 @@ pub async fn get_key( State(state): State, axum::extract::Path(key_id): axum::extract::Path, ) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, }; match kms.get_key(&key_id).await { @@ -118,9 +152,9 @@ pub async fn delete_key( State(state): State, axum::extract::Path(key_id): axum::extract::Path, ) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, }; match kms.delete_key(&key_id).await { @@ -134,9 +168,9 @@ pub async fn enable_key( State(state): State, axum::extract::Path(key_id): axum::extract::Path, ) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, }; match kms.enable_key(&key_id).await { @@ -150,9 +184,9 @@ pub async fn disable_key( State(state): State, axum::extract::Path(key_id): axum::extract::Path, ) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, }; match kms.disable_key(&key_id).await { @@ -163,32 +197,26 @@ pub async fn disable_key( } pub async fn encrypt(State(state): State, body: Body) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, + }; + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, }; - let body_bytes = match http_body_util::BodyExt::collect(body).await { - Ok(c) => c.to_bytes(), - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid request body"), + let key_id = match require_str(&req, &["KeyId", "key_id"], "Missing KeyId") { + Ok(value) => value, + Err(response) => return response, }; - - let req: serde_json::Value = match serde_json::from_slice(&body_bytes) { - Ok(v) => v, - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid JSON"), + let plaintext_b64 = match require_str(&req, &["Plaintext", "plaintext"], "Missing Plaintext") { + Ok(value) => value, + Err(response) => return response, }; - - let key_id = match req.get("KeyId").and_then(|v| v.as_str()) { - Some(k) => k, - None => return json_err(StatusCode::BAD_REQUEST, "Missing KeyId"), - }; - let plaintext_b64 = match req.get("Plaintext").and_then(|v| v.as_str()) { - Some(p) => p, - None => return json_err(StatusCode::BAD_REQUEST, "Missing Plaintext"), - }; - let plaintext = match B64.decode(plaintext_b64) { - Ok(p) => p, - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid base64 Plaintext"), + let plaintext = match decode_b64(plaintext_b64, "Plaintext") { + Ok(value) => value, + Err(response) => return response, }; match kms.encrypt_data(key_id, &plaintext).await { @@ -201,32 +229,30 @@ pub async fn encrypt(State(state): State, body: Body) -> Response { } pub async fn decrypt(State(state): State, body: Body) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, + }; + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, }; - let body_bytes = match http_body_util::BodyExt::collect(body).await { - Ok(c) => c.to_bytes(), - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid request body"), + let key_id = match require_str(&req, &["KeyId", "key_id"], "Missing KeyId") { + Ok(value) => value, + Err(response) => return response, }; - - let req: serde_json::Value = match serde_json::from_slice(&body_bytes) { - Ok(v) => v, - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid JSON"), + let ciphertext_b64 = match require_str( + &req, + &["CiphertextBlob", "ciphertext_blob"], + "Missing CiphertextBlob", + ) { + Ok(value) => value, + Err(response) => return response, }; - - let key_id = match req.get("KeyId").and_then(|v| v.as_str()) { - Some(k) => k, - None => return json_err(StatusCode::BAD_REQUEST, "Missing KeyId"), - }; - let ct_b64 = match req.get("CiphertextBlob").and_then(|v| v.as_str()) { - Some(c) => c, - None => return json_err(StatusCode::BAD_REQUEST, "Missing CiphertextBlob"), - }; - let ciphertext = match B64.decode(ct_b64) { - Ok(c) => c, - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid base64"), + let ciphertext = match decode_b64(ciphertext_b64, "CiphertextBlob") { + Ok(value) => value, + Err(response) => return response, }; match kms.decrypt_data(key_id, &ciphertext).await { @@ -239,39 +265,276 @@ pub async fn decrypt(State(state): State, body: Body) -> Response { } pub async fn generate_data_key(State(state): State, body: Body) -> Response { - let kms = match &state.kms { - Some(k) => k, - None => return json_err(StatusCode::SERVICE_UNAVAILABLE, "KMS not enabled"), + generate_data_key_inner(state, body, true).await +} + +pub async fn generate_data_key_without_plaintext( + State(state): State, + body: Body, +) -> Response { + generate_data_key_inner(state, body, false).await +} + +async fn generate_data_key_inner(state: AppState, body: Body, include_plaintext: bool) -> Response { + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, + }; + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, }; - let body_bytes = match http_body_util::BodyExt::collect(body).await { - Ok(c) => c.to_bytes(), - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid request body"), - }; - - let req: serde_json::Value = match serde_json::from_slice(&body_bytes) { - Ok(v) => v, - Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid JSON"), - }; - - let key_id = match req.get("KeyId").and_then(|v| v.as_str()) { - Some(k) => k, - None => return json_err(StatusCode::BAD_REQUEST, "Missing KeyId"), + let key_id = match require_str(&req, &["KeyId", "key_id"], "Missing KeyId") { + Ok(value) => value, + Err(response) => return response, }; let num_bytes = req .get("NumberOfBytes") .and_then(|v| v.as_u64()) .unwrap_or(32) as usize; - if num_bytes < 1 || num_bytes > 1024 { + if !(1..=1024).contains(&num_bytes) { return json_err(StatusCode::BAD_REQUEST, "NumberOfBytes must be 1-1024"); } match kms.generate_data_key(key_id, num_bytes).await { - Ok((plaintext, wrapped)) => json_ok(json!({ - "KeyId": key_id, - "Plaintext": B64.encode(&plaintext), - "CiphertextBlob": B64.encode(&wrapped), + Ok((plaintext, wrapped)) => { + let mut value = json!({ + "KeyId": key_id, + "CiphertextBlob": B64.encode(&wrapped), + }); + if include_plaintext { + value["Plaintext"] = json!(B64.encode(&plaintext)); + } + json_ok(value) + } + Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()), + } +} + +pub async fn re_encrypt(State(state): State, body: Body) -> Response { + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, + }; + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, + }; + + let ciphertext_b64 = match require_str( + &req, + &["CiphertextBlob", "ciphertext_blob"], + "CiphertextBlob is required", + ) { + Ok(value) => value, + Err(response) => return response, + }; + let destination_key_id = match require_str( + &req, + &["DestinationKeyId", "destination_key_id"], + "DestinationKeyId is required", + ) { + Ok(value) => value, + Err(response) => return response, + }; + let ciphertext = match decode_b64(ciphertext_b64, "CiphertextBlob") { + Ok(value) => value, + Err(response) => return response, + }; + + let keys = kms.list_keys().await; + let mut source_key_id: Option = None; + let mut plaintext: Option> = None; + for key in keys { + if !key.enabled { + continue; + } + if let Ok(value) = kms.decrypt_data(&key.key_id, &ciphertext).await { + source_key_id = Some(key.key_id); + plaintext = Some(value); + break; + } + } + + let Some(source_key_id) = source_key_id else { + return json_err( + StatusCode::BAD_REQUEST, + "Could not determine source key for CiphertextBlob", + ); + }; + let plaintext = plaintext.unwrap_or_default(); + + match kms.encrypt_data(destination_key_id, &plaintext).await { + Ok(new_ciphertext) => json_ok(json!({ + "CiphertextBlob": B64.encode(&new_ciphertext), + "SourceKeyId": source_key_id, + "KeyId": destination_key_id, + })), + Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()), + } +} + +pub async fn generate_random(State(state): State, body: Body) -> Response { + if let Err(response) = require_kms(&state) { + return response; + } + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, + }; + let num_bytes = req + .get("NumberOfBytes") + .and_then(|v| v.as_u64()) + .unwrap_or(32) as usize; + + if !(1..=1024).contains(&num_bytes) { + return json_err(StatusCode::BAD_REQUEST, "NumberOfBytes must be 1-1024"); + } + + let mut bytes = vec![0u8; num_bytes]; + rand::thread_rng().fill_bytes(&mut bytes); + json_ok(json!({ + "Plaintext": B64.encode(bytes), + })) +} + +pub async fn client_generate_key(State(state): State) -> Response { + let _ = state; + + let mut key = [0u8; 32]; + rand::thread_rng().fill_bytes(&mut key); + json_ok(json!({ + "Key": B64.encode(key), + "Algorithm": "AES-256-GCM", + "KeySize": 32, + })) +} + +pub async fn client_encrypt(State(state): State, body: Body) -> Response { + let _ = state; + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, + }; + let plaintext_b64 = + match require_str(&req, &["Plaintext", "plaintext"], "Plaintext is required") { + Ok(value) => value, + Err(response) => return response, + }; + let key_b64 = match require_str(&req, &["Key", "key"], "Key is required") { + Ok(value) => value, + Err(response) => return response, + }; + + let plaintext = match decode_b64(plaintext_b64, "Plaintext") { + Ok(value) => value, + Err(response) => return response, + }; + let key_bytes = match decode_b64(key_b64, "Key") { + Ok(value) => value, + Err(response) => return response, + }; + if key_bytes.len() != 32 { + return json_err(StatusCode::BAD_REQUEST, "Key must decode to 32 bytes"); + } + + let cipher = match Aes256Gcm::new_from_slice(&key_bytes) { + Ok(cipher) => cipher, + Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid encryption key"), + }; + let mut nonce_bytes = [0u8; 12]; + rand::thread_rng().fill_bytes(&mut nonce_bytes); + let nonce = Nonce::from_slice(&nonce_bytes); + + match cipher.encrypt(nonce, plaintext.as_ref()) { + Ok(ciphertext) => json_ok(json!({ + "Ciphertext": B64.encode(ciphertext), + "Nonce": B64.encode(nonce_bytes), + "Algorithm": "AES-256-GCM", + })), + Err(e) => json_err(StatusCode::BAD_REQUEST, &e.to_string()), + } +} + +pub async fn client_decrypt(State(state): State, body: Body) -> Response { + let _ = state; + let req = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, + }; + let ciphertext_b64 = match require_str( + &req, + &["Ciphertext", "ciphertext"], + "Ciphertext is required", + ) { + Ok(value) => value, + Err(response) => return response, + }; + let nonce_b64 = match require_str(&req, &["Nonce", "nonce"], "Nonce is required") { + Ok(value) => value, + Err(response) => return response, + }; + let key_b64 = match require_str(&req, &["Key", "key"], "Key is required") { + Ok(value) => value, + Err(response) => return response, + }; + + let ciphertext = match decode_b64(ciphertext_b64, "Ciphertext") { + Ok(value) => value, + Err(response) => return response, + }; + let nonce_bytes = match decode_b64(nonce_b64, "Nonce") { + Ok(value) => value, + Err(response) => return response, + }; + let key_bytes = match decode_b64(key_b64, "Key") { + Ok(value) => value, + Err(response) => return response, + }; + if key_bytes.len() != 32 { + return json_err(StatusCode::BAD_REQUEST, "Key must decode to 32 bytes"); + } + if nonce_bytes.len() != 12 { + return json_err(StatusCode::BAD_REQUEST, "Nonce must decode to 12 bytes"); + } + + let cipher = match Aes256Gcm::new_from_slice(&key_bytes) { + Ok(cipher) => cipher, + Err(_) => return json_err(StatusCode::BAD_REQUEST, "Invalid encryption key"), + }; + let nonce = Nonce::from_slice(&nonce_bytes); + + match cipher.decrypt(nonce, ciphertext.as_ref()) { + Ok(plaintext) => json_ok(json!({ + "Plaintext": B64.encode(plaintext), + })), + Err(e) => json_err(StatusCode::BAD_REQUEST, &e.to_string()), + } +} + +pub async fn materials( + State(state): State, + axum::extract::Path(key_id): axum::extract::Path, + body: Body, +) -> Response { + let kms = match require_kms(&state) { + Ok(kms) => kms, + Err(response) => return response, + }; + let _ = match read_json(body).await { + Ok(req) => req, + Err(response) => return response, + }; + + match kms.generate_data_key(&key_id, 32).await { + Ok((plaintext, wrapped)) => json_ok(json!({ + "PlaintextKey": B64.encode(plaintext), + "EncryptedKey": B64.encode(wrapped), + "KeyId": key_id, + "Algorithm": "AES-256-GCM", + "KeyWrapAlgorithm": "kms", })), Err(e) => json_err(StatusCode::INTERNAL_SERVER_ERROR, &e.to_string()), } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/mod.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/mod.rs index 1ba61d4..2f63435 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/mod.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/mod.rs @@ -4,6 +4,7 @@ mod config; pub mod kms; mod select; pub mod ui; +pub mod ui_api; pub mod ui_pages; use std::collections::HashMap; @@ -15,6 +16,7 @@ use axum::response::{IntoResponse, Response}; use base64::engine::general_purpose::URL_SAFE; use base64::Engine; use chrono::{DateTime, Utc}; +use serde_json::json; use myfsio_common::error::{S3Error, S3ErrorCode}; use myfsio_common::types::PartInfo; @@ -25,7 +27,8 @@ use tokio_util::io::ReaderStream; use crate::state::AppState; fn s3_error_response(err: S3Error) -> Response { - let status = StatusCode::from_u16(err.http_status()).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); + let status = + StatusCode::from_u16(err.http_status()).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); let resource = if err.resource.is_empty() { "/".to_string() } else { @@ -35,12 +38,7 @@ fn s3_error_response(err: S3Error) -> Response { .with_resource(resource) .with_request_id(uuid::Uuid::new_v4().simple().to_string()) .to_xml(); - ( - status, - [("content-type", "application/xml")], - body, - ) - .into_response() + (status, [("content-type", "application/xml")], body).into_response() } fn storage_err_response(err: myfsio_storage::error::StorageError) -> Response { @@ -51,17 +49,25 @@ pub async fn list_buckets(State(state): State) -> Response { match state.storage.list_buckets().await { Ok(buckets) => { let xml = myfsio_xml::response::list_buckets_xml("myfsio", "myfsio", &buckets); - ( - StatusCode::OK, - [("content-type", "application/xml")], - xml, - ) - .into_response() + (StatusCode::OK, [("content-type", "application/xml")], xml).into_response() } Err(e) => storage_err_response(e), } } +pub async fn health_check() -> Response { + ( + StatusCode::OK, + [("content-type", "application/json")], + json!({ + "status": "ok", + "version": env!("CARGO_PKG_VERSION"), + }) + .to_string(), + ) + .into_response() +} + pub async fn create_bucket( State(state): State, Path(bucket): Path, @@ -109,14 +115,12 @@ pub async fn create_bucket( } match state.storage.create_bucket(&bucket).await { - Ok(()) => { - ( - StatusCode::OK, - [("location", format!("/{}", bucket).as_str())], - "", - ) - .into_response() - } + Ok(()) => ( + StatusCode::OK, + [("location", format!("/{}", bucket).as_str())], + "", + ) + .into_response(), Err(e) => storage_err_response(e), } } @@ -162,9 +166,7 @@ pub async fn get_bucket( Query(query): Query, ) -> Response { if !matches!(state.storage.bucket_exists(&bucket).await, Ok(true)) { - return storage_err_response( - myfsio_storage::error::StorageError::BucketNotFound(bucket), - ); + return storage_err_response(myfsio_storage::error::StorageError::BucketNotFound(bucket)); } if query.quota.is_some() { @@ -258,8 +260,16 @@ pub async fn get_bucket( let params = myfsio_common::types::ListParams { max_keys, continuation_token: effective_start.clone(), - prefix: if prefix.is_empty() { None } else { Some(prefix.clone()) }, - start_after: if is_v2 { query.start_after.clone() } else { None }, + prefix: if prefix.is_empty() { + None + } else { + Some(prefix.clone()) + }, + start_after: if is_v2 { + query.start_after.clone() + } else { + None + }, }; match state.storage.list_objects(&bucket, ¶ms).await { Ok(result) => { @@ -411,19 +421,16 @@ pub async fn delete_bucket( } } -pub async fn head_bucket( - State(state): State, - Path(bucket): Path, -) -> Response { +pub async fn head_bucket(State(state): State, Path(bucket): Path) -> Response { match state.storage.bucket_exists(&bucket).await { Ok(true) => { let mut headers = HeaderMap::new(); headers.insert("x-amz-bucket-region", state.config.region.parse().unwrap()); (StatusCode::OK, headers).into_response() } - Ok(false) => storage_err_response( - myfsio_storage::error::StorageError::BucketNotFound(bucket), - ), + Ok(false) => { + storage_err_response(myfsio_storage::error::StorageError::BucketNotFound(bucket)) + } Err(e) => storage_err_response(e), } } @@ -458,22 +465,34 @@ pub struct ObjectQuery { fn apply_response_overrides(headers: &mut HeaderMap, query: &ObjectQuery) { if let Some(ref v) = query.response_content_type { - if let Ok(val) = v.parse() { headers.insert("content-type", val); } + if let Ok(val) = v.parse() { + headers.insert("content-type", val); + } } if let Some(ref v) = query.response_content_disposition { - if let Ok(val) = v.parse() { headers.insert("content-disposition", val); } + if let Ok(val) = v.parse() { + headers.insert("content-disposition", val); + } } if let Some(ref v) = query.response_content_language { - if let Ok(val) = v.parse() { headers.insert("content-language", val); } + if let Ok(val) = v.parse() { + headers.insert("content-language", val); + } } if let Some(ref v) = query.response_content_encoding { - if let Ok(val) = v.parse() { headers.insert("content-encoding", val); } + if let Ok(val) = v.parse() { + headers.insert("content-encoding", val); + } } if let Some(ref v) = query.response_cache_control { - if let Ok(val) = v.parse() { headers.insert("cache-control", val); } + if let Ok(val) = v.parse() { + headers.insert("cache-control", val); + } } if let Some(ref v) = query.response_expires { - if let Ok(val) = v.parse() { headers.insert("expires", val); } + if let Ok(val) = v.parse() { + headers.insert("expires", val); + } } } @@ -490,12 +509,18 @@ fn guessed_content_type(key: &str, explicit: Option<&str>) -> String { } fn is_aws_chunked(headers: &HeaderMap) -> bool { - if let Some(enc) = headers.get("content-encoding").and_then(|v| v.to_str().ok()) { + if let Some(enc) = headers + .get("content-encoding") + .and_then(|v| v.to_str().ok()) + { if enc.to_ascii_lowercase().contains("aws-chunked") { return true; } } - if let Some(sha) = headers.get("x-amz-content-sha256").and_then(|v| v.to_str().ok()) { + if let Some(sha) = headers + .get("x-amz-content-sha256") + .and_then(|v| v.to_str().ok()) + { let lower = sha.to_ascii_lowercase(); if lower.starts_with("streaming-") { return true; @@ -535,7 +560,10 @@ pub async fn put_object( if let Some(ref upload_id) = query.upload_id { if let Some(part_number) = query.part_number { - if let Some(copy_source) = headers.get("x-amz-copy-source").and_then(|v| v.to_str().ok()) { + if let Some(copy_source) = headers + .get("x-amz-copy-source") + .and_then(|v| v.to_str().ok()) + { let range = headers .get("x-amz-copy-source-range") .and_then(|v| v.to_str().ok()); @@ -562,15 +590,16 @@ pub async fn put_object( } } - if let Some(copy_source) = headers.get("x-amz-copy-source").and_then(|v| v.to_str().ok()) { + if let Some(copy_source) = headers + .get("x-amz-copy-source") + .and_then(|v| v.to_str().ok()) + { return copy_object_handler(&state, copy_source, &bucket, &key, &headers).await; } let content_type = guessed_content_type( &key, - headers - .get("content-type") - .and_then(|v| v.to_str().ok()), + headers.get("content-type").and_then(|v| v.to_str().ok()), ); let mut metadata = HashMap::new(); @@ -589,14 +618,18 @@ pub async fn put_object( Box::pin(chunked::decode_body(body)) } else { let stream = tokio_util::io::StreamReader::new( - http_body_util::BodyStream::new(body).map_ok(|frame| { - frame.into_data().unwrap_or_default() - }).map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e)), + http_body_util::BodyStream::new(body) + .map_ok(|frame| frame.into_data().unwrap_or_default()) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e)), ); Box::pin(stream) }; - match state.storage.put_object(&bucket, &key, boxed, Some(metadata)).await { + match state + .storage + .put_object(&bucket, &key, boxed, Some(metadata)) + .await + { Ok(meta) => { if let Some(enc_ctx) = resolve_encryption_context(&state, &bucket, &headers).await { if let Some(ref enc_svc) = state.encryption { @@ -612,26 +645,39 @@ pub async fn put_object( Ok(enc_meta) => { if let Err(e) = tokio::fs::rename(&enc_tmp, &obj_path).await { let _ = tokio::fs::remove_file(&enc_tmp).await; - return storage_err_response(myfsio_storage::error::StorageError::Io(e)); + return storage_err_response( + myfsio_storage::error::StorageError::Io(e), + ); } - let enc_size = tokio::fs::metadata(&obj_path).await.map(|m| m.len()).unwrap_or(0); + let enc_size = tokio::fs::metadata(&obj_path) + .await + .map(|m| m.len()) + .unwrap_or(0); let mut enc_metadata = enc_meta.to_metadata_map(); - let all_meta = match state.storage.get_object_metadata(&bucket, &key).await { - Ok(m) => m, - Err(_) => HashMap::new(), - }; + let all_meta = + match state.storage.get_object_metadata(&bucket, &key).await { + Ok(m) => m, + Err(_) => HashMap::new(), + }; for (k, v) in &all_meta { enc_metadata.entry(k.clone()).or_insert_with(|| v.clone()); } enc_metadata.insert("__size__".to_string(), enc_size.to_string()); - let _ = state.storage.put_object_metadata(&bucket, &key, &enc_metadata).await; + let _ = state + .storage + .put_object_metadata(&bucket, &key, &enc_metadata) + .await; let mut resp_headers = HeaderMap::new(); if let Some(ref etag) = meta.etag { - resp_headers.insert("etag", format!("\"{}\"", etag).parse().unwrap()); + resp_headers + .insert("etag", format!("\"{}\"", etag).parse().unwrap()); } - resp_headers.insert("x-amz-server-side-encryption", enc_ctx.algorithm.as_str().parse().unwrap()); + resp_headers.insert( + "x-amz-server-side-encryption", + enc_ctx.algorithm.as_str().parse().unwrap(), + ); return (StatusCode::OK, resp_headers).into_response(); } Err(e) => { @@ -697,7 +743,11 @@ pub async fn get_object( return range_get_handler(&state, &bucket, &key, range_str, &query).await; } - let all_meta = state.storage.get_object_metadata(&bucket, &key).await.unwrap_or_default(); + let all_meta = state + .storage + .get_object_metadata(&bucket, &key) + .await + .unwrap_or_default(); let enc_meta = myfsio_crypto::encryption::EncryptionMetadata::from_metadata(&all_meta); if let (Some(ref enc_info), Some(ref enc_svc)) = (&enc_meta, &state.encryption) { @@ -712,7 +762,10 @@ pub async fn get_object( let customer_key = extract_sse_c_key(&headers); let ck_ref = customer_key.as_deref(); - if let Err(e) = enc_svc.decrypt_object(&obj_path, &dec_tmp, enc_info, ck_ref).await { + if let Err(e) = enc_svc + .decrypt_object(&obj_path, &dec_tmp, enc_info, ck_ref) + .await + { let _ = tokio::fs::remove_file(&dec_tmp).await; return s3_error_response(S3Error::new( myfsio_common::error::S3ErrorCode::InternalError, @@ -747,10 +800,17 @@ pub async fn get_object( insert_content_type(&mut resp_headers, &key, meta.content_type.as_deref()); resp_headers.insert( "last-modified", - meta.last_modified.format("%a, %d %b %Y %H:%M:%S GMT").to_string().parse().unwrap(), + meta.last_modified + .format("%a, %d %b %Y %H:%M:%S GMT") + .to_string() + .parse() + .unwrap(), ); resp_headers.insert("accept-ranges", "bytes".parse().unwrap()); - resp_headers.insert("x-amz-server-side-encryption", enc_info.algorithm.parse().unwrap()); + resp_headers.insert( + "x-amz-server-side-encryption", + enc_info.algorithm.parse().unwrap(), + ); for (k, v) in &meta.metadata { if let Ok(header_val) = v.parse() { @@ -889,11 +949,7 @@ pub async fn head_object( } } -async fn initiate_multipart_handler( - state: &AppState, - bucket: &str, - key: &str, -) -> Response { +async fn initiate_multipart_handler(state: &AppState, bucket: &str, key: &str) -> Response { match state.storage.initiate_multipart(bucket, key, None).await { Ok(upload_id) => { let xml = myfsio_xml::response::initiate_multipart_upload_xml(bucket, key, &upload_id); @@ -915,14 +971,18 @@ async fn upload_part_handler_with_chunking( Box::pin(chunked::decode_body(body)) } else { let stream = tokio_util::io::StreamReader::new( - http_body_util::BodyStream::new(body).map_ok(|frame| { - frame.into_data().unwrap_or_default() - }).map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e)), + http_body_util::BodyStream::new(body) + .map_ok(|frame| frame.into_data().unwrap_or_default()) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e)), ); Box::pin(stream) }; - match state.storage.upload_part(bucket, upload_id, part_number, boxed).await { + match state + .storage + .upload_part(bucket, upload_id, part_number, boxed) + .await + { Ok(etag) => { let mut headers = HeaderMap::new(); headers.insert("etag", format!("\"{}\"", etag).parse().unwrap()); @@ -1052,7 +1112,11 @@ async fn complete_multipart_handler( }) .collect(); - match state.storage.complete_multipart(bucket, upload_id, &parts).await { + match state + .storage + .complete_multipart(bucket, upload_id, &parts) + .await + { Ok(meta) => { let etag = meta.etag.as_deref().unwrap_or(""); let xml = myfsio_xml::response::complete_multipart_upload_xml( @@ -1067,21 +1131,14 @@ async fn complete_multipart_handler( } } -async fn abort_multipart_handler( - state: &AppState, - bucket: &str, - upload_id: &str, -) -> Response { +async fn abort_multipart_handler(state: &AppState, bucket: &str, upload_id: &str) -> Response { match state.storage.abort_multipart(bucket, upload_id).await { Ok(()) => StatusCode::NO_CONTENT.into_response(), Err(e) => storage_err_response(e), } } -async fn list_multipart_uploads_handler( - state: &AppState, - bucket: &str, -) -> Response { +async fn list_multipart_uploads_handler(state: &AppState, bucket: &str) -> Response { match state.storage.list_multipart_uploads(bucket).await { Ok(uploads) => { let xml = myfsio_xml::response::list_multipart_uploads_xml(bucket, &uploads); @@ -1128,9 +1185,7 @@ async fn object_attributes_handler( .collect(); let all = attrs.is_empty(); - let mut xml = String::from( - "" - ); + let mut xml = String::from(""); xml.push_str(""); if all || attrs.contains("etag") { @@ -1139,10 +1194,7 @@ async fn object_attributes_handler( } } if all || attrs.contains("storageclass") { - let sc = meta - .storage_class - .as_deref() - .unwrap_or("STANDARD"); + let sc = meta.storage_class.as_deref().unwrap_or("STANDARD"); xml.push_str(&format!("{}", xml_escape(sc))); } if all || attrs.contains("objectsize") { @@ -1185,7 +1237,11 @@ async fn copy_object_handler( return resp; } - match state.storage.copy_object(src_bucket, src_key, dst_bucket, dst_key).await { + match state + .storage + .copy_object(src_bucket, src_key, dst_bucket, dst_key) + .await + { Ok(meta) => { let etag = meta.etag.as_deref().unwrap_or(""); let last_modified = myfsio_xml::response::format_s3_datetime(&meta.last_modified); @@ -1196,11 +1252,7 @@ async fn copy_object_handler( } } -async fn delete_objects_handler( - state: &AppState, - bucket: &str, - body: Body, -) -> Response { +async fn delete_objects_handler(state: &AppState, bucket: &str, body: Body) -> Response { let body_bytes = match http_body_util::BodyExt::collect(body).await { Ok(collected) => collected.to_bytes(), Err(_) => { @@ -1289,7 +1341,9 @@ async fn range_get_handler( headers.insert("content-length", length.to_string().parse().unwrap()); headers.insert( "content-range", - format!("bytes {}-{}/{}", start, end, total_size).parse().unwrap(), + format!("bytes {}-{}/{}", start, end, total_size) + .parse() + .unwrap(), ); if let Some(ref etag) = meta.etag { headers.insert("etag", format!("\"{}\"", etag).parse().unwrap()); @@ -1471,7 +1525,10 @@ async fn resolve_encryption_context( bucket: &str, headers: &HeaderMap, ) -> Option { - if let Some(alg) = headers.get("x-amz-server-side-encryption").and_then(|v| v.to_str().ok()) { + if let Some(alg) = headers + .get("x-amz-server-side-encryption") + .and_then(|v| v.to_str().ok()) + { let algorithm = match alg { "AES256" => myfsio_crypto::encryption::SseAlgorithm::Aes256, "aws:kms" => myfsio_crypto::encryption::SseAlgorithm::AwsKms, @@ -1606,11 +1663,21 @@ async fn post_object_form_handler( let key_template = match fields.get("key").cloned() { Some(k) => k, - None => return s3_error_response(S3Error::new(S3ErrorCode::InvalidArgument, "Missing key field")), + None => { + return s3_error_response(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing key field", + )) + } }; let policy_b64 = match fields.get("policy").cloned() { Some(v) => v, - None => return s3_error_response(S3Error::new(S3ErrorCode::InvalidArgument, "Missing policy field")), + None => { + return s3_error_response(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing policy field", + )) + } }; let signature = match fields .iter() @@ -1618,7 +1685,12 @@ async fn post_object_form_handler( .map(|(_, v)| v.clone()) { Some(v) => v, - None => return s3_error_response(S3Error::new(S3ErrorCode::InvalidArgument, "Missing signature")), + None => { + return s3_error_response(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing signature", + )) + } }; let credential = match fields .iter() @@ -1626,7 +1698,12 @@ async fn post_object_form_handler( .map(|(_, v)| v.clone()) { Some(v) => v, - None => return s3_error_response(S3Error::new(S3ErrorCode::InvalidArgument, "Missing credential")), + None => { + return s3_error_response(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing credential", + )) + } }; let algorithm = match fields .iter() @@ -1634,7 +1711,12 @@ async fn post_object_form_handler( .map(|(_, v)| v.clone()) { Some(v) => v, - None => return s3_error_response(S3Error::new(S3ErrorCode::InvalidArgument, "Missing algorithm")), + None => { + return s3_error_response(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing algorithm", + )) + } }; if algorithm != "AWS4-HMAC-SHA256" { return s3_error_response(S3Error::new( @@ -1667,7 +1749,10 @@ async fn post_object_form_handler( match chrono::DateTime::parse_from_rfc3339(&normalized) { Ok(exp_time) => { if Utc::now() > exp_time.with_timezone(&Utc) { - return s3_error_response(S3Error::new(S3ErrorCode::AccessDenied, "Policy expired")); + return s3_error_response(S3Error::new( + S3ErrorCode::AccessDenied, + "Policy expired", + )); } } Err(_) => { @@ -1688,14 +1773,23 @@ async fn post_object_form_handler( }; if let Some(conditions) = policy_value.get("conditions").and_then(|v| v.as_array()) { - if let Err(msg) = validate_post_policy_conditions(bucket, &object_key, conditions, &fields, content_length) { + if let Err(msg) = validate_post_policy_conditions( + bucket, + &object_key, + conditions, + &fields, + content_length, + ) { return s3_error_response(S3Error::new(S3ErrorCode::AccessDenied, msg)); } } let credential_parts: Vec<&str> = credential.split('/').collect(); if credential_parts.len() != 5 { - return s3_error_response(S3Error::new(S3ErrorCode::InvalidArgument, "Invalid credential format")); + return s3_error_response(S3Error::new( + S3ErrorCode::InvalidArgument, + "Invalid credential format", + )); } let access_key = credential_parts[0]; let date_stamp = credential_parts[1]; @@ -1704,9 +1798,15 @@ async fn post_object_form_handler( let secret_key = match state.iam.get_secret_key(access_key) { Some(s) => s, - None => return s3_error_response(S3Error::new(S3ErrorCode::AccessDenied, "Invalid access key")), + None => { + return s3_error_response(S3Error::new( + S3ErrorCode::AccessDenied, + "Invalid access key", + )) + } }; - let signing_key = myfsio_auth::sigv4::derive_signing_key(&secret_key, date_stamp, region, service); + let signing_key = + myfsio_auth::sigv4::derive_signing_key(&secret_key, date_stamp, region, service); let expected = myfsio_auth::sigv4::compute_post_policy_signature(&signing_key, &policy_b64); if !myfsio_auth::sigv4::constant_time_compare(&expected, &signature) { return s3_error_response(S3Error::new( @@ -1717,7 +1817,12 @@ async fn post_object_form_handler( let file_data = match file_bytes { Some(b) => b, - None => return s3_error_response(S3Error::new(S3ErrorCode::InvalidArgument, "Missing file field")), + None => { + return s3_error_response(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing file field", + )) + } }; let mut metadata = HashMap::new(); @@ -1741,7 +1846,11 @@ async fn post_object_form_handler( let cursor = std::io::Cursor::new(file_data.to_vec()); let boxed: myfsio_storage::traits::AsyncReadStream = Box::pin(cursor); - let meta = match state.storage.put_object(bucket, &object_key, boxed, Some(metadata)).await { + let meta = match state + .storage + .put_object(bucket, &object_key, boxed, Some(metadata)) + .await + { Ok(m) => m, Err(e) => return storage_err_response(e), }; diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/select.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/select.rs index 211f35a..cf77f80 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/select.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/select.rs @@ -51,14 +51,12 @@ pub async fn post_select_object_content( let object_path = match state.storage.get_object_path(bucket, key).await { Ok(path) => path, Err(_) => { - return s3_error_response(S3Error::new( - S3ErrorCode::NoSuchKey, - "Object not found", - )); + return s3_error_response(S3Error::new(S3ErrorCode::NoSuchKey, "Object not found")); } }; - let join_res = tokio::task::spawn_blocking(move || execute_select_query(object_path, request)).await; + let join_res = + tokio::task::spawn_blocking(move || execute_select_query(object_path, request)).await; let chunks = match join_res { Ok(Ok(chunks)) => chunks, Ok(Err(message)) => { @@ -79,7 +77,10 @@ pub async fn post_select_object_content( } let stats_payload = build_stats_xml(0, bytes_returned); - events.push(Bytes::from(encode_select_event("Stats", stats_payload.as_bytes()))); + events.push(Bytes::from(encode_select_event( + "Stats", + stats_payload.as_bytes(), + ))); events.push(Bytes::from(encode_select_event("End", b""))); let stream = stream::iter(events.into_iter().map(Ok::)); @@ -166,10 +167,18 @@ fn parse_select_request(payload: &[u8]) -> Result { )); } - let input_node = child(&root, "InputSerialization") - .ok_or_else(|| S3Error::new(S3ErrorCode::InvalidRequest, "InputSerialization is required"))?; - let output_node = child(&root, "OutputSerialization") - .ok_or_else(|| S3Error::new(S3ErrorCode::InvalidRequest, "OutputSerialization is required"))?; + let input_node = child(&root, "InputSerialization").ok_or_else(|| { + S3Error::new( + S3ErrorCode::InvalidRequest, + "InputSerialization is required", + ) + })?; + let output_node = child(&root, "OutputSerialization").ok_or_else(|| { + S3Error::new( + S3ErrorCode::InvalidRequest, + "OutputSerialization is required", + ) + })?; let input_format = parse_input_format(&input_node)?; let output_format = parse_output_format(&output_node)?; @@ -187,8 +196,10 @@ fn parse_input_format(node: &roxmltree::Node<'_, '_>) -> Result) -> Result) -> Result { if let Some(csv_node) = child(node, "CSV") { return Ok(OutputFormat::Csv(CsvOutputConfig { - field_delimiter: child_text(&csv_node, "FieldDelimiter").unwrap_or_else(|| ",".to_string()), - record_delimiter: child_text(&csv_node, "RecordDelimiter").unwrap_or_else(|| "\n".to_string()), - quote_character: child_text(&csv_node, "QuoteCharacter").unwrap_or_else(|| "\"".to_string()), + field_delimiter: child_text(&csv_node, "FieldDelimiter") + .unwrap_or_else(|| ",".to_string()), + record_delimiter: child_text(&csv_node, "RecordDelimiter") + .unwrap_or_else(|| "\n".to_string()), + quote_character: child_text(&csv_node, "QuoteCharacter") + .unwrap_or_else(|| "\"".to_string()), })); } if let Some(json_node) = child(node, "JSON") { return Ok(OutputFormat::Json(JsonOutputConfig { - record_delimiter: child_text(&json_node, "RecordDelimiter").unwrap_or_else(|| "\n".to_string()), + record_delimiter: child_text(&json_node, "RecordDelimiter") + .unwrap_or_else(|| "\n".to_string()), })); } @@ -231,7 +246,10 @@ fn parse_output_format(node: &roxmltree::Node<'_, '_>) -> Result(node: &'a roxmltree::Node<'a, 'input>, name: &str) -> Option> { +fn child<'a, 'input>( + node: &'a roxmltree::Node<'a, 'input>, + name: &str, +) -> Option> { node.children() .find(|n| n.is_element() && n.tag_name().name() == name) } @@ -243,7 +261,8 @@ fn child_text(node: &roxmltree::Node<'_, '_>, name: &str) -> Option { } fn execute_select_query(path: PathBuf, request: SelectRequest) -> Result>, String> { - let conn = Connection::open_in_memory().map_err(|e| format!("DuckDB connection error: {}", e))?; + let conn = + Connection::open_in_memory().map_err(|e| format!("DuckDB connection error: {}", e))?; load_input_table(&conn, &path, &request.input_format)?; @@ -341,7 +360,10 @@ fn collect_csv_chunks( let mut chunks: Vec> = Vec::new(); let mut buffer = String::new(); - while let Some(row) = rows.next().map_err(|e| format!("SQL execution error: {}", e))? { + while let Some(row) = rows + .next() + .map_err(|e| format!("SQL execution error: {}", e))? + { let mut fields: Vec = Vec::with_capacity(col_count); for i in 0..col_count { let value = row @@ -353,7 +375,10 @@ fn collect_csv_chunks( } let mut text = value_ref_to_string(value); - if text.contains(&delimiter) || text.contains("e) || text.contains(&record_delimiter) { + if text.contains(&delimiter) + || text.contains("e) + || text.contains(&record_delimiter) + { text = text.replace("e, &(quote.clone() + "e)); text = format!("{}{}{}", quote, text, quote); } @@ -385,16 +410,16 @@ fn collect_json_chunks( let mut chunks: Vec> = Vec::new(); let mut buffer = String::new(); - while let Some(row) = rows.next().map_err(|e| format!("SQL execution error: {}", e))? { + while let Some(row) = rows + .next() + .map_err(|e| format!("SQL execution error: {}", e))? + { let mut record: HashMap = HashMap::with_capacity(col_count); for i in 0..col_count { let value = row .get_ref(i) .map_err(|e| format!("SQL execution error: {}", e))?; - let key = columns - .get(i) - .cloned() - .unwrap_or_else(|| format!("_{}", i)); + let key = columns.get(i).cloned().unwrap_or_else(|| format!("_{}", i)); record.insert(key, value_ref_to_json(value)); } let line = serde_json::to_string(&record) @@ -452,7 +477,9 @@ fn value_ref_to_json(value: ValueRef<'_>) -> serde_json::Value { ValueRef::Double(v) => serde_json::json!(v), ValueRef::Decimal(v) => serde_json::Value::String(v.to_string()), ValueRef::Text(v) => serde_json::Value::String(String::from_utf8_lossy(v).into_owned()), - ValueRef::Blob(v) => serde_json::Value::String(base64::engine::general_purpose::STANDARD.encode(v)), + ValueRef::Blob(v) => { + serde_json::Value::String(base64::engine::general_purpose::STANDARD.encode(v)) + } _ => serde_json::Value::String(format!("{:?}", value)), } } @@ -477,7 +504,8 @@ fn require_xml_content_type(headers: &HeaderMap) -> Option { } fn s3_error_response(err: S3Error) -> Response { - let status = StatusCode::from_u16(err.http_status()).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); + let status = + StatusCode::from_u16(err.http_status()).unwrap_or(StatusCode::INTERNAL_SERVER_ERROR); let resource = if err.resource.is_empty() { "/".to_string() } else { @@ -487,12 +515,7 @@ fn s3_error_response(err: S3Error) -> Response { .with_resource(resource) .with_request_id(uuid::Uuid::new_v4().simple().to_string()) .to_xml(); - ( - status, - [("content-type", "application/xml")], - body, - ) - .into_response() + (status, [("content-type", "application/xml")], body).into_response() } fn build_stats_xml(bytes_scanned: usize, bytes_returned: usize) -> String { @@ -508,7 +531,10 @@ fn encode_select_event(event_type: &str, payload: &[u8]) -> Vec { let mut headers = Vec::new(); headers.extend(encode_select_header(":event-type", event_type)); if event_type == "Records" { - headers.extend(encode_select_header(":content-type", "application/octet-stream")); + headers.extend(encode_select_header( + ":content-type", + "application/octet-stream", + )); } else if event_type == "Stats" { headers.extend(encode_select_header(":content-type", "text/xml")); } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui.rs index 551af37..6414d5e 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui.rs @@ -1,4 +1,5 @@ use std::collections::HashMap; +use std::error::Error as StdError; use axum::extract::{Extension, Form, State}; use axum::http::{header, HeaderMap, StatusCode}; @@ -100,6 +101,10 @@ pub async fn csrf_error_page( resp } +pub async fn root_redirect() -> Response { + Redirect::to("/ui/buckets").into_response() +} + pub async fn not_found_page( State(state): State, Extension(session): Extension, @@ -119,9 +124,15 @@ pub async fn require_login( return next.run(req).await; } let path = req.uri().path().to_string(); - let query = req.uri().query().map(|q| format!("?{}", q)).unwrap_or_default(); + let query = req + .uri() + .query() + .map(|q| format!("?{}", q)) + .unwrap_or_default(); let next_url = format!("{}{}", path, query); - let encoded = percent_encoding::utf8_percent_encode(&next_url, percent_encoding::NON_ALPHANUMERIC).to_string(); + let encoded = + percent_encoding::utf8_percent_encode(&next_url, percent_encoding::NON_ALPHANUMERIC) + .to_string(); let target = format!("/login?next={}", encoded); Redirect::to(&target).into_response() } @@ -130,22 +141,45 @@ pub fn render(state: &AppState, template: &str, ctx: &Context) -> Response { let engine = match &state.templates { Some(e) => e, None => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Templates not configured").into_response(); + return ( + StatusCode::INTERNAL_SERVER_ERROR, + "Templates not configured", + ) + .into_response(); } }; match engine.render(template, ctx) { Ok(html) => { let mut headers = HeaderMap::new(); - headers.insert(header::CONTENT_TYPE, "text/html; charset=utf-8".parse().unwrap()); + headers.insert( + header::CONTENT_TYPE, + "text/html; charset=utf-8".parse().unwrap(), + ); (StatusCode::OK, headers, html).into_response() } Err(e) => { - tracing::error!("Template render failed ({}): {}", template, e); - ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Template error: {}", e), - ) - .into_response() + let mut detail = format!("{}", e); + let mut src = StdError::source(&e); + while let Some(s) = src { + detail.push_str(" | "); + detail.push_str(&s.to_string()); + src = s.source(); + } + tracing::error!("Template render failed ({}): {}", template, detail); + let fallback_ctx = Context::new(); + let body = if template != "500.html" { + engine + .render("500.html", &fallback_ctx) + .unwrap_or_else(|_| "Internal Server Error".to_string()) + } else { + "Internal Server Error".to_string() + }; + let mut headers = HeaderMap::new(); + headers.insert( + header::CONTENT_TYPE, + "text/html; charset=utf-8".parse().unwrap(), + ); + (StatusCode::INTERNAL_SERVER_ERROR, headers, body).into_response() } } } @@ -159,6 +193,8 @@ pub fn base_context(session: &SessionHandle, endpoint: Option<&str>) -> Context ctx.insert("current_user_display_name", &snapshot.display_name); ctx.insert("current_endpoint", &endpoint.unwrap_or("")); ctx.insert("request_args", &HashMap::::new()); + ctx.insert("null", &serde_json::Value::Null); + ctx.insert("none", &serde_json::Value::Null); ctx } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui_api.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui_api.rs new file mode 100644 index 0000000..c4b3db7 --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui_api.rs @@ -0,0 +1,3506 @@ +use std::collections::{BTreeMap, HashMap}; +use std::io::Cursor; +use std::path::{Component, Path as FsPath, PathBuf}; +use std::sync::{Mutex, OnceLock}; + +use axum::body::{to_bytes, Body}; +use axum::extract::{Extension, Path, Query, State}; +use axum::http::{header, HeaderMap, StatusCode}; +use axum::response::{IntoResponse, Response}; +use axum::Json; +use chrono::{DateTime, Datelike, Timelike, Utc}; +use futures::TryStreamExt; +use http_body_util::BodyStream; +use myfsio_auth::sigv4; +use myfsio_common::constants::{BUCKET_VERSIONS_DIR, SYSTEM_BUCKETS_DIR, SYSTEM_ROOT}; +use myfsio_common::types::{ListParams, PartInfo, Tag}; +use myfsio_crypto::encryption::EncryptionMetadata; +use myfsio_storage::error::StorageError; +use myfsio_storage::traits::StorageEngine; +use percent_encoding::{utf8_percent_encode, AsciiSet, NON_ALPHANUMERIC}; +use roxmltree::Document; +use serde::de::DeserializeOwned; +use serde::Deserialize; +use serde_json::{json, Value}; +use sysinfo::{Disks, System}; +use tokio::io::AsyncReadExt; + +use crate::handlers::{self, ObjectQuery}; +use crate::middleware::session::SessionHandle; +use crate::state::AppState; +use crate::stores::connections::RemoteConnection; + +const UI_KEY_ENCODE_SET: &AsciiSet = &NON_ALPHANUMERIC + .remove(b'-') + .remove(b'_') + .remove(b'.') + .remove(b'~') + .remove(b'/'); + +const PATH_SEGMENT_ENCODE_SET: &AsciiSet = &NON_ALPHANUMERIC + .remove(b'-') + .remove(b'_') + .remove(b'.') + .remove(b'~'); + +const AWS_QUERY_ENCODE_SET: &AsciiSet = &NON_ALPHANUMERIC + .remove(b'-') + .remove(b'_') + .remove(b'.') + .remove(b'~'); + +fn url_templates_for(bucket: &str) -> Value { + json!({ + "download": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/download", bucket), + "preview": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/preview", bucket), + "delete": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/delete", bucket), + "presign": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/presign", bucket), + "metadata": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/metadata", bucket), + "versions": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/versions", bucket), + "restore": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/restore/VERSION_ID_PLACEHOLDER", bucket), + "tags": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/tags", bucket), + "copy": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/copy", bucket), + "move": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/move", bucket), + }) +} + +fn encode_object_key(key: &str) -> String { + utf8_percent_encode(key, UI_KEY_ENCODE_SET).to_string() +} + +fn encode_path_segment(value: &str) -> String { + utf8_percent_encode(value, PATH_SEGMENT_ENCODE_SET).to_string() +} + +fn build_ui_object_url(bucket: &str, key: &str, action: &str) -> String { + format!( + "/ui/buckets/{}/objects/{}/{}", + bucket, + encode_object_key(key), + action + ) +} + +fn human_size(bytes: u64) -> String { + const UNITS: [&str; 6] = ["B", "KB", "MB", "GB", "TB", "PB"]; + let mut size = bytes as f64; + let mut idx = 0; + while size >= 1024.0 && idx < UNITS.len() - 1 { + size /= 1024.0; + idx += 1; + } + if idx == 0 { + format!("{} {}", bytes, UNITS[idx]) + } else { + format!("{:.1} {}", size, UNITS[idx]) + } +} + +fn json_error(status: StatusCode, message: impl Into) -> Response { + (status, Json(json!({ "error": message.into() }))).into_response() +} + +fn json_ok(value: Value) -> Response { + Json(value).into_response() +} + +fn push_issue(result: &mut Value, issue: Value) { + if let Some(items) = result + .get_mut("issues") + .and_then(|value| value.as_array_mut()) + { + items.push(issue); + } +} + +fn storage_status(err: &StorageError) -> StatusCode { + match err { + StorageError::BucketNotFound(_) + | StorageError::ObjectNotFound { .. } + | StorageError::UploadNotFound(_) => StatusCode::NOT_FOUND, + StorageError::InvalidBucketName(_) + | StorageError::InvalidObjectKey(_) + | StorageError::InvalidRange + | StorageError::QuotaExceeded(_) => StatusCode::BAD_REQUEST, + StorageError::BucketAlreadyExists(_) => StatusCode::CONFLICT, + StorageError::BucketNotEmpty(_) => StatusCode::CONFLICT, + StorageError::Io(_) | StorageError::Json(_) | StorageError::Internal(_) => { + StatusCode::INTERNAL_SERVER_ERROR + } + } +} + +fn storage_json_error(err: StorageError) -> Response { + json_error(storage_status(&err), err.to_string()) +} + +fn parse_bool_flag(value: Option<&str>) -> bool { + matches!( + value.map(|v| v.trim().to_ascii_lowercase()), + Some(v) if v == "1" || v == "true" || v == "on" || v == "yes" + ) +} + +fn parse_form_body(bytes: &[u8]) -> HashMap { + String::from_utf8_lossy(bytes) + .split('&') + .filter(|pair| !pair.is_empty()) + .map(|pair| { + let mut parts = pair.splitn(2, '='); + let key = parts.next().unwrap_or_default(); + let value = parts.next().unwrap_or_default(); + (decode_form_value(key), decode_form_value(value)) + }) + .collect() +} + +fn decode_form_value(value: &str) -> String { + percent_encoding::percent_decode_str(&value.replace('+', " ")) + .decode_utf8_lossy() + .into_owned() +} + +fn current_access_key(session: &SessionHandle) -> Option { + session.read(|s| s.user_id.clone()) +} + +fn owner_id_or_default(session: &SessionHandle) -> String { + current_access_key(session).unwrap_or_else(|| "myfsio".to_string()) +} + +fn safe_attachment_filename(key: &str) -> String { + let raw = key.rsplit('/').next().unwrap_or(key); + let sanitized = raw + .replace('"', "'") + .replace('\\', "_") + .chars() + .filter(|c| c.is_ascii_graphic() || *c == ' ') + .collect::(); + if sanitized.trim().is_empty() { + "download".to_string() + } else { + sanitized + } +} + +fn parse_api_base(state: &AppState) -> String { + std::env::var("API_BASE_URL") + .unwrap_or_else(|_| format!("http://{}", state.config.bind_addr)) + .trim_end_matches('/') + .to_string() +} + +fn aws_query_encode(value: &str) -> String { + utf8_percent_encode(value, AWS_QUERY_ENCODE_SET).to_string() +} + +fn xml_escape(value: &str) -> String { + value + .replace('&', "&") + .replace('<', "<") + .replace('>', ">") + .replace('"', """) + .replace('\'', "'") +} + +fn key_relative_path(key: &str) -> Result { + let mut out = PathBuf::new(); + for component in FsPath::new(key).components() { + match component { + Component::Normal(part) => out.push(part), + _ => return Err("Invalid object key".to_string()), + } + } + if out.as_os_str().is_empty() { + return Err("Invalid object key".to_string()); + } + Ok(out) +} + +fn object_live_path(state: &AppState, bucket: &str, key: &str) -> Result { + let rel = key_relative_path(key)?; + Ok(state.config.storage_root.join(bucket).join(rel)) +} + +fn version_root_for_bucket(state: &AppState, bucket: &str) -> PathBuf { + state + .config + .storage_root + .join(SYSTEM_ROOT) + .join(SYSTEM_BUCKETS_DIR) + .join(bucket) + .join(BUCKET_VERSIONS_DIR) +} + +fn version_dir_for_object(state: &AppState, bucket: &str, key: &str) -> Result { + let rel = key_relative_path(key)?; + Ok(version_root_for_bucket(state, bucket).join(rel)) +} + +#[derive(Debug, Clone, Default, Deserialize)] +struct VersionManifest { + #[serde(default)] + version_id: String, + #[serde(default)] + key: String, + #[serde(default)] + size: u64, + #[serde(default)] + archived_at: Option, + #[serde(default)] + etag: Option, + #[serde(default)] + metadata: HashMap, + #[serde(default)] + reason: Option, +} + +fn manifest_timestamp(value: &VersionManifest) -> DateTime { + value + .archived_at + .as_deref() + .and_then(|s| DateTime::parse_from_rfc3339(s).ok()) + .map(|dt| dt.with_timezone(&Utc)) + .unwrap_or_else(Utc::now) +} + +fn manifest_to_json(record: &VersionManifest) -> Value { + let ts = manifest_timestamp(record); + json!({ + "version_id": record.version_id, + "key": record.key, + "size": record.size, + "etag": record.etag, + "archived_at": ts.to_rfc3339(), + "last_modified": ts.to_rfc3339(), + "metadata": record.metadata, + "reason": record.reason.clone().unwrap_or_else(|| "update".to_string()), + "is_latest": false, + }) +} + +fn read_version_manifests_for_object( + state: &AppState, + bucket: &str, + key: &str, +) -> Result, String> { + let version_dir = version_dir_for_object(state, bucket, key)?; + if !version_dir.exists() { + return Ok(Vec::new()); + } + + let mut entries = Vec::new(); + for entry in std::fs::read_dir(&version_dir).map_err(|e| e.to_string())? { + let entry = entry.map_err(|e| e.to_string())?; + if !entry.file_type().map_err(|e| e.to_string())?.is_file() { + continue; + } + if entry.path().extension().and_then(|ext| ext.to_str()) != Some("json") { + continue; + } + let text = std::fs::read_to_string(entry.path()).map_err(|e| e.to_string())?; + let mut manifest: VersionManifest = + serde_json::from_str(&text).map_err(|e| e.to_string())?; + if manifest.version_id.is_empty() { + manifest.version_id = entry + .path() + .file_stem() + .and_then(|s| s.to_str()) + .unwrap_or_default() + .to_string(); + } + if manifest.key.is_empty() { + manifest.key = key.to_string(); + } + entries.push(manifest); + } + + entries.sort_by(|a, b| manifest_timestamp(b).cmp(&manifest_timestamp(a))); + Ok(entries) +} + +async fn read_object_bytes_for_zip( + state: &AppState, + bucket: &str, + key: &str, +) -> Result, String> { + let all_meta = state + .storage + .get_object_metadata(bucket, key) + .await + .map_err(|e| e.to_string())?; + + if let Some(enc_meta) = EncryptionMetadata::from_metadata(&all_meta) { + let enc_svc = state + .encryption + .as_ref() + .ok_or_else(|| "Encryption service is not available".to_string())?; + let obj_path = state + .storage + .get_object_path(bucket, key) + .await + .map_err(|e| e.to_string())?; + let tmp_dir = state.config.storage_root.join(SYSTEM_ROOT).join("tmp"); + let _ = tokio::fs::create_dir_all(&tmp_dir).await; + let dec_tmp = tmp_dir.join(format!("zip-dec-{}", uuid::Uuid::new_v4())); + enc_svc + .decrypt_object(&obj_path, &dec_tmp, &enc_meta, None) + .await + .map_err(|e| e.to_string())?; + let bytes = tokio::fs::read(&dec_tmp).await.map_err(|e| e.to_string())?; + let _ = tokio::fs::remove_file(&dec_tmp).await; + return Ok(bytes); + } + + let (_meta, mut reader) = state + .storage + .get_object(bucket, key) + .await + .map_err(|e| e.to_string())?; + let mut bytes = Vec::new(); + reader + .read_to_end(&mut bytes) + .await + .map_err(|e| e.to_string())?; + Ok(bytes) +} + +fn value_to_string_vec(value: Option<&Value>, field_name: &str) -> Vec { + match value { + Some(Value::Array(items)) => items + .iter() + .filter_map(|v| v.as_str().map(|s| s.to_string())) + .collect(), + Some(Value::String(s)) if !s.trim().is_empty() => vec![s.to_string()], + Some(Value::Null) | None => Vec::new(), + Some(_) => vec![field_name.to_string()], + } +} + +fn xml_child<'a>(node: roxmltree::Node<'a, 'a>, name: &str) -> Option> { + node.children() + .find(|child| child.is_element() && child.tag_name().name() == name) +} + +fn xml_child_text(node: roxmltree::Node<'_, '_>, name: &str) -> Option { + xml_child(node, name) + .and_then(|child| child.text()) + .map(|text| text.trim().to_string()) + .filter(|text| !text.is_empty()) +} + +fn xml_children_texts(node: roxmltree::Node<'_, '_>, name: &str) -> Vec { + node.children() + .filter(|child| child.is_element() && child.tag_name().name() == name) + .filter_map(|child| child.text().map(|text| text.trim().to_string())) + .filter(|text| !text.is_empty()) + .collect() +} + +fn parse_acl_value(value: Option<&Value>, owner: &str) -> Value { + let default_grant = json!({ + "grantee": owner, + "permission": "FULL_CONTROL", + "grantee_type": "CanonicalUser", + "display_name": owner, + "grantee_id": owner, + "grantee_uri": Value::Null, + }); + + let Some(value) = value else { + return json!({ + "owner": owner, + "grants": [default_grant], + "canned_acls": ["private", "public-read", "public-read-write", "authenticated-read"], + }); + }; + + match value { + Value::String(xml) => { + let doc = match Document::parse(xml) { + Ok(doc) => doc, + Err(_) => { + return json!({ + "owner": owner, + "grants": [default_grant], + "canned_acls": ["private", "public-read", "public-read-write", "authenticated-read"], + }); + } + }; + let owner_node = doc + .descendants() + .find(|node| node.is_element() && node.tag_name().name() == "Owner"); + let owner_id = owner_node + .and_then(|node| xml_child_text(node, "ID")) + .unwrap_or_else(|| owner.to_string()); + + let grants = doc + .descendants() + .filter(|node| node.is_element() && node.tag_name().name() == "Grant") + .map(|grant| { + let grantee = xml_child(grant, "Grantee"); + let permission = xml_child_text(grant, "Permission").unwrap_or_default(); + let grantee_id = grantee.and_then(|node| xml_child_text(node, "ID")); + let display_name = grantee.and_then(|node| xml_child_text(node, "DisplayName")); + let grantee_uri = grantee.and_then(|node| xml_child_text(node, "URI")); + let grantee_type = grantee + .and_then(|node| { + node.attributes() + .find(|attr| { + attr.name() == "type" || attr.name().ends_with(":type") + }) + .map(|attr| attr.value().to_string()) + }) + .or_else(|| { + if grantee_uri.is_some() { + Some("Group".to_string()) + } else { + Some("CanonicalUser".to_string()) + } + }) + .unwrap_or_else(|| "CanonicalUser".to_string()); + let grantee_label = display_name + .clone() + .or_else(|| grantee_id.clone()) + .or_else(|| grantee_uri.clone()) + .unwrap_or_else(|| "unknown".to_string()); + + json!({ + "grantee": grantee_label, + "permission": permission, + "grantee_type": grantee_type, + "display_name": display_name, + "grantee_id": grantee_id, + "grantee_uri": grantee_uri, + }) + }) + .collect::>(); + + json!({ + "owner": owner_id, + "grants": if grants.is_empty() { vec![default_grant] } else { grants }, + "canned_acls": ["private", "public-read", "public-read-write", "authenticated-read"], + }) + } + Value::Object(map) => { + let grants = map + .get("grants") + .and_then(|value| value.as_array()) + .cloned() + .unwrap_or_else(|| vec![default_grant]); + json!({ + "owner": map.get("owner").and_then(|v| v.as_str()).unwrap_or(owner), + "grants": grants, + "canned_acls": ["private", "public-read", "public-read-write", "authenticated-read"], + }) + } + _ => json!({ + "owner": owner, + "grants": [default_grant], + "canned_acls": ["private", "public-read", "public-read-write", "authenticated-read"], + }), + } +} + +fn parse_cors_value(value: Option<&Value>) -> Value { + let Some(value) = value else { + return json!({ "rules": [] }); + }; + + match value { + Value::String(xml) => { + let doc = match Document::parse(xml) { + Ok(doc) => doc, + Err(_) => return json!({ "rules": [] }), + }; + let rules = doc + .descendants() + .filter(|node| node.is_element() && node.tag_name().name() == "CORSRule") + .map(|rule| { + let allowed_origins = xml_children_texts(rule, "AllowedOrigin"); + let allowed_methods = xml_children_texts(rule, "AllowedMethod"); + let allowed_headers = xml_children_texts(rule, "AllowedHeader"); + let expose_headers = xml_children_texts(rule, "ExposeHeader"); + let max_age_seconds = + xml_child_text(rule, "MaxAgeSeconds").and_then(|v| v.parse::().ok()); + json!({ + "AllowedOrigins": allowed_origins, + "AllowedMethods": allowed_methods, + "AllowedHeaders": allowed_headers, + "ExposeHeaders": expose_headers, + "MaxAgeSeconds": max_age_seconds, + "allowed_origins": allowed_origins, + "allowed_methods": allowed_methods, + "allowed_headers": allowed_headers, + "expose_headers": expose_headers, + "max_age_seconds": max_age_seconds, + }) + }) + .collect::>(); + json!({ "rules": rules }) + } + Value::Array(rules) => json!({ "rules": rules }), + Value::Object(map) => { + if let Some(rules) = map.get("rules").and_then(|value| value.as_array()) { + json!({ "rules": rules }) + } else { + json!({ "rules": [map] }) + } + } + _ => json!({ "rules": [] }), + } +} + +fn parse_lifecycle_value(value: Option<&Value>) -> Value { + let Some(value) = value else { + return json!({ "rules": [] }); + }; + + match value { + Value::String(xml) => { + let doc = match Document::parse(xml) { + Ok(doc) => doc, + Err(_) => return json!({ "rules": [] }), + }; + let rules = doc + .descendants() + .filter(|node| node.is_element() && node.tag_name().name() == "Rule") + .map(|rule| { + let rule_id = xml_child_text(rule, "ID").unwrap_or_default(); + let status = xml_child_text(rule, "Status").unwrap_or_else(|| "Enabled".to_string()); + let prefix = xml_child(rule, "Filter") + .and_then(|filter| xml_child_text(filter, "Prefix")) + .or_else(|| xml_child_text(rule, "Prefix")) + .unwrap_or_default(); + let expiration_days = xml_child(rule, "Expiration") + .and_then(|node| xml_child_text(node, "Days")) + .and_then(|v| v.parse::().ok()); + let noncurrent_days = xml_child(rule, "NoncurrentVersionExpiration") + .and_then(|node| xml_child_text(node, "NoncurrentDays")) + .and_then(|v| v.parse::().ok()); + let abort_days = xml_child(rule, "AbortIncompleteMultipartUpload") + .and_then(|node| xml_child_text(node, "DaysAfterInitiation")) + .and_then(|v| v.parse::().ok()); + + json!({ + "ID": rule_id, + "Status": status, + "Filter": { "Prefix": prefix }, + "Expiration": expiration_days.map(|days| json!({ "Days": days })), + "NoncurrentVersionExpiration": noncurrent_days.map(|days| json!({ "NoncurrentDays": days })), + "AbortIncompleteMultipartUpload": abort_days.map(|days| json!({ "DaysAfterInitiation": days })), + "id": rule_id, + "status": status, + "prefix": prefix, + "expiration_days": expiration_days, + "noncurrent_days": noncurrent_days, + "abort_mpu_days": abort_days, + }) + }) + .collect::>(); + json!({ "rules": rules }) + } + Value::Array(rules) => json!({ "rules": rules }), + Value::Object(map) => { + if let Some(rules) = map.get("rules").and_then(|value| value.as_array()) { + json!({ "rules": rules }) + } else { + json!({ "rules": [map] }) + } + } + _ => json!({ "rules": [] }), + } +} + +fn bucket_acl_xml_for_canned(owner_id: &str, canned_acl: &str) -> Result { + let mut grants = vec![format!( + "{}{}FULL_CONTROL", + xml_escape(owner_id), + xml_escape(owner_id), + )]; + + match canned_acl { + "private" => {} + "public-read" => grants.push( + "http://acs.amazonaws.com/groups/global/AllUsersREAD".to_string() + ), + "public-read-write" => { + grants.push( + "http://acs.amazonaws.com/groups/global/AllUsersREAD".to_string() + ); + grants.push( + "http://acs.amazonaws.com/groups/global/AllUsersWRITE".to_string() + ); + } + "authenticated-read" => grants.push( + "http://acs.amazonaws.com/groups/global/AuthenticatedUsersREAD".to_string() + ), + _ => return Err(format!("Invalid canned ACL: {}", canned_acl)), + } + + Ok(format!( + "{}{}{}", + xml_escape(owner_id), + xml_escape(owner_id), + grants.join("") + )) +} + +fn cors_xml_from_rules(rules: &[Value]) -> String { + let mut xml = String::from( + "", + ); + for rule in rules { + xml.push_str(""); + for origin in value_to_string_vec(rule.get("AllowedOrigins"), "AllowedOrigin") { + xml.push_str(&format!( + "{}", + xml_escape(&origin) + )); + } + for method in value_to_string_vec(rule.get("AllowedMethods"), "AllowedMethod") { + xml.push_str(&format!( + "{}", + xml_escape(&method) + )); + } + for header in value_to_string_vec(rule.get("AllowedHeaders"), "AllowedHeader") { + xml.push_str(&format!( + "{}", + xml_escape(&header) + )); + } + for header in value_to_string_vec(rule.get("ExposeHeaders"), "ExposeHeader") { + xml.push_str(&format!( + "{}", + xml_escape(&header) + )); + } + if let Some(max_age) = rule.get("MaxAgeSeconds").and_then(|v| v.as_u64()) { + xml.push_str(&format!("{}", max_age)); + } + xml.push_str(""); + } + xml.push_str(""); + xml +} + +fn lifecycle_xml_from_rules(rules: &[Value]) -> String { + let mut xml = String::from( + "", + ); + for rule in rules { + xml.push_str(""); + + let id = rule.get("ID").and_then(|v| v.as_str()).unwrap_or_default(); + if !id.is_empty() { + xml.push_str(&format!("{}", xml_escape(id))); + } + + let status = rule + .get("Status") + .and_then(|v| v.as_str()) + .unwrap_or("Enabled"); + xml.push_str(&format!("{}", xml_escape(status))); + + let prefix = rule + .get("Filter") + .and_then(|v| v.get("Prefix")) + .and_then(|v| v.as_str()) + .or_else(|| rule.get("Prefix").and_then(|v| v.as_str())) + .unwrap_or_default(); + xml.push_str(""); + xml.push_str(&format!("{}", xml_escape(prefix))); + xml.push_str(""); + + if let Some(days) = rule + .get("Expiration") + .and_then(|v| v.get("Days")) + .and_then(|v| v.as_u64()) + { + xml.push_str(&format!("{}", days)); + } + + if let Some(days) = rule + .get("NoncurrentVersionExpiration") + .and_then(|v| v.get("NoncurrentDays")) + .and_then(|v| v.as_u64()) + { + xml.push_str(&format!( + "{}", + days + )); + } + + if let Some(days) = rule + .get("AbortIncompleteMultipartUpload") + .and_then(|v| v.get("DaysAfterInitiation")) + .and_then(|v| v.as_u64()) + { + xml.push_str(&format!( + "{}", + days + )); + } + + xml.push_str(""); + } + xml.push_str(""); + xml +} + +fn zip_dos_time(dt: DateTime) -> (u16, u16) { + let year = dt.year().clamp(1980, 2107) as u16; + let month = dt.month() as u16; + let day = dt.day() as u16; + let hour = dt.hour() as u16; + let minute = dt.minute() as u16; + let second = (dt.second() / 2) as u16; + let dos_time = (hour << 11) | (minute << 5) | second; + let dos_date = ((year - 1980) << 9) | (month << 5) | day; + (dos_time, dos_date) +} + +fn write_u16(buf: &mut Vec, value: u16) { + buf.extend_from_slice(&value.to_le_bytes()); +} + +fn write_u32(buf: &mut Vec, value: u32) { + buf.extend_from_slice(&value.to_le_bytes()); +} + +fn build_zip_archive(entries: Vec<(String, Vec, DateTime)>) -> Result, String> { + #[derive(Clone)] + struct CentralEntry { + name: Vec, + crc32: u32, + size: u32, + offset: u32, + mod_time: u16, + mod_date: u16, + } + + let mut output = Vec::new(); + let mut central_entries = Vec::new(); + + for (name, data, modified) in entries { + if data.len() > u32::MAX as usize { + return Err(format!("Object '{}' is too large for ZIP export", name)); + } + let offset = output.len(); + if offset > u32::MAX as usize { + return Err("ZIP archive is too large".to_string()); + } + + let name_bytes = name.into_bytes(); + let mut hasher = crc32fast::Hasher::new(); + hasher.update(&data); + let crc32 = hasher.finalize(); + let size = data.len() as u32; + let (mod_time, mod_date) = zip_dos_time(modified); + let flags = 0x0800u16; + + write_u32(&mut output, 0x04034b50); + write_u16(&mut output, 20); + write_u16(&mut output, flags); + write_u16(&mut output, 0); + write_u16(&mut output, mod_time); + write_u16(&mut output, mod_date); + write_u32(&mut output, crc32); + write_u32(&mut output, size); + write_u32(&mut output, size); + write_u16(&mut output, name_bytes.len() as u16); + write_u16(&mut output, 0); + output.extend_from_slice(&name_bytes); + output.extend_from_slice(&data); + + central_entries.push(CentralEntry { + name: name_bytes, + crc32, + size, + offset: offset as u32, + mod_time, + mod_date, + }); + } + + let central_start = output.len(); + for entry in ¢ral_entries { + write_u32(&mut output, 0x02014b50); + write_u16(&mut output, 20); + write_u16(&mut output, 20); + write_u16(&mut output, 0x0800); + write_u16(&mut output, 0); + write_u16(&mut output, entry.mod_time); + write_u16(&mut output, entry.mod_date); + write_u32(&mut output, entry.crc32); + write_u32(&mut output, entry.size); + write_u32(&mut output, entry.size); + write_u16(&mut output, entry.name.len() as u16); + write_u16(&mut output, 0); + write_u16(&mut output, 0); + write_u16(&mut output, 0); + write_u16(&mut output, 0); + write_u32(&mut output, 0); + write_u32(&mut output, entry.offset); + output.extend_from_slice(&entry.name); + } + + let central_size = output.len() - central_start; + if central_entries.len() > u16::MAX as usize + || central_start > u32::MAX as usize + || central_size > u32::MAX as usize + { + return Err("ZIP archive exceeds classic ZIP limits".to_string()); + } + + write_u32(&mut output, 0x06054b50); + write_u16(&mut output, 0); + write_u16(&mut output, 0); + write_u16(&mut output, central_entries.len() as u16); + write_u16(&mut output, central_entries.len() as u16); + write_u32(&mut output, central_size as u32); + write_u32(&mut output, central_start as u32); + write_u16(&mut output, 0); + + Ok(output) +} + +fn dangerous_preview_content_type(content_type: Option<&str>, key: &str) -> Option { + let guessed = content_type + .map(|value| value.to_ascii_lowercase()) + .unwrap_or_else(|| { + mime_guess::from_path(key) + .first_raw() + .unwrap_or("application/octet-stream") + .to_ascii_lowercase() + }); + + match guessed.split(';').next().unwrap_or_default().trim() { + "text/html" + | "text/xml" + | "application/xml" + | "application/xhtml+xml" + | "image/svg+xml" => Some("text/plain; charset=utf-8".to_string()), + _ => None, + } +} + +async fn parse_json_body(body: Body) -> Result { + let bytes = to_bytes(body, usize::MAX) + .await + .map_err(|_| json_error(StatusCode::BAD_REQUEST, "Failed to read request body"))?; + serde_json::from_slice::(&bytes) + .map_err(|e| json_error(StatusCode::BAD_REQUEST, format!("Invalid JSON body: {}", e))) +} + +#[derive(Deserialize, Default)] +pub struct ListObjectsQuery { + #[serde(default)] + pub max_keys: Option, + #[serde(default)] + pub continuation_token: Option, + #[serde(default)] + pub prefix: Option, + #[serde(default)] + pub start_after: Option, +} + +pub async fn list_bucket_objects( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + Query(q): Query, +) -> Response { + if !matches!(state.storage.bucket_exists(&bucket_name).await, Ok(true)) { + return json_error(StatusCode::NOT_FOUND, "Bucket not found"); + } + + let max_keys = q.max_keys.unwrap_or(1000).min(5000); + let params = ListParams { + max_keys, + continuation_token: q.continuation_token.clone(), + prefix: q.prefix.clone(), + start_after: q.start_after.clone(), + }; + + let versioning_enabled = state + .storage + .is_versioning_enabled(&bucket_name) + .await + .unwrap_or(false); + + let stats = state.storage.bucket_stats(&bucket_name).await.ok(); + let total_count = stats.as_ref().map(|s| s.objects).unwrap_or(0); + + match state.storage.list_objects(&bucket_name, ¶ms).await { + Ok(res) => { + let objects: Vec = res + .objects + .iter() + .map(|o| { + json!({ + "key": o.key, + "size": o.size, + "last_modified": o.last_modified.to_rfc3339(), + "last_modified_iso": o.last_modified.to_rfc3339(), + "last_modified_display": o.last_modified.format("%Y-%m-%d %H:%M:%S").to_string(), + "etag": o.etag.clone().unwrap_or_default(), + "storage_class": o.storage_class.clone().unwrap_or_else(|| "STANDARD".to_string()), + "content_type": o.content_type.clone().unwrap_or_default(), + "download_url": build_ui_object_url(&bucket_name, &o.key, "download"), + "preview_url": build_ui_object_url(&bucket_name, &o.key, "preview"), + "delete_endpoint": build_ui_object_url(&bucket_name, &o.key, "delete"), + "presign_endpoint": build_ui_object_url(&bucket_name, &o.key, "presign"), + "metadata_url": build_ui_object_url(&bucket_name, &o.key, "metadata"), + "versions_endpoint": build_ui_object_url(&bucket_name, &o.key, "versions"), + "restore_template": format!( + "/ui/buckets/{}/objects/{}/restore/VERSION_ID_PLACEHOLDER", + bucket_name, + encode_object_key(&o.key) + ), + "tags_url": build_ui_object_url(&bucket_name, &o.key, "tags"), + "copy_url": build_ui_object_url(&bucket_name, &o.key, "copy"), + "move_url": build_ui_object_url(&bucket_name, &o.key, "move"), + }) + }) + .collect(); + + Json(json!({ + "versioning_enabled": versioning_enabled, + "total_count": total_count, + "is_truncated": res.is_truncated, + "next_continuation_token": res.next_continuation_token, + "url_templates": url_templates_for(&bucket_name), + "objects": objects, + })) + .into_response() + } + Err(e) => storage_json_error(e), + } +} + +#[derive(Deserialize, Default)] +pub struct StreamObjectsQuery { + #[serde(default)] + pub prefix: Option, + #[serde(default)] + pub delimiter: Option, +} + +pub async fn stream_bucket_objects( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + Query(q): Query, +) -> Response { + if !matches!(state.storage.bucket_exists(&bucket_name).await, Ok(true)) { + return (StatusCode::NOT_FOUND, "Bucket not found").into_response(); + } + + let versioning_enabled = state + .storage + .is_versioning_enabled(&bucket_name) + .await + .unwrap_or(false); + let stats = state.storage.bucket_stats(&bucket_name).await.ok(); + let total_count = stats.as_ref().map(|s| s.objects).unwrap_or(0); + + let mut lines: Vec = Vec::new(); + lines.push( + json!({ + "type": "meta", + "url_templates": url_templates_for(&bucket_name), + "versioning_enabled": versioning_enabled, + }) + .to_string(), + ); + lines.push(json!({ "type": "count", "total_count": total_count }).to_string()); + + let use_delimiter = q.delimiter.as_deref() == Some("/"); + let prefix = q.prefix.clone().unwrap_or_default(); + + if use_delimiter { + let params = myfsio_common::types::ShallowListParams { + prefix: prefix.clone(), + delimiter: "/".to_string(), + max_keys: 5000, + continuation_token: None, + }; + match state + .storage + .list_objects_shallow(&bucket_name, ¶ms) + .await + { + Ok(res) => { + for p in &res.common_prefixes { + lines.push(json!({ "type": "folder", "prefix": p }).to_string()); + } + for o in &res.objects { + lines.push( + json!({ + "type": "object", + "key": o.key, + "size": o.size, + "last_modified": o.last_modified.to_rfc3339(), + "last_modified_iso": o.last_modified.to_rfc3339(), + "last_modified_display": o.last_modified.format("%Y-%m-%d %H:%M:%S").to_string(), + "etag": o.etag.clone().unwrap_or_default(), + "storage_class": o.storage_class.clone().unwrap_or_else(|| "STANDARD".to_string()), + }) + .to_string(), + ); + } + } + Err(e) => lines.push(json!({ "type": "error", "error": e.to_string() }).to_string()), + } + } else { + let mut token: Option = None; + loop { + let params = ListParams { + max_keys: 1000, + continuation_token: token.clone(), + prefix: if prefix.is_empty() { + None + } else { + Some(prefix.clone()) + }, + start_after: None, + }; + match state.storage.list_objects(&bucket_name, ¶ms).await { + Ok(res) => { + for o in &res.objects { + lines.push( + json!({ + "type": "object", + "key": o.key, + "size": o.size, + "last_modified": o.last_modified.to_rfc3339(), + "last_modified_iso": o.last_modified.to_rfc3339(), + "last_modified_display": o.last_modified.format("%Y-%m-%d %H:%M:%S").to_string(), + "etag": o.etag.clone().unwrap_or_default(), + "storage_class": o.storage_class.clone().unwrap_or_else(|| "STANDARD".to_string()), + }) + .to_string(), + ); + } + if !res.is_truncated || res.next_continuation_token.is_none() { + break; + } + token = res.next_continuation_token; + } + Err(e) => { + lines.push(json!({ "type": "error", "error": e.to_string() }).to_string()); + break; + } + } + } + } + + lines.push(json!({ "type": "done" }).to_string()); + + let body = lines.join("\n") + "\n"; + let mut headers = HeaderMap::new(); + headers.insert( + header::CONTENT_TYPE, + "application/x-ndjson; charset=utf-8".parse().unwrap(), + ); + (StatusCode::OK, headers, body).into_response() +} + +pub async fn list_bucket_folders( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + Query(q): Query, +) -> Response { + if !matches!(state.storage.bucket_exists(&bucket_name).await, Ok(true)) { + return json_error(StatusCode::NOT_FOUND, "Bucket not found"); + } + + let prefix = q.prefix.clone().unwrap_or_default(); + let params = myfsio_common::types::ShallowListParams { + prefix: prefix.clone(), + delimiter: "/".to_string(), + max_keys: 5000, + continuation_token: None, + }; + match state + .storage + .list_objects_shallow(&bucket_name, ¶ms) + .await + { + Ok(res) => Json(json!({ + "prefixes": res.common_prefixes, + "current_prefix": prefix, + })) + .into_response(), + Err(e) => storage_json_error(e), + } +} + +pub async fn list_copy_targets( + State(state): State, + Extension(_session): Extension, + Path(_bucket_name): Path, +) -> Response { + let buckets: Vec = state + .storage + .list_buckets() + .await + .map(|list| list.into_iter().map(|b| b.name).collect()) + .unwrap_or_default(); + Json(json!({ "buckets": buckets })).into_response() +} + +pub async fn json_not_implemented() -> Response { + json_error( + StatusCode::NOT_IMPLEMENTED, + "This feature is not implemented yet", + ) +} + +#[derive(Deserialize)] +pub struct ConnectionTestPayload { + pub endpoint_url: String, + pub access_key: String, + pub secret_key: String, + #[serde(default = "default_region")] + pub region: String, +} + +fn default_region() -> String { + "us-east-1".to_string() +} + +pub async fn test_connection( + State(state): State, + Extension(_session): Extension, + body: Body, +) -> Response { + let payload: ConnectionTestPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(_) => { + return ( + StatusCode::BAD_REQUEST, + Json(json!({ + "status": "error", + "message": "Invalid JSON payload", + })), + ) + .into_response() + } + }; + + if payload.endpoint_url.trim().is_empty() + || payload.access_key.trim().is_empty() + || payload.secret_key.trim().is_empty() + { + return ( + StatusCode::BAD_REQUEST, + Json(json!({ + "status": "error", + "message": "Missing credentials", + })), + ) + .into_response(); + } + + let connection = RemoteConnection { + id: "test".to_string(), + name: "Test".to_string(), + endpoint_url: payload.endpoint_url.trim().to_string(), + access_key: payload.access_key.trim().to_string(), + secret_key: payload.secret_key.trim().to_string(), + region: payload.region.trim().to_string(), + }; + + if state.replication.check_endpoint(&connection).await { + Json(json!({ + "status": "ok", + "message": "Connection successful", + })) + .into_response() + } else { + ( + StatusCode::BAD_REQUEST, + Json(json!({ + "status": "error", + "message": format!("Connection failed or endpoint is unreachable: {}", connection.endpoint_url), + })), + ) + .into_response() + } +} + +pub async fn connection_health( + State(state): State, + Extension(_session): Extension, + Path(connection_id): Path, +) -> Response { + let Some(connection) = state.connections.get(&connection_id) else { + return ( + StatusCode::NOT_FOUND, + Json(json!({ + "healthy": false, + "error": "Connection not found", + })), + ) + .into_response(); + }; + + let healthy = state.replication.check_endpoint(&connection).await; + Json(json!({ + "healthy": healthy, + "error": if healthy { + Value::Null + } else { + Value::String(format!("Cannot reach endpoint: {}", connection.endpoint_url)) + } + })) + .into_response() +} + +async fn peer_health_payload(state: &AppState, site_id: &str) -> Result { + let Some(registry) = &state.site_registry else { + return Err(json_error( + StatusCode::NOT_FOUND, + "Site registry not available", + )); + }; + let Some(peer) = registry.get_peer(site_id) else { + return Err(json_error(StatusCode::NOT_FOUND, "Peer not found")); + }; + + let checked_at = chrono::Utc::now().timestamp_millis() as f64 / 1000.0; + let mut healthy = false; + let mut error: Option = None; + + if let Some(connection_id) = peer.connection_id.as_deref() { + if let Some(connection) = state.connections.get(connection_id) { + healthy = state.replication.check_endpoint(&connection).await; + if !healthy { + error = Some(format!( + "Cannot reach endpoint: {}", + connection.endpoint_url + )); + } + } else { + error = Some(format!("Connection '{}' not found", connection_id)); + } + } else { + error = Some("No connection configured for this peer".to_string()); + } + + registry.update_health(site_id, healthy); + Ok(json!({ + "site_id": site_id, + "is_healthy": healthy, + "checked_at": checked_at, + "error": error, + })) +} + +pub async fn peer_health( + State(state): State, + Extension(_session): Extension, + Path(site_id): Path, +) -> Response { + match peer_health_payload(&state, &site_id).await { + Ok(payload) => Json(payload).into_response(), + Err(response) => response, + } +} + +pub async fn peer_sync_stats( + State(state): State, + Extension(_session): Extension, + Path(site_id): Path, +) -> Response { + let Some(registry) = &state.site_registry else { + return json_error(StatusCode::NOT_FOUND, "Site registry not available"); + }; + let Some(peer) = registry.get_peer(&site_id) else { + return json_error(StatusCode::NOT_FOUND, "Peer not found"); + }; + let Some(connection_id) = peer.connection_id.as_deref() else { + return json_error(StatusCode::BAD_REQUEST, "No connection configured"); + }; + + let rules = state.replication.list_rules(); + let mut buckets: Vec = Vec::new(); + let mut buckets_syncing = 0u64; + let mut objects_synced = 0u64; + let mut objects_pending = 0u64; + let mut objects_failed = 0u64; + let mut bytes_synced = 0u64; + let mut last_sync_at: Option = None; + + for rule in rules + .into_iter() + .filter(|rule| rule.target_connection_id == connection_id) + { + buckets_syncing += 1; + objects_synced += rule.stats.objects_synced; + objects_pending += rule.stats.objects_pending; + bytes_synced += rule.stats.bytes_synced; + if let Some(sync_at) = rule.stats.last_sync_at { + if last_sync_at + .map(|current| sync_at > current) + .unwrap_or(true) + { + last_sync_at = Some(sync_at); + } + } + + let failures = state.replication.get_failure_count(&rule.bucket_name) as u64; + objects_failed += failures; + buckets.push(json!({ + "bucket_name": rule.bucket_name, + "target_bucket": rule.target_bucket, + "mode": rule.mode, + "enabled": rule.enabled, + "last_sync_at": rule.stats.last_sync_at, + "objects_synced": rule.stats.objects_synced, + "objects_pending": rule.stats.objects_pending, + "failures": failures, + })); + } + + Json(json!({ + "buckets_syncing": buckets_syncing, + "objects_synced": objects_synced, + "objects_pending": objects_pending, + "objects_failed": objects_failed, + "bytes_synced": bytes_synced, + "last_sync_at": last_sync_at, + "buckets": buckets, + })) + .into_response() +} + +pub async fn peer_bidirectional_status( + State(state): State, + Extension(_session): Extension, + Path(site_id): Path, +) -> Response { + let Some(registry) = &state.site_registry else { + return json_error(StatusCode::NOT_FOUND, "Site registry not available"); + }; + let Some(peer) = registry.get_peer(&site_id) else { + return json_error(StatusCode::NOT_FOUND, "Peer not found"); + }; + + let local_site = registry.get_local_site(); + let local_bidirectional_rules: Vec = state + .replication + .list_rules() + .into_iter() + .filter(|rule| { + peer.connection_id + .as_deref() + .map(|connection_id| rule.target_connection_id == connection_id) + .unwrap_or(false) + && rule.mode == crate::services::replication::MODE_BIDIRECTIONAL + }) + .map(|rule| { + json!({ + "bucket_name": rule.bucket_name, + "target_bucket": rule.target_bucket, + "enabled": rule.enabled, + }) + }) + .collect(); + + let mut result = json!({ + "site_id": site_id, + "local_site_id": local_site.as_ref().map(|site| site.site_id.clone()), + "local_endpoint": local_site.as_ref().map(|site| site.endpoint.clone()), + "local_bidirectional_rules": local_bidirectional_rules, + "local_site_sync_enabled": state.config.site_sync_enabled, + "remote_status": Value::Null, + "issues": Vec::::new(), + "is_fully_configured": false, + }); + + if local_site + .as_ref() + .map(|site| site.site_id.trim().is_empty()) + .unwrap_or(true) + { + push_issue( + &mut result, + json!({ + "code": "NO_LOCAL_SITE_ID", + "message": "Local site identity not configured", + "severity": "error", + }), + ); + } + if local_site + .as_ref() + .map(|site| site.endpoint.trim().is_empty()) + .unwrap_or(true) + { + push_issue( + &mut result, + json!({ + "code": "NO_LOCAL_ENDPOINT", + "message": "Local site endpoint not configured (remote site cannot reach back)", + "severity": "error", + }), + ); + } + + let Some(connection_id) = peer.connection_id.as_deref() else { + push_issue( + &mut result, + json!({ + "code": "NO_CONNECTION", + "message": "No connection configured for this peer", + "severity": "error", + }), + ); + return Json(result).into_response(); + }; + + let Some(connection) = state.connections.get(connection_id) else { + push_issue( + &mut result, + json!({ + "code": "CONNECTION_NOT_FOUND", + "message": format!("Connection '{}' not found", connection_id), + "severity": "error", + }), + ); + return Json(result).into_response(); + }; + + if result["local_bidirectional_rules"] + .as_array() + .map(|rules| rules.is_empty()) + .unwrap_or(true) + { + push_issue( + &mut result, + json!({ + "code": "NO_LOCAL_BIDIRECTIONAL_RULES", + "message": "No bidirectional replication rules configured on this site", + "severity": "warning", + }), + ); + } + if !state.config.site_sync_enabled { + push_issue( + &mut result, + json!({ + "code": "SITE_SYNC_DISABLED", + "message": "Site sync worker is disabled (SITE_SYNC_ENABLED=false). Pull operations will not work.", + "severity": "warning", + }), + ); + } + if !state.replication.check_endpoint(&connection).await { + push_issue( + &mut result, + json!({ + "code": "REMOTE_UNREACHABLE", + "message": "Remote endpoint is not reachable", + "severity": "error", + }), + ); + return Json(result).into_response(); + } + + let admin_url = format!( + "{}/admin/sites", + connection.endpoint_url.trim_end_matches('/') + ); + match reqwest::Client::new() + .get(&admin_url) + .header("accept", "application/json") + .header("x-access-key", &connection.access_key) + .header("x-secret-key", &connection.secret_key) + .timeout(std::time::Duration::from_secs(10)) + .send() + .await + { + Ok(resp) if resp.status().is_success() => match resp.json::().await { + Ok(remote_data) => { + let remote_local = remote_data.get("local").cloned().unwrap_or(Value::Null); + let remote_peers = remote_data + .get("peers") + .and_then(|value| value.as_array()) + .cloned() + .unwrap_or_default(); + let mut has_peer_for_us = false; + let mut peer_connection_configured = false; + + for remote_peer in &remote_peers { + let matches_site = local_site + .as_ref() + .map(|site| { + remote_peer.get("site_id").and_then(|v| v.as_str()) + == Some(site.site_id.as_str()) + || remote_peer.get("endpoint").and_then(|v| v.as_str()) + == Some(site.endpoint.as_str()) + }) + .unwrap_or(false); + if matches_site { + has_peer_for_us = true; + peer_connection_configured = remote_peer + .get("connection_id") + .and_then(|v| v.as_str()) + .map(|v| !v.trim().is_empty()) + .unwrap_or(false); + break; + } + } + + result["remote_status"] = json!({ + "reachable": true, + "local_site": remote_local, + "site_sync_enabled": Value::Null, + "has_peer_for_us": has_peer_for_us, + "peer_connection_configured": peer_connection_configured, + "has_bidirectional_rules_for_us": Value::Null, + }); + + if !has_peer_for_us { + push_issue( + &mut result, + json!({ + "code": "REMOTE_NO_PEER_FOR_US", + "message": "Remote site does not have this site registered as a peer", + "severity": "error", + }), + ); + } else if !peer_connection_configured { + push_issue( + &mut result, + json!({ + "code": "REMOTE_NO_CONNECTION_FOR_US", + "message": "Remote site has us as peer but no connection configured (cannot push back)", + "severity": "error", + }), + ); + } + } + Err(_) => { + result["remote_status"] = json!({ + "reachable": true, + "invalid_response": true, + }); + push_issue( + &mut result, + json!({ + "code": "REMOTE_INVALID_RESPONSE", + "message": "Remote admin API returned invalid JSON", + "severity": "warning", + }), + ); + } + }, + Ok(resp) + if resp.status() == StatusCode::UNAUTHORIZED + || resp.status() == StatusCode::FORBIDDEN => + { + result["remote_status"] = json!({ + "reachable": true, + "admin_access_denied": true, + }); + push_issue( + &mut result, + json!({ + "code": "REMOTE_ADMIN_ACCESS_DENIED", + "message": "Cannot verify remote configuration (admin access denied)", + "severity": "warning", + }), + ); + } + Ok(resp) => { + result["remote_status"] = json!({ + "reachable": true, + "admin_api_error": resp.status().as_u16(), + }); + push_issue( + &mut result, + json!({ + "code": "REMOTE_ADMIN_API_ERROR", + "message": format!("Remote admin API returned status {}", resp.status().as_u16()), + "severity": "warning", + }), + ); + } + Err(_) => { + result["remote_status"] = json!({ + "reachable": false, + "error": "Connection failed", + }); + push_issue( + &mut result, + json!({ + "code": "REMOTE_ADMIN_UNREACHABLE", + "message": "Could not reach remote admin API", + "severity": "warning", + }), + ); + } + } + + let has_errors = result["issues"] + .as_array() + .map(|items| { + items.iter().any(|issue| { + issue.get("severity").and_then(|value| value.as_str()) == Some("error") + }) + }) + .unwrap_or(true); + result["is_fully_configured"] = json!( + !has_errors + && result["local_bidirectional_rules"] + .as_array() + .map(|rules| !rules.is_empty()) + .unwrap_or(false) + ); + + Json(result).into_response() +} + +#[derive(Clone, Copy)] +struct MetricsSettingsSnapshot { + enabled: bool, + retention_hours: u64, + interval_minutes: u64, +} + +static METRICS_SETTINGS: OnceLock> = OnceLock::new(); + +fn metrics_settings_snapshot(state: &AppState) -> MetricsSettingsSnapshot { + *METRICS_SETTINGS + .get_or_init(|| { + Mutex::new(MetricsSettingsSnapshot { + enabled: state.config.metrics_history_enabled, + retention_hours: state.config.metrics_history_retention_hours, + interval_minutes: state.config.metrics_history_interval_minutes, + }) + }) + .lock() + .unwrap() +} + +pub async fn metrics_settings(State(state): State) -> Response { + let settings = metrics_settings_snapshot(&state); + Json(json!({ + "enabled": settings.enabled, + "retention_hours": settings.retention_hours, + "interval_minutes": settings.interval_minutes, + })) + .into_response() +} + +pub async fn update_metrics_settings(State(state): State, body: Body) -> Response { + let payload: Value = parse_json_body(body).await.unwrap_or_else(|_| json!({})); + let mut settings = METRICS_SETTINGS + .get_or_init(|| { + Mutex::new(MetricsSettingsSnapshot { + enabled: state.config.metrics_history_enabled, + retention_hours: state.config.metrics_history_retention_hours, + interval_minutes: state.config.metrics_history_interval_minutes, + }) + }) + .lock() + .unwrap(); + let enabled = payload + .get("enabled") + .and_then(|value| value.as_bool()) + .unwrap_or(settings.enabled); + let retention_hours = payload + .get("retention_hours") + .and_then(|value| value.as_u64()) + .unwrap_or(settings.retention_hours) + .max(1); + let interval_minutes = payload + .get("interval_minutes") + .and_then(|value| value.as_u64()) + .unwrap_or(settings.interval_minutes) + .max(1); + *settings = MetricsSettingsSnapshot { + enabled, + retention_hours, + interval_minutes, + }; + + Json(json!({ + "enabled": enabled, + "retention_hours": retention_hours, + "interval_minutes": interval_minutes, + })) + .into_response() +} + +#[derive(Deserialize, Default)] +struct MultipartInitPayload { + #[serde(default)] + object_key: String, + #[serde(default)] + metadata: Option>, +} + +pub async fn upload_object( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + headers: HeaderMap, + body: Body, +) -> Response { + let content_type = match headers + .get(header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()) + { + Some(value) + if value + .to_ascii_lowercase() + .starts_with("multipart/form-data") => + { + value.to_string() + } + _ => return json_error(StatusCode::BAD_REQUEST, "Expected multipart form upload"), + }; + + let boundary = match multer::parse_boundary(&content_type) { + Ok(value) => value, + Err(_) => return json_error(StatusCode::BAD_REQUEST, "Missing multipart boundary"), + }; + + let stream = BodyStream::new(body) + .map_ok(|frame| frame.into_data().unwrap_or_default()) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e)); + let mut multipart = multer::Multipart::new(stream, boundary); + + let mut object_key: Option = None; + let mut metadata_raw: Option = None; + let mut file_name: Option = None; + let mut file_content_type: Option = None; + let mut file_bytes: Option> = None; + + while let Some(field) = match multipart.next_field().await { + Ok(field) => field, + Err(e) => { + return json_error( + StatusCode::BAD_REQUEST, + format!("Malformed multipart body: {}", e), + ) + } + } { + let name = field.name().unwrap_or_default().to_string(); + match name.as_str() { + "object_key" => match field.text().await { + Ok(value) if !value.trim().is_empty() => { + object_key = Some(value.trim().to_string()) + } + _ => {} + }, + "metadata" => match field.text().await { + Ok(value) if !value.trim().is_empty() => metadata_raw = Some(value), + _ => {} + }, + "object" => { + file_name = field.file_name().map(|s| s.to_string()); + file_content_type = field.content_type().map(|mime| mime.to_string()); + match field.bytes().await { + Ok(bytes) => file_bytes = Some(bytes.to_vec()), + Err(e) => { + return json_error( + StatusCode::BAD_REQUEST, + format!("Failed to read upload: {}", e), + ) + } + } + } + _ => { + let _ = field.bytes().await; + } + } + } + + let bytes = match file_bytes { + Some(bytes) if !bytes.is_empty() => bytes, + _ => return json_error(StatusCode::BAD_REQUEST, "Choose a file to upload"), + }; + + let key = object_key + .or(file_name.clone()) + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) + .ok_or_else(|| json_error(StatusCode::BAD_REQUEST, "Object key is required")); + let key = match key { + Ok(key) => key, + Err(response) => return response, + }; + + let metadata = if let Some(raw) = metadata_raw { + match serde_json::from_str::>(&raw) { + Ok(map) => Some( + map.into_iter() + .map(|(k, v)| (k, v.as_str().unwrap_or(&v.to_string()).to_string())) + .collect::>(), + ), + Err(_) => return json_error(StatusCode::BAD_REQUEST, "Metadata must be a JSON object"), + } + } else { + None + }; + + let mut upload_headers = HeaderMap::new(); + if let Some(content_type) = file_content_type.as_deref() { + if let Ok(value) = content_type.parse() { + upload_headers.insert(header::CONTENT_TYPE, value); + } + } + if let Some(metadata) = &metadata { + for (key, value) in metadata { + let header_name = format!("x-amz-meta-{}", key); + if let Ok(name) = header_name.parse::() { + if let Ok(value) = value.parse() { + upload_headers.insert(name, value); + } + } + } + } + + let response = handlers::put_object( + State(state), + Path((bucket_name.clone(), key.clone())), + Query(ObjectQuery::default()), + upload_headers, + Body::from(bytes), + ) + .await; + + if !response.status().is_success() { + return response; + } + + let mut message = format!("Uploaded '{}'", key); + if metadata.is_some() { + message.push_str(" with metadata"); + } + json_ok(json!({ + "status": "ok", + "message": message, + "key": key, + })) +} + +pub async fn initiate_multipart_upload( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + body: Body, +) -> Response { + let payload: MultipartInitPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + let object_key = payload.object_key.trim(); + if object_key.is_empty() { + return json_error(StatusCode::BAD_REQUEST, "object_key is required"); + } + + match state + .storage + .initiate_multipart(&bucket_name, object_key, payload.metadata) + .await + { + Ok(upload_id) => json_ok(json!({ "upload_id": upload_id })), + Err(err) => storage_json_error(err), + } +} + +#[derive(Deserialize, Default)] +pub struct MultipartPartQuery { + #[serde(rename = "partNumber")] + part_number: Option, +} + +pub async fn upload_multipart_part( + State(state): State, + Extension(_session): Extension, + Path((bucket_name, upload_id)): Path<(String, String)>, + Query(query): Query, + body: Body, +) -> Response { + let Some(part_number) = query.part_number else { + return json_error(StatusCode::BAD_REQUEST, "partNumber is required"); + }; + if !(1..=10_000).contains(&part_number) { + return json_error( + StatusCode::BAD_REQUEST, + "partNumber must be between 1 and 10000", + ); + } + + let bytes = match to_bytes(body, usize::MAX).await { + Ok(bytes) if !bytes.is_empty() => bytes, + Ok(_) => return json_error(StatusCode::BAD_REQUEST, "Empty request body"), + Err(_) => return json_error(StatusCode::BAD_REQUEST, "Failed to read request body"), + }; + let reader: myfsio_storage::traits::AsyncReadStream = Box::pin(Cursor::new(bytes.to_vec())); + match state + .storage + .upload_part(&bucket_name, &upload_id, part_number, reader) + .await + { + Ok(etag) => json_ok(json!({ "etag": etag, "part_number": part_number })), + Err(err) => storage_json_error(err), + } +} + +#[derive(Deserialize, Default)] +struct CompleteMultipartPayload { + #[serde(default)] + parts: Vec, +} + +#[derive(Deserialize, Default)] +struct CompleteMultipartPartPayload { + #[serde(default, alias = "PartNumber")] + part_number: u32, + #[serde(default, alias = "ETag")] + etag: String, +} + +pub async fn complete_multipart_upload( + State(state): State, + Extension(_session): Extension, + Path((bucket_name, upload_id)): Path<(String, String)>, + body: Body, +) -> Response { + let payload: CompleteMultipartPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + if payload.parts.is_empty() { + return json_error(StatusCode::BAD_REQUEST, "parts array required"); + } + + let parts = payload + .parts + .iter() + .map(|part| PartInfo { + part_number: part.part_number, + etag: part.etag.trim_matches('"').to_string(), + }) + .collect::>(); + + match state + .storage + .complete_multipart(&bucket_name, &upload_id, &parts) + .await + { + Ok(meta) => json_ok(json!({ + "key": meta.key, + "size": meta.size, + "etag": meta.etag.unwrap_or_default(), + "last_modified": meta.last_modified.to_rfc3339(), + })), + Err(err) => storage_json_error(err), + } +} + +pub async fn abort_multipart_upload( + State(state): State, + Extension(_session): Extension, + Path((bucket_name, upload_id)): Path<(String, String)>, +) -> Response { + match state + .storage + .abort_multipart(&bucket_name, &upload_id) + .await + { + Ok(()) => json_ok(json!({ "status": "aborted" })), + Err(err) => storage_json_error(err), + } +} + +async fn get_bucket_config_json( + state: &AppState, + bucket: &str, +) -> Result { + state.storage.get_bucket_config(bucket).await +} + +pub async fn bucket_acl( + State(state): State, + Extension(session): Extension, + Path(bucket_name): Path, +) -> Response { + match get_bucket_config_json(&state, &bucket_name).await { + Ok(config) => Json(parse_acl_value( + config.acl.as_ref(), + &owner_id_or_default(&session), + )) + .into_response(), + Err(err) => storage_json_error(err), + } +} + +#[derive(Deserialize, Default)] +struct BucketAclPayload { + #[serde(default)] + canned_acl: String, +} + +pub async fn update_bucket_acl( + State(state): State, + Extension(session): Extension, + Path(bucket_name): Path, + body: Body, +) -> Response { + let payload: BucketAclPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + if payload.canned_acl.trim().is_empty() { + return json_error(StatusCode::BAD_REQUEST, "canned_acl is required"); + } + + let acl_xml = match bucket_acl_xml_for_canned( + &owner_id_or_default(&session), + payload.canned_acl.trim(), + ) { + Ok(xml) => xml, + Err(message) => return json_error(StatusCode::BAD_REQUEST, message), + }; + + match state.storage.get_bucket_config(&bucket_name).await { + Ok(mut config) => { + config.acl = Some(Value::String(acl_xml)); + match state.storage.set_bucket_config(&bucket_name, &config).await { + Ok(()) => json_ok(json!({ + "status": "ok", + "message": format!("ACL set to {}", payload.canned_acl.trim()), + })), + Err(err) => storage_json_error(err), + } + } + Err(err) => storage_json_error(err), + } +} + +pub async fn bucket_cors( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, +) -> Response { + match get_bucket_config_json(&state, &bucket_name).await { + Ok(config) => Json(parse_cors_value(config.cors.as_ref())).into_response(), + Err(err) => storage_json_error(err), + } +} + +#[derive(Deserialize, Default)] +struct BucketCorsPayload { + #[serde(default)] + rules: Vec, +} + +pub async fn update_bucket_cors( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + body: Body, +) -> Response { + let payload: BucketCorsPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + match state.storage.get_bucket_config(&bucket_name).await { + Ok(mut config) => { + config.cors = if payload.rules.is_empty() { + None + } else { + Some(Value::String(cors_xml_from_rules(&payload.rules))) + }; + match state.storage.set_bucket_config(&bucket_name, &config).await { + Ok(()) => json_ok(json!({ + "status": "ok", + "message": "CORS configuration saved", + "rules": payload.rules, + })), + Err(err) => storage_json_error(err), + } + } + Err(err) => storage_json_error(err), + } +} + +pub async fn bucket_lifecycle( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, +) -> Response { + match get_bucket_config_json(&state, &bucket_name).await { + Ok(config) => Json(parse_lifecycle_value(config.lifecycle.as_ref())).into_response(), + Err(err) => storage_json_error(err), + } +} + +#[derive(Deserialize, Default)] +struct BucketLifecyclePayload { + #[serde(default)] + rules: Vec, +} + +pub async fn update_bucket_lifecycle( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + body: Body, +) -> Response { + let payload: BucketLifecyclePayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + match state.storage.get_bucket_config(&bucket_name).await { + Ok(mut config) => { + config.lifecycle = if payload.rules.is_empty() { + None + } else { + Some(Value::String(lifecycle_xml_from_rules(&payload.rules))) + }; + match state.storage.set_bucket_config(&bucket_name, &config).await { + Ok(()) => json_ok(json!({ + "status": "ok", + "message": "Lifecycle rules saved", + "rules": payload.rules, + })), + Err(err) => storage_json_error(err), + } + } + Err(err) => storage_json_error(err), + } +} + +async fn serve_object_download_or_preview( + state: AppState, + bucket: String, + key: String, + headers: HeaderMap, + is_download: bool, +) -> Response { + let content_type = state + .storage + .head_object(&bucket, &key) + .await + .ok() + .and_then(|meta| meta.content_type); + + let mut query = ObjectQuery::default(); + if is_download { + query.response_content_disposition = Some(format!( + "attachment; filename=\"{}\"", + safe_attachment_filename(&key) + )); + } else if let Some(forced) = dangerous_preview_content_type(content_type.as_deref(), &key) { + query.response_content_type = Some(forced); + } + + let mut response = + handlers::get_object(State(state), Path((bucket, key)), Query(query), headers).await; + response + .headers_mut() + .insert("x-content-type-options", "nosniff".parse().unwrap()); + response +} + +async fn object_metadata_json(state: &AppState, bucket: &str, key: &str) -> Response { + let head = match state.storage.head_object(bucket, key).await { + Ok(meta) => meta, + Err(err) => return storage_json_error(err), + }; + let metadata = state + .storage + .get_object_metadata(bucket, key) + .await + .unwrap_or_default(); + + let mut out = metadata.clone(); + if let Some(content_type) = head.content_type { + out.insert("Content-Type".to_string(), content_type); + } + let display_length = metadata + .get("__size__") + .cloned() + .unwrap_or_else(|| head.size.to_string()); + out.insert("Content-Length".to_string(), display_length); + if let Some(algorithm) = metadata.get("x-amz-server-side-encryption") { + out.insert( + "x-amz-server-side-encryption".to_string(), + algorithm.to_string(), + ); + } + Json(json!({ "metadata": out })).into_response() +} + +async fn object_versions_json(state: &AppState, bucket: &str, key: &str) -> Response { + match read_version_manifests_for_object(state, bucket, key) { + Ok(entries) => Json(json!({ + "versions": entries.into_iter().map(|entry| manifest_to_json(&entry)).collect::>(), + })) + .into_response(), + Err(err) => json_error(StatusCode::BAD_REQUEST, err), + } +} + +async fn object_tags_json(state: &AppState, bucket: &str, key: &str) -> Response { + match state.storage.get_object_tags(bucket, key).await { + Ok(tags) => Json(json!({ + "tags": tags.into_iter().map(|tag| json!({ "Key": tag.key, "Value": tag.value })).collect::>(), + })) + .into_response(), + Err(err) => storage_json_error(err), + } +} + +#[derive(Deserialize, Default)] +struct PresignPayload { + #[serde(default = "default_presign_method")] + method: String, + #[serde(default)] + expires_in: Option, +} + +fn default_presign_method() -> String { + "GET".to_string() +} + +async fn object_presign_json( + state: &AppState, + session: &SessionHandle, + bucket: &str, + key: &str, + body: Body, +) -> Response { + let payload: PresignPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + let method = payload.method.trim().to_ascii_uppercase(); + if !matches!(method.as_str(), "GET" | "PUT" | "DELETE") { + return json_error( + StatusCode::BAD_REQUEST, + "Method must be GET, PUT, or DELETE", + ); + } + + let access_key = match current_access_key(session) { + Some(key) => key, + None => return json_error(StatusCode::FORBIDDEN, "Missing authenticated session"), + }; + let secret_key = match state.iam.get_secret_key(&access_key) { + Some(secret) => secret, + None => { + return json_error( + StatusCode::FORBIDDEN, + "Session credentials are no longer valid", + ) + } + }; + + let min_expiry = state.config.presigned_url_min_expiry; + let max_expiry = state.config.presigned_url_max_expiry; + let expires = payload + .expires_in + .unwrap_or(900) + .clamp(min_expiry, max_expiry); + + let api_base = parse_api_base(state); + let parsed = match reqwest::Url::parse(&api_base) { + Ok(url) => url, + Err(err) => { + return json_error( + StatusCode::INTERNAL_SERVER_ERROR, + format!("Invalid API_BASE_URL: {}", err), + ) + } + }; + let host = match parsed.host_str() { + Some(host) => { + if let Some(port) = parsed.port() { + format!("{}:{}", host, port) + } else { + host.to_string() + } + } + None => { + return json_error( + StatusCode::INTERNAL_SERVER_ERROR, + "Unable to determine API host", + ) + } + }; + + let now = Utc::now(); + let amz_date = now.format("%Y%m%dT%H%M%SZ").to_string(); + let date_stamp = now.format("%Y%m%d").to_string(); + let region = state.config.region.as_str(); + let credential = format!("{}/{}/{}/s3/aws4_request", access_key, date_stamp, region); + + let canonical_uri = format!("/{}/{}", bucket, encode_object_key(key)); + let mut query_params = vec![ + ( + "X-Amz-Algorithm".to_string(), + "AWS4-HMAC-SHA256".to_string(), + ), + ("X-Amz-Credential".to_string(), credential.clone()), + ("X-Amz-Date".to_string(), amz_date.clone()), + ("X-Amz-Expires".to_string(), expires.to_string()), + ("X-Amz-SignedHeaders".to_string(), "host".to_string()), + ]; + query_params.sort_by(|a, b| a.0.cmp(&b.0).then_with(|| a.1.cmp(&b.1))); + + let canonical_query = query_params + .iter() + .map(|(name, value)| format!("{}={}", aws_query_encode(name), aws_query_encode(value))) + .collect::>() + .join("&"); + let canonical_request = format!( + "{}\n{}\n{}\nhost:{}\n\nhost\nUNSIGNED-PAYLOAD", + method, canonical_uri, canonical_query, host + ); + let scope = format!("{}/{}/s3/aws4_request", date_stamp, region); + let string_to_sign = sigv4::build_string_to_sign(&amz_date, &scope, &canonical_request); + let signing_key = sigv4::derive_signing_key(&secret_key, &date_stamp, region, "s3"); + let signature = sigv4::compute_signature(&signing_key, &string_to_sign); + + let final_query = format!("{}&X-Amz-Signature={}", canonical_query, signature); + let final_url = format!("{}{}?{}", api_base, canonical_uri, final_query); + + Json(json!({ + "url": final_url, + "method": method, + "expires_in": expires, + })) + .into_response() +} + +#[derive(Deserialize, Default)] +struct ObjectTagsPayload { + #[serde(default)] + tags: Vec, +} + +#[derive(Deserialize, Default)] +struct ObjectTagPayload { + #[serde(default, alias = "Key", alias = "key")] + key: String, + #[serde(default, alias = "Value", alias = "value")] + value: String, +} + +async fn update_object_tags(state: &AppState, bucket: &str, key: &str, body: Body) -> Response { + let payload: ObjectTagsPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + if payload.tags.len() > 50 { + return json_error(StatusCode::BAD_REQUEST, "Maximum 50 tags allowed"); + } + + let tags = payload + .tags + .iter() + .filter(|tag| !tag.key.trim().is_empty()) + .map(|tag| Tag { + key: tag.key.trim().to_string(), + value: tag.value.to_string(), + }) + .collect::>(); + + let result = if tags.is_empty() { + state.storage.delete_object_tags(bucket, key).await + } else { + state.storage.set_object_tags(bucket, key, &tags).await + }; + + match result { + Ok(()) => Json(json!({ + "status": "ok", + "message": "Tags saved", + "tags": tags.into_iter().map(|tag| json!({ "Key": tag.key, "Value": tag.value })).collect::>(), + })) + .into_response(), + Err(err) => storage_json_error(err), + } +} + +#[derive(Deserialize, Default)] +struct CopyMovePayload { + #[serde(default)] + dest_bucket: String, + #[serde(default)] + dest_key: String, +} + +async fn copy_object_json(state: &AppState, bucket: &str, key: &str, body: Body) -> Response { + let payload: CopyMovePayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + let dest_bucket = payload.dest_bucket.trim(); + let dest_key = payload.dest_key.trim(); + if dest_bucket.is_empty() || dest_key.is_empty() { + return json_error( + StatusCode::BAD_REQUEST, + "dest_bucket and dest_key are required", + ); + } + + match state + .storage + .copy_object(bucket, key, dest_bucket, dest_key) + .await + { + Ok(_) => Json(json!({ + "status": "ok", + "message": format!("Copied to {}/{}", dest_bucket, dest_key), + "dest_bucket": dest_bucket, + "dest_key": dest_key, + })) + .into_response(), + Err(err) => storage_json_error(err), + } +} + +async fn move_object_json(state: &AppState, bucket: &str, key: &str, body: Body) -> Response { + let payload: CopyMovePayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + let dest_bucket = payload.dest_bucket.trim(); + let dest_key = payload.dest_key.trim(); + if dest_bucket.is_empty() || dest_key.is_empty() { + return json_error( + StatusCode::BAD_REQUEST, + "dest_bucket and dest_key are required", + ); + } + if dest_bucket == bucket && dest_key == key { + return json_error( + StatusCode::BAD_REQUEST, + "Cannot move object to the same location", + ); + } + + match state.storage.copy_object(bucket, key, dest_bucket, dest_key).await { + Ok(_) => match state.storage.delete_object(bucket, key).await { + Ok(()) => Json(json!({ + "status": "ok", + "message": format!("Moved to {}/{}", dest_bucket, dest_key), + "dest_bucket": dest_bucket, + "dest_key": dest_key, + })) + .into_response(), + Err(_) => Json(json!({ + "status": "partial", + "message": format!("Copied to {}/{} but failed to delete source", dest_bucket, dest_key), + "dest_bucket": dest_bucket, + "dest_key": dest_key, + })) + .into_response(), + }, + Err(err) => storage_json_error(err), + } +} + +async fn purge_object_versions_for_key( + state: &AppState, + bucket: &str, + key: &str, +) -> Result<(), String> { + if let Ok(version_dir) = version_dir_for_object(state, bucket, key) { + if version_dir.exists() { + std::fs::remove_dir_all(&version_dir).map_err(|e| e.to_string())?; + } + } + Ok(()) +} + +async fn delete_object_json( + state: &AppState, + bucket: &str, + key: &str, + headers: &HeaderMap, + body: Body, +) -> Response { + let body_bytes = match to_bytes(body, usize::MAX).await { + Ok(bytes) => bytes, + Err(_) => return json_error(StatusCode::BAD_REQUEST, "Failed to read request body"), + }; + + let content_type = headers + .get(header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()) + .unwrap_or_default(); + let form = if content_type.starts_with("application/x-www-form-urlencoded") { + parse_form_body(&body_bytes) + } else { + HashMap::new() + }; + let purge_versions = parse_bool_flag(form.get("purge_versions").map(|s| s.as_str())); + + if purge_versions { + if let Err(err) = state.storage.delete_object(bucket, key).await { + return storage_json_error(err); + } + if let Err(err) = purge_object_versions_for_key(state, bucket, key).await { + return json_error(StatusCode::BAD_REQUEST, err); + } + return Json(json!({ + "status": "ok", + "message": format!("Permanently deleted '{}' and all versions", key), + })) + .into_response(); + } + + match state.storage.delete_object(bucket, key).await { + Ok(()) => Json(json!({ + "status": "ok", + "message": format!("Deleted '{}'", key), + })) + .into_response(), + Err(err) => storage_json_error(err), + } +} + +async fn restore_object_version_json( + state: &AppState, + bucket: &str, + key: &str, + version_id: &str, +) -> Response { + let version_dir = match version_dir_for_object(state, bucket, key) { + Ok(path) => path, + Err(err) => return json_error(StatusCode::BAD_REQUEST, err), + }; + let data_path = version_dir.join(format!("{}.bin", version_id)); + let meta_path = version_dir.join(format!("{}.json", version_id)); + if !data_path.exists() || !meta_path.exists() { + return json_error(StatusCode::NOT_FOUND, "Version not found"); + } + + let manifest_text = match std::fs::read_to_string(&meta_path) { + Ok(text) => text, + Err(err) => return json_error(StatusCode::BAD_REQUEST, err.to_string()), + }; + let manifest: VersionManifest = match serde_json::from_str(&manifest_text) { + Ok(manifest) => manifest, + Err(err) => return json_error(StatusCode::BAD_REQUEST, err.to_string()), + }; + + let live_exists = state.storage.head_object(bucket, key).await.is_ok(); + let versioning_enabled = state + .storage + .is_versioning_enabled(bucket) + .await + .unwrap_or(false); + if live_exists { + if let Err(err) = state.storage.delete_object(bucket, key).await { + return storage_json_error(err); + } + } + + let destination = match object_live_path(state, bucket, key) { + Ok(path) => path, + Err(err) => return json_error(StatusCode::BAD_REQUEST, err), + }; + if let Some(parent) = destination.parent() { + if let Err(err) = tokio::fs::create_dir_all(parent).await { + return json_error(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()); + } + } + if let Err(err) = tokio::fs::copy(&data_path, &destination).await { + return json_error(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()); + } + if let Err(err) = state + .storage + .put_object_metadata(bucket, key, &manifest.metadata) + .await + { + return storage_json_error(err); + } + + let mut message = format!("Restored '{}'", key); + if live_exists && versioning_enabled { + message.push_str(" (previous current version was archived)"); + } + Json(json!({ "status": "ok", "message": message })).into_response() +} + +#[derive(Debug, Clone, Copy)] +enum ObjectGetAction { + Download, + Preview, + Metadata, + Versions, + Tags, +} + +#[derive(Debug, Clone)] +enum ObjectPostAction { + Delete, + Presign, + Tags, + Copy, + Move, + Restore(String), +} + +fn parse_object_get_action(rest: &str) -> Option<(String, ObjectGetAction)> { + for (suffix, action) in [ + ("/download", ObjectGetAction::Download), + ("/preview", ObjectGetAction::Preview), + ("/metadata", ObjectGetAction::Metadata), + ("/versions", ObjectGetAction::Versions), + ("/tags", ObjectGetAction::Tags), + ] { + if let Some(key) = rest.strip_suffix(suffix) { + return Some((key.to_string(), action)); + } + } + None +} + +fn parse_object_post_action(rest: &str) -> Option<(String, ObjectPostAction)> { + if let Some((key, version_id)) = rest.rsplit_once("/restore/") { + return Some(( + key.to_string(), + ObjectPostAction::Restore(version_id.to_string()), + )); + } + for (suffix, action) in [ + ("/delete", ObjectPostAction::Delete), + ("/presign", ObjectPostAction::Presign), + ("/tags", ObjectPostAction::Tags), + ("/copy", ObjectPostAction::Copy), + ("/move", ObjectPostAction::Move), + ] { + if let Some(key) = rest.strip_suffix(suffix) { + return Some((key.to_string(), action)); + } + } + None +} + +pub async fn object_get_dispatch( + State(state): State, + Extension(session): Extension, + Path((bucket_name, rest)): Path<(String, String)>, + headers: HeaderMap, +) -> Response { + let Some((key, action)) = parse_object_get_action(&rest) else { + return json_error(StatusCode::NOT_FOUND, "Unknown object action"); + }; + + match action { + ObjectGetAction::Download => { + serve_object_download_or_preview(state, bucket_name, key, headers, true).await + } + ObjectGetAction::Preview => { + serve_object_download_or_preview(state, bucket_name, key, headers, false).await + } + ObjectGetAction::Metadata => object_metadata_json(&state, &bucket_name, &key).await, + ObjectGetAction::Versions => object_versions_json(&state, &bucket_name, &key).await, + ObjectGetAction::Tags => { + let _ = session; + object_tags_json(&state, &bucket_name, &key).await + } + } +} + +pub async fn object_post_dispatch( + State(state): State, + Extension(session): Extension, + Path((bucket_name, rest)): Path<(String, String)>, + headers: HeaderMap, + body: Body, +) -> Response { + let Some((key, action)) = parse_object_post_action(&rest) else { + return json_error(StatusCode::NOT_FOUND, "Unknown object action"); + }; + + match action { + ObjectPostAction::Delete => { + delete_object_json(&state, &bucket_name, &key, &headers, body).await + } + ObjectPostAction::Presign => { + object_presign_json(&state, &session, &bucket_name, &key, body).await + } + ObjectPostAction::Tags => update_object_tags(&state, &bucket_name, &key, body).await, + ObjectPostAction::Copy => copy_object_json(&state, &bucket_name, &key, body).await, + ObjectPostAction::Move => move_object_json(&state, &bucket_name, &key, body).await, + ObjectPostAction::Restore(version_id) => { + restore_object_version_json(&state, &bucket_name, &key, &version_id).await + } + } +} + +#[derive(Deserialize, Default)] +struct BulkKeysPayload { + #[serde(default)] + keys: Vec, + #[serde(default)] + purge_versions: bool, +} + +async fn expand_bulk_keys( + state: &AppState, + bucket: &str, + keys: &[String], +) -> Result, StorageError> { + let mut expanded = Vec::new(); + for key in keys { + if key.ends_with('/') { + let params = ListParams { + max_keys: 5000, + continuation_token: None, + prefix: Some(key.clone()), + start_after: None, + }; + let objects = state.storage.list_objects(bucket, ¶ms).await?; + for object in objects.objects { + expanded.push(object.key); + } + } else { + expanded.push(key.clone()); + } + } + let mut unique = BTreeMap::new(); + for key in expanded { + unique.entry(key.clone()).or_insert(key); + } + Ok(unique.into_values().collect()) +} + +pub async fn bulk_delete_objects( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + body: Body, +) -> Response { + let payload: BulkKeysPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + let cleaned = payload + .keys + .into_iter() + .map(|key| key.trim().to_string()) + .filter(|key| !key.is_empty()) + .collect::>(); + if cleaned.is_empty() { + return json_error( + StatusCode::BAD_REQUEST, + "Select at least one object to delete", + ); + } + + let keys = match expand_bulk_keys(&state, &bucket_name, &cleaned).await { + Ok(keys) => keys, + Err(err) => return storage_json_error(err), + }; + if keys.is_empty() { + return json_error( + StatusCode::BAD_REQUEST, + "No objects found under the selected folders", + ); + } + + let mut deleted = Vec::new(); + let mut errors = Vec::new(); + + for key in keys { + match state.storage.delete_object(&bucket_name, &key).await { + Ok(()) => { + if payload.purge_versions { + if let Err(err) = + purge_object_versions_for_key(&state, &bucket_name, &key).await + { + errors.push(json!({ "key": key, "error": err })); + continue; + } + } + deleted.push(key); + } + Err(err) => errors.push(json!({ "key": key, "error": err.to_string() })), + } + } + + if deleted.is_empty() && !errors.is_empty() { + return ( + StatusCode::BAD_REQUEST, + Json(json!({ + "status": "error", + "message": "Unable to delete the selected objects", + "deleted": deleted, + "errors": errors, + })), + ) + .into_response(); + } + + let mut message = format!( + "Deleted {} object{}", + deleted.len(), + if deleted.len() == 1 { "" } else { "s" } + ); + if payload.purge_versions && !deleted.is_empty() { + message.push_str(" (including archived versions)"); + } + if !errors.is_empty() { + message.push_str(&format!("; {} failed", errors.len())); + } + + Json(json!({ + "status": if errors.is_empty() { "ok" } else { "partial" }, + "message": message, + "deleted": deleted, + "errors": errors, + })) + .into_response() +} + +pub async fn bulk_download_objects( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + body: Body, +) -> Response { + let payload: BulkKeysPayload = match parse_json_body(body).await { + Ok(payload) => payload, + Err(response) => return response, + }; + + let cleaned = payload + .keys + .into_iter() + .map(|key| key.trim().to_string()) + .filter(|key| !key.is_empty()) + .collect::>(); + if cleaned.is_empty() { + return json_error( + StatusCode::BAD_REQUEST, + "Select at least one object to download", + ); + } + + let keys = match expand_bulk_keys(&state, &bucket_name, &cleaned).await { + Ok(keys) => keys, + Err(err) => return storage_json_error(err), + }; + if keys.is_empty() { + return json_error( + StatusCode::BAD_REQUEST, + "No objects found under the selected folders", + ); + } + + let mut total_bytes = 0u64; + let mut archive_entries = Vec::new(); + for key in keys { + match state.storage.head_object(&bucket_name, &key).await { + Ok(meta) => { + total_bytes = total_bytes.saturating_add(meta.size); + match read_object_bytes_for_zip(&state, &bucket_name, &key).await { + Ok(bytes) => archive_entries.push((key, bytes, meta.last_modified)), + Err(err) => return json_error(StatusCode::BAD_REQUEST, err), + } + } + Err(err) => return storage_json_error(err), + } + } + + let max_total_bytes = 256 * 1024 * 1024u64; + if total_bytes > max_total_bytes { + return json_error( + StatusCode::BAD_REQUEST, + "Total download size exceeds 256 MB limit. Select fewer objects.", + ); + } + + let zip_bytes = match build_zip_archive(archive_entries) { + Ok(bytes) => bytes, + Err(err) => return json_error(StatusCode::BAD_REQUEST, err), + }; + + let mut headers = HeaderMap::new(); + headers.insert(header::CONTENT_TYPE, "application/zip".parse().unwrap()); + headers.insert( + header::CONTENT_DISPOSITION, + format!("attachment; filename=\"{}-download.zip\"", bucket_name) + .parse() + .unwrap(), + ); + (StatusCode::OK, headers, zip_bytes).into_response() +} + +pub async fn archived_objects( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, +) -> Response { + let versions_root = version_root_for_bucket(&state, &bucket_name); + if !versions_root.exists() { + return Json(json!({ "objects": [] })).into_response(); + } + + let mut grouped: BTreeMap> = BTreeMap::new(); + let mut stack = vec![versions_root]; + + while let Some(current) = stack.pop() { + let read_dir = match std::fs::read_dir(¤t) { + Ok(entries) => entries, + Err(_) => continue, + }; + for entry in read_dir.flatten() { + let file_type = match entry.file_type() { + Ok(file_type) => file_type, + Err(_) => continue, + }; + if file_type.is_dir() { + stack.push(entry.path()); + continue; + } + if entry.path().extension().and_then(|ext| ext.to_str()) != Some("json") { + continue; + } + let text = match std::fs::read_to_string(entry.path()) { + Ok(text) => text, + Err(_) => continue, + }; + let manifest = match serde_json::from_str::(&text) { + Ok(manifest) => manifest, + Err(_) => continue, + }; + if manifest.key.is_empty() { + continue; + } + grouped + .entry(manifest.key.clone()) + .or_default() + .push(manifest); + } + } + + let mut objects = Vec::new(); + for (key, mut versions) in grouped { + let live_exists = object_live_path(&state, &bucket_name, &key) + .map(|path| path.exists()) + .unwrap_or(false); + if live_exists { + continue; + } + versions.sort_by(|a, b| manifest_timestamp(b).cmp(&manifest_timestamp(a))); + let latest = versions.first().map(|record| manifest_to_json(record)); + objects.push(json!({ + "key": key, + "versions": versions.len(), + "total_size": versions.iter().map(|entry| entry.size).sum::(), + "latest": latest, + "restore_url": versions.first().map(|record| format!( + "/ui/buckets/{}/archived/{}/restore/{}", + bucket_name, + encode_object_key(&record.key), + encode_path_segment(&record.version_id) + )), + "purge_url": format!( + "/ui/buckets/{}/archived/{}/purge", + bucket_name, + encode_object_key(&key) + ), + })); + } + + Json(json!({ "objects": objects })).into_response() +} + +pub async fn archived_post_dispatch( + State(state): State, + Extension(_session): Extension, + Path((bucket_name, rest)): Path<(String, String)>, +) -> Response { + if let Some((key, version_id)) = rest.rsplit_once("/restore/") { + return restore_object_version_json(&state, &bucket_name, key, version_id).await; + } + if let Some(key) = rest.strip_suffix("/purge") { + match purge_object_versions_for_key(&state, &bucket_name, key).await { + Ok(()) => { + let _ = state.storage.delete_object(&bucket_name, key).await; + Json(json!({ + "status": "ok", + "message": format!("Removed archived versions for '{}'", key), + })) + .into_response() + } + Err(err) => json_error(StatusCode::BAD_REQUEST, err), + } + } else { + json_error(StatusCode::NOT_FOUND, "Unknown archived object action") + } +} + +pub async fn gc_status_ui( + State(state): State, + Extension(_session): Extension, +) -> Response { + match &state.gc { + Some(gc) => Json(gc.status().await).into_response(), + None => Json(json!({ + "enabled": false, + "message": "GC is not enabled. Set GC_ENABLED=true to enable." + })) + .into_response(), + } +} + +pub async fn gc_run_ui( + State(state): State, + Extension(_session): Extension, + body: Body, +) -> Response { + let Some(gc) = &state.gc else { + return json_error(StatusCode::BAD_REQUEST, "GC is not enabled"); + }; + let payload: Value = parse_json_body(body).await.unwrap_or_else(|_| json!({})); + let dry_run = payload + .get("dry_run") + .and_then(|value| value.as_bool()) + .unwrap_or(false); + match gc.run_now(dry_run).await { + Ok(result) => Json(result).into_response(), + Err(err) => json_error(StatusCode::CONFLICT, err), + } +} + +pub async fn gc_history_ui( + State(state): State, + Extension(_session): Extension, + Query(params): Query>, +) -> Response { + let limit = params.get("limit").and_then(|v| v.parse::().ok()); + match &state.gc { + Some(gc) => Json(apply_history_limit(gc.history().await, limit)).into_response(), + None => Json(json!({ "executions": [] })).into_response(), + } +} + +pub async fn integrity_status_ui( + State(state): State, + Extension(_session): Extension, +) -> Response { + match &state.integrity { + Some(checker) => Json(checker.status().await).into_response(), + None => Json(json!({ + "enabled": false, + "message": "Integrity checker is not enabled. Set INTEGRITY_ENABLED=true to enable." + })) + .into_response(), + } +} + +pub async fn integrity_run_ui( + State(state): State, + Extension(_session): Extension, + body: Body, +) -> Response { + let Some(checker) = &state.integrity else { + return json_error(StatusCode::BAD_REQUEST, "Integrity checker is not enabled"); + }; + let payload: Value = parse_json_body(body).await.unwrap_or_else(|_| json!({})); + let dry_run = payload + .get("dry_run") + .and_then(|value| value.as_bool()) + .unwrap_or(false); + let auto_heal = payload + .get("auto_heal") + .and_then(|value| value.as_bool()) + .unwrap_or(false); + match checker.run_now(dry_run, auto_heal).await { + Ok(result) => Json(result).into_response(), + Err(err) => json_error(StatusCode::CONFLICT, err), + } +} + +pub async fn integrity_history_ui( + State(state): State, + Extension(_session): Extension, + Query(params): Query>, +) -> Response { + let limit = params.get("limit").and_then(|v| v.parse::().ok()); + match &state.integrity { + Some(checker) => Json(apply_history_limit(checker.history().await, limit)).into_response(), + None => Json(json!({ "executions": [] })).into_response(), + } +} + +fn apply_history_limit(mut value: Value, limit: Option) -> Value { + if let Some(limit) = limit { + if let Some(arr) = value.get_mut("executions").and_then(|v| v.as_array_mut()) { + if arr.len() > limit { + arr.truncate(limit); + } + } + } + value +} + +pub async fn bucket_stub_json(Extension(_session): Extension) -> Response { + Json(json!({"status": "not_implemented", "items": []})).into_response() +} + +pub async fn lifecycle_history_stub( + State(state): State, + Extension(_session): Extension, + Path(_bucket_name): Path, +) -> Response { + Json(json!({ + "enabled": state.config.lifecycle_enabled, + "executions": [], + "total": 0, + })) + .into_response() +} + +#[derive(Deserialize, Default)] +pub struct ReplicationFailuresQuery { + #[serde(default)] + pub limit: Option, + #[serde(default)] + pub offset: Option, +} + +#[derive(Deserialize)] +pub struct ReplicationObjectKeyQuery { + pub object_key: String, +} + +pub async fn replication_status( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, +) -> Response { + let Some(rule) = state.replication.get_rule(&bucket_name) else { + return json_error(StatusCode::NOT_FOUND, "No replication rule"); + }; + + let (endpoint_healthy, endpoint_error) = match state.connections.get(&rule.target_connection_id) + { + Some(conn) => { + let healthy = state.replication.check_endpoint(&conn).await; + let error = if healthy { + None + } else { + Some(format!("Cannot reach endpoint: {}", conn.endpoint_url)) + }; + (healthy, error) + } + None => (false, Some("Target connection not found".to_string())), + }; + + json_ok(json!({ + "enabled": rule.enabled, + "target_bucket": rule.target_bucket, + "target_connection_id": rule.target_connection_id, + "mode": rule.mode, + "objects_synced": rule.stats.objects_synced, + "objects_pending": rule.stats.objects_pending, + "objects_orphaned": rule.stats.objects_orphaned, + "bytes_synced": rule.stats.bytes_synced, + "last_sync_at": rule.stats.last_sync_at, + "last_sync_key": rule.stats.last_sync_key, + "endpoint_healthy": endpoint_healthy, + "endpoint_error": endpoint_error, + })) +} + +pub async fn replication_failures( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + Query(q): Query, +) -> Response { + let limit = q.limit.unwrap_or(50).clamp(1, 500); + let offset = q.offset.unwrap_or(0); + let failures = state + .replication + .get_failed_items(&bucket_name, limit, offset); + let total = state.replication.get_failure_count(&bucket_name); + json_ok(json!({ + "failures": failures, + "total": total, + "limit": limit, + "offset": offset, + })) +} + +pub async fn retry_replication_failure( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + Query(q): Query, +) -> Response { + let object_key = q.object_key.trim(); + if object_key.is_empty() { + return json_error(StatusCode::BAD_REQUEST, "object_key is required"); + } + + if state + .replication + .retry_failed(&bucket_name, object_key) + .await + { + json_ok(json!({ + "status": "submitted", + "object_key": object_key, + })) + } else { + json_error(StatusCode::BAD_REQUEST, "Failed to submit retry") + } +} + +pub async fn retry_all_replication_failures( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, +) -> Response { + let (submitted, skipped) = state.replication.retry_all(&bucket_name).await; + json_ok(json!({ + "status": "submitted", + "submitted": submitted, + "skipped": skipped, + })) +} + +pub async fn dismiss_replication_failure( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, + Query(q): Query, +) -> Response { + let object_key = q.object_key.trim(); + if object_key.is_empty() { + return json_error(StatusCode::BAD_REQUEST, "object_key is required"); + } + + if state.replication.dismiss_failure(&bucket_name, object_key) { + json_ok(json!({ + "status": "dismissed", + "object_key": object_key, + })) + } else { + json_error(StatusCode::NOT_FOUND, "Failure not found") + } +} + +pub async fn clear_replication_failures( + State(state): State, + Extension(_session): Extension, + Path(bucket_name): Path, +) -> Response { + state.replication.clear_failures(&bucket_name); + json_ok(json!({ "status": "cleared" })) +} + +static SERVER_START_TIME: std::sync::OnceLock = std::sync::OnceLock::new(); +static SYSINFO: std::sync::OnceLock> = std::sync::OnceLock::new(); + +async fn sample_system() -> (f64, u64, u64) { + let lock = SYSINFO.get_or_init(|| { + let mut system = System::new(); + system.refresh_cpu_usage(); + system.refresh_memory(); + Mutex::new(system) + }); + { + let mut system = lock.lock().unwrap(); + system.refresh_cpu_usage(); + } + tokio::time::sleep(sysinfo::MINIMUM_CPU_UPDATE_INTERVAL).await; + let mut system = lock.lock().unwrap(); + system.refresh_cpu_usage(); + system.refresh_memory(); + let cpu_percent = system.global_cpu_usage() as f64; + let mem_total = system.total_memory(); + let mem_used = system.used_memory(); + (cpu_percent, mem_used, mem_total) +} + +fn normalize_path_for_mount(path: &FsPath) -> String { + let canonical = path.canonicalize().unwrap_or_else(|_| path.to_path_buf()); + let raw = canonical.to_string_lossy().to_string(); + let stripped = raw.strip_prefix(r"\\?\").unwrap_or(&raw); + stripped.to_lowercase() +} + +fn sample_disk(path: &FsPath) -> (u64, u64) { + let disks = Disks::new_with_refreshed_list(); + let path_str = normalize_path_for_mount(path); + let mut best: Option<(usize, u64, u64)> = None; + for disk in disks.list() { + let mount_raw = disk.mount_point().to_string_lossy().to_string(); + let mount = mount_raw + .strip_prefix(r"\\?\") + .unwrap_or(&mount_raw) + .to_lowercase(); + let total = disk.total_space(); + let free = disk.available_space(); + if path_str.starts_with(&mount) { + let len = mount.len(); + match best { + Some((best_len, _, _)) if len <= best_len => {} + _ => best = Some((len, total, free)), + } + } + } + best.map(|(_, total, free)| (total, free)).unwrap_or((0, 0)) +} + +pub async fn collect_metrics(state: &AppState) -> Value { + let start_time = *SERVER_START_TIME.get_or_init(std::time::Instant::now); + let uptime_days = start_time.elapsed().as_secs_f64() / 86400.0; + + let buckets_list = state.storage.list_buckets().await.unwrap_or_default(); + let bucket_count = buckets_list.len() as u64; + + let mut total_objects: u64 = 0; + let mut total_bytes: u64 = 0; + let mut total_versions: u64 = 0; + for bucket in &buckets_list { + if let Ok(stats) = state.storage.bucket_stats(&bucket.name).await { + total_objects += stats.objects; + total_bytes += stats.bytes; + total_versions += stats.version_count; + } + } + + let (cpu_percent, mem_used, mem_total) = sample_system().await; + let mem_pct = if mem_total > 0 { + (mem_used as f64 / mem_total as f64) * 100.0 + } else { + 0.0 + }; + + let (disk_total, disk_free) = sample_disk(&state.config.storage_root); + let disk_used = disk_total.saturating_sub(disk_free); + let disk_pct = if disk_total > 0 { + (disk_used as f64 / disk_total as f64) * 100.0 + } else { + 0.0 + }; + + json!({ + "cpu_percent": cpu_percent, + "memory": { + "percent": mem_pct, + "used": human_size(mem_used), + "total": human_size(mem_total), + }, + "disk": { + "percent": disk_pct, + "free": human_size(disk_free), + "total": human_size(disk_total), + }, + "app": { + "storage_used": human_size(total_bytes), + "buckets": bucket_count, + "objects": total_objects, + "versions": total_versions, + "uptime_days": uptime_days.floor() as u64, + }, + }) +} + +pub async fn metrics_api(State(state): State) -> Response { + Json(collect_metrics(&state).await).into_response() +} + +#[derive(Deserialize, Default)] +pub struct HoursQuery { + #[serde(default)] + pub hours: Option, +} + +pub async fn metrics_history( + State(state): State, + Query(q): Query, +) -> Response { + let settings = metrics_settings_snapshot(&state); + match &state.system_metrics { + Some(metrics) => Json(json!({ + "enabled": settings.enabled, + "history": metrics.get_history(q.hours).await, + "interval_minutes": settings.interval_minutes, + "retention_hours": settings.retention_hours, + "hours_requested": q.hours.unwrap_or(settings.retention_hours), + })) + .into_response(), + None => Json(json!({ + "enabled": settings.enabled, + "history": [], + "interval_minutes": settings.interval_minutes, + "retention_hours": settings.retention_hours, + "hours_requested": q.hours.unwrap_or(settings.retention_hours), + })) + .into_response(), + } +} + +pub async fn metrics_operations(State(state): State) -> Response { + match &state.metrics { + Some(metrics) => { + let stats = metrics.get_current_stats(); + Json(json!({ + "enabled": true, + "stats": stats, + })) + .into_response() + } + None => Json(json!({ + "enabled": false, + "stats": null, + })) + .into_response(), + } +} + +pub async fn metrics_operations_history( + State(state): State, + Query(q): Query, +) -> Response { + match &state.metrics { + Some(metrics) => { + let history = metrics.get_history(q.hours.or(Some(24))); + Json(json!({ + "enabled": true, + "history": history, + "interval_minutes": 5, + })) + .into_response() + } + None => Json(json!({ + "enabled": false, + "history": [], + "interval_minutes": 5, + })) + .into_response(), + } +} diff --git a/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui_pages.rs b/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui_pages.rs index 92012b5..4b300a3 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui_pages.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/handlers/ui_pages.rs @@ -1,7 +1,7 @@ use std::collections::HashMap; -use axum::extract::{Extension, Path, Query, State}; -use axum::http::StatusCode; +use axum::extract::{Extension, Form, Path, Query, State}; +use axum::http::{header, HeaderMap, StatusCode}; use axum::response::{IntoResponse, Redirect, Response}; use serde_json::{json, Value}; use tera::Context; @@ -20,41 +20,104 @@ pub fn register_ui_endpoints(engine: &TemplateEngine) { ("ui.bucket_detail", "/ui/buckets/{bucket_name}"), ("ui.create_bucket", "/ui/buckets/create"), ("ui.delete_bucket", "/ui/buckets/{bucket_name}/delete"), - ("ui.update_bucket_versioning", "/ui/buckets/{bucket_name}/versioning"), + ( + "ui.update_bucket_versioning", + "/ui/buckets/{bucket_name}/versioning", + ), ("ui.update_bucket_quota", "/ui/buckets/{bucket_name}/quota"), - ("ui.update_bucket_encryption", "/ui/buckets/{bucket_name}/encryption"), - ("ui.update_bucket_policy", "/ui/buckets/{bucket_name}/policy"), - ("ui.update_bucket_replication", "/ui/buckets/{bucket_name}/replication"), - ("ui.update_bucket_website", "/ui/buckets/{bucket_name}/website"), + ( + "ui.update_bucket_encryption", + "/ui/buckets/{bucket_name}/encryption", + ), + ( + "ui.update_bucket_policy", + "/ui/buckets/{bucket_name}/policy", + ), + ( + "ui.update_bucket_replication", + "/ui/buckets/{bucket_name}/replication", + ), + ( + "ui.update_bucket_website", + "/ui/buckets/{bucket_name}/website", + ), ("ui.upload_object", "/ui/buckets/{bucket_name}/upload"), - ("ui.bulk_delete_objects", "/ui/buckets/{bucket_name}/bulk-delete"), - ("ui.bulk_download_objects", "/ui/buckets/{bucket_name}/bulk-download"), + ( + "ui.bulk_delete_objects", + "/ui/buckets/{bucket_name}/bulk-delete", + ), + ( + "ui.bulk_download_objects", + "/ui/buckets/{bucket_name}/bulk-download", + ), ("ui.archived_objects", "/ui/buckets/{bucket_name}/archived"), - ("ui.initiate_multipart_upload", "/ui/buckets/{bucket_name}/multipart/initiate"), - ("ui.upload_multipart_part", "/ui/buckets/{bucket_name}/multipart/{upload_id}/part/{part_number}"), - ("ui.complete_multipart_upload", "/ui/buckets/{bucket_name}/multipart/{upload_id}/complete"), - ("ui.abort_multipart_upload", "/ui/buckets/{bucket_name}/multipart/{upload_id}/abort"), - ("ui.get_lifecycle_history", "/ui/buckets/{bucket_name}/lifecycle/history"), - ("ui.get_replication_status", "/ui/buckets/{bucket_name}/replication/status"), - ("ui.get_replication_failures", "/ui/buckets/{bucket_name}/replication/failures"), - ("ui.clear_replication_failures", "/ui/buckets/{bucket_name}/replication/failures/clear"), - ("ui.retry_all_replication_failures", "/ui/buckets/{bucket_name}/replication/failures/retry-all"), - ("ui.retry_replication_failure", "/ui/buckets/{bucket_name}/replication/failures/retry"), - ("ui.dismiss_replication_failure", "/ui/buckets/{bucket_name}/replication/failures/dismiss"), + ( + "ui.initiate_multipart_upload", + "/ui/buckets/{bucket_name}/multipart/initiate", + ), + ( + "ui.upload_multipart_part", + "/ui/buckets/{bucket_name}/multipart/{upload_id}/part", + ), + ( + "ui.complete_multipart_upload", + "/ui/buckets/{bucket_name}/multipart/{upload_id}/complete", + ), + ( + "ui.abort_multipart_upload", + "/ui/buckets/{bucket_name}/multipart/{upload_id}/abort", + ), + ( + "ui.get_lifecycle_history", + "/ui/buckets/{bucket_name}/lifecycle/history", + ), + ( + "ui.get_replication_status", + "/ui/buckets/{bucket_name}/replication/status", + ), + ( + "ui.get_replication_failures", + "/ui/buckets/{bucket_name}/replication/failures", + ), + ( + "ui.clear_replication_failures", + "/ui/buckets/{bucket_name}/replication/failures/clear", + ), + ( + "ui.retry_all_replication_failures", + "/ui/buckets/{bucket_name}/replication/failures/retry-all", + ), + ( + "ui.retry_replication_failure", + "/ui/buckets/{bucket_name}/replication/failures/retry", + ), + ( + "ui.dismiss_replication_failure", + "/ui/buckets/{bucket_name}/replication/failures/dismiss", + ), ("ui.replication_wizard", "/ui/replication/new"), - ("ui.create_peer_replication_rules", "/ui/replication/create"), + ( + "ui.create_peer_replication_rules", + "/ui/sites/peers/{site_id}/replication-rules", + ), ("ui.iam_dashboard", "/ui/iam"), ("ui.create_iam_user", "/ui/iam/users"), ("ui.update_iam_user", "/ui/iam/users/{user_id}"), ("ui.delete_iam_user", "/ui/iam/users/{user_id}/delete"), ("ui.update_iam_policies", "/ui/iam/users/{user_id}/policies"), ("ui.update_iam_expiry", "/ui/iam/users/{user_id}/expiry"), - ("ui.rotate_iam_secret", "/ui/iam/users/{user_id}/rotate-secret"), + ( + "ui.rotate_iam_secret", + "/ui/iam/users/{user_id}/rotate-secret", + ), ("ui.connections_dashboard", "/ui/connections"), ("ui.create_connection", "/ui/connections/create"), ("ui.update_connection", "/ui/connections/{connection_id}"), - ("ui.delete_connection", "/ui/connections/{connection_id}/delete"), - ("ui.test_connection", "/ui/connections/{connection_id}/test"), + ( + "ui.delete_connection", + "/ui/connections/{connection_id}/delete", + ), + ("ui.test_connection", "/ui/connections/test"), ("ui.sites_dashboard", "/ui/sites"), ("ui.update_local_site", "/ui/sites/local"), ("ui.add_peer_site", "/ui/sites/peers"), @@ -65,22 +128,40 @@ pub fn register_ui_endpoints(engine: &TemplateEngine) { ("ui.system_gc_history", "/ui/system/gc/history"), ("ui.system_integrity_status", "/ui/system/integrity/status"), ("ui.system_integrity_run", "/ui/system/integrity/run"), - ("ui.system_integrity_history", "/ui/system/integrity/history"), + ( + "ui.system_integrity_history", + "/ui/system/integrity/history", + ), ("ui.website_domains_dashboard", "/ui/website-domains"), ("ui.create_website_domain", "/ui/website-domains/create"), ("ui.update_website_domain", "/ui/website-domains/{domain}"), - ("ui.delete_website_domain", "/ui/website-domains/{domain}/delete"), + ( + "ui.delete_website_domain", + "/ui/website-domains/{domain}/delete", + ), ("ui.docs_page", "/ui/docs"), ]); } -fn page_context( - state: &AppState, - session: &SessionHandle, - endpoint: &str, -) -> Context { +fn page_context(state: &AppState, session: &SessionHandle, endpoint: &str) -> Context { let mut ctx = base_context(session, Some(endpoint)); - ctx.insert("principal", &session.read(|s| s.user_id.clone())); + let principal = session.read(|s| { + s.user_id.as_ref().map(|uid| { + json!({ + "access_key": uid, + "user_id": uid, + "display_name": s + .display_name + .clone() + .unwrap_or_else(|| uid.clone()), + "is_admin": true, + }) + }) + }); + match principal { + Some(p) => ctx.insert("principal", &p), + None => ctx.insert("principal", &Value::Null), + } ctx.insert("can_manage_iam", &true); ctx.insert("can_manage_replication", &true); ctx.insert("can_manage_sites", &true); @@ -94,6 +175,112 @@ fn page_context( ctx } +fn human_size(bytes: u64) -> String { + const UNITS: [&str; 6] = ["B", "KB", "MB", "GB", "TB", "PB"]; + let mut size = bytes as f64; + let mut idx = 0usize; + while size >= 1024.0 && idx < UNITS.len() - 1 { + size /= 1024.0; + idx += 1; + } + if idx == 0 { + format!("{} {}", bytes, UNITS[idx]) + } else { + format!("{:.1} {}", size, UNITS[idx]) + } +} + +fn wants_json(headers: &HeaderMap) -> bool { + headers + .get("x-requested-with") + .and_then(|value| value.to_str().ok()) + .map(|value| value.eq_ignore_ascii_case("xmlhttprequest")) + .unwrap_or(false) + || headers + .get(header::ACCEPT) + .and_then(|value| value.to_str().ok()) + .map(|value| value.contains("application/json")) + .unwrap_or(false) +} + +fn bucket_tab_redirect(bucket_name: &str, tab: &str) -> Response { + Redirect::to(&format!("/ui/buckets/{}?tab={}", bucket_name, tab)).into_response() +} + +fn default_public_policy(bucket_name: &str) -> String { + serde_json::to_string_pretty(&json!({ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AllowList", + "Effect": "Allow", + "Principal": "*", + "Action": ["s3:ListBucket"], + "Resource": [format!("arn:aws:s3:::{}", bucket_name)], + }, + { + "Sid": "AllowRead", + "Effect": "Allow", + "Principal": "*", + "Action": ["s3:GetObject"], + "Resource": [format!("arn:aws:s3:::{}/*", bucket_name)], + } + ] + })) + .unwrap_or_else(|_| "{}".to_string()) +} + +fn parse_api_base(state: &AppState) -> (String, String) { + let api_base = std::env::var("API_BASE_URL") + .unwrap_or_else(|_| format!("http://{}", state.config.bind_addr)) + .trim_end_matches('/') + .to_string(); + let api_host = api_base + .split("://") + .nth(1) + .unwrap_or(&api_base) + .split('/') + .next() + .unwrap_or("") + .to_string(); + (api_base, api_host) +} + +fn config_encryption_to_ui(value: Option<&Value>) -> Value { + match value { + Some(Value::Object(map)) => Value::Object(map.clone()), + Some(Value::String(s)) => { + serde_json::from_str(s).unwrap_or_else(|_| json!({ "Rules": [] })) + } + _ => json!({ "Rules": [] }), + } +} + +fn config_website_to_ui(value: Option<&Value>) -> Value { + match value { + Some(Value::Object(map)) => Value::Object(map.clone()), + Some(Value::String(s)) => serde_json::from_str(s).unwrap_or(Value::Null), + _ => Value::Null, + } +} + +fn bucket_access_descriptor( + policy: Option<&Value>, + bucket_name: &str, +) -> (&'static str, &'static str) { + let Some(policy) = policy else { + return ("Private", "bg-secondary-subtle text-secondary-emphasis"); + }; + + let default_policy = default_public_policy(bucket_name); + let default_policy_value: Value = serde_json::from_str(&default_policy).unwrap_or(Value::Null); + if *policy == default_policy_value { + return ("Public Read", "bg-warning-subtle text-warning-emphasis"); + } + + ("Custom policy", "bg-info-subtle text-info-emphasis") +} + pub async fn buckets_overview( State(state): State, Extension(session): Extension, @@ -108,24 +295,33 @@ pub async fn buckets_overview( } }; - let items: Vec = buckets - .iter() - .map(|b| { - json!({ - "meta": { - "name": b.name, - "creation_date": b.creation_date.to_rfc3339(), - }, - "summary": { - "human_size": "0 B", - "objects": 0, - }, - "detail_url": format!("/ui/buckets/{}", b.name), - "access_badge": "bg-secondary bg-opacity-10 text-secondary", - "access_label": "Private", - }) - }) - .collect(); + let mut items: Vec = Vec::with_capacity(buckets.len()); + for b in &buckets { + let stats = state.storage.bucket_stats(&b.name).await.ok(); + let total_bytes = stats.as_ref().map(|s| s.total_bytes()).unwrap_or(0); + let total_objects = stats.as_ref().map(|s| s.total_objects()).unwrap_or(0); + let policy = state + .storage + .get_bucket_config(&b.name) + .await + .ok() + .and_then(|cfg| cfg.policy); + let (access_label, access_badge) = bucket_access_descriptor(policy.as_ref(), &b.name); + + items.push(json!({ + "meta": { + "name": b.name, + "creation_date": b.creation_date.to_rfc3339(), + }, + "summary": { + "human_size": human_size(total_bytes), + "objects": total_objects, + }, + "detail_url": format!("/ui/buckets/{}", b.name), + "access_badge": access_badge, + "access_label": access_label, + })); + } ctx.insert("buckets", &items); render(&state, "buckets.html", &ctx) @@ -135,32 +331,269 @@ pub async fn bucket_detail( State(state): State, Extension(session): Extension, Path(bucket_name): Path, + Query(request_args): Query>, ) -> Response { if !matches!(state.storage.bucket_exists(&bucket_name).await, Ok(true)) { return (StatusCode::NOT_FOUND, "Bucket not found").into_response(); } let mut ctx = page_context(&state, &session, "ui.bucket_detail"); + ctx.insert("request_args", &request_args); + let bucket_meta = state + .storage + .list_buckets() + .await + .ok() + .and_then(|list| list.into_iter().find(|b| b.name == bucket_name)); + let bucket_config = state + .storage + .get_bucket_config(&bucket_name) + .await + .unwrap_or_default(); + let bucket_stats = state + .storage + .bucket_stats(&bucket_name) + .await + .unwrap_or_default(); + let replication_rule = state.replication.get_rule(&bucket_name); + let target_conn = replication_rule + .as_ref() + .and_then(|rule| state.connections.get(&rule.target_connection_id)); + let versioning_enabled = state + .storage + .is_versioning_enabled(&bucket_name) + .await + .unwrap_or(false); + let encryption_config = config_encryption_to_ui(bucket_config.encryption.as_ref()); + let website_config = config_website_to_ui(bucket_config.website.as_ref()); + let quota = bucket_config.quota.clone(); + let max_bytes = quota.as_ref().and_then(|q| q.max_bytes); + let max_objects = quota.as_ref().and_then(|q| q.max_objects); + let bucket_policy = bucket_config.policy.clone().unwrap_or(Value::Null); + let bucket_policy_text = if bucket_policy.is_null() { + String::new() + } else { + serde_json::to_string_pretty(&bucket_policy).unwrap_or_else(|_| bucket_policy.to_string()) + }; + let default_policy = default_public_policy(&bucket_name); + let default_policy_value: Value = serde_json::from_str(&default_policy).unwrap_or(Value::Null); + let preset_choice = if bucket_policy.is_null() { + "private" + } else if bucket_policy == default_policy_value { + "public" + } else { + "custom" + }; ctx.insert("bucket_name", &bucket_name); - ctx.insert("bucket", &json!({ "name": bucket_name })); + ctx.insert( + "bucket", + &json!({ + "name": bucket_name, + "creation_date": bucket_meta + .as_ref() + .map(|b| b.creation_date.to_rfc3339()) + .unwrap_or_else(|| chrono::Utc::now().to_rfc3339()), + }), + ); ctx.insert("objects", &Vec::::new()); ctx.insert("prefixes", &Vec::::new()); - ctx.insert("total_objects", &0); - ctx.insert("total_bytes", &0); - ctx.insert("max_objects", &Value::Null); - ctx.insert("max_bytes", &Value::Null); - ctx.insert("versioning_status", &"Disabled"); - ctx.insert("encryption_config", &json!({ "Rules": [] })); - ctx.insert("replication_rules", &Vec::::new()); - ctx.insert("website_config", &Value::Null); - ctx.insert("bucket_policy", &""); - ctx.insert("connections", &Vec::::new()); + ctx.insert("total_objects", &bucket_stats.total_objects()); + ctx.insert("total_bytes", &bucket_stats.total_bytes()); + ctx.insert("current_objects", &bucket_stats.objects); + ctx.insert("current_bytes", &bucket_stats.bytes); + ctx.insert("version_count", &bucket_stats.version_count); + ctx.insert("version_bytes", &bucket_stats.version_bytes); + ctx.insert("max_objects", &max_objects); + ctx.insert("max_bytes", &max_bytes); + ctx.insert("has_max_objects", &max_objects.is_some()); + ctx.insert("has_max_bytes", &max_bytes.is_some()); + ctx.insert( + "obj_pct", + &max_objects + .map(|m| { + ((bucket_stats.total_objects() as f64 / m.max(1) as f64) * 100.0).round() as u64 + }) + .unwrap_or(0), + ); + ctx.insert( + "bytes_pct", + &max_bytes + .map(|m| ((bucket_stats.total_bytes() as f64 / m.max(1) as f64) * 100.0).round() as u64) + .unwrap_or(0), + ); + ctx.insert("has_quota", "a.is_some()); + ctx.insert("versioning_enabled", &versioning_enabled); + ctx.insert( + "versioning_status", + &(if versioning_enabled { + "Enabled" + } else { + "Disabled" + }), + ); + ctx.insert("encryption_config", &encryption_config); + ctx.insert("enc_rules", &Vec::::new()); + ctx.insert("enc_algorithm", &""); + ctx.insert("enc_kms_key", &""); + let replication_rules = replication_rule + .clone() + .and_then(|rule| serde_json::to_value(rule).ok()) + .map(|rule| vec![rule]) + .unwrap_or_default(); + ctx.insert("replication_rules", &replication_rules); + ctx.insert( + "replication_rule", + &replication_rule + .clone() + .and_then(|rule| serde_json::to_value(rule).ok()) + .unwrap_or(Value::Null), + ); + ctx.insert("website_config", &website_config); + ctx.insert("bucket_policy", &bucket_policy); + ctx.insert("bucket_policy_text", &bucket_policy_text); + ctx.insert("preset_choice", &preset_choice); + let conns: Vec = state + .connections + .list() + .into_iter() + .map(|c| { + json!({ + "id": c.id, + "name": c.name, + "endpoint_url": c.endpoint_url, + "region": c.region, + "access_key": c.access_key, + }) + }) + .collect(); + ctx.insert("connections", &conns); ctx.insert("current_prefix", &""); ctx.insert("parent_prefix", &""); ctx.insert("has_more", &false); ctx.insert("next_token", &""); - ctx.insert("active_tab", &"objects"); - ctx.insert("multipart_uploads", &Vec::::new()); + ctx.insert( + "active_tab", + &request_args + .get("tab") + .cloned() + .unwrap_or_else(|| "objects".to_string()), + ); + let multipart_uploads: Vec = state + .storage + .list_multipart_uploads(&bucket_name) + .await + .unwrap_or_default() + .into_iter() + .map(|u| { + json!({ + "upload_id": u.upload_id, + "key": u.key, + "initiated": u.initiated.to_rfc3339(), + }) + }) + .collect(); + ctx.insert("multipart_uploads", &multipart_uploads); + ctx.insert( + "target_conn", + &target_conn + .as_ref() + .and_then(|conn| serde_json::to_value(conn).ok()) + .unwrap_or(Value::Null), + ); + ctx.insert( + "target_conn_name", + &target_conn + .as_ref() + .map(|conn| conn.name.clone()) + .unwrap_or_default(), + ); + ctx.insert("default_policy", &default_policy); + ctx.insert("can_manage_cors", &true); + ctx.insert("can_manage_lifecycle", &true); + ctx.insert("can_manage_quota", &true); + ctx.insert("can_manage_versioning", &true); + ctx.insert("can_manage_website", &true); + ctx.insert("can_edit_policy", &true); + ctx.insert("is_replication_admin", &true); + ctx.insert("lifecycle_enabled", &state.config.lifecycle_enabled); + ctx.insert("site_sync_enabled", &state.config.site_sync_enabled); + ctx.insert( + "website_hosting_enabled", + &state.config.website_hosting_enabled, + ); + let website_domains: Vec = state + .website_domains + .as_ref() + .map(|store| { + store + .list_all() + .into_iter() + .filter_map(|entry| { + if entry.get("bucket").and_then(|v| v.as_str()) == Some(bucket_name.as_str()) { + entry + .get("domain") + .and_then(|v| v.as_str()) + .map(|s| s.to_string()) + } else { + None + } + }) + .collect() + }) + .unwrap_or_default(); + ctx.insert("website_domains", &website_domains); + let kms_keys: Vec = if let Some(kms) = &state.kms { + kms.list_keys() + .await + .into_iter() + .map(|key| { + json!({ + "key_id": key.key_id, + "description": key.description, + }) + }) + .collect() + } else { + Vec::new() + }; + ctx.insert("kms_keys", &kms_keys); + ctx.insert( + "bucket_stats", + &json!({ + "bytes": bucket_stats.bytes, + "objects": bucket_stats.objects, + "total_bytes": bucket_stats.total_bytes(), + "total_objects": bucket_stats.total_objects(), + "version_bytes": bucket_stats.version_bytes, + "version_count": bucket_stats.version_count + }), + ); + ctx.insert( + "bucket_quota", + &json!({ "max_bytes": max_bytes, "max_objects": max_objects }), + ); + ctx.insert( + "buckets_for_copy_url", + &format!("/ui/buckets/{}/copy-targets", bucket_name), + ); + ctx.insert("acl_url", &format!("/ui/buckets/{}/acl", bucket_name)); + ctx.insert("cors_url", &format!("/ui/buckets/{}/cors", bucket_name)); + ctx.insert( + "folders_url", + &format!("/ui/buckets/{}/folders", bucket_name), + ); + ctx.insert( + "lifecycle_url", + &format!("/ui/buckets/{}/lifecycle", bucket_name), + ); + ctx.insert( + "objects_api_url", + &format!("/ui/buckets/{}/objects", bucket_name), + ); + ctx.insert( + "objects_stream_url", + &format!("/ui/buckets/{}/objects/stream", bucket_name), + ); render(&state, "bucket_detail.html", &ctx) } @@ -169,43 +602,936 @@ pub async fn iam_dashboard( Extension(session): Extension, ) -> Response { let mut ctx = page_context(&state, &session, "ui.iam_dashboard"); - let users: Vec = state - .iam - .list_users() + let now = chrono::Utc::now(); + let soon = now + chrono::Duration::days(7); + let raw_users = state.iam.list_users().await; + let mut users: Vec = Vec::with_capacity(raw_users.len()); + for u in raw_users.iter() { + let user_id = u + .get("user_id") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let display_name = u + .get("display_name") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let enabled = u.get("enabled").and_then(|v| v.as_bool()).unwrap_or(true); + let access_key = u + .get("access_keys") + .and_then(|v| v.as_array()) + .and_then(|arr| { + arr.iter().find_map(|k| { + k.get("access_key") + .and_then(|x| x.as_str()) + .map(|s| s.to_string()) + }) + }) + .unwrap_or_default(); + + let detail = state.iam.get_user(&user_id).await; + let policies = detail + .as_ref() + .and_then(|d| d.get("policies").cloned()) + .unwrap_or(Value::Array(Vec::new())); + let expires_at = detail + .as_ref() + .and_then(|d| d.get("expires_at").cloned()) + .unwrap_or(Value::Null); + let is_admin = policies + .as_array() + .map(|items| { + items.iter().any(|policy| { + policy + .get("actions") + .and_then(|value| value.as_array()) + .map(|actions| { + actions + .iter() + .any(|action| matches!(action.as_str(), Some("*") | Some("iam:*"))) + }) + .unwrap_or(false) + }) + }) + .unwrap_or(false); + let expires_dt = expires_at.as_str().and_then(|value| { + chrono::DateTime::parse_from_rfc3339(value) + .ok() + .map(|dt| dt.with_timezone(&chrono::Utc)) + }); + let is_expired = expires_dt.map(|dt| dt <= now).unwrap_or(false); + let is_expiring_soon = expires_dt.map(|dt| dt > now && dt <= soon).unwrap_or(false); + let access_keys = u + .get("access_keys") + .cloned() + .unwrap_or(Value::Array(Vec::new())); + + users.push(json!({ + "user_id": user_id, + "access_key": access_key, + "display_name": display_name, + "enabled": enabled, + "is_enabled": enabled, + "expires_at": expires_at, + "is_admin": is_admin, + "is_expired": is_expired, + "is_expiring_soon": is_expiring_soon, + "access_keys": access_keys, + "policies": policies, + "policy_count": u.get("policy_count").cloned().unwrap_or(Value::from(0)), + })); + } + let all_buckets: Vec = state + .storage + .list_buckets() .await - .into_iter() - .map(|u| { - let mut map = u.as_object().cloned().unwrap_or_default(); - map.entry("policies".to_string()).or_insert(Value::Array(Vec::new())); - map.entry("expires_at".to_string()).or_insert(Value::Null); - map.entry("is_enabled".to_string()).or_insert(Value::Bool(true)); - map.entry("display_name".to_string()) - .or_insert_with(|| Value::String(String::new())); - Value::Object(map) - }) - .collect(); + .map(|list| list.into_iter().map(|b| b.name).collect()) + .unwrap_or_default(); ctx.insert("users", &users); ctx.insert("iam_locked", &false); - ctx.insert("now_iso", &chrono::Utc::now().to_rfc3339()); - ctx.insert( - "soon_iso", - &(chrono::Utc::now() + chrono::Duration::days(7)).to_rfc3339(), - ); - ctx.insert("all_buckets", &Vec::::new()); + ctx.insert("locked_reason", &""); + ctx.insert("iam_disabled", &false); + ctx.insert("all_buckets", &all_buckets); + ctx.insert("disclosed_secret", &Value::Null); + let config_doc = + serde_json::to_string_pretty(&state.iam.export_config(true)).unwrap_or_default(); + ctx.insert("config_document", &config_doc); + ctx.insert("config_summary", &json!({ "user_count": users.len() })); render(&state, "iam.html", &ctx) } +#[derive(serde::Deserialize)] +pub struct CreateIamUserForm { + pub display_name: Option, + pub access_key: Option, + pub secret_key: Option, + pub policies: Option, + pub expires_at: Option, + #[serde(default)] + pub csrf_token: String, +} + +fn parse_policies(raw: &str) -> Result, String> { + let trimmed = raw.trim(); + if trimmed.is_empty() { + return Ok(vec![]); + } + serde_json::from_str::>(trimmed) + .map_err(|e| format!("Invalid policies JSON: {}", e)) +} + +fn normalize_expires_at(raw: Option) -> Result, String> { + let Some(value) = raw else { + return Ok(None); + }; + let trimmed = value.trim(); + if trimmed.is_empty() { + return Ok(None); + } + if let Ok(dt) = chrono::DateTime::parse_from_rfc3339(trimmed) { + return Ok(Some(dt.with_timezone(&chrono::Utc).to_rfc3339())); + } + if let Ok(naive) = chrono::NaiveDateTime::parse_from_str(trimmed, "%Y-%m-%dT%H:%M") { + return Ok(Some(naive.and_utc().to_rfc3339())); + } + if let Ok(naive) = chrono::NaiveDateTime::parse_from_str(trimmed, "%Y-%m-%dT%H:%M:%S") { + return Ok(Some(naive.and_utc().to_rfc3339())); + } + Err("Invalid expiry date format".to_string()) +} + +pub async fn create_iam_user( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let wants_json = wants_json(&headers); + let display_name = form + .display_name + .as_deref() + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| "Unnamed".to_string()); + + if display_name.len() > 64 { + let message = "Display name must be 64 characters or fewer".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/iam").into_response(); + } + + let policies = match form.policies.as_deref().map(parse_policies) { + Some(Ok(p)) if !p.is_empty() => Some(p), + Some(Ok(_)) | None => None, + Some(Err(e)) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + return Redirect::to("/ui/iam").into_response(); + } + }; + + let expires_at = match normalize_expires_at(form.expires_at) { + Ok(v) => v, + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + return Redirect::to("/ui/iam").into_response(); + } + }; + + let custom_access_key = form + .access_key + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()); + let custom_secret_key = form + .secret_key + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()); + + match state.iam.create_user( + &display_name, + policies.clone(), + custom_access_key, + custom_secret_key, + expires_at, + ) { + Ok(created) => { + let user_id = created + .get("user_id") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let access_key = created + .get("access_key") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let secret_key = created + .get("secret_key") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let message = format!("Created user {}", access_key); + if wants_json { + return axum::Json(json!({ + "success": true, + "message": message, + "user_id": user_id, + "access_key": access_key, + "secret_key": secret_key, + "display_name": display_name, + "expires_at": created.get("expires_at").cloned().unwrap_or(Value::Null), + "policies": policies.unwrap_or_default(), + })) + .into_response(); + } + session + .write(|s| s.push_flash("success", format!("{}. Copy the secret now.", message))); + Redirect::to("/ui/iam").into_response() + } + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + Redirect::to("/ui/iam").into_response() + } + } +} + +#[derive(serde::Deserialize)] +pub struct UpdateIamUserForm { + pub display_name: Option, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn update_iam_user( + State(state): State, + Extension(session): Extension, + Path(user_id): Path, + headers: HeaderMap, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let wants_json = wants_json(&headers); + let display_name = form + .display_name + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()); + + match state.iam.update_user(&user_id, display_name, None) { + Ok(()) => { + if wants_json { + let display_name = state + .iam + .get_user(&user_id) + .await + .and_then(|user| { + user.get("display_name") + .and_then(|value| value.as_str()) + .map(ToString::to_string) + }) + .unwrap_or_default(); + return axum::Json(json!({ + "success": true, + "user_id": user_id, + "display_name": display_name, + })) + .into_response(); + } + session.write(|s| s.push_flash("success", "User updated.")); + Redirect::to("/ui/iam").into_response() + } + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + Redirect::to("/ui/iam").into_response() + } + } +} + +pub async fn delete_iam_user( + State(state): State, + Extension(session): Extension, + Path(user_id): Path, + headers: HeaderMap, +) -> Response { + let wants_json = wants_json(&headers); + match state.iam.delete_user(&user_id) { + Ok(()) => { + if wants_json { + return axum::Json(json!({ "success": true })).into_response(); + } + session.write(|s| s.push_flash("success", "User deleted.")); + Redirect::to("/ui/iam").into_response() + } + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + Redirect::to("/ui/iam").into_response() + } + } +} + +#[derive(serde::Deserialize)] +pub struct UpdateIamPoliciesForm { + pub policies: String, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn update_iam_policies( + State(state): State, + Extension(session): Extension, + Path(user_id): Path, + headers: HeaderMap, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let wants_json = wants_json(&headers); + let policies = match parse_policies(&form.policies) { + Ok(p) => p, + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + return Redirect::to("/ui/iam").into_response(); + } + }; + + match state.iam.update_user_policies(&user_id, policies) { + Ok(()) => { + if wants_json { + let policies = state + .iam + .get_user(&user_id) + .await + .and_then(|user| user.get("policies").cloned()) + .unwrap_or_else(|| Value::Array(Vec::new())); + return axum::Json(json!({ + "success": true, + "user_id": user_id, + "policies": policies, + })) + .into_response(); + } + session.write(|s| s.push_flash("success", "Policies updated.")); + Redirect::to("/ui/iam").into_response() + } + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + Redirect::to("/ui/iam").into_response() + } + } +} + +#[derive(serde::Deserialize)] +pub struct UpdateIamExpiryForm { + pub expires_at: Option, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn update_iam_expiry( + State(state): State, + Extension(session): Extension, + Path(user_id): Path, + headers: HeaderMap, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let wants_json = wants_json(&headers); + let expires_at = match normalize_expires_at(form.expires_at) { + Ok(v) => v, + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + return Redirect::to("/ui/iam").into_response(); + } + }; + + match state.iam.update_user(&user_id, None, Some(expires_at)) { + Ok(()) => { + if wants_json { + return axum::Json(json!({ "success": true })).into_response(); + } + session.write(|s| s.push_flash("success", "Expiry updated.")); + Redirect::to("/ui/iam").into_response() + } + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + Redirect::to("/ui/iam").into_response() + } + } +} + +pub async fn rotate_iam_secret( + State(state): State, + Extension(session): Extension, + Path(user_id): Path, + headers: HeaderMap, +) -> Response { + let wants_json = wants_json(&headers); + match state.iam.rotate_secret(&user_id) { + Ok(result) => { + let access_key = result + .get("access_key") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let secret_key = result + .get("secret_key") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + if wants_json { + return axum::Json(json!({ + "success": true, + "access_key": access_key, + "secret_key": secret_key, + })) + .into_response(); + } + session + .write(|s| s.push_flash("success", format!("Secret rotated for {}.", access_key))); + Redirect::to("/ui/iam").into_response() + } + Err(e) => { + if wants_json { + return (StatusCode::BAD_REQUEST, axum::Json(json!({ "error": e }))) + .into_response(); + } + session.write(|s| s.push_flash("danger", e)); + Redirect::to("/ui/iam").into_response() + } + } +} + pub async fn sites_dashboard( State(state): State, Extension(session): Extension, ) -> Response { let mut ctx = page_context(&state, &session, "ui.sites_dashboard"); - ctx.insert("local_site", &Value::Null); - ctx.insert("peers", &Vec::::new()); + + let local_site = state + .site_registry + .as_ref() + .and_then(|reg| reg.get_local_site()) + .map(|s| { + json!({ + "site_id": s.site_id, + "display_name": s.display_name, + "endpoint": s.endpoint, + "region": s.region, + "priority": s.priority, + }) + }) + .unwrap_or(Value::Null); + + let peers: Vec = state + .site_registry + .as_ref() + .map(|reg| { + reg.list_peers() + .into_iter() + .map(|p| { + json!({ + "site_id": p.site_id, + "display_name": p.display_name, + "endpoint": p.endpoint, + "region": p.region, + "priority": p.priority, + "connection_id": p.connection_id, + "is_healthy": p.is_healthy, + "last_health_check": p.last_health_check, + }) + }) + .collect() + }) + .unwrap_or_default(); + + let peers_with_stats: Vec = peers + .iter() + .cloned() + .map(|peer| { + let has_connection = peer + .get("connection_id") + .and_then(|value| value.as_str()) + .map(|value| !value.is_empty()) + .unwrap_or(false); + json!({ + "peer": peer, + "has_connection": has_connection, + "buckets_syncing": 0, + "has_bidirectional": false, + }) + }) + .collect(); + + let conns: Vec = state + .connections + .list() + .into_iter() + .map(|c| { + json!({ + "id": c.id, + "name": c.name, + "endpoint_url": c.endpoint_url, + "region": c.region, + "access_key": c.access_key, + }) + }) + .collect(); + + ctx.insert("local_site", &local_site); + ctx.insert("peers", &peers); + ctx.insert("peers_with_stats", &peers_with_stats); + ctx.insert("connections", &conns); + ctx.insert( + "config_site_id", + &std::env::var("SITE_ID").unwrap_or_default(), + ); + ctx.insert( + "config_site_endpoint", + &std::env::var("SITE_ENDPOINT").unwrap_or_default(), + ); + ctx.insert( + "config_site_region", + &std::env::var("SITE_REGION").unwrap_or_else(|_| state.config.region.clone()), + ); ctx.insert("topology", &json!({"sites": [], "connections": []})); render(&state, "sites.html", &ctx) } +#[derive(serde::Deserialize)] +pub struct LocalSiteForm { + pub site_id: String, + #[serde(default)] + pub endpoint: String, + #[serde(default = "default_site_region")] + pub region: String, + #[serde(default = "default_site_priority")] + pub priority: i32, + #[serde(default)] + pub display_name: String, + #[serde(default)] + pub csrf_token: String, +} + +#[derive(serde::Deserialize)] +pub struct PeerSiteForm { + pub site_id: String, + pub endpoint: String, + #[serde(default = "default_site_region")] + pub region: String, + #[serde(default = "default_site_priority")] + pub priority: i32, + #[serde(default)] + pub display_name: String, + #[serde(default)] + pub connection_id: String, + #[serde(default)] + pub csrf_token: String, +} + +#[derive(serde::Deserialize, Default)] +pub struct DeletePeerSiteForm { + #[serde(default)] + pub csrf_token: String, +} + +fn default_site_region() -> String { + "us-east-1".to_string() +} + +fn default_site_priority() -> i32 { + 100 +} + +pub async fn update_local_site( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + let wants_json = wants_json(&headers); + let site_id = form.site_id.trim().to_string(); + if site_id.is_empty() { + let message = "Site ID is required.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + } + + let Some(registry) = &state.site_registry else { + let message = "Site registry is not available.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + }; + + let existing = registry.get_local_site(); + let site = crate::services::site_registry::SiteInfo { + site_id: site_id.clone(), + endpoint: form.endpoint.trim().to_string(), + region: form.region.trim().to_string(), + priority: form.priority, + display_name: { + let display_name = form.display_name.trim(); + if display_name.is_empty() { + site_id.clone() + } else { + display_name.to_string() + } + }, + created_at: existing.and_then(|site| site.created_at), + }; + registry.set_local_site(site); + + let message = "Local site configuration updated".to_string(); + if wants_json { + return axum::Json(json!({ "ok": true, "message": message })).into_response(); + } + session.write(|s| s.push_flash("success", message)); + Redirect::to("/ui/sites").into_response() +} + +pub async fn add_peer_site( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + let wants_json = wants_json(&headers); + let site_id = form.site_id.trim().to_string(); + let endpoint = form.endpoint.trim().to_string(); + if site_id.is_empty() { + let message = "Site ID is required.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + } + if endpoint.is_empty() { + let message = "Endpoint is required.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + } + + let Some(registry) = &state.site_registry else { + let message = "Site registry is not available.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + }; + + if registry.get_peer(&site_id).is_some() { + let message = format!("Peer site '{}' already exists.", site_id); + if wants_json { + return ( + StatusCode::CONFLICT, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + } + + let connection_id = { + let value = form.connection_id.trim(); + if value.is_empty() { + None + } else { + Some(value.to_string()) + } + }; + if let Some(connection_id) = connection_id.as_deref() { + if state.connections.get(connection_id).is_none() { + let message = format!("Connection '{}' not found.", connection_id); + if wants_json { + return ( + StatusCode::NOT_FOUND, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + } + } + + let has_connection = connection_id.is_some(); + let peer = crate::services::site_registry::PeerSite { + site_id: site_id.clone(), + endpoint, + region: form.region.trim().to_string(), + priority: form.priority, + display_name: { + let display_name = form.display_name.trim(); + if display_name.is_empty() { + site_id.clone() + } else { + display_name.to_string() + } + }, + connection_id: connection_id.clone(), + created_at: None, + is_healthy: false, + last_health_check: None, + }; + registry.add_peer(peer); + + let message = format!("Peer site '{}' added.", site_id); + if wants_json { + let redirect = if has_connection { + Some(format!("/ui/replication/new?site_id={}", site_id)) + } else { + None + }; + return axum::Json(json!({ + "ok": true, + "message": message, + "redirect": redirect, + })) + .into_response(); + } + session.write(|s| s.push_flash("success", message)); + if has_connection { + return Redirect::to(&format!("/ui/replication/new?site_id={}", site_id)).into_response(); + } + Redirect::to("/ui/sites").into_response() +} + +pub async fn update_peer_site( + State(state): State, + Extension(session): Extension, + Path(site_id): Path, + headers: HeaderMap, + Form(form): Form, +) -> Response { + let wants_json = wants_json(&headers); + let Some(registry) = &state.site_registry else { + let message = "Site registry is not available.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + }; + + let Some(existing) = registry.get_peer(&site_id) else { + let message = format!("Peer site '{}' not found.", site_id); + if wants_json { + return ( + StatusCode::NOT_FOUND, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + }; + + let connection_id = { + let value = form.connection_id.trim(); + if value.is_empty() { + None + } else { + Some(value.to_string()) + } + }; + if let Some(connection_id) = connection_id.as_deref() { + if state.connections.get(connection_id).is_none() { + let message = format!("Connection '{}' not found.", connection_id); + if wants_json { + return ( + StatusCode::NOT_FOUND, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + } + } + + let peer = crate::services::site_registry::PeerSite { + site_id: site_id.clone(), + endpoint: form.endpoint.trim().to_string(), + region: form.region.trim().to_string(), + priority: form.priority, + display_name: { + let display_name = form.display_name.trim(); + if display_name.is_empty() { + site_id.clone() + } else { + display_name.to_string() + } + }, + connection_id, + created_at: existing.created_at, + is_healthy: existing.is_healthy, + last_health_check: existing.last_health_check, + }; + registry.update_peer(peer); + + let message = format!("Peer site '{}' updated.", site_id); + if wants_json { + return axum::Json(json!({ "ok": true, "message": message })).into_response(); + } + session.write(|s| s.push_flash("success", message)); + Redirect::to("/ui/sites").into_response() +} + +pub async fn delete_peer_site( + State(state): State, + Extension(session): Extension, + Path(site_id): Path, + headers: HeaderMap, + Form(_form): Form, +) -> Response { + let wants_json = wants_json(&headers); + let Some(registry) = &state.site_registry else { + let message = "Site registry is not available.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/sites").into_response(); + }; + + if registry.delete_peer(&site_id) { + let message = format!("Peer site '{}' deleted.", site_id); + if wants_json { + return axum::Json(json!({ "ok": true, "message": message })).into_response(); + } + session.write(|s| s.push_flash("success", message)); + } else { + let message = format!("Peer site '{}' not found.", site_id); + if wants_json { + return ( + StatusCode::NOT_FOUND, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + } + + Redirect::to("/ui/sites").into_response() +} + pub async fn connections_dashboard( State(state): State, Extension(session): Extension, @@ -233,24 +1559,211 @@ pub async fn metrics_dashboard( Extension(session): Extension, ) -> Response { let mut ctx = page_context(&state, &session, "ui.metrics_dashboard"); - ctx.insert("metrics_enabled", &state.config.metrics_enabled); + ctx.insert( + "metrics_enabled", + &(state.config.metrics_enabled || state.config.metrics_history_enabled), + ); + ctx.insert( + "metrics_history_enabled", + &state.config.metrics_history_enabled, + ); + ctx.insert("operation_metrics_enabled", &state.config.metrics_enabled); ctx.insert("history", &Vec::::new()); ctx.insert("operation_metrics", &Vec::::new()); - ctx.insert("summary", &json!({})); + + let metrics = crate::handlers::ui_api::collect_metrics(&state).await; + let cpu_percent = metrics + .get("cpu_percent") + .and_then(|v| v.as_f64()) + .unwrap_or(0.0); + let memory = metrics + .get("memory") + .cloned() + .unwrap_or_else(|| json!({ "percent": 0, "total": "0 B", "used": "0 B" })); + let disk = metrics + .get("disk") + .cloned() + .unwrap_or_else(|| json!({ "percent": 0, "free": "0 B", "total": "0 B" })); + let app = metrics.get("app").cloned().unwrap_or_else(|| { + json!({ + "buckets": 0, "objects": 0, "storage_used": "0 B", + "uptime_days": 0, "versions": 0, + }) + }); + let mem_pct = memory + .get("percent") + .and_then(|v| v.as_f64()) + .unwrap_or(0.0); + let disk_pct = disk.get("percent").and_then(|v| v.as_f64()).unwrap_or(0.0); + let has_issues = cpu_percent > 80.0 || mem_pct > 85.0 || disk_pct > 90.0; + + ctx.insert("cpu_percent", &cpu_percent); + ctx.insert("memory", &memory); + ctx.insert("disk", &disk); + ctx.insert("app", &app); + ctx.insert("has_issues", &has_issues); + ctx.insert( + "summary", + &json!({ + "app": app, + "cpu_percent": cpu_percent, + "disk": disk, + "memory": memory, + "has_issues": has_issues, + }), + ); render(&state, "metrics.html", &ctx) } +fn format_history_timestamp(timestamp: Option) -> String { + let Some(timestamp) = timestamp else { + return "-".to_string(); + }; + let millis = (timestamp * 1000.0).round() as i64; + chrono::DateTime::::from_timestamp_millis(millis) + .map(|dt| dt.format("%Y-%m-%d %H:%M:%S UTC").to_string()) + .unwrap_or_else(|| "-".to_string()) +} + +fn format_byte_count(bytes: u64) -> String { + const UNITS: [&str; 5] = ["B", "KB", "MB", "GB", "TB"]; + let mut value = bytes as f64; + let mut unit = 0usize; + while value >= 1024.0 && unit < UNITS.len() - 1 { + value /= 1024.0; + unit += 1; + } + if unit == 0 { + format!("{} {}", bytes, UNITS[unit]) + } else { + format!("{value:.1} {}", UNITS[unit]) + } +} + +fn decorate_gc_history(executions: &[Value]) -> Vec { + executions + .iter() + .cloned() + .map(|mut execution| { + let timestamp = execution.get("timestamp").and_then(|value| value.as_f64()); + let bytes_freed = execution + .get("result") + .and_then(|value| value.get("temp_bytes_freed")) + .and_then(|value| value.as_u64()) + .unwrap_or(0); + if let Some(obj) = execution.as_object_mut() { + obj.insert( + "timestamp_display".to_string(), + Value::String(format_history_timestamp(timestamp)), + ); + obj.insert( + "bytes_freed_display".to_string(), + Value::String(format_byte_count(bytes_freed)), + ); + } + execution + }) + .collect() +} + +fn decorate_integrity_history(executions: &[Value]) -> Vec { + executions + .iter() + .cloned() + .map(|mut execution| { + let timestamp = execution.get("timestamp").and_then(|value| value.as_f64()); + if let Some(obj) = execution.as_object_mut() { + obj.insert( + "timestamp_display".to_string(), + Value::String(format_history_timestamp(timestamp)), + ); + } + execution + }) + .collect() +} + pub async fn system_dashboard( State(state): State, Extension(session): Extension, ) -> Response { let mut ctx = page_context(&state, &session, "ui.system_dashboard"); + + let gc_status = match &state.gc { + Some(gc) => gc.status().await, + None => json!({ + "dry_run": false, + "enabled": false, + "interval_hours": 6, + "lock_file_max_age_hours": 1, + "multipart_max_age_days": 7, + "running": false, + "scanning": false, + "scan_elapsed_seconds": Value::Null, + "temp_file_max_age_hours": 24, + }), + }; + let gc_history = match &state.gc { + Some(gc) => gc + .history() + .await + .get("executions") + .and_then(|value| value.as_array()) + .map(|values| decorate_gc_history(values)) + .unwrap_or_default(), + None => Vec::new(), + }; + + let integrity_status = match &state.integrity { + Some(checker) => checker.status().await, + None => json!({ + "auto_heal": false, + "batch_size": 100, + "dry_run": false, + "enabled": false, + "interval_hours": 24, + "running": false, + "scanning": false, + "scan_elapsed_seconds": Value::Null, + }), + }; + let integrity_history = match &state.integrity { + Some(checker) => checker + .history() + .await + .get("executions") + .and_then(|value| value.as_array()) + .map(|values| decorate_integrity_history(values)) + .unwrap_or_default(), + None => Vec::new(), + }; + ctx.insert("gc_enabled", &state.config.gc_enabled); ctx.insert("integrity_enabled", &state.config.integrity_enabled); - ctx.insert("gc_history", &Vec::::new()); - ctx.insert("integrity_history", &Vec::::new()); - ctx.insert("gc_status", &json!({"running": false})); - ctx.insert("integrity_status", &json!({"running": false})); + ctx.insert("gc_history", &gc_history); + ctx.insert("integrity_history", &integrity_history); + ctx.insert("gc_status", &gc_status); + ctx.insert("integrity_status", &integrity_status); + ctx.insert("app_version", &env!("CARGO_PKG_VERSION")); + ctx.insert("display_timezone", &"UTC"); + ctx.insert("platform", &std::env::consts::OS); + ctx.insert( + "storage_root", + &state.config.storage_root.display().to_string(), + ); + ctx.insert("total_issues", &0); + let features = vec![ + json!({"label": "Encryption (SSE-S3)", "enabled": state.config.encryption_enabled}), + json!({"label": "KMS", "enabled": state.config.kms_enabled}), + json!({"label": "Versioning Lifecycle", "enabled": state.config.lifecycle_enabled}), + json!({"label": "Metrics History", "enabled": state.config.metrics_history_enabled}), + json!({"label": "Operation Metrics", "enabled": state.config.metrics_enabled}), + json!({"label": "Site Sync", "enabled": state.config.site_sync_enabled}), + json!({"label": "Website Hosting", "enabled": state.config.website_hosting_enabled}), + json!({"label": "Garbage Collection", "enabled": state.config.gc_enabled}), + json!({"label": "Integrity Scanner", "enabled": state.config.integrity_enabled}), + ]; + ctx.insert("features", &features); render(&state, "system.html", &ctx) } @@ -259,26 +1772,327 @@ pub async fn website_domains_dashboard( Extension(session): Extension, ) -> Response { let mut ctx = page_context(&state, &session, "ui.website_domains_dashboard"); - ctx.insert("domains", &Vec::::new()); + let buckets: Vec = state + .storage + .list_buckets() + .await + .map(|list| list.into_iter().map(|b| b.name).collect()) + .unwrap_or_default(); + let mappings = state + .website_domains + .as_ref() + .map(|store| { + let mut mappings = store.list_all(); + mappings.sort_by(|a, b| { + let a_domain = a + .get("domain") + .and_then(|value| value.as_str()) + .unwrap_or(""); + let b_domain = b + .get("domain") + .and_then(|value| value.as_str()) + .unwrap_or(""); + a_domain.cmp(b_domain) + }); + mappings + }) + .unwrap_or_default(); + ctx.insert("domains", &mappings); + ctx.insert("mappings", &mappings); + ctx.insert("buckets", &buckets); render(&state, "website_domains.html", &ctx) } pub async fn replication_wizard( State(state): State, Extension(session): Extension, + Query(q): Query>, ) -> Response { let mut ctx = page_context(&state, &session, "ui.replication_wizard"); - ctx.insert("connections", &Vec::::new()); - ctx.insert("local_site", &Value::Null); - ctx.insert("peers", &Vec::::new()); + + let site_id = q.get("site_id").cloned().unwrap_or_default(); + let peer_record = state + .site_registry + .as_ref() + .and_then(|reg| { + if site_id.is_empty() { + reg.list_peers().into_iter().next() + } else { + reg.get_peer(&site_id) + } + }) + .map(|p| { + json!({ + "site_id": p.site_id, + "display_name": p.display_name, + "endpoint": p.endpoint, + "region": p.region, + "connection_id": p.connection_id, + }) + }) + .unwrap_or_else(|| { + json!({ + "site_id": site_id, + "display_name": "", + "endpoint": "", + "region": "us-east-1", + }) + }); + let peer_connection_id = peer_record + .get("connection_id") + .and_then(|v| v.as_str()) + .unwrap_or_default() + .to_string(); + + let local_site = state + .site_registry + .as_ref() + .and_then(|reg| reg.get_local_site()) + .map(|s| { + json!({ + "site_id": s.site_id, + "display_name": s.display_name, + "endpoint": s.endpoint, + "region": s.region, + }) + }) + .unwrap_or(Value::Null); + + let peers: Vec = state + .site_registry + .as_ref() + .map(|reg| { + reg.list_peers() + .into_iter() + .map(|p| { + json!({ + "site_id": p.site_id, + "display_name": p.display_name, + "endpoint": p.endpoint, + "region": p.region, + "connection_id": p.connection_id, + }) + }) + .collect() + }) + .unwrap_or_default(); + + let all_rules = state.replication.list_rules(); + let bucket_names: Vec = state + .storage + .list_buckets() + .await + .map(|list| list.into_iter().map(|b| b.name).collect()) + .unwrap_or_default(); + let buckets: Vec = bucket_names + .into_iter() + .map(|bucket_name| { + let existing_rule = all_rules + .iter() + .find(|rule| rule.bucket_name == bucket_name); + let has_rule_for_peer = existing_rule + .map(|rule| rule.target_connection_id == peer_connection_id) + .unwrap_or(false); + json!({ + "name": bucket_name, + "has_rule": has_rule_for_peer, + "existing_mode": if has_rule_for_peer { + existing_rule.map(|rule| rule.mode.clone()) + } else { + None:: + }, + "existing_target": if has_rule_for_peer { + existing_rule.map(|rule| rule.target_bucket.clone()) + } else { + None:: + }, + }) + }) + .collect(); + + let conns: Vec = state + .connections + .list() + .into_iter() + .map(|c| { + json!({ + "id": c.id, + "name": c.name, + "endpoint_url": c.endpoint_url, + "region": c.region, + "access_key": c.access_key, + }) + }) + .collect(); + + let connection = conns + .iter() + .find(|conn| { + conn.get("id") + .and_then(|value| value.as_str()) + .map(|id| id == peer_connection_id) + .unwrap_or(false) + }) + .cloned() + .or_else(|| conns.first().cloned()) + .unwrap_or_else( + || json!({ "id": "", "name": "", "endpoint_url": "", "region": "", "access_key": "" }), + ); + + ctx.insert("peer", &peer_record); + ctx.insert("peers", &peers); + ctx.insert("local_site", &local_site); + ctx.insert("connections", &conns); + ctx.insert("connection", &connection); + ctx.insert("buckets", &buckets); render(&state, "replication_wizard.html", &ctx) } +#[derive(serde::Deserialize)] +pub struct CreatePeerReplicationRulesForm { + #[serde(default)] + pub mode: String, + #[serde(default)] + pub buckets: Vec, + #[serde(default)] + pub csrf_token: String, + #[serde(flatten)] + pub extras: HashMap, +} + +pub async fn create_peer_replication_rules( + State(state): State, + Extension(session): Extension, + Path(site_id): Path, + Form(form): Form, +) -> Response { + create_peer_replication_rules_impl(state, session, site_id, form).await +} + +pub async fn create_peer_replication_rules_from_query( + State(state): State, + Extension(session): Extension, + Query(q): Query>, + Form(form): Form, +) -> Response { + let site_id = q.get("site_id").cloned().unwrap_or_default(); + create_peer_replication_rules_impl(state, session, site_id, form).await +} + +async fn create_peer_replication_rules_impl( + state: AppState, + session: SessionHandle, + site_id: String, + form: CreatePeerReplicationRulesForm, +) -> Response { + let Some(registry) = &state.site_registry else { + session.write(|s| s.push_flash("danger", "Site registry is not available.")); + return Redirect::to("/ui/sites").into_response(); + }; + let Some(peer) = registry.get_peer(&site_id) else { + session.write(|s| s.push_flash("danger", format!("Peer site '{}' not found.", site_id))); + return Redirect::to("/ui/sites").into_response(); + }; + let Some(connection_id) = peer.connection_id.clone() else { + session.write(|s| { + s.push_flash( + "danger", + "This peer has no connection configured. Add a connection first.", + ) + }); + return Redirect::to("/ui/sites").into_response(); + }; + if state.connections.get(&connection_id).is_none() { + session.write(|s| { + s.push_flash( + "danger", + format!("Connection '{}' was not found.", connection_id), + ) + }); + return Redirect::to("/ui/sites").into_response(); + } + + let mode = match form.mode.trim() { + crate::services::replication::MODE_ALL => crate::services::replication::MODE_ALL, + crate::services::replication::MODE_BIDIRECTIONAL => { + crate::services::replication::MODE_BIDIRECTIONAL + } + _ => crate::services::replication::MODE_NEW_ONLY, + } + .to_string(); + + if form.buckets.is_empty() { + session.write(|s| s.push_flash("warning", "No buckets selected.")); + return Redirect::to("/ui/sites").into_response(); + } + + let mut created = 0usize; + let mut created_existing = Vec::new(); + + for bucket_name in form.buckets { + let target_key = format!("target_{}", bucket_name); + let target_bucket = form + .extras + .get(&target_key) + .map(|value| value.trim()) + .filter(|value| !value.is_empty()) + .unwrap_or(bucket_name.as_str()) + .to_string(); + + let rule = crate::services::replication::ReplicationRule { + bucket_name: bucket_name.clone(), + target_connection_id: connection_id.clone(), + target_bucket, + enabled: true, + mode: mode.clone(), + created_at: Some(chrono::Utc::now().timestamp_millis() as f64 / 1000.0), + stats: Default::default(), + sync_deletions: true, + last_pull_at: None, + filter_prefix: None, + }; + + state.replication.set_rule(rule); + created += 1; + if mode == crate::services::replication::MODE_ALL { + created_existing.push(bucket_name); + } + } + + for bucket_name in created_existing { + state + .replication + .clone() + .schedule_existing_objects_sync(bucket_name); + } + + if created > 0 { + session.write(|s| { + s.push_flash( + "success", + format!( + "Created {} replication rule(s) for {}.", + created, + if peer.display_name.is_empty() { + peer.site_id.as_str() + } else { + peer.display_name.as_str() + } + ), + ) + }); + } + Redirect::to("/ui/sites").into_response() +} + pub async fn docs_page( State(state): State, Extension(session): Extension, ) -> Response { - let ctx = page_context(&state, &session, "ui.docs_page"); + let mut ctx = page_context(&state, &session, "ui.docs_page"); + let (api_base, api_host) = parse_api_base(&state); + ctx.insert("api_base", &api_base); + ctx.insert("api_host", &api_host); render(&state, "docs.html", &ctx) } @@ -292,22 +2106,905 @@ pub struct CreateBucketForm { pub async fn create_bucket( State(state): State, Extension(session): Extension, + headers: HeaderMap, axum::extract::Form(form): axum::extract::Form, ) -> Response { - match state.storage.create_bucket(form.bucket_name.trim()).await { + let wants_json = wants_json(&headers); + let bucket_name = form.bucket_name.trim().to_string(); + + if bucket_name.is_empty() { + let message = "Bucket name is required".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/buckets").into_response(); + } + + match state.storage.create_bucket(&bucket_name).await { Ok(()) => { - session.write(|s| s.push_flash("success", format!("Bucket '{}' created.", form.bucket_name))); + let message = format!("Bucket '{}' created.", bucket_name); + if wants_json { + return axum::Json(json!({ + "success": true, + "message": message, + "bucket_name": bucket_name, + })) + .into_response(); + } + session.write(|s| s.push_flash("success", message)); } Err(e) => { - session.write(|s| s.push_flash("danger", format!("Failed to create bucket: {}", e))); + let message = format!("Failed to create bucket: {}", e); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); } } Redirect::to("/ui/buckets").into_response() } -pub async fn stub_post( - Extension(session): Extension, +#[derive(serde::Deserialize)] +pub struct UpdateBucketVersioningForm { + pub state: String, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn delete_bucket( + State(state): State, + Path(bucket_name): Path, ) -> Response { + match state.storage.delete_bucket(&bucket_name).await { + Ok(()) => axum::Json(json!({ + "ok": true, + "message": format!("Bucket '{}' deleted.", bucket_name), + })) + .into_response(), + Err(e) => ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(), + } +} + +pub async fn update_bucket_versioning( + State(state): State, + Path(bucket_name): Path, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let enabled = form.state.eq_ignore_ascii_case("enable"); + match state.storage.set_versioning(&bucket_name, enabled).await { + Ok(()) => axum::Json(json!({ + "ok": true, + "enabled": enabled, + "message": if enabled { "Versioning enabled." } else { "Versioning suspended." }, + })) + .into_response(), + Err(e) => ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(), + } +} + +fn empty_string_as_none<'de, D, T>(deserializer: D) -> Result, D::Error> +where + D: serde::Deserializer<'de>, + T: std::str::FromStr, + T::Err: std::fmt::Display, +{ + use serde::Deserialize; + let opt = Option::::deserialize(deserializer)?; + match opt.as_deref() { + None | Some("") => Ok(None), + Some(s) => s.parse::().map(Some).map_err(serde::de::Error::custom), + } +} + +#[derive(serde::Deserialize)] +pub struct UpdateBucketQuotaForm { + pub action: String, + #[serde(default, deserialize_with = "empty_string_as_none")] + pub max_mb: Option, + #[serde(default, deserialize_with = "empty_string_as_none")] + pub max_objects: Option, + #[serde(default)] + pub csrf_token: String, +} + +#[derive(serde::Deserialize)] +pub struct UpdateBucketReplicationForm { + pub action: String, + #[serde(default)] + pub target_connection_id: String, + #[serde(default)] + pub target_bucket: String, + #[serde(default)] + pub replication_mode: String, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn update_bucket_replication( + State(state): State, + Extension(session): Extension, + Path(bucket_name): Path, + headers: HeaderMap, + Form(form): Form, +) -> Response { + let wants_json = wants_json(&headers); + + let respond = |ok: bool, status: StatusCode, message: String, extra: Value| -> Response { + if wants_json { + let mut payload = json!({ + "ok": ok, + "message": message, + }); + if let Some(obj) = payload.as_object_mut() { + if let Some(extra_obj) = extra.as_object() { + for (key, value) in extra_obj { + obj.insert(key.clone(), value.clone()); + } + } + } + return (status, axum::Json(payload)).into_response(); + } + + session.write(|s| s.push_flash(if ok { "success" } else { "danger" }, message)); + bucket_tab_redirect(&bucket_name, "replication") + }; + + match form.action.as_str() { + "delete" => { + state.replication.delete_rule(&bucket_name); + respond( + true, + StatusCode::OK, + "Replication configuration removed.".to_string(), + json!({ "action": "delete", "enabled": false }), + ) + } + "pause" => { + let Some(mut rule) = state.replication.get_rule(&bucket_name) else { + return respond( + false, + StatusCode::NOT_FOUND, + "No replication configuration to pause.".to_string(), + json!({ "error": "No replication configuration to pause" }), + ); + }; + rule.enabled = false; + state.replication.set_rule(rule); + respond( + true, + StatusCode::OK, + "Replication paused.".to_string(), + json!({ "action": "pause", "enabled": false }), + ) + } + "resume" => { + let Some(mut rule) = state.replication.get_rule(&bucket_name) else { + return respond( + false, + StatusCode::NOT_FOUND, + "No replication configuration to resume.".to_string(), + json!({ "error": "No replication configuration to resume" }), + ); + }; + rule.enabled = true; + let mode = rule.mode.clone(); + state.replication.set_rule(rule); + + let message = if mode == crate::services::replication::MODE_ALL { + state + .replication + .clone() + .schedule_existing_objects_sync(bucket_name.clone()); + "Replication resumed. Existing object sync will continue in the background." + .to_string() + } else { + "Replication resumed.".to_string() + }; + + respond( + true, + StatusCode::OK, + message, + json!({ "action": "resume", "enabled": true, "mode": mode }), + ) + } + "create" => { + let target_connection_id = form.target_connection_id.trim(); + let target_bucket = form.target_bucket.trim(); + if target_connection_id.is_empty() || target_bucket.is_empty() { + return respond( + false, + StatusCode::BAD_REQUEST, + "Target connection and bucket are required.".to_string(), + json!({ "error": "Target connection and bucket are required" }), + ); + } + if state.connections.get(target_connection_id).is_none() { + return respond( + false, + StatusCode::BAD_REQUEST, + "Target connection was not found.".to_string(), + json!({ "error": "Target connection was not found" }), + ); + } + + let mode = match form.replication_mode.trim() { + crate::services::replication::MODE_ALL => crate::services::replication::MODE_ALL, + crate::services::replication::MODE_BIDIRECTIONAL => { + crate::services::replication::MODE_BIDIRECTIONAL + } + _ => crate::services::replication::MODE_NEW_ONLY, + }; + + state + .replication + .set_rule(crate::services::replication::ReplicationRule { + bucket_name: bucket_name.clone(), + target_connection_id: target_connection_id.to_string(), + target_bucket: target_bucket.to_string(), + enabled: true, + mode: mode.to_string(), + created_at: Some( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs_f64()) + .unwrap_or(0.0), + ), + stats: crate::services::replication::ReplicationStats::default(), + sync_deletions: true, + last_pull_at: None, + filter_prefix: None, + }); + + let message = if mode == crate::services::replication::MODE_ALL { + state + .replication + .clone() + .schedule_existing_objects_sync(bucket_name.clone()); + "Replication configured. Existing object sync will continue in the background." + .to_string() + } else { + "Replication configured. New uploads will be replicated.".to_string() + }; + + respond( + true, + StatusCode::OK, + message, + json!({ + "action": "create", + "enabled": true, + "mode": mode, + "target_connection_id": target_connection_id, + "target_bucket": target_bucket, + }), + ) + } + _ => respond( + false, + StatusCode::BAD_REQUEST, + "Invalid replication action.".to_string(), + json!({ "error": "Invalid action" }), + ), + } +} + +#[derive(serde::Deserialize)] +pub struct ConnectionForm { + pub name: String, + pub endpoint_url: String, + pub access_key: String, + #[serde(default)] + pub secret_key: String, + #[serde(default = "default_connection_region")] + pub region: String, + #[serde(default)] + pub csrf_token: String, +} + +fn default_connection_region() -> String { + "us-east-1".to_string() +} + +pub async fn create_connection( + State(state): State, + Extension(session): Extension, + headers: HeaderMap, + Form(form): Form, +) -> Response { + let wants_json = wants_json(&headers); + let name = form.name.trim(); + let endpoint = form.endpoint_url.trim(); + let access_key = form.access_key.trim(); + let secret_key = form.secret_key.trim(); + let region = form.region.trim(); + + if name.is_empty() || endpoint.is_empty() || access_key.is_empty() || secret_key.is_empty() { + let message = "All connection fields are required.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/connections").into_response(); + } + + let connection = crate::stores::connections::RemoteConnection { + id: uuid::Uuid::new_v4().to_string(), + name: name.to_string(), + endpoint_url: endpoint.to_string(), + access_key: access_key.to_string(), + secret_key: secret_key.to_string(), + region: if region.is_empty() { + default_connection_region() + } else { + region.to_string() + }, + }; + + match state.connections.add(connection.clone()) { + Ok(()) => { + let message = format!("Connection '{}' created.", connection.name); + if wants_json { + axum::Json(json!({ + "ok": true, + "message": message, + "connection": { + "id": connection.id, + "name": connection.name, + "endpoint_url": connection.endpoint_url, + "access_key": connection.access_key, + "region": connection.region, + } + })) + .into_response() + } else { + session.write(|s| s.push_flash("success", message)); + Redirect::to("/ui/connections").into_response() + } + } + Err(err) => { + let message = format!("Failed to create connection: {}", err); + if wants_json { + ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response() + } else { + session.write(|s| s.push_flash("danger", message)); + Redirect::to("/ui/connections").into_response() + } + } + } +} + +pub async fn update_connection( + State(state): State, + Extension(session): Extension, + Path(connection_id): Path, + headers: HeaderMap, + Form(form): Form, +) -> Response { + let wants_json = wants_json(&headers); + let Some(mut connection) = state.connections.get(&connection_id) else { + let message = "Connection not found.".to_string(); + if wants_json { + return ( + StatusCode::NOT_FOUND, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/connections").into_response(); + }; + + let name = form.name.trim(); + let endpoint = form.endpoint_url.trim(); + let access_key = form.access_key.trim(); + let secret_key = form.secret_key.trim(); + let region = form.region.trim(); + + if name.is_empty() || endpoint.is_empty() || access_key.is_empty() { + let message = "Name, endpoint, and access key are required.".to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to("/ui/connections").into_response(); + } + + connection.name = name.to_string(); + connection.endpoint_url = endpoint.to_string(); + connection.access_key = access_key.to_string(); + if !secret_key.is_empty() { + connection.secret_key = secret_key.to_string(); + } + connection.region = if region.is_empty() { + default_connection_region() + } else { + region.to_string() + }; + + match state.connections.add(connection.clone()) { + Ok(()) => { + let message = format!("Connection '{}' updated.", connection.name); + if wants_json { + axum::Json(json!({ + "ok": true, + "message": message, + "connection": { + "id": connection.id, + "name": connection.name, + "endpoint_url": connection.endpoint_url, + "access_key": connection.access_key, + "region": connection.region, + } + })) + .into_response() + } else { + session.write(|s| s.push_flash("success", message)); + Redirect::to("/ui/connections").into_response() + } + } + Err(err) => { + let message = format!("Failed to update connection: {}", err); + if wants_json { + ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response() + } else { + session.write(|s| s.push_flash("danger", message)); + Redirect::to("/ui/connections").into_response() + } + } + } +} + +#[derive(serde::Deserialize, Default)] +pub struct DeleteConnectionForm { + #[serde(default)] + pub csrf_token: String, +} + +pub async fn delete_connection( + State(state): State, + Extension(session): Extension, + Path(connection_id): Path, + headers: HeaderMap, + Form(_form): Form, +) -> Response { + let wants_json = wants_json(&headers); + match state.connections.delete(&connection_id) { + Ok(true) => { + let message = "Connection deleted.".to_string(); + if wants_json { + axum::Json(json!({ "ok": true, "message": message })).into_response() + } else { + session.write(|s| s.push_flash("success", message)); + Redirect::to("/ui/connections").into_response() + } + } + Ok(false) => { + let message = "Connection not found.".to_string(); + if wants_json { + ( + StatusCode::NOT_FOUND, + axum::Json(json!({ "error": message })), + ) + .into_response() + } else { + session.write(|s| s.push_flash("danger", message)); + Redirect::to("/ui/connections").into_response() + } + } + Err(err) => { + let message = format!("Failed to delete connection: {}", err); + if wants_json { + ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response() + } else { + session.write(|s| s.push_flash("danger", message)); + Redirect::to("/ui/connections").into_response() + } + } + } +} + +#[derive(serde::Deserialize)] +pub struct WebsiteDomainForm { + pub bucket: String, + #[serde(default)] + pub domain: String, + #[serde(default)] + pub csrf_token: String, +} + +#[derive(serde::Deserialize, Default)] +pub struct WebsiteDomainDeleteForm { + #[serde(default)] + pub csrf_token: String, +} + +pub async fn create_website_domain( + State(state): State, + Extension(session): Extension, + Form(form): Form, +) -> Response { + let Some(store) = &state.website_domains else { + session.write(|s| s.push_flash("danger", "Website hosting is not enabled.")); + return Redirect::to("/ui/website-domains").into_response(); + }; + + let domain = crate::services::website_domains::normalize_domain(&form.domain); + let bucket = form.bucket.trim().to_string(); + if !crate::services::website_domains::is_valid_domain(&domain) { + session.write(|s| s.push_flash("danger", "Enter a valid domain name.")); + return Redirect::to("/ui/website-domains").into_response(); + } + match state.storage.bucket_exists(&bucket).await { + Ok(true) => {} + _ => { + session + .write(|s| s.push_flash("danger", format!("Bucket '{}' does not exist.", bucket))); + return Redirect::to("/ui/website-domains").into_response(); + } + } + store.set_mapping(&domain, &bucket); + session.write(|s| { + s.push_flash( + "success", + format!("Domain '{}' mapped to '{}'.", domain, bucket), + ) + }); + Redirect::to("/ui/website-domains").into_response() +} + +pub async fn update_website_domain( + State(state): State, + Extension(session): Extension, + Path(domain): Path, + Form(form): Form, +) -> Response { + let Some(store) = &state.website_domains else { + session.write(|s| s.push_flash("danger", "Website hosting is not enabled.")); + return Redirect::to("/ui/website-domains").into_response(); + }; + + let domain = crate::services::website_domains::normalize_domain(&domain); + let bucket = form.bucket.trim().to_string(); + match state.storage.bucket_exists(&bucket).await { + Ok(true) => {} + _ => { + session + .write(|s| s.push_flash("danger", format!("Bucket '{}' does not exist.", bucket))); + return Redirect::to("/ui/website-domains").into_response(); + } + } + if store.get_bucket(&domain).is_none() { + session.write(|s| s.push_flash("danger", format!("Domain '{}' was not found.", domain))); + return Redirect::to("/ui/website-domains").into_response(); + } + store.set_mapping(&domain, &bucket); + session.write(|s| s.push_flash("success", format!("Domain '{}' updated.", domain))); + Redirect::to("/ui/website-domains").into_response() +} + +pub async fn delete_website_domain( + State(state): State, + Extension(session): Extension, + Path(domain): Path, + Form(_form): Form, +) -> Response { + let Some(store) = &state.website_domains else { + session.write(|s| s.push_flash("danger", "Website hosting is not enabled.")); + return Redirect::to("/ui/website-domains").into_response(); + }; + + let domain = crate::services::website_domains::normalize_domain(&domain); + if store.delete_mapping(&domain) { + session.write(|s| s.push_flash("success", format!("Domain '{}' removed.", domain))); + } else { + session.write(|s| s.push_flash("danger", format!("Domain '{}' was not found.", domain))); + } + Redirect::to("/ui/website-domains").into_response() +} + +pub async fn update_bucket_quota( + State(state): State, + Path(bucket_name): Path, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let mut config = match state.storage.get_bucket_config(&bucket_name).await { + Ok(cfg) => cfg, + Err(e) => { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(); + } + }; + + if form.action.eq_ignore_ascii_case("remove") { + config.quota = None; + } else { + config.quota = Some(myfsio_common::types::QuotaConfig { + max_bytes: form.max_mb.map(|mb| mb.saturating_mul(1024 * 1024)), + max_objects: form.max_objects, + }); + } + + match state.storage.set_bucket_config(&bucket_name, &config).await { + Ok(()) => axum::Json(json!({ + "ok": true, + "has_quota": config.quota.is_some(), + "max_bytes": config.quota.as_ref().and_then(|q| q.max_bytes), + "max_objects": config.quota.as_ref().and_then(|q| q.max_objects), + "message": if config.quota.is_some() { "Quota settings saved." } else { "Quota removed." }, + })) + .into_response(), + Err(e) => ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(), + } +} + +#[derive(serde::Deserialize)] +pub struct UpdateBucketEncryptionForm { + pub action: String, + #[serde(default)] + pub algorithm: String, + #[serde(default)] + pub kms_key_id: String, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn update_bucket_encryption( + State(state): State, + Path(bucket_name): Path, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let mut config = match state.storage.get_bucket_config(&bucket_name).await { + Ok(cfg) => cfg, + Err(e) => { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(); + } + }; + + if form.action.eq_ignore_ascii_case("disable") { + config.encryption = None; + } else { + let mut inner = json!({ + "SSEAlgorithm": if form.algorithm == "aws:kms" { "aws:kms" } else { "AES256" } + }); + if form.algorithm == "aws:kms" && !form.kms_key_id.trim().is_empty() { + inner["KMSMasterKeyID"] = Value::String(form.kms_key_id.trim().to_string()); + } + config.encryption = Some(json!({ + "Rules": [{ + "ApplyServerSideEncryptionByDefault": inner + }] + })); + } + + match state.storage.set_bucket_config(&bucket_name, &config).await { + Ok(()) => { + let algorithm = config + .encryption + .as_ref() + .and_then(|value| value.get("Rules")) + .and_then(|rules| rules.as_array()) + .and_then(|rules| rules.first()) + .and_then(|rule| rule.get("ApplyServerSideEncryptionByDefault")) + .and_then(|inner| inner.get("SSEAlgorithm")) + .and_then(|v| v.as_str()) + .unwrap_or("AES256"); + axum::Json(json!({ + "ok": true, + "enabled": config.encryption.is_some(), + "algorithm": algorithm, + "message": if config.encryption.is_some() { "Encryption settings saved." } else { "Encryption disabled." }, + })) + .into_response() + } + Err(e) => ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(), + } +} + +#[derive(serde::Deserialize)] +pub struct UpdateBucketPolicyForm { + pub mode: String, + #[serde(default)] + pub policy_document: String, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn update_bucket_policy( + State(state): State, + Extension(session): Extension, + Path(bucket_name): Path, + headers: HeaderMap, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let wants_json = wants_json(&headers); + let redirect_url = format!("/ui/buckets/{}?tab=permissions", bucket_name); + let mut config = match state.storage.get_bucket_config(&bucket_name).await { + Ok(cfg) => cfg, + Err(e) => { + let message = e.to_string(); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to(&redirect_url).into_response(); + } + }; + + if form.mode.eq_ignore_ascii_case("delete") { + config.policy = None; + } else { + let policy: Value = match serde_json::from_str(&form.policy_document) { + Ok(value) => value, + Err(e) => { + let message = format!("Invalid policy JSON: {}", e); + if wants_json { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response(); + } + session.write(|s| s.push_flash("danger", message)); + return Redirect::to(&redirect_url).into_response(); + } + }; + config.policy = Some(policy); + } + + match state.storage.set_bucket_config(&bucket_name, &config).await { + Ok(()) => { + let message = if config.policy.is_some() { + "Bucket policy saved." + } else { + "Bucket policy deleted." + }; + if wants_json { + axum::Json(json!({ + "ok": true, + "message": message, + })) + .into_response() + } else { + session.write(|s| s.push_flash("success", message)); + Redirect::to(&redirect_url).into_response() + } + } + Err(e) => { + let message = e.to_string(); + if wants_json { + ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": message })), + ) + .into_response() + } else { + session.write(|s| s.push_flash("danger", message)); + Redirect::to(&redirect_url).into_response() + } + } + } +} + +#[derive(serde::Deserialize)] +pub struct UpdateBucketWebsiteForm { + pub action: String, + #[serde(default)] + pub index_document: String, + #[serde(default)] + pub error_document: String, + #[serde(default)] + pub csrf_token: String, +} + +pub async fn update_bucket_website( + State(state): State, + Path(bucket_name): Path, + axum::extract::Form(form): axum::extract::Form, +) -> Response { + let mut config = match state.storage.get_bucket_config(&bucket_name).await { + Ok(cfg) => cfg, + Err(e) => { + return ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(); + } + }; + + if form.action.eq_ignore_ascii_case("disable") { + config.website = None; + } else { + let index_document = if form.index_document.trim().is_empty() { + "index.html".to_string() + } else { + form.index_document.trim().to_string() + }; + let error_document = form.error_document.trim().to_string(); + config.website = Some(json!({ + "index_document": index_document, + "error_document": if error_document.is_empty() { Value::Null } else { Value::String(error_document) } + })); + } + + match state.storage.set_bucket_config(&bucket_name, &config).await { + Ok(()) => { + let website = config.website.clone().unwrap_or(Value::Null); + axum::Json(json!({ + "ok": true, + "enabled": !website.is_null(), + "index_document": website.get("index_document").and_then(|v| v.as_str()).unwrap_or("index.html"), + "error_document": website.get("error_document").and_then(|v| v.as_str()).unwrap_or(""), + "message": if website.is_null() { "Website hosting disabled." } else { "Website settings saved." }, + })) + .into_response() + } + Err(e) => ( + StatusCode::BAD_REQUEST, + axum::Json(json!({ "error": e.to_string() })), + ) + .into_response(), + } +} + +pub async fn stub_post(Extension(session): Extension) -> Response { session.write(|s| s.push_flash("info", "This action is not yet implemented in the Rust UI.")); Redirect::to("/ui/buckets").into_response() } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/lib.rs b/rust/myfsio-engine/crates/myfsio-server/src/lib.rs index 34f6ebf..93ee066 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/lib.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/lib.rs @@ -9,24 +9,249 @@ pub mod templates; use axum::Router; -pub const SERVER_HEADER: &str = concat!("MyFSIO-Rust/", env!("CARGO_PKG_VERSION")); +pub const SERVER_HEADER: &str = "MyFSIO"; pub fn create_ui_router(state: state::AppState) -> Router { - use axum::routing::{get, post}; + use axum::routing::{delete, get, post, put}; use handlers::ui; + use handlers::ui_api; use handlers::ui_pages; let protected = Router::new() + .route("/", get(ui::root_redirect)) + .route("/ui", get(ui::root_redirect)) + .route("/ui/", get(ui::root_redirect)) .route("/ui/buckets", get(ui_pages::buckets_overview)) .route("/ui/buckets/create", post(ui_pages::create_bucket)) .route("/ui/buckets/{bucket_name}", get(ui_pages::bucket_detail)) + .route( + "/ui/buckets/{bucket_name}/delete", + post(ui_pages::delete_bucket), + ) + .route( + "/ui/buckets/{bucket_name}/versioning", + post(ui_pages::update_bucket_versioning), + ) + .route( + "/ui/buckets/{bucket_name}/quota", + post(ui_pages::update_bucket_quota), + ) + .route( + "/ui/buckets/{bucket_name}/encryption", + post(ui_pages::update_bucket_encryption), + ) + .route( + "/ui/buckets/{bucket_name}/policy", + post(ui_pages::update_bucket_policy), + ) + .route( + "/ui/buckets/{bucket_name}/replication", + post(ui_pages::update_bucket_replication), + ) + .route( + "/ui/buckets/{bucket_name}/website", + post(ui_pages::update_bucket_website), + ) + .route( + "/ui/buckets/{bucket_name}/upload", + post(ui_api::upload_object), + ) + .route( + "/ui/buckets/{bucket_name}/multipart/initiate", + post(ui_api::initiate_multipart_upload), + ) + .route( + "/ui/buckets/{bucket_name}/multipart/{upload_id}/part", + put(ui_api::upload_multipart_part), + ) + .route( + "/ui/buckets/{bucket_name}/multipart/{upload_id}/complete", + post(ui_api::complete_multipart_upload), + ) + .route( + "/ui/buckets/{bucket_name}/multipart/{upload_id}/abort", + delete(ui_api::abort_multipart_upload), + ) + .route( + "/ui/buckets/{bucket_name}/objects", + get(ui_api::list_bucket_objects), + ) + .route( + "/ui/buckets/{bucket_name}/objects/stream", + get(ui_api::stream_bucket_objects), + ) + .route( + "/ui/buckets/{bucket_name}/folders", + get(ui_api::list_bucket_folders), + ) + .route( + "/ui/buckets/{bucket_name}/copy-targets", + get(ui_api::list_copy_targets), + ) + .route( + "/ui/buckets/{bucket_name}/objects/{*rest}", + get(ui_api::object_get_dispatch).post(ui_api::object_post_dispatch), + ) + .route( + "/ui/buckets/{bucket_name}/acl", + get(ui_api::bucket_acl).post(ui_api::update_bucket_acl), + ) + .route( + "/ui/buckets/{bucket_name}/cors", + get(ui_api::bucket_cors).post(ui_api::update_bucket_cors), + ) + .route( + "/ui/buckets/{bucket_name}/lifecycle", + get(ui_api::bucket_lifecycle).post(ui_api::update_bucket_lifecycle), + ) + .route( + "/ui/buckets/{bucket_name}/lifecycle/history", + get(ui_api::lifecycle_history_stub), + ) + .route( + "/ui/buckets/{bucket_name}/replication/status", + get(ui_api::replication_status), + ) + .route( + "/ui/buckets/{bucket_name}/replication/failures", + get(ui_api::replication_failures).delete(ui_api::clear_replication_failures), + ) + .route( + "/ui/buckets/{bucket_name}/replication/failures/retry", + post(ui_api::retry_replication_failure), + ) + .route( + "/ui/buckets/{bucket_name}/replication/failures/retry-all", + post(ui_api::retry_all_replication_failures), + ) + .route( + "/ui/buckets/{bucket_name}/replication/failures/dismiss", + delete(ui_api::dismiss_replication_failure), + ) + .route( + "/ui/buckets/{bucket_name}/replication/failures/clear", + delete(ui_api::clear_replication_failures), + ) + .route( + "/ui/buckets/{bucket_name}/bulk-delete", + post(ui_api::bulk_delete_objects), + ) + .route( + "/ui/buckets/{bucket_name}/bulk-download", + post(ui_api::bulk_download_objects), + ) + .route( + "/ui/buckets/{bucket_name}/archived", + get(ui_api::archived_objects), + ) + .route( + "/ui/buckets/{bucket_name}/archived/{*rest}", + post(ui_api::archived_post_dispatch), + ) .route("/ui/iam", get(ui_pages::iam_dashboard)) + .route("/ui/iam/users", post(ui_pages::create_iam_user)) + .route("/ui/iam/users/{user_id}", post(ui_pages::update_iam_user)) + .route( + "/ui/iam/users/{user_id}/delete", + post(ui_pages::delete_iam_user), + ) + .route( + "/ui/iam/users/{user_id}/policies", + post(ui_pages::update_iam_policies), + ) + .route( + "/ui/iam/users/{user_id}/expiry", + post(ui_pages::update_iam_expiry), + ) + .route( + "/ui/iam/users/{user_id}/rotate-secret", + post(ui_pages::rotate_iam_secret), + ) + .route("/ui/connections/create", post(ui_pages::create_connection)) + .route("/ui/connections/test", post(ui_api::test_connection)) + .route( + "/ui/connections/{connection_id}", + post(ui_pages::update_connection), + ) + .route( + "/ui/connections/{connection_id}/delete", + post(ui_pages::delete_connection), + ) + .route( + "/ui/connections/{connection_id}/health", + get(ui_api::connection_health), + ) .route("/ui/sites", get(ui_pages::sites_dashboard)) + .route("/ui/sites/local", post(ui_pages::update_local_site)) + .route("/ui/sites/peers", post(ui_pages::add_peer_site)) + .route( + "/ui/sites/peers/{site_id}/update", + post(ui_pages::update_peer_site), + ) + .route( + "/ui/sites/peers/{site_id}/delete", + post(ui_pages::delete_peer_site), + ) + .route("/ui/sites/peers/{site_id}/health", get(ui_api::peer_health)) + .route( + "/ui/sites/peers/{site_id}/sync-stats", + get(ui_api::peer_sync_stats), + ) + .route( + "/ui/sites/peers/{site_id}/bidirectional-status", + get(ui_api::peer_bidirectional_status), + ) .route("/ui/connections", get(ui_pages::connections_dashboard)) .route("/ui/metrics", get(ui_pages::metrics_dashboard)) + .route( + "/ui/metrics/settings", + get(ui_api::metrics_settings).put(ui_api::update_metrics_settings), + ) + .route("/ui/metrics/api", get(ui_api::metrics_api)) + .route("/ui/metrics/history", get(ui_api::metrics_history)) + .route("/ui/metrics/operations", get(ui_api::metrics_operations)) + .route( + "/ui/metrics/operations/history", + get(ui_api::metrics_operations_history), + ) .route("/ui/system", get(ui_pages::system_dashboard)) - .route("/ui/website-domains", get(ui_pages::website_domains_dashboard)) + .route("/ui/system/gc/status", get(ui_api::gc_status_ui)) + .route("/ui/system/gc/run", post(ui_api::gc_run_ui)) + .route("/ui/system/gc/history", get(ui_api::gc_history_ui)) + .route( + "/ui/system/integrity/status", + get(ui_api::integrity_status_ui), + ) + .route("/ui/system/integrity/run", post(ui_api::integrity_run_ui)) + .route( + "/ui/system/integrity/history", + get(ui_api::integrity_history_ui), + ) + .route( + "/ui/website-domains", + get(ui_pages::website_domains_dashboard), + ) + .route( + "/ui/website-domains/create", + post(ui_pages::create_website_domain), + ) + .route( + "/ui/website-domains/{domain}", + post(ui_pages::update_website_domain), + ) + .route( + "/ui/website-domains/{domain}/delete", + post(ui_pages::delete_website_domain), + ) .route("/ui/replication/new", get(ui_pages::replication_wizard)) + .route( + "/ui/replication/create", + post(ui_pages::create_peer_replication_rules_from_query), + ) + .route( + "/ui/sites/peers/{site_id}/replication-rules", + post(ui_pages::create_peer_replication_rules), + ) .route("/ui/docs", get(ui_pages::docs_page)) .layer(axum::middleware::from_fn(ui::require_login)); @@ -40,18 +265,29 @@ pub fn create_ui_router(state: state::AppState) -> Router { secure: false, }; + let static_service = tower_http::services::ServeDir::new(&state.config.static_dir); + protected .merge(public) + .fallback(ui::not_found_page) .layer(axum::middleware::from_fn(middleware::csrf_layer)) .layer(axum::middleware::from_fn_with_state( session_state, middleware::session_layer, )) + .layer(axum::middleware::from_fn_with_state( + state.clone(), + middleware::ui_metrics_layer, + )) .with_state(state) + .nest_service("/static", static_service) + .layer(axum::middleware::from_fn(middleware::server_header)) + .layer(tower_http::compression::CompressionLayer::new()) } pub fn create_router(state: state::AppState) -> Router { let mut router = Router::new() + .route("/myfsio/health", axum::routing::get(handlers::health_check)) .route("/", axum::routing::get(handlers::list_buckets)) .route( "/{bucket}", @@ -61,6 +297,14 @@ pub fn create_router(state: state::AppState) -> Router { .head(handlers::head_bucket) .post(handlers::post_bucket), ) + .route( + "/{bucket}/", + axum::routing::put(handlers::create_bucket) + .get(handlers::get_bucket) + .delete(handlers::delete_bucket) + .head(handlers::head_bucket) + .post(handlers::post_bucket), + ) .route( "/{bucket}/{*key}", axum::routing::put(handlers::put_object) @@ -72,53 +316,189 @@ pub fn create_router(state: state::AppState) -> Router { if state.config.kms_enabled { router = router - .route("/kms/keys", axum::routing::get(handlers::kms::list_keys).post(handlers::kms::create_key)) - .route("/kms/keys/{key_id}", axum::routing::get(handlers::kms::get_key).delete(handlers::kms::delete_key)) - .route("/kms/keys/{key_id}/enable", axum::routing::post(handlers::kms::enable_key)) - .route("/kms/keys/{key_id}/disable", axum::routing::post(handlers::kms::disable_key)) + .route( + "/kms/keys", + axum::routing::get(handlers::kms::list_keys).post(handlers::kms::create_key), + ) + .route( + "/kms/keys/{key_id}", + axum::routing::get(handlers::kms::get_key).delete(handlers::kms::delete_key), + ) + .route( + "/kms/keys/{key_id}/enable", + axum::routing::post(handlers::kms::enable_key), + ) + .route( + "/kms/keys/{key_id}/disable", + axum::routing::post(handlers::kms::disable_key), + ) .route("/kms/encrypt", axum::routing::post(handlers::kms::encrypt)) .route("/kms/decrypt", axum::routing::post(handlers::kms::decrypt)) - .route("/kms/generate-data-key", axum::routing::post(handlers::kms::generate_data_key)); + .route( + "/kms/generate-data-key", + axum::routing::post(handlers::kms::generate_data_key), + ) + .route( + "/kms/generate-data-key-without-plaintext", + axum::routing::post(handlers::kms::generate_data_key_without_plaintext), + ) + .route( + "/kms/re-encrypt", + axum::routing::post(handlers::kms::re_encrypt), + ) + .route( + "/kms/generate-random", + axum::routing::post(handlers::kms::generate_random), + ) + .route( + "/kms/client/generate-key", + axum::routing::post(handlers::kms::client_generate_key), + ) + .route( + "/kms/client/encrypt", + axum::routing::post(handlers::kms::client_encrypt), + ) + .route( + "/kms/client/decrypt", + axum::routing::post(handlers::kms::client_decrypt), + ) + .route( + "/kms/materials/{key_id}", + axum::routing::post(handlers::kms::materials), + ); } router = router - .route("/admin/site/local", axum::routing::get(handlers::admin::get_local_site).put(handlers::admin::update_local_site)) - .route("/admin/site/all", axum::routing::get(handlers::admin::list_all_sites)) - .route("/admin/site/peers", axum::routing::post(handlers::admin::register_peer_site)) - .route("/admin/site/peers/{site_id}", axum::routing::get(handlers::admin::get_peer_site).put(handlers::admin::update_peer_site).delete(handlers::admin::delete_peer_site)) - .route("/admin/site/peers/{site_id}/health", axum::routing::post(handlers::admin::check_peer_health)) - .route("/admin/site/topology", axum::routing::get(handlers::admin::get_topology)) - .route("/admin/site/peers/{site_id}/bidirectional-status", axum::routing::get(handlers::admin::check_bidirectional_status)) - .route("/admin/iam/users", axum::routing::get(handlers::admin::iam_list_users)) - .route("/admin/iam/users/{identifier}", axum::routing::get(handlers::admin::iam_get_user)) - .route("/admin/iam/users/{identifier}/policies", axum::routing::get(handlers::admin::iam_get_user_policies)) - .route("/admin/iam/users/{identifier}/access-keys", axum::routing::post(handlers::admin::iam_create_access_key)) - .route("/admin/iam/users/{identifier}/access-keys/{access_key}", axum::routing::delete(handlers::admin::iam_delete_access_key)) - .route("/admin/iam/users/{identifier}/disable", axum::routing::post(handlers::admin::iam_disable_user)) - .route("/admin/iam/users/{identifier}/enable", axum::routing::post(handlers::admin::iam_enable_user)) - .route("/admin/website-domains", axum::routing::get(handlers::admin::list_website_domains).post(handlers::admin::create_website_domain)) - .route("/admin/website-domains/{domain}", axum::routing::get(handlers::admin::get_website_domain).put(handlers::admin::update_website_domain).delete(handlers::admin::delete_website_domain)) - .route("/admin/gc/status", axum::routing::get(handlers::admin::gc_status)) - .route("/admin/gc/run", axum::routing::post(handlers::admin::gc_run)) - .route("/admin/gc/history", axum::routing::get(handlers::admin::gc_history)) - .route("/admin/integrity/status", axum::routing::get(handlers::admin::integrity_status)) - .route("/admin/integrity/run", axum::routing::post(handlers::admin::integrity_run)) - .route("/admin/integrity/history", axum::routing::get(handlers::admin::integrity_history)); + .route( + "/admin/site", + axum::routing::get(handlers::admin::get_local_site) + .put(handlers::admin::update_local_site), + ) + .route( + "/admin/sites", + axum::routing::get(handlers::admin::list_all_sites) + .post(handlers::admin::register_peer_site), + ) + .route( + "/admin/sites/{site_id}", + axum::routing::get(handlers::admin::get_peer_site) + .put(handlers::admin::update_peer_site) + .delete(handlers::admin::delete_peer_site), + ) + .route( + "/admin/sites/{site_id}/health", + axum::routing::get(handlers::admin::check_peer_health) + .post(handlers::admin::check_peer_health), + ) + .route( + "/admin/sites/{site_id}/bidirectional-status", + axum::routing::get(handlers::admin::check_bidirectional_status), + ) + .route( + "/admin/topology", + axum::routing::get(handlers::admin::get_topology), + ) + .route( + "/admin/site/local", + axum::routing::get(handlers::admin::get_local_site) + .put(handlers::admin::update_local_site), + ) + .route( + "/admin/site/all", + axum::routing::get(handlers::admin::list_all_sites), + ) + .route( + "/admin/site/peers", + axum::routing::post(handlers::admin::register_peer_site), + ) + .route( + "/admin/site/peers/{site_id}", + axum::routing::get(handlers::admin::get_peer_site) + .put(handlers::admin::update_peer_site) + .delete(handlers::admin::delete_peer_site), + ) + .route( + "/admin/site/peers/{site_id}/health", + axum::routing::post(handlers::admin::check_peer_health), + ) + .route( + "/admin/site/topology", + axum::routing::get(handlers::admin::get_topology), + ) + .route( + "/admin/site/peers/{site_id}/bidirectional-status", + axum::routing::get(handlers::admin::check_bidirectional_status), + ) + .route( + "/admin/iam/users", + axum::routing::get(handlers::admin::iam_list_users), + ) + .route( + "/admin/iam/users/{identifier}", + axum::routing::get(handlers::admin::iam_get_user), + ) + .route( + "/admin/iam/users/{identifier}/policies", + axum::routing::get(handlers::admin::iam_get_user_policies), + ) + .route( + "/admin/iam/users/{identifier}/access-keys", + axum::routing::post(handlers::admin::iam_create_access_key), + ) + .route( + "/admin/iam/users/{identifier}/access-keys/{access_key}", + axum::routing::delete(handlers::admin::iam_delete_access_key), + ) + .route( + "/admin/iam/users/{identifier}/disable", + axum::routing::post(handlers::admin::iam_disable_user), + ) + .route( + "/admin/iam/users/{identifier}/enable", + axum::routing::post(handlers::admin::iam_enable_user), + ) + .route( + "/admin/website-domains", + axum::routing::get(handlers::admin::list_website_domains) + .post(handlers::admin::create_website_domain), + ) + .route( + "/admin/website-domains/{domain}", + axum::routing::get(handlers::admin::get_website_domain) + .put(handlers::admin::update_website_domain) + .delete(handlers::admin::delete_website_domain), + ) + .route( + "/admin/gc/status", + axum::routing::get(handlers::admin::gc_status), + ) + .route( + "/admin/gc/run", + axum::routing::post(handlers::admin::gc_run), + ) + .route( + "/admin/gc/history", + axum::routing::get(handlers::admin::gc_history), + ) + .route( + "/admin/integrity/status", + axum::routing::get(handlers::admin::integrity_status), + ) + .route( + "/admin/integrity/run", + axum::routing::post(handlers::admin::integrity_run), + ) + .route( + "/admin/integrity/history", + axum::routing::get(handlers::admin::integrity_history), + ); - let mut router = router + router .layer(axum::middleware::from_fn_with_state( state.clone(), middleware::auth_layer, )) .layer(axum::middleware::from_fn(middleware::server_header)) - .with_state(state.clone()); - - if state.config.ui_enabled { - let static_service = tower_http::services::ServeDir::new(&state.config.static_dir); - router = router - .nest_service("/static", static_service) - .merge(create_ui_router(state)); - } - - router + .layer(tower_http::compression::CompressionLayer::new()) + .with_state(state) } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/main.rs b/rust/myfsio-engine/crates/myfsio-server/src/main.rs index ece0c93..204511b 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/main.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/main.rs @@ -3,8 +3,18 @@ use myfsio_server::config::ServerConfig; use myfsio_server::state::AppState; #[derive(Parser)] -#[command(name = "myfsio", version, about = "MyFSIO S3-compatible storage engine")] +#[command( + name = "myfsio", + version, + about = "MyFSIO S3-compatible storage engine" +)] struct Cli { + #[arg(long, help = "Validate configuration and exit")] + check_config: bool, + #[arg(long, help = "Show configuration summary and exit")] + show_config: bool, + #[arg(long, help = "Reset admin credentials and exit")] + reset_cred: bool, #[command(subcommand)] command: Option, } @@ -17,9 +27,30 @@ enum Command { #[tokio::main] async fn main() { + load_env_files(); tracing_subscriber::fmt::init(); let cli = Cli::parse(); + let config = ServerConfig::from_env(); + + if cli.reset_cred { + reset_admin_credentials(&config); + return; + } + if cli.check_config || cli.show_config { + print_config_summary(&config); + if cli.check_config { + let issues = validate_config(&config); + for issue in &issues { + println!("{issue}"); + } + if issues.iter().any(|issue| issue.starts_with("CRITICAL:")) { + std::process::exit(1); + } + } + return; + } + match cli.command.unwrap_or(Command::Serve) { Command::Version => { println!("myfsio {}", env!("CARGO_PKG_VERSION")); @@ -28,19 +59,24 @@ async fn main() { Command::Serve => {} } - let config = ServerConfig::from_env(); + ensure_iam_bootstrap(&config); let bind_addr = config.bind_addr; + let ui_bind_addr = config.ui_bind_addr; - tracing::info!("MyFSIO Rust Engine starting on {}", bind_addr); + tracing::info!("MyFSIO Rust Engine starting — API on {}", bind_addr); + if config.ui_enabled { + tracing::info!("UI will bind on {}", ui_bind_addr); + } tracing::info!("Storage root: {}", config.storage_root.display()); tracing::info!("Region: {}", config.region); tracing::info!( - "Encryption: {}, KMS: {}, GC: {}, Lifecycle: {}, Integrity: {}, Metrics: {}, UI: {}", + "Encryption: {}, KMS: {}, GC: {}, Lifecycle: {}, Integrity: {}, Metrics History: {}, Operation Metrics: {}, UI: {}", config.encryption_enabled, config.kms_enabled, config.gc_enabled, config.lifecycle_enabled, config.integrity_enabled, + config.metrics_history_enabled, config.metrics_enabled, config.ui_enabled ); @@ -68,13 +104,17 @@ async fn main() { tracing::info!("Metrics collector background service started"); } + if let Some(ref system_metrics) = state.system_metrics { + bg_handles.push(system_metrics.clone().start_background()); + tracing::info!("System metrics history collector started"); + } + if config.lifecycle_enabled { - let lifecycle = std::sync::Arc::new( - myfsio_server::services::lifecycle::LifecycleService::new( + let lifecycle = + std::sync::Arc::new(myfsio_server::services::lifecycle::LifecycleService::new( state.storage.clone(), myfsio_server::services::lifecycle::LifecycleConfig::default(), - ), - ); + )); bg_handles.push(lifecycle.start_background()); tracing::info!("Lifecycle manager background service started"); } @@ -87,15 +127,21 @@ async fn main() { tracing::info!("Site sync worker started"); } - let app = myfsio_server::create_router(state); + let ui_enabled = config.ui_enabled; + let api_app = myfsio_server::create_router(state.clone()); + let ui_app = if ui_enabled { + Some(myfsio_server::create_ui_router(state.clone())) + } else { + None + }; - let listener = match tokio::net::TcpListener::bind(bind_addr).await { + let api_listener = match tokio::net::TcpListener::bind(bind_addr).await { Ok(listener) => listener, Err(err) => { if err.kind() == std::io::ErrorKind::AddrInUse { - tracing::error!("Port already in use: {}", bind_addr); + tracing::error!("API port already in use: {}", bind_addr); } else { - tracing::error!("Failed to bind {}: {}", bind_addr, err); + tracing::error!("Failed to bind API {}: {}", bind_addr, err); } for handle in bg_handles { handle.abort(); @@ -103,17 +149,67 @@ async fn main() { std::process::exit(1); } }; - tracing::info!("Listening on {}", bind_addr); + tracing::info!("API listening on {}", bind_addr); - if let Err(err) = axum::serve(listener, app) - .with_graceful_shutdown(shutdown_signal()) - .await - { - tracing::error!("Server exited with error: {}", err); - for handle in bg_handles { - handle.abort(); + let ui_listener = if let Some(ref app) = ui_app { + let _ = app; + match tokio::net::TcpListener::bind(ui_bind_addr).await { + Ok(listener) => { + tracing::info!("UI listening on {}", ui_bind_addr); + Some(listener) + } + Err(err) => { + if err.kind() == std::io::ErrorKind::AddrInUse { + tracing::error!("UI port already in use: {}", ui_bind_addr); + } else { + tracing::error!("Failed to bind UI {}: {}", ui_bind_addr, err); + } + for handle in bg_handles { + handle.abort(); + } + std::process::exit(1); + } + } + } else { + None + }; + + let shutdown = shutdown_signal_shared(); + let api_shutdown = shutdown.clone(); + let api_task = tokio::spawn(async move { + axum::serve(api_listener, api_app) + .with_graceful_shutdown(async move { + api_shutdown.notified().await; + }) + .await + }); + + let ui_task = if let (Some(listener), Some(app)) = (ui_listener, ui_app) { + let ui_shutdown = shutdown.clone(); + Some(tokio::spawn(async move { + axum::serve(listener, app) + .with_graceful_shutdown(async move { + ui_shutdown.notified().await; + }) + .await + })) + } else { + None + }; + + tokio::signal::ctrl_c() + .await + .expect("Failed to listen for Ctrl+C"); + tracing::info!("Shutdown signal received"); + shutdown.notify_waiters(); + + if let Err(err) = api_task.await.unwrap_or(Ok(())) { + tracing::error!("API server exited with error: {}", err); + } + if let Some(task) = ui_task { + if let Err(err) = task.await.unwrap_or(Ok(())) { + tracing::error!("UI server exited with error: {}", err); } - std::process::exit(1); } for handle in bg_handles { @@ -121,9 +217,209 @@ async fn main() { } } -async fn shutdown_signal() { - tokio::signal::ctrl_c() - .await - .expect("Failed to listen for Ctrl+C"); - tracing::info!("Shutdown signal received"); +fn print_config_summary(config: &ServerConfig) { + println!("MyFSIO Rust Configuration"); + println!("Version: {}", env!("CARGO_PKG_VERSION")); + println!("API bind: {}", config.bind_addr); + println!("UI bind: {}", config.ui_bind_addr); + println!("UI enabled: {}", config.ui_enabled); + println!("Storage root: {}", config.storage_root.display()); + println!("IAM config: {}", config.iam_config_path.display()); + println!("Region: {}", config.region); + println!("Encryption enabled: {}", config.encryption_enabled); + println!("KMS enabled: {}", config.kms_enabled); + println!("GC enabled: {}", config.gc_enabled); + println!("Integrity enabled: {}", config.integrity_enabled); + println!("Lifecycle enabled: {}", config.lifecycle_enabled); + println!( + "Website hosting enabled: {}", + config.website_hosting_enabled + ); + println!("Site sync enabled: {}", config.site_sync_enabled); + println!( + "Metrics history enabled: {}", + config.metrics_history_enabled + ); + println!("Operation metrics enabled: {}", config.metrics_enabled); +} + +fn validate_config(config: &ServerConfig) -> Vec { + let mut issues = Vec::new(); + + if config.ui_enabled && config.bind_addr == config.ui_bind_addr { + issues.push( + "CRITICAL: API and UI bind addresses cannot be identical when UI is enabled." + .to_string(), + ); + } + if config.presigned_url_min_expiry > config.presigned_url_max_expiry { + issues.push("CRITICAL: PRESIGNED_URL_MIN_EXPIRY_SECONDS cannot exceed PRESIGNED_URL_MAX_EXPIRY_SECONDS.".to_string()); + } + if let Err(err) = std::fs::create_dir_all(&config.storage_root) { + issues.push(format!( + "CRITICAL: Cannot create storage root {}: {}", + config.storage_root.display(), + err + )); + } + if let Some(parent) = config.iam_config_path.parent() { + if let Err(err) = std::fs::create_dir_all(parent) { + issues.push(format!( + "CRITICAL: Cannot create IAM config directory {}: {}", + parent.display(), + err + )); + } + } + if config.encryption_enabled && config.secret_key.is_none() { + issues.push( + "WARNING: ENCRYPTION_ENABLED=true but SECRET_KEY is not configured; secure-at-rest config encryption is unavailable.".to_string(), + ); + } + if config.site_sync_enabled && !config.website_hosting_enabled { + issues.push( + "INFO: SITE_SYNC_ENABLED=true without WEBSITE_HOSTING_ENABLED; this is valid but unrelated.".to_string(), + ); + } + + issues +} + +fn shutdown_signal_shared() -> std::sync::Arc { + std::sync::Arc::new(tokio::sync::Notify::new()) +} + +fn load_env_files() { + let cwd = std::env::current_dir().ok(); + let mut candidates: Vec = Vec::new(); + candidates.push(std::path::PathBuf::from("/opt/myfsio/myfsio.env")); + if let Some(ref dir) = cwd { + candidates.push(dir.join(".env")); + candidates.push(dir.join("myfsio.env")); + for ancestor in dir.ancestors().skip(1).take(4) { + candidates.push(ancestor.join(".env")); + candidates.push(ancestor.join("myfsio.env")); + } + } + + let mut seen = std::collections::HashSet::new(); + for path in candidates { + if !seen.insert(path.clone()) { + continue; + } + if path.is_file() { + match dotenvy::from_path_override(&path) { + Ok(()) => eprintln!("Loaded env file: {}", path.display()), + Err(e) => eprintln!("Failed to load env file {}: {}", path.display(), e), + } + } + } +} + +fn ensure_iam_bootstrap(config: &ServerConfig) { + let iam_path = &config.iam_config_path; + if iam_path.exists() { + return; + } + + let access_key = std::env::var("ADMIN_ACCESS_KEY") + .ok() + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| format!("AK{}", uuid::Uuid::new_v4().simple())); + let secret_key = std::env::var("ADMIN_SECRET_KEY") + .ok() + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| format!("SK{}", uuid::Uuid::new_v4().simple())); + + let user_id = format!("u-{}", &uuid::Uuid::new_v4().simple().to_string()[..16]); + let created_at = chrono::Utc::now().to_rfc3339(); + + let body = serde_json::json!({ + "version": 2, + "users": [{ + "user_id": user_id, + "display_name": "Local Admin", + "enabled": true, + "access_keys": [{ + "access_key": access_key, + "secret_key": secret_key, + "status": "active", + "created_at": created_at, + }], + "policies": [{ + "bucket": "*", + "actions": ["*"], + "prefix": "*", + }] + }] + }); + + let json = match serde_json::to_string_pretty(&body) { + Ok(s) => s, + Err(e) => { + tracing::error!("Failed to serialize IAM bootstrap config: {}", e); + return; + } + }; + + if let Some(parent) = iam_path.parent() { + if let Err(e) = std::fs::create_dir_all(parent) { + tracing::error!( + "Failed to create IAM config dir {}: {}", + parent.display(), + e + ); + return; + } + } + + if let Err(e) = std::fs::write(iam_path, json) { + tracing::error!( + "Failed to write IAM bootstrap config {}: {}", + iam_path.display(), + e + ); + return; + } + + tracing::info!("============================================================"); + tracing::info!("MYFSIO - ADMIN CREDENTIALS INITIALIZED"); + tracing::info!("============================================================"); + tracing::info!("Access Key: {}", access_key); + tracing::info!("Secret Key: {}", secret_key); + tracing::info!("Saved to: {}", iam_path.display()); + tracing::info!("============================================================"); +} + +fn reset_admin_credentials(config: &ServerConfig) { + if let Some(parent) = config.iam_config_path.parent() { + if let Err(err) = std::fs::create_dir_all(parent) { + eprintln!( + "Failed to create IAM config directory {}: {}", + parent.display(), + err + ); + std::process::exit(1); + } + } + + if config.iam_config_path.exists() { + let backup = config + .iam_config_path + .with_extension(format!("bak-{}", chrono::Utc::now().timestamp())); + if let Err(err) = std::fs::rename(&config.iam_config_path, &backup) { + eprintln!( + "Failed to back up existing IAM config {}: {}", + config.iam_config_path.display(), + err + ); + std::process::exit(1); + } + println!("Backed up existing IAM config to {}", backup.display()); + } + + ensure_iam_bootstrap(config); + println!("Admin credentials reset."); } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/middleware/auth.rs b/rust/myfsio-engine/crates/myfsio-server/src/middleware/auth.rs index 5dfa0d5..db2283d 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/middleware/auth.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/middleware/auth.rs @@ -1,5 +1,5 @@ use axum::extract::{Request, State}; -use axum::http::{Method, StatusCode}; +use axum::http::{header, HeaderMap, Method, StatusCode}; use axum::middleware::Next; use axum::response::{IntoResponse, Response}; @@ -7,51 +7,470 @@ use chrono::{NaiveDateTime, Utc}; use myfsio_auth::sigv4; use myfsio_common::error::{S3Error, S3ErrorCode}; use myfsio_common::types::Principal; +use myfsio_storage::traits::StorageEngine; +use serde_json::Value; +use std::time::Instant; +use tokio::io::AsyncReadExt; use crate::state::AppState; -pub async fn auth_layer( - State(state): State, - mut req: Request, - next: Next, +fn website_error_response( + status: StatusCode, + body: Option>, + content_type: &str, ) -> Response { - let uri = req.uri().clone(); - let path = uri.path().to_string(); + let mut headers = HeaderMap::new(); + headers.insert(header::CONTENT_TYPE, content_type.parse().unwrap()); + headers.insert(header::ACCEPT_RANGES, "bytes".parse().unwrap()); + if let Some(ref body) = body { + headers.insert( + header::CONTENT_LENGTH, + body.len().to_string().parse().unwrap(), + ); + (status, headers, body.clone()).into_response() + } else { + (status, headers).into_response() + } +} - if path == "/" && req.method() == axum::http::Method::GET { - match try_auth(&state, &req) { - AuthResult::Ok(principal) => { - if let Err(err) = authorize_request(&state, &principal, &req) { - return error_response(err, &path); - } - req.extensions_mut().insert(principal); - } - AuthResult::Denied(err) => return error_response(err, &path), - AuthResult::NoAuth => { - return error_response( - S3Error::new(S3ErrorCode::AccessDenied, "Missing credentials"), - &path, - ); - } +fn parse_range_header(range_header: &str, total_size: u64) -> Option<(u64, u64)> { + let range_spec = range_header.strip_prefix("bytes=")?; + if let Some(suffix) = range_spec.strip_prefix('-') { + let suffix_len: u64 = suffix.parse().ok()?; + if suffix_len == 0 || suffix_len > total_size { + return None; } - return next.run(req).await; + return Some((total_size - suffix_len, total_size - 1)); } - match try_auth(&state, &req) { - AuthResult::Ok(principal) => { - if let Err(err) = authorize_request(&state, &principal, &req) { - return error_response(err, &path); + let (start_str, end_str) = range_spec.split_once('-')?; + let start: u64 = start_str.parse().ok()?; + let end = if end_str.is_empty() { + total_size.saturating_sub(1) + } else { + end_str + .parse::() + .ok()? + .min(total_size.saturating_sub(1)) + }; + + if start > end || start >= total_size { + return None; + } + Some((start, end)) +} + +fn website_content_type(key: &str, metadata: &std::collections::HashMap) -> String { + metadata + .get("__content_type__") + .filter(|value| !value.trim().is_empty()) + .cloned() + .unwrap_or_else(|| { + mime_guess::from_path(key) + .first_raw() + .unwrap_or("application/octet-stream") + .to_string() + }) +} + +fn parse_website_config(value: &Value) -> Option<(String, Option)> { + match value { + Value::Object(map) => { + let index_document = map + .get("index_document") + .or_else(|| map.get("IndexDocument")) + .and_then(|v| v.as_str()) + .unwrap_or("index.html") + .to_string(); + let error_document = map + .get("error_document") + .or_else(|| map.get("ErrorDocument")) + .and_then(|v| v.as_str()) + .map(|v| v.to_string()); + Some((index_document, error_document)) + } + Value::String(raw) => { + if let Ok(json) = serde_json::from_str::(raw) { + return parse_website_config(&json); } - req.extensions_mut().insert(principal); - next.run(req).await + let doc = roxmltree::Document::parse(raw).ok()?; + let index_document = doc + .descendants() + .find(|node| node.is_element() && node.tag_name().name() == "Suffix") + .and_then(|node| node.text()) + .map(|text| text.trim().to_string()) + .filter(|text| !text.is_empty()) + .unwrap_or_else(|| "index.html".to_string()); + let error_document = doc + .descendants() + .find(|node| node.is_element() && node.tag_name().name() == "Key") + .and_then(|node| node.text()) + .map(|text| text.trim().to_string()) + .filter(|text| !text.is_empty()); + Some((index_document, error_document)) } - AuthResult::Denied(err) => error_response(err, &path), - AuthResult::NoAuth => { - error_response( - S3Error::new(S3ErrorCode::AccessDenied, "Missing credentials"), - &path, + _ => None, + } +} + +async fn serve_website_document( + state: &AppState, + bucket: &str, + key: &str, + method: &axum::http::Method, + range_header: Option<&str>, + status: StatusCode, +) -> Option { + let metadata = state.storage.get_object_metadata(bucket, key).await.ok()?; + let (meta, mut reader) = state.storage.get_object(bucket, key).await.ok()?; + let content_type = website_content_type(key, &metadata); + + if method == axum::http::Method::HEAD { + let mut headers = HeaderMap::new(); + headers.insert(header::CONTENT_TYPE, content_type.parse().unwrap()); + headers.insert( + header::CONTENT_LENGTH, + meta.size.to_string().parse().unwrap(), + ); + headers.insert(header::ACCEPT_RANGES, "bytes".parse().unwrap()); + return Some((status, headers).into_response()); + } + + let mut bytes = Vec::new(); + if reader.read_to_end(&mut bytes).await.is_err() { + return None; + } + + let mut headers = HeaderMap::new(); + headers.insert(header::CONTENT_TYPE, content_type.parse().unwrap()); + headers.insert(header::ACCEPT_RANGES, "bytes".parse().unwrap()); + + if status == StatusCode::OK { + if let Some(range_header) = range_header { + let Some((start, end)) = parse_range_header(range_header, bytes.len() as u64) else { + let mut range_headers = HeaderMap::new(); + range_headers.insert( + header::CONTENT_RANGE, + format!("bytes */{}", bytes.len()).parse().unwrap(), + ); + return Some((StatusCode::RANGE_NOT_SATISFIABLE, range_headers).into_response()); + }; + let body = bytes[start as usize..=end as usize].to_vec(); + headers.insert( + header::CONTENT_RANGE, + format!("bytes {}-{}/{}", start, end, bytes.len()) + .parse() + .unwrap(), + ); + headers.insert( + header::CONTENT_LENGTH, + body.len().to_string().parse().unwrap(), + ); + return Some((StatusCode::PARTIAL_CONTENT, headers, body).into_response()); + } + } + + headers.insert( + header::CONTENT_LENGTH, + bytes.len().to_string().parse().unwrap(), + ); + Some((status, headers, bytes).into_response()) +} + +async fn maybe_serve_website( + state: &AppState, + method: Method, + host: String, + uri_path: String, + range_header: Option, +) -> Option { + if !state.config.website_hosting_enabled { + return None; + } + if method != axum::http::Method::GET && method != axum::http::Method::HEAD { + return None; + } + let request_path = uri_path.trim_start_matches('/').to_string(); + let store = state.website_domains.as_ref()?; + let bucket = store.get_bucket(&host)?; + if !matches!(state.storage.bucket_exists(&bucket).await, Ok(true)) { + return Some(website_error_response( + StatusCode::NOT_FOUND, + None, + "text/plain; charset=utf-8", + )); + } + + let bucket_config = state.storage.get_bucket_config(&bucket).await.ok()?; + let Some(website_config) = bucket_config.website.as_ref() else { + return Some(website_error_response( + StatusCode::NOT_FOUND, + None, + "text/plain; charset=utf-8", + )); + }; + let Some((index_document, error_document)) = parse_website_config(website_config) else { + return Some(website_error_response( + StatusCode::NOT_FOUND, + None, + "text/plain; charset=utf-8", + )); + }; + + let mut object_key = if request_path.is_empty() || uri_path.ends_with('/') { + if request_path.is_empty() { + index_document.clone() + } else { + format!("{}{}", request_path, index_document) + } + } else { + request_path.clone() + }; + + let exists = state + .storage + .head_object(&bucket, &object_key) + .await + .is_ok(); + if !exists && !request_path.is_empty() && !request_path.ends_with('/') { + let alternate = format!("{}/{}", request_path, index_document); + if state.storage.head_object(&bucket, &alternate).await.is_ok() { + object_key = alternate; + } else if let Some(error_key) = error_document.as_deref() { + return serve_website_document( + state, + &bucket, + error_key, + &method, + range_header.as_deref(), + StatusCode::NOT_FOUND, ) + .await + .or_else(|| { + Some(website_error_response( + StatusCode::NOT_FOUND, + None, + "text/plain; charset=utf-8", + )) + }); + } else { + return Some(website_error_response( + StatusCode::NOT_FOUND, + None, + "text/plain; charset=utf-8", + )); } + } else if !exists { + if let Some(error_key) = error_document.as_deref() { + return serve_website_document( + state, + &bucket, + error_key, + &method, + range_header.as_deref(), + StatusCode::NOT_FOUND, + ) + .await + .or_else(|| { + Some(website_error_response( + StatusCode::NOT_FOUND, + None, + "text/plain; charset=utf-8", + )) + }); + } + return Some(website_error_response( + StatusCode::NOT_FOUND, + None, + "text/plain; charset=utf-8", + )); + } + + serve_website_document( + state, + &bucket, + &object_key, + &method, + range_header.as_deref(), + StatusCode::OK, + ) + .await +} + +pub async fn auth_layer(State(state): State, mut req: Request, next: Next) -> Response { + let start = Instant::now(); + let uri = req.uri().clone(); + let path = uri.path().to_string(); + let method = req.method().clone(); + let query = uri.query().unwrap_or("").to_string(); + let copy_source = req + .headers() + .get("x-amz-copy-source") + .and_then(|v| v.to_str().ok()) + .map(|value| value.to_string()); + let endpoint_type = classify_endpoint(&path, &query); + let bytes_in = req + .headers() + .get(axum::http::header::CONTENT_LENGTH) + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.parse::().ok()) + .unwrap_or(0); + + let host = req + .headers() + .get(header::HOST) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.split(':').next()) + .map(|value| value.trim().to_ascii_lowercase()); + let range_header = req + .headers() + .get(header::RANGE) + .and_then(|value| value.to_str().ok()) + .map(|value| value.to_string()); + + let response = if path == "/myfsio/health" || path == "/health" { + next.run(req).await + } else if let Some(response) = maybe_serve_website( + &state, + method.clone(), + host.unwrap_or_default(), + path.clone(), + range_header, + ) + .await + { + response + } else { + match try_auth(&state, &req) { + AuthResult::NoAuth => match authorize_request( + &state, + None, + &method, + &path, + &query, + copy_source.as_deref(), + ) + .await + { + Ok(()) => next.run(req).await, + Err(err) => error_response(err, &path), + }, + AuthResult::Ok(principal) => { + if let Err(err) = authorize_request( + &state, + Some(&principal), + &method, + &path, + &query, + copy_source.as_deref(), + ) + .await + { + error_response(err, &path) + } else { + req.extensions_mut().insert(principal); + next.run(req).await + } + } + AuthResult::Denied(err) => error_response(err, &path), + } + }; + + if let Some(metrics) = &state.metrics { + let latency_ms = start.elapsed().as_secs_f64() * 1000.0; + let status = response.status().as_u16(); + let bytes_out = response + .headers() + .get(axum::http::header::CONTENT_LENGTH) + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.parse::().ok()) + .unwrap_or(0); + let error_code = if status >= 400 { + Some(s3_code_for_status(status)) + } else { + None + }; + metrics.record_request( + method.as_str(), + endpoint_type, + status, + latency_ms, + bytes_in, + bytes_out, + error_code, + ); + } + + response +} + +fn classify_endpoint(path: &str, query: &str) -> &'static str { + if path == "/" { + return "list_buckets"; + } + let segments: Vec<&str> = path + .trim_start_matches('/') + .split('/') + .filter(|s| !s.is_empty()) + .collect(); + if segments.is_empty() { + return "other"; + } + if segments.len() == 1 { + if query.contains("uploads") { + return "list_multipart_uploads"; + } + if query.contains("versioning") { + return "bucket_versioning"; + } + if query.contains("lifecycle") { + return "bucket_lifecycle"; + } + if query.contains("policy") { + return "bucket_policy"; + } + if query.contains("website") { + return "bucket_website"; + } + if query.contains("encryption") { + return "bucket_encryption"; + } + if query.contains("replication") { + return "bucket_replication"; + } + return "bucket"; + } + if query.contains("uploadId") { + return "multipart_part"; + } + if query.contains("uploads") { + return "multipart_init"; + } + if query.contains("tagging") { + return "object_tagging"; + } + if query.contains("acl") { + return "object_acl"; + } + "object" +} + +fn s3_code_for_status(status: u16) -> &'static str { + match status { + 400 => "BadRequest", + 401 => "Unauthorized", + 403 => "AccessDenied", + 404 => "NotFound", + 405 => "MethodNotAllowed", + 409 => "Conflict", + 411 => "MissingContentLength", + 412 => "PreconditionFailed", + 413 => "EntityTooLarge", + 416 => "InvalidRange", + 500 => "InternalError", + 501 => "NotImplemented", + 503 => "ServiceUnavailable", + _ => "Other", } } @@ -61,20 +480,45 @@ enum AuthResult { NoAuth, } -fn authorize_request(state: &AppState, principal: &Principal, req: &Request) -> Result<(), S3Error> { - let path = req.uri().path(); +async fn authorize_request( + state: &AppState, + principal: Option<&Principal>, + method: &Method, + path: &str, + query: &str, + copy_source: Option<&str>, +) -> Result<(), S3Error> { + if path == "/myfsio/health" || path == "/health" { + return Ok(()); + } if path == "/" { - if state.iam.authorize(principal, None, "list", None) { - return Ok(()); + if let Some(principal) = principal { + if state.iam.authorize(principal, None, "list", None) { + return Ok(()); + } + return Err(S3Error::new(S3ErrorCode::AccessDenied, "Access denied")); } - return Err(S3Error::new(S3ErrorCode::AccessDenied, "Access denied")); + return Err(S3Error::new( + S3ErrorCode::AccessDenied, + "Missing credentials", + )); } if path.starts_with("/admin/") || path.starts_with("/kms/") { - return Ok(()); + return if principal.is_some() { + Ok(()) + } else { + Err(S3Error::new( + S3ErrorCode::AccessDenied, + "Missing credentials", + )) + }; } - let mut segments = path.trim_start_matches('/').split('/').filter(|s| !s.is_empty()); + let mut segments = path + .trim_start_matches('/') + .split('/') + .filter(|s| !s.is_empty()); let bucket = match segments.next() { Some(b) => b, None => { @@ -82,29 +526,25 @@ fn authorize_request(state: &AppState, principal: &Principal, req: &Request) -> } }; let remaining: Vec<&str> = segments.collect(); - let query = req.uri().query().unwrap_or(""); if remaining.is_empty() { - let action = resolve_bucket_action(req.method(), query); - if state.iam.authorize(principal, Some(bucket), action, None) { - return Ok(()); - } - return Err(S3Error::new(S3ErrorCode::AccessDenied, "Access denied")); + let action = resolve_bucket_action(method, query); + return authorize_action(state, principal, bucket, action, None).await; } let object_key = remaining.join("/"); - if req.method() == Method::PUT { - if let Some(copy_source) = req - .headers() - .get("x-amz-copy-source") - .and_then(|v| v.to_str().ok()) - { + if *method == Method::PUT { + if let Some(copy_source) = copy_source { let source = copy_source.strip_prefix('/').unwrap_or(copy_source); if let Some((src_bucket, src_key)) = source.split_once('/') { let source_allowed = - state.iam.authorize(principal, Some(src_bucket), "read", Some(src_key)); + authorize_action(state, principal, src_bucket, "read", Some(src_key)) + .await + .is_ok(); let dest_allowed = - state.iam.authorize(principal, Some(bucket), "write", Some(&object_key)); + authorize_action(state, principal, bucket, "write", Some(&object_key)) + .await + .is_ok(); if source_allowed && dest_allowed { return Ok(()); } @@ -113,15 +553,270 @@ fn authorize_request(state: &AppState, principal: &Principal, req: &Request) -> } } - let action = resolve_object_action(req.method(), query); - if state - .iam - .authorize(principal, Some(bucket), action, Some(&object_key)) - { + let action = resolve_object_action(method, query); + authorize_action(state, principal, bucket, action, Some(&object_key)).await +} + +async fn authorize_action( + state: &AppState, + principal: Option<&Principal>, + bucket: &str, + action: &str, + object_key: Option<&str>, +) -> Result<(), S3Error> { + let iam_allowed = principal + .map(|principal| { + state + .iam + .authorize(principal, Some(bucket), action, object_key) + }) + .unwrap_or(false); + let policy_decision = evaluate_bucket_policy( + state, + principal.map(|principal| principal.access_key.as_str()), + bucket, + action, + object_key, + ) + .await; + + if matches!(policy_decision, PolicyDecision::Deny) { + return Err(S3Error::new( + S3ErrorCode::AccessDenied, + "Access denied by bucket policy", + )); + } + if iam_allowed || matches!(policy_decision, PolicyDecision::Allow) { return Ok(()); } - Err(S3Error::new(S3ErrorCode::AccessDenied, "Access denied")) + if principal.is_some() { + Err(S3Error::new(S3ErrorCode::AccessDenied, "Access denied")) + } else { + Err(S3Error::new( + S3ErrorCode::AccessDenied, + "Missing credentials", + )) + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum PolicyDecision { + Allow, + Deny, + Neutral, +} + +async fn evaluate_bucket_policy( + state: &AppState, + access_key: Option<&str>, + bucket: &str, + action: &str, + object_key: Option<&str>, +) -> PolicyDecision { + let config = match state.storage.get_bucket_config(bucket).await { + Ok(config) => config, + Err(_) => return PolicyDecision::Neutral, + }; + let policy: &Value = match config.policy.as_ref() { + Some(policy) => policy, + None => return PolicyDecision::Neutral, + }; + let mut decision = PolicyDecision::Neutral; + + match policy.get("Statement") { + Some(Value::Array(items)) => { + for statement in items.iter() { + match evaluate_policy_statement(statement, access_key, bucket, action, object_key) { + PolicyDecision::Deny => return PolicyDecision::Deny, + PolicyDecision::Allow => decision = PolicyDecision::Allow, + PolicyDecision::Neutral => {} + } + } + } + Some(statement) => { + return evaluate_policy_statement(statement, access_key, bucket, action, object_key); + } + None => return PolicyDecision::Neutral, + } + + decision +} + +fn evaluate_policy_statement( + statement: &Value, + access_key: Option<&str>, + bucket: &str, + action: &str, + object_key: Option<&str>, +) -> PolicyDecision { + if !statement_matches_principal(statement, access_key) + || !statement_matches_action(statement, action) + || !statement_matches_resource(statement, bucket, object_key) + { + return PolicyDecision::Neutral; + } + + match statement + .get("Effect") + .and_then(|value| value.as_str()) + .map(|value| value.to_ascii_lowercase()) + .as_deref() + { + Some("deny") => PolicyDecision::Deny, + Some("allow") => PolicyDecision::Allow, + _ => PolicyDecision::Neutral, + } +} + +fn statement_matches_principal(statement: &Value, access_key: Option<&str>) -> bool { + match statement.get("Principal") { + Some(principal) => principal_value_matches(principal, access_key), + None => false, + } +} + +fn principal_value_matches(value: &Value, access_key: Option<&str>) -> bool { + match value { + Value::String(token) => token == "*" || access_key == Some(token.as_str()), + Value::Array(items) => items + .iter() + .any(|item| principal_value_matches(item, access_key)), + Value::Object(map) => map + .values() + .any(|item| principal_value_matches(item, access_key)), + _ => false, + } +} + +fn statement_matches_action(statement: &Value, action: &str) -> bool { + match statement.get("Action") { + Some(Value::String(value)) => policy_action_matches(value, action), + Some(Value::Array(items)) => items.iter().any(|item| { + item.as_str() + .map(|value| policy_action_matches(value, action)) + .unwrap_or(false) + }), + _ => false, + } +} + +fn policy_action_matches(policy_action: &str, requested_action: &str) -> bool { + let normalized_policy_action = normalize_policy_action(policy_action); + normalized_policy_action == "*" || normalized_policy_action == requested_action +} + +fn normalize_policy_action(action: &str) -> String { + let normalized = action.trim().to_ascii_lowercase(); + if normalized == "*" { + return normalized; + } + match normalized.as_str() { + "s3:listbucket" + | "s3:listallmybuckets" + | "s3:listbucketversions" + | "s3:listmultipartuploads" + | "s3:listparts" => "list".to_string(), + "s3:getobject" + | "s3:getobjectversion" + | "s3:getobjecttagging" + | "s3:getobjectversiontagging" + | "s3:getobjectacl" + | "s3:getbucketversioning" + | "s3:headobject" + | "s3:headbucket" => "read".to_string(), + "s3:putobject" + | "s3:createbucket" + | "s3:putobjecttagging" + | "s3:putbucketversioning" + | "s3:createmultipartupload" + | "s3:uploadpart" + | "s3:completemultipartupload" + | "s3:abortmultipartupload" + | "s3:copyobject" => "write".to_string(), + "s3:deleteobject" + | "s3:deleteobjectversion" + | "s3:deletebucket" + | "s3:deleteobjecttagging" => "delete".to_string(), + "s3:putobjectacl" | "s3:putbucketacl" | "s3:getbucketacl" => "share".to_string(), + "s3:putbucketpolicy" | "s3:getbucketpolicy" | "s3:deletebucketpolicy" => { + "policy".to_string() + } + "s3:getreplicationconfiguration" + | "s3:putreplicationconfiguration" + | "s3:deletereplicationconfiguration" + | "s3:replicateobject" + | "s3:replicatetags" + | "s3:replicatedelete" => "replication".to_string(), + "s3:getlifecycleconfiguration" + | "s3:putlifecycleconfiguration" + | "s3:deletelifecycleconfiguration" + | "s3:getbucketlifecycle" + | "s3:putbucketlifecycle" => "lifecycle".to_string(), + "s3:getbucketcors" | "s3:putbucketcors" | "s3:deletebucketcors" => "cors".to_string(), + other => other.to_string(), + } +} + +fn statement_matches_resource(statement: &Value, bucket: &str, object_key: Option<&str>) -> bool { + match statement.get("Resource") { + Some(Value::String(resource)) => resource_matches(resource, bucket, object_key), + Some(Value::Array(items)) => items.iter().any(|item| { + item.as_str() + .map(|resource| resource_matches(resource, bucket, object_key)) + .unwrap_or(false) + }), + _ => false, + } +} + +fn resource_matches(resource: &str, bucket: &str, object_key: Option<&str>) -> bool { + let remainder = match resource.strip_prefix("arn:aws:s3:::") { + Some(value) => value, + None => return false, + }; + + match remainder.split_once('/') { + Some((resource_bucket, resource_key)) => object_key + .map(|key| wildcard_match(bucket, resource_bucket) && wildcard_match(key, resource_key)) + .unwrap_or(false), + None => object_key.is_none() && wildcard_match(bucket, remainder), + } +} + +fn wildcard_match(value: &str, pattern: &str) -> bool { + let value = value.as_bytes(); + let pattern = pattern.as_bytes(); + let mut value_idx = 0usize; + let mut pattern_idx = 0usize; + let mut star_idx: Option = None; + let mut match_idx = 0usize; + + while value_idx < value.len() { + if pattern_idx < pattern.len() + && (pattern[pattern_idx] == b'?' + || pattern[pattern_idx].eq_ignore_ascii_case(&value[value_idx])) + { + value_idx += 1; + pattern_idx += 1; + } else if pattern_idx < pattern.len() && pattern[pattern_idx] == b'*' { + star_idx = Some(pattern_idx); + pattern_idx += 1; + match_idx = value_idx; + } else if let Some(star) = star_idx { + pattern_idx = star + 1; + match_idx += 1; + value_idx = match_idx; + } else { + return false; + } + } + + while pattern_idx < pattern.len() && pattern[pattern_idx] == b'*' { + pattern_idx += 1; + } + + pattern_idx == pattern.len() } fn resolve_bucket_action(method: &Method, query: &str) -> &'static str { @@ -186,10 +881,18 @@ fn resolve_bucket_action(method: &Method, query: &str) -> &'static str { fn resolve_object_action(method: &Method, query: &str) -> &'static str { if has_query_key(query, "tagging") { - return if *method == Method::GET { "read" } else { "write" }; + return if *method == Method::GET { + "read" + } else { + "write" + }; } if has_query_key(query, "acl") { - return if *method == Method::GET { "read" } else { "write" }; + return if *method == Method::GET { + "read" + } else { + "write" + }; } if has_query_key(query, "retention") || has_query_key(query, "legal-hold") { return "object_lock"; @@ -241,14 +944,16 @@ fn try_auth(state: &AppState, req: &Request) -> AuthResult { } if let (Some(ak), Some(sk)) = ( - req.headers().get("x-access-key").and_then(|v| v.to_str().ok()), - req.headers().get("x-secret-key").and_then(|v| v.to_str().ok()), + req.headers() + .get("x-access-key") + .and_then(|v| v.to_str().ok()), + req.headers() + .get("x-secret-key") + .and_then(|v| v.to_str().ok()), ) { return match state.iam.authenticate(ak, sk) { Some(principal) => AuthResult::Ok(principal), - None => AuthResult::Denied( - S3Error::from_code(S3ErrorCode::SignatureDoesNotMatch), - ), + None => AuthResult::Denied(S3Error::from_code(S3ErrorCode::SignatureDoesNotMatch)), }; } @@ -263,9 +968,10 @@ fn verify_sigv4_header(state: &AppState, req: &Request, auth_str: &str) -> AuthR .collect(); if parts.len() != 3 { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Malformed Authorization header"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Malformed Authorization header", + )); } let credential = parts[0].strip_prefix("Credential=").unwrap_or(""); @@ -274,9 +980,10 @@ fn verify_sigv4_header(state: &AppState, req: &Request, auth_str: &str) -> AuthR let cred_parts: Vec<&str> = credential.split('/').collect(); if cred_parts.len() != 5 { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Malformed credential"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Malformed credential", + )); } let access_key = cred_parts[0]; @@ -292,21 +999,22 @@ fn verify_sigv4_header(state: &AppState, req: &Request, auth_str: &str) -> AuthR .unwrap_or(""); if amz_date.is_empty() { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::AccessDenied, "Missing Date header"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::AccessDenied, + "Missing Date header", + )); } - if let Some(err) = check_timestamp_freshness(amz_date, state.config.sigv4_timestamp_tolerance_secs) { + if let Some(err) = + check_timestamp_freshness(amz_date, state.config.sigv4_timestamp_tolerance_secs) + { return AuthResult::Denied(err); } let secret_key = match state.iam.get_secret_key(access_key) { Some(sk) => sk, None => { - return AuthResult::Denied( - S3Error::from_code(S3ErrorCode::InvalidAccessKeyId), - ); + return AuthResult::Denied(S3Error::from_code(S3ErrorCode::InvalidAccessKeyId)); } }; @@ -350,16 +1058,12 @@ fn verify_sigv4_header(state: &AppState, req: &Request, auth_str: &str) -> AuthR ); if !verified { - return AuthResult::Denied( - S3Error::from_code(S3ErrorCode::SignatureDoesNotMatch), - ); + return AuthResult::Denied(S3Error::from_code(S3ErrorCode::SignatureDoesNotMatch)); } match state.iam.get_principal(access_key) { Some(p) => AuthResult::Ok(p), - None => AuthResult::Denied( - S3Error::from_code(S3ErrorCode::InvalidAccessKeyId), - ), + None => AuthResult::Denied(S3Error::from_code(S3ErrorCode::InvalidAccessKeyId)), } } @@ -374,9 +1078,10 @@ fn verify_sigv4_query(state: &AppState, req: &Request) -> AuthResult { let credential = match param_map.get("X-Amz-Credential") { Some(c) => *c, None => { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Missing X-Amz-Credential"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing X-Amz-Credential", + )); } }; @@ -387,33 +1092,37 @@ fn verify_sigv4_query(state: &AppState, req: &Request) -> AuthResult { let provided_signature = match param_map.get("X-Amz-Signature") { Some(s) => *s, None => { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Missing X-Amz-Signature"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing X-Amz-Signature", + )); } }; let amz_date = match param_map.get("X-Amz-Date") { Some(d) => *d, None => { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Missing X-Amz-Date"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing X-Amz-Date", + )); } }; let expires_str = match param_map.get("X-Amz-Expires") { Some(e) => *e, None => { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Missing X-Amz-Expires"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Missing X-Amz-Expires", + )); } }; let cred_parts: Vec<&str> = credential.split('/').collect(); if cred_parts.len() != 5 { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Malformed credential"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Malformed credential", + )); } let access_key = cred_parts[0]; @@ -424,44 +1133,44 @@ fn verify_sigv4_query(state: &AppState, req: &Request) -> AuthResult { let expires: u64 = match expires_str.parse() { Ok(e) => e, Err(_) => { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "Invalid X-Amz-Expires"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "Invalid X-Amz-Expires", + )); } }; if expires < state.config.presigned_url_min_expiry || expires > state.config.presigned_url_max_expiry { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::InvalidArgument, "X-Amz-Expires out of range"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::InvalidArgument, + "X-Amz-Expires out of range", + )); } - if let Ok(request_time) = - NaiveDateTime::parse_from_str(amz_date, "%Y%m%dT%H%M%SZ") - { + if let Ok(request_time) = NaiveDateTime::parse_from_str(amz_date, "%Y%m%dT%H%M%SZ") { let request_utc = request_time.and_utc(); let now = Utc::now(); let elapsed = (now - request_utc).num_seconds(); if elapsed > expires as i64 { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::AccessDenied, "Request has expired"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::AccessDenied, + "Request has expired", + )); } if elapsed < -(state.config.sigv4_timestamp_tolerance_secs as i64) { - return AuthResult::Denied( - S3Error::new(S3ErrorCode::AccessDenied, "Request is too far in the future"), - ); + return AuthResult::Denied(S3Error::new( + S3ErrorCode::AccessDenied, + "Request is too far in the future", + )); } } let secret_key = match state.iam.get_secret_key(access_key) { Some(sk) => sk, None => { - return AuthResult::Denied( - S3Error::from_code(S3ErrorCode::InvalidAccessKeyId), - ); + return AuthResult::Denied(S3Error::from_code(S3ErrorCode::InvalidAccessKeyId)); } }; @@ -505,16 +1214,12 @@ fn verify_sigv4_query(state: &AppState, req: &Request) -> AuthResult { ); if !verified { - return AuthResult::Denied( - S3Error::from_code(S3ErrorCode::SignatureDoesNotMatch), - ); + return AuthResult::Denied(S3Error::from_code(S3ErrorCode::SignatureDoesNotMatch)); } match state.iam.get_principal(access_key) { Some(p) => AuthResult::Ok(p), - None => AuthResult::Denied( - S3Error::from_code(S3ErrorCode::InvalidAccessKeyId), - ), + None => AuthResult::Denied(S3Error::from_code(S3ErrorCode::InvalidAccessKeyId)), } } @@ -543,10 +1248,7 @@ fn parse_query_params(query: &str) -> Vec<(String, String)> { let mut parts = pair.splitn(2, '='); let key = parts.next()?; let value = parts.next().unwrap_or(""); - Some(( - urlencoding_decode(key), - urlencoding_decode(value), - )) + Some((urlencoding_decode(key), urlencoding_decode(value))) }) .collect() } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/middleware/mod.rs b/rust/myfsio-engine/crates/myfsio-server/src/middleware/mod.rs index a7e6347..b8ab0ef 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/middleware/mod.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/middleware/mod.rs @@ -4,15 +4,82 @@ pub mod session; pub use auth::auth_layer; pub use session::{csrf_layer, session_layer, SessionHandle, SessionLayerState}; -use axum::extract::Request; +use axum::extract::{Request, State}; use axum::middleware::Next; use axum::response::Response; +use std::time::Instant; + +use crate::state::AppState; pub async fn server_header(req: Request, next: Next) -> Response { let mut resp = next.run(req).await; - resp.headers_mut().insert( - "server", - crate::SERVER_HEADER.parse().unwrap(), - ); + resp.headers_mut() + .insert("server", crate::SERVER_HEADER.parse().unwrap()); resp } + +pub async fn ui_metrics_layer(State(state): State, req: Request, next: Next) -> Response { + let metrics = match state.metrics.clone() { + Some(m) => m, + None => return next.run(req).await, + }; + let start = Instant::now(); + let method = req.method().clone(); + let path = req.uri().path().to_string(); + let endpoint_type = classify_ui_endpoint(&path); + let bytes_in = req + .headers() + .get(axum::http::header::CONTENT_LENGTH) + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.parse::().ok()) + .unwrap_or(0); + + let response = next.run(req).await; + + let latency_ms = start.elapsed().as_secs_f64() * 1000.0; + let status = response.status().as_u16(); + let bytes_out = response + .headers() + .get(axum::http::header::CONTENT_LENGTH) + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.parse::().ok()) + .unwrap_or(0); + let error_code = if status >= 400 { Some("UIError") } else { None }; + metrics.record_request( + method.as_str(), + endpoint_type, + status, + latency_ms, + bytes_in, + bytes_out, + error_code, + ); + + response +} + +fn classify_ui_endpoint(path: &str) -> &'static str { + if path.contains("/upload") { + "ui_upload" + } else if path.starts_with("/ui/buckets/") { + "ui_bucket" + } else if path.starts_with("/ui/iam") { + "ui_iam" + } else if path.starts_with("/ui/sites") { + "ui_sites" + } else if path.starts_with("/ui/connections") { + "ui_connections" + } else if path.starts_with("/ui/metrics") { + "ui_metrics" + } else if path.starts_with("/ui/system") { + "ui_system" + } else if path.starts_with("/ui/website-domains") { + "ui_website_domains" + } else if path.starts_with("/ui/replication") { + "ui_replication" + } else if path.starts_with("/login") || path.starts_with("/logout") { + "ui_auth" + } else { + "ui_other" + } +} diff --git a/rust/myfsio-engine/crates/myfsio-server/src/middleware/session.rs b/rust/myfsio-engine/crates/myfsio-server/src/middleware/session.rs index 3fc5347..60618bc 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/middleware/session.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/middleware/session.rs @@ -62,18 +62,14 @@ pub async fn session_layer( ) -> Response { let cookie_id = extract_session_cookie(&req); - let (session_id, session_data, is_new) = match cookie_id.and_then(|id| { - state - .store - .get(&id) - .map(|data| (id.clone(), data)) - }) { - Some((id, data)) => (id, data, false), - None => { - let (id, data) = state.store.create(); - (id, data, true) - } - }; + let (session_id, session_data, is_new) = + match cookie_id.and_then(|id| state.store.get(&id).map(|data| (id.clone(), data))) { + Some((id, data)) => (id, data, false), + None => { + let (id, data) = state.store.create(); + (id, data, true) + } + }; let handle = SessionHandle::new(session_id.clone(), session_data); req.extensions_mut().insert(handle.clone()); @@ -95,6 +91,8 @@ pub async fn session_layer( } pub async fn csrf_layer(req: Request, next: Next) -> Response { + const CSRF_HEADER_ALIAS: &str = "x-csrftoken"; + let method = req.method().clone(); let needs_check = matches!( method, @@ -126,29 +124,33 @@ pub async fn csrf_layer(req: Request, next: Next) -> Response { let header_token = req .headers() .get(CSRF_HEADER_NAME) + .or_else(|| req.headers().get(CSRF_HEADER_ALIAS)) .and_then(|v| v.to_str().ok()) .map(|s| s.to_string()); - if let Some(token) = header_token { - if csrf_tokens_match(&expected, &token) { + if let Some(token) = header_token.as_deref() { + if csrf_tokens_match(&expected, token) { return next.run(req).await; } } + let content_type = req + .headers() + .get(header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()) + .unwrap_or("") + .to_string(); + let (parts, body) = req.into_parts(); let bytes = match axum::body::to_bytes(body, usize::MAX).await { Ok(b) => b, Err(_) => return (StatusCode::BAD_REQUEST, "Body read failed").into_response(), }; - let content_type = parts - .headers - .get(header::CONTENT_TYPE) - .and_then(|v| v.to_str().ok()) - .unwrap_or(""); - let form_token = if content_type.starts_with("application/x-www-form-urlencoded") { extract_form_token(&bytes) + } else if content_type.starts_with("multipart/form-data") { + extract_multipart_token(&content_type, &bytes) } else { None }; @@ -160,9 +162,32 @@ pub async fn csrf_layer(req: Request, next: Next) -> Response { } } + tracing::warn!( + path = %parts.uri.path(), + content_type = %content_type, + expected_len = expected.len(), + header_present = header_token.is_some(), + "CSRF token mismatch" + ); (StatusCode::FORBIDDEN, "Invalid CSRF token").into_response() } +fn extract_multipart_token(content_type: &str, body: &[u8]) -> Option { + let boundary = multer::parse_boundary(content_type).ok()?; + let prefix = format!("--{}", boundary); + let text = std::str::from_utf8(body).ok()?; + let needle = "name=\"csrf_token\""; + let idx = text.find(needle)?; + let after = &text[idx + needle.len()..]; + let body_start = after.find("\r\n\r\n")? + 4; + let tail = &after[body_start..]; + let end = tail + .find(&format!("\r\n--{}", prefix.trim_start_matches("--"))) + .or_else(|| tail.find("\r\n--")) + .unwrap_or(tail.len()); + Some(tail[..end].trim().to_string()) +} + fn extract_session_cookie(req: &Request) -> Option { let raw = req.headers().get(header::COOKIE)?.to_str().ok()?; for pair in raw.split(';') { diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/access_logging.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/access_logging.rs new file mode 100644 index 0000000..0bc03bc --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/access_logging.rs @@ -0,0 +1,105 @@ +use parking_lot::RwLock; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct LoggingConfiguration { + pub target_bucket: String, + #[serde(default)] + pub target_prefix: String, + #[serde(default = "default_enabled")] + pub enabled: bool, +} + +fn default_enabled() -> bool { + true +} + +#[derive(Serialize, Deserialize)] +struct StoredLoggingFile { + #[serde(rename = "LoggingEnabled")] + logging_enabled: Option, +} + +#[derive(Serialize, Deserialize)] +struct StoredLoggingEnabled { + #[serde(rename = "TargetBucket")] + target_bucket: String, + #[serde(rename = "TargetPrefix", default)] + target_prefix: String, +} + +pub struct AccessLoggingService { + storage_root: PathBuf, + cache: RwLock>>, +} + +impl AccessLoggingService { + pub fn new(storage_root: &Path) -> Self { + Self { + storage_root: storage_root.to_path_buf(), + cache: RwLock::new(HashMap::new()), + } + } + + fn config_path(&self, bucket: &str) -> PathBuf { + self.storage_root + .join(".myfsio.sys") + .join("buckets") + .join(bucket) + .join("logging.json") + } + + pub fn get(&self, bucket: &str) -> Option { + if let Some(cached) = self.cache.read().get(bucket).cloned() { + return cached; + } + + let path = self.config_path(bucket); + let config = if path.exists() { + std::fs::read_to_string(&path) + .ok() + .and_then(|s| serde_json::from_str::(&s).ok()) + .and_then(|f| f.logging_enabled) + .map(|e| LoggingConfiguration { + target_bucket: e.target_bucket, + target_prefix: e.target_prefix, + enabled: true, + }) + } else { + None + }; + + self.cache + .write() + .insert(bucket.to_string(), config.clone()); + config + } + + pub fn set(&self, bucket: &str, config: LoggingConfiguration) -> std::io::Result<()> { + let path = self.config_path(bucket); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + let stored = StoredLoggingFile { + logging_enabled: Some(StoredLoggingEnabled { + target_bucket: config.target_bucket.clone(), + target_prefix: config.target_prefix.clone(), + }), + }; + let json = serde_json::to_string_pretty(&stored) + .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?; + std::fs::write(&path, json)?; + self.cache.write().insert(bucket.to_string(), Some(config)); + Ok(()) + } + + pub fn delete(&self, bucket: &str) { + let path = self.config_path(bucket); + if path.exists() { + let _ = std::fs::remove_file(&path); + } + self.cache.write().insert(bucket.to_string(), None); + } +} diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/gc.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/gc.rs index e1d930f..37bb402 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/gc.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/gc.rs @@ -28,6 +28,7 @@ pub struct GcService { storage_root: PathBuf, config: GcConfig, running: Arc>, + started_at: Arc>>, history: Arc>>, history_path: PathBuf, } @@ -53,6 +54,7 @@ impl GcService { storage_root, config, running: Arc::new(RwLock::new(false)), + started_at: Arc::new(RwLock::new(None)), history: Arc::new(RwLock::new(history)), history_path, } @@ -60,9 +62,17 @@ impl GcService { pub async fn status(&self) -> Value { let running = *self.running.read().await; + let scan_elapsed_seconds = self + .started_at + .read() + .await + .as_ref() + .map(|started| started.elapsed().as_secs_f64()); json!({ "enabled": true, "running": running, + "scanning": running, + "scan_elapsed_seconds": scan_elapsed_seconds, "interval_hours": self.config.interval_hours, "temp_file_max_age_hours": self.config.temp_file_max_age_hours, "multipart_max_age_days": self.config.multipart_max_age_days, @@ -73,7 +83,9 @@ impl GcService { pub async fn history(&self) -> Value { let history = self.history.read().await; - json!({ "executions": *history }) + let mut executions: Vec = history.iter().cloned().collect(); + executions.reverse(); + json!({ "executions": executions }) } pub async fn run_now(&self, dry_run: bool) -> Result { @@ -84,12 +96,14 @@ impl GcService { } *running = true; } + *self.started_at.write().await = Some(Instant::now()); let start = Instant::now(); let result = self.execute_gc(dry_run || self.config.dry_run).await; let elapsed = start.elapsed().as_secs_f64(); *self.running.write().await = false; + *self.started_at.write().await = None; let mut result_json = result.clone(); if let Some(obj) = result_json.as_object_mut() { @@ -124,9 +138,12 @@ impl GcService { let mut errors: Vec = Vec::new(); let now = std::time::SystemTime::now(); - let temp_max_age = std::time::Duration::from_secs_f64(self.config.temp_file_max_age_hours * 3600.0); - let multipart_max_age = std::time::Duration::from_secs(self.config.multipart_max_age_days * 86400); - let lock_max_age = std::time::Duration::from_secs_f64(self.config.lock_file_max_age_hours * 3600.0); + let temp_max_age = + std::time::Duration::from_secs_f64(self.config.temp_file_max_age_hours * 3600.0); + let multipart_max_age = + std::time::Duration::from_secs(self.config.multipart_max_age_days * 86400); + let lock_max_age = + std::time::Duration::from_secs_f64(self.config.lock_file_max_age_hours * 3600.0); let tmp_dir = self.storage_root.join(".myfsio.sys").join("tmp"); if tmp_dir.exists() { @@ -140,7 +157,10 @@ impl GcService { let size = metadata.len(); if !dry_run { if let Err(e) = std::fs::remove_file(entry.path()) { - errors.push(format!("Failed to remove temp file: {}", e)); + errors.push(format!( + "Failed to remove temp file: {}", + e + )); continue; } } @@ -242,7 +262,10 @@ impl GcService { if let Some(parent) = self.history_path.parent() { let _ = std::fs::create_dir_all(parent); } - let _ = std::fs::write(&self.history_path, serde_json::to_string_pretty(&data).unwrap_or_default()); + let _ = std::fs::write( + &self.history_path, + serde_json::to_string_pretty(&data).unwrap_or_default(), + ); } pub fn start_background(self: Arc) -> tokio::task::JoinHandle<()> { diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/integrity.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/integrity.rs index 7cf2f03..0d7ef03 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/integrity.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/integrity.rs @@ -1,11 +1,17 @@ +use myfsio_common::constants::{ + BUCKET_META_DIR, BUCKET_VERSIONS_DIR, INDEX_FILE, SYSTEM_BUCKETS_DIR, SYSTEM_ROOT, +}; use myfsio_storage::fs_backend::FsStorageBackend; -use myfsio_storage::traits::StorageEngine; -use serde_json::{json, Value}; -use std::path::PathBuf; +use serde_json::{json, Map, Value}; +use std::collections::{HashMap, HashSet}; +use std::path::{Path, PathBuf}; use std::sync::Arc; use std::time::Instant; use tokio::sync::RwLock; +const MAX_ISSUES: usize = 500; +const INTERNAL_FOLDERS: &[&str] = &[".meta", ".versions", ".multipart"]; + pub struct IntegrityConfig { pub interval_hours: f64, pub batch_size: usize, @@ -17,7 +23,7 @@ impl Default for IntegrityConfig { fn default() -> Self { Self { interval_hours: 24.0, - batch_size: 1000, + batch_size: 10_000, auto_heal: false, dry_run: false, } @@ -25,21 +31,70 @@ impl Default for IntegrityConfig { } pub struct IntegrityService { + #[allow(dead_code)] storage: Arc, + storage_root: PathBuf, config: IntegrityConfig, running: Arc>, + started_at: Arc>>, history: Arc>>, history_path: PathBuf, } +#[derive(Default)] +struct ScanState { + objects_scanned: u64, + buckets_scanned: u64, + corrupted_objects: u64, + orphaned_objects: u64, + phantom_metadata: u64, + stale_versions: u64, + etag_cache_inconsistencies: u64, + issues: Vec, + errors: Vec, +} + +impl ScanState { + fn batch_exhausted(&self, batch_size: usize) -> bool { + self.objects_scanned >= batch_size as u64 + } + + fn push_issue(&mut self, issue_type: &str, bucket: &str, key: &str, detail: String) { + if self.issues.len() < MAX_ISSUES { + self.issues.push(json!({ + "issue_type": issue_type, + "bucket": bucket, + "key": key, + "detail": detail, + })); + } + } + + fn into_json(self, elapsed: f64) -> Value { + json!({ + "objects_scanned": self.objects_scanned, + "buckets_scanned": self.buckets_scanned, + "corrupted_objects": self.corrupted_objects, + "orphaned_objects": self.orphaned_objects, + "phantom_metadata": self.phantom_metadata, + "stale_versions": self.stale_versions, + "etag_cache_inconsistencies": self.etag_cache_inconsistencies, + "issues_healed": 0, + "issues": self.issues, + "errors": self.errors, + "execution_time_seconds": elapsed, + }) + } +} + impl IntegrityService { pub fn new( storage: Arc, - storage_root: &std::path::Path, + storage_root: &Path, config: IntegrityConfig, ) -> Self { let history_path = storage_root - .join(".myfsio.sys") + .join(SYSTEM_ROOT) .join("config") .join("integrity_history.json"); @@ -55,8 +110,10 @@ impl IntegrityService { Self { storage, + storage_root: storage_root.to_path_buf(), config, running: Arc::new(RwLock::new(false)), + started_at: Arc::new(RwLock::new(None)), history: Arc::new(RwLock::new(history)), history_path, } @@ -64,9 +121,17 @@ impl IntegrityService { pub async fn status(&self) -> Value { let running = *self.running.read().await; + let scan_elapsed_seconds = self + .started_at + .read() + .await + .as_ref() + .map(|started| started.elapsed().as_secs_f64()); json!({ "enabled": true, "running": running, + "scanning": running, + "scan_elapsed_seconds": scan_elapsed_seconds, "interval_hours": self.config.interval_hours, "batch_size": self.config.batch_size, "auto_heal": self.config.auto_heal, @@ -76,7 +141,9 @@ impl IntegrityService { pub async fn history(&self) -> Value { let history = self.history.read().await; - json!({ "executions": *history }) + let mut executions: Vec = history.iter().cloned().collect(); + executions.reverse(); + json!({ "executions": executions }) } pub async fn run_now(&self, dry_run: bool, auto_heal: bool) -> Result { @@ -87,23 +154,31 @@ impl IntegrityService { } *running = true; } + *self.started_at.write().await = Some(Instant::now()); let start = Instant::now(); - let result = self.check_integrity(dry_run, auto_heal).await; + let storage_root = self.storage_root.clone(); + let batch_size = self.config.batch_size; + let result = + tokio::task::spawn_blocking(move || scan_all_buckets(&storage_root, batch_size)) + .await + .unwrap_or_else(|e| { + let mut st = ScanState::default(); + st.errors.push(format!("scan task failed: {}", e)); + st + }); let elapsed = start.elapsed().as_secs_f64(); *self.running.write().await = false; + *self.started_at.write().await = None; - let mut result_json = result.clone(); - if let Some(obj) = result_json.as_object_mut() { - obj.insert("execution_time_seconds".to_string(), json!(elapsed)); - } + let result_json = result.into_json(elapsed); let record = json!({ "timestamp": chrono::Utc::now().timestamp_millis() as f64 / 1000.0, "dry_run": dry_run, "auto_heal": auto_heal, - "result": result_json, + "result": result_json.clone(), }); { @@ -116,62 +191,7 @@ impl IntegrityService { } self.save_history().await; - Ok(result) - } - - async fn check_integrity(&self, _dry_run: bool, _auto_heal: bool) -> Value { - let buckets = match self.storage.list_buckets().await { - Ok(b) => b, - Err(e) => return json!({"error": e.to_string()}), - }; - - let mut objects_scanned = 0u64; - let mut corrupted = 0u64; - let mut phantom_metadata = 0u64; - let mut errors: Vec = Vec::new(); - - for bucket in &buckets { - let params = myfsio_common::types::ListParams { - max_keys: self.config.batch_size, - ..Default::default() - }; - let objects = match self.storage.list_objects(&bucket.name, ¶ms).await { - Ok(r) => r.objects, - Err(e) => { - errors.push(format!("{}: {}", bucket.name, e)); - continue; - } - }; - - for obj in &objects { - objects_scanned += 1; - match self.storage.get_object_path(&bucket.name, &obj.key).await { - Ok(path) => { - if !path.exists() { - phantom_metadata += 1; - } else if let Some(ref expected_etag) = obj.etag { - match myfsio_crypto::hashing::md5_file(&path) { - Ok(actual_etag) => { - if &actual_etag != expected_etag { - corrupted += 1; - } - } - Err(e) => errors.push(format!("{}:{}: {}", bucket.name, obj.key, e)), - } - } - } - Err(e) => errors.push(format!("{}:{}: {}", bucket.name, obj.key, e)), - } - } - } - - json!({ - "objects_scanned": objects_scanned, - "buckets_scanned": buckets.len(), - "corrupted_objects": corrupted, - "phantom_metadata": phantom_metadata, - "errors": errors, - }) + Ok(result_json) } async fn save_history(&self) { @@ -202,3 +222,511 @@ impl IntegrityService { }) } } + +fn scan_all_buckets(storage_root: &Path, batch_size: usize) -> ScanState { + let mut state = ScanState::default(); + let buckets = match list_bucket_names(storage_root) { + Ok(b) => b, + Err(e) => { + state.errors.push(format!("list buckets: {}", e)); + return state; + } + }; + + for bucket in &buckets { + if state.batch_exhausted(batch_size) { + break; + } + state.buckets_scanned += 1; + + let bucket_path = storage_root.join(bucket); + let meta_root = storage_root + .join(SYSTEM_ROOT) + .join(SYSTEM_BUCKETS_DIR) + .join(bucket) + .join(BUCKET_META_DIR); + + let index_entries = collect_index_entries(&meta_root); + + check_corrupted(&mut state, bucket, &bucket_path, &index_entries, batch_size); + check_phantom(&mut state, bucket, &bucket_path, &index_entries, batch_size); + check_orphaned(&mut state, bucket, &bucket_path, &index_entries, batch_size); + check_stale_versions(&mut state, storage_root, bucket, batch_size); + check_etag_cache(&mut state, storage_root, bucket, &index_entries, batch_size); + } + + state +} + +fn list_bucket_names(storage_root: &Path) -> std::io::Result> { + let mut names = Vec::new(); + if !storage_root.exists() { + return Ok(names); + } + for entry in std::fs::read_dir(storage_root)? { + let entry = entry?; + let name = entry.file_name().to_string_lossy().to_string(); + if name == SYSTEM_ROOT { + continue; + } + if entry.file_type().map(|t| t.is_dir()).unwrap_or(false) { + names.push(name); + } + } + Ok(names) +} + +#[allow(dead_code)] +struct IndexEntryInfo { + entry: Value, + index_file: PathBuf, + key_name: String, +} + +fn collect_index_entries(meta_root: &Path) -> HashMap { + let mut out: HashMap = HashMap::new(); + if !meta_root.exists() { + return out; + } + + let mut stack: Vec = vec![meta_root.to_path_buf()]; + while let Some(dir) = stack.pop() { + let rd = match std::fs::read_dir(&dir) { + Ok(r) => r, + Err(_) => continue, + }; + for entry in rd.flatten() { + let path = entry.path(); + let ft = match entry.file_type() { + Ok(t) => t, + Err(_) => continue, + }; + if ft.is_dir() { + stack.push(path); + continue; + } + if entry.file_name().to_string_lossy() != INDEX_FILE { + continue; + } + let rel_dir = match path.parent().and_then(|p| p.strip_prefix(meta_root).ok()) { + Some(p) => p.to_path_buf(), + None => continue, + }; + let dir_prefix = if rel_dir.as_os_str().is_empty() { + String::new() + } else { + rel_dir + .components() + .map(|c| c.as_os_str().to_string_lossy().to_string()) + .collect::>() + .join("/") + }; + + let content = match std::fs::read_to_string(&path) { + Ok(c) => c, + Err(_) => continue, + }; + let index_data: Map = match serde_json::from_str(&content) { + Ok(Value::Object(m)) => m, + _ => continue, + }; + + for (key_name, entry_val) in index_data { + let full_key = if dir_prefix.is_empty() { + key_name.clone() + } else { + format!("{}/{}", dir_prefix, key_name) + }; + out.insert( + full_key, + IndexEntryInfo { + entry: entry_val, + index_file: path.clone(), + key_name, + }, + ); + } + } + } + out +} + +fn stored_etag(entry: &Value) -> Option { + entry + .get("metadata") + .and_then(|m| m.get("__etag__")) + .and_then(|v| v.as_str()) + .map(|s| s.to_string()) +} + +fn check_corrupted( + state: &mut ScanState, + bucket: &str, + bucket_path: &Path, + entries: &HashMap, + batch_size: usize, +) { + let mut keys: Vec<&String> = entries.keys().collect(); + keys.sort(); + + for full_key in keys { + if state.batch_exhausted(batch_size) { + return; + } + let info = &entries[full_key]; + let object_path = bucket_path.join(full_key); + if !object_path.exists() { + continue; + } + state.objects_scanned += 1; + + let Some(stored) = stored_etag(&info.entry) else { + continue; + }; + + match myfsio_crypto::hashing::md5_file(&object_path) { + Ok(actual) => { + if actual != stored { + state.corrupted_objects += 1; + state.push_issue( + "corrupted_object", + bucket, + full_key, + format!("stored_etag={} actual_etag={}", stored, actual), + ); + } + } + Err(e) => state + .errors + .push(format!("hash {}/{}: {}", bucket, full_key, e)), + } + } +} + +fn check_phantom( + state: &mut ScanState, + bucket: &str, + bucket_path: &Path, + entries: &HashMap, + batch_size: usize, +) { + let mut keys: Vec<&String> = entries.keys().collect(); + keys.sort(); + + for full_key in keys { + if state.batch_exhausted(batch_size) { + return; + } + state.objects_scanned += 1; + let object_path = bucket_path.join(full_key); + if !object_path.exists() { + state.phantom_metadata += 1; + state.push_issue( + "phantom_metadata", + bucket, + full_key, + "metadata entry without file on disk".to_string(), + ); + } + } +} + +fn check_orphaned( + state: &mut ScanState, + bucket: &str, + bucket_path: &Path, + entries: &HashMap, + batch_size: usize, +) { + let indexed: HashSet<&String> = entries.keys().collect(); + let mut stack: Vec<(PathBuf, String)> = vec![(bucket_path.to_path_buf(), String::new())]; + + while let Some((dir, prefix)) = stack.pop() { + if state.batch_exhausted(batch_size) { + return; + } + let rd = match std::fs::read_dir(&dir) { + Ok(r) => r, + Err(_) => continue, + }; + for entry in rd.flatten() { + if state.batch_exhausted(batch_size) { + return; + } + let name = entry.file_name().to_string_lossy().to_string(); + let ft = match entry.file_type() { + Ok(t) => t, + Err(_) => continue, + }; + if ft.is_dir() { + if prefix.is_empty() && INTERNAL_FOLDERS.contains(&name.as_str()) { + continue; + } + let new_prefix = if prefix.is_empty() { + name + } else { + format!("{}/{}", prefix, name) + }; + stack.push((entry.path(), new_prefix)); + } else if ft.is_file() { + let full_key = if prefix.is_empty() { + name + } else { + format!("{}/{}", prefix, name) + }; + state.objects_scanned += 1; + if !indexed.contains(&full_key) { + state.orphaned_objects += 1; + state.push_issue( + "orphaned_object", + bucket, + &full_key, + "file exists without metadata entry".to_string(), + ); + } + } + } + } +} + +fn check_stale_versions( + state: &mut ScanState, + storage_root: &Path, + bucket: &str, + batch_size: usize, +) { + let versions_root = storage_root + .join(SYSTEM_ROOT) + .join(SYSTEM_BUCKETS_DIR) + .join(bucket) + .join(BUCKET_VERSIONS_DIR); + if !versions_root.exists() { + return; + } + + let mut stack: Vec = vec![versions_root.clone()]; + while let Some(dir) = stack.pop() { + if state.batch_exhausted(batch_size) { + return; + } + let rd = match std::fs::read_dir(&dir) { + Ok(r) => r, + Err(_) => continue, + }; + + let mut bin_stems: HashMap = HashMap::new(); + let mut json_stems: HashMap = HashMap::new(); + let mut subdirs: Vec = Vec::new(); + + for entry in rd.flatten() { + let ft = match entry.file_type() { + Ok(t) => t, + Err(_) => continue, + }; + let path = entry.path(); + if ft.is_dir() { + subdirs.push(path); + continue; + } + let name = entry.file_name().to_string_lossy().to_string(); + if let Some(stem) = name.strip_suffix(".bin") { + bin_stems.insert(stem.to_string(), path); + } else if let Some(stem) = name.strip_suffix(".json") { + json_stems.insert(stem.to_string(), path); + } + } + + for (stem, path) in &bin_stems { + if state.batch_exhausted(batch_size) { + return; + } + state.objects_scanned += 1; + if !json_stems.contains_key(stem) { + state.stale_versions += 1; + let key = path + .strip_prefix(&versions_root) + .map(|p| p.to_string_lossy().replace('\\', "/")) + .unwrap_or_else(|_| path.display().to_string()); + state.push_issue( + "stale_version", + bucket, + &key, + "version data without manifest".to_string(), + ); + } + } + + for (stem, path) in &json_stems { + if state.batch_exhausted(batch_size) { + return; + } + state.objects_scanned += 1; + if !bin_stems.contains_key(stem) { + state.stale_versions += 1; + let key = path + .strip_prefix(&versions_root) + .map(|p| p.to_string_lossy().replace('\\', "/")) + .unwrap_or_else(|_| path.display().to_string()); + state.push_issue( + "stale_version", + bucket, + &key, + "version manifest without data".to_string(), + ); + } + } + + stack.extend(subdirs); + } +} + +fn check_etag_cache( + state: &mut ScanState, + storage_root: &Path, + bucket: &str, + entries: &HashMap, + batch_size: usize, +) { + let etag_index_path = storage_root + .join(SYSTEM_ROOT) + .join(SYSTEM_BUCKETS_DIR) + .join(bucket) + .join("etag_index.json"); + if !etag_index_path.exists() { + return; + } + + let cache: HashMap = match std::fs::read_to_string(&etag_index_path) + .ok() + .and_then(|s| serde_json::from_str(&s).ok()) + { + Some(Value::Object(m)) => m.into_iter().collect(), + _ => return, + }; + + for (full_key, cached_val) in cache { + if state.batch_exhausted(batch_size) { + return; + } + state.objects_scanned += 1; + let Some(cached_etag) = cached_val.as_str() else { + continue; + }; + let Some(info) = entries.get(&full_key) else { + continue; + }; + let Some(stored) = stored_etag(&info.entry) else { + continue; + }; + if cached_etag != stored { + state.etag_cache_inconsistencies += 1; + state.push_issue( + "etag_cache_inconsistency", + bucket, + &full_key, + format!("cached_etag={} index_etag={}", cached_etag, stored), + ); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::fs; + + fn md5_hex(bytes: &[u8]) -> String { + myfsio_crypto::hashing::md5_bytes(bytes) + } + + fn write_index(meta_dir: &Path, entries: &[(&str, &str)]) { + fs::create_dir_all(meta_dir).unwrap(); + let mut map = Map::new(); + for (name, etag) in entries { + map.insert( + name.to_string(), + json!({ "metadata": { "__etag__": etag } }), + ); + } + fs::write( + meta_dir.join(INDEX_FILE), + serde_json::to_string(&Value::Object(map)).unwrap(), + ) + .unwrap(); + } + + #[test] + fn scan_detects_each_issue_type() { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + let bucket = "testbucket"; + let bucket_path = root.join(bucket); + let meta_root = root + .join(SYSTEM_ROOT) + .join(SYSTEM_BUCKETS_DIR) + .join(bucket) + .join(BUCKET_META_DIR); + fs::create_dir_all(&bucket_path).unwrap(); + + let clean_bytes = b"clean file contents"; + let clean_etag = md5_hex(clean_bytes); + fs::write(bucket_path.join("clean.txt"), clean_bytes).unwrap(); + + let corrupted_bytes = b"actual content"; + fs::write(bucket_path.join("corrupted.txt"), corrupted_bytes).unwrap(); + + fs::write(bucket_path.join("orphan.txt"), b"no metadata").unwrap(); + + write_index( + &meta_root, + &[ + ("clean.txt", &clean_etag), + ("corrupted.txt", "00000000000000000000000000000000"), + ("phantom.txt", "deadbeefdeadbeefdeadbeefdeadbeef"), + ], + ); + + let versions_root = root + .join(SYSTEM_ROOT) + .join(SYSTEM_BUCKETS_DIR) + .join(bucket) + .join(BUCKET_VERSIONS_DIR) + .join("someobject"); + fs::create_dir_all(&versions_root).unwrap(); + fs::write(versions_root.join("v1.bin"), b"orphan bin").unwrap(); + fs::write(versions_root.join("v2.json"), b"{}").unwrap(); + + let etag_index = root + .join(SYSTEM_ROOT) + .join(SYSTEM_BUCKETS_DIR) + .join(bucket) + .join("etag_index.json"); + fs::write( + &etag_index, + serde_json::to_string(&json!({ "clean.txt": "stale-cached-etag" })).unwrap(), + ) + .unwrap(); + + let state = scan_all_buckets(root, 10_000); + + assert_eq!(state.corrupted_objects, 1, "corrupted"); + assert_eq!(state.phantom_metadata, 1, "phantom"); + assert_eq!(state.orphaned_objects, 1, "orphaned"); + assert_eq!(state.stale_versions, 2, "stale versions"); + assert_eq!(state.etag_cache_inconsistencies, 1, "etag cache"); + assert_eq!(state.buckets_scanned, 1); + assert!( + state.errors.is_empty(), + "unexpected errors: {:?}", + state.errors + ); + } + + #[test] + fn skips_system_root_as_bucket() { + let tmp = tempfile::tempdir().unwrap(); + fs::create_dir_all(tmp.path().join(SYSTEM_ROOT).join("config")).unwrap(); + let state = scan_all_buckets(tmp.path(), 100); + assert_eq!(state.buckets_scanned, 0); + } +} diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/lifecycle.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/lifecycle.rs index b68706a..071e6ab 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/lifecycle.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/lifecycle.rs @@ -66,7 +66,10 @@ impl LifecycleService { None => continue, }; - let rules = match lifecycle.as_str().and_then(|s| serde_json::from_str::(s).ok()) { + let rules = match lifecycle + .as_str() + .and_then(|s| serde_json::from_str::(s).ok()) + { Some(v) => v, None => continue, }; @@ -93,7 +96,11 @@ impl LifecycleService { let cutoff = chrono::Utc::now() - chrono::Duration::days(days as i64); let params = myfsio_common::types::ListParams { max_keys: 1000, - prefix: if prefix.is_empty() { None } else { Some(prefix.to_string()) }, + prefix: if prefix.is_empty() { + None + } else { + Some(prefix.to_string()) + }, ..Default::default() }; if let Ok(result) = self.storage.list_objects(&bucket.name, ¶ms).await { @@ -101,7 +108,8 @@ impl LifecycleService { if obj.last_modified < cutoff { match self.storage.delete_object(&bucket.name, &obj.key).await { Ok(()) => total_expired += 1, - Err(e) => errors.push(format!("{}:{}: {}", bucket.name, obj.key, e)), + Err(e) => errors + .push(format!("{}:{}: {}", bucket.name, obj.key, e)), } } } @@ -112,12 +120,18 @@ impl LifecycleService { if let Some(abort) = rule.get("AbortIncompleteMultipartUpload") { if let Some(days) = abort.get("DaysAfterInitiation").and_then(|d| d.as_u64()) { let cutoff = chrono::Utc::now() - chrono::Duration::days(days as i64); - if let Ok(uploads) = self.storage.list_multipart_uploads(&bucket.name).await { + if let Ok(uploads) = self.storage.list_multipart_uploads(&bucket.name).await + { for upload in &uploads { if upload.initiated < cutoff { - match self.storage.abort_multipart(&bucket.name, &upload.upload_id).await { + match self + .storage + .abort_multipart(&bucket.name, &upload.upload_id) + .await + { Ok(()) => total_multipart_aborted += 1, - Err(e) => errors.push(format!("abort {}: {}", upload.upload_id, e)), + Err(e) => errors + .push(format!("abort {}: {}", upload.upload_id, e)), } } } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/metrics.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/metrics.rs index 84e08e2..59314bc 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/metrics.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/metrics.rs @@ -165,8 +165,9 @@ impl MetricsService { .ok() .and_then(|s| serde_json::from_str::(&s).ok()) .and_then(|v| { - v.get("snapshots") - .and_then(|s| serde_json::from_value::>(s.clone()).ok()) + v.get("snapshots").and_then(|s| { + serde_json::from_value::>(s.clone()).ok() + }) }) .unwrap_or_default() } else { @@ -218,7 +219,9 @@ impl MetricsService { if let Some(code) = error_code { *inner.error_codes.entry(code.to_string()).or_insert(0) += 1; } - inner.totals.record(latency_ms, success, bytes_in, bytes_out); + inner + .totals + .record(latency_ms, success, bytes_in, bytes_out); } pub fn get_current_stats(&self) -> Value { diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/mod.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/mod.rs index 9612243..9da9f01 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/mod.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/mod.rs @@ -1,9 +1,11 @@ +pub mod access_logging; pub mod gc; -pub mod lifecycle; pub mod integrity; +pub mod lifecycle; pub mod metrics; pub mod replication; pub mod s3_client; pub mod site_registry; pub mod site_sync; +pub mod system_metrics; pub mod website_domains; diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/replication.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/replication.rs index 9a1e7ca..922174f 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/replication.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/replication.rs @@ -8,6 +8,7 @@ use parking_lot::Mutex; use serde::{Deserialize, Serialize}; use tokio::sync::Semaphore; +use myfsio_common::types::ListParams; use myfsio_storage::fs_backend::FsStorageBackend; use myfsio_storage::traits::StorageEngine; @@ -124,7 +125,10 @@ impl ReplicationFailureStore { } let trimmed = &failures[..failures.len().min(self.max_failures_per_bucket)]; let data = serde_json::json!({ "failures": trimmed }); - let _ = std::fs::write(&path, serde_json::to_string_pretty(&data).unwrap_or_default()); + let _ = std::fs::write( + &path, + serde_json::to_string_pretty(&data).unwrap_or_default(), + ); } pub fn load(&self, bucket: &str) -> Vec { @@ -148,7 +152,10 @@ impl ReplicationFailureStore { pub fn add(&self, bucket: &str, failure: ReplicationFailure) { let mut failures = self.load(bucket); - if let Some(existing) = failures.iter_mut().find(|f| f.object_key == failure.object_key) { + if let Some(existing) = failures + .iter_mut() + .find(|f| f.object_key == failure.object_key) + { existing.failure_count += 1; existing.timestamp = failure.timestamp; existing.error_message = failure.error_message.clone(); @@ -318,7 +325,101 @@ impl ReplicationManager { let manager = self.clone(); tokio::spawn(async move { let _permit = permit; - manager.replicate_task(&bucket, &key, &rule, &connection, &action).await; + manager + .replicate_task(&bucket, &key, &rule, &connection, &action) + .await; + }); + } + + pub async fn replicate_existing_objects(self: Arc, bucket: String) -> usize { + let rule = match self.get_rule(&bucket) { + Some(r) if r.enabled => r, + _ => return 0, + }; + let connection = match self.connections.get(&rule.target_connection_id) { + Some(c) => c, + None => { + tracing::warn!( + "Cannot replicate existing objects for {}: connection {} not found", + bucket, + rule.target_connection_id + ); + return 0; + } + }; + if !self.check_endpoint(&connection).await { + tracing::warn!( + "Cannot replicate existing objects for {}: endpoint {} is unreachable", + bucket, + connection.endpoint_url + ); + return 0; + } + + let mut continuation_token: Option = None; + let mut submitted = 0usize; + + loop { + let page = match self + .storage + .list_objects( + &bucket, + &ListParams { + max_keys: 1000, + continuation_token: continuation_token.clone(), + prefix: rule.filter_prefix.clone(), + start_after: None, + }, + ) + .await + { + Ok(page) => page, + Err(err) => { + tracing::error!( + "Failed to list existing objects for replication in {}: {}", + bucket, + err + ); + break; + } + }; + + let next_token = page.next_continuation_token.clone(); + let is_truncated = page.is_truncated; + + for object in page.objects { + submitted += 1; + self.clone() + .trigger(bucket.clone(), object.key, "write".to_string()) + .await; + } + + if !is_truncated { + break; + } + + continuation_token = next_token; + if continuation_token.is_none() { + break; + } + } + + submitted + } + + pub fn schedule_existing_objects_sync(self: Arc, bucket: String) { + tokio::spawn(async move { + let submitted = self + .clone() + .replicate_existing_objects(bucket.clone()) + .await; + if submitted > 0 { + tracing::info!( + "Scheduled {} existing object(s) for replication in {}", + submitted, + bucket + ); + } }); } @@ -330,7 +431,8 @@ impl ReplicationManager { conn: &RemoteConnection, action: &str, ) { - if object_key.contains("..") || object_key.starts_with('/') || object_key.starts_with('\\') { + if object_key.contains("..") || object_key.starts_with('/') || object_key.starts_with('\\') + { tracing::error!("Invalid object key (path traversal): {}", object_key); return; } @@ -358,7 +460,12 @@ impl ReplicationManager { } Err(err) => { let msg = format!("{:?}", err); - tracing::error!("Replication DELETE failed {}/{}: {}", bucket, object_key, msg); + tracing::error!( + "Replication DELETE failed {}/{}: {}", + bucket, + object_key, + msg + ); self.failures.add( bucket, ReplicationFailure { @@ -414,16 +521,18 @@ impl ReplicationManager { .send() .await { - Ok(_) | Err(_) => upload_object( - &client, - &rule.target_bucket, - object_key, - &src_path, - file_size, - self.streaming_threshold_bytes, - content_type.as_deref(), - ) - .await, + Ok(_) | Err(_) => { + upload_object( + &client, + &rule.target_bucket, + object_key, + &src_path, + file_size, + self.streaming_threshold_bytes, + content_type.as_deref(), + ) + .await + } } } other => other, @@ -577,9 +686,9 @@ async fn upload_object( ))) })? } else { - let bytes = tokio::fs::read(path).await.map_err(|e| { - aws_sdk_s3::error::SdkError::construction_failure(Box::new(e)) - })?; + let bytes = tokio::fs::read(path) + .await + .map_err(|e| aws_sdk_s3::error::SdkError::construction_failure(Box::new(e)))?; ByteStream::from(bytes) }; diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/s3_client.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/s3_client.rs index 431e149..7f085eb 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/s3_client.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/s3_client.rs @@ -37,8 +37,8 @@ pub fn build_client(connection: &RemoteConnection, options: &ClientOptions) -> C .read_timeout(options.read_timeout) .build(); - let retry_config = aws_smithy_types::retry::RetryConfig::standard() - .with_max_attempts(options.max_attempts); + let retry_config = + aws_smithy_types::retry::RetryConfig::standard().with_max_attempts(options.max_attempts); let config = aws_sdk_s3::config::Builder::new() .behavior_version(BehaviorVersion::latest()) diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/site_registry.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/site_registry.rs index f7a2b48..00ab5c5 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/site_registry.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/site_registry.rs @@ -102,7 +102,12 @@ impl SiteRegistry { } pub fn get_peer(&self, site_id: &str) -> Option { - self.data.read().peers.iter().find(|p| p.site_id == site_id).cloned() + self.data + .read() + .peers + .iter() + .find(|p| p.site_id == site_id) + .cloned() } pub fn add_peer(&self, peer: PeerSite) { diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/site_sync.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/site_sync.rs index a067b92..a1e6a67 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/site_sync.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/site_sync.rs @@ -102,7 +102,10 @@ impl SiteSyncWorker { } pub async fn run(self: Arc) { - tracing::info!("Site sync worker started (interval={}s)", self.interval.as_secs()); + tracing::info!( + "Site sync worker started (interval={}s)", + self.interval.as_secs() + ); loop { tokio::select! { _ = tokio::time::sleep(self.interval) => {} @@ -309,11 +312,10 @@ impl SiteSyncWorker { let resp = match req.send().await { Ok(r) => r, Err(err) => { - let msg = format!("{:?}", err); - if msg.contains("NoSuchBucket") { + if is_not_found_error(&err) { return Ok(result); } - return Err(msg); + return Err(format!("{:?}", err)); } }; for obj in resp.contents() { @@ -409,11 +411,9 @@ impl SiteSyncWorker { } }; - let metadata: Option> = head.metadata().map(|m| { - m.iter() - .map(|(k, v)| (k.clone(), v.clone())) - .collect() - }); + let metadata: Option> = head + .metadata() + .map(|m| m.iter().map(|(k, v)| (k.clone(), v.clone())).collect()); let stream = resp.body.into_async_read(); let boxed: Pin> = Box::pin(stream); @@ -428,7 +428,12 @@ impl SiteSyncWorker { true } Err(err) => { - tracing::error!("Store pulled object failed {}/{}: {}", local_bucket, key, err); + tracing::error!( + "Store pulled object failed {}/{}: {}", + local_bucket, + key, + err + ); false } } @@ -483,3 +488,11 @@ fn now_secs() -> f64 { .map(|d| d.as_secs_f64()) .unwrap_or(0.0) } + +fn is_not_found_error(err: &aws_sdk_s3::error::SdkError) -> bool { + let msg = format!("{:?}", err); + msg.contains("NoSuchBucket") + || msg.contains("code: Some(\"NotFound\")") + || msg.contains("code: Some(\"NoSuchBucket\")") + || msg.contains("status: 404") +} diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/system_metrics.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/system_metrics.rs new file mode 100644 index 0000000..6c0849d --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/system_metrics.rs @@ -0,0 +1,203 @@ +use chrono::{DateTime, Utc}; +use myfsio_storage::fs_backend::FsStorageBackend; +use myfsio_storage::traits::StorageEngine; +use serde::{Deserialize, Serialize}; +use serde_json::json; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use sysinfo::{Disks, System}; +use tokio::sync::RwLock; + +#[derive(Debug, Clone)] +pub struct SystemMetricsConfig { + pub interval_minutes: u64, + pub retention_hours: u64, +} + +impl Default for SystemMetricsConfig { + fn default() -> Self { + Self { + interval_minutes: 5, + retention_hours: 24, + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SystemMetricsSnapshot { + pub timestamp: DateTime, + pub cpu_percent: f64, + pub memory_percent: f64, + pub disk_percent: f64, + pub storage_bytes: u64, +} + +pub struct SystemMetricsService { + storage_root: PathBuf, + storage: Arc, + config: SystemMetricsConfig, + history: Arc>>, + history_path: PathBuf, +} + +impl SystemMetricsService { + pub fn new( + storage_root: &Path, + storage: Arc, + config: SystemMetricsConfig, + ) -> Self { + let history_path = storage_root + .join(".myfsio.sys") + .join("config") + .join("metrics_history.json"); + + let mut history = if history_path.exists() { + std::fs::read_to_string(&history_path) + .ok() + .and_then(|s| serde_json::from_str::(&s).ok()) + .and_then(|v| { + v.get("history").and_then(|h| { + serde_json::from_value::>(h.clone()).ok() + }) + }) + .unwrap_or_default() + } else { + Vec::new() + }; + prune_history(&mut history, config.retention_hours); + + Self { + storage_root: storage_root.to_path_buf(), + storage, + config, + history: Arc::new(RwLock::new(history)), + history_path, + } + } + + pub async fn get_history(&self, hours: Option) -> Vec { + let mut history = self.history.read().await.clone(); + prune_history(&mut history, hours.unwrap_or(self.config.retention_hours)); + history + } + + async fn take_snapshot(&self) { + let snapshot = collect_snapshot(&self.storage_root, &self.storage).await; + let mut history = self.history.write().await; + history.push(snapshot); + prune_history(&mut history, self.config.retention_hours); + drop(history); + self.save_history().await; + } + + async fn save_history(&self) { + let history = self.history.read().await; + let data = json!({ "history": *history }); + if let Some(parent) = self.history_path.parent() { + let _ = std::fs::create_dir_all(parent); + } + let _ = std::fs::write( + &self.history_path, + serde_json::to_string_pretty(&data).unwrap_or_default(), + ); + } + + pub fn start_background(self: Arc) -> tokio::task::JoinHandle<()> { + let interval = + std::time::Duration::from_secs(self.config.interval_minutes.saturating_mul(60)); + tokio::spawn(async move { + self.take_snapshot().await; + let mut timer = tokio::time::interval(interval); + loop { + timer.tick().await; + self.take_snapshot().await; + } + }) + } +} + +fn prune_history(history: &mut Vec, retention_hours: u64) { + let cutoff = Utc::now() - chrono::Duration::hours(retention_hours as i64); + history.retain(|item| item.timestamp > cutoff); +} + +fn sample_system_now() -> (f64, f64) { + let mut system = System::new(); + system.refresh_cpu_usage(); + std::thread::sleep(sysinfo::MINIMUM_CPU_UPDATE_INTERVAL); + system.refresh_cpu_usage(); + system.refresh_memory(); + + let cpu_percent = system.global_cpu_usage() as f64; + let memory_percent = if system.total_memory() > 0 { + (system.used_memory() as f64 / system.total_memory() as f64) * 100.0 + } else { + 0.0 + }; + (cpu_percent, memory_percent) +} + +fn normalize_path_for_mount(path: &Path) -> String { + let canonical = path.canonicalize().unwrap_or_else(|_| path.to_path_buf()); + let raw = canonical.to_string_lossy().to_string(); + let stripped = raw.strip_prefix(r"\\?\").unwrap_or(&raw); + stripped.to_lowercase() +} + +fn sample_disk(path: &Path) -> (u64, u64) { + let disks = Disks::new_with_refreshed_list(); + let path_str = normalize_path_for_mount(path); + let mut best: Option<(usize, u64, u64)> = None; + + for disk in disks.list() { + let mount_raw = disk.mount_point().to_string_lossy().to_string(); + let mount = mount_raw + .strip_prefix(r"\\?\") + .unwrap_or(&mount_raw) + .to_lowercase(); + let total = disk.total_space(); + let free = disk.available_space(); + if path_str.starts_with(&mount) { + let len = mount.len(); + match best { + Some((best_len, _, _)) if len <= best_len => {} + _ => best = Some((len, total, free)), + } + } + } + + best.map(|(_, total, free)| (total, free)).unwrap_or((0, 0)) +} + +async fn collect_snapshot( + storage_root: &Path, + storage: &Arc, +) -> SystemMetricsSnapshot { + let (cpu_percent, memory_percent) = sample_system_now(); + let (disk_total, disk_free) = sample_disk(storage_root); + let disk_percent = if disk_total > 0 { + ((disk_total - disk_free) as f64 / disk_total as f64) * 100.0 + } else { + 0.0 + }; + + let mut storage_bytes = 0u64; + let buckets = storage.list_buckets().await.unwrap_or_default(); + for bucket in buckets { + if let Ok(stats) = storage.bucket_stats(&bucket.name).await { + storage_bytes += stats.total_bytes(); + } + } + + SystemMetricsSnapshot { + timestamp: Utc::now(), + cpu_percent: round2(cpu_percent), + memory_percent: round2(memory_percent), + disk_percent: round2(disk_percent), + storage_bytes, + } +} + +fn round2(value: f64) -> f64 { + (value * 100.0).round() / 100.0 +} diff --git a/rust/myfsio-engine/crates/myfsio-server/src/services/website_domains.rs b/rust/myfsio-engine/crates/myfsio-server/src/services/website_domains.rs index 7ce27cf..5060d52 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/services/website_domains.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/services/website_domains.rs @@ -64,7 +64,10 @@ impl WebsiteDomainStore { } pub fn set_mapping(&self, domain: &str, bucket: &str) { - self.data.write().mappings.insert(domain.to_string(), bucket.to_string()); + self.data + .write() + .mappings + .insert(domain.to_string(), bucket.to_string()); self.save(); } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/session.rs b/rust/myfsio-engine/crates/myfsio-server/src/session.rs index d35aedd..dbed866 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/session.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/session.rs @@ -27,21 +27,18 @@ pub struct SessionData { pub csrf_token: String, pub flash: Vec, pub extra: HashMap, - created_at: Instant, last_accessed: Instant, } impl SessionData { pub fn new() -> Self { - let now = Instant::now(); Self { user_id: None, display_name: None, csrf_token: generate_token(CSRF_TOKEN_BYTES), flash: Vec::new(), extra: HashMap::new(), - created_at: now, - last_accessed: now, + last_accessed: Instant::now(), } } diff --git a/rust/myfsio-engine/crates/myfsio-server/src/state.rs b/rust/myfsio-engine/crates/myfsio-server/src/state.rs index 3aeeaa1..34c25ab 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/state.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/state.rs @@ -2,16 +2,18 @@ use std::sync::Arc; use std::time::Duration; use crate::config::ServerConfig; -use crate::session::SessionStore; -use crate::templates::TemplateEngine; +use crate::services::access_logging::AccessLoggingService; use crate::services::gc::GcService; use crate::services::integrity::IntegrityService; use crate::services::metrics::MetricsService; use crate::services::replication::ReplicationManager; use crate::services::site_registry::SiteRegistry; use crate::services::site_sync::SiteSyncWorker; +use crate::services::system_metrics::SystemMetricsService; use crate::services::website_domains::WebsiteDomainStore; +use crate::session::SessionStore; use crate::stores::connections::ConnectionStore; +use crate::templates::TemplateEngine; use myfsio_auth::iam::IamService; use myfsio_crypto::encryption::EncryptionService; use myfsio_crypto::kms::KmsService; @@ -27,6 +29,7 @@ pub struct AppState { pub gc: Option>, pub integrity: Option>, pub metrics: Option>, + pub system_metrics: Option>, pub site_registry: Option>, pub website_domains: Option>, pub connections: Arc, @@ -34,6 +37,7 @@ pub struct AppState { pub site_sync: Option>, pub templates: Option>, pub sessions: Arc, + pub access_logging: Arc, } impl AppState { @@ -66,7 +70,23 @@ impl AppState { let metrics = if config.metrics_enabled { Some(Arc::new(MetricsService::new( &config.storage_root, - crate::services::metrics::MetricsConfig::default(), + crate::services::metrics::MetricsConfig { + interval_minutes: config.metrics_interval_minutes, + retention_hours: config.metrics_retention_hours, + }, + ))) + } else { + None + }; + + let system_metrics = if config.metrics_history_enabled { + Some(Arc::new(SystemMetricsService::new( + &config.storage_root, + storage.clone(), + crate::services::system_metrics::SystemMetricsConfig { + interval_minutes: config.metrics_history_interval_minutes, + retention_hours: config.metrics_history_retention_hours, + }, ))) } else { None @@ -111,6 +131,7 @@ impl AppState { }; let templates = init_templates(&config.templates_dir); + let access_logging = Arc::new(AccessLoggingService::new(&config.storage_root)); Self { config, storage, @@ -120,6 +141,7 @@ impl AppState { gc, integrity, metrics, + system_metrics, site_registry, website_domains, connections, @@ -127,6 +149,7 @@ impl AppState { site_sync, templates, sessions: Arc::new(SessionStore::new(Duration::from_secs(60 * 60 * 12))), + access_logging, } } @@ -149,9 +172,7 @@ impl AppState { let encryption = if config.encryption_enabled { match myfsio_crypto::kms::load_or_create_master_key(&keys_dir).await { - Ok(master_key) => { - Some(Arc::new(EncryptionService::new(master_key, kms.clone()))) - } + Ok(master_key) => Some(Arc::new(EncryptionService::new(master_key, kms.clone()))), Err(e) => { tracing::error!("Failed to initialize encryption: {}", e); None diff --git a/rust/myfsio-engine/crates/myfsio-server/src/templates.rs b/rust/myfsio-engine/crates/myfsio-server/src/templates.rs index 46491f7..153fce3 100644 --- a/rust/myfsio-engine/crates/myfsio-server/src/templates.rs +++ b/rust/myfsio-engine/crates/myfsio-server/src/templates.rs @@ -6,7 +6,8 @@ use parking_lot::RwLock; use serde_json::Value; use tera::{Context, Error as TeraError, Tera}; -pub type EndpointResolver = Arc) -> Option + Send + Sync>; +pub type EndpointResolver = + Arc) -> Option + Send + Sync>; #[derive(Clone)] pub struct TemplateEngine { @@ -17,10 +18,10 @@ pub struct TemplateEngine { impl TemplateEngine { pub fn new(template_glob: &str) -> Result { let mut tera = Tera::new(template_glob)?; + tera.set_escape_fn(html_escape); register_filters(&mut tera); - let endpoints: Arc>> = - Arc::new(RwLock::new(HashMap::new())); + let endpoints: Arc>> = Arc::new(RwLock::new(HashMap::new())); register_functions(&mut tera, endpoints.clone()); @@ -52,9 +53,25 @@ impl TemplateEngine { } } +fn html_escape(input: &str) -> String { + let mut out = String::with_capacity(input.len()); + for c in input.chars() { + match c { + '&' => out.push_str("&"), + '<' => out.push_str("<"), + '>' => out.push_str(">"), + '"' => out.push_str("""), + '\'' => out.push_str("'"), + _ => out.push(c), + } + } + out +} + fn register_filters(tera: &mut Tera) { tera.register_filter("format_datetime", format_datetime_filter); tera.register_filter("filesizeformat", filesizeformat_filter); + tera.register_filter("slice", slice_filter); } fn register_functions(tera: &mut Tera, endpoints: Arc>>) { @@ -67,10 +84,7 @@ fn register_functions(tera: &mut Tera, endpoints: Arc) -> tera: Value::String(s) => DateTime::parse_from_rfc3339(s) .ok() .map(|d| d.with_timezone(&Utc)) - .or_else(|| DateTime::parse_from_rfc2822(s).ok().map(|d| d.with_timezone(&Utc))), + .or_else(|| { + DateTime::parse_from_rfc2822(s) + .ok() + .map(|d| d.with_timezone(&Utc)) + }), Value::Number(n) => n.as_f64().and_then(|f| { let secs = f as i64; let nanos = ((f - secs as f64) * 1_000_000_000.0) as u32; @@ -170,6 +188,51 @@ fn format_datetime_filter(value: &Value, args: &HashMap) -> tera: } } +fn slice_filter(value: &Value, args: &HashMap) -> tera::Result { + let start = args.get("start").and_then(|v| v.as_i64()).unwrap_or(0); + let end = args.get("end").and_then(|v| v.as_i64()); + + match value { + Value::String(s) => { + let chars: Vec = s.chars().collect(); + let len = chars.len() as i64; + let norm = |i: i64| -> usize { + if i < 0 { + (len + i).max(0) as usize + } else { + i.min(len) as usize + } + }; + let s_idx = norm(start); + let e_idx = match end { + Some(e) => norm(e), + None => len as usize, + }; + let e_idx = e_idx.max(s_idx); + Ok(Value::String(chars[s_idx..e_idx].iter().collect())) + } + Value::Array(arr) => { + let len = arr.len() as i64; + let norm = |i: i64| -> usize { + if i < 0 { + (len + i).max(0) as usize + } else { + i.min(len) as usize + } + }; + let s_idx = norm(start); + let e_idx = match end { + Some(e) => norm(e), + None => len as usize, + }; + let e_idx = e_idx.max(s_idx); + Ok(Value::Array(arr[s_idx..e_idx].to_vec())) + } + Value::Null => Ok(Value::String(String::new())), + _ => Err(tera::Error::msg("slice: unsupported value type")), + } +} + fn filesizeformat_filter(value: &Value, _args: &HashMap) -> tera::Result { let bytes = match value { Value::Number(n) => n.as_f64().unwrap_or(0.0), @@ -205,7 +268,10 @@ mod tests { engine.register_endpoints(&[ ("ui.buckets_overview", "/ui/buckets"), ("ui.bucket_detail", "/ui/buckets/{bucket_name}"), - ("ui.abort_multipart_upload", "/ui/buckets/{bucket_name}/multipart/{upload_id}/abort"), + ( + "ui.abort_multipart_upload", + "/ui/buckets/{bucket_name}/multipart/{upload_id}/abort", + ), ]); engine } @@ -220,7 +286,10 @@ mod tests { #[test] fn static_url() { let e = test_engine(); - let out = render_inline(&e, "{{ url_for(endpoint='static', filename='css/main.css') }}"); + let out = render_inline( + &e, + "{{ url_for(endpoint='static', filename='css/main.css') }}", + ); assert_eq!(out, "/static/css/main.css"); } @@ -267,7 +336,11 @@ mod tests { .get_template_names() .map(|s| s.to_string()) .collect(); - assert!(names.len() >= 10, "expected 10+ templates, got {}", names.len()); + assert!( + names.len() >= 10, + "expected 10+ templates, got {}", + names.len() + ); } #[test] diff --git a/rust/myfsio-engine/crates/myfsio-server/static/css/main.css b/rust/myfsio-engine/crates/myfsio-server/static/css/main.css new file mode 100644 index 0000000..e48174d --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/css/main.css @@ -0,0 +1,3157 @@ +:root { + --myfsio-body-bg: #f5f6fa; + --myfsio-text: #0f172a; + --myfsio-card-bg: #ffffff; + --myfsio-card-border: #e2e8f0; + --myfsio-muted: #475569; + --myfsio-input-bg: #ffffff; + --myfsio-input-border: #cbd5f5; + --myfsio-nav-gradient: linear-gradient(90deg, #0f172a, #1d4ed8); + --myfsio-nav-link: rgba(255, 255, 255, 0.85); + --myfsio-nav-link-hover: #ffffff; + --myfsio-preview-bg: #f8f9fb; + --myfsio-policy-bg: #0f172a; + --myfsio-policy-fg: #e2e8f0; + --myfsio-hover-bg: rgba(59, 130, 246, 0.12); + --myfsio-accent: #3b82f6; + --myfsio-accent-hover: #2563eb; + --myfsio-tag-key-bg: #e0e7ff; + --myfsio-tag-key-text: #3730a3; + --myfsio-tag-value-bg: #f0f1fa; + --myfsio-tag-value-text: #4338ca; + --myfsio-tag-border: #c7d2fe; + --myfsio-tag-delete-hover: #ef4444; +} + +[data-theme='dark'] { + --myfsio-body-bg: #0b1120; + --myfsio-text: #e2e8f0; + --myfsio-card-bg: #1a1f2e; + --myfsio-card-border: #2d3548; + --myfsio-muted: #94a3b8; + --myfsio-input-bg: #111827; + --myfsio-input-border: #374151; + --myfsio-nav-gradient: linear-gradient(90deg, #020617, #1e3a8a); + --myfsio-nav-link: rgba(248, 250, 252, 0.85); + --myfsio-nav-link-hover: #ffffff; + --myfsio-preview-bg: #1f2937; + --myfsio-policy-bg: #0f1419; + --myfsio-policy-fg: #f8fafc; + --myfsio-hover-bg: rgba(59, 130, 246, 0.2); + --myfsio-accent: #60a5fa; + --myfsio-accent-hover: #3b82f6; + --myfsio-tag-key-bg: #312e81; + --myfsio-tag-key-text: #c7d2fe; + --myfsio-tag-value-bg: #1e1b4b; + --myfsio-tag-value-text: #a5b4fc; + --myfsio-tag-border: #4338ca; + --myfsio-tag-delete-hover: #f87171; +} + +[data-theme='dark'] body, +[data-theme='dark'] html { + color-scheme: dark; +} + +body { + background-color: var(--myfsio-body-bg); + color: var(--myfsio-text); + transition: background-color 0.3s ease, color 0.3s ease; +} + +html, body { + min-height: 100%; +} + +main { + color: var(--myfsio-text); + background-color: var(--myfsio-body-bg); +} + +html { + background-color: var(--myfsio-body-bg); + scroll-behavior: smooth; +} + +.text-muted, +.form-text { + color: var(--myfsio-muted) !important; +} + +.table-responsive { + border-radius: 0.5rem; + overflow-x: auto; + -webkit-overflow-scrolling: touch; +} +.message-stack { position: sticky; top: 1rem; z-index: 100; } + +.table-responsive table { + min-width: 600px; +} + +.table-responsive table th, +.table-responsive table td { + white-space: nowrap; +} + +.table-responsive table td.text-wrap { + white-space: normal; + min-width: 200px; +} +code { font-size: 0.85rem; } + +code { + background-color: rgba(15, 23, 42, 0.08); + color: var(--myfsio-text); + padding: 0.15rem 0.4rem; + border-radius: 0.25rem; +} + +[data-theme='dark'] code { + background-color: rgba(148, 163, 184, 0.15); + color: #93c5fd; +} + +.card, +.card-header, +.modal-content, +.dropdown-menu, +.list-group-item { + background-color: var(--myfsio-card-bg); + color: var(--myfsio-text); + border-color: var(--myfsio-card-border); +} + +.bg-panel { + background-color: var(--myfsio-preview-bg); + color: var(--myfsio-text); + border-color: var(--myfsio-card-border) !important; +} + +.border-dashed { + border-style: dashed !important; +} + +.card { + box-shadow: 0 1px 3px 0 rgba(0, 0, 0, 0.1), 0 1px 2px 0 rgba(0, 0, 0, 0.06); +} + +[data-theme='dark'] .card { + box-shadow: 0 1px 3px 0 rgba(0, 0, 0, 0.3), 0 1px 2px 0 rgba(0, 0, 0, 0.2); +} + +.card-header { + font-weight: 500; +} + +.drop-zone { + position: relative; + transition: all 0.2s ease; +} + +.drop-zone.drag-over { + background-color: var(--myfsio-hover-bg); + border: 2px dashed var(--myfsio-input-border); +} + +.drop-zone.drag-over::after { + content: 'Drop files here to upload'; + position: absolute; + top: 50%; + left: 50%; + transform: translate(-50%, -50%); + font-size: 1.5rem; + font-weight: 600; + color: var(--myfsio-muted); + pointer-events: none; + z-index: 10; +} + +.drop-zone.drag-over table { + opacity: 0.3; +} + +.modal-header, +.modal-footer { + border-color: var(--myfsio-card-border); +} + +:root { + --sidebar-width: 260px; + --sidebar-collapsed-width: 72px; + --mobile-header-height: 56px; + --sidebar-bg: linear-gradient(180deg, #0f172a 0%, #1e293b 100%); + --sidebar-border: rgba(255, 255, 255, 0.08); + --sidebar-link-color: rgba(255, 255, 255, 0.7); + --sidebar-link-hover: rgba(255, 255, 255, 0.95); + --sidebar-link-active-bg: rgba(59, 130, 246, 0.2); + --sidebar-link-active-border: #3b82f6; + --sidebar-section-color: rgba(255, 255, 255, 0.4); +} + +[data-theme='dark'] { + --sidebar-bg: linear-gradient(180deg, #020617 0%, #0f172a 100%); + --sidebar-border: rgba(255, 255, 255, 0.06); + --sidebar-link-active-bg: rgba(59, 130, 246, 0.25); +} + +body { + display: flex; + min-height: 100vh; +} + +.main-wrapper { + flex: 1; + display: flex; + flex-direction: column; + min-width: 0; + transition: margin-left 0.3s cubic-bezier(0.4, 0, 0.2, 1); +} + +@media (min-width: 992px) { + .main-wrapper { + margin-left: var(--sidebar-width); + } + + body.sidebar-is-collapsed .main-wrapper { + margin-left: var(--sidebar-collapsed-width); + } +} + +.main-content { + flex: 1; + padding: 1.5rem; + max-width: 100%; + overflow-x: hidden; +} + +@media (min-width: 992px) { + .main-content { + padding: 2rem 2.5rem; + } +} + +.mobile-header { + position: fixed; + top: 0; + left: 0; + right: 0; + height: var(--mobile-header-height); + background: var(--sidebar-bg); + display: flex; + align-items: center; + justify-content: space-between; + padding: 0 1rem; + z-index: 1030; + box-shadow: 0 2px 8px rgba(0, 0, 0, 0.15); +} + +.sidebar-toggle-btn { + width: 40px; + height: 40px; + display: flex; + align-items: center; + justify-content: center; + background: rgba(255, 255, 255, 0.1); + border: none; + border-radius: 10px; + color: white; + cursor: pointer; + transition: all 0.2s ease; +} + +.sidebar-toggle-btn:hover { + background: rgba(255, 255, 255, 0.15); + transform: scale(1.05); +} + +.mobile-brand { + display: flex; + align-items: center; + gap: 0.5rem; + text-decoration: none; + color: white; + font-weight: 600; + font-size: 1.1rem; +} + +.mobile-brand img { + border-radius: 8px; +} + +.theme-toggle-mobile { + width: 40px; + height: 40px; + display: flex; + align-items: center; + justify-content: center; + background: rgba(255, 255, 255, 0.1); + border: none; + border-radius: 10px; + color: white; + cursor: pointer; + transition: all 0.2s ease; +} + +.theme-toggle-mobile:hover { + background: rgba(255, 255, 255, 0.15); +} + +@media (max-width: 991.98px) { + .main-wrapper { + padding-top: var(--mobile-header-height); + } +} + +.sidebar { + position: fixed; + top: 0; + left: 0; + width: var(--sidebar-width); + height: 100vh; + background: var(--sidebar-bg); + display: flex; + flex-direction: column; + z-index: 1040; + border-right: 1px solid var(--sidebar-border); + transition: width 0.3s cubic-bezier(0.4, 0, 0.2, 1); + overflow: visible; +} + +.sidebar-collapsed { + width: var(--sidebar-collapsed-width); +} + +html.sidebar-will-collapse .sidebar { + width: var(--sidebar-collapsed-width); +} + +html.sidebar-will-collapse .main-wrapper { + margin-left: var(--sidebar-collapsed-width); +} + +html.sidebar-will-collapse .sidebar, +html.sidebar-will-collapse .sidebar *, +html.sidebar-will-collapse .main-wrapper { + transition: none !important; +} + +html.sidebar-will-collapse .sidebar-title, +html.sidebar-will-collapse .sidebar-link-text, +html.sidebar-will-collapse .nav-section-title, +html.sidebar-will-collapse .theme-toggle-text, +html.sidebar-will-collapse .sidebar-user .user-info, +html.sidebar-will-collapse .sidebar-logout-btn .logout-text, +html.sidebar-will-collapse .sidebar-collapse-btn { + display: none !important; + opacity: 0 !important; + width: 0 !important; +} + +html.sidebar-will-collapse .sidebar-header { + justify-content: center; + padding: 1rem 0; +} + +html.sidebar-will-collapse .sidebar-brand { + justify-content: center; +} + +html.sidebar-will-collapse .nav-section { + padding: 0; + width: 100%; +} + +html.sidebar-will-collapse .sidebar-nav { + align-items: center; +} + +html.sidebar-will-collapse .sidebar-link { + justify-content: center; + align-items: center; + width: 48px; + height: 48px; + padding: 0; + margin: 0.25rem auto; + gap: 0; +} + +html.sidebar-will-collapse .sidebar-footer { + padding: 0.75rem 0.5rem; + align-items: center; +} + +html.sidebar-will-collapse .theme-toggle-sidebar, +html.sidebar-will-collapse .sidebar-user, +html.sidebar-will-collapse .sidebar-logout-btn { + justify-content: center; + width: 48px; + height: 48px; + padding: 0; + margin: 0 auto; +} + +html.sidebar-will-collapse .sidebar-user { + background: transparent; +} + +.sidebar-header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 1.25rem 1rem; + border-bottom: 1px solid var(--sidebar-border); + min-height: 70px; + gap: 0.5rem; + overflow: visible; +} + +.sidebar-collapsed .sidebar-header { + justify-content: center; + padding: 1rem 0; + min-height: 70px; +} + +.sidebar-collapsed .sidebar-logo { + width: 36px; + height: 36px; +} + +.sidebar-collapsed .sidebar-brand { + justify-content: center; + cursor: pointer; + position: relative; + gap: 0; +} + +.sidebar-collapsed .sidebar-brand::after { + content: 'Click to expand'; + position: absolute; + left: calc(100% + 12px); + top: 50%; + transform: translateY(-50%); + background: #1e293b; + color: white; + padding: 0.5rem 0.875rem; + border-radius: 8px; + font-size: 0.8rem; + font-weight: 500; + white-space: nowrap; + opacity: 0; + visibility: hidden; + transition: all 0.2s ease; + z-index: 1050; + box-shadow: 0 4px 12px rgba(0, 0, 0, 0.2); +} + +.sidebar-collapsed .sidebar-brand:hover::after { + opacity: 1; + visibility: visible; +} + +.sidebar-collapsed .sidebar-collapse-btn { + display: none; +} + +.sidebar-brand { + display: flex; + align-items: center; + gap: 0.75rem; + text-decoration: none; + color: white; + overflow: visible; +} + +.sidebar-logo { + border-radius: 10px; + box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3); + flex-shrink: 0; + transition: transform 0.2s ease; +} + +.sidebar-brand:hover .sidebar-logo { + transform: scale(1.05); +} + +.sidebar-title { + font-weight: 700; + font-size: 1.25rem; + letter-spacing: -0.02em; + white-space: nowrap; + opacity: 1; + transition: opacity 0.2s ease; +} + +.sidebar-collapsed .sidebar-title { + opacity: 0; + width: 0; +} + +.sidebar-collapse-btn { + width: 32px; + height: 32px; + display: flex; + align-items: center; + justify-content: center; + background: rgba(255, 255, 255, 0.1); + border: none; + border-radius: 8px; + color: rgba(255, 255, 255, 0.7); + cursor: pointer; + transition: all 0.2s ease; + flex-shrink: 0; +} + +.sidebar-collapse-btn:hover { + background: rgba(255, 255, 255, 0.15); + color: white; +} + +.sidebar-collapse-btn svg { + transition: transform 0.3s ease; +} + +.sidebar-collapsed .sidebar-collapse-btn svg { + transform: rotate(180deg); +} + +.sidebar-body { + flex: 1; + padding: 1rem 0; +} + +.sidebar-nav { + display: flex; + flex-direction: column; + gap: 0.5rem; + overflow: visible; +} + +.nav-section { + padding: 0 0.75rem; + margin-bottom: 0.5rem; + overflow: visible; +} + +.nav-section-title { + display: block; + font-size: 0.7rem; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.08em; + color: var(--sidebar-section-color); + padding: 0.5rem 0.75rem; + white-space: nowrap; + overflow: hidden; + transition: opacity 0.2s ease; +} + +.sidebar-collapsed .nav-section-title { + opacity: 0; + height: 0; + padding: 0; + margin: 0; +} + +.sidebar-link { + display: flex; + align-items: center; + gap: 0.875rem; + padding: 0.75rem 1rem; + color: var(--sidebar-link-color); + text-decoration: none; + border-radius: 10px; + transition: all 0.2s ease; + position: relative; + overflow: visible; + margin: 0.5rem; +} + +.sidebar-link::before { + content: ''; + position: absolute; + left: 0; + top: 50%; + transform: translateY(-50%); + width: 3px; + height: 0; + background: var(--sidebar-link-active-border); + border-radius: 0 2px 2px 0; + transition: height 0.2s ease; +} + +.sidebar-link:hover { + color: var(--sidebar-link-hover); + background: rgba(255, 255, 255, 0.08); +} + +.sidebar-link.active { + color: white; + background: var(--sidebar-link-active-bg); +} + +.sidebar-link.active::before { + height: 60%; +} + +.sidebar-link svg { + flex-shrink: 0; + opacity: 0.9; + transition: transform 0.2s ease; +} + +.sidebar-link:hover svg { + transform: scale(1.1); +} + +.sidebar-link-text { + white-space: nowrap; + overflow: hidden; + transition: opacity 0.2s ease, width 0.2s ease; +} + +.sidebar-collapsed .sidebar-link-text { + opacity: 0; + width: 0; +} + +.sidebar-collapsed .sidebar-link { + justify-content: center; + align-items: center; + padding: 0; + margin: 0.25rem auto; + width: 48px; + height: 48px; + gap: 0; +} + +.sidebar-collapsed .sidebar-link svg { + width: 20px; + height: 20px; + flex-shrink: 0; +} + +.sidebar-collapsed .nav-section { + padding: 0; + width: 100%; +} + +.sidebar-collapsed .sidebar-nav { + align-items: center; +} + +.sidebar-collapsed .sidebar-link[data-tooltip]::after { + content: attr(data-tooltip); + position: absolute; + left: calc(100% + 12px); + top: 50%; + transform: translateY(-50%); + background: #1e293b; + color: white; + padding: 0.5rem 0.875rem; + border-radius: 8px; + font-size: 0.85rem; + font-weight: 500; + white-space: nowrap; + opacity: 0; + visibility: hidden; + transition: all 0.2s ease; + z-index: 1050; + box-shadow: 0 4px 12px rgba(0, 0, 0, 0.2); +} + +.sidebar-collapsed .sidebar-link[data-tooltip]:hover::after { + opacity: 1; + visibility: visible; +} + +.sidebar-footer { + padding: 1rem; + border-top: 1px solid var(--sidebar-border); + display: flex; + flex-direction: column; + gap: 0.75rem; + overflow: visible; +} + +.sidebar-collapsed .sidebar-footer { + padding: 0.75rem 0.5rem; + align-items: center; +} + +.theme-toggle-sidebar { + display: flex; + align-items: center; + gap: 0.75rem; + width: 100%; + padding: 0.75rem 1rem; + background: rgba(255, 255, 255, 0.08); + border: none; + border-radius: 10px; + color: rgba(255, 255, 255, 0.8); + cursor: pointer; + transition: all 0.2s ease; +} + +.theme-toggle-sidebar:hover { + background: rgba(255, 255, 255, 0.12); + color: white; +} + +.theme-toggle-text { + font-size: 0.875rem; + font-weight: 500; + white-space: nowrap; + overflow: hidden; + transition: opacity 0.2s ease; +} + +.sidebar-collapsed .theme-toggle-text { + opacity: 0; + width: 0; + display: none; +} + +.sidebar-collapsed .theme-toggle-sidebar { + justify-content: center; + padding: 0; + width: 48px; + height: 48px; + margin: 0 auto; + position: relative; +} + +.sidebar-collapsed .theme-toggle-sidebar svg { + width: 18px; + height: 18px; + flex-shrink: 0; +} + +.sidebar-collapsed .theme-toggle-sidebar::after { + content: 'Toggle theme'; + position: absolute; + left: calc(100% + 12px); + top: 50%; + transform: translateY(-50%); + background: #1e293b; + color: white; + padding: 0.5rem 0.875rem; + border-radius: 8px; + font-size: 0.8rem; + font-weight: 500; + white-space: nowrap; + opacity: 0; + visibility: hidden; + transition: all 0.2s ease; + z-index: 1050; + box-shadow: 0 4px 12px rgba(0, 0, 0, 0.2); +} + +.sidebar-collapsed .theme-toggle-sidebar:hover::after { + opacity: 1; + visibility: visible; +} + +.sidebar-user { + display: flex; + align-items: center; + gap: 0.75rem; + padding: 0.75rem; + background: rgba(255, 255, 255, 0.05); + border-radius: 10px; + overflow: hidden; +} + +.sidebar-user .user-avatar { + width: 36px; + height: 36px; + border-radius: 10px; + background: linear-gradient(135deg, #3b82f6, #8b5cf6); + display: flex; + align-items: center; + justify-content: center; + color: white; + flex-shrink: 0; +} + +.sidebar-user .user-info { + overflow: hidden; + transition: opacity 0.2s ease, width 0.2s ease; +} + +.sidebar-collapsed .sidebar-user .user-info { + opacity: 0; + width: 0; + display: none; +} + +.sidebar-collapsed .sidebar-user { + justify-content: center; + padding: 0; + width: 48px; + height: 48px; + margin: 0 auto; + background: transparent; + position: relative; +} + +.sidebar-collapsed .sidebar-user .user-avatar { + width: 36px; + height: 36px; +} + +.sidebar-collapsed .sidebar-user::after { + content: attr(data-username); + position: absolute; + left: calc(100% + 12px); + top: 50%; + transform: translateY(-50%); + background: #1e293b; + color: white; + padding: 0.5rem 0.875rem; + border-radius: 8px; + font-size: 0.8rem; + font-weight: 500; + white-space: nowrap; + opacity: 0; + visibility: hidden; + transition: all 0.2s ease; + z-index: 1050; + box-shadow: 0 4px 12px rgba(0, 0, 0, 0.2); +} + +.sidebar-collapsed .sidebar-user:hover::after { + opacity: 1; + visibility: visible; +} + +.sidebar-user .user-name { + font-size: 0.875rem; + font-weight: 600; + color: white; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.sidebar-user .user-key { + font-size: 0.75rem; + color: rgba(255, 255, 255, 0.5); + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.sidebar-logout-btn { + display: flex; + align-items: center; + justify-content: center; + gap: 0.75rem; + width: 100%; + padding: 0.75rem 1rem; + background: rgba(239, 68, 68, 0.15); + border: 1px solid rgba(239, 68, 68, 0.3); + border-radius: 10px; + color: #fca5a5; + font-size: 0.875rem; + font-weight: 500; + cursor: pointer; + transition: all 0.2s ease; +} + +.sidebar-logout-btn:hover { + background: rgba(239, 68, 68, 0.25); + border-color: rgba(239, 68, 68, 0.5); + color: #fecaca; +} + +.sidebar-collapsed .sidebar-logout-btn .logout-text { + display: none; +} + +.sidebar-collapsed .sidebar-logout-btn { + justify-content: center; + padding: 0; + width: 48px; + height: 48px; + margin: 0 auto; + position: relative; +} + +.sidebar-collapsed .sidebar-logout-btn svg { + width: 18px; + height: 18px; + flex-shrink: 0; +} + +.sidebar-collapsed .sidebar-logout-btn::after { + content: 'Sign out'; + position: absolute; + left: calc(100% + 12px); + top: 50%; + transform: translateY(-50%); + background: #1e293b; + color: white; + padding: 0.5rem 0.875rem; + border-radius: 8px; + font-size: 0.8rem; + font-weight: 500; + white-space: nowrap; + opacity: 0; + visibility: hidden; + transition: all 0.2s ease; + z-index: 1050; + box-shadow: 0 4px 12px rgba(0, 0, 0, 0.2); +} + +.sidebar-collapsed .sidebar-logout-btn:hover::after { + opacity: 1; + visibility: visible; +} + +.sidebar-offcanvas { + width: 280px !important; + background: var(--sidebar-bg) !important; + border-right: none !important; +} + +.sidebar-offcanvas .sidebar-header { + background: transparent; +} + +.sidebar-offcanvas .sidebar-body { + display: flex; + flex-direction: column; + padding: 0; +} + +.sidebar-offcanvas .sidebar-nav { + flex: 1; + padding: 1rem 0; +} + +.sidebar-offcanvas .sidebar-footer { + margin-top: auto; +} + +.myfsio-nav { + background: var(--myfsio-nav-gradient); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); + backdrop-filter: blur(8px); + -webkit-backdrop-filter: blur(8px); +} + +.myfsio-nav .navbar-brand { + color: #fff; + font-weight: 600; + letter-spacing: -0.02em; + display: inline-flex; + align-items: center; + gap: 0.5rem; +} + +.myfsio-logo { + border-radius: 0.35rem; + box-shadow: 0 0 6px rgba(15, 23, 42, 0.35); + background-color: rgba(255, 255, 255, 0.1); +} + +.myfsio-title { + display: inline-block; +} + +.myfsio-nav .nav-link { + color: var(--myfsio-nav-link); + transition: color 0.2s ease; +} + +.myfsio-nav .nav-link:hover { + color: var(--myfsio-nav-link-hover); +} + +.myfsio-nav .nav-link.nav-link-muted { opacity: 0.75; } + +.myfsio-nav .nav-link.nav-link-muted .badge { + color: #0f172a; + background-color: #fef08a; +} + +[data-theme='dark'] .myfsio-nav .nav-link.nav-link-muted .badge { + color: #0f172a; + background-color: #fde047; +} + +.myfsio-nav .navbar-toggler { + border-color: rgba(255, 255, 255, 0.6); +} + +.myfsio-nav .navbar-toggler-icon { + filter: invert(1); +} + +.docs-hero { + background: var(--myfsio-nav-gradient); + color: #fff !important; + border: 1px solid rgba(255, 255, 255, 0.2); + box-shadow: 0 15px 35px rgba(15, 23, 42, 0.3); +} + +.docs-hero * { + color: inherit; +} + +.docs-callout { + background-color: rgba(15, 23, 42, 0.35); + border: 1px solid rgba(255, 255, 255, 0.35); + border-radius: 0.75rem; + padding: 1rem 1.25rem; +} + +.docs-callout code { + color: #fff; + background-color: rgba(0, 0, 0, 0.2); +} + +[data-theme='dark'] .docs-callout { + background-color: rgba(2, 6, 23, 0.55); + border-color: rgba(255, 255, 255, 0.25); +} + +.docs-feature-card + .docs-feature-card { + margin-top: 1.25rem; +} + +.docs-checklist { + padding-left: 1.25rem; + display: flex; + flex-direction: column; + gap: 0.35rem; +} + +.docs-checklist li { + margin: 0; +} + +.docs-section { + border: 1px solid var(--myfsio-card-border); + border-radius: 1rem; +} + +.docs-section-kicker { + display: inline-flex; + align-items: center; + justify-content: center; + width: 40px; + height: 40px; + border-radius: 999px; + background: rgba(59, 130, 246, 0.12); + color: #1d4ed8; + font-weight: 600; +} + +[data-theme='dark'] .docs-section-kicker { + background: rgba(59, 130, 246, 0.25); + color: #93c5fd; +} + +.docs-steps { + counter-reset: docs-step; + margin: 1rem 0 1.25rem; + padding-left: 1.25rem; +} + +.docs-steps li { + margin-bottom: 0.4rem; +} + +.docs-highlight { + background: rgba(59, 130, 246, 0.08); + border-radius: 0.75rem; + padding: 1rem 1.25rem; + border: 1px solid rgba(59, 130, 246, 0.2); +} + +[data-theme='dark'] .docs-highlight { + background: rgba(59, 130, 246, 0.18); + border-color: rgba(59, 130, 246, 0.35); +} + +.docs-pill-list { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); + gap: 1.5rem; +} + +.docs-pill-list ul { + padding-left: 1.1rem; + margin-bottom: 0; +} + +.docs-table thead { + text-transform: uppercase; + font-size: 0.75rem; + letter-spacing: 0.08em; +} + +[data-theme='dark'] .docs-table .table-secondary, +[data-theme='dark'] .docs-section .table-secondary { + --bs-table-bg: rgba(148, 163, 184, 0.14); + --bs-table-striped-bg: rgba(148, 163, 184, 0.16); + --bs-table-hover-bg: rgba(148, 163, 184, 0.2); + --bs-table-color: var(--myfsio-text); + color: var(--myfsio-text); +} + +[data-theme='dark'] .docs-table .table-secondary th, +[data-theme='dark'] .docs-table .table-secondary td, +[data-theme='dark'] .docs-table .table-secondary strong, +[data-theme='dark'] .docs-table .table-secondary code, +[data-theme='dark'] .docs-section .table-secondary th, +[data-theme='dark'] .docs-section .table-secondary td, +[data-theme='dark'] .docs-section .table-secondary strong, +[data-theme='dark'] .docs-section .table-secondary code { + color: var(--myfsio-text); +} + +.main-content:has(.docs-sidebar) { + overflow-x: visible; +} + +.docs-sidebar { + position: sticky; + top: 1.5rem; + border-radius: 1rem; + border: 1px solid var(--myfsio-card-border); + max-height: calc(100vh - 3rem); + overflow-y: auto; +} + +.docs-sidebar-callouts { + display: flex; + flex-direction: column; + gap: 0.85rem; + padding: 1rem; + border-radius: 0.75rem; + background-color: rgba(15, 23, 42, 0.04); +} + +[data-theme='dark'] .docs-sidebar-callouts { + background-color: rgba(248, 250, 252, 0.05); +} + +.docs-sidebar-callouts code { + font-size: 0.85rem; +} + +.docs-toc a { + color: var(--myfsio-text); + text-decoration: none; + display: inline-flex; + gap: 0.35rem; + align-items: center; + padding: 0.2rem 0; +} + +.docs-toc a:hover { + color: #2563eb; +} + +.docs-sidebar-mobile { + border-radius: 0.75rem; + border: 1px solid var(--myfsio-card-border); +} + +.docs-sidebar-mobile .docs-toc { + display: flex; + flex-wrap: wrap; + gap: 0.5rem 1rem; + padding-top: 0.5rem; +} + +.docs-sidebar-mobile .docs-toc li { + flex: 1 0 45%; +} + +.min-width-0 { + min-width: 0; +} + +.alert pre { + max-width: 100%; + overflow-x: auto; + -webkit-overflow-scrolling: touch; +} + +.iam-user-card { + position: relative; + border: 1px solid var(--myfsio-card-border) !important; + border-radius: 1rem !important; + overflow: visible; + transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1); +} + +.iam-user-card:hover { + transform: translateY(-2px); + box-shadow: 0 8px 24px -4px rgba(0, 0, 0, 0.12), 0 4px 8px -4px rgba(0, 0, 0, 0.08); + border-color: var(--myfsio-accent) !important; +} + +[data-theme='dark'] .iam-user-card:hover { + box-shadow: 0 8px 24px -4px rgba(0, 0, 0, 0.4), 0 4px 8px -4px rgba(0, 0, 0, 0.3); +} + + +.iam-role-badge { + display: inline-flex; + align-items: center; + padding: 0.25em 0.65em; + border-radius: 999px; + font-size: 0.7rem; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.03em; +} + +.iam-role-admin { + background: rgba(245, 158, 11, 0.15); + color: #d97706; +} + +[data-theme='dark'] .iam-role-admin { + background: rgba(245, 158, 11, 0.25); + color: #fbbf24; +} + +.iam-role-user { + background: rgba(59, 130, 246, 0.12); + color: #2563eb; +} + +[data-theme='dark'] .iam-role-user { + background: rgba(59, 130, 246, 0.2); + color: #60a5fa; +} + +.iam-perm-badge { + display: inline-flex; + align-items: center; + gap: 0.25rem; + padding: 0.3em 0.6em; + border-radius: 999px; + font-size: 0.75rem; + font-weight: 500; + background: rgba(59, 130, 246, 0.08); + color: var(--myfsio-text); + border: 1px solid rgba(59, 130, 246, 0.15); +} + +[data-theme='dark'] .iam-perm-badge { + background: rgba(59, 130, 246, 0.15); + border-color: rgba(59, 130, 246, 0.25); +} + +.iam-copy-key { + display: inline-flex; + align-items: center; + justify-content: center; + width: 22px; + height: 22px; + padding: 0; + border: none; + background: transparent; + color: var(--myfsio-muted); + border-radius: 4px; + cursor: pointer; + transition: all 0.15s ease; + flex-shrink: 0; +} + +.iam-copy-key:hover { + background: var(--myfsio-hover-bg); + color: var(--myfsio-text); +} + +.iam-no-results { + text-align: center; + padding: 2rem 1rem; + color: var(--myfsio-muted); +} + +@media (max-width: 768px) { + .iam-user-card:hover { + transform: none; + } +} + +.user-avatar-lg { + width: 48px; + height: 48px; + border-radius: 12px; +} + +.btn-icon { + padding: 0.25rem; + line-height: 1; + border: none; + background: transparent; + color: var(--myfsio-muted); + border-radius: 0.375rem; +} + +.btn-icon:hover { + background: var(--myfsio-hover-bg); + color: var(--myfsio-text); +} + +.badge { + font-weight: 500; + padding: 0.35em 0.65em; + font-size: 0.8125rem; +} + +.theme-toggle { + min-width: auto; + width: 38px; + height: 32px; + padding: 0; + display: inline-flex; + align-items: center; + justify-content: center; + border-radius: 999px; + transition: all 0.2s ease; +} + +.theme-toggle:hover { + transform: translateY(-1px); +} + +.theme-toggle .theme-icon { + transition: opacity 0.2s ease, transform 0.2s ease; +} + +[data-bs-theme="light"] #themeToggleSun, +[data-bs-theme="light"] #themeToggleSunMobile { + display: none !important; +} +[data-bs-theme="light"] #themeToggleMoon, +[data-bs-theme="light"] #themeToggleMoonMobile { + display: inline-block !important; +} +[data-bs-theme="dark"] #themeToggleSun, +[data-bs-theme="dark"] #themeToggleSunMobile { + display: inline-block !important; +} +[data-bs-theme="dark"] #themeToggleMoon, +[data-bs-theme="dark"] #themeToggleMoonMobile { + display: none !important; +} + +.config-copy { + position: absolute; + top: 0.5rem; + right: 0.5rem; + opacity: 0.8; + transition: opacity 0.2s; + background-color: rgba(0, 0, 0, 0.5); + border: none; + color: white; +} + +.config-copy:hover { + opacity: 1; + background-color: rgba(0, 0, 0, 0.7); + color: white; +} + +.bucket-table td:last-child, +.bucket-table th:last-child { white-space: nowrap; } + +.object-key { + max-width: 0; + width: 100%; + overflow: hidden; + text-overflow: ellipsis; +} + +.object-key .fw-medium { + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.object-key .text-muted { + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.preview-card { top: 1rem; } + +.preview-stage { + background-color: var(--myfsio-preview-bg); + overflow: hidden; + border-color: var(--myfsio-card-border) !important; +} + +.preview-stage:has(#preview-placeholder:not(.d-none)) { + min-height: 0; +} + +.preview-stage:has(#preview-image:not(.d-none)), +.preview-stage:has(#preview-video:not(.d-none)), +.preview-stage:has(#preview-iframe:not(.d-none)) { + min-height: 200px; +} + +#preview-placeholder { + padding: 2rem 1rem; +} + +#preview-text { + padding: 1rem 1.125rem; + max-height: 360px; + overflow: auto; + white-space: pre-wrap; + word-break: break-word; + font-family: 'SFMono-Regular', 'Menlo', 'Consolas', 'Liberation Mono', monospace; + font-size: .8rem; + line-height: 1.6; + tab-size: 4; + color: var(--myfsio-text); + background: transparent; +} + +.upload-progress-stack { + display: flex; + flex-direction: column; + gap: 0.75rem; + max-height: 300px; + overflow-y: auto; +} + +.upload-progress-item { + border: 1px solid var(--myfsio-card-border); + border-radius: 0.75rem; + background-color: var(--myfsio-card-bg); + padding: 0.875rem 1rem; + transition: border-color 0.2s ease, background-color 0.2s ease, box-shadow 0.2s ease; +} + +.upload-progress-item[data-state='uploading'] { + border-color: rgba(59, 130, 246, 0.4); + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.08); +} + +.upload-progress-item[data-state='success'] { + border-color: rgba(34, 197, 94, 0.6); + background-color: rgba(34, 197, 94, 0.04); +} + +.upload-progress-item[data-state='error'] { + border-color: rgba(239, 68, 68, 0.7); + background-color: rgba(239, 68, 68, 0.04); +} + +.upload-progress-item .file-name { + font-weight: 500; + word-break: break-all; + margin-bottom: 0.25rem; +} + +.upload-progress-item .file-size { + font-size: 0.75rem; + color: var(--myfsio-muted); +} + +.upload-progress-item .upload-status { + font-size: 0.8rem; + color: var(--myfsio-muted); +} + +.upload-progress-item .upload-status.success { + color: #16a34a; +} + +.upload-progress-item .upload-status.error { + color: #dc2626; +} + +.upload-progress-item .progress-container { + margin-top: 0.5rem; +} + +.upload-progress-item .progress { + height: 6px; + border-radius: 999px; + overflow: hidden; +} + +.upload-progress-item .progress-bar { + transition: width 0.2s ease; +} + +.upload-progress-item .progress-text { + font-size: 0.7rem; + color: var(--myfsio-muted); + margin-top: 0.25rem; + display: flex; + justify-content: space-between; +} + +.progress-thin { + height: 0.35rem; + background-color: rgba(15, 23, 42, 0.1); +} + +[data-theme='dark'] .progress-thin { + background-color: rgba(248, 250, 252, 0.15); +} + +[data-theme='dark'] .upload-progress-item .upload-status.success { + color: #4ade80; +} + +[data-theme='dark'] .upload-progress-item .upload-status.error { + color: #f87171; +} + +#deleteObjectKey { + word-break: break-all; + max-width: 100%; +} + +.preview-stage img, +.preview-stage video, +.preview-stage iframe { + border: 0; + max-height: 360px; +} + +.upload-dropzone { + border: 2px dashed var(--myfsio-card-border); + border-radius: 0.75rem; + padding: 1.5rem; + cursor: pointer; + transition: border-color 0.2s ease, background-color 0.2s ease; + position: relative; + overflow: hidden; +} + +.upload-dropzone::before { + content: ''; + position: absolute; + inset: 0; + background: linear-gradient(135deg, rgba(59, 130, 246, 0.03) 0%, rgba(139, 92, 246, 0.03) 100%); + opacity: 0; + transition: opacity 0.3s ease; +} + +.upload-dropzone:hover::before, +.upload-dropzone.is-dragover::before { + opacity: 1; +} + +.upload-dropzone.is-dragover { + background-color: rgba(59, 130, 246, 0.08); + border-color: #3b82f6; +} + +.upload-dropzone.upload-locked { + background-color: rgba(59, 130, 246, 0.05); + border-color: #3b82f6; + border-style: dashed; +} + +.upload-dropzone.upload-locked::after { + content: 'Drop more files to add to queue'; + display: block; + margin-top: 0.5rem; + font-size: 0.8rem; + color: #3b82f6; + font-weight: 500; +} + +.metadata-stack .metadata-entry + .metadata-entry { + margin-top: 0.75rem; +} + +.metadata-stack .metadata-key { + text-transform: uppercase; + letter-spacing: 0.05em; + color: var(--myfsio-muted); +} + +.metadata-stack .metadata-value { + font-weight: 600; +} + +.policy-preview { + background-color: var(--myfsio-policy-bg); + color: var(--myfsio-policy-fg); + border-radius: 0.5rem; + padding: 1rem; + font-size: 0.85rem; + max-height: 320px; + overflow: auto; + border: 1px solid var(--myfsio-card-border); +} + +.policy-editor-disabled { + opacity: 0.72; + cursor: not-allowed; +} + +.objects-table-container { + max-height: 600px; + overflow-y: auto; +} + +.objects-table-container thead { + position: sticky; + top: 0; + z-index: 10; +} + +.objects-table-container thead th { + background-color: var(--myfsio-preview-bg); + border-bottom: 2px solid var(--myfsio-card-border); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.04); +} + +[data-theme='dark'] .objects-table-container thead th { + background-color: #1a2234; + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.15); +} + +.btn-group form { display: inline; } + +.font-monospace { font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, monospace; } + +.table { + color: var(--myfsio-text); + background-color: var(--myfsio-card-bg); + border-collapse: separate; + border-spacing: 0; + font-size: 0.9rem; +} + +.table th, +.table td { + border-color: var(--myfsio-card-border); + padding: 0.875rem 1rem; + vertical-align: middle; +} + +.table th { + font-weight: 600; + font-size: 0.75rem; + text-transform: uppercase; + letter-spacing: 0.05em; + color: var(--myfsio-muted); + border-bottom: 2px solid var(--myfsio-card-border); +} + +.table td { + border-bottom: 1px solid rgba(0, 0, 0, 0.05); +} + +[data-theme='dark'] .table td { + border-bottom: 1px solid rgba(255, 255, 255, 0.05); +} + +.table tbody tr:last-child td { + border-bottom: none; +} + +.table-light th { + background-color: var(--myfsio-preview-bg); +} + +[data-theme='dark'] .table-light th { + background-color: rgba(248, 250, 252, 0.03); + color: var(--myfsio-muted); +} + +.table-hover tbody tr { + transition: all 0.15s ease; +} + +.table-hover tbody tr:hover { + background-color: var(--myfsio-hover-bg); + cursor: pointer; +} + +.table-hover tbody tr:hover td { + border-bottom-color: transparent; +} + +.table thead { + background-color: var(--myfsio-preview-bg); + color: var(--myfsio-muted); +} + +[data-theme='dark'] .table thead { + background-color: rgba(248, 250, 252, 0.03); + color: var(--myfsio-muted); +} + +.table-striped tbody tr:nth-of-type(odd) { + background-color: rgba(0, 0, 0, 0.015); +} + +[data-theme='dark'] .table-striped tbody tr:nth-of-type(odd) { + background-color: rgba(255, 255, 255, 0.02); +} + +.form-control, +.form-select { + background-color: var(--myfsio-input-bg); + color: var(--myfsio-text); + border-color: var(--myfsio-input-border); + transition: border-color 0.15s ease, box-shadow 0.15s ease; +} + +.form-control::placeholder { + color: var(--myfsio-muted); + opacity: 0.6; +} + +[data-theme='dark'] .form-control::placeholder { + opacity: 0.5; +} + +.form-control:focus, +.form-select:focus { + background-color: var(--myfsio-input-bg); + color: var(--myfsio-text); + border-color: #3b82f6; + box-shadow: 0 0 0 0.2rem rgba(59, 130, 246, 0.25); +} + +.alert { + color: var(--myfsio-text); + border-color: var(--myfsio-card-border); + border-width: 1px; + border-left-width: 4px; +} + +.alert-success { + background-color: rgba(34, 197, 94, 0.1); + border-left-color: #22c55e; +} + +[data-theme='dark'] .alert-success { + background-color: rgba(34, 197, 94, 0.15); + color: #86efac; +} + +.alert-danger { + background-color: rgba(239, 68, 68, 0.1); + border-left-color: #ef4444; +} + +[data-theme='dark'] .alert-danger { + background-color: rgba(239, 68, 68, 0.15); + color: #fca5a5; +} + +.alert-warning { + background-color: rgba(251, 191, 36, 0.1); + border-left-color: #fbbf24; +} + +[data-theme='dark'] .alert-warning { + background-color: rgba(251, 191, 36, 0.15); + color: #fde047; +} + +.alert-info { + background-color: rgba(59, 130, 246, 0.1); + border-left-color: #3b82f6; +} + +[data-theme='dark'] .alert-info { + background-color: rgba(59, 130, 246, 0.15); + color: #93c5fd; +} + +.btn { + color: inherit; + transition: all 0.2s ease; +} + +.btn:hover { + transform: translateY(-1px); +} + +.btn:active { + transform: translateY(0); +} + +.btn-icon { + width: 36px; + height: 36px; + padding: 0.4rem; + display: inline-flex; + align-items: center; + justify-content: center; +} + +.btn-group-sm .btn-icon { + width: 34px; + height: 34px; +} + +[data-theme='dark'] .btn-outline-secondary { + color: #e2e8f0; + border-color: #475569; +} + +[data-theme='dark'] .btn-outline-secondary:hover { + background-color: rgba(148, 163, 184, 0.2); + border-color: #64748b; + color: #f8fafc; +} + +[data-theme='dark'] .btn-outline-danger { + color: #fca5a5; + border-color: #f87171; +} + +[data-theme='dark'] .btn-outline-danger:hover { + background-color: rgba(248, 113, 113, 0.2); + border-color: #ef4444; + color: #fecaca; +} + +[data-theme='dark'] .btn-outline-primary { + color: #93c5fd; + border-color: #60a5fa; +} + +[data-theme='dark'] .btn-outline-primary:hover { + background-color: rgba(59, 130, 246, 0.2); + border-color: #3b82f6; + color: #bfdbfe; +} + +[data-theme='dark'] .btn-primary { + background-color: #2563eb; + border-color: #1d4ed8; + color: #ffffff; +} + +[data-theme='dark'] .btn-primary:hover { + background-color: #1d4ed8; + border-color: #1e40af; +} + +.btn-primary { + color: #ffffff; + background: linear-gradient(135deg, #3b82f6 0%, #2563eb 100%); + border: none; + box-shadow: 0 2px 4px rgba(59, 130, 246, 0.3); +} + +.btn-primary:hover { + background: linear-gradient(135deg, #2563eb 0%, #1d4ed8 100%); + box-shadow: 0 4px 12px rgba(59, 130, 246, 0.4); +} + +.btn-primary:active { + box-shadow: 0 1px 2px rgba(59, 130, 246, 0.3); +} + +[data-theme='dark'] .btn-danger { + background-color: #dc2626; + border-color: #b91c1c; +} + +[data-theme='dark'] .btn-danger:hover { + background-color: #b91c1c; + border-color: #991b1b; +} + +.badge.text-bg-info { + background-color: #bae6fd; + color: #0f172a; +} + +[data-theme='dark'] .badge.text-bg-info { + background-color: #0ea5e9; + color: #e2e8f0; +} + +[data-theme='dark'] .badge.text-bg-warning { + background-color: #fde047; + color: #0f172a; +} + +[data-theme='dark'] .badge.text-bg-secondary { + background-color: #475569; + color: #e2e8f0; +} + +[data-theme='dark'] .badge.text-bg-success { + background-color: #22c55e; + color: #ffffff; +} + +[data-theme='dark'] .badge.text-bg-primary { + background-color: #3b82f6; + color: #ffffff; +} + +.dropdown-menu { + border-color: var(--myfsio-card-border); +} + +[data-theme='dark'] .form-label, +[data-theme='dark'] label, +[data-theme='dark'] .modal-title, +[data-theme='dark'] .fw-semibold { + color: var(--myfsio-text); +} + +.modal-backdrop.show { + opacity: 0.6; +} + +[data-theme='dark'] .btn-close { + filter: invert(1) grayscale(100%) brightness(200%); +} + +[data-theme='dark'] .border { + border-color: var(--myfsio-card-border) !important; +} + +.btn-link { + color: #3b82f6; + text-decoration: none; +} + +.btn-link:hover { + color: #2563eb; + text-decoration: underline; + transform: none; +} + +[data-theme='dark'] .btn-link { + color: #60a5fa; +} + +[data-theme='dark'] .btn-link:hover { + color: #93c5fd; +} + +[data-theme='dark'] .input-group-text { + background-color: var(--myfsio-input-bg); + color: var(--myfsio-text); + border-color: var(--myfsio-input-border); +} + +.page-header { + margin-bottom: 2rem; +} + +.page-header h1 { + font-weight: 600; + letter-spacing: -0.02em; +} + +@keyframes pulse { + 0%, 100% { opacity: 1; } + 50% { opacity: 0.5; } +} + +.loading { + animation: pulse 2s cubic-bezier(0.4, 0, 0.6, 1) infinite; +} + +[data-theme='dark'] .text-primary { + color: #60a5fa !important; +} + +[data-theme='dark'] .text-success { + color: #86efac !important; +} + +[data-theme='dark'] .text-danger { + color: #fca5a5 !important; +} + +[data-theme='dark'] .text-warning { + color: #fde047 !important; +} + +[data-theme='dark'] .lead { + color: var(--myfsio-muted); +} + +.btn-sm { + transition: all 0.15s ease; +} + +[data-theme='dark'] .btn-outline-light { + color: #f8fafc; + border-color: rgba(248, 250, 252, 0.3); +} + +[data-theme='dark'] .btn-outline-light:hover { + background-color: rgba(248, 250, 252, 0.1); + border-color: rgba(248, 250, 252, 0.5); +} + +pre { + background-color: rgba(15, 23, 42, 0.05); + border: 1px solid var(--myfsio-card-border); + border-radius: 0.5rem; + padding: 1rem; + overflow-x: auto; + font-size: 0.875rem; + line-height: 1.6; +} + +[data-theme='dark'] pre { + background-color: #111827; + border-color: rgba(148, 163, 184, 0.24); + color: #e5eefb; +} + +pre code { + background: none; + padding: 0; + color: inherit; +} + +[data-theme='dark'] .docs-section .bg-light { + background-color: #182235 !important; + border: 1px solid rgba(148, 163, 184, 0.18); + color: #e5eefb; +} + +[data-theme='dark'] .docs-section .bg-light .text-muted { + color: #a9b6c8 !important; +} + +.docs-section + .docs-section { + margin-top: 1.25rem; +} + +.breadcrumb { + background-color: transparent; + padding: 0.5rem 0; + font-size: 0.9rem; +} + +.breadcrumb-item + .breadcrumb-item::before { + content: "›"; + color: var(--myfsio-muted); +} + +.breadcrumb-item a { + color: var(--myfsio-text); + text-decoration: none; + transition: color 0.2s ease; +} + +.breadcrumb-item a:hover { + color: #3b82f6; + text-decoration: underline; +} + +[data-theme='dark'] .breadcrumb-item a:hover { + color: #60a5fa; +} + +.breadcrumb-item.active { + color: var(--myfsio-muted); +} + +.bi { + vertical-align: -0.125em; +} + +.sticky-top { + top: 1.5rem; +} + +.card-body dl:last-child { + margin-bottom: 0; +} + +.text-center svg { + display: inline-block; +} + +[data-theme='dark'] .input-group .btn-outline-primary { + background-color: transparent; +} + +.text-nowrap { + white-space: nowrap; +} + +.alert svg { + flex-shrink: 0; +} + +[data-object-row]:hover { + background-color: var(--myfsio-hover-bg) !important; +} + +.folder-row { + background-color: var(--myfsio-section-bg); + transition: background-color 0.15s ease; +} + +.folder-row:hover { + background-color: var(--myfsio-hover-bg) !important; +} + +.folder-row td:first-child { + padding-left: 0.5rem; +} + +.btn-group-sm .btn { + padding: 0.25rem 0.6rem; + font-size: 0.875rem; +} + +.modal-content { + border: none; + border-radius: 1rem; + box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); + overflow: hidden; +} + +.modal-header { + background-color: var(--myfsio-card-bg); + border-bottom: 1px solid var(--myfsio-card-border); + padding: 1.25rem 1.5rem; +} + +.modal-header.border-0 { + border-bottom: none; +} + +.modal-title { + font-weight: 600; + display: flex; + align-items: center; + gap: 0.5rem; +} + +.modal-body { + padding: 1.5rem; + overflow-wrap: break-word; + word-wrap: break-word; + word-break: break-word; +} + +.modal-footer { + background-color: var(--myfsio-preview-bg); + border-top: 1px solid var(--myfsio-card-border); + padding: 1rem 1.5rem; + gap: 0.75rem; +} + +.modal-footer.border-0 { + border-top: none; + background-color: var(--myfsio-card-bg); +} + +[data-theme='dark'] .modal-footer { + background-color: rgba(0, 0, 0, 0.2); +} + +[data-theme='dark'] .modal-footer.border-0 { + background-color: var(--myfsio-card-bg); +} + +.modal-icon { + width: 48px; + height: 48px; + border-radius: 50%; + display: flex; + align-items: center; + justify-content: center; + margin: 0 auto 1rem; +} + +.modal-icon-danger { + background: rgba(239, 68, 68, 0.1); + color: #ef4444; +} + +.modal-icon-warning { + background: rgba(251, 191, 36, 0.1); + color: #f59e0b; +} + +.modal-icon-success { + background: rgba(34, 197, 94, 0.1); + color: #22c55e; +} + +.modal-icon-info { + background: rgba(59, 130, 246, 0.1); + color: #3b82f6; +} + +[data-theme='dark'] .modal-icon-danger { + background: rgba(239, 68, 68, 0.2); + color: #f87171; +} + +[data-theme='dark'] .modal-icon-warning { + background: rgba(251, 191, 36, 0.2); + color: #fbbf24; +} + +[data-theme='dark'] .modal-icon-success { + background: rgba(34, 197, 94, 0.2); + color: #4ade80; +} + +[data-theme='dark'] .modal-icon-info { + background: rgba(59, 130, 246, 0.2); + color: #60a5fa; +} + +.modal .alert { + border-radius: 0.75rem; +} + +.modal .form-control, +.modal .form-select { + border-radius: 0.5rem; +} + +.modal .btn { + border-radius: 0.5rem; + padding: 0.5rem 1rem; + font-weight: 500; +} + +.modal .btn-sm { + padding: 0.375rem 0.75rem; +} + +.modal .list-group { + border-radius: 0.75rem; + overflow: hidden; +} + +.modal .list-group-item { + border-color: var(--myfsio-card-border); +} + +.modal-backdrop { + backdrop-filter: blur(4px); + -webkit-backdrop-filter: blur(4px); +} + +.user-avatar { + width: 40px; + height: 40px; + border-radius: 50%; + background: linear-gradient(135deg, #3b82f6, #8b5cf6); + display: flex; + align-items: center; + justify-content: center; + color: white; + flex-shrink: 0; +} + +.connection-icon { + width: 32px; + height: 32px; + border-radius: 0.5rem; + background: var(--myfsio-preview-bg); + display: flex; + align-items: center; + justify-content: center; + color: var(--myfsio-muted); + flex-shrink: 0; +} + +[data-theme='dark'] .connection-icon { + background: rgba(255, 255, 255, 0.1); +} + +.bucket-card { + position: relative; + transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1); + border: 1px solid var(--myfsio-card-border) !important; + overflow: hidden; + border-radius: 1rem !important; +} + +.bucket-card::before { + content: ''; + position: absolute; + top: 0; + left: 0; + right: 0; + height: 4px; + background: linear-gradient(90deg, #3b82f6, #8b5cf6); + opacity: 0; + transition: opacity 0.2s ease; +} + +.bucket-card:hover { + transform: translateY(-2px); + box-shadow: 0 8px 24px -4px rgba(0, 0, 0, 0.12), 0 4px 8px -4px rgba(0, 0, 0, 0.08); + border-color: var(--myfsio-accent) !important; +} + +.bucket-card:hover::before { + opacity: 1; +} + +[data-theme='dark'] .bucket-card:hover { + box-shadow: 0 8px 24px -4px rgba(0, 0, 0, 0.4), 0 4px 8px -4px rgba(0, 0, 0, 0.3); +} + +.bucket-card .card-body { + padding: 1.25rem 1.5rem; +} + +.bucket-card .card-footer { + padding: 0.75rem 1.5rem; +} + +.bucket-card .bucket-icon { + width: 44px; + height: 44px; + display: flex; + align-items: center; + justify-content: center; + border-radius: 12px; + background: linear-gradient(135deg, rgba(59, 130, 246, 0.15) 0%, rgba(139, 92, 246, 0.15) 100%); + color: var(--myfsio-accent); + transition: transform 0.2s ease; +} + +.bucket-card:hover .bucket-icon { + transform: scale(1.05); +} + +[data-theme='dark'] .bucket-card .bucket-icon { + background: linear-gradient(135deg, rgba(59, 130, 246, 0.25) 0%, rgba(139, 92, 246, 0.25) 100%); +} + +.bucket-card .bucket-stats { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 1rem; + padding: 1rem; + margin-top: 1rem; + background: var(--myfsio-preview-bg); + border-radius: 0.75rem; +} + +.bucket-card .bucket-stat { + text-align: center; +} + +.bucket-card .bucket-stat-value { + font-size: 1.25rem; + font-weight: 700; + color: var(--myfsio-text); + line-height: 1.2; +} + +.bucket-card .bucket-stat-label { + font-size: 0.75rem; + text-transform: uppercase; + letter-spacing: 0.05em; + color: var(--myfsio-muted); + margin-top: 0.25rem; +} + +.bucket-card .bucket-name { + font-size: 1.1rem; + font-weight: 600; + color: var(--myfsio-text); + margin: 0; + line-height: 1.3; +} + +.bucket-card .bucket-access-badge { + font-size: 0.7rem; + padding: 0.35em 0.75em; + border-radius: 999px; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.03em; +} + +.form-control:focus, +.form-select:focus, +.btn:focus-visible { + outline: none; + box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.3), 0 0 0 1px rgba(59, 130, 246, 0.5); +} + +@keyframes shimmer { + 0% { background-position: -200% 0; } + 100% { background-position: 200% 0; } +} + +.skeleton { + background: linear-gradient(90deg, var(--myfsio-card-bg) 25%, var(--myfsio-hover-bg) 50%, var(--myfsio-card-bg) 75%); + background-size: 200% 100%; + animation: shimmer 1.5s infinite; + border-radius: 4px; +} + +.toast-container { + position: fixed; + bottom: 1.5rem; + right: 1.5rem; + z-index: 1100; + display: flex; + flex-direction: column; + gap: 0.75rem; +} + +.toast-item { + background: var(--myfsio-card-bg); + border: 1px solid var(--myfsio-card-border); + border-radius: 0.75rem; + padding: 1rem 1.25rem; + box-shadow: 0 10px 25px -5px rgba(0, 0, 0, 0.1), 0 8px 10px -6px rgba(0, 0, 0, 0.1); + display: flex; + align-items: center; + gap: 0.75rem; + animation: slideInRight 0.3s ease-out; + max-width: 360px; +} + +@keyframes slideInRight { + from { opacity: 0; transform: translateX(100%); } + to { opacity: 1; transform: translateX(0); } +} + +.empty-state { + padding: 3rem 2rem; + text-align: center; +} + +.empty-state-icon { + width: 80px; + height: 80px; + margin: 0 auto 1.5rem; + display: flex; + align-items: center; + justify-content: center; + background: linear-gradient(135deg, rgba(59, 130, 246, 0.1) 0%, rgba(139, 92, 246, 0.1) 100%); + border-radius: 50%; + color: #3b82f6; +} + +[data-theme='dark'] .empty-state-icon { + background: linear-gradient(135deg, rgba(59, 130, 246, 0.2) 0%, rgba(139, 92, 246, 0.2) 100%); + color: #60a5fa; +} + +.metric-card { + position: relative; + overflow: hidden; +} + +.metric-card::after { + content: ''; + position: absolute; + top: 0; + right: 0; + width: 120px; + height: 120px; + background: linear-gradient(135deg, transparent 40%, rgba(59, 130, 246, 0.05) 100%); + border-radius: 50%; + transform: translate(30%, -30%); +} + +[data-theme='dark'] .metric-card::after { + background: linear-gradient(135deg, transparent 40%, rgba(59, 130, 246, 0.1) 100%); +} + +.progress { + overflow: visible; + border-radius: 999px; +} + +.progress-bar { + border-radius: 999px; + position: relative; + transition: width 0.6s ease; +} + +.icon-box { + transition: all 0.2s ease; +} + +.card:hover .icon-box { + transform: scale(1.1) rotate(5deg); +} + +.search-input-wrapper { + position: relative; +} + +.search-input-wrapper .form-control { + padding-left: 2.75rem; + border-radius: 999px; + transition: all 0.2s ease; +} + +.search-input-wrapper .form-control:focus { + padding-left: 2.75rem; +} + +.search-input-wrapper .search-icon { + position: absolute; + left: 1rem; + top: 50%; + transform: translateY(-50%); + color: var(--myfsio-muted); + pointer-events: none; + transition: color 0.2s ease; +} + +.search-input-wrapper:focus-within .search-icon { + color: #3b82f6; +} + +.nav-tabs { + border-bottom: 2px solid var(--myfsio-card-border); +} + +.nav-tabs .nav-link { + border: none; + color: var(--myfsio-muted); + padding: 0.75rem 1.25rem; + font-weight: 500; + position: relative; + transition: all 0.2s ease; +} + +.nav-tabs .nav-link::after { + content: ''; + position: absolute; + bottom: -2px; + left: 0; + right: 0; + height: 2px; + background: linear-gradient(90deg, #3b82f6, #8b5cf6); + transform: scaleX(0); + transition: transform 0.2s ease; +} + +.nav-tabs .nav-link:hover { + color: var(--myfsio-text); + border: none; +} + +.nav-tabs .nav-link.active { + background: transparent; + color: #3b82f6; + border: none; +} + +.nav-tabs .nav-link.active::after { + transform: scaleX(1); +} + +[data-theme='dark'] .nav-tabs .nav-link.active { + color: #60a5fa; +} + +.file-type-badge { + display: inline-flex; + align-items: center; + padding: 0.25rem 0.5rem; + border-radius: 4px; + font-size: 0.7rem; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.05em; +} + +.file-type-image { background: rgba(139, 92, 246, 0.15); color: #7c3aed; } +.file-type-video { background: rgba(236, 72, 153, 0.15); color: #db2777; } +.file-type-audio { background: rgba(245, 158, 11, 0.15); color: #d97706; } +.file-type-document { background: rgba(59, 130, 246, 0.15); color: #2563eb; } +.file-type-archive { background: rgba(34, 197, 94, 0.15); color: #16a34a; } +.file-type-code { background: rgba(99, 102, 241, 0.15); color: #4f46e5; } + +[data-theme='dark'] .file-type-image { background: rgba(139, 92, 246, 0.25); color: #a78bfa; } +[data-theme='dark'] .file-type-video { background: rgba(236, 72, 153, 0.25); color: #f472b6; } +[data-theme='dark'] .file-type-audio { background: rgba(245, 158, 11, 0.25); color: #fbbf24; } +[data-theme='dark'] .file-type-document { background: rgba(59, 130, 246, 0.25); color: #60a5fa; } +[data-theme='dark'] .file-type-archive { background: rgba(34, 197, 94, 0.25); color: #4ade80; } +[data-theme='dark'] .file-type-code { background: rgba(99, 102, 241, 0.25); color: #818cf8; } + +.table-hover [data-object-row] .btn-group { + opacity: 0.5; + transition: opacity 0.2s ease; +} + +.table-hover [data-object-row]:hover .btn-group { + opacity: 1; +} + +::-webkit-scrollbar { + width: 8px; + height: 8px; +} + +::-webkit-scrollbar-track { + background: var(--myfsio-body-bg); + border-radius: 4px; +} + +::-webkit-scrollbar-thumb { + background: var(--myfsio-muted); + border-radius: 4px; + opacity: 0.5; +} + +::-webkit-scrollbar-thumb:hover { + background: var(--myfsio-text); +} + +.modal.fade .modal-dialog { + transform: scale(0.95) translateY(-20px); + transition: transform 0.2s ease-out; +} + +.modal.show .modal-dialog { + transform: scale(1) translateY(0); +} + +.tooltip-inner { + background: var(--myfsio-policy-bg); + padding: 0.5rem 0.75rem; + border-radius: 6px; + font-size: 0.8125rem; +} + +@keyframes countUp { + from { opacity: 0; transform: translateY(10px); } + to { opacity: 1; transform: translateY(0); } +} + +.stat-value { + animation: countUp 0.5s ease-out; +} + +.action-btn-group { + display: flex; + gap: 0.5rem; +} + +.action-btn-group .btn { + border-radius: 8px; +} + +@keyframes livePulse { + 0%, 100% { opacity: 1; } + 50% { opacity: 0.4; } +} + +.live-indicator { + width: 8px; + height: 8px; + background: #22c55e; + border-radius: 50%; + animation: livePulse 2s infinite; +} + +.login-card { + border: none; + border-radius: 1rem; + overflow: hidden; +} + +.login-card::before { + content: ''; + position: absolute; + top: 0; + left: 0; + right: 0; + height: 4px; + background: linear-gradient(90deg, #3b82f6, #8b5cf6, #ec4899); +} + +@media (max-width: 768px) { + .bucket-card:hover { + transform: none; + } + + .objects-table-container { + max-height: 60vh; + } + + .preview-card { + position: relative !important; + top: 0 !important; + } + + .card-body .table-responsive { + margin: -1rem; + padding: 0; + width: calc(100% + 2rem); + } + + .card-body .table-responsive table { + margin-bottom: 0; + } + + .table th, + .table td { + padding: 0.5rem 0.75rem; + } + + .table-responsive::after { + content: ''; + position: absolute; + top: 0; + right: 0; + bottom: 0; + width: 20px; + background: linear-gradient(to left, var(--myfsio-card-bg), transparent); + pointer-events: none; + opacity: 0; + transition: opacity 0.3s; + } + + .table-responsive:not(:hover)::after { + opacity: 0.8; + } +} + +*:focus-visible { + outline: 2px solid #3b82f6; + outline-offset: 2px; +} + +* { + transition-property: background-color, border-color, color, fill, stroke; + transition-duration: 0s; + transition-timing-function: ease; +} + +body.theme-transitioning, +body.theme-transitioning * { + transition-duration: 0.3s !important; +} + +.status-badge { + display: inline-flex; + align-items: center; + gap: 0.375rem; +} + +.status-badge-dot { + width: 6px; + height: 6px; + border-radius: 50%; +} + +.status-badge-success .status-badge-dot { background: #22c55e; } +.status-badge-warning .status-badge-dot { background: #f59e0b; } +.status-badge-danger .status-badge-dot { background: #ef4444; } +.status-badge-info .status-badge-dot { background: #3b82f6; } + +.bucket-list-item { + display: flex; + align-items: center; + padding: 1rem 1.25rem; + background: var(--myfsio-card-bg); + border: 1px solid var(--myfsio-card-border); + border-radius: 0.75rem; + transition: all 0.2s ease; + gap: 1rem; +} + +.bucket-list-item:hover { + border-color: rgba(59, 130, 246, 0.3); + background: var(--myfsio-hover-bg); +} + +.text-gradient { + background: linear-gradient(135deg, #3b82f6, #8b5cf6); + -webkit-background-clip: text; + -webkit-text-fill-color: transparent; + background-clip: text; +} + +.shadow-glow { + box-shadow: 0 0 20px rgba(59, 130, 246, 0.3); +} + +.border-gradient { + border: 2px solid transparent; + background: linear-gradient(var(--myfsio-card-bg), var(--myfsio-card-bg)) padding-box, linear-gradient(135deg, #3b82f6, #8b5cf6) border-box; +} + +#objects-table .dropdown-menu { + position: fixed !important; + z-index: 1050; +} + +.btn-icon.dropdown-toggle::after { + display: none; +} + +.floating-upload-progress { + position: fixed; + bottom: 1.5rem; + right: 1.5rem; + z-index: 1055; + min-width: 320px; + max-width: 400px; + animation: slideInUp 0.3s ease-out; +} + +@keyframes slideInUp { + from { opacity: 0; transform: translateY(20px); } + to { opacity: 1; transform: translateY(0); } +} + +.floating-upload-content { + background: var(--myfsio-card-bg); + border: 1px solid var(--myfsio-card-border); + border-radius: 0.75rem; + padding: 1rem; + box-shadow: 0 10px 25px -5px rgba(0, 0, 0, 0.15), 0 8px 10px -6px rgba(0, 0, 0, 0.1); +} + +[data-theme='dark'] .floating-upload-content { + box-shadow: 0 10px 25px -5px rgba(0, 0, 0, 0.4), 0 8px 10px -6px rgba(0, 0, 0, 0.3); +} + +.objects-header-responsive { + display: flex; + flex-wrap: wrap; + gap: 0.5rem; + align-items: center; +} + +.objects-header-responsive > .header-title { + flex: 0 0 auto; +} + +.objects-header-responsive > .header-actions { + display: flex; + flex-wrap: wrap; + gap: 0.5rem; + align-items: center; + flex: 1; + justify-content: flex-end; +} + +@media (max-width: 640px) { + .objects-header-responsive { + flex-direction: column; + align-items: stretch; + } + + .objects-header-responsive > .header-title { + margin-bottom: 0.5rem; + } + + .objects-header-responsive > .header-actions { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 0.5rem; + } + + .objects-header-responsive > .header-actions .btn { + justify-content: center; + } + + .objects-header-responsive > .header-actions .search-wrapper { + grid-column: span 2; + } + + .objects-header-responsive > .header-actions .search-wrapper input { + max-width: 100% !important; + width: 100%; + } + + .objects-header-responsive > .header-actions .bulk-actions { + grid-column: span 2; + display: flex; + gap: 0.5rem; + } + + .objects-header-responsive > .header-actions .bulk-actions .btn { + flex: 1; + } +} + +.modal { + z-index: 1055; +} + +.modal-backdrop { + z-index: 1050; +} + +@media (min-width: 992px) { + .toast-container { + right: 1.5rem; + } +} + +@media (min-width: 992px) { + .floating-upload-progress { + right: 1.5rem; + } +} + +.sidebar-body::-webkit-scrollbar { + width: 4px; +} + +.sidebar-body::-webkit-scrollbar-track { + background: transparent; +} + +.sidebar-body::-webkit-scrollbar-thumb { + background: rgba(255, 255, 255, 0.2); + border-radius: 2px; +} + +.sidebar-body::-webkit-scrollbar-thumb:hover { + background: rgba(255, 255, 255, 0.3); +} + +@keyframes sidebarLinkPulse { + 0%, 100% { box-shadow: 0 0 0 0 rgba(59, 130, 246, 0.4); } + 50% { box-shadow: 0 0 0 4px rgba(59, 130, 246, 0); } +} + +.sidebar-link.active { + animation: sidebarLinkPulse 2s ease-in-out infinite; +} + +.offcanvas-backdrop.show { + backdrop-filter: blur(4px); + -webkit-backdrop-filter: blur(4px); +} + +body:has(.login-card) .sidebar, +body:has(.login-card) .mobile-header { + display: none !important; +} + +body:has(.login-card) .main-wrapper { + margin-left: 0 !important; + padding-top: 0 !important; +} + +.context-menu { + position: fixed; + z-index: 1060; + min-width: 180px; + background: var(--myfsio-card-bg); + border: 1px solid var(--myfsio-card-border); + border-radius: 0.5rem; + box-shadow: 0 10px 25px -5px rgba(0, 0, 0, 0.15), 0 8px 10px -6px rgba(0, 0, 0, 0.1); + padding: 0.25rem 0; + font-size: 0.875rem; +} + +[data-theme='dark'] .context-menu { + box-shadow: 0 10px 25px -5px rgba(0, 0, 0, 0.4), 0 8px 10px -6px rgba(0, 0, 0, 0.3); +} + +.context-menu-item { + display: flex; + align-items: center; + gap: 0.625rem; + padding: 0.5rem 0.875rem; + color: var(--myfsio-text); + cursor: pointer; + transition: background-color 0.1s ease; + border: none; + background: none; + width: 100%; + text-align: left; + font-size: inherit; +} + +.context-menu-item:hover { + background-color: var(--myfsio-hover-bg); +} + +.context-menu-item.text-danger:hover { + background-color: rgba(239, 68, 68, 0.1); +} + +.context-menu-divider { + height: 1px; + background: var(--myfsio-card-border); + margin: 0.25rem 0; +} + +.context-menu-shortcut { + margin-left: auto; + font-size: 0.75rem; + color: var(--myfsio-muted); +} + +.kbd-shortcuts-list { + display: flex; + flex-direction: column; + gap: 0.5rem; +} + +.kbd-shortcuts-list .shortcut-row { + display: flex; + align-items: center; + justify-content: space-between; + padding: 0.375rem 0; +} + +.kbd-shortcuts-list kbd { + display: inline-flex; + align-items: center; + justify-content: center; + min-width: 1.75rem; + padding: 0.2rem 0.5rem; + font-family: inherit; + font-size: 0.75rem; + font-weight: 600; + background: var(--myfsio-preview-bg); + border: 1px solid var(--myfsio-card-border); + border-radius: 0.25rem; + box-shadow: 0 1px 0 1px rgba(0, 0, 0, 0.05); + color: var(--myfsio-text); +} + +[data-theme='dark'] .kbd-shortcuts-list kbd { + background: rgba(255, 255, 255, 0.1); + box-shadow: 0 1px 0 1px rgba(0, 0, 0, 0.2); +} + +.sort-dropdown .dropdown-item.active, +.sort-dropdown .dropdown-item:active { + background-color: var(--myfsio-hover-bg); + color: var(--myfsio-text); +} + +.sort-dropdown .dropdown-item { + font-size: 0.875rem; + padding: 0.375rem 1rem; +} + +.tag-pill { + display: inline-flex; + border-radius: 9999px; + border: 1px solid var(--myfsio-tag-border); + overflow: hidden; + font-size: 0.75rem; + line-height: 1; +} + +.tag-pill-key { + padding: 0.3rem 0.5rem; + background: var(--myfsio-tag-key-bg); + color: var(--myfsio-tag-key-text); + font-weight: 600; +} + +.tag-pill-value { + padding: 0.3rem 0.5rem; + background: var(--myfsio-tag-value-bg); + color: var(--myfsio-tag-value-text); + font-weight: 400; +} + +.tag-editor-card { + background: var(--myfsio-preview-bg); + border-radius: 0.5rem; + padding: 0.75rem; +} + +.tag-editor-header, +.tag-editor-row { + display: grid; + grid-template-columns: 1fr 1fr 28px; + gap: 0.5rem; + align-items: center; +} + +.tag-editor-header { + padding-bottom: 0.375rem; + border-bottom: 1px solid var(--myfsio-card-border); + margin-bottom: 0.5rem; +} + +.tag-editor-header span { + font-size: 0.7rem; + font-weight: 600; + text-transform: uppercase; + color: var(--myfsio-muted); + letter-spacing: 0.05em; +} + +.tag-editor-row { + margin-bottom: 0.375rem; +} + +.tag-editor-delete { + display: inline-flex; + align-items: center; + justify-content: center; + width: 28px; + height: 28px; + border: none; + background: transparent; + color: var(--myfsio-muted); + border-radius: 0.375rem; + cursor: pointer; + transition: color 0.15s, background 0.15s; +} + +.tag-editor-delete:hover { + color: var(--myfsio-tag-delete-hover); + background: rgba(239, 68, 68, 0.1); +} + +.tag-editor-actions { + display: flex; + align-items: center; + gap: 0.5rem; + margin-top: 0.75rem; + padding-top: 0.5rem; + border-top: 1px solid var(--myfsio-card-border); +} + +@media (prefers-reduced-motion: reduce) { + *, + *::before, + *::after { + animation-duration: 0.01ms !important; + animation-iteration-count: 1 !important; + transition-duration: 0.01ms !important; + } +} + +@media print { + .sidebar, + .mobile-header { + display: none !important; + } + + .main-wrapper { + margin-left: 0 !important; + padding-top: 0 !important; + } +} diff --git a/rust/myfsio-engine/crates/myfsio-server/static/images/MyFSIO.ico b/rust/myfsio-engine/crates/myfsio-server/static/images/MyFSIO.ico new file mode 100644 index 0000000..58ea56f Binary files /dev/null and b/rust/myfsio-engine/crates/myfsio-server/static/images/MyFSIO.ico differ diff --git a/rust/myfsio-engine/crates/myfsio-server/static/images/MyFSIO.png b/rust/myfsio-engine/crates/myfsio-server/static/images/MyFSIO.png new file mode 100644 index 0000000..e425feb Binary files /dev/null and b/rust/myfsio-engine/crates/myfsio-server/static/images/MyFSIO.png differ diff --git a/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-main.js b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-main.js new file mode 100644 index 0000000..75040f5 --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-main.js @@ -0,0 +1,4827 @@ +(function () { + 'use strict'; + + const { formatBytes, escapeHtml, fallbackCopy, setupJsonAutoIndent } = window.BucketDetailUtils || { + formatBytes: (bytes) => { + if (!Number.isFinite(bytes)) return `${bytes} bytes`; + const units = ['bytes', 'KB', 'MB', 'GB', 'TB']; + let i = 0; + let size = bytes; + while (size >= 1024 && i < units.length - 1) { + size /= 1024; + i++; + } + return `${size.toFixed(i === 0 ? 0 : 1)} ${units[i]}`; + }, + escapeHtml: (value) => { + if (value === null || value === undefined) return ''; + return String(value) + .replace(/&/g, '&') + .replace(//g, '>') + .replace(/"/g, '"') + .replace(/'/g, '''); + }, + fallbackCopy: () => false, + setupJsonAutoIndent: () => { } + }; + + setupJsonAutoIndent(document.getElementById('policyDocument')); + + const getFileTypeIcon = (key) => { + const ext = (key.split('.').pop() || '').toLowerCase(); + const iconMap = { + image: ['jpg', 'jpeg', 'png', 'gif', 'svg', 'webp', 'ico', 'bmp', 'tiff', 'tif'], + document: ['pdf', 'doc', 'docx', 'txt', 'rtf', 'odt', 'pages'], + spreadsheet: ['xls', 'xlsx', 'csv', 'ods', 'numbers'], + archive: ['zip', 'rar', '7z', 'tar', 'gz', 'bz2', 'xz', 'tgz'], + code: ['js', 'ts', 'jsx', 'tsx', 'py', 'java', 'cpp', 'c', 'h', 'hpp', 'cs', 'go', 'rs', 'rb', 'php', 'html', 'htm', 'css', 'scss', 'sass', 'less', 'json', 'xml', 'yaml', 'yml', 'md', 'sh', 'bat', 'ps1', 'sql'], + audio: ['mp3', 'wav', 'flac', 'ogg', 'aac', 'm4a', 'wma', 'aiff'], + video: ['mp4', 'avi', 'mov', 'mkv', 'webm', 'wmv', 'flv', 'm4v', 'mpeg', 'mpg'], + }; + const icons = { + image: ` + + + `, + document: ` + + + `, + spreadsheet: ` + + `, + archive: ` + + + `, + code: ` + + + `, + audio: ` + + + + `, + video: ` + + `, + default: ` + + `, + }; + for (const [type, extensions] of Object.entries(iconMap)) { + if (extensions.includes(ext)) { + return icons[type]; + } + } + return icons.default; + }; + + const selectAllCheckbox = document.querySelector('[data-select-all]'); + const bulkDeleteButton = document.querySelector('[data-bulk-delete-trigger]'); + const bulkDeleteLabel = bulkDeleteButton?.querySelector('[data-bulk-delete-label]'); + const bulkDeleteModalEl = document.getElementById('bulkDeleteModal'); + const bulkDeleteModal = bulkDeleteModalEl ? new bootstrap.Modal(bulkDeleteModalEl) : null; + const bulkDeleteList = document.getElementById('bulkDeleteList'); + const bulkDeleteCount = document.getElementById('bulkDeleteCount'); + const bulkDeleteStatus = document.getElementById('bulkDeleteStatus'); + const bulkDeleteConfirm = document.getElementById('bulkDeleteConfirm'); + const bulkDeletePurge = document.getElementById('bulkDeletePurge'); + const previewPanel = document.getElementById('preview-panel'); + const previewEmpty = document.getElementById('preview-empty'); + const previewKey = document.getElementById('preview-key'); + const previewSize = document.getElementById('preview-size'); + const previewModified = document.getElementById('preview-modified'); + const previewEtag = document.getElementById('preview-etag'); + const previewMetadata = document.getElementById('preview-metadata'); + const previewMetadataList = document.getElementById('preview-metadata-list'); + const previewPlaceholder = document.getElementById('preview-placeholder'); + const previewPlaceholderDefault = previewPlaceholder ? previewPlaceholder.innerHTML : ''; + const previewErrorAlert = document.getElementById('preview-error-alert'); + const previewDetailsMeta = document.getElementById('preview-details-meta'); + const previewImage = document.getElementById('preview-image'); + const previewVideo = document.getElementById('preview-video'); + const previewAudio = document.getElementById('preview-audio'); + const previewText = document.getElementById('preview-text'); + const previewIframe = document.getElementById('preview-iframe'); + const downloadButton = document.getElementById('downloadButton'); + const presignButton = document.getElementById('presignButton'); + const presignModalEl = document.getElementById('presignModal'); + const presignModal = presignModalEl ? new bootstrap.Modal(presignModalEl) : null; + const presignMethod = document.getElementById('presignMethod'); + const presignTtl = document.getElementById('presignTtl'); + const presignLink = document.getElementById('presignLink'); + const copyPresignLink = document.getElementById('copyPresignLink'); + const copyPresignDefaultLabel = copyPresignLink?.textContent?.trim() || 'Copy'; + const generatePresignButton = document.getElementById('generatePresignButton'); + const policyForm = document.getElementById('bucketPolicyForm'); + const policyTextarea = document.getElementById('policyDocument'); + const policyPreset = document.getElementById('policyPreset'); + const policyMode = document.getElementById('policyMode'); + const uploadForm = document.querySelector('[data-upload-form]'); + const uploadModalEl = document.getElementById('uploadModal'); + const uploadModal = uploadModalEl ? bootstrap.Modal.getOrCreateInstance(uploadModalEl) : null; + const uploadFileInput = uploadForm?.querySelector('input[name="object"]'); + const uploadDropZone = uploadForm?.querySelector('[data-dropzone]'); + const uploadDropZoneLabel = uploadDropZone?.querySelector('[data-dropzone-label]'); + const messageModalEl = document.getElementById('messageModal'); + const messageModal = messageModalEl ? new bootstrap.Modal(messageModalEl) : null; + const messageModalTitle = document.getElementById('messageModalTitle'); + const messageModalBody = document.getElementById('messageModalBody'); + const messageModalAction = document.getElementById('messageModalAction'); + let messageModalActionHandler = null; + let isGeneratingPresign = false; + const objectsContainer = document.querySelector('.objects-table-container[data-bucket]'); + const bulkDeleteEndpoint = objectsContainer?.dataset.bulkDeleteEndpoint || ''; + const objectsApiUrl = objectsContainer?.dataset.objectsApi || ''; + const objectsStreamUrl = objectsContainer?.dataset.objectsStream || ''; + const versionPanel = document.getElementById('version-panel'); + const versionList = document.getElementById('version-list'); + const refreshVersionsButton = document.getElementById('refreshVersionsButton'); + let archivedCard = document.getElementById('archived-objects-card'); + let archivedBody = archivedCard?.querySelector('[data-archived-body]'); + let archivedCountBadge = archivedCard?.querySelector('[data-archived-count]'); + let archivedRefreshButton = archivedCard?.querySelector('[data-archived-refresh]'); + let archivedEndpoint = archivedCard?.dataset.archivedEndpoint; + let versioningEnabled = objectsContainer?.dataset.versioning === 'true'; + const versionsCache = new Map(); + let activeRow = null; + const selectedRows = new Map(); + let bulkDeleting = false; + if (presignButton) presignButton.disabled = true; + if (generatePresignButton) generatePresignButton.disabled = true; + if (downloadButton) downloadButton.classList.add('disabled'); + + const objectCountBadge = document.getElementById('object-count-badge'); + const loadMoreContainer = document.getElementById('load-more-container'); + const loadMoreSpinner = document.getElementById('load-more-spinner'); + const loadMoreStatus = document.getElementById('load-more-status'); + const objectsLoadingRow = document.getElementById('objects-loading-row'); + let nextContinuationToken = null; + let totalObjectCount = 0; + let loadedObjectCount = 0; + let isLoadingObjects = false; + let hasMoreObjects = false; + let currentFilterTerm = ''; + let currentSortField = 'name'; + let currentSortDir = 'asc'; + let pageSize = 5000; + let currentPrefix = ''; + let allObjects = []; + let streamFolders = []; + let useDelimiterMode = true; + let urlTemplates = null; + let streamAbortController = null; + let useStreaming = !!objectsStreamUrl; + let streamingComplete = false; + const STREAM_RENDER_BATCH = 500; + let pendingStreamObjects = []; + let streamRenderScheduled = false; + + const buildUrlFromTemplate = (template, key) => { + if (!template) return ''; + return template.replace('KEY_PLACEHOLDER', encodeURIComponent(key).replace(/%2F/g, '/')); + }; + + const ROW_HEIGHT = 53; + const BUFFER_ROWS = 10; + let visibleItems = []; + let renderedRange = { start: 0, end: 0 }; + + let memoizedVisibleItems = null; + let memoizedInputs = { objectCount: -1, folderCount: -1, prefix: null, filterTerm: null }; + + const createObjectRow = (obj, displayKey = null) => { + const tr = document.createElement('tr'); + tr.dataset.objectRow = ''; + tr.dataset.key = obj.key; + tr.dataset.size = obj.size; + tr.dataset.lastModified = obj.lastModified ?? obj.last_modified ?? ''; + tr.dataset.lastModifiedDisplay = obj.lastModifiedDisplay ?? obj.last_modified_display ?? new Date(obj.lastModified || obj.last_modified).toLocaleString(); + tr.dataset.lastModifiedIso = obj.lastModifiedIso ?? obj.last_modified_iso ?? obj.lastModified ?? obj.last_modified ?? ''; + tr.dataset.etag = obj.etag ?? ''; + tr.dataset.previewUrl = obj.previewUrl ?? obj.preview_url ?? ''; + tr.dataset.downloadUrl = obj.downloadUrl ?? obj.download_url ?? ''; + tr.dataset.presignEndpoint = obj.presignEndpoint ?? obj.presign_endpoint ?? ''; + tr.dataset.deleteEndpoint = obj.deleteEndpoint ?? obj.delete_endpoint ?? ''; + tr.dataset.metadataUrl = obj.metadataUrl ?? obj.metadata_url ?? ''; + tr.dataset.versionsEndpoint = obj.versionsEndpoint ?? obj.versions_endpoint ?? ''; + tr.dataset.restoreTemplate = obj.restoreTemplate ?? obj.restore_template ?? ''; + tr.dataset.tagsUrl = obj.tagsUrl ?? obj.tags_url ?? ''; + tr.dataset.copyUrl = obj.copyUrl ?? obj.copy_url ?? ''; + tr.dataset.moveUrl = obj.moveUrl ?? obj.move_url ?? ''; + + const keyToShow = displayKey || obj.key; + const lastModDisplay = obj.lastModifiedDisplay || obj.last_modified_display || new Date(obj.lastModified || obj.last_modified).toLocaleDateString(); + + tr.innerHTML = ` + + + + +
+ ${getFileTypeIcon(obj.key)} + ${escapeHtml(keyToShow)} +
+
Modified ${escapeHtml(lastModDisplay)}
+ + + ${formatBytes(obj.size)} + + +
+ + + +
+ +
    +
    • +
    • +
    • +
    • +
+
+
+ + `; + + return tr; + }; + + const showEmptyState = () => { + if (!objectsTableBody) return; + objectsTableBody.innerHTML = ` + + +
+
+ + + + +
+
No objects yet
+

Drag and drop files here or click Upload to get started.

+ +
+ + + `; + }; + + const showLoadError = (message) => { + if (!objectsTableBody) return; + objectsTableBody.innerHTML = ` + + +
+ + + +

Failed to load objects

+

${escapeHtml(message)}

+ +
+ + + `; + }; + + let bucketTotalObjects = objectsContainer ? parseInt(objectsContainer.dataset.bucketTotalObjects || '0', 10) : 0; + + const updateObjectCountBadge = () => { + if (!objectCountBadge) return; + if (useDelimiterMode) { + const total = bucketTotalObjects || totalObjectCount; + objectCountBadge.textContent = `${total.toLocaleString()} object${total !== 1 ? 's' : ''}`; + } else { + objectCountBadge.textContent = `${totalObjectCount.toLocaleString()} object${totalObjectCount !== 1 ? 's' : ''}`; + } + }; + + let topSpacer = null; + let bottomSpacer = null; + + const initVirtualScrollElements = () => { + if (!objectsTableBody) return; + + if (!topSpacer) { + topSpacer = document.createElement('tr'); + topSpacer.id = 'virtual-top-spacer'; + topSpacer.innerHTML = ''; + } + if (!bottomSpacer) { + bottomSpacer = document.createElement('tr'); + bottomSpacer.id = 'virtual-bottom-spacer'; + bottomSpacer.innerHTML = ''; + } + }; + + const computeVisibleItems = (forceRecompute = false) => { + const currentInputs = { + objectCount: allObjects.length, + folderCount: streamFolders.length, + prefix: currentPrefix, + filterTerm: currentFilterTerm, + sortField: currentSortField, + sortDir: currentSortDir + }; + + if (!forceRecompute && + memoizedVisibleItems !== null && + memoizedInputs.objectCount === currentInputs.objectCount && + memoizedInputs.folderCount === currentInputs.folderCount && + memoizedInputs.prefix === currentInputs.prefix && + memoizedInputs.filterTerm === currentInputs.filterTerm && + memoizedInputs.sortField === currentInputs.sortField && + memoizedInputs.sortDir === currentInputs.sortDir) { + return memoizedVisibleItems; + } + + const items = []; + + if (searchResults !== null) { + searchResults.forEach(obj => { + items.push({ type: 'file', data: obj, displayKey: obj.key }); + }); + } else if (useDelimiterMode && streamFolders.length > 0) { + streamFolders.forEach(folderPath => { + const folderName = folderPath.slice(currentPrefix.length).replace(/\/$/, ''); + items.push({ type: 'folder', path: folderPath, displayKey: folderName }); + }); + allObjects.forEach(obj => { + const remainder = obj.key.slice(currentPrefix.length); + if (!remainder) return; + items.push({ type: 'file', data: obj, displayKey: remainder }); + }); + } else { + const folders = new Set(); + + allObjects.forEach(obj => { + if (!obj.key.startsWith(currentPrefix)) return; + + const remainder = obj.key.slice(currentPrefix.length); + + if (!remainder) return; + + const isFolderMarker = obj.key.endsWith('/') && obj.size === 0; + const slashIndex = remainder.indexOf('/'); + + if (slashIndex === -1 && !isFolderMarker) { + items.push({ type: 'file', data: obj, displayKey: remainder }); + } else { + const effectiveSlashIndex = isFolderMarker && slashIndex === remainder.length - 1 + ? slashIndex + : (slashIndex === -1 ? remainder.length - 1 : slashIndex); + const folderName = remainder.slice(0, effectiveSlashIndex); + const folderPath = currentPrefix + folderName + '/'; + if (!folders.has(folderPath)) { + folders.add(folderPath); + items.push({ type: 'folder', path: folderPath, displayKey: folderName }); + } + } + }); + } + + items.sort((a, b) => { + if (a.type === 'folder' && b.type === 'file') return -1; + if (a.type === 'file' && b.type === 'folder') return 1; + if (a.type === 'folder' && b.type === 'folder') { + return a.path.localeCompare(b.path); + } + const dir = currentSortDir === 'asc' ? 1 : -1; + if (currentSortField === 'size') { + return (a.data.size - b.data.size) * dir; + } + if (currentSortField === 'date') { + const aTime = new Date(a.data.lastModified || a.data.last_modified || 0).getTime(); + const bTime = new Date(b.data.lastModified || b.data.last_modified || 0).getTime(); + return (aTime - bTime) * dir; + } + return a.data.key.localeCompare(b.data.key) * dir; + }); + + memoizedVisibleItems = items; + memoizedInputs = currentInputs; + return items; + }; + + const renderVirtualRows = () => { + if (!objectsTableBody || !scrollContainer) return; + + const containerHeight = scrollContainer.clientHeight; + const scrollTop = scrollContainer.scrollTop; + + const startIndex = Math.max(0, Math.floor(scrollTop / ROW_HEIGHT) - BUFFER_ROWS); + const endIndex = Math.min(visibleItems.length, Math.ceil((scrollTop + containerHeight) / ROW_HEIGHT) + BUFFER_ROWS); + + if (startIndex === renderedRange.start && endIndex === renderedRange.end) return; + + renderedRange = { start: startIndex, end: endIndex }; + + objectsTableBody.innerHTML = ''; + + initVirtualScrollElements(); + topSpacer.querySelector('td').style.height = `${startIndex * ROW_HEIGHT}px`; + objectsTableBody.appendChild(topSpacer); + + for (let i = startIndex; i < endIndex; i++) { + const item = visibleItems[i]; + if (!item) continue; + + let row; + if (item.type === 'folder') { + row = createFolderRow(item.path, item.displayKey); + } else { + row = createObjectRow(item.data, item.displayKey); + } + row.dataset.virtualIndex = i; + objectsTableBody.appendChild(row); + } + + const remainingRows = visibleItems.length - endIndex; + bottomSpacer.querySelector('td').style.height = `${remainingRows * ROW_HEIGHT}px`; + objectsTableBody.appendChild(bottomSpacer); + + attachRowHandlers(); + }; + + let scrollTimeout = null; + const handleVirtualScroll = () => { + if (scrollTimeout) cancelAnimationFrame(scrollTimeout); + scrollTimeout = requestAnimationFrame(renderVirtualRows); + }; + + const refreshVirtualList = () => { + visibleItems = computeVisibleItems(); + renderedRange = { start: -1, end: -1 }; + + if (visibleItems.length === 0) { + if (allObjects.length === 0 && streamFolders.length === 0 && !hasMoreObjects) { + showEmptyState(); + } else { + objectsTableBody.innerHTML = ` + + +
+
+ + + +
+
Empty folder
+

This folder contains no objects${hasMoreObjects ? ' yet. Loading more...' : '.'}

+
+ + + `; + } + } else { + renderVirtualRows(); + } + + updateFolderViewStatus(); + }; + + const updateFolderViewStatus = () => { + const folderViewStatusEl = document.getElementById('folder-view-status'); + if (!folderViewStatusEl) return; + folderViewStatusEl.classList.add('d-none'); + }; + + const processStreamObject = (obj) => { + const key = obj.key; + return { + key: key, + size: obj.size, + lastModified: obj.last_modified, + lastModifiedDisplay: obj.last_modified_display, + lastModifiedIso: obj.last_modified_iso, + etag: obj.etag, + previewUrl: urlTemplates ? buildUrlFromTemplate(urlTemplates.preview, key) : '', + downloadUrl: urlTemplates ? buildUrlFromTemplate(urlTemplates.download, key) : '', + presignEndpoint: urlTemplates ? buildUrlFromTemplate(urlTemplates.presign, key) : '', + deleteEndpoint: urlTemplates ? buildUrlFromTemplate(urlTemplates.delete, key) : '', + metadataUrl: urlTemplates ? buildUrlFromTemplate(urlTemplates.metadata, key) : '', + versionsEndpoint: urlTemplates ? buildUrlFromTemplate(urlTemplates.versions, key) : '', + restoreTemplate: urlTemplates ? urlTemplates.restore.replace('KEY_PLACEHOLDER', encodeURIComponent(key).replace(/%2F/g, '/')) : '', + tagsUrl: urlTemplates ? buildUrlFromTemplate(urlTemplates.tags, key) : '', + copyUrl: urlTemplates ? buildUrlFromTemplate(urlTemplates.copy, key) : '', + moveUrl: urlTemplates ? buildUrlFromTemplate(urlTemplates.move, key) : '' + }; + }; + + let lastStreamRenderTime = 0; + const STREAM_RENDER_THROTTLE_MS = 500; + + const buildBottomStatusText = (complete) => { + if (!complete) { + const countText = totalObjectCount > 0 ? ` of ${totalObjectCount.toLocaleString()}` : ''; + return `${loadedObjectCount.toLocaleString()}${countText} loading...`; + } + const parts = []; + if (useDelimiterMode && streamFolders.length > 0) { + parts.push(`${streamFolders.length.toLocaleString()} folder${streamFolders.length !== 1 ? 's' : ''}`); + } + parts.push(`${loadedObjectCount.toLocaleString()} object${loadedObjectCount !== 1 ? 's' : ''}`); + return parts.join(', '); + }; + + const flushPendingStreamObjects = () => { + if (pendingStreamObjects.length > 0) { + const batch = pendingStreamObjects.splice(0, pendingStreamObjects.length); + batch.forEach(obj => { + loadedObjectCount++; + allObjects.push(obj); + }); + } + updateObjectCountBadge(); + if (loadMoreStatus) { + loadMoreStatus.textContent = buildBottomStatusText(streamingComplete); + } + if (objectsLoadingRow && objectsLoadingRow.parentNode) { + const loadingText = objectsLoadingRow.querySelector('p'); + if (loadingText) { + const countText = totalObjectCount > 0 ? ` of ${totalObjectCount.toLocaleString()}` : ''; + loadingText.textContent = `Loading ${loadedObjectCount.toLocaleString()}${countText} objects...`; + } + } + const now = performance.now(); + if (!streamingComplete && now - lastStreamRenderTime < STREAM_RENDER_THROTTLE_MS) { + streamRenderScheduled = false; + return; + } + lastStreamRenderTime = now; + refreshVirtualList(); + streamRenderScheduled = false; + }; + + const scheduleStreamRender = () => { + if (streamRenderScheduled) return; + streamRenderScheduled = true; + requestAnimationFrame(flushPendingStreamObjects); + }; + + const loadObjectsStreaming = async () => { + if (isLoadingObjects) return; + isLoadingObjects = true; + streamingComplete = false; + + if (objectsLoadingRow) objectsLoadingRow.style.display = ''; + nextContinuationToken = null; + loadedObjectCount = 0; + totalObjectCount = 0; + allObjects = []; + streamFolders = []; + memoizedVisibleItems = null; + memoizedInputs = { objectCount: -1, folderCount: -1, prefix: null, filterTerm: null }; + pendingStreamObjects = []; + lastStreamRenderTime = 0; + + streamAbortController = new AbortController(); + + try { + const params = new URLSearchParams(); + if (currentPrefix) params.set('prefix', currentPrefix); + if (useDelimiterMode) params.set('delimiter', '/'); + + const response = await fetch(`${objectsStreamUrl}?${params}`, { + signal: streamAbortController.signal + }); + if (!response.ok) { + throw new Error(`HTTP ${response.status}`); + } + + if (objectsLoadingRow) { + const loadingText = objectsLoadingRow.querySelector('p'); + if (loadingText) loadingText.textContent = 'Receiving objects...'; + } + + const reader = response.body.getReader(); + const decoder = new TextDecoder(); + let buffer = ''; + + while (true) { + const { done, value } = await reader.read(); + if (done) break; + + buffer += decoder.decode(value, { stream: true }); + const lines = buffer.split('\n'); + buffer = lines.pop() || ''; + + for (const line of lines) { + if (!line.trim()) continue; + try { + const msg = JSON.parse(line); + switch (msg.type) { + case 'meta': + urlTemplates = msg.url_templates; + versioningEnabled = msg.versioning_enabled; + if (objectsContainer) { + objectsContainer.dataset.versioning = versioningEnabled ? 'true' : 'false'; + } + break; + case 'count': + totalObjectCount = msg.total_count || 0; + if (objectsLoadingRow) { + const loadingText = objectsLoadingRow.querySelector('p'); + if (loadingText) loadingText.textContent = `Loading 0 of ${totalObjectCount.toLocaleString()} objects...`; + } + break; + case 'folder': + streamFolders.push(msg.prefix); + scheduleStreamRender(); + break; + case 'object': + pendingStreamObjects.push(processStreamObject(msg)); + if (pendingStreamObjects.length >= STREAM_RENDER_BATCH) { + scheduleStreamRender(); + } + break; + case 'error': + throw new Error(msg.error); + case 'done': + streamingComplete = true; + break; + } + } catch (parseErr) { + console.warn('Failed to parse stream line:', line, parseErr); + } + } + if (pendingStreamObjects.length > 0) { + scheduleStreamRender(); + } + } + + if (buffer.trim()) { + try { + const msg = JSON.parse(buffer); + if (msg.type === 'object') { + pendingStreamObjects.push(processStreamObject(msg)); + } else if (msg.type === 'done') { + streamingComplete = true; + } + } catch (e) { } + } + + streamingComplete = true; + flushPendingStreamObjects(); + hasMoreObjects = false; + totalObjectCount = loadedObjectCount; + if (!currentPrefix && !useDelimiterMode) bucketTotalObjects = totalObjectCount; + updateObjectCountBadge(); + + if (objectsLoadingRow && objectsLoadingRow.parentNode) { + objectsLoadingRow.remove(); + } + + if (loadMoreStatus) { + loadMoreStatus.textContent = buildBottomStatusText(true); + } + refreshVirtualList(); + renderBreadcrumb(currentPrefix); + + } catch (error) { + if (error.name === 'AbortError') return; + console.error('Streaming failed, falling back to paginated:', error); + useStreaming = false; + isLoadingObjects = false; + await loadObjectsPaginated(false); + return; + } finally { + isLoadingObjects = false; + streamAbortController = null; + } + }; + + const loadObjectsPaginated = async (append = false) => { + if (isLoadingObjects) return; + isLoadingObjects = true; + + if (!append) { + if (objectsLoadingRow) objectsLoadingRow.style.display = ''; + nextContinuationToken = null; + loadedObjectCount = 0; + totalObjectCount = 0; + allObjects = []; + streamFolders = []; + memoizedVisibleItems = null; + memoizedInputs = { objectCount: -1, folderCount: -1, prefix: null, filterTerm: null }; + } + + if (append && loadMoreSpinner) { + loadMoreSpinner.classList.remove('d-none'); + } + + try { + const params = new URLSearchParams({ max_keys: String(pageSize) }); + if (nextContinuationToken) { + params.set('continuation_token', nextContinuationToken); + } + + const response = await fetch(`${objectsApiUrl}?${params}`); + if (!response.ok) { + const data = await response.json().catch(() => ({})); + throw new Error(data.error || `HTTP ${response.status}`); + } + + const data = await response.json(); + + versioningEnabled = data.versioning_enabled; + if (objectsContainer) { + objectsContainer.dataset.versioning = versioningEnabled ? 'true' : 'false'; + } + + totalObjectCount = data.total_count || 0; + if (!append && !currentPrefix && !useDelimiterMode) bucketTotalObjects = totalObjectCount; + nextContinuationToken = data.next_continuation_token; + + if (!append && objectsLoadingRow) { + objectsLoadingRow.remove(); + } + + if (data.url_templates && !urlTemplates) { + urlTemplates = data.url_templates; + } + + data.objects.forEach(obj => { + loadedObjectCount++; + allObjects.push(processStreamObject(obj)); + }); + + updateObjectCountBadge(); + hasMoreObjects = data.is_truncated; + + if (loadMoreStatus) { + if (data.is_truncated) { + loadMoreStatus.textContent = `${loadedObjectCount.toLocaleString()} of ${totalObjectCount.toLocaleString()} loaded`; + } else { + loadMoreStatus.textContent = `${loadedObjectCount.toLocaleString()} objects`; + } + } + + refreshVirtualList(); + renderBreadcrumb(currentPrefix); + + } catch (error) { + console.error('Failed to load objects:', error); + if (!append) { + showLoadError(error.message); + } else { + showMessage({ title: 'Load Failed', body: error.message, variant: 'danger' }); + } + } finally { + isLoadingObjects = false; + if (loadMoreSpinner) { + loadMoreSpinner.classList.add('d-none'); + } + } + }; + + const loadObjects = async (append = false) => { + if (useStreaming && !append) { + return loadObjectsStreaming(); + } + return loadObjectsPaginated(append); + }; + + const attachRowHandlers = () => { + const objectRows = document.querySelectorAll('[data-object-row]'); + objectRows.forEach(row => { + if (row.dataset.handlersAttached) return; + row.dataset.handlersAttached = 'true'; + + const deleteBtn = row.querySelector('[data-delete-object]'); + deleteBtn?.addEventListener('click', (e) => { + e.stopPropagation(); + const deleteModalEl = document.getElementById('deleteObjectModal'); + const deleteModal = deleteModalEl ? bootstrap.Modal.getOrCreateInstance(deleteModalEl) : null; + const deleteObjectForm = document.getElementById('deleteObjectForm'); + const deleteObjectKey = document.getElementById('deleteObjectKey'); + if (deleteModal && deleteObjectForm) { + deleteObjectForm.setAttribute('action', row.dataset.deleteEndpoint); + if (deleteObjectKey) deleteObjectKey.textContent = row.dataset.key; + deleteModal.show(); + } + }); + + const selectCheckbox = row.querySelector('[data-object-select]'); + selectCheckbox?.addEventListener('click', (event) => event.stopPropagation()); + selectCheckbox?.addEventListener('change', () => { + toggleRowSelection(row, selectCheckbox.checked); + }); + + if (selectedRows.has(row.dataset.key)) { + selectCheckbox.checked = true; + row.classList.add('table-active'); + } + + if (activeRow && activeRow.dataset.key === row.dataset.key) { + row.classList.add('table-active'); + activeRow = row; + } + }); + + const folderRows = document.querySelectorAll('.folder-row'); + folderRows.forEach(row => { + if (row.dataset.handlersAttached) return; + row.dataset.handlersAttached = 'true'; + + const folderPath = row.dataset.folderPath; + + const checkbox = row.querySelector('[data-folder-select]'); + checkbox?.addEventListener('change', (e) => { + e.stopPropagation(); + if (checkbox.checked) { + selectedRows.set(folderPath, { key: folderPath, isFolder: true }); + } else { + selectedRows.delete(folderPath); + } + const folderObjects = allObjects.filter(obj => obj.key.startsWith(folderPath)); + folderObjects.forEach(obj => { + if (checkbox.checked) { + selectedRows.set(obj.key, obj); + } else { + selectedRows.delete(obj.key); + } + }); + updateBulkDeleteState(); + }); + + const folderBtn = row.querySelector('button'); + folderBtn?.addEventListener('click', (e) => { + e.stopPropagation(); + navigateToFolder(folderPath); + }); + + row.addEventListener('click', (e) => { + if (e.target.closest('[data-folder-select]') || e.target.closest('button')) return; + navigateToFolder(folderPath); + }); + }); + + updateBulkDeleteState(); + }; + + const scrollSentinel = document.getElementById('scroll-sentinel'); + const scrollContainer = document.querySelector('.objects-table-container'); + + if (scrollContainer) { + scrollContainer.addEventListener('scroll', handleVirtualScroll, { passive: true }); + } + + if (scrollSentinel && scrollContainer) { + const containerObserver = new IntersectionObserver((entries) => { + entries.forEach(entry => { + if (entry.isIntersecting && hasMoreObjects && !isLoadingObjects) { + loadObjects(true); + } + }); + }, { + root: scrollContainer, + rootMargin: '500px', + threshold: 0 + }); + containerObserver.observe(scrollSentinel); + + const viewportObserver = new IntersectionObserver((entries) => { + entries.forEach(entry => { + if (entry.isIntersecting && hasMoreObjects && !isLoadingObjects) { + loadObjects(true); + } + }); + }, { + root: null, + rootMargin: '500px', + threshold: 0 + }); + viewportObserver.observe(scrollSentinel); + } + + + if (objectsApiUrl) { + loadObjects(); + } + + const folderBreadcrumb = document.getElementById('folder-breadcrumb'); + const objectsTableBody = document.querySelector('#objects-table tbody'); + + if (objectsTableBody) { + objectsTableBody.addEventListener('click', (e) => { + const row = e.target.closest('[data-object-row]'); + if (!row) return; + + if (e.target.closest('[data-delete-object]') || e.target.closest('[data-object-select]') || e.target.closest('a') || e.target.closest('.dropdown')) { + return; + } + + selectRow(row); + }); + } + + const hasFolders = () => streamFolders.length > 0 || allObjects.some(obj => obj.key.includes('/')); + + const getFoldersAtPrefix = (prefix) => { + const folders = new Set(); + const files = []; + + allObjects.forEach(obj => { + const key = obj.key; + if (!key.startsWith(prefix)) return; + + const remainder = key.slice(prefix.length); + const slashIndex = remainder.indexOf('/'); + + if (slashIndex === -1) { + + files.push(obj); + } else { + + const folderName = remainder.slice(0, slashIndex + 1); + folders.add(prefix + folderName); + } + }); + + return { folders: Array.from(folders).sort(), files }; + }; + + const countObjectsInFolder = (folderPrefix) => { + if (useDelimiterMode) { + return { count: 0, mayHaveMore: true }; + } + const count = allObjects.filter(obj => obj.key.startsWith(folderPrefix)).length; + return { count, mayHaveMore: hasMoreObjects }; + }; + + const renderBreadcrumb = (prefix) => { + if (!folderBreadcrumb) return; + + if (!prefix && !hasFolders()) { + folderBreadcrumb.classList.add('d-none'); + return; + } + + folderBreadcrumb.classList.remove('d-none'); + const ol = folderBreadcrumb.querySelector('ol'); + ol.innerHTML = ''; + + const rootLi = document.createElement('li'); + rootLi.className = 'breadcrumb-item'; + if (!prefix) { + rootLi.classList.add('active'); + rootLi.setAttribute('aria-current', 'page'); + rootLi.innerHTML = ` + + + + Root + `; + } else { + rootLi.innerHTML = ` + + + + + Root + + `; + } + ol.appendChild(rootLi); + + if (prefix) { + const parts = prefix.split('/').filter(Boolean); + let accumulated = ''; + parts.forEach((part, index) => { + accumulated += part + '/'; + const li = document.createElement('li'); + li.className = 'breadcrumb-item'; + + if (index === parts.length - 1) { + li.classList.add('active'); + li.setAttribute('aria-current', 'page'); + li.textContent = part; + } else { + const a = document.createElement('a'); + a.href = '#'; + a.className = 'text-decoration-none'; + a.dataset.folderNav = accumulated; + a.textContent = part; + li.appendChild(a); + } + ol.appendChild(li); + }); + } + + ol.querySelectorAll('[data-folder-nav]').forEach(link => { + link.addEventListener('click', (e) => { + e.preventDefault(); + navigateToFolder(link.dataset.folderNav); + }); + }); + }; + + const getObjectsInFolder = (folderPrefix) => { + return allObjects.filter(obj => obj.key.startsWith(folderPrefix)); + }; + + const createFolderRow = (folderPath, displayName = null) => { + const folderName = displayName || folderPath.slice(currentPrefix.length).replace(/\/$/, ''); + const { count: objectCount, mayHaveMore } = countObjectsInFolder(folderPath); + let countLine = ''; + if (useDelimiterMode) { + countLine = ''; + } else { + const countDisplay = mayHaveMore ? `${objectCount}+` : objectCount; + countLine = `
${countDisplay} object${objectCount !== 1 ? 's' : ''}
`; + } + + const tr = document.createElement('tr'); + tr.className = 'folder-row'; + tr.dataset.folderPath = folderPath; + tr.style.cursor = 'pointer'; + + tr.innerHTML = ` + + + + +
+ + + + ${escapeHtml(folderName)}/ +
+ ${countLine} + + + — + + + + + `; + + return tr; + }; + + const navigateToFolder = (prefix) => { + if (streamAbortController) { + streamAbortController.abort(); + streamAbortController = null; + } + + currentPrefix = prefix; + + if (scrollContainer) scrollContainer.scrollTop = 0; + + selectedRows.clear(); + + if (typeof updateBulkDeleteState === 'function') { + updateBulkDeleteState(); + } + + if (previewPanel) previewPanel.classList.add('d-none'); + if (previewEmpty) previewEmpty.classList.remove('d-none'); + activeRow = null; + + isLoadingObjects = false; + loadObjects(false); + }; + + const renderObjectsView = () => { + if (!objectsTableBody) return; + + const { folders, files } = getFoldersAtPrefix(currentPrefix); + + objectsTableBody.innerHTML = ''; + + folders.forEach(folderPath => { + objectsTableBody.appendChild(createFolderRow(folderPath)); + }); + + files.forEach(obj => { + objectsTableBody.appendChild(obj.element); + obj.element.style.display = ''; + + const keyCell = obj.element.querySelector('.object-key .fw-medium'); + if (keyCell && currentPrefix) { + const displayName = obj.key.slice(currentPrefix.length); + keyCell.textContent = displayName; + keyCell.closest('.object-key').title = obj.key; + } else if (keyCell) { + keyCell.textContent = obj.key; + } + }); + + allObjects.forEach(obj => { + if (!files.includes(obj)) { + obj.element.style.display = 'none'; + } + }); + + if (folders.length === 0 && files.length === 0) { + const emptyRow = document.createElement('tr'); + emptyRow.innerHTML = ` + +
+
+ + + +
+
Empty folder
+

This folder contains no objects.

+
+ + `; + objectsTableBody.appendChild(emptyRow); + } + + if (typeof updateBulkDeleteState === 'function') { + updateBulkDeleteState(); + } + }; + + const showMessage = ({ title = 'Notice', body = '', bodyHtml = null, variant = 'info', actionText = null, onAction = null }) => { + if (!actionText && !onAction && window.showToast) { + window.showToast(body || title, title, variant); + return; + } + if (!messageModal) { + window.alert(body || title); + return; + } + document.querySelectorAll('.modal.show').forEach(modal => { + const instance = bootstrap.Modal.getInstance(modal); + if (instance && modal.id !== 'messageModal') { + instance.hide(); + } + }); + const iconEl = document.getElementById('messageModalIcon'); + if (iconEl) { + const iconPaths = { + success: '', + danger: '', + warning: '', + info: '' + }; + const iconColors = { success: 'text-success', danger: 'text-danger', warning: 'text-warning', info: 'text-primary' }; + iconEl.innerHTML = iconPaths[variant] || iconPaths.info; + iconEl.classList.remove('text-success', 'text-danger', 'text-warning', 'text-primary'); + iconEl.classList.add(iconColors[variant] || 'text-primary'); + } + messageModalTitle.textContent = title; + if (bodyHtml) { + messageModalBody.innerHTML = bodyHtml; + } else { + messageModalBody.textContent = body; + } + messageModalActionHandler = null; + const variantClass = { + success: 'btn-success', + danger: 'btn-danger', + warning: 'btn-warning', + info: 'btn-primary', + }; + Object.values(variantClass).forEach((cls) => messageModalAction.classList.remove(cls)); + if (actionText && typeof onAction === 'function') { + messageModalAction.textContent = actionText; + messageModalAction.classList.remove('d-none'); + messageModalAction.classList.add(variantClass[variant] || 'btn-primary'); + messageModalActionHandler = onAction; + } else { + messageModalAction.classList.add('d-none'); + } + setTimeout(() => messageModal.show(), 150); + }; + + messageModalAction?.addEventListener('click', () => { + if (typeof messageModalActionHandler === 'function') { + messageModalActionHandler(); + } + messageModal?.hide(); + }); + + messageModalEl?.addEventListener('hidden.bs.modal', () => { + messageModalActionHandler = null; + messageModalAction.classList.add('d-none'); + }); + + const normalizePolicyTemplate = (rawTemplate) => { + if (!rawTemplate) { + return ''; + } + try { + let parsed = JSON.parse(rawTemplate); + if (typeof parsed === 'string') { + parsed = JSON.parse(parsed); + } + return JSON.stringify(parsed, null, 2); + } catch { + return rawTemplate; + } + }; + + let publicPolicyTemplate = normalizePolicyTemplate(policyTextarea?.dataset.publicTemplate || ''); + let customPolicyDraft = policyTextarea?.value || ''; + const policyReadonlyHint = document.getElementById('policyReadonlyHint'); + const presetButtons = Array.from(document.querySelectorAll('.preset-btn[data-preset]')); + + const setActivePolicyPreset = (preset) => { + if (policyPreset) { + policyPreset.value = preset; + } + presetButtons.forEach(button => { + button.classList.toggle('active', button.dataset.preset === preset); + }); + }; + + const setPolicyTextareaState = (readonly) => { + if (!policyTextarea) return; + if (readonly) { + policyTextarea.setAttribute('readonly', 'readonly'); + policyTextarea.classList.add('bg-body-secondary'); + policyTextarea.classList.add('policy-editor-disabled'); + policyTextarea.setAttribute('aria-disabled', 'true'); + } else { + policyTextarea.removeAttribute('readonly'); + policyTextarea.classList.remove('bg-body-secondary'); + policyTextarea.classList.remove('policy-editor-disabled'); + policyTextarea.removeAttribute('aria-disabled'); + } + }; + + const applyPolicyPreset = (preset) => { + if (!policyTextarea || !policyMode) return; + const isPresetMode = preset === 'private' || preset === 'public'; + if (policyReadonlyHint) { + policyReadonlyHint.classList.toggle('d-none', !isPresetMode); + } + switch (preset) { + case 'private': + setPolicyTextareaState(true); + policyTextarea.value = ''; + policyMode.value = 'delete'; + break; + case 'public': + setPolicyTextareaState(true); + policyTextarea.value = publicPolicyTemplate || ''; + policyMode.value = 'upsert'; + break; + default: + setPolicyTextareaState(false); + policyTextarea.value = customPolicyDraft; + policyMode.value = 'upsert'; + break; + } + }; + + policyTextarea?.addEventListener('input', () => { + if (policyPreset?.value === 'custom') { + customPolicyDraft = policyTextarea.value; + } + }); + + presetButtons.forEach(btn => { + btn.addEventListener('click', () => { + setActivePolicyPreset(btn.dataset.preset); + applyPolicyPreset(btn.dataset.preset); + }); + }); + + if (policyPreset) { + setActivePolicyPreset(policyPreset.value || policyPreset.dataset.default || 'custom'); + applyPolicyPreset(policyPreset.value || policyPreset.dataset.default || 'custom'); + } + + policyForm?.addEventListener('submit', () => { + if (!policyMode || !policyPreset || !policyTextarea) { + return; + } + if (policyPreset.value === 'private') { + policyMode.value = 'delete'; + policyTextarea.value = ''; + } else if (policyPreset.value === 'public') { + policyMode.value = 'upsert'; + policyTextarea.value = publicPolicyTemplate || policyTextarea.value; + } else { + policyMode.value = 'upsert'; + } + }); + + const bulkActionsWrapper = document.getElementById('bulk-actions-wrapper'); + const updateBulkDeleteState = () => { + const selectedCount = selectedRows.size; + if (bulkDeleteButton) { + const shouldShow = Boolean(bulkDeleteEndpoint) && (selectedCount > 0 || bulkDeleting); + bulkDeleteButton.disabled = !bulkDeleteEndpoint || selectedCount === 0 || bulkDeleting; + if (bulkDeleteLabel) { + bulkDeleteLabel.textContent = selectedCount ? `Delete (${selectedCount})` : 'Delete'; + } + if (bulkActionsWrapper) { + bulkActionsWrapper.classList.toggle('d-none', !shouldShow); + } + } + if (bulkDeleteConfirm) { + bulkDeleteConfirm.disabled = selectedCount === 0 || bulkDeleting; + } + if (selectAllCheckbox) { + const filesInView = visibleItems.filter(item => item.type === 'file'); + const foldersInView = visibleItems.filter(item => item.type === 'folder'); + const total = filesInView.length + foldersInView.length; + const fileSelectedCount = filesInView.filter(item => selectedRows.has(item.data.key)).length; + const folderSelectedCount = foldersInView.filter(item => selectedRows.has(item.path)).length; + const visibleSelectedCount = fileSelectedCount + folderSelectedCount; + selectAllCheckbox.disabled = total === 0; + selectAllCheckbox.checked = visibleSelectedCount > 0 && visibleSelectedCount === total && total > 0; + selectAllCheckbox.indeterminate = visibleSelectedCount > 0 && visibleSelectedCount < total; + } + }; + + function toggleRowSelection(row, shouldSelect) { + if (!row || !row.dataset.key) return; + if (shouldSelect) { + selectedRows.set(row.dataset.key, row); + } else { + selectedRows.delete(row.dataset.key); + } + updateBulkDeleteState(); + } + + const renderBulkDeletePreview = () => { + if (!bulkDeleteList) return; + const keys = Array.from(selectedRows.keys()); + bulkDeleteList.innerHTML = ''; + if (bulkDeleteCount) { + const folderCount = keys.filter(k => k.endsWith('/')).length; + const objectCount = keys.length - folderCount; + const parts = []; + if (folderCount) parts.push(`${folderCount} folder${folderCount !== 1 ? 's' : ''}`); + if (objectCount) parts.push(`${objectCount} object${objectCount !== 1 ? 's' : ''}`); + bulkDeleteCount.textContent = `${parts.join(' and ')} selected`; + } + if (!keys.length) { + const empty = document.createElement('li'); + empty.className = 'list-group-item py-2 small text-muted'; + empty.textContent = 'No objects selected.'; + bulkDeleteList.appendChild(empty); + if (bulkDeleteStatus) { + bulkDeleteStatus.textContent = ''; + } + return; + } + const preview = keys.slice(0, 6); + preview.forEach((key) => { + const item = document.createElement('li'); + item.className = 'list-group-item py-1 small text-break'; + item.textContent = key; + bulkDeleteList.appendChild(item); + }); + if (bulkDeleteStatus) { + bulkDeleteStatus.textContent = keys.length > preview.length ? `+${keys.length - preview.length} more not shown` : ''; + } + }; + + const openBulkDeleteModal = () => { + if (!bulkDeleteModal) { + return; + } + if (selectedRows.size === 0) { + showMessage({ title: 'Select objects', body: 'Choose at least one object to delete.', variant: 'warning' }); + return; + } + renderBulkDeletePreview(); + if (bulkDeletePurge) { + bulkDeletePurge.checked = false; + } + if (bulkDeleteConfirm) { + bulkDeleteConfirm.disabled = bulkDeleting; + bulkDeleteConfirm.textContent = bulkDeleting ? 'Deleting…' : 'Delete objects'; + } + bulkDeleteModal.show(); + }; + + const performBulkDelete = async () => { + if (!bulkDeleteEndpoint || selectedRows.size === 0 || !bulkDeleteConfirm) { + return; + } + bulkDeleting = true; + bulkDeleteConfirm.disabled = true; + bulkDeleteConfirm.textContent = 'Deleting…'; + updateBulkDeleteState(); + const payload = { + keys: Array.from(selectedRows.keys()), + }; + if (versioningEnabled && bulkDeletePurge?.checked) { + payload.purge_versions = true; + } + try { + const response = await fetch(bulkDeleteEndpoint, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'X-Requested-With': 'XMLHttpRequest', + }, + body: JSON.stringify(payload), + }); + let data = {}; + try { + data = await response.json(); + } catch { + data = {}; + } + if (!response.ok || data.error) { + throw new Error(data.error || data.message || 'Unable to delete selected objects'); + } + bulkDeleteModal?.hide(); + const deletedCount = Array.isArray(data.deleted) ? data.deleted.length : selectedRows.size; + const errorCount = Array.isArray(data.errors) ? data.errors.length : 0; + const messageParts = []; + if (deletedCount) { + messageParts.push(`${deletedCount} deleted`); + } + if (errorCount) { + messageParts.push(`${errorCount} failed`); + } + const summary = messageParts.length ? messageParts.join(', ') : 'Bulk delete finished'; + showMessage({ title: 'Bulk delete complete', body: data.message || summary, variant: errorCount ? 'warning' : 'success' }); + selectedRows.clear(); + previewEmpty.classList.remove('d-none'); + previewPanel.classList.add('d-none'); + activeRow = null; + loadObjects(false); + } catch (error) { + bulkDeleteModal?.hide(); + showMessage({ title: 'Delete failed', body: (error && error.message) || 'Unable to delete selected objects', variant: 'danger' }); + } finally { + bulkDeleting = false; + if (bulkDeleteConfirm) { + bulkDeleteConfirm.disabled = false; + bulkDeleteConfirm.textContent = 'Delete objects'; + } + updateBulkDeleteState(); + } + }; + + const updateGeneratePresignState = () => { + if (!generatePresignButton) return; + if (isGeneratingPresign) { + generatePresignButton.disabled = true; + generatePresignButton.textContent = 'Generating…'; + return; + } + generatePresignButton.textContent = 'Generate link'; + generatePresignButton.disabled = !activeRow; + }; + + const requestPresignedUrl = async () => { + if (!activeRow) { + showMessage({ title: 'Select an object', body: 'Choose an object before generating a presigned URL.', variant: 'warning' }); + return; + } + const endpoint = activeRow.dataset.presignEndpoint; + if (!endpoint) { + showMessage({ title: 'Unavailable', body: 'Presign endpoint unavailable for this object.', variant: 'danger' }); + return; + } + if (isGeneratingPresign) { + return; + } + isGeneratingPresign = true; + updateGeneratePresignState(); + presignLink.value = ''; + try { + const payload = { + method: presignMethod?.value || 'GET', + expires_in: Number(presignTtl?.value) || 900, + }; + const response = await fetch(endpoint, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '' }, + body: JSON.stringify(payload), + }); + const data = await response.json(); + if (!response.ok) { + throw new Error(data.error || 'Unable to generate presigned URL'); + } + presignLink.value = data.url; + } catch (error) { + presignModal?.hide(); + showMessage({ title: 'Presign failed', body: (error && error.message) || 'Unable to generate presigned URL', variant: 'danger' }); + } finally { + isGeneratingPresign = false; + updateGeneratePresignState(); + } + }; + + const INTERNAL_METADATA_KEYS = new Set([ + '__etag__', + '__size__', + '__content_type__', + '__last_modified__', + '__storage_class__', + ]); + + const isInternalKey = (key) => INTERNAL_METADATA_KEYS.has(key.toLowerCase()); + + const renderMetadata = (metadata) => { + if (!previewMetadata || !previewMetadataList) return; + previewMetadataList.innerHTML = ''; + if (!metadata) { + previewMetadata.classList.add('d-none'); + return; + } + const userMetadata = Object.entries(metadata).filter(([key]) => !isInternalKey(key)); + if (userMetadata.length === 0) { + previewMetadata.classList.add('d-none'); + return; + } + previewMetadata.classList.remove('d-none'); + userMetadata.forEach(([key, value]) => { + const wrapper = document.createElement('div'); + wrapper.className = 'metadata-entry'; + const label = document.createElement('div'); + label.className = 'metadata-key small'; + label.textContent = key; + const val = document.createElement('div'); + val.className = 'metadata-value text-break'; + val.textContent = value; + wrapper.appendChild(label); + wrapper.appendChild(val); + previewMetadataList.appendChild(wrapper); + }); + }; + + const describeVersionReason = (reason) => { + switch (reason) { + case 'delete': + return 'delete marker'; + case 'restore-overwrite': + return 'restore overwrite'; + default: + return reason || 'update'; + } + }; + + const confirmVersionRestore = (row, version, label = null, onConfirm) => { + if (!version) return; + const timestamp = (version.archived_at || version.last_modified) ? new Date(version.archived_at || version.last_modified).toLocaleString() : version.version_id; + const sizeLabel = formatBytes(Number(version.size) || 0); + const reasonLabel = describeVersionReason(version.reason); + const targetLabel = label || row?.dataset.key || 'this object'; + const metadata = version.metadata && typeof version.metadata === 'object' ? Object.entries(version.metadata) : []; + const metadataHtml = metadata.length + ? `
Metadata
${metadata + .map( + ([key, value]) => + `
${escapeHtml(key)}
${escapeHtml(value)}
` + ) + .join('')}
` + : ''; + const summaryHtml = ` +
+
Target: ${escapeHtml(targetLabel)}
+
Version ID: ${escapeHtml(version.version_id)}
+
Timestamp: ${escapeHtml(timestamp)}
+
Size: ${escapeHtml(sizeLabel)}
+
Reason: ${escapeHtml(reasonLabel)}
+
+ ${metadataHtml} + `; + const fallbackText = `Restore ${targetLabel} from ${timestamp}? Size ${sizeLabel}. Reason: ${reasonLabel}.`; + showMessage({ + title: 'Restore archived version?', + body: fallbackText, + bodyHtml: summaryHtml, + variant: 'warning', + actionText: 'Restore version', + onAction: () => { + if (typeof onConfirm === 'function') { + onConfirm(); + } else { + restoreVersion(row, version); + } + }, + }); + }; + + const updateArchivedCount = (count) => { + if (!archivedCountBadge) return; + const label = count === 1 ? 'item' : 'items'; + archivedCountBadge.textContent = `${count} ${label}`; + }; + + function renderArchivedRows(items) { + if (!archivedBody) return; + archivedBody.innerHTML = ''; + if (!items || items.length === 0) { + archivedBody.innerHTML = 'No archived-only objects.'; + updateArchivedCount(0); + return; + } + updateArchivedCount(items.length); + items.forEach((item) => { + const row = document.createElement('tr'); + + const keyCell = document.createElement('td'); + const keyLabel = document.createElement('div'); + keyLabel.className = 'fw-semibold text-break'; + keyLabel.textContent = item.key; + const badgeWrap = document.createElement('div'); + badgeWrap.className = 'mt-1'; + const badge = document.createElement('span'); + badge.className = 'badge text-bg-warning'; + badge.textContent = 'Archived'; + badgeWrap.appendChild(badge); + keyCell.appendChild(keyLabel); + keyCell.appendChild(badgeWrap); + + const latestCell = document.createElement('td'); + if (item.latest) { + const ts = (item.latest.archived_at || item.latest.last_modified) ? new Date(item.latest.archived_at || item.latest.last_modified).toLocaleString() : item.latest.version_id; + const sizeLabel = formatBytes(Number(item.latest.size) || 0); + latestCell.innerHTML = `
${ts}
${sizeLabel} · ${describeVersionReason(item.latest.reason)}
`; + } else { + latestCell.innerHTML = 'Unknown'; + } + + const countCell = document.createElement('td'); + countCell.className = 'text-end text-muted'; + countCell.textContent = item.versions; + + const actionsCell = document.createElement('td'); + actionsCell.className = 'text-end'; + const btnGroup = document.createElement('div'); + btnGroup.className = 'btn-group btn-group-sm'; + + const restoreButton = document.createElement('button'); + restoreButton.type = 'button'; + restoreButton.className = 'btn btn-outline-primary'; + restoreButton.textContent = 'Restore'; + restoreButton.disabled = !item.latest || !item.restore_url; + restoreButton.addEventListener('click', () => confirmVersionRestore(null, item.latest, item.key, () => restoreArchivedObject(item))); + + const purgeButton = document.createElement('button'); + purgeButton.type = 'button'; + purgeButton.className = 'btn btn-outline-danger'; + purgeButton.textContent = 'Delete versions'; + purgeButton.addEventListener('click', () => confirmArchivedPurge(item)); + + btnGroup.appendChild(restoreButton); + btnGroup.appendChild(purgeButton); + actionsCell.appendChild(btnGroup); + + row.appendChild(keyCell); + row.appendChild(latestCell); + row.appendChild(countCell); + row.appendChild(actionsCell); + archivedBody.appendChild(row); + }); + } + + async function restoreArchivedObject(item) { + if (!item?.restore_url) return; + try { + const response = await fetch(item.restore_url, { method: 'POST' }); + let data = {}; + try { + data = await response.json(); + } catch { + data = {}; + } + if (!response.ok) { + throw new Error(data.error || 'Unable to restore archived object'); + } + showMessage({ title: 'Restore scheduled', body: data.message || 'Object restored from archive.', variant: 'success' }); + await loadArchivedObjects(); + loadObjects(false); + } catch (error) { + showMessage({ title: 'Restore failed', body: (error && error.message) || 'Unable to restore archived object', variant: 'danger' }); + } + } + + async function purgeArchivedObject(item) { + if (!item?.purge_url) return; + try { + const response = await fetch(item.purge_url, { + method: 'POST', + headers: { 'X-Requested-With': 'XMLHttpRequest' }, + }); + let data = {}; + try { + data = await response.json(); + } catch { + data = {}; + } + if (!response.ok) { + throw new Error(data.error || 'Unable to delete archived versions'); + } + showMessage({ title: 'Archived versions removed', body: data.message || 'All archived data for this key has been deleted.', variant: 'success' }); + await loadArchivedObjects(); + } catch (error) { + showMessage({ title: 'Delete failed', body: (error && error.message) || 'Unable to delete archived versions', variant: 'danger' }); + } + } + + function confirmArchivedPurge(item) { + const label = item?.key || 'this object'; + const count = item?.versions || 0; + const countLabel = count === 1 ? 'version' : 'versions'; + showMessage({ + title: 'Delete archived versions?', + body: `Permanently remove ${count} archived ${countLabel} for ${label}? This cannot be undone.`, + variant: 'danger', + actionText: 'Delete versions', + onAction: () => purgeArchivedObject(item), + }); + } + + async function loadArchivedObjects() { + if (!archivedEndpoint || !archivedBody) return; + archivedBody.innerHTML = 'Loading…'; + try { + const response = await fetch(archivedEndpoint); + let data = {}; + try { + data = await response.json(); + } catch { + data = {}; + } + if (!response.ok) { + throw new Error(data.error || 'Unable to load archived objects'); + } + const items = Array.isArray(data.objects) ? data.objects : []; + renderArchivedRows(items); + } catch (error) { + archivedBody.innerHTML = `${(error && error.message) || 'Unable to load archived objects'}`; + updateArchivedCount(0); + } + } + + if (archivedRefreshButton) { + archivedRefreshButton.addEventListener('click', () => loadArchivedObjects()); + } + if (archivedCard && archivedEndpoint) { + loadArchivedObjects(); + } + + const propertiesTab = document.getElementById('properties-tab'); + if (propertiesTab) { + propertiesTab.addEventListener('shown.bs.tab', () => { + if (archivedCard && archivedEndpoint) { + loadArchivedObjects(); + } + }); + } + + async function restoreVersion(row, version) { + if (!row || !version?.version_id) return; + const template = row.dataset.restoreTemplate; + if (!template) return; + const url = template.replace('VERSION_ID_PLACEHOLDER', version.version_id); + try { + const response = await fetch(url, { method: 'POST' }); + let data = {}; + try { + data = await response.json(); + } catch { + data = {}; + } + if (!response.ok) { + throw new Error(data.error || 'Unable to restore version'); + } + const endpoint = row.dataset.versionsEndpoint; + if (endpoint) { + versionsCache.delete(endpoint); + } + await loadObjectVersions(row, { force: true }); + showMessage({ title: 'Version restored', body: data.message || 'The selected version has been restored.', variant: 'success' }); + loadObjects(false); + } catch (error) { + showMessage({ title: 'Restore failed', body: (error && error.message) || 'Unable to restore version', variant: 'danger' }); + } + } + + function renderVersionEntries(entries, row) { + if (!versionList) return; + if (!entries || entries.length === 0) { + versionList.innerHTML = '

No previous versions yet.

'; + return; + } + versionList.innerHTML = ''; + entries.forEach((entry, index) => { + const versionNumber = index + 1; + const item = document.createElement('div'); + item.className = 'd-flex align-items-center justify-content-between py-2 border-bottom'; + const textStack = document.createElement('div'); + textStack.className = 'me-3'; + const heading = document.createElement('div'); + heading.className = 'd-flex align-items-center'; + const badge = document.createElement('span'); + badge.className = 'badge text-bg-secondary me-2'; + badge.textContent = `#${versionNumber}`; + const title = document.createElement('div'); + title.className = 'fw-semibold small'; + const timestamp = (entry.archived_at || entry.last_modified) ? new Date(entry.archived_at || entry.last_modified).toLocaleString() : entry.version_id; + title.textContent = timestamp; + heading.appendChild(badge); + heading.appendChild(title); + const meta = document.createElement('div'); + meta.className = 'text-muted small'; + const reason = describeVersionReason(entry.reason); + const sizeLabel = formatBytes(Number(entry.size) || 0); + meta.textContent = `${sizeLabel} · ${reason}`; + textStack.appendChild(heading); + textStack.appendChild(meta); + const restoreButton = document.createElement('button'); + restoreButton.type = 'button'; + restoreButton.className = 'btn btn-outline-primary btn-sm'; + restoreButton.textContent = 'Restore'; + restoreButton.addEventListener('click', () => confirmVersionRestore(row, entry)); + item.appendChild(textStack); + item.appendChild(restoreButton); + versionList.appendChild(item); + }); + } + + async function loadObjectVersions(row, { force = false } = {}) { + if (!versionPanel || !versionList || !versioningEnabled) { + versionPanel?.classList.add('d-none'); + return; + } + if (!row) { + versionPanel.classList.add('d-none'); + return; + } + const endpoint = row.dataset.versionsEndpoint; + if (!endpoint) { + versionPanel.classList.add('d-none'); + return; + } + versionPanel.classList.remove('d-none'); + if (!force && versionsCache.has(endpoint)) { + renderVersionEntries(versionsCache.get(endpoint), row); + return; + } + versionList.innerHTML = '
Loading versions…
'; + try { + const response = await fetch(endpoint); + let data = {}; + try { + data = await response.json(); + } catch { + data = {}; + } + if (!response.ok) { + throw new Error(data.error || 'Unable to load versions'); + } + const entries = Array.isArray(data.versions) ? data.versions : []; + versionsCache.set(endpoint, entries); + renderVersionEntries(entries, row); + } catch (error) { + versionList.innerHTML = `

${(error && error.message) || 'Unable to load versions'}

`; + } + } + + renderMetadata(null); + const deleteModalEl = document.getElementById('deleteObjectModal'); + const deleteModal = deleteModalEl ? new bootstrap.Modal(deleteModalEl) : null; + const deleteObjectForm = document.getElementById('deleteObjectForm'); + const deleteObjectKey = document.getElementById('deleteObjectKey'); + + if (deleteObjectForm) { + deleteObjectForm.addEventListener('submit', async (e) => { + e.preventDefault(); + const submitBtn = deleteObjectForm.querySelector('[type="submit"]'); + const originalHtml = submitBtn ? submitBtn.innerHTML : ''; + try { + if (submitBtn) { + submitBtn.disabled = true; + submitBtn.innerHTML = 'Deleting...'; + } + const formData = new FormData(deleteObjectForm); + const csrfToken = formData.get('csrf_token') || (window.getCsrfToken ? window.getCsrfToken() : ''); + const formAction = deleteObjectForm.getAttribute('action'); + const response = await fetch(formAction, { + method: 'POST', + headers: { + 'X-CSRFToken': csrfToken, + 'Accept': 'application/json', + 'X-Requested-With': 'XMLHttpRequest' + }, + body: formData + }); + const contentType = response.headers.get('content-type') || ''; + if (!contentType.includes('application/json')) { + throw new Error('Server returned an unexpected response. Please try again.'); + } + const data = await response.json(); + if (!response.ok) { + throw new Error(data.error || 'Unable to delete object'); + } + if (deleteModal) deleteModal.hide(); + showMessage({ title: 'Object deleted', body: data.message || 'The object has been deleted.', variant: 'success' }); + previewEmpty.classList.remove('d-none'); + previewPanel.classList.add('d-none'); + activeRow = null; + loadObjects(false); + } catch (err) { + if (deleteModal) deleteModal.hide(); + showMessage({ title: 'Delete failed', body: err.message || 'Unable to delete object', variant: 'danger' }); + } finally { + if (submitBtn) { + submitBtn.disabled = false; + submitBtn.innerHTML = originalHtml; + } + } + }); + } + + const resetPreviewMedia = () => { + [previewImage, previewVideo, previewAudio, previewIframe].forEach((el) => { + if (!el) return; + el.classList.add('d-none'); + if (el.tagName === 'IMG') { + el.removeAttribute('src'); + el.onload = null; + } + if (el.tagName === 'VIDEO' || el.tagName === 'AUDIO') { + el.pause(); + el.removeAttribute('src'); + } + if (el.tagName === 'IFRAME') { + el.setAttribute('src', 'about:blank'); + } + }); + if (previewText) { + previewText.classList.add('d-none'); + previewText.textContent = ''; + } + previewPlaceholder.innerHTML = previewPlaceholderDefault; + previewPlaceholder.classList.remove('d-none'); + }; + + let previewFailed = false; + + const handlePreviewError = () => { + previewFailed = true; + if (downloadButton) { + downloadButton.classList.add('disabled'); + downloadButton.removeAttribute('href'); + } + if (presignButton) presignButton.disabled = true; + if (generatePresignButton) generatePresignButton.disabled = true; + if (previewDetailsMeta) previewDetailsMeta.classList.add('d-none'); + if (previewMetadata) previewMetadata.classList.add('d-none'); + const tagsPanel = document.getElementById('preview-tags'); + if (tagsPanel) tagsPanel.classList.add('d-none'); + const versionPanel = document.getElementById('version-panel'); + if (versionPanel) versionPanel.classList.add('d-none'); + if (previewErrorAlert) { + previewErrorAlert.textContent = 'Unable to load object \u2014 it may have been deleted, or the server returned an error.'; + previewErrorAlert.classList.remove('d-none'); + } + }; + + const clearPreviewError = () => { + previewFailed = false; + if (previewErrorAlert) previewErrorAlert.classList.add('d-none'); + if (previewDetailsMeta) previewDetailsMeta.classList.remove('d-none'); + }; + + async function fetchMetadata(metadataUrl) { + if (!metadataUrl) return null; + try { + const resp = await fetch(metadataUrl); + if (resp.ok) { + const data = await resp.json(); + return data.metadata || {}; + } + } catch (e) { + console.warn('Failed to load metadata', e); + } + return null; + } + + async function selectRow(row) { + document.querySelectorAll('[data-object-row]').forEach((r) => r.classList.remove('table-active')); + row.classList.add('table-active'); + previewEmpty.classList.add('d-none'); + previewPanel.classList.remove('d-none'); + activeRow = row; + renderMetadata(null); + clearPreviewError(); + + previewKey.textContent = row.dataset.key; + previewSize.textContent = formatBytes(Number(row.dataset.size)); + previewModified.textContent = row.dataset.lastModifiedIso || row.dataset.lastModified; + previewEtag.textContent = row.dataset.etag; + downloadButton.href = row.dataset.downloadUrl; + downloadButton.classList.remove('disabled'); + if (presignButton) { + presignButton.dataset.endpoint = row.dataset.presignEndpoint; + presignButton.disabled = false; + } + if (generatePresignButton) { + generatePresignButton.disabled = false; + } + updateGeneratePresignState(); + if (versioningEnabled) { + loadObjectVersions(row); + } + + resetPreviewMedia(); + const previewUrl = row.dataset.previewUrl; + const lower = row.dataset.key.toLowerCase(); + if (previewUrl && lower.match(/\.(png|jpg|jpeg|gif|webp|svg|ico|bmp)$/)) { + previewPlaceholder.innerHTML = '
Loading preview\u2026
'; + const currentRow = row; + fetch(previewUrl) + .then((r) => { + if (activeRow !== currentRow) return; + if (!r.ok) { + previewPlaceholder.innerHTML = '
Failed to load preview
'; + handlePreviewError(); + return; + } + return r.blob(); + }) + .then((blob) => { + if (!blob || activeRow !== currentRow) return; + const url = URL.createObjectURL(blob); + previewImage.onload = () => { + if (activeRow !== currentRow) { URL.revokeObjectURL(url); return; } + previewImage.classList.remove('d-none'); + previewPlaceholder.classList.add('d-none'); + }; + previewImage.onerror = () => { + if (activeRow !== currentRow) { URL.revokeObjectURL(url); return; } + URL.revokeObjectURL(url); + previewPlaceholder.innerHTML = '
Failed to load preview
'; + }; + previewImage.src = url; + }) + .catch(() => { + if (activeRow !== currentRow) return; + previewPlaceholder.innerHTML = '
Failed to load preview
'; + handlePreviewError(); + }); + } else if (previewUrl && lower.match(/\.(mp4|webm|ogv|mov|avi|mkv)$/)) { + const currentRow = row; + previewVideo.onerror = () => { + if (activeRow !== currentRow) return; + previewVideo.classList.add('d-none'); + previewPlaceholder.classList.remove('d-none'); + previewPlaceholder.innerHTML = '
Failed to load preview
'; + handlePreviewError(); + }; + previewVideo.src = previewUrl; + previewVideo.classList.remove('d-none'); + previewPlaceholder.classList.add('d-none'); + } else if (previewUrl && lower.match(/\.(mp3|wav|flac|ogg|aac|m4a|wma)$/)) { + const currentRow = row; + previewAudio.onerror = () => { + if (activeRow !== currentRow) return; + previewAudio.classList.add('d-none'); + previewPlaceholder.classList.remove('d-none'); + previewPlaceholder.innerHTML = '
Failed to load preview
'; + handlePreviewError(); + }; + previewAudio.src = previewUrl; + previewAudio.classList.remove('d-none'); + previewPlaceholder.classList.add('d-none'); + } else if (previewUrl && lower.match(/\.(pdf)$/)) { + const currentRow = row; + previewIframe.onerror = () => { + if (activeRow !== currentRow) return; + previewIframe.classList.add('d-none'); + previewPlaceholder.classList.remove('d-none'); + previewPlaceholder.innerHTML = '
Failed to load preview
'; + handlePreviewError(); + }; + previewIframe.src = previewUrl; + previewIframe.style.minHeight = '500px'; + previewIframe.classList.remove('d-none'); + previewPlaceholder.classList.add('d-none'); + } else if (previewUrl && previewText && lower.match(/\.(txt|log|json|md|csv|xml|html|htm|js|ts|py|java|c|cpp|h|css|scss|yaml|yml|toml|ini|cfg|conf|sh|bat|rs|go|rb|php|sql|r|swift|kt|scala|pl|lua|zig|ex|exs|hs|erl|ps1|psm1|psd1|fish|zsh|env|properties|gradle|makefile|dockerfile|vagrantfile|gitignore|gitattributes|editorconfig|eslintrc|prettierrc)$/)) { + previewText.textContent = 'Loading\u2026'; + previewText.classList.remove('d-none'); + previewPlaceholder.classList.add('d-none'); + const currentRow = row; + fetch(previewUrl) + .then((r) => { + if (!r.ok) throw new Error(r.statusText); + const len = parseInt(r.headers.get('Content-Length') || '0', 10); + if (len > 512 * 1024) { + return r.text().then((t) => t.slice(0, 512 * 1024) + '\n\n--- Truncated (file too large for preview) ---'); + } + return r.text(); + }) + .then((text) => { + if (activeRow !== currentRow) return; + previewText.textContent = text; + }) + .catch(() => { + if (activeRow !== currentRow) return; + previewText.classList.add('d-none'); + previewPlaceholder.classList.remove('d-none'); + previewPlaceholder.innerHTML = '
Failed to load preview
'; + handlePreviewError(); + }); + } + + const metadataUrl = row.dataset.metadataUrl; + if (metadataUrl) { + const metadata = await fetchMetadata(metadataUrl); + if (activeRow === row && !previewFailed) { + renderMetadata(metadata); + } + } + } + + updateBulkDeleteState(); + + function initFolderNavigation() { + if (hasFolders()) { + renderBreadcrumb(currentPrefix); + renderObjectsView(); + } + if (typeof updateFolderViewStatus === 'function') { + updateFolderViewStatus(); + } + if (typeof updateFilterWarning === 'function') { + updateFilterWarning(); + } + } + + bulkDeleteButton?.addEventListener('click', () => openBulkDeleteModal()); + bulkDeleteConfirm?.addEventListener('click', () => performBulkDelete()); + + const filterWarning = document.getElementById('filter-warning'); + const filterWarningText = document.getElementById('filter-warning-text'); + const folderViewStatus = document.getElementById('folder-view-status'); + + const updateFilterWarning = () => { + if (!filterWarning) return; + const isFiltering = currentFilterTerm.length > 0; + if (isFiltering && hasMoreObjects) { + filterWarning.classList.remove('d-none'); + } else { + filterWarning.classList.add('d-none'); + } + }; + + let searchDebounceTimer = null; + let searchAbortController = null; + let searchResults = null; + + const performServerSearch = async (term) => { + if (searchAbortController) searchAbortController.abort(); + searchAbortController = new AbortController(); + + try { + const params = new URLSearchParams({ q: term, limit: '500' }); + if (currentPrefix) params.set('prefix', currentPrefix); + const searchUrl = objectsStreamUrl.replace('/stream', '/search'); + const response = await fetch(`${searchUrl}?${params}`, { + signal: searchAbortController.signal + }); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + const data = await response.json(); + searchResults = (data.results || []).map(obj => processStreamObject(obj)); + memoizedVisibleItems = null; + memoizedInputs = { objectCount: -1, folderCount: -1, prefix: null, filterTerm: null }; + refreshVirtualList(); + if (loadMoreStatus) { + const countText = searchResults.length.toLocaleString(); + const truncated = data.truncated ? '+' : ''; + loadMoreStatus.textContent = `${countText}${truncated} result${searchResults.length !== 1 ? 's' : ''}`; + } + } catch (e) { + if (e.name === 'AbortError') return; + if (loadMoreStatus) { + loadMoreStatus.textContent = 'Search failed'; + } + } + }; + + document.getElementById('object-search')?.addEventListener('input', (event) => { + const newTerm = event.target.value.toLowerCase(); + const wasFiltering = currentFilterTerm.length > 0; + const isFiltering = newTerm.length > 0; + currentFilterTerm = newTerm; + + clearTimeout(searchDebounceTimer); + + if (isFiltering) { + searchDebounceTimer = setTimeout(() => performServerSearch(newTerm), 300); + return; + } + + if (!isFiltering && wasFiltering) { + if (searchAbortController) searchAbortController.abort(); + searchResults = null; + memoizedVisibleItems = null; + memoizedInputs = { objectCount: -1, folderCount: -1, prefix: null, filterTerm: null }; + if (loadMoreStatus) { + loadMoreStatus.textContent = buildBottomStatusText(streamingComplete); + } + } + + updateFilterWarning(); + refreshVirtualList(); + }); + + document.querySelectorAll('[data-sort-field]').forEach(el => { + el.addEventListener('click', (e) => { + e.preventDefault(); + const field = el.dataset.sortField; + const dir = el.dataset.sortDir || 'asc'; + currentSortField = field; + currentSortDir = dir; + document.querySelectorAll('[data-sort-field]').forEach(s => s.classList.remove('active')); + el.classList.add('active'); + var label = document.getElementById('sort-dropdown-label'); + if (label) label.textContent = el.textContent.trim(); + refreshVirtualList(); + }); + }); + + document.addEventListener('keydown', (e) => { + if (e.target.tagName === 'INPUT' || e.target.tagName === 'TEXTAREA' || e.target.tagName === 'SELECT' || e.target.isContentEditable) return; + + if (e.key === '/' && !e.ctrlKey && !e.metaKey) { + e.preventDefault(); + document.getElementById('object-search')?.focus(); + } + + if (e.key === '?' && !e.ctrlKey && !e.metaKey) { + e.preventDefault(); + var kbModal = document.getElementById('keyboardShortcutsModal'); + if (kbModal) { + var instance = bootstrap.Modal.getOrCreateInstance(kbModal); + instance.toggle(); + } + } + + if (e.key === 'Escape') { + var searchInput = document.getElementById('object-search'); + if (searchInput && document.activeElement === searchInput) { + searchInput.value = ''; + const wasFiltering = currentFilterTerm.length > 0; + currentFilterTerm = ''; + if (wasFiltering) { + clearTimeout(searchDebounceTimer); + if (searchAbortController) searchAbortController.abort(); + searchResults = null; + memoizedVisibleItems = null; + memoizedInputs = { objectCount: -1, folderCount: -1, prefix: null, filterTerm: null }; + if (loadMoreStatus) { + loadMoreStatus.textContent = buildBottomStatusText(streamingComplete); + } + } + refreshVirtualList(); + searchInput.blur(); + } + } + + if (e.key === 'Delete' && !e.ctrlKey && !e.metaKey) { + if (selectedRows.size > 0 && bulkDeleteButton && !bulkDeleteButton.disabled) { + bulkDeleteButton.click(); + } + } + + if (e.key === 'a' && (e.ctrlKey || e.metaKey)) { + if (visibleItems.length > 0 && selectAllCheckbox) { + e.preventDefault(); + selectAllCheckbox.checked = true; + selectAllCheckbox.dispatchEvent(new Event('change')); + } + } + }); + + const ctxMenu = document.getElementById('objectContextMenu'); + let ctxTargetRow = null; + + const hideContextMenu = () => { + if (ctxMenu) ctxMenu.classList.add('d-none'); + ctxTargetRow = null; + }; + + if (ctxMenu) { + document.addEventListener('click', hideContextMenu); + document.addEventListener('contextmenu', (e) => { + const row = e.target.closest('[data-object-row]'); + if (!row) { hideContextMenu(); return; } + e.preventDefault(); + ctxTargetRow = row; + + const x = Math.min(e.clientX, window.innerWidth - 200); + const y = Math.min(e.clientY, window.innerHeight - 200); + ctxMenu.style.left = x + 'px'; + ctxMenu.style.top = y + 'px'; + ctxMenu.classList.remove('d-none'); + }); + + ctxMenu.querySelectorAll('[data-ctx-action]').forEach(btn => { + btn.addEventListener('click', () => { + if (!ctxTargetRow) return; + const action = btn.dataset.ctxAction; + const key = ctxTargetRow.dataset.key; + const bucket = objectsContainer?.dataset.bucket || ''; + + if (action === 'download') { + const url = ctxTargetRow.dataset.downloadUrl; + if (url) window.open(url, '_blank'); + } else if (action === 'copy-path') { + const s3Path = 's3://' + bucket + '/' + key; + if (navigator.clipboard) { + navigator.clipboard.writeText(s3Path).then(() => { + if (window.showToast) window.showToast('Copied: ' + s3Path, 'Copied', 'success'); + }); + } + } else if (action === 'presign') { + selectRow(ctxTargetRow); + presignLink.value = ''; + presignModal?.show(); + requestPresignedUrl(); + } else if (action === 'delete') { + const deleteEndpoint = ctxTargetRow.dataset.deleteEndpoint; + if (deleteEndpoint) { + selectRow(ctxTargetRow); + const deleteModalEl = document.getElementById('deleteObjectModal'); + const deleteModal = deleteModalEl ? bootstrap.Modal.getOrCreateInstance(deleteModalEl) : null; + const deleteObjectForm = document.getElementById('deleteObjectForm'); + const deleteObjectKey = document.getElementById('deleteObjectKey'); + if (deleteModal && deleteObjectForm) { + deleteObjectForm.setAttribute('action', deleteEndpoint); + if (deleteObjectKey) deleteObjectKey.textContent = key; + deleteModal.show(); + } + } + } + hideContextMenu(); + }); + }); + } + + refreshVersionsButton?.addEventListener('click', () => { + if (!activeRow) { + versionList.innerHTML = '

Select an object to view versions.

'; + return; + } + const endpoint = activeRow.dataset.versionsEndpoint; + if (endpoint) { + versionsCache.delete(endpoint); + } + loadObjectVersions(activeRow, { force: true }); + }); + + presignButton?.addEventListener('click', () => { + if (!activeRow) { + showMessage({ title: 'Select an object', body: 'Choose an object before generating a presigned URL.', variant: 'warning' }); + return; + } + presignLink.value = ''; + presignModal?.show(); + requestPresignedUrl(); + }); + + generatePresignButton?.addEventListener('click', () => { + requestPresignedUrl(); + }); + + copyPresignLink?.addEventListener('click', async () => { + if (!presignLink?.value) { + return; + } + + const fallbackCopy = (text) => { + const textArea = document.createElement('textarea'); + textArea.value = text; + textArea.style.position = 'fixed'; + textArea.style.left = '-999999px'; + textArea.style.top = '-999999px'; + document.body.appendChild(textArea); + textArea.focus(); + textArea.select(); + let success = false; + try { + success = document.execCommand('copy'); + } catch (err) { + success = false; + } + textArea.remove(); + return success; + }; + + let copied = false; + + if (navigator.clipboard && window.isSecureContext) { + try { + await navigator.clipboard.writeText(presignLink.value); + copied = true; + } catch (error) { + + } + } + + if (!copied) { + copied = fallbackCopy(presignLink.value); + } + + if (copied) { + copyPresignLink.textContent = 'Copied!'; + window.setTimeout(() => { + copyPresignLink.textContent = copyPresignDefaultLabel; + }, 1500); + } else { + showMessage({ title: 'Copy Failed', body: 'Unable to copy link to clipboard. Please select the link and copy manually.', variant: 'warning' }); + } + }); + + if (uploadForm && uploadFileInput) { + const uploadSubmitBtn = document.getElementById('uploadSubmitBtn'); + const uploadCancelBtn = document.getElementById('uploadCancelBtn'); + const uploadBtnText = document.getElementById('uploadBtnText'); + const bulkUploadProgress = document.getElementById('bulkUploadProgress'); + const bulkUploadStatus = document.getElementById('bulkUploadStatus'); + const bulkUploadCounter = document.getElementById('bulkUploadCounter'); + const bulkUploadProgressBar = document.getElementById('bulkUploadProgressBar'); + const bulkUploadCurrentFile = document.getElementById('bulkUploadCurrentFile'); + const bulkUploadResults = document.getElementById('bulkUploadResults'); + const bulkUploadSuccessAlert = document.getElementById('bulkUploadSuccessAlert'); + const bulkUploadErrorAlert = document.getElementById('bulkUploadErrorAlert'); + const bulkUploadSuccessCount = document.getElementById('bulkUploadSuccessCount'); + const bulkUploadErrorCount = document.getElementById('bulkUploadErrorCount'); + const bulkUploadErrorList = document.getElementById('bulkUploadErrorList'); + const uploadKeyPrefix = document.getElementById('uploadKeyPrefix'); + const singleFileOptions = document.getElementById('singleFileOptions'); + const floatingProgress = document.getElementById('floatingUploadProgress'); + const floatingProgressBar = document.getElementById('floatingUploadProgressBar'); + const floatingProgressStatus = document.getElementById('floatingUploadStatus'); + const floatingProgressTitle = document.getElementById('floatingUploadTitle'); + const floatingProgressExpand = document.getElementById('floatingUploadExpand'); + const floatingProgressCancel = document.getElementById('floatingUploadCancel'); + const uploadQueueContainer = document.getElementById('uploadQueueContainer'); + const uploadQueueList = document.getElementById('uploadQueueList'); + const uploadQueueCount = document.getElementById('uploadQueueCount'); + const clearUploadQueueBtn = document.getElementById('clearUploadQueueBtn'); + let isUploading = false; + let uploadQueue = []; + let activeXHRs = []; + let activeMultipartUpload = null; + let uploadCancelled = false; + let uploadStats = { + totalFiles: 0, + completedFiles: 0, + totalBytes: 0, + uploadedBytes: 0, + currentFileBytes: 0, + currentFileLoaded: 0, + currentFileName: '' + }; + + window.addEventListener('beforeunload', (e) => { + if (isUploading) { + e.preventDefault(); + e.returnValue = 'Upload in progress. Are you sure you want to leave?'; + return e.returnValue; + } + }); + + const showFloatingProgress = () => { + if (floatingProgress) { + floatingProgress.classList.remove('d-none'); + } + }; + + const hideFloatingProgress = () => { + if (floatingProgress) { + floatingProgress.classList.add('d-none'); + } + }; + + const updateFloatingProgress = () => { + const { totalFiles, completedFiles, totalBytes, uploadedBytes, currentFileLoaded, currentFileName } = uploadStats; + const effectiveUploaded = uploadedBytes + currentFileLoaded; + + if (floatingProgressBar && totalBytes > 0) { + const percent = Math.round((effectiveUploaded / totalBytes) * 100); + floatingProgressBar.style.width = `${percent}%`; + } + if (floatingProgressStatus) { + const bytesText = `${formatBytes(effectiveUploaded)} / ${formatBytes(totalBytes)}`; + const queuedCount = uploadQueue.length; + let statusText = `${completedFiles}/${totalFiles} files`; + if (queuedCount > 0) { + statusText += ` (+${queuedCount} queued)`; + } + statusText += ` • ${bytesText}`; + floatingProgressStatus.textContent = statusText; + } + if (floatingProgressTitle) { + const remaining = totalFiles - completedFiles; + const queuedCount = uploadQueue.length; + let title = `Uploading ${remaining} file${remaining !== 1 ? 's' : ''}`; + if (queuedCount > 0) { + title += ` (+${queuedCount} queued)`; + } + floatingProgressTitle.textContent = title + '...'; + } + }; + + floatingProgressExpand?.addEventListener('click', () => { + if (uploadModal) { + uploadModal.show(); + } + }); + + const cancelAllUploads = async () => { + uploadCancelled = true; + + activeXHRs.forEach(xhr => { + try { xhr.abort(); } catch { } + }); + activeXHRs = []; + + if (activeMultipartUpload) { + const { abortUrl } = activeMultipartUpload; + const csrfToken = document.querySelector('input[name="csrf_token"]')?.value; + try { + await fetch(abortUrl, { method: 'DELETE', headers: { 'X-CSRFToken': csrfToken || '' } }); + } catch { } + activeMultipartUpload = null; + } + + uploadQueue = []; + isProcessingQueue = false; + isUploading = false; + setUploadLockState(false); + hideFloatingProgress(); + resetUploadUI(); + + showMessage({ title: 'Upload cancelled', body: 'All uploads have been cancelled.', variant: 'info' }); + loadObjects(false); + }; + + floatingProgressCancel?.addEventListener('click', () => { + cancelAllUploads(); + }); + + const refreshUploadDropLabel = () => { + if (!uploadDropZoneLabel) return; + if (isUploading) { + uploadDropZoneLabel.textContent = 'Drop files here to add to queue'; + if (singleFileOptions) singleFileOptions.classList.add('d-none'); + return; + } + const files = uploadFileInput.files; + if (!files || files.length === 0) { + uploadDropZoneLabel.textContent = 'No file selected'; + if (singleFileOptions) singleFileOptions.classList.remove('d-none'); + return; + } + uploadDropZoneLabel.textContent = files.length === 1 ? files[0].name : `${files.length} files selected`; + + if (singleFileOptions) { + singleFileOptions.classList.toggle('d-none', files.length > 1); + } + }; + + const updateUploadBtnText = () => { + if (!uploadBtnText) return; + if (isUploading) { + const files = uploadFileInput.files; + if (files && files.length > 0) { + uploadBtnText.textContent = `Add ${files.length} to queue`; + if (uploadSubmitBtn) uploadSubmitBtn.disabled = false; + } else { + uploadBtnText.textContent = 'Uploading...'; + } + return; + } + const files = uploadFileInput.files; + if (!files || files.length <= 1) { + uploadBtnText.textContent = 'Upload'; + } else { + uploadBtnText.textContent = `Upload ${files.length} files`; + } + }; + + const resetUploadUI = () => { + if (bulkUploadProgress) bulkUploadProgress.classList.add('d-none'); + if (bulkUploadResults) bulkUploadResults.classList.add('d-none'); + if (bulkUploadSuccessAlert) bulkUploadSuccessAlert.classList.remove('d-none'); + if (bulkUploadErrorAlert) bulkUploadErrorAlert.classList.add('d-none'); + if (bulkUploadErrorList) bulkUploadErrorList.innerHTML = ''; + if (uploadSubmitBtn) uploadSubmitBtn.disabled = false; + if (uploadFileInput) uploadFileInput.disabled = false; + const progressStack = document.querySelector('[data-upload-progress]'); + if (progressStack) progressStack.innerHTML = ''; + if (uploadDropZone) { + uploadDropZone.classList.remove('upload-locked'); + uploadDropZone.style.pointerEvents = ''; + } + isUploading = false; + hideFloatingProgress(); + }; + + const MULTIPART_THRESHOLD = 8 * 1024 * 1024; + const CHUNK_SIZE = 8 * 1024 * 1024; + const uploadProgressStack = document.querySelector('[data-upload-progress]'); + const multipartInitUrl = uploadForm.dataset.multipartInitUrl; + const multipartPartTemplate = uploadForm.dataset.multipartPartTemplate; + const multipartCompleteTemplate = uploadForm.dataset.multipartCompleteTemplate; + const multipartAbortTemplate = uploadForm.dataset.multipartAbortTemplate; + + const createProgressItem = (file) => { + const item = document.createElement('div'); + item.className = 'upload-progress-item'; + item.dataset.state = 'uploading'; + item.innerHTML = ` +
+
+
${escapeHtml(file.name)}
+
${formatBytes(file.size)}
+
+
Preparing...
+
+
+
+
+
+
+ 0 B + 0% +
+
+ `; + return item; + }; + + const updateProgressItem = (item, { loaded, total, status, state, error }) => { + if (state) item.dataset.state = state; + const statusEl = item.querySelector('.upload-status'); + const progressBar = item.querySelector('.progress-bar'); + const progressLoaded = item.querySelector('.progress-loaded'); + const progressPercent = item.querySelector('.progress-percent'); + + if (status) { + statusEl.textContent = status; + statusEl.className = 'upload-status text-end ms-2'; + if (state === 'success') statusEl.classList.add('success'); + if (state === 'error') statusEl.classList.add('error'); + } + if (typeof loaded === 'number' && typeof total === 'number' && total > 0) { + const percent = Math.round((loaded / total) * 100); + progressBar.style.width = `${percent}%`; + progressLoaded.textContent = `${formatBytes(loaded)} / ${formatBytes(total)}`; + progressPercent.textContent = `${percent}%`; + } + if (error) { + const progressContainer = item.querySelector('.progress-container'); + if (progressContainer) { + progressContainer.innerHTML = `
${escapeHtml(error)}
`; + } + } + }; + + const uploadMultipart = async (file, objectKey, metadata, progressItem) => { + const csrfToken = document.querySelector('input[name="csrf_token"]')?.value; + + if (uploadCancelled) throw new Error('Upload cancelled'); + + updateProgressItem(progressItem, { status: 'Initiating...', loaded: 0, total: file.size }); + const initResp = await fetch(multipartInitUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': csrfToken || '' }, + body: JSON.stringify({ object_key: objectKey, metadata }) + }); + if (!initResp.ok) { + const err = await initResp.json().catch(() => ({})); + throw new Error(err.error || 'Failed to initiate upload'); + } + const { upload_id } = await initResp.json(); + + const partUrl = multipartPartTemplate.replace('UPLOAD_ID_PLACEHOLDER', upload_id); + const completeUrl = multipartCompleteTemplate.replace('UPLOAD_ID_PLACEHOLDER', upload_id); + const abortUrl = multipartAbortTemplate.replace('UPLOAD_ID_PLACEHOLDER', upload_id); + + activeMultipartUpload = { upload_id, abortUrl }; + + const parts = []; + const totalParts = Math.ceil(file.size / CHUNK_SIZE); + let uploadedBytes = 0; + + try { + for (let partNumber = 1; partNumber <= totalParts; partNumber++) { + if (uploadCancelled) throw new Error('Upload cancelled'); + + const start = (partNumber - 1) * CHUNK_SIZE; + const end = Math.min(start + CHUNK_SIZE, file.size); + const chunk = file.slice(start, end); + + updateProgressItem(progressItem, { + status: `Part ${partNumber}/${totalParts}`, + loaded: uploadedBytes, + total: file.size + }); + uploadStats.currentFileLoaded = uploadedBytes; + updateFloatingProgress(); + + const partResp = await fetch(`${partUrl}?partNumber=${partNumber}`, { + method: 'PUT', + headers: { + 'X-CSRFToken': csrfToken || '', + 'Content-Type': 'application/octet-stream' + }, + body: chunk + }); + + if (uploadCancelled) throw new Error('Upload cancelled'); + + if (!partResp.ok) { + const err = await partResp.json().catch(() => ({})); + throw new Error(err.error || `Part ${partNumber} failed`); + } + + const partData = await partResp.json(); + parts.push({ part_number: partNumber, etag: partData.etag }); + uploadedBytes += chunk.size; + + updateProgressItem(progressItem, { + loaded: uploadedBytes, + total: file.size + }); + uploadStats.currentFileLoaded = uploadedBytes; + updateFloatingProgress(); + } + + updateProgressItem(progressItem, { status: 'Completing...', loaded: file.size, total: file.size }); + const completeResp = await fetch(completeUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': csrfToken || '' }, + body: JSON.stringify({ parts }) + }); + + if (!completeResp.ok) { + const err = await completeResp.json().catch(() => ({})); + throw new Error(err.error || 'Failed to complete upload'); + } + + activeMultipartUpload = null; + return await completeResp.json(); + } catch (err) { + if (!uploadCancelled) { + try { + await fetch(abortUrl, { method: 'DELETE', headers: { 'X-CSRFToken': csrfToken || '' } }); + } catch { } + } + activeMultipartUpload = null; + throw err; + } + }; + + const uploadRegular = async (file, objectKey, metadata, progressItem) => { + return new Promise((resolve, reject) => { + const formData = new FormData(); + formData.append('object', file); + formData.append('object_key', objectKey); + if (metadata) formData.append('metadata', JSON.stringify(metadata)); + const csrfToken = document.querySelector('input[name="csrf_token"]')?.value; + if (csrfToken) formData.append('csrf_token', csrfToken); + + const xhr = new XMLHttpRequest(); + activeXHRs.push(xhr); + xhr.open('POST', uploadForm.action, true); + xhr.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); + xhr.setRequestHeader('X-CSRFToken', csrfToken || ''); + + const removeXHR = () => { + const idx = activeXHRs.indexOf(xhr); + if (idx > -1) activeXHRs.splice(idx, 1); + }; + + xhr.upload.addEventListener('progress', (e) => { + if (e.lengthComputable) { + updateProgressItem(progressItem, { + status: 'Uploading...', + loaded: e.loaded, + total: e.total + }); + uploadStats.currentFileLoaded = e.loaded; + updateFloatingProgress(); + } + }); + + xhr.addEventListener('load', () => { + removeXHR(); + if (xhr.status >= 200 && xhr.status < 300) { + try { + const data = JSON.parse(xhr.responseText); + if (data.status === 'error') { + reject(new Error(data.message || 'Upload failed')); + } else { + resolve(data); + } + } catch { + resolve({}); + } + } else { + try { + const data = JSON.parse(xhr.responseText); + reject(new Error(data.message || `Upload failed (${xhr.status})`)); + } catch { + reject(new Error(`Upload failed (${xhr.status})`)); + } + } + }); + + xhr.addEventListener('error', () => { removeXHR(); reject(new Error('Network error')); }); + xhr.addEventListener('abort', () => { removeXHR(); reject(new Error('Upload cancelled')); }); + + xhr.send(formData); + }); + }; + + const uploadSingleFile = async (file, keyPrefix = '', metadata = null, progressItem = null) => { + const objectKey = keyPrefix ? `${keyPrefix}${file.name}` : file.name; + const shouldUseMultipart = file.size >= MULTIPART_THRESHOLD && multipartInitUrl; + + if (!progressItem && uploadProgressStack) { + progressItem = createProgressItem(file); + uploadProgressStack.appendChild(progressItem); + } + + try { + let result; + if (shouldUseMultipart) { + updateProgressItem(progressItem, { status: 'Multipart upload...', loaded: 0, total: file.size }); + result = await uploadMultipart(file, objectKey, metadata, progressItem); + } else { + updateProgressItem(progressItem, { status: 'Uploading...', loaded: 0, total: file.size }); + result = await uploadRegular(file, objectKey, metadata, progressItem); + } + updateProgressItem(progressItem, { state: 'success', status: 'Complete', loaded: file.size, total: file.size }); + return result; + } catch (err) { + updateProgressItem(progressItem, { state: 'error', status: 'Failed', error: err.message }); + throw err; + } + }; + + const setUploadLockState = (locked) => { + if (uploadDropZone) { + uploadDropZone.classList.toggle('upload-locked', locked); + } + }; + + let uploadSuccessFiles = []; + let uploadErrorFiles = []; + let isProcessingQueue = false; + + const updateQueueListDisplay = () => { + if (!uploadQueueList || !uploadQueueContainer || !uploadQueueCount) return; + if (uploadQueue.length === 0) { + uploadQueueContainer.classList.add('d-none'); + return; + } + uploadQueueContainer.classList.remove('d-none'); + uploadQueueCount.textContent = uploadQueue.length; + uploadQueueList.innerHTML = uploadQueue.map((item, idx) => ` +
  • + + + + + ${escapeHtml(item.file.name)} + + ${formatBytes(item.file.size)} +
    • + `).join(''); + }; + + const addFilesToQueue = (files, keyPrefix, metadata) => { + for (const file of files) { + uploadQueue.push({ file, keyPrefix, metadata }); + uploadStats.totalFiles++; + uploadStats.totalBytes += file.size; + } + updateFloatingProgress(); + updateQueueListDisplay(); + }; + + const clearUploadQueue = () => { + const clearedCount = uploadQueue.length; + if (clearedCount === 0) return; + for (const item of uploadQueue) { + uploadStats.totalFiles--; + uploadStats.totalBytes -= item.file.size; + } + uploadQueue.length = 0; + updateFloatingProgress(); + updateQueueListDisplay(); + }; + + if (clearUploadQueueBtn) { + clearUploadQueueBtn.addEventListener('click', clearUploadQueue); + } + + const processUploadQueue = async () => { + if (isProcessingQueue) return; + isProcessingQueue = true; + + while (uploadQueue.length > 0 && !uploadCancelled) { + const item = uploadQueue.shift(); + const { file, keyPrefix, metadata } = item; + updateQueueListDisplay(); + + uploadStats.currentFileName = file.name; + uploadStats.currentFileBytes = file.size; + uploadStats.currentFileLoaded = 0; + + if (bulkUploadCounter) { + const queuedCount = uploadQueue.length; + let counterText = `${uploadStats.completedFiles + 1}/${uploadStats.totalFiles}`; + if (queuedCount > 0) { + counterText += ` (+${queuedCount} queued)`; + } + bulkUploadCounter.textContent = counterText; + } + if (bulkUploadCurrentFile) { + bulkUploadCurrentFile.textContent = `Uploading: ${file.name}`; + } + if (bulkUploadProgressBar) { + const percent = Math.round(((uploadStats.completedFiles + 1) / uploadStats.totalFiles) * 100); + bulkUploadProgressBar.style.width = `${percent}%`; + } + updateFloatingProgress(); + + try { + await uploadSingleFile(file, keyPrefix, metadata); + uploadSuccessFiles.push(file.name); + } catch (error) { + uploadErrorFiles.push({ name: file.name, error: error.message || 'Unknown error' }); + } + + uploadStats.uploadedBytes += file.size; + uploadStats.completedFiles++; + uploadStats.currentFileLoaded = 0; + updateFloatingProgress(); + } + + isProcessingQueue = false; + + if (uploadQueue.length === 0 && !uploadCancelled) { + finishUploadSession(); + } + }; + + const finishUploadSession = () => { + if (bulkUploadProgress) bulkUploadProgress.classList.add('d-none'); + if (bulkUploadResults) bulkUploadResults.classList.remove('d-none'); + hideFloatingProgress(); + + if (bulkUploadSuccessCount) bulkUploadSuccessCount.textContent = uploadSuccessFiles.length; + if (uploadSuccessFiles.length === 0 && bulkUploadSuccessAlert) { + bulkUploadSuccessAlert.classList.add('d-none'); + } + + if (uploadErrorFiles.length > 0) { + if (bulkUploadErrorCount) bulkUploadErrorCount.textContent = uploadErrorFiles.length; + if (bulkUploadErrorAlert) bulkUploadErrorAlert.classList.remove('d-none'); + if (bulkUploadErrorList) { + bulkUploadErrorList.innerHTML = uploadErrorFiles + .map(f => `
  • ${escapeHtml(f.name)}: ${escapeHtml(f.error)}
    • `) + .join(''); + } + } + + isUploading = false; + setUploadLockState(false); + refreshUploadDropLabel(); + updateUploadBtnText(); + updateQueueListDisplay(); + + if (uploadSubmitBtn) uploadSubmitBtn.disabled = false; + if (uploadFileInput) { + uploadFileInput.disabled = false; + uploadFileInput.value = ''; + } + + const previousKey = activeRow?.dataset.key || null; + loadObjects(false).then(() => { + if (previousKey) { + const newRow = document.querySelector(`[data-object-row][data-key="${CSS.escape(previousKey)}"]`); + if (newRow) { + selectRow(newRow); + if (versioningEnabled) loadObjectVersions(newRow, { force: true }); + } + } + }); + + const successCount = uploadSuccessFiles.length; + const errorCount = uploadErrorFiles.length; + if (successCount > 0 && errorCount > 0) { + showMessage({ title: 'Upload complete', body: `${successCount} uploaded, ${errorCount} failed.`, variant: 'warning' }); + } else if (successCount > 0) { + showMessage({ title: 'Upload complete', body: `${successCount} object(s) uploaded successfully.`, variant: 'success' }); + } else if (errorCount > 0) { + showMessage({ title: 'Upload failed', body: `${errorCount} file(s) failed to upload.`, variant: 'danger' }); + } + }; + + const performBulkUpload = async (files) => { + if (!files || files.length === 0) return; + + const keyPrefix = (uploadKeyPrefix?.value || '').trim(); + const metadataRaw = uploadForm.querySelector('textarea[name="metadata"]')?.value?.trim(); + let metadata = null; + if (metadataRaw) { + try { + metadata = JSON.parse(metadataRaw); + } catch { + showMessage({ title: 'Invalid metadata', body: 'Metadata must be valid JSON.', variant: 'danger' }); + return; + } + } + + if (!isUploading) { + isUploading = true; + uploadCancelled = false; + uploadSuccessFiles = []; + uploadErrorFiles = []; + uploadStats = { + totalFiles: 0, + completedFiles: 0, + totalBytes: 0, + uploadedBytes: 0, + currentFileBytes: 0, + currentFileLoaded: 0, + currentFileName: '' + }; + + if (bulkUploadProgress) bulkUploadProgress.classList.remove('d-none'); + if (bulkUploadResults) bulkUploadResults.classList.add('d-none'); + if (uploadSubmitBtn) uploadSubmitBtn.disabled = true; + refreshUploadDropLabel(); + updateUploadBtnText(); + + if (uploadModal) uploadModal.hide(); + showFloatingProgress(); + } + + const fileCount = files.length; + addFilesToQueue(Array.from(files), keyPrefix, metadata); + + if (uploadFileInput) { + uploadFileInput.value = ''; + } + refreshUploadDropLabel(); + updateUploadBtnText(); + + processUploadQueue(); + }; + + refreshUploadDropLabel(); + uploadFileInput.addEventListener('change', () => { + refreshUploadDropLabel(); + updateUploadBtnText(); + if (!isUploading) { + resetUploadUI(); + } + }); + uploadDropZone?.addEventListener('click', () => { + uploadFileInput?.click(); + }); + + uploadForm.addEventListener('submit', async (event) => { + event.preventDefault(); + const files = uploadFileInput.files; + if (!files || files.length === 0) return; + + const keyPrefix = (uploadKeyPrefix?.value || '').trim(); + + if (uploadSubmitBtn) { + uploadSubmitBtn.disabled = true; + if (uploadBtnText) uploadBtnText.textContent = 'Uploading...'; + } + + await performBulkUpload(Array.from(files)); + }); + + uploadModalEl?.addEventListener('show.bs.modal', () => { + if (hasFolders() && currentPrefix) { + uploadKeyPrefix.value = currentPrefix; + + const advancedToggle = document.querySelector('[data-bs-target="#advancedUploadOptions"]'); + const advancedCollapse = document.getElementById('advancedUploadOptions'); + if (advancedToggle && advancedCollapse && !advancedCollapse.classList.contains('show')) { + new bootstrap.Collapse(advancedCollapse, { show: true }); + } + } else if (uploadKeyPrefix) { + + uploadKeyPrefix.value = ''; + } + }); + + uploadModalEl?.addEventListener('hide.bs.modal', (event) => { + if (isUploading) { + showFloatingProgress(); + } + }); + + uploadModalEl?.addEventListener('hidden.bs.modal', () => { + if (!isUploading) { + resetUploadUI(); + uploadFileInput.value = ''; + refreshUploadDropLabel(); + updateUploadBtnText(); + } + }); + + uploadModalEl?.addEventListener('show.bs.modal', () => { + if (isUploading) { + hideFloatingProgress(); + } + }); + + const preventDefaults = (event) => { + event.preventDefault(); + event.stopPropagation(); + }; + + const wireDropTarget = (target, { highlightClass = '', autoOpenModal = false } = {}) => { + if (!target) return; + ['dragenter', 'dragover'].forEach((eventName) => { + target.addEventListener(eventName, (event) => { + preventDefaults(event); + if (highlightClass) { + target.classList.add(highlightClass); + } + }); + }); + ['dragleave', 'drop'].forEach((eventName) => { + target.addEventListener(eventName, (event) => { + preventDefaults(event); + if (highlightClass) { + target.classList.remove(highlightClass); + } + }); + }); + target.addEventListener('drop', (event) => { + if (!event.dataTransfer?.files?.length) { + return; + } + if (isUploading) { + performBulkUpload(event.dataTransfer.files); + } else { + if (uploadFileInput) { + uploadFileInput.files = event.dataTransfer.files; + uploadFileInput.dispatchEvent(new Event('change', { bubbles: true })); + } + if (autoOpenModal && uploadModal) { + uploadModal.show(); + } + } + }); + }; + + if (uploadDropZone) { + wireDropTarget(uploadDropZone, { highlightClass: 'is-dragover' }); + } + + if (objectsContainer) { + wireDropTarget(objectsContainer, { highlightClass: 'drag-over', autoOpenModal: true }); + } + } + + const bulkDownloadButton = document.querySelector('[data-bulk-download-trigger]'); + const bulkDownloadEndpoint = document.getElementById('objects-drop-zone')?.dataset.bulkDownloadEndpoint; + + const updateBulkDownloadState = () => { + if (!bulkDownloadButton) return; + const selectedCount = document.querySelectorAll('[data-object-select]:checked').length; + bulkDownloadButton.disabled = selectedCount === 0; + }; + + selectAllCheckbox?.addEventListener('change', (event) => { + const shouldSelect = Boolean(event.target?.checked); + + const filesInView = visibleItems.filter(item => item.type === 'file'); + + filesInView.forEach(item => { + if (shouldSelect) { + selectedRows.set(item.data.key, item.data); + } else { + selectedRows.delete(item.data.key); + } + }); + + const foldersInView = visibleItems.filter(item => item.type === 'folder'); + foldersInView.forEach(item => { + if (shouldSelect) { + selectedRows.set(item.path, { key: item.path, isFolder: true }); + } else { + selectedRows.delete(item.path); + } + }); + + document.querySelectorAll('[data-folder-select]').forEach(cb => { + cb.checked = shouldSelect; + }); + + document.querySelectorAll('[data-object-row]').forEach((row) => { + const checkbox = row.querySelector('[data-object-select]'); + if (checkbox) { + checkbox.checked = shouldSelect; + } + }); + + updateBulkDeleteState(); + setTimeout(updateBulkDownloadState, 0); + }); + + bulkDownloadButton?.addEventListener('click', async () => { + if (!bulkDownloadEndpoint) return; + const selected = Array.from(selectedRows.keys()); + if (selected.length === 0) return; + + bulkDownloadButton.disabled = true; + const originalHtml = bulkDownloadButton.innerHTML; + bulkDownloadButton.innerHTML = ' Downloading...'; + + try { + const response = await fetch(bulkDownloadEndpoint, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '', + }, + body: JSON.stringify({ keys: selected }), + }); + + if (!response.ok) { + const data = await response.json().catch(() => ({})); + throw new Error(data.error || 'Download failed'); + } + + const blob = await response.blob(); + const url = window.URL.createObjectURL(blob); + const a = document.createElement('a'); + a.href = url; + a.download = `${document.getElementById('objects-drop-zone').dataset.bucket}-download.zip`; + document.body.appendChild(a); + a.click(); + window.URL.revokeObjectURL(url); + a.remove(); + } catch (error) { + showMessage({ title: 'Download Failed', body: error.message, variant: 'danger' }); + } finally { + bulkDownloadButton.disabled = false; + bulkDownloadButton.innerHTML = originalHtml; + } + }); + + const replicationStatsContainer = document.getElementById('replication-stats-cards'); + if (replicationStatsContainer) { + const statusEndpoint = replicationStatsContainer.dataset.statusEndpoint; + const syncedEl = replicationStatsContainer.querySelector('[data-stat="synced"]'); + const pendingEl = replicationStatsContainer.querySelector('[data-stat="pending"]'); + const orphanedEl = replicationStatsContainer.querySelector('[data-stat="orphaned"]'); + const bytesEl = replicationStatsContainer.querySelector('[data-stat="bytes"]'); + const lastSyncEl = document.getElementById('replication-last-sync'); + const lastSyncTimeEl = document.querySelector('[data-stat="last-sync-time"]'); + const lastSyncKeyEl = document.querySelector('[data-stat="last-sync-key"]'); + const endpointWarning = document.getElementById('replication-endpoint-warning'); + const endpointErrorEl = document.getElementById('replication-endpoint-error'); + const statusAlert = document.getElementById('replication-status-alert'); + const statusBadge = document.getElementById('replication-status-badge'); + const statusText = document.getElementById('replication-status-text'); + const pauseForm = document.getElementById('pause-replication-form'); + + const loadReplicationStats = async () => { + try { + const resp = await fetch(statusEndpoint); + if (!resp.ok) throw new Error('Failed to fetch stats'); + const data = await resp.json(); + + // Handle endpoint health status + if (data.endpoint_healthy === false) { + // Show warning and hide success alert + if (endpointWarning) { + endpointWarning.classList.remove('d-none'); + if (endpointErrorEl && data.endpoint_error) { + endpointErrorEl.textContent = data.endpoint_error + '. Replication is paused until the endpoint is available.'; + } + } + if (statusAlert) statusAlert.classList.add('d-none'); + + // Update status badge to show "Paused" with warning styling + if (statusBadge) { + statusBadge.className = 'badge bg-warning-subtle text-warning px-3 py-2'; + statusBadge.innerHTML = ` + + + + Paused (Endpoint Unavailable)`; + } + + // Hide the pause button since replication is effectively already paused + if (pauseForm) pauseForm.classList.add('d-none'); + } else { + // Hide warning and show success alert + if (endpointWarning) endpointWarning.classList.add('d-none'); + if (statusAlert) statusAlert.classList.remove('d-none'); + + // Restore status badge to show "Enabled" + if (statusBadge) { + statusBadge.className = 'badge bg-success-subtle text-success px-3 py-2'; + statusBadge.innerHTML = ` + + + + Enabled`; + } + + // Show the pause button + if (pauseForm) pauseForm.classList.remove('d-none'); + } + + if (syncedEl) syncedEl.textContent = data.objects_synced; + if (pendingEl) { + pendingEl.textContent = data.objects_pending; + if (data.objects_pending > 0) pendingEl.classList.add('text-warning'); + } + if (orphanedEl) orphanedEl.textContent = data.objects_orphaned; + if (bytesEl) bytesEl.textContent = formatBytes(data.bytes_synced); + + if (data.last_sync_at && lastSyncEl) { + lastSyncEl.style.display = ''; + const date = new Date(data.last_sync_at * 1000); + if (lastSyncTimeEl) lastSyncTimeEl.textContent = date.toLocaleString(); + if (lastSyncKeyEl && data.last_sync_key) { + lastSyncKeyEl.innerHTML = ' — ' + escapeHtml(data.last_sync_key) + ''; + } + } + } catch (err) { + console.error('Failed to load replication stats:', err); + if (syncedEl) syncedEl.textContent = '—'; + if (pendingEl) pendingEl.textContent = '—'; + if (orphanedEl) orphanedEl.textContent = '—'; + if (bytesEl) bytesEl.textContent = '—'; + } + }; + + loadReplicationStats(); + + if (window.pollingManager) { + window.pollingManager.start('replication', loadReplicationStats); + } + + const refreshBtn = document.querySelector('[data-refresh-replication]'); + refreshBtn?.addEventListener('click', () => { + + if (syncedEl) syncedEl.innerHTML = ''; + if (pendingEl) pendingEl.innerHTML = ''; + if (orphanedEl) orphanedEl.innerHTML = ''; + if (bytesEl) bytesEl.innerHTML = ''; + loadReplicationStats(); + loadReplicationFailures(); + }); + + const failuresCard = document.getElementById('replication-failures-card'); + const failuresBody = document.getElementById('replication-failures-body'); + const failureCountBadge = document.getElementById('replication-failure-count'); + const retryAllBtn = document.getElementById('retry-all-failures-btn'); + const clearFailuresBtn = document.getElementById('clear-failures-btn'); + const showMoreFailuresBtn = document.getElementById('show-more-failures'); + const failuresPagination = document.getElementById('replication-failures-pagination'); + const failuresShownCount = document.getElementById('failures-shown-count'); + const clearFailuresModal = document.getElementById('clearFailuresModal'); + const confirmClearFailuresBtn = document.getElementById('confirmClearFailuresBtn'); + const clearFailuresModalInstance = clearFailuresModal ? new bootstrap.Modal(clearFailuresModal) : null; + + let failuresExpanded = false; + let currentFailures = []; + + const loadReplicationFailures = async () => { + if (!failuresCard) return; + + const endpoint = failuresCard.dataset.failuresEndpoint; + const limit = failuresExpanded ? 50 : 5; + + try { + const resp = await fetch(`${endpoint}?limit=${limit}`); + if (!resp.ok) throw new Error('Failed to fetch failures'); + const data = await resp.json(); + + currentFailures = data.failures; + const total = data.total; + + if (total > 0) { + failuresCard.style.display = ''; + failureCountBadge.textContent = total; + renderFailures(currentFailures); + + if (total > 5 && !failuresExpanded) { + failuresPagination.style.display = ''; + failuresShownCount.textContent = `Showing ${Math.min(5, total)} of ${total}`; + } else { + failuresPagination.style.display = 'none'; + } + } else { + failuresCard.style.display = 'none'; + } + } catch (err) { + console.error('Failed to load replication failures:', err); + } + }; + + const renderFailures = (failures) => { + if (!failuresBody) return; + failuresBody.innerHTML = failures.map(f => ` + + + ${escapeHtml(f.object_key)} + + + ${escapeHtml(f.error_message)} + + ${new Date(f.timestamp * 1000).toLocaleString()} + ${f.failure_count} + + + + + + `).join(''); + }; + + window.retryFailure = async (btn, objectKey) => { + const originalHtml = btn.innerHTML; + btn.disabled = true; + btn.innerHTML = ''; + const endpoint = failuresCard.dataset.retryEndpoint.replace('__KEY__', encodeURIComponent(objectKey)); + try { + const resp = await fetch(endpoint, { method: 'POST' }); + if (resp.ok) { + loadReplicationFailures(); + } + } catch (err) { + console.error('Failed to retry:', err); + btn.disabled = false; + btn.innerHTML = originalHtml; + } + }; + + window.dismissFailure = async (btn, objectKey) => { + const originalHtml = btn.innerHTML; + btn.disabled = true; + btn.innerHTML = ''; + const endpoint = failuresCard.dataset.dismissEndpoint.replace('__KEY__', encodeURIComponent(objectKey)); + try { + const resp = await fetch(endpoint, { method: 'DELETE' }); + if (resp.ok) { + loadReplicationFailures(); + } + } catch (err) { + console.error('Failed to dismiss:', err); + btn.disabled = false; + btn.innerHTML = originalHtml; + } + }; + + retryAllBtn?.addEventListener('click', async () => { + const btn = retryAllBtn; + const originalHtml = btn.innerHTML; + btn.disabled = true; + btn.innerHTML = 'Retrying...'; + const endpoint = failuresCard.dataset.retryAllEndpoint; + try { + const resp = await fetch(endpoint, { method: 'POST' }); + if (resp.ok) { + loadReplicationFailures(); + } + } catch (err) { + console.error('Failed to retry all:', err); + } finally { + btn.disabled = false; + btn.innerHTML = originalHtml; + } + }); + + clearFailuresBtn?.addEventListener('click', () => { + clearFailuresModalInstance?.show(); + }); + + confirmClearFailuresBtn?.addEventListener('click', async () => { + const btn = confirmClearFailuresBtn; + const originalHtml = btn.innerHTML; + btn.disabled = true; + btn.innerHTML = 'Clearing...'; + const endpoint = failuresCard.dataset.clearEndpoint; + try { + const resp = await fetch(endpoint, { method: 'DELETE' }); + if (resp.ok) { + clearFailuresModalInstance?.hide(); + loadReplicationFailures(); + } + } catch (err) { + console.error('Failed to clear failures:', err); + } finally { + btn.disabled = false; + btn.innerHTML = originalHtml; + } + }); + + showMoreFailuresBtn?.addEventListener('click', () => { + failuresExpanded = !failuresExpanded; + showMoreFailuresBtn.textContent = failuresExpanded ? 'Show less' : 'Show more...'; + loadReplicationFailures(); + }); + + loadReplicationFailures(); + } + + const algoAes256Radio = document.getElementById('algo_aes256'); + const algoKmsRadio = document.getElementById('algo_kms'); + const kmsKeySection = document.getElementById('kmsKeySection'); + const encryptionForm = document.getElementById('encryptionForm'); + const encryptionAction = document.getElementById('encryptionAction'); + const disableEncryptionBtn = document.getElementById('disableEncryptionBtn'); + + const updateKmsKeyVisibility = () => { + if (!kmsKeySection) return; + const showKms = algoKmsRadio?.checked; + kmsKeySection.style.display = showKms ? '' : 'none'; + }; + + algoAes256Radio?.addEventListener('change', updateKmsKeyVisibility); + algoKmsRadio?.addEventListener('change', updateKmsKeyVisibility); + + const targetBucketInput = document.getElementById('target_bucket'); + const targetBucketFeedback = document.getElementById('target_bucket_feedback'); + + const validateBucketName = (name) => { + if (!name) return { valid: false, error: 'Bucket name is required' }; + if (name.length < 3) return { valid: false, error: 'Bucket name must be at least 3 characters' }; + if (name.length > 63) return { valid: false, error: 'Bucket name must be 63 characters or less' }; + if (!/^[a-z0-9]/.test(name)) return { valid: false, error: 'Bucket name must start with a lowercase letter or number' }; + if (!/[a-z0-9]$/.test(name)) return { valid: false, error: 'Bucket name must end with a lowercase letter or number' }; + if (/[A-Z]/.test(name)) return { valid: false, error: 'Bucket name must not contain uppercase letters' }; + if (/_/.test(name)) return { valid: false, error: 'Bucket name must not contain underscores' }; + if (/\.\.|--/.test(name)) return { valid: false, error: 'Bucket name must not contain consecutive periods or hyphens' }; + if (/^\d+\.\d+\.\d+\.\d+$/.test(name)) return { valid: false, error: 'Bucket name must not be formatted as an IP address' }; + if (!/^[a-z0-9][a-z0-9.-]*[a-z0-9]$/.test(name) && name.length > 2) return { valid: false, error: 'Bucket name contains invalid characters. Use only lowercase letters, numbers, hyphens, and periods.' }; + return { valid: true, error: null }; + }; + + const updateBucketNameValidation = () => { + if (!targetBucketInput || !targetBucketFeedback) return; + const name = targetBucketInput.value.trim(); + if (!name) { + targetBucketInput.classList.remove('is-valid', 'is-invalid'); + targetBucketFeedback.textContent = ''; + return; + } + const result = validateBucketName(name); + targetBucketInput.classList.toggle('is-valid', result.valid); + targetBucketInput.classList.toggle('is-invalid', !result.valid); + targetBucketFeedback.textContent = result.error || ''; + }; + + targetBucketInput?.addEventListener('input', updateBucketNameValidation); + targetBucketInput?.addEventListener('blur', updateBucketNameValidation); + + const replicationForm = targetBucketInput?.closest('form'); + replicationForm?.addEventListener('submit', (e) => { + const name = targetBucketInput.value.trim(); + const result = validateBucketName(name); + if (!result.valid) { + e.preventDefault(); + updateBucketNameValidation(); + targetBucketInput.focus(); + return false; + } + }); + + const formatPolicyBtn = document.getElementById('formatPolicyBtn'); + const policyValidationStatus = document.getElementById('policyValidationStatus'); + const policyValidBadge = document.getElementById('policyValidBadge'); + const policyInvalidBadge = document.getElementById('policyInvalidBadge'); + const policyErrorDetail = document.getElementById('policyErrorDetail'); + + const validatePolicyJson = () => { + if (!policyTextarea || !policyValidationStatus) return; + const value = policyTextarea.value.trim(); + if (!value) { + policyValidationStatus.classList.add('d-none'); + policyErrorDetail?.classList.add('d-none'); + return; + } + policyValidationStatus.classList.remove('d-none'); + try { + JSON.parse(value); + policyValidBadge?.classList.remove('d-none'); + policyInvalidBadge?.classList.add('d-none'); + policyErrorDetail?.classList.add('d-none'); + } catch (err) { + policyValidBadge?.classList.add('d-none'); + policyInvalidBadge?.classList.remove('d-none'); + if (policyErrorDetail) { + policyErrorDetail.textContent = err.message; + policyErrorDetail.classList.remove('d-none'); + } + } + }; + + policyTextarea?.addEventListener('input', validatePolicyJson); + policyTextarea?.addEventListener('blur', validatePolicyJson); + + formatPolicyBtn?.addEventListener('click', () => { + if (!policyTextarea) return; + const value = policyTextarea.value.trim(); + if (!value) return; + try { + const parsed = JSON.parse(value); + policyTextarea.value = JSON.stringify(parsed, null, 2); + validatePolicyJson(); + } catch (err) { + validatePolicyJson(); + } + }); + + if (policyTextarea && policyPreset?.value === 'custom') { + validatePolicyJson(); + } + + const lifecycleCard = document.getElementById('lifecycle-rules-card'); + const lifecycleUrl = lifecycleCard?.dataset.lifecycleUrl; + const lifecycleRulesBody = document.getElementById('lifecycle-rules-body'); + const addLifecycleRuleModalEl = document.getElementById('addLifecycleRuleModal'); + const addLifecycleRuleModal = addLifecycleRuleModalEl ? new bootstrap.Modal(addLifecycleRuleModalEl) : null; + let lifecycleRules = []; + + const loadLifecycleRules = async () => { + if (!lifecycleUrl || !lifecycleRulesBody) return; + lifecycleRulesBody.innerHTML = 'Loading...'; + try { + const resp = await fetch(lifecycleUrl); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to load lifecycle rules'); + lifecycleRules = data.rules || []; + renderLifecycleRules(); + } catch (err) { + lifecycleRulesBody.innerHTML = `${escapeHtml(err.message)}`; + } + }; + + const renderLifecycleRules = () => { + if (!lifecycleRulesBody) return; + if (lifecycleRules.length === 0) { + lifecycleRulesBody.innerHTML = 'No lifecycle rules configured'; + return; + } + lifecycleRulesBody.innerHTML = lifecycleRules.map((rule, idx) => { + const expiration = rule.Expiration?.Days ? `${rule.Expiration.Days}d` : '-'; + const noncurrent = rule.NoncurrentVersionExpiration?.NoncurrentDays ? `${rule.NoncurrentVersionExpiration.NoncurrentDays}d` : '-'; + const abortMpu = rule.AbortIncompleteMultipartUpload?.DaysAfterInitiation ? `${rule.AbortIncompleteMultipartUpload.DaysAfterInitiation}d` : '-'; + const statusClass = rule.Status === 'Enabled' ? 'bg-success' : 'bg-secondary'; + return ` + ${escapeHtml(rule.ID || '')} + ${escapeHtml(rule.Filter?.Prefix || '*')} + ${escapeHtml(rule.Status)} + ${expiration} + ${noncurrent} + ${abortMpu} + +
    + + +
    + + `; + }).join(''); + }; + + window.editLifecycleRule = (idx) => { + const rule = lifecycleRules[idx]; + if (!rule) return; + document.getElementById('lifecycleRuleId').value = rule.ID || ''; + document.getElementById('lifecycleRuleStatus').value = rule.Status || 'Enabled'; + document.getElementById('lifecycleRulePrefix').value = rule.Filter?.Prefix || ''; + document.getElementById('lifecycleExpirationDays').value = rule.Expiration?.Days || ''; + document.getElementById('lifecycleNoncurrentDays').value = rule.NoncurrentVersionExpiration?.NoncurrentDays || ''; + document.getElementById('lifecycleAbortMpuDays').value = rule.AbortIncompleteMultipartUpload?.DaysAfterInitiation || ''; + window.editingLifecycleIdx = idx; + addLifecycleRuleModal?.show(); + }; + + window.editingLifecycleIdx = null; + + window.deleteLifecycleRule = async (idx) => { + lifecycleRules.splice(idx, 1); + await saveLifecycleRules(); + }; + + const saveLifecycleRules = async () => { + if (!lifecycleUrl) return; + try { + const resp = await fetch(lifecycleUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '' }, + body: JSON.stringify({ rules: lifecycleRules }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to save'); + showMessage({ title: 'Lifecycle rules saved', body: 'Configuration updated successfully.', variant: 'success' }); + renderLifecycleRules(); + } catch (err) { + showMessage({ title: 'Save failed', body: err.message, variant: 'danger' }); + } + }; + + document.getElementById('addLifecycleRuleConfirm')?.addEventListener('click', async () => { + const ruleId = document.getElementById('lifecycleRuleId')?.value?.trim(); + const status = document.getElementById('lifecycleRuleStatus')?.value || 'Enabled'; + const prefix = document.getElementById('lifecycleRulePrefix')?.value?.trim() || ''; + const expDays = parseInt(document.getElementById('lifecycleExpirationDays')?.value) || 0; + const ncDays = parseInt(document.getElementById('lifecycleNoncurrentDays')?.value) || 0; + const abortDays = parseInt(document.getElementById('lifecycleAbortMpuDays')?.value) || 0; + if (!ruleId) { showMessage({ title: 'Validation error', body: 'Rule ID is required', variant: 'warning' }); return; } + if (expDays === 0 && ncDays === 0 && abortDays === 0) { showMessage({ title: 'Validation error', body: 'At least one action is required', variant: 'warning' }); return; } + const rule = { ID: ruleId, Status: status, Filter: { Prefix: prefix } }; + if (expDays > 0) rule.Expiration = { Days: expDays }; + if (ncDays > 0) rule.NoncurrentVersionExpiration = { NoncurrentDays: ncDays }; + if (abortDays > 0) rule.AbortIncompleteMultipartUpload = { DaysAfterInitiation: abortDays }; + if (typeof window.editingLifecycleIdx === 'number' && window.editingLifecycleIdx !== null) { + lifecycleRules[window.editingLifecycleIdx] = rule; + window.editingLifecycleIdx = null; + } else { + lifecycleRules.push(rule); + } + await saveLifecycleRules(); + addLifecycleRuleModal?.hide(); + document.getElementById('lifecycleRuleId').value = ''; + document.getElementById('lifecycleRulePrefix').value = ''; + document.getElementById('lifecycleExpirationDays').value = ''; + document.getElementById('lifecycleNoncurrentDays').value = ''; + document.getElementById('lifecycleAbortMpuDays').value = ''; + document.getElementById('lifecycleRuleStatus').value = 'Enabled'; + }); + + const corsCard = document.getElementById('cors-rules-card'); + const corsUrl = corsCard?.dataset.corsUrl; + const corsRulesBody = document.getElementById('cors-rules-body'); + const addCorsRuleModalEl = document.getElementById('addCorsRuleModal'); + const addCorsRuleModal = addCorsRuleModalEl ? new bootstrap.Modal(addCorsRuleModalEl) : null; + let corsRules = []; + + const loadCorsRules = async () => { + if (!corsUrl || !corsRulesBody) return; + corsRulesBody.innerHTML = 'Loading...'; + try { + const resp = await fetch(corsUrl); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to load CORS rules'); + corsRules = data.rules || []; + renderCorsRules(); + } catch (err) { + corsRulesBody.innerHTML = `${escapeHtml(err.message)}`; + } + }; + + const renderCorsRules = () => { + if (!corsRulesBody) return; + if (corsRules.length === 0) { + corsRulesBody.innerHTML = 'No CORS rules configured'; + return; + } + corsRulesBody.innerHTML = corsRules.map((rule, idx) => { + const origins = (rule.AllowedOrigins || []).map(o => `${escapeHtml(o)}`).join(', '); + const methods = (rule.AllowedMethods || []).map(m => `${escapeHtml(m)}`).join(' '); + const headers = (rule.AllowedHeaders || []).slice(0, 3).map(h => `${escapeHtml(h)}`).join(', '); + return ` + ${origins || 'None'} + ${methods || 'None'} + ${headers || '*'} + ${rule.MaxAgeSeconds || '-'} + +
    + + +
    + + `; + }).join(''); + }; + + window.editCorsRule = (idx) => { + const rule = corsRules[idx]; + if (!rule) return; + document.getElementById('corsAllowedOrigins').value = (rule.AllowedOrigins || []).join('\n'); + document.getElementById('corsAllowedHeaders').value = (rule.AllowedHeaders || []).join('\n'); + document.getElementById('corsExposeHeaders').value = (rule.ExposeHeaders || []).join('\n'); + document.getElementById('corsMaxAge').value = rule.MaxAgeSeconds || ''; + document.getElementById('corsMethodGet').checked = (rule.AllowedMethods || []).includes('GET'); + document.getElementById('corsMethodPut').checked = (rule.AllowedMethods || []).includes('PUT'); + document.getElementById('corsMethodPost').checked = (rule.AllowedMethods || []).includes('POST'); + document.getElementById('corsMethodDelete').checked = (rule.AllowedMethods || []).includes('DELETE'); + document.getElementById('corsMethodHead').checked = (rule.AllowedMethods || []).includes('HEAD'); + window.editingCorsIdx = idx; + addCorsRuleModal?.show(); + }; + + window.editingCorsIdx = null; + + window.deleteCorsRule = async (idx) => { + corsRules.splice(idx, 1); + await saveCorsRules(); + }; + + const saveCorsRules = async () => { + if (!corsUrl) return; + try { + const resp = await fetch(corsUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '' }, + body: JSON.stringify({ rules: corsRules }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to save'); + showMessage({ title: 'CORS rules saved', body: 'Configuration updated successfully.', variant: 'success' }); + renderCorsRules(); + } catch (err) { + showMessage({ title: 'Save failed', body: err.message, variant: 'danger' }); + } + }; + + document.getElementById('addCorsRuleConfirm')?.addEventListener('click', async () => { + const originsRaw = document.getElementById('corsAllowedOrigins')?.value?.trim() || ''; + const origins = originsRaw.split('\n').map(s => s.trim()).filter(Boolean); + const methods = []; + if (document.getElementById('corsMethodGet')?.checked) methods.push('GET'); + if (document.getElementById('corsMethodPut')?.checked) methods.push('PUT'); + if (document.getElementById('corsMethodPost')?.checked) methods.push('POST'); + if (document.getElementById('corsMethodDelete')?.checked) methods.push('DELETE'); + if (document.getElementById('corsMethodHead')?.checked) methods.push('HEAD'); + const headersRaw = document.getElementById('corsAllowedHeaders')?.value?.trim() || ''; + const headers = headersRaw.split('\n').map(s => s.trim()).filter(Boolean); + const exposeRaw = document.getElementById('corsExposeHeaders')?.value?.trim() || ''; + const expose = exposeRaw.split('\n').map(s => s.trim()).filter(Boolean); + const maxAge = parseInt(document.getElementById('corsMaxAge')?.value) || 0; + if (origins.length === 0) { showMessage({ title: 'Validation error', body: 'At least one origin is required', variant: 'warning' }); return; } + if (methods.length === 0) { showMessage({ title: 'Validation error', body: 'At least one method is required', variant: 'warning' }); return; } + const rule = { AllowedOrigins: origins, AllowedMethods: methods }; + if (headers.length > 0) rule.AllowedHeaders = headers; + if (expose.length > 0) rule.ExposeHeaders = expose; + if (maxAge > 0) rule.MaxAgeSeconds = maxAge; + if (typeof window.editingCorsIdx === 'number' && window.editingCorsIdx !== null) { + corsRules[window.editingCorsIdx] = rule; + window.editingCorsIdx = null; + } else { + corsRules.push(rule); + } + await saveCorsRules(); + addCorsRuleModal?.hide(); + document.getElementById('corsAllowedOrigins').value = ''; + document.getElementById('corsAllowedHeaders').value = ''; + document.getElementById('corsExposeHeaders').value = ''; + document.getElementById('corsMaxAge').value = ''; + document.getElementById('corsMethodGet').checked = false; + document.getElementById('corsMethodPut').checked = false; + document.getElementById('corsMethodPost').checked = false; + document.getElementById('corsMethodDelete').checked = false; + document.getElementById('corsMethodHead').checked = false; + }); + + const aclCard = document.getElementById('bucket-acl-card'); + const aclUrl = aclCard?.dataset.aclUrl; + const aclOwnerEl = document.getElementById('acl-owner'); + const aclGrantsList = document.getElementById('acl-grants-list'); + const aclLoading = document.getElementById('acl-loading'); + const aclContent = document.getElementById('acl-content'); + const cannedAclSelect = document.getElementById('cannedAclSelect'); + + const loadAcl = async () => { + if (!aclUrl) return; + try { + const resp = await fetch(aclUrl); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to load ACL'); + if (aclOwnerEl) aclOwnerEl.textContent = data.owner || '-'; + if (aclGrantsList) { + const grants = data.grants || []; + if (grants.length === 0) { + aclGrantsList.innerHTML = '
    No grants
    '; + } else { + aclGrantsList.innerHTML = grants.map(g => `
    ${escapeHtml(g.grantee)}${escapeHtml(g.permission)}
    `).join(''); + } + } + if (aclLoading) aclLoading.classList.add('d-none'); + if (aclContent) aclContent.classList.remove('d-none'); + } catch (err) { + if (aclLoading) aclLoading.classList.add('d-none'); + if (aclContent) aclContent.classList.remove('d-none'); + if (aclGrantsList) aclGrantsList.innerHTML = `
    ${escapeHtml(err.message)}
    `; + } + }; + + cannedAclSelect?.addEventListener('change', async () => { + const canned = cannedAclSelect.value; + if (!canned || !aclUrl) return; + try { + const resp = await fetch(aclUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '' }, + body: JSON.stringify({ canned_acl: canned }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to set ACL'); + showMessage({ title: 'ACL updated', body: `Bucket ACL set to "${canned}"`, variant: 'success' }); + await loadAcl(); + } catch (err) { + showMessage({ title: 'ACL update failed', body: err.message, variant: 'danger' }); + } + }); + + document.querySelectorAll('[data-set-acl]').forEach(btn => { + btn.addEventListener('click', async () => { + const canned = btn.dataset.setAcl; + if (!canned || !aclUrl) return; + btn.disabled = true; + const originalText = btn.innerHTML; + btn.innerHTML = ''; + try { + const resp = await fetch(aclUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '' }, + body: JSON.stringify({ canned_acl: canned }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to set ACL'); + showMessage({ title: 'ACL updated', body: `Bucket ACL set to "${canned}"`, variant: 'success' }); + await loadAcl(); + } catch (err) { + showMessage({ title: 'ACL update failed', body: err.message, variant: 'danger' }); + } finally { + btn.innerHTML = originalText; + btn.disabled = false; + } + }); + }); + + document.getElementById('objects-table')?.addEventListener('show.bs.dropdown', function (e) { + const dropdown = e.target.closest('.dropdown'); + const menu = dropdown?.querySelector('.dropdown-menu'); + const btn = e.target; + if (!menu || !btn) return; + const btnRect = btn.getBoundingClientRect(); + menu.style.position = 'fixed'; + menu.style.top = (btnRect.bottom + 4) + 'px'; + menu.style.left = 'auto'; + menu.style.right = (window.innerWidth - btnRect.right) + 'px'; + menu.style.transform = 'none'; + }); + + const previewTagsPanel = document.getElementById('preview-tags'); + const previewTagsList = document.getElementById('preview-tags-list'); + const previewTagsEmpty = document.getElementById('preview-tags-empty'); + const previewTagsCount = document.getElementById('preview-tags-count'); + const previewTagsEditor = document.getElementById('preview-tags-editor'); + const previewTagsInputs = document.getElementById('preview-tags-inputs'); + const editTagsButton = document.getElementById('editTagsButton'); + const addTagRow = document.getElementById('addTagRow'); + const saveTagsButton = document.getElementById('saveTagsButton'); + const cancelTagsButton = document.getElementById('cancelTagsButton'); + let currentObjectTags = []; + let isEditingTags = false; + let savedObjectTags = []; + + const loadObjectTags = async (row) => { + if (!row || !previewTagsPanel) return; + if (previewFailed) { + previewTagsPanel.classList.add('d-none'); + return; + } + const tagsUrl = row.dataset.tagsUrl; + if (!tagsUrl) { + previewTagsPanel.classList.add('d-none'); + return; + } + previewTagsPanel.classList.remove('d-none'); + try { + const resp = await fetch(tagsUrl); + const data = await resp.json(); + currentObjectTags = data.tags || []; + renderObjectTags(); + } catch (err) { + currentObjectTags = []; + renderObjectTags(); + } + }; + + const renderObjectTags = () => { + if (!previewTagsList || !previewTagsEmpty || !previewTagsCount) return; + previewTagsCount.textContent = currentObjectTags.length; + if (currentObjectTags.length === 0) { + previewTagsList.innerHTML = ''; + previewTagsEmpty.classList.remove('d-none'); + } else { + previewTagsEmpty.classList.add('d-none'); + previewTagsList.innerHTML = currentObjectTags.map(t => `${escapeHtml(t.Key)}${escapeHtml(t.Value)}`).join(''); + } + }; + + const syncTagInputs = () => { + previewTagsInputs?.querySelectorAll('.tag-editor-row').forEach((row, idx) => { + if (idx < currentObjectTags.length) { + currentObjectTags[idx].Key = row.querySelector(`[data-tag-key="${idx}"]`)?.value || ''; + currentObjectTags[idx].Value = row.querySelector(`[data-tag-value="${idx}"]`)?.value || ''; + } + }); + }; + + const renderTagEditor = () => { + if (!previewTagsInputs) return; + previewTagsInputs.innerHTML = currentObjectTags.map((t, idx) => ` +
    + + + +
    + `).join(''); + }; + + window.removeTagRow = (idx) => { + syncTagInputs(); + currentObjectTags.splice(idx, 1); + renderTagEditor(); + }; + + editTagsButton?.addEventListener('click', () => { + savedObjectTags = currentObjectTags.map(t => ({ Key: t.Key, Value: t.Value })); + isEditingTags = true; + previewTagsList.classList.add('d-none'); + previewTagsEmpty.classList.add('d-none'); + previewTagsEditor?.classList.remove('d-none'); + const card = previewTagsEditor?.querySelector('.tag-editor-card'); + if (card) { + card.style.opacity = '0'; + card.style.transition = 'opacity 0.2s ease'; + requestAnimationFrame(() => { card.style.opacity = '1'; }); + } + renderTagEditor(); + }); + + cancelTagsButton?.addEventListener('click', () => { + isEditingTags = false; + currentObjectTags = savedObjectTags.map(t => ({ Key: t.Key, Value: t.Value })); + previewTagsEditor?.classList.add('d-none'); + previewTagsList.classList.remove('d-none'); + renderObjectTags(); + }); + + addTagRow?.addEventListener('click', () => { + if (currentObjectTags.length >= 10) { + showMessage({ title: 'Limit reached', body: 'Maximum 10 tags allowed per object.', variant: 'warning' }); + return; + } + syncTagInputs(); + currentObjectTags.push({ Key: '', Value: '' }); + renderTagEditor(); + }); + + saveTagsButton?.addEventListener('click', async () => { + if (!activeRow) return; + const tagsUrl = activeRow.dataset.tagsUrl; + if (!tagsUrl) return; + const inputs = previewTagsInputs?.querySelectorAll('.tag-editor-row'); + const newTags = []; + inputs?.forEach((group, idx) => { + const key = group.querySelector(`[data-tag-key="${idx}"]`)?.value?.trim() || ''; + const value = group.querySelector(`[data-tag-value="${idx}"]`)?.value?.trim() || ''; + if (key) newTags.push({ Key: key, Value: value }); + }); + try { + const resp = await fetch(tagsUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '' }, + body: JSON.stringify({ tags: newTags }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to save tags'); + currentObjectTags = newTags; + isEditingTags = false; + previewTagsEditor?.classList.add('d-none'); + previewTagsList.classList.remove('d-none'); + renderObjectTags(); + showMessage({ title: 'Tags saved', body: 'Object tags updated successfully.', variant: 'success' }); + } catch (err) { + showMessage({ title: 'Save failed', body: err.message, variant: 'danger' }); + } + }); + + const copyMoveModalEl = document.getElementById('copyMoveModal'); + const copyMoveModal = copyMoveModalEl ? new bootstrap.Modal(copyMoveModalEl) : null; + const copyMoveActionLabel = document.getElementById('copyMoveActionLabel'); + const copyMoveConfirmLabel = document.getElementById('copyMoveConfirmLabel'); + const copyMoveSource = document.getElementById('copyMoveSource'); + const copyMoveDestBucket = document.getElementById('copyMoveDestBucket'); + const copyMoveDestKey = document.getElementById('copyMoveDestKey'); + const copyMoveConfirm = document.getElementById('copyMoveConfirm'); + const bucketsForCopyUrl = objectsContainer?.dataset.bucketsForCopyUrl; + let copyMoveAction = 'copy'; + let copyMoveSourceKey = ''; + + window.openCopyMoveModal = async (action, key) => { + copyMoveAction = action; + copyMoveSourceKey = key; + if (copyMoveActionLabel) copyMoveActionLabel.textContent = action === 'move' ? 'Move' : 'Copy'; + if (copyMoveConfirmLabel) copyMoveConfirmLabel.textContent = action === 'move' ? 'Move' : 'Copy'; + if (copyMoveSource) copyMoveSource.textContent = key; + if (copyMoveDestKey) copyMoveDestKey.value = key; + if (copyMoveDestBucket) { + copyMoveDestBucket.innerHTML = ''; + try { + const resp = await fetch(bucketsForCopyUrl); + const data = await resp.json(); + const buckets = data.buckets || []; + copyMoveDestBucket.innerHTML = buckets.map(b => ``).join(''); + } catch { + copyMoveDestBucket.innerHTML = ''; + } + } + copyMoveModal?.show(); + }; + + copyMoveConfirm?.addEventListener('click', async () => { + const destBucket = copyMoveDestBucket?.value; + const destKey = copyMoveDestKey?.value?.trim(); + if (!destBucket || !destKey) { showMessage({ title: 'Validation error', body: 'Destination bucket and key are required', variant: 'warning' }); return; } + const actionUrl = copyMoveAction === 'move' + ? urlTemplates?.move?.replace('KEY_PLACEHOLDER', encodeURIComponent(copyMoveSourceKey).replace(/%2F/g, '/')) + : urlTemplates?.copy?.replace('KEY_PLACEHOLDER', encodeURIComponent(copyMoveSourceKey).replace(/%2F/g, '/')); + if (!actionUrl) { showMessage({ title: 'Error', body: 'Copy/move URL not configured', variant: 'danger' }); return; } + try { + const resp = await fetch(actionUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': window.getCsrfToken ? window.getCsrfToken() : '' }, + body: JSON.stringify({ dest_bucket: destBucket, dest_key: destKey }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || `Failed to ${copyMoveAction} object`); + showMessage({ title: `Object ${copyMoveAction === 'move' ? 'moved' : 'copied'}`, body: `Successfully ${copyMoveAction === 'move' ? 'moved' : 'copied'} to ${destBucket}/${destKey}`, variant: 'success' }); + copyMoveModal?.hide(); + if (copyMoveAction === 'move') { + previewEmpty.classList.remove('d-none'); + previewPanel.classList.add('d-none'); + activeRow = null; + loadObjects(false); + } + } catch (err) { + showMessage({ title: `${copyMoveAction === 'move' ? 'Move' : 'Copy'} failed`, body: err.message, variant: 'danger' }); + } + }); + + const originalSelectRow = selectRow; + selectRow = async (row) => { + await originalSelectRow(row); + loadObjectTags(row); + }; + + if (lifecycleCard) loadLifecycleRules(); + + const lifecycleHistoryCard = document.getElementById('lifecycle-history-card'); + const lifecycleHistoryBody = document.getElementById('lifecycle-history-body'); + const lifecycleHistoryPagination = document.getElementById('lifecycle-history-pagination'); + const showMoreHistoryBtn = document.getElementById('show-more-history'); + const historyShownCount = document.getElementById('history-shown-count'); + let historyExpanded = false; + + const loadLifecycleHistory = async () => { + if (!lifecycleHistoryCard || !lifecycleHistoryBody) return; + + const endpoint = lifecycleHistoryCard.dataset.historyEndpoint; + const limit = historyExpanded ? 50 : 5; + + lifecycleHistoryBody.innerHTML = 'Loading...'; + + try { + const resp = await fetch(`${endpoint}?limit=${limit}`); + if (!resp.ok) throw new Error('Failed to fetch history'); + const data = await resp.json(); + + if (!data.enabled) { + lifecycleHistoryBody.innerHTML = 'Lifecycle enforcement is not enabled'; + return; + } + + const executions = data.executions || []; + const total = data.total || 0; + + if (executions.length === 0) { + lifecycleHistoryBody.innerHTML = 'No executions recorded yet'; + lifecycleHistoryPagination.style.display = 'none'; + return; + } + + lifecycleHistoryBody.innerHTML = executions.map(e => { + const date = new Date(e.timestamp * 1000); + const hasErrors = e.errors && e.errors.length > 0; + const hasActivity = e.objects_deleted > 0 || e.versions_deleted > 0 || e.uploads_aborted > 0; + let statusBadge; + if (hasErrors) { + statusBadge = 'Errors'; + } else if (hasActivity) { + statusBadge = 'Success'; + } else { + statusBadge = 'No action'; + } + const errorTooltip = hasErrors ? ` title="${escapeHtml(e.errors.join('; '))}"` : ''; + return ` + ${date.toLocaleString()} + ${e.objects_deleted} + ${e.versions_deleted} + ${e.uploads_aborted} + ${statusBadge} + `; + }).join(''); + + if (total > 5 && !historyExpanded) { + lifecycleHistoryPagination.style.display = ''; + historyShownCount.textContent = `Showing ${Math.min(5, total)} of ${total}`; + } else { + lifecycleHistoryPagination.style.display = 'none'; + } + } catch (err) { + console.error('Failed to load lifecycle history:', err); + lifecycleHistoryBody.innerHTML = 'Failed to load history'; + } + }; + + showMoreHistoryBtn?.addEventListener('click', () => { + historyExpanded = !historyExpanded; + showMoreHistoryBtn.textContent = historyExpanded ? 'Show less' : 'Show more...'; + loadLifecycleHistory(); + }); + + if (lifecycleHistoryCard) { + loadLifecycleHistory(); + if (window.pollingManager) { + window.pollingManager.start('lifecycle', loadLifecycleHistory); + } + } + + if (corsCard) loadCorsRules(); + if (aclCard) loadAcl(); + + function updateVersioningBadge(enabled) { + var badge = document.querySelector('.badge.rounded-pill'); + if (!badge) return; + badge.classList.remove('text-bg-success', 'text-bg-secondary'); + badge.classList.add(enabled ? 'text-bg-success' : 'text-bg-secondary'); + var icon = '' + + '' + + ''; + badge.innerHTML = icon + (enabled ? 'Versioning On' : 'Versioning Off'); + versioningEnabled = enabled; + } + + function interceptForm(formId, options) { + var form = document.getElementById(formId); + if (!form) return; + + form.addEventListener('submit', function (e) { + e.preventDefault(); + window.UICore.submitFormAjax(form, { + successMessage: options.successMessage || 'Operation completed', + onSuccess: function (data) { + if (options.onSuccess) options.onSuccess(data); + if (options.closeModal) { + var modal = bootstrap.Modal.getInstance(document.getElementById(options.closeModal)); + if (modal) modal.hide(); + } + if (options.reload) { + setTimeout(function () { location.reload(); }, 500); + } + } + }); + }); + } + + function updateVersioningCard(enabled) { + var card = document.getElementById('bucket-versioning-card'); + if (!card) return; + var cardBody = card.querySelector('.card-body'); + if (!cardBody) return; + + var enabledHtml = '' + + ''; + + var disabledHtml = '' + + '
    ' + + '' + + '' + + '
    '; + + cardBody.innerHTML = enabled ? enabledHtml : disabledHtml; + + var archivedCardEl = document.getElementById('archived-objects-card'); + if (archivedCardEl) { + archivedCardEl.style.display = enabled ? '' : 'none'; + } else if (enabled) { + var endpoint = window.BucketDetailConfig?.endpoints?.archivedObjects || ''; + if (endpoint) { + var html = '
    ' + + '
    ' + + '
    ' + + '' + + '' + + 'Archived Objects
    ' + + '
    ' + + '0 items' + + '
    ' + + '
    ' + + '

    Objects that have been deleted while versioning is enabled. Their previous versions remain available until you restore or purge them.

    ' + + '
    ' + + '' + + '' + + '' + + '' + + '' + + '' + + '' + + '
    ' + + '' + + 'KeyLatest VersionVersionsActions
    ' + + '' + + '' + + 'No archived objects
    '; + card.insertAdjacentHTML('afterend', html); + archivedCard = document.getElementById('archived-objects-card'); + archivedBody = archivedCard.querySelector('[data-archived-body]'); + archivedCountBadge = archivedCard.querySelector('[data-archived-count]'); + archivedRefreshButton = archivedCard.querySelector('[data-archived-refresh]'); + archivedEndpoint = endpoint; + archivedRefreshButton.addEventListener('click', function() { loadArchivedObjects(); }); + loadArchivedObjects(); + } + } + + var dropZone = document.getElementById('objects-drop-zone'); + if (dropZone) { + dropZone.setAttribute('data-versioning', enabled ? 'true' : 'false'); + } + + var bulkPurgeWrap = document.getElementById('bulkDeletePurgeWrap'); + if (bulkPurgeWrap) { + bulkPurgeWrap.classList.toggle('d-none', !enabled); + } + var singleDeleteVerWrap = document.getElementById('deleteObjectVersioningWrap'); + if (singleDeleteVerWrap) { + singleDeleteVerWrap.classList.toggle('d-none', !enabled); + } + + if (!enabled) { + var newForm = document.getElementById('enableVersioningForm'); + if (newForm) { + newForm.setAttribute('action', window.BucketDetailConfig?.endpoints?.versioning || ''); + newForm.addEventListener('submit', function (e) { + e.preventDefault(); + window.UICore.submitFormAjax(newForm, { + successMessage: 'Versioning enabled', + onSuccess: function () { + updateVersioningBadge(true); + updateVersioningCard(true); + } + }); + }); + } + } + } + + function updateEncryptionCard(enabled, algorithm) { + var encCard = document.getElementById('bucket-encryption-card'); + if (!encCard) return; + var alertContainer = encCard.querySelector('.alert'); + if (alertContainer) { + if (enabled) { + alertContainer.className = 'alert alert-success d-flex align-items-start mb-4'; + var algoText = algorithm === 'aws:kms' ? 'KMS' : 'AES-256'; + alertContainer.innerHTML = '' + + '' + + '' + + '
    Default encryption enabled (' + algoText + ')' + + '

    All new objects uploaded to this bucket will be automatically encrypted.

    '; + } else { + alertContainer.className = 'alert alert-secondary d-flex align-items-start mb-4'; + alertContainer.innerHTML = '' + + '' + + '
    Default encryption disabled' + + '

    Objects are stored without default encryption. You can enable server-side encryption below.

    '; + } + } + var disableBtn = document.getElementById('disableEncryptionBtn'); + if (disableBtn) { + disableBtn.style.display = enabled ? '' : 'none'; + } + } + + function updateQuotaCard(hasQuota, maxBytes, maxObjects) { + var quotaCard = document.getElementById('bucket-quota-card'); + if (!quotaCard) return; + var alertContainer = quotaCard.querySelector('.alert'); + if (alertContainer) { + if (hasQuota) { + alertContainer.className = 'alert alert-info d-flex align-items-start mb-4'; + var quotaParts = []; + if (maxBytes) quotaParts.push(formatBytes(maxBytes) + ' storage'); + if (maxObjects) quotaParts.push(maxObjects.toLocaleString() + ' objects'); + alertContainer.innerHTML = '' + + '' + + '
    Storage quota active' + + '

    This bucket is limited to ' + quotaParts.join(' and ') + '.

    '; + } else { + alertContainer.className = 'alert alert-secondary d-flex align-items-start mb-4'; + alertContainer.innerHTML = '' + + '' + + '' + + '
    No storage quota' + + '

    This bucket has no storage or object count limits. Set limits below to control usage.

    '; + } + } + var removeBtn = document.getElementById('removeQuotaBtn'); + if (removeBtn) { + removeBtn.style.display = hasQuota ? '' : 'none'; + } + var maxMbInput = document.getElementById('max_mb'); + var maxObjInput = document.getElementById('max_objects'); + if (maxMbInput) maxMbInput.value = maxBytes ? Math.floor(maxBytes / 1048576) : ''; + if (maxObjInput) maxObjInput.value = maxObjects || ''; + } + + function updatePolicyCard(hasPolicy, preset) { + var policyCard = document.querySelector('#permissions-pane .card'); + if (!policyCard) return; + var alertContainer = policyCard.querySelector('.alert'); + if (alertContainer) { + if (hasPolicy) { + alertContainer.className = 'alert alert-info d-flex align-items-start mb-4'; + alertContainer.innerHTML = '' + + '' + + '
    Policy attached' + + '

    A bucket policy is attached to this bucket. Access is granted via both IAM and bucket policy rules.

    '; + } else { + alertContainer.className = 'alert alert-secondary d-flex align-items-start mb-4'; + alertContainer.innerHTML = '' + + '' + + '' + + '
    IAM only' + + '

    No bucket policy is attached. Access is controlled by IAM policies only.

    '; + } + } + document.querySelectorAll('.preset-btn').forEach(function (btn) { + btn.classList.remove('active'); + if (btn.dataset.preset === preset) btn.classList.add('active'); + }); + var presetInputEl = document.getElementById('policyPreset'); + if (presetInputEl) presetInputEl.value = preset; + var deletePolicyBtn = document.getElementById('deletePolicyBtn'); + if (deletePolicyBtn) { + deletePolicyBtn.style.display = hasPolicy ? '' : 'none'; + } + } + + interceptForm('enableVersioningForm', { + successMessage: 'Versioning enabled', + onSuccess: function (data) { + updateVersioningBadge(true); + updateVersioningCard(true); + } + }); + + interceptForm('suspendVersioningForm', { + successMessage: 'Versioning suspended', + closeModal: 'suspendVersioningModal', + onSuccess: function (data) { + updateVersioningBadge(false); + updateVersioningCard(false); + } + }); + + interceptForm('encryptionForm', { + successMessage: 'Encryption settings saved', + onSuccess: function (data) { + updateEncryptionCard(data.enabled !== false, data.algorithm || 'AES256'); + } + }); + + interceptForm('quotaForm', { + successMessage: 'Quota settings saved', + onSuccess: function (data) { + updateQuotaCard(data.has_quota, data.max_bytes, data.max_objects); + } + }); + + interceptForm('websiteForm', { + successMessage: 'Website settings saved', + onSuccess: function (data) { + updateWebsiteCard(data.enabled !== false, data.index_document, data.error_document); + } + }); + + interceptForm('bucketPolicyForm', { + successMessage: 'Bucket policy saved', + onSuccess: function (data) { + var policyModeEl = document.getElementById('policyMode'); + var policyPresetEl = document.getElementById('policyPreset'); + var preset = policyModeEl && policyModeEl.value === 'delete' ? 'private' : + (policyPresetEl?.value || 'custom'); + updatePolicyCard(preset !== 'private', preset); + } + }); + + var deletePolicyForm = document.getElementById('deletePolicyForm'); + if (deletePolicyForm) { + deletePolicyForm.addEventListener('submit', function (e) { + e.preventDefault(); + window.UICore.submitFormAjax(deletePolicyForm, { + successMessage: 'Bucket policy deleted', + onSuccess: function (data) { + var modal = bootstrap.Modal.getInstance(document.getElementById('deletePolicyModal')); + if (modal) modal.hide(); + updatePolicyCard(false, 'private'); + var policyTextarea = document.getElementById('policyDocument'); + if (policyTextarea) policyTextarea.value = ''; + } + }); + }); + } + + var confirmDisableEncBtn = document.getElementById('confirmDisableEncryptionBtn'); + if (confirmDisableEncBtn) { + confirmDisableEncBtn.addEventListener('click', function () { + var form = document.getElementById('encryptionForm'); + if (!form) return; + document.getElementById('encryptionAction').value = 'disable'; + var modalEl = document.getElementById('disableEncryptionModal'); + var modal = modalEl ? bootstrap.Modal.getInstance(modalEl) : null; + window.UICore.submitFormAjax(form, { + successMessage: 'Encryption disabled', + onSuccess: function (data) { + document.getElementById('encryptionAction').value = 'enable'; + if (modal) modal.hide(); + updateEncryptionCard(false, null); + }, + onError: function () { + document.getElementById('encryptionAction').value = 'enable'; + if (modal) modal.hide(); + } + }); + }); + } + + var removeQuotaBtn = document.getElementById('removeQuotaBtn'); + if (removeQuotaBtn) { + removeQuotaBtn.addEventListener('click', function () { + var form = document.getElementById('quotaForm'); + if (!form) return; + document.getElementById('quotaAction').value = 'remove'; + window.UICore.submitFormAjax(form, { + successMessage: 'Quota removed', + onSuccess: function (data) { + document.getElementById('quotaAction').value = 'set'; + updateQuotaCard(false, null, null); + } + }); + }); + } + + function updateWebsiteCard(enabled, indexDoc, errorDoc) { + var card = document.getElementById('bucket-website-card'); + if (!card) return; + var alertContainer = card.querySelector('.alert'); + if (alertContainer) { + if (enabled) { + alertContainer.className = 'alert alert-success d-flex align-items-start mb-4'; + var detail = 'Index: ' + escapeHtml(indexDoc || 'index.html') + ''; + if (errorDoc) detail += '
    Error: ' + escapeHtml(errorDoc) + ''; + alertContainer.innerHTML = '' + + '' + + '
    Website hosting is enabled' + + '

    ' + detail + '

    '; + } else { + alertContainer.className = 'alert alert-secondary d-flex align-items-start mb-4'; + alertContainer.innerHTML = '' + + '' + + '' + + '
    Website hosting is disabled' + + '

    Enable website hosting to serve bucket contents as a static website.

    '; + } + } + var disableBtn = document.getElementById('disableWebsiteBtn'); + if (disableBtn) { + disableBtn.style.display = enabled ? '' : 'none'; + } + var submitBtn = document.getElementById('websiteSubmitBtn'); + if (submitBtn) { + submitBtn.classList.remove('btn-primary', 'btn-success'); + submitBtn.classList.add(enabled ? 'btn-primary' : 'btn-success'); + } + var submitLabel = document.getElementById('websiteSubmitLabel'); + if (submitLabel) { + submitLabel.textContent = enabled ? 'Save Website Settings' : 'Enable Website Hosting'; + } + } + + var disableWebsiteBtn = document.getElementById('disableWebsiteBtn'); + if (disableWebsiteBtn) { + disableWebsiteBtn.addEventListener('click', function () { + var form = document.getElementById('websiteForm'); + if (!form) return; + document.getElementById('websiteAction').value = 'disable'; + window.UICore.submitFormAjax(form, { + successMessage: 'Website hosting disabled', + onSuccess: function (data) { + document.getElementById('websiteAction').value = 'enable'; + updateWebsiteCard(false, null, null); + } + }); + }); + } + + function reloadReplicationPane() { + var replicationPane = document.getElementById('replication-pane'); + if (!replicationPane) return; + fetch(window.location.pathname + '?tab=replication', { + headers: { 'X-Requested-With': 'XMLHttpRequest' } + }) + .then(function (resp) { return resp.text(); }) + .then(function (html) { + var parser = new DOMParser(); + var doc = parser.parseFromString(html, 'text/html'); + var newPane = doc.getElementById('replication-pane'); + if (newPane) { + replicationPane.innerHTML = newPane.innerHTML; + initReplicationForms(); + initReplicationStats(); + } + }) + .catch(function (err) { + console.error('Failed to reload replication pane:', err); + }); + } + + function initReplicationForms() { + document.querySelectorAll('form[action*="replication"]').forEach(function (form) { + if (form.dataset.ajaxBound) return; + form.dataset.ajaxBound = 'true'; + var actionInput = form.querySelector('input[name="action"]'); + if (!actionInput) return; + var action = actionInput.value; + + form.addEventListener('submit', function (e) { + e.preventDefault(); + var msg = action === 'pause' ? 'Replication paused' : + action === 'resume' ? 'Replication resumed' : + action === 'delete' ? 'Replication disabled' : + action === 'create' ? 'Replication configured' : 'Operation completed'; + window.UICore.submitFormAjax(form, { + successMessage: msg, + onSuccess: function (data) { + var modal = bootstrap.Modal.getInstance(document.getElementById('disableReplicationModal')); + if (modal) modal.hide(); + reloadReplicationPane(); + } + }); + }); + }); + } + + function initReplicationStats() { + var statsContainer = document.getElementById('replication-stats-cards'); + if (!statsContainer) return; + var statusEndpoint = statsContainer.dataset.statusEndpoint; + if (!statusEndpoint) return; + + var syncedEl = statsContainer.querySelector('[data-stat="synced"]'); + var pendingEl = statsContainer.querySelector('[data-stat="pending"]'); + var orphanedEl = statsContainer.querySelector('[data-stat="orphaned"]'); + var bytesEl = statsContainer.querySelector('[data-stat="bytes"]'); + + fetch(statusEndpoint) + .then(function (resp) { return resp.json(); }) + .then(function (data) { + if (syncedEl) syncedEl.textContent = data.objects_synced || 0; + if (pendingEl) pendingEl.textContent = data.objects_pending || 0; + if (orphanedEl) orphanedEl.textContent = data.objects_orphaned || 0; + if (bytesEl) bytesEl.textContent = formatBytes(data.bytes_synced || 0); + }) + .catch(function (err) { + console.error('Failed to load replication stats:', err); + }); + } + + initReplicationForms(); + initReplicationStats(); + + var deleteBucketForm = document.getElementById('deleteBucketForm'); + if (deleteBucketForm) { + deleteBucketForm.addEventListener('submit', function (e) { + e.preventDefault(); + window.UICore.submitFormAjax(deleteBucketForm, { + onSuccess: function () { + sessionStorage.setItem('flashMessage', JSON.stringify({ title: 'Bucket deleted', variant: 'success' })); + window.location.href = window.BucketDetailConfig?.endpoints?.bucketsOverview || '/ui/buckets'; + } + }); + }); + } + + window.BucketDetailConfig = window.BucketDetailConfig || {}; + +})(); diff --git a/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-operations.js b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-operations.js new file mode 100644 index 0000000..d5791d4 --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-operations.js @@ -0,0 +1,192 @@ +window.BucketDetailOperations = (function() { + 'use strict'; + + let showMessage = function() {}; + let escapeHtml = function(s) { return s; }; + + function init(config) { + showMessage = config.showMessage || showMessage; + escapeHtml = config.escapeHtml || escapeHtml; + } + + async function loadLifecycleRules(card, endpoint) { + if (!card || !endpoint) return; + const body = card.querySelector('[data-lifecycle-body]'); + if (!body) return; + + try { + const response = await fetch(endpoint); + const data = await response.json(); + + if (!response.ok) { + body.innerHTML = `${escapeHtml(data.error || 'Failed to load')}`; + return; + } + + const rules = data.rules || []; + if (rules.length === 0) { + body.innerHTML = 'No lifecycle rules configured'; + return; + } + + body.innerHTML = rules.map(rule => { + const actions = []; + if (rule.expiration_days) actions.push(`Delete after ${rule.expiration_days} days`); + if (rule.noncurrent_days) actions.push(`Delete old versions after ${rule.noncurrent_days} days`); + if (rule.abort_mpu_days) actions.push(`Abort incomplete MPU after ${rule.abort_mpu_days} days`); + + return ` + + ${escapeHtml(rule.id)} + ${escapeHtml(rule.prefix || '(all)')} + ${actions.map(a => `
    ${escapeHtml(a)}
    `).join('')} + + ${escapeHtml(rule.status)} + + + + + + `; + }).join(''); + } catch (err) { + body.innerHTML = `${escapeHtml(err.message)}`; + } + } + + async function loadCorsRules(card, endpoint) { + if (!card || !endpoint) return; + const body = document.getElementById('cors-rules-body'); + if (!body) return; + + try { + const response = await fetch(endpoint); + const data = await response.json(); + + if (!response.ok) { + body.innerHTML = `${escapeHtml(data.error || 'Failed to load')}`; + return; + } + + const rules = data.rules || []; + if (rules.length === 0) { + body.innerHTML = 'No CORS rules configured'; + return; + } + + body.innerHTML = rules.map((rule, idx) => ` + + ${(rule.allowed_origins || []).map(o => `${escapeHtml(o)}`).join('')} + ${(rule.allowed_methods || []).map(m => `${escapeHtml(m)}`).join('')} + ${(rule.allowed_headers || []).slice(0, 3).join(', ')}${(rule.allowed_headers || []).length > 3 ? '...' : ''} + ${rule.max_age_seconds || 0}s + + + + + `).join(''); + } catch (err) { + body.innerHTML = `${escapeHtml(err.message)}`; + } + } + + async function loadAcl(card, endpoint) { + if (!card || !endpoint) return; + const body = card.querySelector('[data-acl-body]'); + if (!body) return; + + try { + const response = await fetch(endpoint); + const data = await response.json(); + + if (!response.ok) { + body.innerHTML = `${escapeHtml(data.error || 'Failed to load')}`; + return; + } + + const grants = data.grants || []; + if (grants.length === 0) { + body.innerHTML = 'No ACL grants configured'; + return; + } + + body.innerHTML = grants.map(grant => { + const grantee = grant.grantee_type === 'CanonicalUser' + ? grant.display_name || grant.grantee_id + : grant.grantee_uri || grant.grantee_type; + return ` + + ${escapeHtml(grantee)} + ${escapeHtml(grant.permission)} + ${escapeHtml(grant.grantee_type)} + + `; + }).join(''); + } catch (err) { + body.innerHTML = `${escapeHtml(err.message)}`; + } + } + + async function deleteLifecycleRule(ruleId) { + if (!confirm(`Delete lifecycle rule "${ruleId}"?`)) return; + const card = document.getElementById('lifecycle-rules-card'); + if (!card) return; + const endpoint = card.dataset.lifecycleUrl; + const csrfToken = window.getCsrfToken ? window.getCsrfToken() : ''; + + try { + const resp = await fetch(endpoint, { + method: 'DELETE', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': csrfToken }, + body: JSON.stringify({ rule_id: ruleId }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to delete'); + showMessage({ title: 'Rule deleted', body: `Lifecycle rule "${ruleId}" has been deleted.`, variant: 'success' }); + loadLifecycleRules(card, endpoint); + } catch (err) { + showMessage({ title: 'Delete failed', body: err.message, variant: 'danger' }); + } + } + + async function deleteCorsRule(index) { + if (!confirm('Delete this CORS rule?')) return; + const card = document.getElementById('cors-rules-card'); + if (!card) return; + const endpoint = card.dataset.corsUrl; + const csrfToken = window.getCsrfToken ? window.getCsrfToken() : ''; + + try { + const resp = await fetch(endpoint, { + method: 'DELETE', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': csrfToken }, + body: JSON.stringify({ rule_index: index }) + }); + const data = await resp.json(); + if (!resp.ok) throw new Error(data.error || 'Failed to delete'); + showMessage({ title: 'Rule deleted', body: 'CORS rule has been deleted.', variant: 'success' }); + loadCorsRules(card, endpoint); + } catch (err) { + showMessage({ title: 'Delete failed', body: err.message, variant: 'danger' }); + } + } + + return { + init: init, + loadLifecycleRules: loadLifecycleRules, + loadCorsRules: loadCorsRules, + loadAcl: loadAcl, + deleteLifecycleRule: deleteLifecycleRule, + deleteCorsRule: deleteCorsRule + }; +})(); diff --git a/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-upload.js b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-upload.js new file mode 100644 index 0000000..6e0c6a7 --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-upload.js @@ -0,0 +1,600 @@ +window.BucketDetailUpload = (function() { + 'use strict'; + + const MULTIPART_THRESHOLD = 8 * 1024 * 1024; + const CHUNK_SIZE = 8 * 1024 * 1024; + const MAX_PART_RETRIES = 3; + const RETRY_BASE_DELAY_MS = 1000; + + let state = { + isUploading: false, + uploadProgress: { current: 0, total: 0, currentFile: '' } + }; + + let elements = {}; + let callbacks = {}; + + function init(config) { + elements = { + uploadForm: config.uploadForm, + uploadFileInput: config.uploadFileInput, + uploadModal: config.uploadModal, + uploadModalEl: config.uploadModalEl, + uploadSubmitBtn: config.uploadSubmitBtn, + uploadCancelBtn: config.uploadCancelBtn, + uploadBtnText: config.uploadBtnText, + uploadDropZone: config.uploadDropZone, + uploadDropZoneLabel: config.uploadDropZoneLabel, + uploadProgressStack: config.uploadProgressStack, + uploadKeyPrefix: config.uploadKeyPrefix, + singleFileOptions: config.singleFileOptions, + bulkUploadProgress: config.bulkUploadProgress, + bulkUploadStatus: config.bulkUploadStatus, + bulkUploadCounter: config.bulkUploadCounter, + bulkUploadProgressBar: config.bulkUploadProgressBar, + bulkUploadCurrentFile: config.bulkUploadCurrentFile, + bulkUploadResults: config.bulkUploadResults, + bulkUploadSuccessAlert: config.bulkUploadSuccessAlert, + bulkUploadErrorAlert: config.bulkUploadErrorAlert, + bulkUploadSuccessCount: config.bulkUploadSuccessCount, + bulkUploadErrorCount: config.bulkUploadErrorCount, + bulkUploadErrorList: config.bulkUploadErrorList, + floatingProgress: config.floatingProgress, + floatingProgressBar: config.floatingProgressBar, + floatingProgressStatus: config.floatingProgressStatus, + floatingProgressTitle: config.floatingProgressTitle, + floatingProgressExpand: config.floatingProgressExpand + }; + + callbacks = { + showMessage: config.showMessage || function() {}, + formatBytes: config.formatBytes || function(b) { return b + ' bytes'; }, + escapeHtml: config.escapeHtml || function(s) { return s; }, + onUploadComplete: config.onUploadComplete || function() {}, + hasFolders: config.hasFolders || function() { return false; }, + getCurrentPrefix: config.getCurrentPrefix || function() { return ''; } + }; + + setupEventListeners(); + setupBeforeUnload(); + } + + function isUploading() { + return state.isUploading; + } + + function setupBeforeUnload() { + window.addEventListener('beforeunload', (e) => { + if (state.isUploading) { + e.preventDefault(); + e.returnValue = 'Upload in progress. Are you sure you want to leave?'; + return e.returnValue; + } + }); + } + + function showFloatingProgress() { + if (elements.floatingProgress) { + elements.floatingProgress.classList.remove('d-none'); + } + } + + function hideFloatingProgress() { + if (elements.floatingProgress) { + elements.floatingProgress.classList.add('d-none'); + } + } + + function updateFloatingProgress(current, total, currentFile) { + state.uploadProgress = { current, total, currentFile: currentFile || '' }; + if (elements.floatingProgressBar && total > 0) { + const percent = Math.round((current / total) * 100); + elements.floatingProgressBar.style.width = `${percent}%`; + } + if (elements.floatingProgressStatus) { + if (currentFile) { + elements.floatingProgressStatus.textContent = `${current}/${total} files - ${currentFile}`; + } else { + elements.floatingProgressStatus.textContent = `${current}/${total} files completed`; + } + } + if (elements.floatingProgressTitle) { + elements.floatingProgressTitle.textContent = `Uploading ${total} file${total !== 1 ? 's' : ''}...`; + } + } + + function refreshUploadDropLabel() { + if (!elements.uploadDropZoneLabel || !elements.uploadFileInput) return; + const files = elements.uploadFileInput.files; + if (!files || files.length === 0) { + elements.uploadDropZoneLabel.textContent = 'No file selected'; + if (elements.singleFileOptions) elements.singleFileOptions.classList.remove('d-none'); + return; + } + elements.uploadDropZoneLabel.textContent = files.length === 1 ? files[0].name : `${files.length} files selected`; + if (elements.singleFileOptions) { + elements.singleFileOptions.classList.toggle('d-none', files.length > 1); + } + } + + function updateUploadBtnText() { + if (!elements.uploadBtnText || !elements.uploadFileInput) return; + const files = elements.uploadFileInput.files; + if (!files || files.length <= 1) { + elements.uploadBtnText.textContent = 'Upload'; + } else { + elements.uploadBtnText.textContent = `Upload ${files.length} files`; + } + } + + function resetUploadUI() { + if (elements.bulkUploadProgress) elements.bulkUploadProgress.classList.add('d-none'); + if (elements.bulkUploadResults) elements.bulkUploadResults.classList.add('d-none'); + if (elements.bulkUploadSuccessAlert) elements.bulkUploadSuccessAlert.classList.remove('d-none'); + if (elements.bulkUploadErrorAlert) elements.bulkUploadErrorAlert.classList.add('d-none'); + if (elements.bulkUploadErrorList) elements.bulkUploadErrorList.innerHTML = ''; + if (elements.uploadSubmitBtn) elements.uploadSubmitBtn.disabled = false; + if (elements.uploadFileInput) elements.uploadFileInput.disabled = false; + if (elements.uploadProgressStack) elements.uploadProgressStack.innerHTML = ''; + if (elements.uploadDropZone) { + elements.uploadDropZone.classList.remove('upload-locked'); + elements.uploadDropZone.style.pointerEvents = ''; + } + state.isUploading = false; + hideFloatingProgress(); + } + + function setUploadLockState(locked) { + if (elements.uploadDropZone) { + elements.uploadDropZone.classList.toggle('upload-locked', locked); + elements.uploadDropZone.style.pointerEvents = locked ? 'none' : ''; + } + if (elements.uploadFileInput) { + elements.uploadFileInput.disabled = locked; + } + } + + function createProgressItem(file) { + const item = document.createElement('div'); + item.className = 'upload-progress-item'; + item.dataset.state = 'uploading'; + item.innerHTML = ` +
    +
    +
    ${callbacks.escapeHtml(file.name)}
    +
    ${callbacks.formatBytes(file.size)}
    +
    +
    Preparing...
    +
    +
    +
    +
    +
    +
    + 0 B + 0% +
    +
    + `; + return item; + } + + function updateProgressItem(item, { loaded, total, status, progressState, error }) { + if (progressState) item.dataset.state = progressState; + const statusEl = item.querySelector('.upload-status'); + const progressBar = item.querySelector('.progress-bar'); + const progressLoaded = item.querySelector('.progress-loaded'); + const progressPercent = item.querySelector('.progress-percent'); + + if (status) { + statusEl.textContent = status; + statusEl.className = 'upload-status text-end ms-2'; + if (progressState === 'success') statusEl.classList.add('success'); + if (progressState === 'error') statusEl.classList.add('error'); + } + if (typeof loaded === 'number' && typeof total === 'number' && total > 0) { + const percent = Math.round((loaded / total) * 100); + progressBar.style.width = `${percent}%`; + progressLoaded.textContent = `${callbacks.formatBytes(loaded)} / ${callbacks.formatBytes(total)}`; + progressPercent.textContent = `${percent}%`; + } + if (error) { + const progressContainer = item.querySelector('.progress-container'); + if (progressContainer) { + progressContainer.innerHTML = `
    ${callbacks.escapeHtml(error)}
    `; + } + } + } + + function uploadPartXHR(url, chunk, csrfToken, baseBytes, fileSize, progressItem, partNumber, totalParts) { + return new Promise((resolve, reject) => { + const xhr = new XMLHttpRequest(); + xhr.open('PUT', url, true); + xhr.setRequestHeader('X-CSRFToken', csrfToken || ''); + + xhr.upload.addEventListener('progress', (e) => { + if (e.lengthComputable) { + updateProgressItem(progressItem, { + status: `Part ${partNumber}/${totalParts}`, + loaded: baseBytes + e.loaded, + total: fileSize + }); + } + }); + + xhr.addEventListener('load', () => { + if (xhr.status >= 200 && xhr.status < 300) { + try { + resolve(JSON.parse(xhr.responseText)); + } catch { + reject(new Error(`Part ${partNumber}: invalid response`)); + } + } else { + try { + const data = JSON.parse(xhr.responseText); + reject(new Error(data.error || `Part ${partNumber} failed (${xhr.status})`)); + } catch { + reject(new Error(`Part ${partNumber} failed (${xhr.status})`)); + } + } + }); + + xhr.addEventListener('error', () => reject(new Error(`Part ${partNumber}: network error`))); + xhr.addEventListener('abort', () => reject(new Error(`Part ${partNumber}: aborted`))); + + xhr.send(chunk); + }); + } + + async function uploadPartWithRetry(url, chunk, csrfToken, baseBytes, fileSize, progressItem, partNumber, totalParts) { + let lastError; + for (let attempt = 0; attempt <= MAX_PART_RETRIES; attempt++) { + try { + return await uploadPartXHR(url, chunk, csrfToken, baseBytes, fileSize, progressItem, partNumber, totalParts); + } catch (err) { + lastError = err; + if (attempt < MAX_PART_RETRIES) { + const delay = RETRY_BASE_DELAY_MS * Math.pow(2, attempt); + updateProgressItem(progressItem, { + status: `Part ${partNumber}/${totalParts} retry ${attempt + 1}/${MAX_PART_RETRIES}...`, + loaded: baseBytes, + total: fileSize + }); + await new Promise(r => setTimeout(r, delay)); + } + } + } + throw lastError; + } + + async function uploadMultipart(file, objectKey, metadata, progressItem, urls) { + const csrfToken = document.querySelector('input[name="csrf_token"]')?.value; + + updateProgressItem(progressItem, { status: 'Initiating...', loaded: 0, total: file.size }); + const initResp = await fetch(urls.initUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': csrfToken || '' }, + body: JSON.stringify({ object_key: objectKey, metadata }) + }); + if (!initResp.ok) { + const err = await initResp.json().catch(() => ({})); + throw new Error(err.error || 'Failed to initiate upload'); + } + const { upload_id } = await initResp.json(); + + const partUrl = urls.partTemplate.replace('UPLOAD_ID_PLACEHOLDER', upload_id); + const completeUrl = urls.completeTemplate.replace('UPLOAD_ID_PLACEHOLDER', upload_id); + const abortUrl = urls.abortTemplate.replace('UPLOAD_ID_PLACEHOLDER', upload_id); + + const parts = []; + const totalParts = Math.ceil(file.size / CHUNK_SIZE); + let uploadedBytes = 0; + + try { + for (let partNumber = 1; partNumber <= totalParts; partNumber++) { + const start = (partNumber - 1) * CHUNK_SIZE; + const end = Math.min(start + CHUNK_SIZE, file.size); + const chunk = file.slice(start, end); + + const partData = await uploadPartWithRetry( + `${partUrl}?partNumber=${partNumber}`, + chunk, csrfToken, uploadedBytes, file.size, + progressItem, partNumber, totalParts + ); + + parts.push({ part_number: partNumber, etag: partData.etag }); + uploadedBytes += (end - start); + + updateProgressItem(progressItem, { + loaded: uploadedBytes, + total: file.size + }); + } + + updateProgressItem(progressItem, { status: 'Completing...', loaded: file.size, total: file.size }); + const completeResp = await fetch(completeUrl, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-CSRFToken': csrfToken || '' }, + body: JSON.stringify({ parts }) + }); + + if (!completeResp.ok) { + const err = await completeResp.json().catch(() => ({})); + throw new Error(err.error || 'Failed to complete upload'); + } + + return await completeResp.json(); + } catch (err) { + try { + await fetch(abortUrl, { method: 'DELETE', headers: { 'X-CSRFToken': csrfToken || '' } }); + } catch {} + throw err; + } + } + + async function uploadRegular(file, objectKey, metadata, progressItem, formAction) { + return new Promise((resolve, reject) => { + const formData = new FormData(); + formData.append('object', file); + formData.append('object_key', objectKey); + if (metadata) formData.append('metadata', JSON.stringify(metadata)); + const csrfToken = document.querySelector('input[name="csrf_token"]')?.value; + if (csrfToken) formData.append('csrf_token', csrfToken); + + const xhr = new XMLHttpRequest(); + xhr.open('POST', formAction, true); + xhr.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); + xhr.setRequestHeader('X-CSRFToken', csrfToken || ''); + + xhr.upload.addEventListener('progress', (e) => { + if (e.lengthComputable) { + updateProgressItem(progressItem, { + status: 'Uploading...', + loaded: e.loaded, + total: e.total + }); + } + }); + + xhr.addEventListener('load', () => { + if (xhr.status >= 200 && xhr.status < 300) { + try { + const data = JSON.parse(xhr.responseText); + if (data.status === 'error') { + reject(new Error(data.message || 'Upload failed')); + } else { + resolve(data); + } + } catch { + resolve({}); + } + } else { + try { + const data = JSON.parse(xhr.responseText); + reject(new Error(data.message || `Upload failed (${xhr.status})`)); + } catch { + reject(new Error(`Upload failed (${xhr.status})`)); + } + } + }); + + xhr.addEventListener('error', () => reject(new Error('Network error'))); + xhr.addEventListener('abort', () => reject(new Error('Upload aborted'))); + + xhr.send(formData); + }); + } + + async function uploadSingleFile(file, keyPrefix, metadata, progressItem, urls) { + const objectKey = keyPrefix ? `${keyPrefix}${file.name}` : file.name; + const shouldUseMultipart = file.size >= MULTIPART_THRESHOLD && urls.initUrl; + + if (!progressItem && elements.uploadProgressStack) { + progressItem = createProgressItem(file); + elements.uploadProgressStack.appendChild(progressItem); + } + + try { + let result; + if (shouldUseMultipart) { + updateProgressItem(progressItem, { status: 'Multipart upload...', loaded: 0, total: file.size }); + result = await uploadMultipart(file, objectKey, metadata, progressItem, urls); + } else { + updateProgressItem(progressItem, { status: 'Uploading...', loaded: 0, total: file.size }); + result = await uploadRegular(file, objectKey, metadata, progressItem, urls.formAction); + } + updateProgressItem(progressItem, { progressState: 'success', status: 'Complete', loaded: file.size, total: file.size }); + return result; + } catch (err) { + updateProgressItem(progressItem, { progressState: 'error', status: 'Failed', error: err.message }); + throw err; + } + } + + async function performBulkUpload(files, urls) { + if (state.isUploading || !files || files.length === 0) return; + + state.isUploading = true; + setUploadLockState(true); + const keyPrefix = (elements.uploadKeyPrefix?.value || '').trim(); + const metadataRaw = elements.uploadForm?.querySelector('textarea[name="metadata"]')?.value?.trim(); + let metadata = null; + if (metadataRaw) { + try { + metadata = JSON.parse(metadataRaw); + } catch { + callbacks.showMessage({ title: 'Invalid metadata', body: 'Metadata must be valid JSON.', variant: 'danger' }); + resetUploadUI(); + return; + } + } + + if (elements.bulkUploadProgress) elements.bulkUploadProgress.classList.remove('d-none'); + if (elements.bulkUploadResults) elements.bulkUploadResults.classList.add('d-none'); + if (elements.uploadSubmitBtn) elements.uploadSubmitBtn.disabled = true; + if (elements.uploadFileInput) elements.uploadFileInput.disabled = true; + + const successFiles = []; + const errorFiles = []; + const total = files.length; + + updateFloatingProgress(0, total, files[0]?.name || ''); + + for (let i = 0; i < total; i++) { + const file = files[i]; + const current = i + 1; + + if (elements.bulkUploadCounter) elements.bulkUploadCounter.textContent = `${current}/${total}`; + if (elements.bulkUploadCurrentFile) elements.bulkUploadCurrentFile.textContent = `Uploading: ${file.name}`; + if (elements.bulkUploadProgressBar) { + const percent = Math.round((current / total) * 100); + elements.bulkUploadProgressBar.style.width = `${percent}%`; + } + updateFloatingProgress(i, total, file.name); + + try { + await uploadSingleFile(file, keyPrefix, metadata, null, urls); + successFiles.push(file.name); + } catch (error) { + errorFiles.push({ name: file.name, error: error.message || 'Unknown error' }); + } + } + updateFloatingProgress(total, total); + + if (elements.bulkUploadProgress) elements.bulkUploadProgress.classList.add('d-none'); + if (elements.bulkUploadResults) elements.bulkUploadResults.classList.remove('d-none'); + + if (elements.bulkUploadSuccessCount) elements.bulkUploadSuccessCount.textContent = successFiles.length; + if (successFiles.length === 0 && elements.bulkUploadSuccessAlert) { + elements.bulkUploadSuccessAlert.classList.add('d-none'); + } + + if (errorFiles.length > 0) { + if (elements.bulkUploadErrorCount) elements.bulkUploadErrorCount.textContent = errorFiles.length; + if (elements.bulkUploadErrorAlert) elements.bulkUploadErrorAlert.classList.remove('d-none'); + if (elements.bulkUploadErrorList) { + elements.bulkUploadErrorList.innerHTML = errorFiles + .map(f => `
  • ${callbacks.escapeHtml(f.name)}: ${callbacks.escapeHtml(f.error)}
    • `) + .join(''); + } + } + + state.isUploading = false; + setUploadLockState(false); + + if (successFiles.length > 0) { + if (elements.uploadBtnText) elements.uploadBtnText.textContent = 'Refreshing...'; + callbacks.onUploadComplete(successFiles, errorFiles); + } else { + if (elements.uploadSubmitBtn) elements.uploadSubmitBtn.disabled = false; + if (elements.uploadFileInput) elements.uploadFileInput.disabled = false; + } + } + + function setupEventListeners() { + if (elements.uploadFileInput) { + elements.uploadFileInput.addEventListener('change', () => { + if (state.isUploading) return; + refreshUploadDropLabel(); + updateUploadBtnText(); + resetUploadUI(); + }); + } + + if (elements.uploadDropZone) { + elements.uploadDropZone.addEventListener('click', () => { + if (state.isUploading) return; + elements.uploadFileInput?.click(); + }); + } + + if (elements.floatingProgressExpand) { + elements.floatingProgressExpand.addEventListener('click', () => { + if (elements.uploadModal) { + elements.uploadModal.show(); + } + }); + } + + if (elements.uploadModalEl) { + elements.uploadModalEl.addEventListener('hide.bs.modal', () => { + if (state.isUploading) { + showFloatingProgress(); + } + }); + + elements.uploadModalEl.addEventListener('hidden.bs.modal', () => { + if (!state.isUploading) { + resetUploadUI(); + if (elements.uploadFileInput) elements.uploadFileInput.value = ''; + refreshUploadDropLabel(); + updateUploadBtnText(); + } + }); + + elements.uploadModalEl.addEventListener('show.bs.modal', () => { + if (state.isUploading) { + hideFloatingProgress(); + } + if (callbacks.hasFolders() && callbacks.getCurrentPrefix()) { + if (elements.uploadKeyPrefix) { + elements.uploadKeyPrefix.value = callbacks.getCurrentPrefix(); + } + } else if (elements.uploadKeyPrefix) { + elements.uploadKeyPrefix.value = ''; + } + }); + } + } + + function wireDropTarget(target, options) { + const { highlightClass = '', autoOpenModal = false } = options || {}; + if (!target) return; + + const preventDefaults = (event) => { + event.preventDefault(); + event.stopPropagation(); + }; + + ['dragenter', 'dragover'].forEach((eventName) => { + target.addEventListener(eventName, (event) => { + preventDefaults(event); + if (state.isUploading) return; + if (highlightClass) { + target.classList.add(highlightClass); + } + }); + }); + + ['dragleave', 'drop'].forEach((eventName) => { + target.addEventListener(eventName, (event) => { + preventDefaults(event); + if (highlightClass) { + target.classList.remove(highlightClass); + } + }); + }); + + target.addEventListener('drop', (event) => { + if (state.isUploading) return; + if (!event.dataTransfer?.files?.length || !elements.uploadFileInput) { + return; + } + elements.uploadFileInput.files = event.dataTransfer.files; + elements.uploadFileInput.dispatchEvent(new Event('change', { bubbles: true })); + if (autoOpenModal && elements.uploadModal) { + elements.uploadModal.show(); + } + }); + } + + return { + init: init, + isUploading: isUploading, + performBulkUpload: performBulkUpload, + wireDropTarget: wireDropTarget, + resetUploadUI: resetUploadUI, + refreshUploadDropLabel: refreshUploadDropLabel, + updateUploadBtnText: updateUploadBtnText + }; +})(); diff --git a/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-utils.js b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-utils.js new file mode 100644 index 0000000..43cc91e --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/js/bucket-detail-utils.js @@ -0,0 +1,120 @@ +window.BucketDetailUtils = (function() { + 'use strict'; + + function setupJsonAutoIndent(textarea) { + if (!textarea) return; + + textarea.addEventListener('keydown', function(e) { + if (e.key === 'Enter') { + e.preventDefault(); + + const start = this.selectionStart; + const end = this.selectionEnd; + const value = this.value; + + const lineStart = value.lastIndexOf('\n', start - 1) + 1; + const currentLine = value.substring(lineStart, start); + + const indentMatch = currentLine.match(/^(\s*)/); + let indent = indentMatch ? indentMatch[1] : ''; + + const trimmedLine = currentLine.trim(); + const lastChar = trimmedLine.slice(-1); + + let newIndent = indent; + let insertAfter = ''; + + if (lastChar === '{' || lastChar === '[') { + newIndent = indent + ' '; + + const charAfterCursor = value.substring(start, start + 1).trim(); + if ((lastChar === '{' && charAfterCursor === '}') || + (lastChar === '[' && charAfterCursor === ']')) { + insertAfter = '\n' + indent; + } + } else if (lastChar === ',' || lastChar === ':') { + newIndent = indent; + } + + const insertion = '\n' + newIndent + insertAfter; + const newValue = value.substring(0, start) + insertion + value.substring(end); + + this.value = newValue; + + const newCursorPos = start + 1 + newIndent.length; + this.selectionStart = this.selectionEnd = newCursorPos; + + this.dispatchEvent(new Event('input', { bubbles: true })); + } + + if (e.key === 'Tab') { + e.preventDefault(); + const start = this.selectionStart; + const end = this.selectionEnd; + + if (e.shiftKey) { + const lineStart = this.value.lastIndexOf('\n', start - 1) + 1; + const lineContent = this.value.substring(lineStart, start); + if (lineContent.startsWith(' ')) { + this.value = this.value.substring(0, lineStart) + + this.value.substring(lineStart + 2); + this.selectionStart = this.selectionEnd = Math.max(lineStart, start - 2); + } + } else { + this.value = this.value.substring(0, start) + ' ' + this.value.substring(end); + this.selectionStart = this.selectionEnd = start + 2; + } + + this.dispatchEvent(new Event('input', { bubbles: true })); + } + }); + } + + function formatBytes(bytes) { + if (!Number.isFinite(bytes)) return `${bytes} bytes`; + const units = ['bytes', 'KB', 'MB', 'GB', 'TB']; + let i = 0; + let size = bytes; + while (size >= 1024 && i < units.length - 1) { + size /= 1024; + i++; + } + return `${size.toFixed(i === 0 ? 0 : 1)} ${units[i]}`; + } + + function escapeHtml(value) { + if (value === null || value === undefined) return ''; + return String(value) + .replace(/&/g, '&') + .replace(//g, '>') + .replace(/"/g, '"') + .replace(/'/g, '''); + } + + function fallbackCopy(text) { + const textArea = document.createElement('textarea'); + textArea.value = text; + textArea.style.position = 'fixed'; + textArea.style.left = '-9999px'; + textArea.style.top = '-9999px'; + document.body.appendChild(textArea); + textArea.focus(); + textArea.select(); + let success = false; + try { + success = document.execCommand('copy'); + } catch { + success = false; + } + document.body.removeChild(textArea); + return success; + } + + return { + setupJsonAutoIndent: setupJsonAutoIndent, + formatBytes: formatBytes, + escapeHtml: escapeHtml, + fallbackCopy: fallbackCopy + }; +})(); diff --git a/rust/myfsio-engine/crates/myfsio-server/static/js/connections-management.js b/rust/myfsio-engine/crates/myfsio-server/static/js/connections-management.js new file mode 100644 index 0000000..8d22a6c --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/js/connections-management.js @@ -0,0 +1,343 @@ +window.ConnectionsManagement = (function() { + 'use strict'; + + var endpoints = {}; + var csrfToken = ''; + + function init(config) { + endpoints = config.endpoints || {}; + csrfToken = config.csrfToken || ''; + + setupEventListeners(); + checkAllConnectionHealth(); + } + + function togglePassword(id) { + var input = document.getElementById(id); + if (input) { + input.type = input.type === 'password' ? 'text' : 'password'; + } + } + + async function testConnection(formId, resultId) { + var form = document.getElementById(formId); + var resultDiv = document.getElementById(resultId); + if (!form || !resultDiv) return; + + var formData = new FormData(form); + var data = {}; + formData.forEach(function(value, key) { + if (key !== 'csrf_token') { + data[key] = value; + } + }); + + resultDiv.innerHTML = '
    Testing connection...
    '; + + var controller = new AbortController(); + var timeoutId = setTimeout(function() { controller.abort(); }, 20000); + + try { + var response = await fetch(endpoints.test, { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'X-CSRFToken': csrfToken + }, + body: JSON.stringify(data), + signal: controller.signal + }); + clearTimeout(timeoutId); + + var result = await response.json(); + if (response.ok) { + resultDiv.innerHTML = '
    ' + + '' + + '' + + '' + window.UICore.escapeHtml(result.message) + '
    '; + } else { + resultDiv.innerHTML = '
    ' + + '' + + '' + + '' + window.UICore.escapeHtml(result.message) + '
    '; + } + } catch (error) { + clearTimeout(timeoutId); + var message = error.name === 'AbortError' + ? 'Connection test timed out - endpoint may be unreachable' + : 'Connection failed: Network error'; + resultDiv.innerHTML = '
    ' + + '' + + '' + + '' + message + '
    '; + } + } + + async function checkConnectionHealth(connectionId, statusEl) { + if (!statusEl) return; + + try { + var controller = new AbortController(); + var timeoutId = setTimeout(function() { controller.abort(); }, 10000); + + var response = await fetch(endpoints.healthTemplate.replace('CONNECTION_ID', connectionId), { + signal: controller.signal + }); + clearTimeout(timeoutId); + + var data = await response.json(); + if (data.healthy) { + statusEl.innerHTML = '' + + ''; + statusEl.setAttribute('data-status', 'healthy'); + statusEl.setAttribute('title', 'Connected'); + } else { + statusEl.innerHTML = '' + + ''; + statusEl.setAttribute('data-status', 'unhealthy'); + statusEl.setAttribute('title', data.error || 'Unreachable'); + } + } catch (error) { + statusEl.innerHTML = '' + + ''; + statusEl.setAttribute('data-status', 'unknown'); + statusEl.setAttribute('title', 'Could not check status'); + } + } + + function checkAllConnectionHealth() { + var rows = document.querySelectorAll('tr[data-connection-id]'); + rows.forEach(function(row, index) { + var connectionId = row.getAttribute('data-connection-id'); + var statusEl = row.querySelector('.connection-status'); + if (statusEl) { + setTimeout(function() { + checkConnectionHealth(connectionId, statusEl); + }, index * 200); + } + }); + } + + function updateConnectionCount() { + var countBadge = document.querySelector('.badge.bg-primary.bg-opacity-10.text-primary.fs-6'); + if (countBadge) { + var remaining = document.querySelectorAll('tr[data-connection-id]').length; + countBadge.textContent = remaining + ' connection' + (remaining !== 1 ? 's' : ''); + } + } + + function createConnectionRowHtml(conn) { + var ak = conn.access_key || ''; + var maskedKey = ak.length > 12 ? ak.slice(0, 8) + '...' + ak.slice(-4) : ak; + + return '' + + '' + + '' + + '' + + '' + + '
    ' + + '
    ' + + '
    ' + + '' + window.UICore.escapeHtml(conn.name) + '' + + '
    ' + + '' + window.UICore.escapeHtml(conn.endpoint_url) + '' + + '' + window.UICore.escapeHtml(conn.region) + '' + + '' + window.UICore.escapeHtml(maskedKey) + '' + + '
    ' + + '' + + '' + + '
    '; + } + + function setupEventListeners() { + var testBtn = document.getElementById('testConnectionBtn'); + if (testBtn) { + testBtn.addEventListener('click', function() { + testConnection('createConnectionForm', 'testResult'); + }); + } + + var editTestBtn = document.getElementById('editTestConnectionBtn'); + if (editTestBtn) { + editTestBtn.addEventListener('click', function() { + testConnection('editConnectionForm', 'editTestResult'); + }); + } + + var editModal = document.getElementById('editConnectionModal'); + if (editModal) { + editModal.addEventListener('show.bs.modal', function(event) { + var button = event.relatedTarget; + if (!button) return; + + var id = button.getAttribute('data-id'); + + document.getElementById('edit_name').value = button.getAttribute('data-name') || ''; + document.getElementById('edit_endpoint_url').value = button.getAttribute('data-endpoint') || ''; + document.getElementById('edit_region').value = button.getAttribute('data-region') || ''; + document.getElementById('edit_access_key').value = button.getAttribute('data-access') || ''; + document.getElementById('edit_secret_key').value = ''; + document.getElementById('edit_secret_key').placeholder = '(unchanged — leave blank to keep current)'; + document.getElementById('edit_secret_key').required = false; + document.getElementById('editTestResult').innerHTML = ''; + + var form = document.getElementById('editConnectionForm'); + form.action = endpoints.updateTemplate.replace('CONNECTION_ID', id); + }); + } + + var deleteModal = document.getElementById('deleteConnectionModal'); + if (deleteModal) { + deleteModal.addEventListener('show.bs.modal', function(event) { + var button = event.relatedTarget; + if (!button) return; + + var id = button.getAttribute('data-id'); + var name = button.getAttribute('data-name'); + + document.getElementById('deleteConnectionName').textContent = name; + var form = document.getElementById('deleteConnectionForm'); + form.action = endpoints.deleteTemplate.replace('CONNECTION_ID', id); + }); + } + + var createForm = document.getElementById('createConnectionForm'); + if (createForm) { + createForm.addEventListener('submit', function(e) { + e.preventDefault(); + window.UICore.submitFormAjax(createForm, { + successMessage: 'Connection created', + onSuccess: function(data) { + createForm.reset(); + document.getElementById('testResult').innerHTML = ''; + + if (data.connection) { + var emptyState = document.querySelector('.empty-state'); + if (emptyState) { + var cardBody = emptyState.closest('.card-body'); + if (cardBody) { + cardBody.innerHTML = '
    ' + + '' + + '' + + '' + + '' + + '' + + '
    StatusNameEndpointRegionAccess KeyActions
    '; + } + } + + var tbody = document.querySelector('table tbody'); + if (tbody) { + tbody.insertAdjacentHTML('beforeend', createConnectionRowHtml(data.connection)); + var newRow = tbody.lastElementChild; + var statusEl = newRow.querySelector('.connection-status'); + if (statusEl) { + checkConnectionHealth(data.connection.id, statusEl); + } + } + updateConnectionCount(); + } else { + location.reload(); + } + } + }); + }); + } + + var editForm = document.getElementById('editConnectionForm'); + if (editForm) { + editForm.addEventListener('submit', function(e) { + e.preventDefault(); + window.UICore.submitFormAjax(editForm, { + successMessage: 'Connection updated', + onSuccess: function(data) { + var modal = bootstrap.Modal.getInstance(document.getElementById('editConnectionModal')); + if (modal) modal.hide(); + + var connId = editForm.action.split('/').slice(-2)[0]; + var row = document.querySelector('tr[data-connection-id="' + connId + '"]'); + if (row && data.connection) { + var nameCell = row.querySelector('.fw-medium'); + if (nameCell) nameCell.textContent = data.connection.name; + + var endpointCell = row.querySelector('.text-truncate'); + if (endpointCell) { + endpointCell.textContent = data.connection.endpoint_url; + endpointCell.title = data.connection.endpoint_url; + } + + var regionBadge = row.querySelector('.badge.bg-primary'); + if (regionBadge) regionBadge.textContent = data.connection.region; + + var accessCode = row.querySelector('code.small'); + if (accessCode && data.connection.access_key) { + var ak = data.connection.access_key; + accessCode.textContent = ak.slice(0, 8) + '...' + ak.slice(-4); + } + + var editBtn = row.querySelector('[data-bs-target="#editConnectionModal"]'); + if (editBtn) { + editBtn.setAttribute('data-name', data.connection.name); + editBtn.setAttribute('data-endpoint', data.connection.endpoint_url); + editBtn.setAttribute('data-region', data.connection.region); + editBtn.setAttribute('data-access', data.connection.access_key); + } + + var deleteBtn = row.querySelector('[data-bs-target="#deleteConnectionModal"]'); + if (deleteBtn) { + deleteBtn.setAttribute('data-name', data.connection.name); + } + + var statusEl = row.querySelector('.connection-status'); + if (statusEl) { + checkConnectionHealth(connId, statusEl); + } + } + } + }); + }); + } + + var deleteForm = document.getElementById('deleteConnectionForm'); + if (deleteForm) { + deleteForm.addEventListener('submit', function(e) { + e.preventDefault(); + window.UICore.submitFormAjax(deleteForm, { + successMessage: 'Connection deleted', + onSuccess: function(data) { + var modal = bootstrap.Modal.getInstance(document.getElementById('deleteConnectionModal')); + if (modal) modal.hide(); + + var connId = deleteForm.action.split('/').slice(-2)[0]; + var row = document.querySelector('tr[data-connection-id="' + connId + '"]'); + if (row) { + row.remove(); + } + + updateConnectionCount(); + + if (document.querySelectorAll('tr[data-connection-id]').length === 0) { + location.reload(); + } + } + }); + }); + } + } + + return { + init: init, + togglePassword: togglePassword, + testConnection: testConnection, + checkConnectionHealth: checkConnectionHealth + }; +})(); diff --git a/rust/myfsio-engine/crates/myfsio-server/static/js/iam-management.js b/rust/myfsio-engine/crates/myfsio-server/static/js/iam-management.js new file mode 100644 index 0000000..ce9b8f3 --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/js/iam-management.js @@ -0,0 +1,846 @@ +window.IAMManagement = (function() { + 'use strict'; + + var users = []; + var currentUserKey = null; + var endpoints = {}; + var csrfToken = ''; + var iamLocked = false; + + var policyModal = null; + var editUserModal = null; + var deleteUserModal = null; + var rotateSecretModal = null; + var expiryModal = null; + var currentRotateKey = null; + var currentEditKey = null; + var currentDeleteKey = null; + var currentEditAccessKey = null; + var currentDeleteAccessKey = null; + var currentExpiryKey = null; + var currentExpiryAccessKey = null; + + var ALL_S3_ACTIONS = [ + 'list', 'read', 'write', 'delete', 'share', 'policy', + 'replication', 'lifecycle', 'cors', + 'create_bucket', 'delete_bucket', + 'versioning', 'tagging', 'encryption', 'quota', + 'object_lock', 'notification', 'logging', 'website' + ]; + + var policyTemplates = { + full: [{ bucket: '*', actions: ['list', 'read', 'write', 'delete', 'share', 'policy', 'create_bucket', 'delete_bucket', 'replication', 'lifecycle', 'cors', 'versioning', 'tagging', 'encryption', 'quota', 'object_lock', 'notification', 'logging', 'website', 'iam:*'] }], + readonly: [{ bucket: '*', actions: ['list', 'read'] }], + writer: [{ bucket: '*', actions: ['list', 'read', 'write'] }], + operator: [{ bucket: '*', actions: ['list', 'read', 'write', 'delete', 'create_bucket', 'delete_bucket'] }], + bucketadmin: [{ bucket: '*', actions: ['list', 'read', 'write', 'delete', 'share', 'policy', 'create_bucket', 'delete_bucket', 'versioning', 'tagging', 'encryption', 'cors', 'lifecycle', 'quota', 'object_lock', 'notification', 'logging', 'website', 'replication'] }] + }; + + function isAdminUser(policies) { + if (!policies || !policies.length) return false; + return policies.some(function(p) { + return p.actions && (p.actions.indexOf('iam:*') >= 0 || p.actions.indexOf('*') >= 0); + }); + } + + function getPermissionLevel(actions) { + if (!actions || !actions.length) return 'Custom (0)'; + if (actions.indexOf('*') >= 0) return 'Full Access'; + if (actions.length >= ALL_S3_ACTIONS.length) { + var hasAll = ALL_S3_ACTIONS.every(function(a) { return actions.indexOf(a) >= 0; }); + if (hasAll) return 'Full Access'; + } + var has = function(a) { return actions.indexOf(a) >= 0; }; + if (has('list') && has('read') && has('write') && has('delete')) return 'Read + Write + Delete'; + if (has('list') && has('read') && has('write')) return 'Read + Write'; + if (has('list') && has('read')) return 'Read Only'; + return 'Custom (' + actions.length + ')'; + } + + function getBucketLabel(bucket) { + return bucket === '*' ? 'All Buckets' : bucket; + } + + function buildUserUrl(template, userId) { + return template.replace('USER_ID', encodeURIComponent(userId)); + } + + function getUserByIdentifier(identifier) { + return users.find(function(u) { + return u.user_id === identifier || u.access_key === identifier; + }) || null; + } + + function getUserById(userId) { + return users.find(function(u) { return u.user_id === userId; }) || null; + } + + function init(config) { + users = config.users || []; + currentUserKey = config.currentUserKey || null; + endpoints = config.endpoints || {}; + csrfToken = config.csrfToken || ''; + iamLocked = config.iamLocked || false; + + if (iamLocked) return; + + initModals(); + setupJsonAutoIndent(); + setupCopyButtons(); + setupPolicyEditor(); + setupCreateUserModal(); + setupEditUserModal(); + setupDeleteUserModal(); + setupRotateSecretModal(); + setupExpiryModal(); + setupFormHandlers(); + setupSearch(); + setupCopyAccessKeyButtons(); + } + + function initModals() { + var policyModalEl = document.getElementById('policyEditorModal'); + var editModalEl = document.getElementById('editUserModal'); + var deleteModalEl = document.getElementById('deleteUserModal'); + var rotateModalEl = document.getElementById('rotateSecretModal'); + var expiryModalEl = document.getElementById('expiryModal'); + + if (policyModalEl) policyModal = new bootstrap.Modal(policyModalEl); + if (editModalEl) editUserModal = new bootstrap.Modal(editModalEl); + if (deleteModalEl) deleteUserModal = new bootstrap.Modal(deleteModalEl); + if (rotateModalEl) rotateSecretModal = new bootstrap.Modal(rotateModalEl); + if (expiryModalEl) expiryModal = new bootstrap.Modal(expiryModalEl); + } + + function setupJsonAutoIndent() { + window.UICore.setupJsonAutoIndent(document.getElementById('policyEditorDocument')); + window.UICore.setupJsonAutoIndent(document.getElementById('createUserPolicies')); + } + + function setupCopyButtons() { + document.querySelectorAll('.config-copy').forEach(function(button) { + button.addEventListener('click', async function() { + var targetId = button.dataset.copyTarget; + var target = document.getElementById(targetId); + if (!target) return; + await window.UICore.copyToClipboard(target.innerText, button, 'Copy JSON'); + }); + }); + + var accessKeyCopyButton = document.querySelector('[data-access-key-copy]'); + if (accessKeyCopyButton) { + accessKeyCopyButton.addEventListener('click', async function() { + var accessKeyInput = document.getElementById('disclosedAccessKeyValue'); + if (!accessKeyInput) return; + await window.UICore.copyToClipboard(accessKeyInput.value, accessKeyCopyButton, 'Copy'); + }); + } + + var secretCopyButton = document.querySelector('[data-secret-copy]'); + if (secretCopyButton) { + secretCopyButton.addEventListener('click', async function() { + var secretInput = document.getElementById('disclosedSecretValue'); + if (!secretInput) return; + await window.UICore.copyToClipboard(secretInput.value, secretCopyButton, 'Copy'); + }); + } + } + + function getUserPolicies(identifier) { + var user = getUserByIdentifier(identifier); + return user ? JSON.stringify(user.policies, null, 2) : ''; + } + + function applyPolicyTemplate(name, textareaEl) { + if (policyTemplates[name] && textareaEl) { + textareaEl.value = JSON.stringify(policyTemplates[name], null, 2); + } + } + + function setupPolicyEditor() { + var userLabelEl = document.getElementById('policyEditorUserLabel'); + var userInputEl = document.getElementById('policyEditorUserId'); + var textareaEl = document.getElementById('policyEditorDocument'); + + document.querySelectorAll('[data-policy-template]').forEach(function(button) { + button.addEventListener('click', function() { + applyPolicyTemplate(button.dataset.policyTemplate, textareaEl); + }); + }); + + document.querySelectorAll('[data-policy-editor]').forEach(function(button) { + button.addEventListener('click', function() { + var userId = button.dataset.userId; + var accessKey = button.dataset.accessKey || userId; + if (!userId) return; + + userLabelEl.textContent = accessKey; + userInputEl.value = userId; + textareaEl.value = getUserPolicies(userId); + + policyModal.show(); + }); + }); + } + + function generateSecureHex(byteCount) { + var arr = new Uint8Array(byteCount); + crypto.getRandomValues(arr); + return Array.from(arr).map(function(b) { return b.toString(16).padStart(2, '0'); }).join(''); + } + + function generateSecureBase64(byteCount) { + var arr = new Uint8Array(byteCount); + crypto.getRandomValues(arr); + var binary = ''; + for (var i = 0; i < arr.length; i++) { + binary += String.fromCharCode(arr[i]); + } + return btoa(binary).replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, ''); + } + + function setupCreateUserModal() { + var createUserPoliciesEl = document.getElementById('createUserPolicies'); + + document.querySelectorAll('[data-create-policy-template]').forEach(function(button) { + button.addEventListener('click', function() { + applyPolicyTemplate(button.dataset.createPolicyTemplate, createUserPoliciesEl); + }); + }); + + var genAccessKeyBtn = document.getElementById('generateAccessKeyBtn'); + if (genAccessKeyBtn) { + genAccessKeyBtn.addEventListener('click', function() { + var input = document.getElementById('createUserAccessKey'); + if (input) input.value = generateSecureHex(8); + }); + } + + var genSecretKeyBtn = document.getElementById('generateSecretKeyBtn'); + if (genSecretKeyBtn) { + genSecretKeyBtn.addEventListener('click', function() { + var input = document.getElementById('createUserSecretKey'); + if (input) input.value = generateSecureBase64(24); + }); + } + } + + function setupEditUserModal() { + var editUserForm = document.getElementById('editUserForm'); + var editUserDisplayName = document.getElementById('editUserDisplayName'); + + document.querySelectorAll('[data-edit-user]').forEach(function(btn) { + btn.addEventListener('click', function() { + var key = btn.dataset.userId; + var accessKey = btn.dataset.accessKey || key; + var name = btn.dataset.displayName; + currentEditKey = key; + currentEditAccessKey = accessKey; + editUserDisplayName.value = name; + editUserForm.action = buildUserUrl(endpoints.updateUser, key); + editUserModal.show(); + }); + }); + } + + function setupDeleteUserModal() { + var deleteUserForm = document.getElementById('deleteUserForm'); + var deleteUserLabel = document.getElementById('deleteUserLabel'); + var deleteSelfWarning = document.getElementById('deleteSelfWarning'); + + document.querySelectorAll('[data-delete-user]').forEach(function(btn) { + btn.addEventListener('click', function() { + var key = btn.dataset.userId; + var accessKey = btn.dataset.accessKey || key; + currentDeleteKey = key; + currentDeleteAccessKey = accessKey; + deleteUserLabel.textContent = accessKey; + deleteUserForm.action = buildUserUrl(endpoints.deleteUser, key); + + if (accessKey === currentUserKey) { + deleteSelfWarning.classList.remove('d-none'); + } else { + deleteSelfWarning.classList.add('d-none'); + } + + deleteUserModal.show(); + }); + }); + } + + function setupRotateSecretModal() { + var rotateUserLabel = document.getElementById('rotateUserLabel'); + var confirmRotateBtn = document.getElementById('confirmRotateBtn'); + var rotateCancelBtn = document.getElementById('rotateCancelBtn'); + var rotateDoneBtn = document.getElementById('rotateDoneBtn'); + var rotateSecretConfirm = document.getElementById('rotateSecretConfirm'); + var rotateSecretResult = document.getElementById('rotateSecretResult'); + var newSecretKeyInput = document.getElementById('newSecretKey'); + var copyNewSecretBtn = document.getElementById('copyNewSecret'); + + document.querySelectorAll('[data-rotate-user]').forEach(function(btn) { + btn.addEventListener('click', function() { + currentRotateKey = btn.dataset.userId; + rotateUserLabel.textContent = btn.dataset.accessKey || currentRotateKey; + + rotateSecretConfirm.classList.remove('d-none'); + rotateSecretResult.classList.add('d-none'); + confirmRotateBtn.classList.remove('d-none'); + rotateCancelBtn.classList.remove('d-none'); + rotateDoneBtn.classList.add('d-none'); + + rotateSecretModal.show(); + }); + }); + + if (confirmRotateBtn) { + confirmRotateBtn.addEventListener('click', async function() { + if (!currentRotateKey) return; + + window.UICore.setButtonLoading(confirmRotateBtn, true, 'Rotating...'); + + try { + var url = buildUserUrl(endpoints.rotateSecret, currentRotateKey); + var response = await fetch(url, { + method: 'POST', + headers: { + 'Accept': 'application/json', + 'X-CSRFToken': csrfToken + } + }); + + if (!response.ok) { + var data = await response.json(); + throw new Error(data.error || 'Failed to rotate secret'); + } + + var data = await response.json(); + newSecretKeyInput.value = data.secret_key; + + rotateSecretConfirm.classList.add('d-none'); + rotateSecretResult.classList.remove('d-none'); + confirmRotateBtn.classList.add('d-none'); + rotateCancelBtn.classList.add('d-none'); + rotateDoneBtn.classList.remove('d-none'); + + } catch (err) { + if (window.showToast) { + window.showToast(err.message, 'Error', 'danger'); + } + rotateSecretModal.hide(); + } finally { + window.UICore.setButtonLoading(confirmRotateBtn, false); + } + }); + } + + if (copyNewSecretBtn) { + copyNewSecretBtn.addEventListener('click', async function() { + await window.UICore.copyToClipboard(newSecretKeyInput.value, copyNewSecretBtn, 'Copy'); + }); + } + + if (rotateDoneBtn) { + rotateDoneBtn.addEventListener('click', function() { + window.location.reload(); + }); + } + } + + function openExpiryModal(key, expiresAt) { + currentExpiryKey = key; + var user = getUserByIdentifier(key); + var label = document.getElementById('expiryUserLabel'); + var input = document.getElementById('expiryDateInput'); + var form = document.getElementById('expiryForm'); + if (label) label.textContent = currentExpiryAccessKey || (user ? user.access_key : key); + if (expiresAt) { + try { + var dt = new Date(expiresAt); + var local = new Date(dt.getTime() - dt.getTimezoneOffset() * 60000); + if (input) input.value = local.toISOString().slice(0, 16); + } catch(e) { + if (input) input.value = ''; + } + } else { + if (input) input.value = ''; + } + if (form) form.action = buildUserUrl(endpoints.updateExpiry, key); + var modalEl = document.getElementById('expiryModal'); + if (modalEl) { + var modal = bootstrap.Modal.getOrCreateInstance(modalEl); + modal.show(); + } + } + + function setupExpiryModal() { + document.querySelectorAll('[data-expiry-user]').forEach(function(btn) { + btn.addEventListener('click', function(e) { + e.preventDefault(); + currentExpiryAccessKey = btn.dataset.accessKey || btn.dataset.userId; + openExpiryModal(btn.dataset.userId, btn.dataset.expiresAt || ''); + }); + }); + + document.querySelectorAll('[data-expiry-preset]').forEach(function(btn) { + btn.addEventListener('click', function() { + var preset = btn.dataset.expiryPreset; + var input = document.getElementById('expiryDateInput'); + if (!input) return; + if (preset === 'clear') { + input.value = ''; + return; + } + var now = new Date(); + var ms = 0; + if (preset === '1h') ms = 3600000; + else if (preset === '24h') ms = 86400000; + else if (preset === '7d') ms = 7 * 86400000; + else if (preset === '30d') ms = 30 * 86400000; + else if (preset === '90d') ms = 90 * 86400000; + var future = new Date(now.getTime() + ms); + var local = new Date(future.getTime() - future.getTimezoneOffset() * 60000); + input.value = local.toISOString().slice(0, 16); + }); + }); + + var expiryForm = document.getElementById('expiryForm'); + if (expiryForm) { + expiryForm.addEventListener('submit', function(e) { + e.preventDefault(); + window.UICore.submitFormAjax(expiryForm, { + successMessage: 'Expiry updated', + onSuccess: function() { + var modalEl = document.getElementById('expiryModal'); + if (modalEl) bootstrap.Modal.getOrCreateInstance(modalEl).hide(); + window.location.reload(); + } + }); + }); + } + } + + function createUserCardHtml(user) { + var userId = user.user_id || ''; + var accessKey = user.access_key || userId; + var displayName = user.display_name || accessKey; + var policies = user.policies || []; + var expiresAt = user.expires_at || ''; + var admin = isAdminUser(policies); + var cardClass = 'card h-100 iam-user-card' + (admin ? ' iam-admin-card' : ''); + var roleBadge = admin + ? 'Admin' + : 'User'; + + var policyBadges = ''; + if (policies && policies.length > 0) { + policyBadges = policies.map(function(p) { + var bucketLabel = getBucketLabel(p.bucket); + var permLevel = getPermissionLevel(p.actions); + return '' + + '' + + '' + + '' + window.UICore.escapeHtml(bucketLabel) + ' · ' + window.UICore.escapeHtml(permLevel) + ''; + }).join(''); + } else { + policyBadges = 'No policies'; + } + + var esc = window.UICore.escapeHtml; + return '
    ' + + '
    ' + + '
    ' + + '
    ' + + '
    ' + + '
    ' + + '' + + '' + + '
    ' + + '
    ' + + '
    ' + + '
    ' + esc(displayName) + '
    ' + + roleBadge + + '
    ' + + '
    ' + + '' + esc(accessKey) + '' + + '' + + '
    ' + + '
    ' + + '
    ' + + '' + + '
      ' + + '
      • ' + + '
      • ' + + '
      • ' + + '
      • ' + + '
      • ' + + '
    ' + + '
    ' + + '
    Bucket Permissions
    ' + + '
    ' + policyBadges + '
    ' + + '' + + '
    '; + } + + function attachUserCardHandlers(cardElement, user) { + var userId = user.user_id; + var accessKey = user.access_key; + var displayName = user.display_name; + var expiresAt = user.expires_at || ''; + var editBtn = cardElement.querySelector('[data-edit-user]'); + if (editBtn) { + editBtn.addEventListener('click', function() { + currentEditKey = userId; + currentEditAccessKey = accessKey; + document.getElementById('editUserDisplayName').value = displayName; + document.getElementById('editUserForm').action = buildUserUrl(endpoints.updateUser, userId); + editUserModal.show(); + }); + } + + var deleteBtn = cardElement.querySelector('[data-delete-user]'); + if (deleteBtn) { + deleteBtn.addEventListener('click', function() { + currentDeleteKey = userId; + currentDeleteAccessKey = accessKey; + document.getElementById('deleteUserLabel').textContent = accessKey; + document.getElementById('deleteUserForm').action = buildUserUrl(endpoints.deleteUser, userId); + var deleteSelfWarning = document.getElementById('deleteSelfWarning'); + if (accessKey === currentUserKey) { + deleteSelfWarning.classList.remove('d-none'); + } else { + deleteSelfWarning.classList.add('d-none'); + } + deleteUserModal.show(); + }); + } + + var rotateBtn = cardElement.querySelector('[data-rotate-user]'); + if (rotateBtn) { + rotateBtn.addEventListener('click', function() { + currentRotateKey = userId; + document.getElementById('rotateUserLabel').textContent = accessKey; + document.getElementById('rotateSecretConfirm').classList.remove('d-none'); + document.getElementById('rotateSecretResult').classList.add('d-none'); + document.getElementById('confirmRotateBtn').classList.remove('d-none'); + document.getElementById('rotateCancelBtn').classList.remove('d-none'); + document.getElementById('rotateDoneBtn').classList.add('d-none'); + rotateSecretModal.show(); + }); + } + + var expiryBtn = cardElement.querySelector('[data-expiry-user]'); + if (expiryBtn) { + expiryBtn.addEventListener('click', function(e) { + e.preventDefault(); + currentExpiryAccessKey = accessKey; + openExpiryModal(userId, expiresAt); + }); + } + + var policyBtn = cardElement.querySelector('[data-policy-editor]'); + if (policyBtn) { + policyBtn.addEventListener('click', function() { + document.getElementById('policyEditorUserLabel').textContent = accessKey; + document.getElementById('policyEditorUserId').value = userId; + document.getElementById('policyEditorDocument').value = getUserPolicies(userId); + policyModal.show(); + }); + } + + var copyBtn = cardElement.querySelector('[data-copy-access-key]'); + if (copyBtn) { + copyBtn.addEventListener('click', function() { + copyAccessKey(copyBtn); + }); + } + } + + function updateUserCount() { + var countEl = document.querySelector('.card-header .text-muted.small'); + if (countEl) { + var count = document.querySelectorAll('.iam-user-card').length; + countEl.textContent = count + ' user' + (count !== 1 ? 's' : '') + ' configured'; + } + } + + function setupFormHandlers() { + var createUserForm = document.querySelector('#createUserModal form'); + if (createUserForm) { + createUserForm.addEventListener('submit', function(e) { + e.preventDefault(); + window.UICore.submitFormAjax(createUserForm, { + successMessage: 'User created', + onSuccess: function(data) { + var modal = bootstrap.Modal.getInstance(document.getElementById('createUserModal')); + if (modal) modal.hide(); + createUserForm.reset(); + + var existingAlert = document.querySelector('.alert.alert-info.border-0.shadow-sm'); + if (existingAlert) existingAlert.remove(); + + if (data.secret_key) { + var alertHtml = ''; + var container = document.querySelector('.page-header'); + if (container) { + container.insertAdjacentHTML('afterend', alertHtml); + document.getElementById('copyNewUserAccessKey').addEventListener('click', async function() { + await window.UICore.copyToClipboard(data.access_key, this, 'Copy'); + }); + document.getElementById('copyNewUserSecret').addEventListener('click', async function() { + await window.UICore.copyToClipboard(data.secret_key, this, 'Copy'); + }); + } + } + + var usersGrid = document.querySelector('.row.g-3'); + var emptyState = document.querySelector('.empty-state'); + if (emptyState) { + var emptyCol = emptyState.closest('.col-12'); + if (emptyCol) emptyCol.remove(); + if (!usersGrid) { + var cardBody = document.querySelector('.card-body.px-4.pb-4'); + if (cardBody) { + cardBody.innerHTML = '
    '; + usersGrid = cardBody.querySelector('.row.g-3'); + } + } + } + + if (usersGrid) { + var newUser = { + user_id: data.user_id, + access_key: data.access_key, + display_name: data.display_name, + expires_at: data.expires_at || '', + policies: data.policies || [] + }; + var cardHtml = createUserCardHtml(newUser); + usersGrid.insertAdjacentHTML('beforeend', cardHtml); + var newCard = usersGrid.lastElementChild; + attachUserCardHandlers(newCard, newUser); + users.push(newUser); + updateUserCount(); + } + } + }); + }); + } + + var policyEditorForm = document.getElementById('policyEditorForm'); + if (policyEditorForm) { + policyEditorForm.addEventListener('submit', function(e) { + e.preventDefault(); + var userInputEl = document.getElementById('policyEditorUserId'); + var userId = userInputEl.value; + if (!userId) return; + + var template = policyEditorForm.dataset.actionTemplate; + policyEditorForm.action = template.replace('USER_ID_PLACEHOLDER', encodeURIComponent(userId)); + + window.UICore.submitFormAjax(policyEditorForm, { + successMessage: 'Policies updated', + onSuccess: function(data) { + policyModal.hide(); + + var userCard = document.querySelector('.iam-user-item[data-user-id="' + userId + '"]'); + if (userCard) { + var cardEl = userCard.querySelector('.iam-user-card'); + var badgeContainer = cardEl ? cardEl.querySelector('[data-policy-badges]') : null; + if (badgeContainer && data.policies) { + var badges = data.policies.map(function(p) { + var bl = getBucketLabel(p.bucket); + var pl = getPermissionLevel(p.actions); + return '' + + '' + + '' + + '' + window.UICore.escapeHtml(bl) + ' · ' + window.UICore.escapeHtml(pl) + ''; + }).join(''); + badgeContainer.innerHTML = badges || 'No policies'; + } + if (cardEl) { + var nowAdmin = isAdminUser(data.policies); + cardEl.classList.toggle('iam-admin-card', nowAdmin); + var roleBadgeEl = cardEl.querySelector('[data-role-badge]'); + if (roleBadgeEl) { + if (nowAdmin) { + roleBadgeEl.className = 'iam-role-badge iam-role-admin'; + roleBadgeEl.textContent = 'Admin'; + } else { + roleBadgeEl.className = 'iam-role-badge iam-role-user'; + roleBadgeEl.textContent = 'User'; + } + } + } + } + + var userIndex = users.findIndex(function(u) { return u.user_id === userId; }); + if (userIndex >= 0 && data.policies) { + users[userIndex].policies = data.policies; + } + } + }); + }); + } + + var editUserForm = document.getElementById('editUserForm'); + if (editUserForm) { + editUserForm.addEventListener('submit', function(e) { + e.preventDefault(); + var key = currentEditKey; + window.UICore.submitFormAjax(editUserForm, { + successMessage: 'User updated', + onSuccess: function(data) { + editUserModal.hide(); + + var newName = data.display_name || document.getElementById('editUserDisplayName').value; + var editBtn = document.querySelector('[data-edit-user][data-user-id="' + key + '"]'); + if (editBtn) { + editBtn.setAttribute('data-display-name', newName); + var card = editBtn.closest('.iam-user-card'); + if (card) { + var nameEl = card.querySelector('h6'); + if (nameEl) { + nameEl.textContent = newName; + nameEl.title = newName; + } + var itemWrapper = card.closest('.iam-user-item'); + if (itemWrapper) { + itemWrapper.setAttribute('data-display-name', newName.toLowerCase()); + } + } + } + + var userIndex = users.findIndex(function(u) { return u.user_id === key; }); + if (userIndex >= 0) { + users[userIndex].display_name = newName; + } + + if (currentEditAccessKey === currentUserKey) { + document.querySelectorAll('.sidebar-user .user-name').forEach(function(el) { + var truncated = newName.length > 16 ? newName.substring(0, 16) + '...' : newName; + el.textContent = truncated; + el.title = newName; + }); + document.querySelectorAll('.sidebar-user[data-username]').forEach(function(el) { + el.setAttribute('data-username', newName); + }); + } + } + }); + }); + } + + var deleteUserForm = document.getElementById('deleteUserForm'); + if (deleteUserForm) { + deleteUserForm.addEventListener('submit', function(e) { + e.preventDefault(); + var key = currentDeleteKey; + window.UICore.submitFormAjax(deleteUserForm, { + successMessage: 'User deleted', + onSuccess: function(data) { + deleteUserModal.hide(); + + if (currentDeleteAccessKey === currentUserKey) { + window.location.href = '/ui/'; + return; + } + + var deleteBtn = document.querySelector('[data-delete-user][data-user-id="' + key + '"]'); + if (deleteBtn) { + var cardCol = deleteBtn.closest('[class*="col-"]'); + if (cardCol) { + cardCol.remove(); + } + } + + users = users.filter(function(u) { return u.user_id !== key; }); + updateUserCount(); + } + }); + }); + } + } + + function setupSearch() { + var searchInput = document.getElementById('iam-user-search'); + if (!searchInput) return; + + searchInput.addEventListener('input', function() { + var query = searchInput.value.toLowerCase().trim(); + var items = document.querySelectorAll('.iam-user-item'); + var noResults = document.getElementById('iam-no-results'); + var visibleCount = 0; + + items.forEach(function(item) { + var name = item.getAttribute('data-display-name') || ''; + var key = item.getAttribute('data-access-key-filter') || ''; + var matches = !query || name.indexOf(query) >= 0 || key.indexOf(query) >= 0; + item.classList.toggle('d-none', !matches); + if (matches) visibleCount++; + }); + + if (noResults) { + noResults.classList.toggle('d-none', visibleCount > 0); + } + }); + } + + function copyAccessKey(btn) { + var key = btn.getAttribute('data-copy-access-key'); + if (!key) return; + var originalHtml = btn.innerHTML; + navigator.clipboard.writeText(key).then(function() { + btn.innerHTML = ''; + btn.style.color = '#22c55e'; + setTimeout(function() { + btn.innerHTML = originalHtml; + btn.style.color = ''; + }, 1200); + }).catch(function() {}); + } + + function setupCopyAccessKeyButtons() { + document.querySelectorAll('[data-copy-access-key]').forEach(function(btn) { + btn.addEventListener('click', function() { + copyAccessKey(btn); + }); + }); + } + + return { + init: init + }; +})(); diff --git a/rust/myfsio-engine/crates/myfsio-server/static/js/ui-core.js b/rust/myfsio-engine/crates/myfsio-server/static/js/ui-core.js new file mode 100644 index 0000000..dde8f10 --- /dev/null +++ b/rust/myfsio-engine/crates/myfsio-server/static/js/ui-core.js @@ -0,0 +1,334 @@ +window.UICore = (function() { + 'use strict'; + + function getCsrfToken() { + const meta = document.querySelector('meta[name="csrf-token"]'); + return meta ? meta.getAttribute('content') : ''; + } + + function formatBytes(bytes) { + if (!Number.isFinite(bytes)) return bytes + ' bytes'; + const units = ['bytes', 'KB', 'MB', 'GB', 'TB']; + let i = 0; + let size = bytes; + while (size >= 1024 && i < units.length - 1) { + size /= 1024; + i++; + } + return size.toFixed(i === 0 ? 0 : 1) + ' ' + units[i]; + } + + function escapeHtml(value) { + if (value === null || value === undefined) return ''; + return String(value) + .replace(/&/g, '&') + .replace(//g, '>') + .replace(/"/g, '"') + .replace(/'/g, '''); + } + + async function submitFormAjax(form, options) { + options = options || {}; + var onSuccess = options.onSuccess || function() {}; + var onError = options.onError || function() {}; + var successMessage = options.successMessage || 'Operation completed'; + + var formData = new FormData(form); + var hasFileInput = !!form.querySelector('input[type="file"]'); + var requestBody = hasFileInput ? formData : new URLSearchParams(formData); + var csrfToken = getCsrfToken(); + var submitBtn = form.querySelector('[type="submit"]'); + var originalHtml = submitBtn ? submitBtn.innerHTML : ''; + + try { + if (submitBtn) { + submitBtn.disabled = true; + submitBtn.innerHTML = 'Saving...'; + } + + var formAction = form.getAttribute('action') || form.action; + var headers = { + 'X-CSRF-Token': csrfToken, + 'Accept': 'application/json', + 'X-Requested-With': 'XMLHttpRequest' + }; + if (!hasFileInput) { + headers['Content-Type'] = 'application/x-www-form-urlencoded;charset=UTF-8'; + } + var response = await fetch(formAction, { + method: form.getAttribute('method') || 'POST', + headers: headers, + body: requestBody, + redirect: 'follow' + }); + + var contentType = response.headers.get('content-type') || ''; + if (!contentType.includes('application/json')) { + throw new Error('Server returned an unexpected response. Please try again.'); + } + + var data = await response.json(); + + if (!response.ok) { + throw new Error(data.error || 'HTTP ' + response.status); + } + + window.showToast(data.message || successMessage, 'Success', 'success'); + onSuccess(data); + + } catch (err) { + window.showToast(err.message, 'Error', 'error'); + onError(err); + } finally { + if (submitBtn) { + submitBtn.disabled = false; + submitBtn.innerHTML = originalHtml; + } + } + } + + function PollingManager() { + this.intervals = {}; + this.callbacks = {}; + this.timers = {}; + this.defaults = { + replication: 30000, + lifecycle: 60000, + connectionHealth: 60000, + bucketStats: 120000 + }; + this._loadSettings(); + } + + PollingManager.prototype._loadSettings = function() { + try { + var stored = localStorage.getItem('myfsio-polling-intervals'); + if (stored) { + var settings = JSON.parse(stored); + for (var key in settings) { + if (settings.hasOwnProperty(key)) { + this.defaults[key] = settings[key]; + } + } + } + } catch (e) { + console.warn('Failed to load polling settings:', e); + } + }; + + PollingManager.prototype.saveSettings = function(settings) { + try { + for (var key in settings) { + if (settings.hasOwnProperty(key)) { + this.defaults[key] = settings[key]; + } + } + localStorage.setItem('myfsio-polling-intervals', JSON.stringify(this.defaults)); + } catch (e) { + console.warn('Failed to save polling settings:', e); + } + }; + + PollingManager.prototype.start = function(key, callback, interval) { + this.stop(key); + var ms = interval !== undefined ? interval : (this.defaults[key] || 30000); + if (ms <= 0) return; + + this.callbacks[key] = callback; + this.intervals[key] = ms; + + callback(); + + var self = this; + this.timers[key] = setInterval(function() { + if (!document.hidden) { + callback(); + } + }, ms); + }; + + PollingManager.prototype.stop = function(key) { + if (this.timers[key]) { + clearInterval(this.timers[key]); + delete this.timers[key]; + } + }; + + PollingManager.prototype.stopAll = function() { + for (var key in this.timers) { + if (this.timers.hasOwnProperty(key)) { + clearInterval(this.timers[key]); + } + } + this.timers = {}; + }; + + PollingManager.prototype.updateInterval = function(key, newInterval) { + var callback = this.callbacks[key]; + this.defaults[key] = newInterval; + this.saveSettings(this.defaults); + if (callback) { + this.start(key, callback, newInterval); + } + }; + + PollingManager.prototype.getSettings = function() { + var result = {}; + for (var key in this.defaults) { + if (this.defaults.hasOwnProperty(key)) { + result[key] = this.defaults[key]; + } + } + return result; + }; + + var pollingManager = new PollingManager(); + + document.addEventListener('visibilitychange', function() { + if (document.hidden) { + pollingManager.stopAll(); + } else { + for (var key in pollingManager.callbacks) { + if (pollingManager.callbacks.hasOwnProperty(key)) { + pollingManager.start(key, pollingManager.callbacks[key], pollingManager.intervals[key]); + } + } + } + }); + + window.addEventListener('beforeunload', function() { + pollingManager.stopAll(); + }); + + return { + getCsrfToken: getCsrfToken, + formatBytes: formatBytes, + escapeHtml: escapeHtml, + submitFormAjax: submitFormAjax, + PollingManager: PollingManager, + pollingManager: pollingManager + }; +})(); + +window.pollingManager = window.UICore.pollingManager; + +window.UICore.copyToClipboard = async function(text, button, originalText) { + try { + await navigator.clipboard.writeText(text); + if (button) { + var prevText = button.textContent; + button.textContent = 'Copied!'; + setTimeout(function() { + button.textContent = originalText || prevText; + }, 1500); + } + return true; + } catch (err) { + console.error('Copy failed:', err); + return false; + } +}; + +window.UICore.setButtonLoading = function(button, isLoading, loadingText) { + if (!button) return; + if (isLoading) { + button._originalHtml = button.innerHTML; + button._originalDisabled = button.disabled; + button.disabled = true; + button.innerHTML = '' + (loadingText || 'Loading...'); + } else { + button.disabled = button._originalDisabled || false; + button.innerHTML = button._originalHtml || button.innerHTML; + } +}; + +window.UICore.updateBadgeCount = function(selector, count, singular, plural) { + var badge = document.querySelector(selector); + if (badge) { + var label = count === 1 ? (singular || '') : (plural || 's'); + badge.textContent = count + ' ' + label; + } +}; + +window.UICore.setupJsonAutoIndent = function(textarea) { + if (!textarea) return; + + textarea.addEventListener('keydown', function(e) { + if (e.key === 'Enter') { + e.preventDefault(); + + var start = this.selectionStart; + var end = this.selectionEnd; + var value = this.value; + + var lineStart = value.lastIndexOf('\n', start - 1) + 1; + var currentLine = value.substring(lineStart, start); + + var indentMatch = currentLine.match(/^(\s*)/); + var indent = indentMatch ? indentMatch[1] : ''; + + var trimmedLine = currentLine.trim(); + var lastChar = trimmedLine.slice(-1); + + var newIndent = indent; + var insertAfter = ''; + + if (lastChar === '{' || lastChar === '[') { + newIndent = indent + ' '; + + var charAfterCursor = value.substring(start, start + 1).trim(); + if ((lastChar === '{' && charAfterCursor === '}') || + (lastChar === '[' && charAfterCursor === ']')) { + insertAfter = '\n' + indent; + } + } else if (lastChar === ',' || lastChar === ':') { + newIndent = indent; + } + + var insertion = '\n' + newIndent + insertAfter; + var newValue = value.substring(0, start) + insertion + value.substring(end); + + this.value = newValue; + + var newCursorPos = start + 1 + newIndent.length; + this.selectionStart = this.selectionEnd = newCursorPos; + + this.dispatchEvent(new Event('input', { bubbles: true })); + } + + if (e.key === 'Tab') { + e.preventDefault(); + var start = this.selectionStart; + var end = this.selectionEnd; + + if (e.shiftKey) { + var lineStart = this.value.lastIndexOf('\n', start - 1) + 1; + var lineContent = this.value.substring(lineStart, start); + if (lineContent.startsWith(' ')) { + this.value = this.value.substring(0, lineStart) + + this.value.substring(lineStart + 2); + this.selectionStart = this.selectionEnd = Math.max(lineStart, start - 2); + } + } else { + this.value = this.value.substring(0, start) + ' ' + this.value.substring(end); + this.selectionStart = this.selectionEnd = start + 2; + } + + this.dispatchEvent(new Event('input', { bubbles: true })); + } + }); +}; + +document.addEventListener('DOMContentLoaded', function() { + var flashMessage = sessionStorage.getItem('flashMessage'); + if (flashMessage) { + sessionStorage.removeItem('flashMessage'); + try { + var msg = JSON.parse(flashMessage); + if (window.showToast) { + window.showToast(msg.body || msg.title, msg.title, msg.variant || 'info'); + } + } catch (e) {} + } +}); diff --git a/rust/myfsio-engine/crates/myfsio-server/templates/base.html b/rust/myfsio-engine/crates/myfsio-server/templates/base.html index 2701a17..e5b9564 100644 --- a/rust/myfsio-engine/crates/myfsio-server/templates/base.html +++ b/rust/myfsio-engine/crates/myfsio-server/templates/base.html @@ -3,7 +3,7 @@ - {% if principal %}{% endif %} + {% if principal %}{% endif %} MyFSIO Console @@ -145,7 +145,7 @@
    - +
    - +
    @@ -655,7 +649,7 @@ {% else %}
    - +
    @@ -817,7 +811,7 @@ {% for key in kms_keys %} {% endfor %} @@ -832,14 +826,14 @@ Save Encryption Settings - {% if enc_algorithm %} - - {% endif %}
    {% else %} @@ -864,7 +858,6 @@
    {% set max_bytes = bucket_quota.max_bytes %} {% set max_objects = bucket_quota.max_objects %} - {% set has_quota = max_bytes != null or max_objects != null %} {% set current_objects = bucket_stats.objects | default(value=0) %} {% set version_count = bucket_stats.version_count | default(value=0) %} {% set total_objects = bucket_stats.total_objects | default(value=current_objects) %} @@ -879,10 +872,10 @@
    {{ total_objects }}
    Total Objects
    - {% if max_objects != null %} + {% if has_max_objects %}
    {% if max_objects > 0 %}{% set obj_pct = total_objects / max_objects * 100 | int %}{% else %}{% set obj_pct = 0 %}{% endif %} -
    +
    {{ obj_pct }}% of {{ max_objects }} limit
    {% else %} @@ -899,10 +892,10 @@
    {{ total_bytes | filesizeformat }}
    Total Storage
    - {% if max_bytes != null %} + {% if has_max_bytes %}
    {% if max_bytes > 0 %}{% set bytes_pct = total_bytes / max_bytes * 100 | int %}{% else %}{% set bytes_pct = 0 %}{% endif %} -
    +
    {{ bytes_pct }}% of {{ max_bytes | filesizeformat }} limit
    {% else %} @@ -924,14 +917,14 @@
    - Storage quota enabled + Storage quota active

    - {% if max_bytes != null and max_objects != null %} - Limited to {{ max_bytes | filesizeformat }} and {{ max_objects }} objects. - {% elif max_bytes != null %} - Limited to {{ max_bytes | filesizeformat }} storage. - {% else %} - Limited to {{ max_objects }} objects. + {% if has_max_bytes and has_max_objects %} + This bucket is limited to {{ max_bytes | filesizeformat }} storage and {{ max_objects }} objects. + {% elif has_max_bytes %} + This bucket is limited to {{ max_bytes | filesizeformat }} storage. + {% elif has_max_objects %} + This bucket is limited to {{ max_objects }} objects. {% endif %}

    @@ -951,14 +944,14 @@ {% if can_manage_quota %}
    - +
    MB
    @@ -968,7 +961,7 @@
    Maximum number of objects allowed. Leave empty for unlimited.
    @@ -1058,7 +1051,7 @@ {% if can_manage_website %} - +
    @@ -1423,7 +1416,7 @@ Refresh - + - + @@ -2056,7 +2049,7 @@ data-multipart-complete-template="{{ url_for(endpoint="ui.complete_multipart_upload", bucket_name=bucket_name, upload_id="UPLOAD_ID_PLACEHOLDER") }}" data-multipart-abort-template="{{ url_for(endpoint="ui.abort_multipart_upload", bucket_name=bucket_name, upload_id="UPLOAD_ID_PLACEHOLDER") }}" > - +

    Upload files to {{ bucket_name }}. You can select multiple files at once.

    @@ -2212,7 +2205,7 @@
    - +
    - +

    Are you sure you want to delete this object? @@ -2440,7 +2433,7 @@

    - +
    + +