From 8f4d4bd0fbb1e1140135db20cb4d060325a34f6d Mon Sep 17 00:00:00 2001 From: kqjy Date: Mon, 12 Jan 2026 08:05:02 +0000 Subject: [PATCH] Update home --- home.md | 569 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 558 insertions(+), 11 deletions(-) diff --git a/home.md b/home.md index ad1c2b3..2a01806 100644 --- a/home.md +++ b/home.md @@ -33,6 +33,63 @@ python run.py --mode api # API only (port 5000) python run.py --mode ui # UI only (port 5100) ``` +### Configuration validation + +Validate your configuration before deploying: + +```bash +# Show configuration summary +python run.py --show-config +./myfsio --show-config + +# Validate and check for issues (exits with code 1 if critical issues found) +python run.py --check-config +./myfsio --check-config +``` + +### Linux Installation (Recommended for Production) + +For production deployments on Linux, use the provided installation script: + +```bash +# Download the binary and install script +# Then run the installer with sudo: +sudo ./scripts/install.sh --binary ./myfsio + +# Or with custom paths: +sudo ./scripts/install.sh \ + --binary ./myfsio \ + --install-dir /opt/myfsio \ + --data-dir /mnt/storage/myfsio \ + --log-dir /var/log/myfsio \ + --api-url https://s3.example.com \ + --user myfsio + +# Non-interactive mode (for automation): +sudo ./scripts/install.sh --binary ./myfsio -y +``` + +The installer will: +1. Create a dedicated system user +2. Set up directories with proper permissions +3. Generate a secure `SECRET_KEY` +4. Create an environment file at `/opt/myfsio/myfsio.env` +5. Install and configure a systemd service + +After installation: +```bash +sudo systemctl start myfsio # Start the service +sudo systemctl enable myfsio # Enable on boot +sudo systemctl status myfsio # Check status +sudo journalctl -u myfsio -f # View logs +``` + +To uninstall: +```bash +sudo ./scripts/uninstall.sh # Full removal +sudo ./scripts/uninstall.sh --keep-data # Keep data directory +``` + ### Docker quickstart The repo now ships a `Dockerfile` so you can run both services in one container: @@ -69,23 +126,143 @@ The repo now tracks a human-friendly release string inside `app/version.py` (see ## 3. Configuration Reference +All configuration is done via environment variables. The table below lists every supported variable. + +### Core Settings + | Variable | Default | Notes | | --- | --- | --- | | `STORAGE_ROOT` | `/data` | Filesystem home for all buckets/objects. | -| `MAX_UPLOAD_SIZE` | `1073741824` | Bytes. Caps incoming uploads in both API + UI. | +| `MAX_UPLOAD_SIZE` | `1073741824` (1 GiB) | Bytes. Caps incoming uploads in both API + UI. | | `UI_PAGE_SIZE` | `100` | `MaxKeys` hint shown in listings. | -| `SECRET_KEY` | `dev-secret-key` | Flask session key for UI auth. | -| `IAM_CONFIG` | `/data/.myfsio.sys/config/iam.json` | Stores users, secrets, and inline policies. | -| `BUCKET_POLICY_PATH` | `/data/.myfsio.sys/config/bucket_policies.json` | Bucket policy store (auto hot-reload). | -| `API_BASE_URL` | `None` | Used by the UI to hit API endpoints (presign/policy). If unset, the UI will auto-detect the host or use `X-Forwarded-*` headers. | +| `SECRET_KEY` | Auto-generated | Flask session key. Auto-generates and persists if not set. **Set explicitly in production.** | +| `API_BASE_URL` | `None` | Public URL for presigned URLs. Required behind proxies. | | `AWS_REGION` | `us-east-1` | Region embedded in SigV4 credential scope. | | `AWS_SERVICE` | `s3` | Service string for SigV4. | -| `ENCRYPTION_ENABLED` | `false` | Enable server-side encryption support. | -| `KMS_ENABLED` | `false` | Enable KMS key management for encryption. | -| `KMS_KEYS_PATH` | `data/kms_keys.json` | Path to store KMS key metadata. | -| `ENCRYPTION_MASTER_KEY_PATH` | `data/master.key` | Path to the master encryption key file. | -Set env vars (or pass overrides to `create_app`) to point the servers at custom paths. +### IAM & Security + +| Variable | Default | Notes | +| --- | --- | --- | +| `IAM_CONFIG` | `data/.myfsio.sys/config/iam.json` | Stores users, secrets, and inline policies. | +| `BUCKET_POLICY_PATH` | `data/.myfsio.sys/config/bucket_policies.json` | Bucket policy store (auto hot-reload). | +| `AUTH_MAX_ATTEMPTS` | `5` | Failed login attempts before lockout. | +| `AUTH_LOCKOUT_MINUTES` | `15` | Lockout duration after max failed attempts. | +| `SESSION_LIFETIME_DAYS` | `30` | How long UI sessions remain valid. | +| `SECRET_TTL_SECONDS` | `300` | TTL for ephemeral secrets (presigned URLs). | +| `UI_ENFORCE_BUCKET_POLICIES` | `false` | Whether the UI should enforce bucket policies. | + +### CORS (Cross-Origin Resource Sharing) + +| Variable | Default | Notes | +| --- | --- | --- | +| `CORS_ORIGINS` | `*` | Comma-separated allowed origins. Use specific domains in production. | +| `CORS_METHODS` | `GET,PUT,POST,DELETE,OPTIONS,HEAD` | Allowed HTTP methods. | +| `CORS_ALLOW_HEADERS` | `*` | Allowed request headers. | +| `CORS_EXPOSE_HEADERS` | `*` | Response headers visible to browsers (e.g., `ETag`). | + +### Rate Limiting + +| Variable | Default | Notes | +| --- | --- | --- | +| `RATE_LIMIT_DEFAULT` | `200 per minute` | Default rate limit for API endpoints. | +| `RATE_LIMIT_STORAGE_URI` | `memory://` | Storage backend for rate limits. Use `redis://host:port` for distributed setups. | + +### Logging + +| Variable | Default | Notes | +| --- | --- | --- | +| `LOG_LEVEL` | `INFO` | Log verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR`. | +| `LOG_TO_FILE` | `true` | Enable file logging. | +| `LOG_DIR` | `/logs` | Directory for log files. | +| `LOG_FILE` | `app.log` | Log filename. | +| `LOG_MAX_BYTES` | `5242880` (5 MB) | Max log file size before rotation. | +| `LOG_BACKUP_COUNT` | `3` | Number of rotated log files to keep. | + +### Encryption + +| Variable | Default | Notes | +| --- | --- | --- | +| `ENCRYPTION_ENABLED` | `false` | Enable server-side encryption support. | +| `ENCRYPTION_MASTER_KEY_PATH` | `data/.myfsio.sys/keys/master.key` | Path to the master encryption key file. | +| `DEFAULT_ENCRYPTION_ALGORITHM` | `AES256` | Default algorithm for new encrypted objects. | +| `KMS_ENABLED` | `false` | Enable KMS key management for encryption. | +| `KMS_KEYS_PATH` | `data/.myfsio.sys/keys/kms_keys.json` | Path to store KMS key metadata. | + + +## Lifecycle Rules + +Lifecycle rules automate object management by scheduling deletions based on object age. + +### Enabling Lifecycle Enforcement + +By default, lifecycle enforcement is disabled. Enable it by setting the environment variable: + +```bash +LIFECYCLE_ENABLED=true python run.py +``` + +Or in your `myfsio.env` file: +``` +LIFECYCLE_ENABLED=true +LIFECYCLE_INTERVAL_SECONDS=3600 # Check interval (default: 1 hour) +``` + +### Configuring Rules + +Once enabled, configure lifecycle rules via: +- **Web UI:** Bucket Details → Lifecycle tab → Add Rule +- **S3 API:** `PUT /?lifecycle` with XML configuration + +### Available Actions + +| Action | Description | +|--------|-------------| +| **Expiration** | Delete current version objects after N days | +| **NoncurrentVersionExpiration** | Delete old versions N days after becoming noncurrent (requires versioning) | +| **AbortIncompleteMultipartUpload** | Clean up incomplete multipart uploads after N days | + +### Example Configuration (XML) + +```xml + + + DeleteOldLogs + Enabled + logs/ + 30 + + +``` + +### Performance Tuning + +| Variable | Default | Notes | +| --- | --- | --- | +| `STREAM_CHUNK_SIZE` | `65536` (64 KB) | Chunk size for streaming large files. | +| `MULTIPART_MIN_PART_SIZE` | `5242880` (5 MB) | Minimum part size for multipart uploads. | +| `BUCKET_STATS_CACHE_TTL` | `60` | Seconds to cache bucket statistics. | +| `BULK_DELETE_MAX_KEYS` | `500` | Maximum keys per bulk delete request. | + +### Server Settings + +| Variable | Default | Notes | +| --- | --- | --- | +| `APP_HOST` | `0.0.0.0` | Network interface to bind to. | +| `APP_PORT` | `5000` | API server port (UI uses 5100). | +| `FLASK_DEBUG` | `0` | Enable Flask debug mode. **Never enable in production.** | + +### Production Checklist + +Before deploying to production, ensure you: + +1. **Set `SECRET_KEY`** - Use a strong, unique value (e.g., `openssl rand -base64 32`) +2. **Restrict CORS** - Set `CORS_ORIGINS` to your specific domains instead of `*` +3. **Configure `API_BASE_URL`** - Required for correct presigned URLs behind proxies +4. **Enable HTTPS** - Use a reverse proxy (nginx, Cloudflare) with TLS termination +5. **Review rate limits** - Adjust `RATE_LIMIT_DEFAULT` based on your needs +6. **Secure master keys** - Back up `ENCRYPTION_MASTER_KEY_PATH` if using encryption +7. **Use `--prod` flag** - Runs with Waitress instead of Flask dev server ### Proxy Configuration @@ -95,6 +272,334 @@ If running behind a reverse proxy (e.g., Nginx, Cloudflare, or a tunnel), ensure The application automatically trusts these headers to generate correct presigned URLs (e.g., `https://s3.example.com/...` instead of `http://127.0.0.1:5000/...`). Alternatively, you can explicitly set `API_BASE_URL` to your public endpoint. +## 4. Upgrading and Updates + +### Version Checking + +The application version is tracked in `app/version.py` and exposed via: +- **Health endpoint:** `GET /healthz` returns JSON with `version` field +- **Metrics dashboard:** Navigate to `/ui/metrics` to see the running version in the System Status card + +To check your current version: + +```bash +# API health endpoint +curl http://localhost:5000/healthz + +# Or inspect version.py directly +cat app/version.py | grep APP_VERSION +``` + +### Pre-Update Backup Procedures + +**Always backup before upgrading to prevent data loss:** + +```bash +# 1. Stop the application +# Ctrl+C if running in terminal, or: +docker stop myfsio # if using Docker + +# 2. Backup configuration files (CRITICAL) +mkdir -p backups/$(date +%Y%m%d_%H%M%S) +cp -r data/.myfsio.sys/config backups/$(date +%Y%m%d_%H%M%S)/ + +# 3. Backup all data (optional but recommended) +tar -czf backups/data_$(date +%Y%m%d_%H%M%S).tar.gz data/ + +# 4. Backup logs for audit trail +cp -r logs backups/$(date +%Y%m%d_%H%M%S)/ +``` + +**Windows PowerShell:** + +```powershell +# Create timestamped backup +$timestamp = Get-Date -Format "yyyyMMdd_HHmmss" +New-Item -ItemType Directory -Path "backups\$timestamp" -Force + +# Backup configs +Copy-Item -Recurse "data\.myfsio.sys\config" "backups\$timestamp\" + +# Backup entire data directory +Compress-Archive -Path "data\" -DestinationPath "backups\data_$timestamp.zip" +``` + +**Critical files to backup:** +- `data/.myfsio.sys/config/iam.json` – User accounts and access keys +- `data/.myfsio.sys/config/bucket_policies.json` – Bucket access policies +- `data/.myfsio.sys/config/kms_keys.json` – Encryption keys (if using KMS) +- `data/.myfsio.sys/config/secret_store.json` – Application secrets + +### Update Procedures + +#### Source Installation Updates + +```bash +# 1. Backup (see above) +# 2. Pull latest code +git fetch origin +git checkout main # or your target branch/tag +git pull + +# 3. Check for dependency changes +pip install -r requirements.txt + +# 4. Review CHANGELOG/release notes for breaking changes +cat CHANGELOG.md # if available + +# 5. Run migration scripts (if any) +# python scripts/migrate_vX_to_vY.py # example + +# 6. Restart application +python run.py +``` + +#### Docker Updates + +```bash +# 1. Backup (see above) +# 2. Pull/rebuild image +docker pull yourregistry/myfsio:latest +# OR rebuild from source: +docker build -t myfsio:latest . + +# 3. Stop and remove old container +docker stop myfsio +docker rm myfsio + +# 4. Start new container with same volumes +docker run -d \ + --name myfsio \ + -p 5000:5000 -p 5100:5100 \ + -v "$(pwd)/data:/app/data" \ + -v "$(pwd)/logs:/app/logs" \ + -e SECRET_KEY="your-secret" \ + myfsio:latest + +# 5. Verify health +curl http://localhost:5000/healthz +``` + +### Version Compatibility Checks + +Before upgrading across major versions, verify compatibility: + +| From Version | To Version | Breaking Changes | Migration Required | +|--------------|------------|------------------|-------------------| +| 0.1.x | 0.2.x | None expected | No | +| 0.1.6 | 0.1.7 | None | No | +| < 0.1.0 | >= 0.1.0 | New IAM config format | Yes - run migration script | + +**Automatic compatibility detection:** + +The application will log warnings on startup if config files need migration: + +``` +WARNING: IAM config format is outdated (v1). Please run: python scripts/migrate_iam.py +``` + +**Manual compatibility check:** + +```bash +# Compare version schemas +python -c "from app.version import APP_VERSION; print(f'Running: {APP_VERSION}')" +python scripts/check_compatibility.py data/.myfsio.sys/config/ +``` + +### Migration Steps for Breaking Changes + +When release notes indicate breaking changes, follow these steps: + +#### Config Format Migrations + +```bash +# 1. Backup first (critical!) +cp data/.myfsio.sys/config/iam.json data/.myfsio.sys/config/iam.json.backup + +# 2. Run provided migration script +python scripts/migrate_iam_v1_to_v2.py + +# 3. Validate migration +python scripts/validate_config.py + +# 4. Test with read-only mode first (if available) +# python run.py --read-only + +# 5. Restart normally +python run.py +``` + +#### Database/Storage Schema Changes + +If object metadata format changes: + +```bash +# 1. Run storage migration script +python scripts/migrate_storage.py --dry-run # preview changes + +# 2. Apply migration +python scripts/migrate_storage.py --apply + +# 3. Verify integrity +python scripts/verify_storage.py +``` + +#### IAM Policy Updates + +If IAM action names change (e.g., `s3:Get` → `s3:GetObject`): + +```bash +# Migration script will update all policies +python scripts/migrate_policies.py \ + --input data/.myfsio.sys/config/iam.json \ + --backup data/.myfsio.sys/config/iam.json.v1 + +# Review changes before committing +python scripts/diff_policies.py \ + data/.myfsio.sys/config/iam.json.v1 \ + data/.myfsio.sys/config/iam.json +``` + +### Rollback Procedures + +If an update causes issues, rollback to the previous version: + +#### Quick Rollback (Source) + +```bash +# 1. Stop application +# Ctrl+C or kill process + +# 2. Revert code +git checkout +# OR +git reset --hard HEAD~1 + +# 3. Restore configs from backup +cp backups/20241213_103000/config/* data/.myfsio.sys/config/ + +# 4. Downgrade dependencies if needed +pip install -r requirements.txt + +# 5. Restart +python run.py +``` + +#### Docker Rollback + +```bash +# 1. Stop current container +docker stop myfsio +docker rm myfsio + +# 2. Start previous version +docker run -d \ + --name myfsio \ + -p 5000:5000 -p 5100:5100 \ + -v "$(pwd)/data:/app/data" \ + -v "$(pwd)/logs:/app/logs" \ + -e SECRET_KEY="your-secret" \ + myfsio:0.1.3 # specify previous version tag + +# 3. Verify +curl http://localhost:5000/healthz +``` + +#### Emergency Config Restore + +If only config is corrupted but code is fine: + +```bash +# Stop app +# Restore from latest backup +cp backups/20241213_103000/config/iam.json data/.myfsio.sys/config/ +cp backups/20241213_103000/config/bucket_policies.json data/.myfsio.sys/config/ + +# Restart app +python run.py +``` + +### Blue-Green Deployment (Zero Downtime) + +For production environments requiring zero downtime: + +```bash +# 1. Run new version on different port (e.g., 5001/5101) +APP_PORT=5001 UI_PORT=5101 python run.py & + +# 2. Health check new instance +curl http://localhost:5001/healthz + +# 3. Update load balancer to route to new ports + +# 4. Monitor for issues + +# 5. Gracefully stop old instance +kill -SIGTERM +``` + +### Post-Update Verification + +After any update, verify functionality: + +```bash +# 1. Health check +curl http://localhost:5000/healthz + +# 2. Login to UI +open http://localhost:5100/ui + +# 3. Test IAM authentication +curl -H "X-Amz-Security-Token: :" \ + http://localhost:5000/ + +# 4. Test presigned URL generation +# Via UI or API + +# 5. Check logs for errors +tail -n 100 logs/myfsio.log +``` + +### Automated Update Scripts + +Create a custom update script for your environment: + +```bash +#!/bin/bash +# update.sh - Automated update with rollback capability + +set -e # Exit on error + +VERSION_NEW="$1" +BACKUP_DIR="backups/$(date +%Y%m%d_%H%M%S)" + +echo "Creating backup..." +mkdir -p "$BACKUP_DIR" +cp -r data/.myfsio.sys/config "$BACKUP_DIR/" + +echo "Updating to version $VERSION_NEW..." +git fetch origin +git checkout "v$VERSION_NEW" +pip install -r requirements.txt + +echo "Starting application..." +python run.py & +APP_PID=$! + +# Wait and health check +sleep 5 +if curl -f http://localhost:5000/healthz; then + echo "Update successful!" +else + echo "Health check failed, rolling back..." + kill $APP_PID + git checkout - + cp -r "$BACKUP_DIR/config/*" data/.myfsio.sys/config/ + python run.py & + exit 1 +fi +``` + ## 4. Authentication & IAM 1. On first boot, `data/.myfsio.sys/config/iam.json` is seeded with `localadmin / localadmin` that has wildcard access. @@ -176,6 +681,48 @@ curl -X PUT http://127.0.0.1:5000/bucket-policy/test \ The UI will reflect this change as soon as the request completes thanks to the hot reload. +### UI Object Browser + +The bucket detail page includes a powerful object browser with the following features: + +#### Folder Navigation + +Objects with forward slashes (`/`) in their keys are displayed as a folder hierarchy. Click a folder row to navigate into it. A breadcrumb navigation bar shows your current path and allows quick navigation back to parent folders or the root. + +#### Pagination & Infinite Scroll + +- Objects load in configurable batches (50, 100, 150, 200, or 250 per page) +- Scroll to the bottom to automatically load more objects (infinite scroll) +- A **Load more** button is available as a fallback for touch devices or when infinite scroll doesn't trigger +- The footer shows the current load status (e.g., "Showing 100 of 500 objects") + +#### Bulk Operations + +- Select multiple objects using checkboxes +- **Bulk Delete**: Delete multiple objects at once +- **Bulk Download**: Download selected objects as individual files + +#### Search & Filter + +Use the search box to filter objects by name in real-time. The filter applies to the currently loaded objects. + +#### Error Handling + +If object loading fails (e.g., network error), a friendly error message is displayed with a **Retry** button to attempt loading again. + +#### Object Preview + +Click any object row to view its details in the preview sidebar: +- File size and last modified date +- ETag (content hash) +- Custom metadata (if present) +- Download and presign (share link) buttons +- Version history (when versioning is enabled) + +#### Drag & Drop Upload + +Drag files directly onto the objects table to upload them to the current bucket and folder path. + ## 6. Presigned URLs - Trigger from the UI using the **Presign** button after selecting an object. @@ -576,4 +1123,4 @@ PUT /bucket-policy/ # Upsert policy DELETE /bucket-policy/ # Delete policy GET /?quota # Get bucket quota PUT /?quota # Set bucket quota (admin only) -``` +``` \ No newline at end of file