diff --git a/home.md b/home.md index 4be6e1f..3b08bde 100644 --- a/home.md +++ b/home.md @@ -145,13 +145,15 @@ All configuration is done via environment variables. The table below lists every | Variable | Default | Notes | | --- | --- | --- | -| `IAM_CONFIG` | `data/.myfsio.sys/config/iam.json` | Stores users, secrets, and inline policies. | +| `IAM_CONFIG` | `data/.myfsio.sys/config/iam.json` | Stores users, secrets, and inline policies. Encrypted at rest when `SECRET_KEY` is set. | | `BUCKET_POLICY_PATH` | `data/.myfsio.sys/config/bucket_policies.json` | Bucket policy store (auto hot-reload). | | `AUTH_MAX_ATTEMPTS` | `5` | Failed login attempts before lockout. | | `AUTH_LOCKOUT_MINUTES` | `15` | Lockout duration after max failed attempts. | | `SESSION_LIFETIME_DAYS` | `30` | How long UI sessions remain valid. | | `SECRET_TTL_SECONDS` | `300` | TTL for ephemeral secrets (presigned URLs). | | `UI_ENFORCE_BUCKET_POLICIES` | `false` | Whether the UI should enforce bucket policies. | +| `ADMIN_ACCESS_KEY` | (none) | Custom access key for the admin user on first run or credential reset. If unset, a random key is generated. | +| `ADMIN_SECRET_KEY` | (none) | Custom secret key for the admin user on first run or credential reset. If unset, a random key is generated. | ### CORS (Cross-Origin Resource Sharing) @@ -250,6 +252,60 @@ Once enabled, configure lifecycle rules via: ``` +## Garbage Collection + +The garbage collector (GC) automatically cleans up orphaned data that accumulates over time: stale temporary files from failed uploads, abandoned multipart uploads, stale lock files, orphaned metadata entries, orphaned version files, and empty directories. + +### Enabling GC + +By default, GC is disabled. Enable it by setting: + +```bash +GC_ENABLED=true python run.py +``` + +Or in your `myfsio.env` file: +``` +GC_ENABLED=true +GC_INTERVAL_HOURS=6 # Run every 6 hours (default) +GC_TEMP_FILE_MAX_AGE_HOURS=24 # Delete temp files older than 24h +GC_MULTIPART_MAX_AGE_DAYS=7 # Delete orphaned multipart uploads older than 7 days +GC_LOCK_FILE_MAX_AGE_HOURS=1 # Delete stale lock files older than 1h +GC_DRY_RUN=false # Set to true to log without deleting +``` + +### What Gets Cleaned + +| Type | Location | Condition | +|------|----------|-----------| +| **Temp files** | `.myfsio.sys/tmp/` | Older than `GC_TEMP_FILE_MAX_AGE_HOURS` | +| **Orphaned multipart uploads** | `.myfsio.sys/multipart/` and `/.multipart/` | Older than `GC_MULTIPART_MAX_AGE_DAYS` | +| **Stale lock files** | `.myfsio.sys/buckets//locks/` | Older than `GC_LOCK_FILE_MAX_AGE_HOURS` | +| **Orphaned metadata** | `.myfsio.sys/buckets//meta/` and `/.meta/` | Object file no longer exists | +| **Orphaned versions** | `.myfsio.sys/buckets//versions/` and `/.versions/` | Main object no longer exists | +| **Empty directories** | Various internal directories | Directory is empty after cleanup | + +### Admin API + +All GC endpoints require admin (`iam:*`) permissions. + +| Method | Route | Description | +|--------|-------|-------------| +| `GET` | `/admin/gc/status` | Get GC status and configuration | +| `POST` | `/admin/gc/run` | Trigger a manual GC run (body: `{"dry_run": true}` for preview) | +| `GET` | `/admin/gc/history` | Get GC execution history (query: `?limit=50&offset=0`) | + +### Dry Run Mode + +Set `GC_DRY_RUN=true` to log what would be deleted without actually removing anything. You can also trigger a one-time dry run via the admin API: + +```bash +curl -X POST "http://localhost:5000/admin/gc/run" \ + -H "X-Access-Key: " -H "X-Secret-Key: " \ + -H "Content-Type: application/json" \ + -d '{"dry_run": true}' +``` + ### Performance Tuning | Variable | Default | Notes | @@ -277,13 +333,14 @@ API responses for JSON, XML, HTML, CSS, and JavaScript are automatically gzip-co Before deploying to production, ensure you: -1. **Set `SECRET_KEY`** - Use a strong, unique value (e.g., `openssl rand -base64 32`) +1. **Set `SECRET_KEY`** - Use a strong, unique value (e.g., `openssl rand -base64 32`). This also enables IAM config encryption at rest. 2. **Restrict CORS** - Set `CORS_ORIGINS` to your specific domains instead of `*` 3. **Configure `API_BASE_URL`** - Required for correct presigned URLs behind proxies 4. **Enable HTTPS** - Use a reverse proxy (nginx, Cloudflare) with TLS termination 5. **Review rate limits** - Adjust `RATE_LIMIT_DEFAULT` based on your needs 6. **Secure master keys** - Back up `ENCRYPTION_MASTER_KEY_PATH` if using encryption 7. **Use `--prod` flag** - Runs with Waitress instead of Flask dev server +8. **Set credential expiry** - Assign `expires_at` to non-admin users for time-limited access ### Proxy Configuration @@ -633,9 +690,10 @@ MyFSIO implements a comprehensive Identity and Access Management (IAM) system th ### Getting Started -1. On first boot, `data/.myfsio.sys/config/iam.json` is created with a randomly generated admin user. The access key and secret key are printed to the console during first startup. If you miss it, check the `iam.json` file directly—credentials are stored in plaintext. +1. On first boot, `data/.myfsio.sys/config/iam.json` is created with a randomly generated admin user. The access key and secret key are printed to the console during first startup. You can set `ADMIN_ACCESS_KEY` and `ADMIN_SECRET_KEY` environment variables to use custom credentials instead of random ones. If `SECRET_KEY` is configured, the IAM config file is encrypted at rest using AES (Fernet). To reset admin credentials later, run `python run.py --reset-cred`. 2. Sign into the UI using the generated credentials, then open **IAM**: - - **Create user**: supply a display name and optional JSON inline policy array. + - **Create user**: supply a display name, optional JSON inline policy array, and optional credential expiry date. + - **Set expiry**: assign an expiration date to any user's credentials. Expired credentials are rejected at authentication time. The UI shows expiry badges and preset durations (1h, 24h, 7d, 30d, 90d). - **Rotate secret**: generates a new secret key; the UI surfaces it once. - **Policy editor**: select a user, paste an array of objects (`{"bucket": "*", "actions": ["list", "read"]}`), and submit. Alias support includes AWS-style verbs (e.g., `s3:GetObject`). 3. Wildcard action `iam:*` is supported for admin user definitions. @@ -653,8 +711,11 @@ The API expects every request to include authentication headers. The UI persists **Security Features:** - **Lockout Protection**: After `AUTH_MAX_ATTEMPTS` (default: 5) failed login attempts, the account is locked for `AUTH_LOCKOUT_MINUTES` (default: 15 minutes). +- **Credential Expiry**: Each user can have an optional `expires_at` timestamp (ISO 8601). Once expired, all API requests using those credentials are rejected. Set or clear expiry via the UI or API. +- **IAM Config Encryption**: When `SECRET_KEY` is set, the IAM config file (`iam.json`) is encrypted at rest using Fernet (AES-256-CBC with HMAC). Existing plaintext configs are automatically encrypted on next load. - **Session Management**: UI sessions remain valid for `SESSION_LIFETIME_DAYS` (default: 30 days). - **Hot Reload**: IAM configuration changes take effect immediately without restart. +- **Credential Reset**: Run `python run.py --reset-cred` to reset admin credentials. Supports `ADMIN_ACCESS_KEY` and `ADMIN_SECRET_KEY` env vars for deterministic keys. ### Permission Model @@ -814,7 +875,8 @@ curl -X POST http://localhost:5000/iam/users \ -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ -d '{ "display_name": "New User", - "policies": [{"bucket": "*", "actions": ["list", "read"]}] + "policies": [{"bucket": "*", "actions": ["list", "read"]}], + "expires_at": "2026-12-31T23:59:59Z" }' # Rotate user secret (requires iam:rotate_key) @@ -827,6 +889,18 @@ curl -X PUT http://localhost:5000/iam/users//policies \ -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ -d '[{"bucket": "*", "actions": ["list", "read", "write"]}]' +# Update credential expiry (requires iam:update_policy) +curl -X POST http://localhost:5000/iam/users//expiry \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d 'expires_at=2026-12-31T23:59:59Z' + +# Remove credential expiry (never expires) +curl -X POST http://localhost:5000/iam/users//expiry \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d 'expires_at=' + # Delete a user (requires iam:delete_user) curl -X DELETE http://localhost:5000/iam/users/ \ -H "X-Access-Key: ..." -H "X-Secret-Key: ..." @@ -838,8 +912,9 @@ When a request is made, permissions are evaluated in this order: 1. **Authentication** – Verify the access key and secret key are valid 2. **Lockout Check** – Ensure the account is not locked due to failed attempts -3. **IAM Policy Check** – Verify the user has the required action for the target bucket -4. **Bucket Policy Check** – If a bucket policy exists, verify it allows the action +3. **Expiry Check** – Reject requests if the user's credentials have expired (`expires_at`) +4. **IAM Policy Check** – Verify the user has the required action for the target bucket +5. **Bucket Policy Check** – If a bucket policy exists, verify it allows the action A request is allowed only if: - The IAM policy grants the action, AND @@ -2413,4 +2488,4 @@ All endpoints require admin (`iam:*`) permissions. ### Web UI - **Per-bucket config:** Bucket Details → Properties tab → "Static Website Hosting" card -- **Domain management:** Sidebar → "Domains" (visible when hosting is enabled and user is admin) \ No newline at end of file +- **Domain management:** Sidebar → "Domains" (visible when hosting is enabled and user is admin)