Update home

2026-02-26 09:42:51 +00:00
parent 0aac42b2b3
commit fc25693215
1 changed files with 193 additions and 6 deletions
--- a/home.md
+++ b/home.md
@@ -7,7 +7,7 @@ This document expands on the README to describe the full workflow for running, c
 MyFSIO ships two Flask entrypoints that share the same storage, IAM, and bucket-policy state:

 - **API server** – Implements the S3-compatible REST API, policy evaluation, and Signature Version 4 presign service.
- **UI server** – Provides the browser console for buckets, IAM, and policies. It proxies to the API for presign operations.
+- **UI server** – Provides the browser console for buckets, IAM, and policies. It proxies all storage operations through the S3 API via boto3 (SigV4-signed), mirroring the architecture used by MinIO and Garage.

 Both servers read `AppConfig`, so editing JSON stores on disk instantly affects both surfaces.

@@ -136,9 +136,10 @@ All configuration is done via environment variables. The table below lists every
 | `MAX_UPLOAD_SIZE` | `1073741824` (1 GiB) | Bytes. Caps incoming uploads in both API + UI. |
 | `UI_PAGE_SIZE` | `100` | `MaxKeys` hint shown in listings. |
 | `SECRET_KEY` | Auto-generated | Flask session key. Auto-generates and persists if not set. **Set explicitly in production.** |
-| `API_BASE_URL` | `None` | Public URL for presigned URLs. Required behind proxies. |
+| `API_BASE_URL` | `http://127.0.0.1:5000` | Internal S3 API URL used by the web UI proxy. Also used for presigned URL generation. Set to your public URL if running behind a reverse proxy. |
 | `AWS_REGION` | `us-east-1` | Region embedded in SigV4 credential scope. |
 | `AWS_SERVICE` | `s3` | Service string for SigV4. |
+| `DISPLAY_TIMEZONE` | `UTC` | Timezone for timestamps in the web UI (e.g., `US/Eastern`, `Asia/Tokyo`). |

 ### IAM & Security

@@ -170,6 +171,7 @@ All configuration is done via environment variables. The table below lists every
 | `RATE_LIMIT_BUCKET_OPS` | `120 per minute` | Rate limit for bucket operations (PUT/DELETE/GET/POST on `/<bucket>`). |
 | `RATE_LIMIT_OBJECT_OPS` | `240 per minute` | Rate limit for object operations (PUT/GET/DELETE/POST on `/<bucket>/<key>`). |
 | `RATE_LIMIT_HEAD_OPS` | `100 per minute` | Rate limit for HEAD requests (bucket and object). |
+| `RATE_LIMIT_ADMIN` | `60 per minute` | Rate limit for admin API endpoints (`/admin/*`). |
 | `RATE_LIMIT_STORAGE_URI` | `memory://` | Storage backend for rate limits. Use `redis://host:port` for distributed setups. |

 ### Server Configuration
@@ -256,6 +258,12 @@ Once enabled, configure lifecycle rules via:
 | `MULTIPART_MIN_PART_SIZE` | `5242880` (5 MB) | Minimum part size for multipart uploads. |
 | `BUCKET_STATS_CACHE_TTL` | `60` | Seconds to cache bucket statistics. |
 | `BULK_DELETE_MAX_KEYS` | `500` | Maximum keys per bulk delete request. |
+| `BULK_DOWNLOAD_MAX_BYTES` | `1073741824` (1 GiB) | Maximum total size for bulk ZIP downloads. |
+| `OBJECT_CACHE_TTL` | `60` | Seconds to cache object metadata. |
+
+#### Gzip Compression
+
+API responses for JSON, XML, HTML, CSS, and JavaScript are automatically gzip-compressed when the client sends `Accept-Encoding: gzip`. Compression activates for responses larger than 500 bytes and is handled by a WSGI middleware (`app/compression.py`). Binary object downloads and streaming responses are never compressed. No configuration is needed.

 ### Server Settings

@@ -285,6 +293,12 @@ If running behind a reverse proxy (e.g., Nginx, Cloudflare, or a tunnel), ensure

 The application automatically trusts these headers to generate correct presigned URLs (e.g., `https://s3.example.com/...` instead of `http://127.0.0.1:5000/...`). Alternatively, you can explicitly set `API_BASE_URL` to your public endpoint.

+| Variable | Default | Notes |
+| --- | --- | --- |
+| `NUM_TRUSTED_PROXIES` | `1` | Number of trusted reverse proxies for `X-Forwarded-*` header processing. |
+| `ALLOWED_REDIRECT_HOSTS` | `""` | Comma-separated whitelist of safe redirect targets. Empty allows only same-host redirects. |
+| `ALLOW_INTERNAL_ENDPOINTS` | `false` | Allow connections to internal/private IPs for webhooks and replication targets. **Keep disabled in production unless needed.** |
+
 ## 4. Upgrading and Updates

 ### Version Checking
@@ -619,13 +633,15 @@ MyFSIO implements a comprehensive Identity and Access Management (IAM) system th

 ### Getting Started

-1. On first boot, `data/.myfsio.sys/config/iam.json` is seeded with `localadmin / localadmin` that has wildcard access.
-2. Sign into the UI using those credentials, then open **IAM**:
+1. On first boot, `data/.myfsio.sys/config/iam.json` is created with a randomly generated admin user. The access key and secret key are printed to the console during first startup. If you miss it, check the `iam.json` file directly—credentials are stored in plaintext.
+2. Sign into the UI using the generated credentials, then open **IAM**:
   - **Create user**: supply a display name and optional JSON inline policy array.
   - **Rotate secret**: generates a new secret key; the UI surfaces it once.
   - **Policy editor**: select a user, paste an array of objects (`{"bucket": "*", "actions": ["list", "read"]}`), and submit. Alias support includes AWS-style verbs (e.g., `s3:GetObject`).
 3. Wildcard action `iam:*` is supported for admin user definitions.

+> **Breaking Change (v0.2.0+):** Previous versions used fixed default credentials (`localadmin/localadmin`). If upgrading from an older version, your existing credentials remain unchanged, but new installations will generate random credentials.
+
 ### Authentication

 The API expects every request to include authentication headers. The UI persists them in the Flask session after login.
@@ -910,7 +926,7 @@ Objects with forward slashes (`/`) in their keys are displayed as a folder hiera

 - Select multiple objects using checkboxes
 - **Bulk Delete**: Delete multiple objects at once
- **Bulk Download**: Download selected objects as individual files
+- **Bulk Download**: Download selected objects as a single ZIP archive (up to `BULK_DOWNLOAD_MAX_BYTES`, default 1 GiB)

 #### Search & Filter

@@ -983,6 +999,7 @@ MyFSIO supports **server-side encryption at rest** to protect your data. When en
 |------|-------------|
 | **AES-256 (SSE-S3)** | Server-managed encryption using a local master key |
 | **KMS (SSE-KMS)** | Encryption using customer-managed keys via the built-in KMS |
+| **SSE-C** | Server-side encryption with customer-provided keys (per-request) |

 ### Enabling Encryption

@@ -1081,6 +1098,44 @@ encrypted, metadata = ClientEncryptionHelper.encrypt_for_upload(plaintext, key)
 decrypted = ClientEncryptionHelper.decrypt_from_download(encrypted, metadata, key)
 ```

+### SSE-C (Customer-Provided Keys)
+
+With SSE-C, you provide your own 256-bit AES encryption key with each request. The server encrypts/decrypts using your key but never stores it. You must supply the same key for both upload and download.
+
+**Required headers:**
+
+| Header | Value |
+|--------|-------|
+| `x-amz-server-side-encryption-customer-algorithm` | `AES256` |
+| `x-amz-server-side-encryption-customer-key` | Base64-encoded 256-bit key |
+| `x-amz-server-side-encryption-customer-key-MD5` | Base64-encoded MD5 of the key |
+
+```bash
+# Generate a 256-bit key
+KEY=$(openssl rand -base64 32)
+KEY_MD5=$(echo -n "$KEY" | base64 -d | openssl dgst -md5 -binary | base64)
+
+# Upload with SSE-C
+curl -X PUT "http://localhost:5000/my-bucket/secret.txt" \
+  -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \
+  -H "x-amz-server-side-encryption-customer-algorithm: AES256" \
+  -H "x-amz-server-side-encryption-customer-key: $KEY" \
+  -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" \
+  --data-binary @secret.txt
+
+# Download with SSE-C (same key required)
+curl "http://localhost:5000/my-bucket/secret.txt" \
+  -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \
+  -H "x-amz-server-side-encryption-customer-algorithm: AES256" \
+  -H "x-amz-server-side-encryption-customer-key: $KEY" \
+  -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5"
+```
+
+**Key points:**
+- SSE-C does not require `ENCRYPTION_ENABLED` or `KMS_ENABLED` — the key is provided per-request
+- If you lose your key, the data is irrecoverable
+- The MD5 header is optional but recommended for integrity verification
+
 ### Important Notes

 - **Existing objects are NOT encrypted** - Only new uploads after enabling encryption are encrypted
@@ -1550,6 +1605,9 @@ GET    /<bucket>?notification           # Get event notifications
 PUT    /<bucket>?notification           # Set event notifications (webhooks)
 GET    /<bucket>?object-lock            # Get object lock configuration
 PUT    /<bucket>?object-lock            # Set object lock configuration
+GET    /<bucket>?website                # Get website configuration
+PUT    /<bucket>?website                # Set website configuration
+DELETE /<bucket>?website                # Delete website configuration
 GET    /<bucket>?uploads                # List active multipart uploads
 GET    /<bucket>?versions               # List object versions
 GET    /<bucket>?location               # Get bucket location/region
@@ -1594,6 +1652,11 @@ PUT    /admin/sites/<site_id>           # Update peer site
 DELETE /admin/sites/<site_id>           # Unregister peer site
 GET    /admin/sites/<site_id>/health    # Check peer health
 GET    /admin/topology                  # Get cluster topology
+GET    /admin/website-domains           # List domain mappings
+POST   /admin/website-domains           # Create domain mapping
+GET    /admin/website-domains/<domain>  # Get domain mapping
+PUT    /admin/website-domains/<domain>  # Update domain mapping
+DELETE /admin/website-domains/<domain>  # Delete domain mapping

 # KMS API
 GET    /kms/keys                        # List KMS keys
@@ -1949,6 +2012,20 @@ curl -X PUT "http://localhost:5000/my-bucket/file.txt" \
  -H "x-amz-meta-newkey: newvalue"
 ```

+### MoveObject (UI)
+
+Move an object to a different key or bucket. This is a UI-only convenience operation that performs a copy followed by a delete of the source. Requires `read` and `delete` on the source, and `write` on the destination.
+
+```bash
+# Move via UI API
+curl -X POST "http://localhost:5100/ui/buckets/my-bucket/objects/old-path/file.txt/move" \
+  -H "Content-Type: application/json" \
+  --cookie "session=..." \
+  -d '{"dest_bucket": "other-bucket", "dest_key": "new-path/file.txt"}'
+```
+
+The move is atomic from the caller's perspective: if the copy succeeds but the delete fails, the object exists in both locations (no data loss).
+
 ### UploadPartCopy

 Copy data from an existing object into a multipart upload part:
@@ -2226,4 +2303,114 @@ curl "http://localhost:5000/my-bucket?list-type=2&start-after=photos/2024/" \
 | `continuation-token` | Token from previous response |
 | `start-after` | Start listing after this key |
 | `fetch-owner` | Include owner info in response |
-| `encoding-type` | Set to `url` for URL-encoded keys
+| `encoding-type` | Set to `url` for URL-encoded keys
+
+## 26. Static Website Hosting
+
+MyFSIO can serve S3 buckets as static websites via custom domain mappings. When a request arrives with a `Host` header matching a mapped domain, MyFSIO resolves the bucket and serves objects directly.
+
+### Enabling
+
+Set the environment variable:
+
+```bash
+WEBSITE_HOSTING_ENABLED=true
+```
+
+When disabled, all website hosting endpoints return 400 and domain-based serving is skipped.
+
+### Configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `WEBSITE_HOSTING_ENABLED` | `false` | Master switch for website hosting |
+
+### Setting Up a Website
+
+**Step 1: Configure the bucket website settings**
+
+```bash
+curl -X PUT "http://localhost:5000/my-site?website" \
+  -H "Authorization: ..." \
+  -d '<?xml version="1.0" encoding="UTF-8"?>
+<WebsiteConfiguration>
+  <IndexDocument><Suffix>index.html</Suffix></IndexDocument>
+  <ErrorDocument><Key>404.html</Key></ErrorDocument>
+</WebsiteConfiguration>'
+```
+
+- `IndexDocument` with `Suffix` is required (must not contain `/`)
+- `ErrorDocument` is optional
+
+**Step 2: Map a domain to the bucket**
+
+```bash
+curl -X POST "http://localhost:5000/admin/website-domains" \
+  -H "Authorization: ..." \
+  -H "Content-Type: application/json" \
+  -d '{"domain": "example.com", "bucket": "my-site"}'
+```
+
+**Step 3: Point your domain to MyFSIO**
+
+For HTTP-only (direct access), point DNS to the MyFSIO host on port 5000.
+
+For HTTPS (recommended), use a reverse proxy. The critical requirement is passing the original `Host` header so MyFSIO can match the domain to a bucket.
+
+**nginx example:**
+
+```nginx
+server {
+    server_name example.com;
+    listen 443 ssl;
+
+    ssl_certificate     /etc/ssl/certs/example.com.pem;
+    ssl_certificate_key /etc/ssl/private/example.com.key;
+
+    location / {
+        proxy_pass http://127.0.0.1:5000;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}
+```
+
+`proxy_set_header Host $host;` is required — without it, MyFSIO cannot match the incoming domain to a bucket. You do not need any path-based routing rules; MyFSIO handles all object resolution internally.
+
+### How Domain Routing Works
+
+1. A request arrives with `Host: example.com`
+2. MyFSIO's `before_request` hook strips the port and looks up the domain in the `WebsiteDomainStore`
+3. If a match is found, it loads the bucket's website config (index/error documents)
+4. Object key resolution:
+   - `/` or trailing `/` → append `index_document` (e.g., `index.html`)
+   - `/path` → try exact match, then try `path/index_document`
+   - Not found → serve `error_document` with 404 status
+5. If no domain match is found, the request falls through to normal S3 API / UI routing
+
+### Domain Mapping Admin API
+
+All endpoints require admin (`iam:*`) permissions.
+
+| Method | Route | Body | Description |
+|--------|-------|------|-------------|
+| `GET` | `/admin/website-domains` | — | List all mappings |
+| `POST` | `/admin/website-domains` | `{"domain": "...", "bucket": "..."}` | Create mapping |
+| `GET` | `/admin/website-domains/<domain>` | — | Get single mapping |
+| `PUT` | `/admin/website-domains/<domain>` | `{"bucket": "..."}` | Update mapping |
+| `DELETE` | `/admin/website-domains/<domain>` | — | Delete mapping |
+
+### Bucket Website API
+
+| Method | Route | Description |
+|--------|-------|-------------|
+| `PUT` | `/<bucket>?website` | Set website config (XML body) |
+| `GET` | `/<bucket>?website` | Get website config (XML response) |
+| `DELETE` | `/<bucket>?website` | Remove website config |
+
+### Web UI
+
+- **Per-bucket config:** Bucket Details → Properties tab → "Static Website Hosting" card
+- **Domain management:** Sidebar → "Domains" (visible when hosting is enabled and user is admin)