diff --git a/home.md b/home.md index ef9bc99..4be6e1f 100644 --- a/home.md +++ b/home.md @@ -7,7 +7,7 @@ This document expands on the README to describe the full workflow for running, c MyFSIO ships two Flask entrypoints that share the same storage, IAM, and bucket-policy state: - **API server** – Implements the S3-compatible REST API, policy evaluation, and Signature Version 4 presign service. -- **UI server** – Provides the browser console for buckets, IAM, and policies. It proxies to the API for presign operations. +- **UI server** – Provides the browser console for buckets, IAM, and policies. It proxies all storage operations through the S3 API via boto3 (SigV4-signed), mirroring the architecture used by MinIO and Garage. Both servers read `AppConfig`, so editing JSON stores on disk instantly affects both surfaces. @@ -136,9 +136,10 @@ All configuration is done via environment variables. The table below lists every | `MAX_UPLOAD_SIZE` | `1073741824` (1 GiB) | Bytes. Caps incoming uploads in both API + UI. | | `UI_PAGE_SIZE` | `100` | `MaxKeys` hint shown in listings. | | `SECRET_KEY` | Auto-generated | Flask session key. Auto-generates and persists if not set. **Set explicitly in production.** | -| `API_BASE_URL` | `None` | Public URL for presigned URLs. Required behind proxies. | +| `API_BASE_URL` | `http://127.0.0.1:5000` | Internal S3 API URL used by the web UI proxy. Also used for presigned URL generation. Set to your public URL if running behind a reverse proxy. | | `AWS_REGION` | `us-east-1` | Region embedded in SigV4 credential scope. | | `AWS_SERVICE` | `s3` | Service string for SigV4. | +| `DISPLAY_TIMEZONE` | `UTC` | Timezone for timestamps in the web UI (e.g., `US/Eastern`, `Asia/Tokyo`). | ### IAM & Security @@ -170,6 +171,7 @@ All configuration is done via environment variables. The table below lists every | `RATE_LIMIT_BUCKET_OPS` | `120 per minute` | Rate limit for bucket operations (PUT/DELETE/GET/POST on `/`). | | `RATE_LIMIT_OBJECT_OPS` | `240 per minute` | Rate limit for object operations (PUT/GET/DELETE/POST on `//`). | | `RATE_LIMIT_HEAD_OPS` | `100 per minute` | Rate limit for HEAD requests (bucket and object). | +| `RATE_LIMIT_ADMIN` | `60 per minute` | Rate limit for admin API endpoints (`/admin/*`). | | `RATE_LIMIT_STORAGE_URI` | `memory://` | Storage backend for rate limits. Use `redis://host:port` for distributed setups. | ### Server Configuration @@ -256,6 +258,12 @@ Once enabled, configure lifecycle rules via: | `MULTIPART_MIN_PART_SIZE` | `5242880` (5 MB) | Minimum part size for multipart uploads. | | `BUCKET_STATS_CACHE_TTL` | `60` | Seconds to cache bucket statistics. | | `BULK_DELETE_MAX_KEYS` | `500` | Maximum keys per bulk delete request. | +| `BULK_DOWNLOAD_MAX_BYTES` | `1073741824` (1 GiB) | Maximum total size for bulk ZIP downloads. | +| `OBJECT_CACHE_TTL` | `60` | Seconds to cache object metadata. | + +#### Gzip Compression + +API responses for JSON, XML, HTML, CSS, and JavaScript are automatically gzip-compressed when the client sends `Accept-Encoding: gzip`. Compression activates for responses larger than 500 bytes and is handled by a WSGI middleware (`app/compression.py`). Binary object downloads and streaming responses are never compressed. No configuration is needed. ### Server Settings @@ -285,6 +293,12 @@ If running behind a reverse proxy (e.g., Nginx, Cloudflare, or a tunnel), ensure The application automatically trusts these headers to generate correct presigned URLs (e.g., `https://s3.example.com/...` instead of `http://127.0.0.1:5000/...`). Alternatively, you can explicitly set `API_BASE_URL` to your public endpoint. +| Variable | Default | Notes | +| --- | --- | --- | +| `NUM_TRUSTED_PROXIES` | `1` | Number of trusted reverse proxies for `X-Forwarded-*` header processing. | +| `ALLOWED_REDIRECT_HOSTS` | `""` | Comma-separated whitelist of safe redirect targets. Empty allows only same-host redirects. | +| `ALLOW_INTERNAL_ENDPOINTS` | `false` | Allow connections to internal/private IPs for webhooks and replication targets. **Keep disabled in production unless needed.** | + ## 4. Upgrading and Updates ### Version Checking @@ -619,13 +633,15 @@ MyFSIO implements a comprehensive Identity and Access Management (IAM) system th ### Getting Started -1. On first boot, `data/.myfsio.sys/config/iam.json` is seeded with `localadmin / localadmin` that has wildcard access. -2. Sign into the UI using those credentials, then open **IAM**: +1. On first boot, `data/.myfsio.sys/config/iam.json` is created with a randomly generated admin user. The access key and secret key are printed to the console during first startup. If you miss it, check the `iam.json` file directly—credentials are stored in plaintext. +2. Sign into the UI using the generated credentials, then open **IAM**: - **Create user**: supply a display name and optional JSON inline policy array. - **Rotate secret**: generates a new secret key; the UI surfaces it once. - **Policy editor**: select a user, paste an array of objects (`{"bucket": "*", "actions": ["list", "read"]}`), and submit. Alias support includes AWS-style verbs (e.g., `s3:GetObject`). 3. Wildcard action `iam:*` is supported for admin user definitions. +> **Breaking Change (v0.2.0+):** Previous versions used fixed default credentials (`localadmin/localadmin`). If upgrading from an older version, your existing credentials remain unchanged, but new installations will generate random credentials. + ### Authentication The API expects every request to include authentication headers. The UI persists them in the Flask session after login. @@ -910,7 +926,7 @@ Objects with forward slashes (`/`) in their keys are displayed as a folder hiera - Select multiple objects using checkboxes - **Bulk Delete**: Delete multiple objects at once -- **Bulk Download**: Download selected objects as individual files +- **Bulk Download**: Download selected objects as a single ZIP archive (up to `BULK_DOWNLOAD_MAX_BYTES`, default 1 GiB) #### Search & Filter @@ -983,6 +999,7 @@ MyFSIO supports **server-side encryption at rest** to protect your data. When en |------|-------------| | **AES-256 (SSE-S3)** | Server-managed encryption using a local master key | | **KMS (SSE-KMS)** | Encryption using customer-managed keys via the built-in KMS | +| **SSE-C** | Server-side encryption with customer-provided keys (per-request) | ### Enabling Encryption @@ -1081,6 +1098,44 @@ encrypted, metadata = ClientEncryptionHelper.encrypt_for_upload(plaintext, key) decrypted = ClientEncryptionHelper.decrypt_from_download(encrypted, metadata, key) ``` +### SSE-C (Customer-Provided Keys) + +With SSE-C, you provide your own 256-bit AES encryption key with each request. The server encrypts/decrypts using your key but never stores it. You must supply the same key for both upload and download. + +**Required headers:** + +| Header | Value | +|--------|-------| +| `x-amz-server-side-encryption-customer-algorithm` | `AES256` | +| `x-amz-server-side-encryption-customer-key` | Base64-encoded 256-bit key | +| `x-amz-server-side-encryption-customer-key-MD5` | Base64-encoded MD5 of the key | + +```bash +# Generate a 256-bit key +KEY=$(openssl rand -base64 32) +KEY_MD5=$(echo -n "$KEY" | base64 -d | openssl dgst -md5 -binary | base64) + +# Upload with SSE-C +curl -X PUT "http://localhost:5000/my-bucket/secret.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-server-side-encryption-customer-algorithm: AES256" \ + -H "x-amz-server-side-encryption-customer-key: $KEY" \ + -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" \ + --data-binary @secret.txt + +# Download with SSE-C (same key required) +curl "http://localhost:5000/my-bucket/secret.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-server-side-encryption-customer-algorithm: AES256" \ + -H "x-amz-server-side-encryption-customer-key: $KEY" \ + -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" +``` + +**Key points:** +- SSE-C does not require `ENCRYPTION_ENABLED` or `KMS_ENABLED` — the key is provided per-request +- If you lose your key, the data is irrecoverable +- The MD5 header is optional but recommended for integrity verification + ### Important Notes - **Existing objects are NOT encrypted** - Only new uploads after enabling encryption are encrypted @@ -1550,6 +1605,9 @@ GET /?notification # Get event notifications PUT /?notification # Set event notifications (webhooks) GET /?object-lock # Get object lock configuration PUT /?object-lock # Set object lock configuration +GET /?website # Get website configuration +PUT /?website # Set website configuration +DELETE /?website # Delete website configuration GET /?uploads # List active multipart uploads GET /?versions # List object versions GET /?location # Get bucket location/region @@ -1594,6 +1652,11 @@ PUT /admin/sites/ # Update peer site DELETE /admin/sites/ # Unregister peer site GET /admin/sites//health # Check peer health GET /admin/topology # Get cluster topology +GET /admin/website-domains # List domain mappings +POST /admin/website-domains # Create domain mapping +GET /admin/website-domains/ # Get domain mapping +PUT /admin/website-domains/ # Update domain mapping +DELETE /admin/website-domains/ # Delete domain mapping # KMS API GET /kms/keys # List KMS keys @@ -1949,6 +2012,20 @@ curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ -H "x-amz-meta-newkey: newvalue" ``` +### MoveObject (UI) + +Move an object to a different key or bucket. This is a UI-only convenience operation that performs a copy followed by a delete of the source. Requires `read` and `delete` on the source, and `write` on the destination. + +```bash +# Move via UI API +curl -X POST "http://localhost:5100/ui/buckets/my-bucket/objects/old-path/file.txt/move" \ + -H "Content-Type: application/json" \ + --cookie "session=..." \ + -d '{"dest_bucket": "other-bucket", "dest_key": "new-path/file.txt"}' +``` + +The move is atomic from the caller's perspective: if the copy succeeds but the delete fails, the object exists in both locations (no data loss). + ### UploadPartCopy Copy data from an existing object into a multipart upload part: @@ -2226,4 +2303,114 @@ curl "http://localhost:5000/my-bucket?list-type=2&start-after=photos/2024/" \ | `continuation-token` | Token from previous response | | `start-after` | Start listing after this key | | `fetch-owner` | Include owner info in response | -| `encoding-type` | Set to `url` for URL-encoded keys \ No newline at end of file +| `encoding-type` | Set to `url` for URL-encoded keys + +## 26. Static Website Hosting + +MyFSIO can serve S3 buckets as static websites via custom domain mappings. When a request arrives with a `Host` header matching a mapped domain, MyFSIO resolves the bucket and serves objects directly. + +### Enabling + +Set the environment variable: + +```bash +WEBSITE_HOSTING_ENABLED=true +``` + +When disabled, all website hosting endpoints return 400 and domain-based serving is skipped. + +### Configuration + +| Variable | Default | Description | +|----------|---------|-------------| +| `WEBSITE_HOSTING_ENABLED` | `false` | Master switch for website hosting | + +### Setting Up a Website + +**Step 1: Configure the bucket website settings** + +```bash +curl -X PUT "http://localhost:5000/my-site?website" \ + -H "Authorization: ..." \ + -d ' + + index.html + 404.html +' +``` + +- `IndexDocument` with `Suffix` is required (must not contain `/`) +- `ErrorDocument` is optional + +**Step 2: Map a domain to the bucket** + +```bash +curl -X POST "http://localhost:5000/admin/website-domains" \ + -H "Authorization: ..." \ + -H "Content-Type: application/json" \ + -d '{"domain": "example.com", "bucket": "my-site"}' +``` + +**Step 3: Point your domain to MyFSIO** + +For HTTP-only (direct access), point DNS to the MyFSIO host on port 5000. + +For HTTPS (recommended), use a reverse proxy. The critical requirement is passing the original `Host` header so MyFSIO can match the domain to a bucket. + +**nginx example:** + +```nginx +server { + server_name example.com; + listen 443 ssl; + + ssl_certificate /etc/ssl/certs/example.com.pem; + ssl_certificate_key /etc/ssl/private/example.com.key; + + location / { + proxy_pass http://127.0.0.1:5000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +`proxy_set_header Host $host;` is required — without it, MyFSIO cannot match the incoming domain to a bucket. You do not need any path-based routing rules; MyFSIO handles all object resolution internally. + +### How Domain Routing Works + +1. A request arrives with `Host: example.com` +2. MyFSIO's `before_request` hook strips the port and looks up the domain in the `WebsiteDomainStore` +3. If a match is found, it loads the bucket's website config (index/error documents) +4. Object key resolution: + - `/` or trailing `/` → append `index_document` (e.g., `index.html`) + - `/path` → try exact match, then try `path/index_document` + - Not found → serve `error_document` with 404 status +5. If no domain match is found, the request falls through to normal S3 API / UI routing + +### Domain Mapping Admin API + +All endpoints require admin (`iam:*`) permissions. + +| Method | Route | Body | Description | +|--------|-------|------|-------------| +| `GET` | `/admin/website-domains` | — | List all mappings | +| `POST` | `/admin/website-domains` | `{"domain": "...", "bucket": "..."}` | Create mapping | +| `GET` | `/admin/website-domains/` | — | Get single mapping | +| `PUT` | `/admin/website-domains/` | `{"bucket": "..."}` | Update mapping | +| `DELETE` | `/admin/website-domains/` | — | Delete mapping | + +### Bucket Website API + +| Method | Route | Description | +|--------|-------|-------------| +| `PUT` | `/?website` | Set website config (XML body) | +| `GET` | `/?website` | Get website config (XML response) | +| `DELETE` | `/?website` | Remove website config | + +### Web UI + +- **Per-bucket config:** Bucket Details → Properties tab → "Static Website Hosting" card +- **Domain management:** Sidebar → "Domains" (visible when hosting is enabled and user is admin) \ No newline at end of file