diff --git a/docs.md b/docs.md index 90110bc..7576a62 100644 --- a/docs.md +++ b/docs.md @@ -139,6 +139,7 @@ All configuration is done via environment variables. The table below lists every | `API_BASE_URL` | `http://127.0.0.1:5000` | Internal S3 API URL used by the web UI proxy. Also used for presigned URL generation. Set to your public URL if running behind a reverse proxy. | | `AWS_REGION` | `us-east-1` | Region embedded in SigV4 credential scope. | | `AWS_SERVICE` | `s3` | Service string for SigV4. | +| `DISPLAY_TIMEZONE` | `UTC` | Timezone for timestamps in the web UI (e.g., `US/Eastern`, `Asia/Tokyo`). | ### IAM & Security @@ -170,6 +171,7 @@ All configuration is done via environment variables. The table below lists every | `RATE_LIMIT_BUCKET_OPS` | `120 per minute` | Rate limit for bucket operations (PUT/DELETE/GET/POST on `/`). | | `RATE_LIMIT_OBJECT_OPS` | `240 per minute` | Rate limit for object operations (PUT/GET/DELETE/POST on `//`). | | `RATE_LIMIT_HEAD_OPS` | `100 per minute` | Rate limit for HEAD requests (bucket and object). | +| `RATE_LIMIT_ADMIN` | `60 per minute` | Rate limit for admin API endpoints (`/admin/*`). | | `RATE_LIMIT_STORAGE_URI` | `memory://` | Storage backend for rate limits. Use `redis://host:port` for distributed setups. | ### Server Configuration @@ -256,6 +258,12 @@ Once enabled, configure lifecycle rules via: | `MULTIPART_MIN_PART_SIZE` | `5242880` (5 MB) | Minimum part size for multipart uploads. | | `BUCKET_STATS_CACHE_TTL` | `60` | Seconds to cache bucket statistics. | | `BULK_DELETE_MAX_KEYS` | `500` | Maximum keys per bulk delete request. | +| `BULK_DOWNLOAD_MAX_BYTES` | `1073741824` (1 GiB) | Maximum total size for bulk ZIP downloads. | +| `OBJECT_CACHE_TTL` | `60` | Seconds to cache object metadata. | + +#### Gzip Compression + +API responses for JSON, XML, HTML, CSS, and JavaScript are automatically gzip-compressed when the client sends `Accept-Encoding: gzip`. Compression activates for responses larger than 500 bytes and is handled by a WSGI middleware (`app/compression.py`). Binary object downloads and streaming responses are never compressed. No configuration is needed. ### Server Settings @@ -285,6 +293,12 @@ If running behind a reverse proxy (e.g., Nginx, Cloudflare, or a tunnel), ensure The application automatically trusts these headers to generate correct presigned URLs (e.g., `https://s3.example.com/...` instead of `http://127.0.0.1:5000/...`). Alternatively, you can explicitly set `API_BASE_URL` to your public endpoint. +| Variable | Default | Notes | +| --- | --- | --- | +| `NUM_TRUSTED_PROXIES` | `1` | Number of trusted reverse proxies for `X-Forwarded-*` header processing. | +| `ALLOWED_REDIRECT_HOSTS` | `""` | Comma-separated whitelist of safe redirect targets. Empty allows only same-host redirects. | +| `ALLOW_INTERNAL_ENDPOINTS` | `false` | Allow connections to internal/private IPs for webhooks and replication targets. **Keep disabled in production unless needed.** | + ## 4. Upgrading and Updates ### Version Checking @@ -912,7 +926,7 @@ Objects with forward slashes (`/`) in their keys are displayed as a folder hiera - Select multiple objects using checkboxes - **Bulk Delete**: Delete multiple objects at once -- **Bulk Download**: Download selected objects as individual files +- **Bulk Download**: Download selected objects as a single ZIP archive (up to `BULK_DOWNLOAD_MAX_BYTES`, default 1 GiB) #### Search & Filter @@ -985,6 +999,7 @@ MyFSIO supports **server-side encryption at rest** to protect your data. When en |------|-------------| | **AES-256 (SSE-S3)** | Server-managed encryption using a local master key | | **KMS (SSE-KMS)** | Encryption using customer-managed keys via the built-in KMS | +| **SSE-C** | Server-side encryption with customer-provided keys (per-request) | ### Enabling Encryption @@ -1083,6 +1098,44 @@ encrypted, metadata = ClientEncryptionHelper.encrypt_for_upload(plaintext, key) decrypted = ClientEncryptionHelper.decrypt_from_download(encrypted, metadata, key) ``` +### SSE-C (Customer-Provided Keys) + +With SSE-C, you provide your own 256-bit AES encryption key with each request. The server encrypts/decrypts using your key but never stores it. You must supply the same key for both upload and download. + +**Required headers:** + +| Header | Value | +|--------|-------| +| `x-amz-server-side-encryption-customer-algorithm` | `AES256` | +| `x-amz-server-side-encryption-customer-key` | Base64-encoded 256-bit key | +| `x-amz-server-side-encryption-customer-key-MD5` | Base64-encoded MD5 of the key | + +```bash +# Generate a 256-bit key +KEY=$(openssl rand -base64 32) +KEY_MD5=$(echo -n "$KEY" | base64 -d | openssl dgst -md5 -binary | base64) + +# Upload with SSE-C +curl -X PUT "http://localhost:5000/my-bucket/secret.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-server-side-encryption-customer-algorithm: AES256" \ + -H "x-amz-server-side-encryption-customer-key: $KEY" \ + -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" \ + --data-binary @secret.txt + +# Download with SSE-C (same key required) +curl "http://localhost:5000/my-bucket/secret.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-server-side-encryption-customer-algorithm: AES256" \ + -H "x-amz-server-side-encryption-customer-key: $KEY" \ + -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" +``` + +**Key points:** +- SSE-C does not require `ENCRYPTION_ENABLED` or `KMS_ENABLED` — the key is provided per-request +- If you lose your key, the data is irrecoverable +- The MD5 header is optional but recommended for integrity verification + ### Important Notes - **Existing objects are NOT encrypted** - Only new uploads after enabling encryption are encrypted @@ -1959,6 +2012,20 @@ curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ -H "x-amz-meta-newkey: newvalue" ``` +### MoveObject (UI) + +Move an object to a different key or bucket. This is a UI-only convenience operation that performs a copy followed by a delete of the source. Requires `read` and `delete` on the source, and `write` on the destination. + +```bash +# Move via UI API +curl -X POST "http://localhost:5100/ui/buckets/my-bucket/objects/old-path/file.txt/move" \ + -H "Content-Type: application/json" \ + --cookie "session=..." \ + -d '{"dest_bucket": "other-bucket", "dest_key": "new-path/file.txt"}' +``` + +The move is atomic from the caller's perspective: if the copy succeeds but the delete fails, the object exists in both locations (no data loss). + ### UploadPartCopy Copy data from an existing object into a multipart upload part: diff --git a/templates/docs.html b/templates/docs.html index 9ab9dcf..2faf377 100644 --- a/templates/docs.html +++ b/templates/docs.html @@ -52,6 +52,11 @@
  • Access Control Lists
  • Object & Bucket Tagging
  • Static Website Hosting
    • +
  • CORS Configuration
    • +
  • PostObject (Form Upload)
    • +
  • List Objects API v2
    • +
  • Upgrading & Updates
    • +
  • Full API Reference
    • @@ -126,6 +131,11 @@ python run.py --mode ui 5000 Listen port (UI uses 5100). + + DISPLAY_TIMEZONE + UTC + Timezone for UI timestamps (e.g., US/Eastern, Asia/Tokyo). + CORS Settings @@ -187,6 +197,11 @@ python run.py --mode ui 100 per minute Rate limit for HEAD requests. + + RATE_LIMIT_ADMIN + 60 per minute + Rate limit for admin API endpoints (/admin/*). + Server Settings @@ -338,6 +353,24 @@ python run.py --mode ui 604800 Maximum presigned URL expiry time (7 days). + + Proxy & Network Settings + + + NUM_TRUSTED_PROXIES + 1 + Number of trusted reverse proxies for X-Forwarded-* headers. + + + ALLOWED_REDIRECT_HOSTS + (empty) + Comma-separated whitelist of safe redirect targets. + + + ALLOW_INTERNAL_ENDPOINTS + false + Allow connections to internal/private IPs (webhooks, replication). + Storage Limits @@ -366,6 +399,16 @@ python run.py --mode ui 50 Max lifecycle history records per bucket. + + OBJECT_CACHE_TTL + 60 + Seconds to cache object metadata. + + + BULK_DOWNLOAD_MAX_BYTES + 1 GB + Max total size for bulk ZIP downloads. + ENCRYPTION_CHUNK_SIZE_BYTES 65536 @@ -491,7 +534,7 @@ sudo journalctl -u myfsio -f # View logs
    • Navigate folder hierarchies using breadcrumbs. Objects with / in keys display as folders.
    • Infinite scroll loads more objects automatically. Choose batch size (50–250) from the footer dropdown.
      • -
    • Bulk select objects for multi-delete or multi-download. Filter by name using the search box.
      • +
    • Bulk select objects for multi-delete or multi-download (ZIP archive, up to 1 GiB). Filter by name using the search box.
    • If loading fails, click Retry to attempt again—no page refresh needed.
    @@ -613,15 +656,75 @@ curl -X PUT {{ api_base }}/demo/notes.txt \ /<bucket>/<key> Delete an object. + + HEAD + /<bucket> + Check if a bucket exists. + + + HEAD + /<bucket>/<key> + Get object metadata without downloading. + + + POST + /<bucket>?delete + Bulk delete objects (XML body). + GET/PUT/DELETE /<bucket>?policy - Fetch, upsert, or remove a bucket policy (S3-compatible). + Bucket policy management. + + + GET/PUT + /<bucket>?versioning + Versioning status. + + + GET/PUT/DELETE + /<bucket>?lifecycle + Lifecycle rules. + + + GET/PUT/DELETE + /<bucket>?cors + CORS configuration. + + + GET/PUT/DELETE + /<bucket>?encryption + Default encryption. + + + GET/PUT + /<bucket>?acl + Bucket ACL. + + + GET/PUT/DELETE + /<bucket>?tagging + Bucket tags. + + + GET/PUT/DELETE + /<bucket>/<key>?tagging + Object tags. + + + POST + /<bucket>/<key>?uploads + Initiate multipart upload. + + + POST + /<bucket>/<key>?select + SQL query (SelectObjectContent). -

    All responses include X-Request-Id for tracing. Logs land in logs/api.log and logs/ui.log.

    +

    All responses include X-Request-Id for tracing. See the Full API Reference for the complete endpoint list. Logs land in logs/api.log and logs/ui.log.

    @@ -1311,6 +1414,10 @@ curl -X PUT "{{ api_base }}/bucket/<bucket>?quota" \ KMS (SSE-KMS) Encryption using customer-managed keys via the built-in KMS + + SSE-C + Server-side encryption with customer-provided keys (per-request) + @@ -1377,6 +1484,54 @@ curl -X DELETE "{{ api_base }}/kms/keys/{key-id}?waiting_period_days=30" \

    Envelope Encryption: Each object is encrypted with a unique Data Encryption Key (DEK). The DEK is then encrypted (wrapped) by the master key or KMS key and stored alongside the ciphertext. On read, the DEK is unwrapped and used to decrypt the object transparently.

    + +

    SSE-C (Customer-Provided Keys)

    +

    With SSE-C, you supply your own 256-bit AES key with each request. The server encrypts/decrypts using your key but never stores it. You must provide the same key for both upload and download.

    +
    + + + + + + + + + + + + + + + + + + + + + +
    HeaderValue
    x-amz-server-side-encryption-customer-algorithmAES256
    x-amz-server-side-encryption-customer-keyBase64-encoded 256-bit key
    x-amz-server-side-encryption-customer-key-MD5Base64-encoded MD5 of the key
    +
    +
    # Generate a 256-bit key
+KEY=$(openssl rand -base64 32)
+KEY_MD5=$(echo -n "$KEY" | base64 -d | openssl dgst -md5 -binary | base64)
+
+# Upload with SSE-C
+curl -X PUT "{{ api_base }}/my-bucket/secret.txt" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-server-side-encryption-customer-algorithm: AES256" \
+  -H "x-amz-server-side-encryption-customer-key: $KEY" \
+  -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5" \
+  --data-binary @secret.txt
+
+# Download with SSE-C (same key required)
+curl "{{ api_base }}/my-bucket/secret.txt" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-server-side-encryption-customer-algorithm: AES256" \
+  -H "x-amz-server-side-encryption-customer-key: $KEY" \
+  -H "x-amz-server-side-encryption-customer-key-MD5: $KEY_MD5"
    +
    + Note: SSE-C does not require ENCRYPTION_ENABLED or KMS_ENABLED. If you lose your key, the data is irrecoverable. +
    @@ -1926,7 +2081,7 @@ curl -X POST "{{ api_base }}/<bucket>/data.csv?select" \ 22

    Advanced S3 Operations

    -

    Copy objects, upload part copies, and use range requests for partial downloads.

    +

    Copy, move, and partially download objects using advanced S3 operations.

    CopyObject

    # Copy within same bucket
@@ -1941,6 +2096,13 @@ curl -X PUT "{{ api_base }}/<bucket>/file.txt" \
   -H "x-amz-metadata-directive: REPLACE" \
   -H "x-amz-meta-newkey: newvalue"
    +

    MoveObject (UI)

    +

    Move an object to a different key or bucket via the UI. Performs a copy then deletes the source. Requires read+delete on source and write on destination.

    +
    # Move via UI API (session-authenticated)
+curl -X POST "http://localhost:5100/ui/buckets/<bucket>/objects/<key>/move" \
+  -H "Content-Type: application/json" --cookie "session=..." \
+  -d '{"dest_bucket": "other-bucket", "dest_key": "new-path/file.txt"}'
    +

    UploadPartCopy

    Copy data from an existing object into a multipart upload part:

    # Copy bytes 0-10485759 from source as part 1
@@ -2193,6 +2355,279 @@ server {
    +
    +
    +
    + 26 +

    CORS Configuration

    +
    +

    Configure per-bucket Cross-Origin Resource Sharing rules to control which origins can access your bucket from a browser.

    + +

    Setting CORS Rules

    +
    # Set CORS configuration
+curl -X PUT "{{ api_base }}/<bucket>?cors" \
+  -H "Content-Type: application/xml" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '<CORSConfiguration>
+  <CORSRule>
+    <AllowedOrigin>https://example.com</AllowedOrigin>
+    <AllowedMethod>GET</AllowedMethod>
+    <AllowedMethod>PUT</AllowedMethod>
+    <AllowedHeader>*</AllowedHeader>
+    <ExposeHeader>ETag</ExposeHeader>
+    <MaxAgeSeconds>3600</MaxAgeSeconds>
+  </CORSRule>
+</CORSConfiguration>'
+
+# Get CORS configuration
+curl "{{ api_base }}/<bucket>?cors" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
+
+# Delete CORS configuration
+curl -X DELETE "{{ api_base }}/<bucket>?cors" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
    + +

    Rule Fields

    +
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    FieldDescription
    AllowedOriginOrigins allowed to make requests (supports * wildcard)
    AllowedMethodHTTP methods: GET, PUT, POST, DELETE, HEAD
    AllowedHeaderRequest headers allowed in preflight (supports *)
    ExposeHeaderResponse headers visible to the browser (e.g., ETag, x-amz-request-id)
    MaxAgeSecondsHow long the browser caches preflight results
    +
    +
    +
    +
    +
    +
    + 27 +

    PostObject (HTML Form Upload)

    +
    +

    Upload objects directly from an HTML form using browser-based POST uploads with policy-based authorization.

    + +

    Form Fields

    +
    + + + + + + + + + + + + + + + + + + +
    FieldDescription
    keyObject key (supports ${filename} variable)
    fileThe file to upload
    policyBase64-encoded policy document (JSON)
    x-amz-signatureHMAC-SHA256 signature of the policy
    x-amz-credentialAccess key / date / region / s3 / aws4_request
    x-amz-algorithmAWS4-HMAC-SHA256
    x-amz-dateISO 8601 date (e.g., 20250101T000000Z)
    Content-TypeMIME type of the uploaded file
    x-amz-meta-*Custom metadata headers
    +
    + +

    Simple Upload (No Signing)

    +
    <form action="{{ api_base }}/my-bucket" method="POST" enctype="multipart/form-data">
+  <input type="hidden" name="key" value="uploads/${filename}">
+  <input type="file" name="file">
+  <button type="submit">Upload</button>
+</form>
    + +

    Signed Upload (With Policy)

    +

    For authenticated uploads, include a base64-encoded policy and SigV4 signature fields. The policy constrains allowed keys, content types, and size limits. See docs.md Section 20 for full signing examples.

    +
    +
    +
    +
    +
    + 28 +

    List Objects API v2

    +
    +

    Use the v2 list API for improved pagination with continuation tokens instead of markers.

    + +

    Usage

    +
    # List with v2 API
+curl "{{ api_base }}/<bucket>?list-type=2&prefix=logs/&delimiter=/&max-keys=100" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
+
+# Paginate with continuation token
+curl "{{ api_base }}/<bucket>?list-type=2&continuation-token=<token>" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
+
+# Start listing after a specific key
+curl "{{ api_base }}/<bucket>?list-type=2&start-after=photos/2025/" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
    + +

    Query Parameters

    +
    + + + + + + + + + + + + + + + + + +
    ParameterDescription
    list-type=2Enables v2 API (required)
    prefixFilter to keys starting with this prefix
    delimiterGroup keys by delimiter (typically / for folders)
    max-keysMaximum objects to return (default 1000)
    continuation-tokenToken from previous response for pagination
    start-afterStart listing after this key (first page only)
    fetch-ownerInclude owner info in response
    encoding-typeSet to url to URL-encode keys in response
    +
    +
    +
    +
    +
    +
    + 29 +

    Upgrading & Updates

    +
    +

    How to safely update MyFSIO to a new version.

    + +

    Pre-Update Backup

    +

    Always back up before updating:

    +
    # Back up configuration
+cp -r data/.myfsio.sys/config/ config-backup/
+
+# Back up data (optional, for critical deployments)
+tar czf myfsio-backup-$(date +%Y%m%d).tar.gz data/
+
+# Back up logs
+cp -r logs/ logs-backup/
    + +

    Update Procedure

    +
      +
    1. Stop the service: sudo systemctl stop myfsio (or kill the process)
      2. +
    2. Pull new version: git pull origin main or download the new binary
      3. +
    3. Install dependencies: pip install -r requirements.txt
      4. +
    4. Validate config: python run.py --check-config
      5. +
    5. Start the service: sudo systemctl start myfsio
      6. +
    6. Verify: curl http://localhost:5000/myfsio/health
      7. +
    + +

    Docker Update

    +
    docker pull myfsio:latest
+docker stop myfsio && docker rm myfsio
+docker run -d --name myfsio -v myfsio-data:/app/data -p 5000:5000 -p 5100:5100 myfsio:latest
    + +

    Rollback

    +

    If something goes wrong, stop the service, restore the backed-up config and data directories, then restart with the previous binary or code version. See docs.md Section 4 for detailed rollback procedures including blue-green deployment strategies.

    +
    +
    +
    +
    +
    + 30 +

    Full API Reference

    +
    +

    Complete list of all S3-compatible, admin, and KMS endpoints.

    +
    # Service
+GET    /myfsio/health                   # Health check
+
+# Bucket Operations
+GET    /                               # List buckets
+PUT    /<bucket>                        # Create bucket
+DELETE /<bucket>                        # Delete bucket
+GET    /<bucket>                        # List objects (?list-type=2)
+HEAD   /<bucket>                        # Check bucket exists
+POST   /<bucket>                        # POST object / form upload
+POST   /<bucket>?delete                 # Bulk delete
+
+# Bucket Configuration
+GET|PUT|DELETE /<bucket>?policy          # Bucket policy
+GET|PUT        /<bucket>?quota           # Bucket quota
+GET|PUT        /<bucket>?versioning      # Versioning
+GET|PUT|DELETE /<bucket>?lifecycle        # Lifecycle rules
+GET|PUT|DELETE /<bucket>?cors             # CORS config
+GET|PUT|DELETE /<bucket>?encryption       # Default encryption
+GET|PUT        /<bucket>?acl              # Bucket ACL
+GET|PUT|DELETE /<bucket>?tagging          # Bucket tags
+GET|PUT|DELETE /<bucket>?replication      # Replication rules
+GET|PUT        /<bucket>?logging          # Access logging
+GET|PUT        /<bucket>?notification     # Event notifications
+GET|PUT        /<bucket>?object-lock      # Object lock config
+GET|PUT|DELETE /<bucket>?website          # Static website
+GET            /<bucket>?uploads          # List multipart uploads
+GET            /<bucket>?versions         # List object versions
+GET            /<bucket>?location         # Bucket region
+
+# Object Operations
+PUT    /<bucket>/<key>                  # Upload object
+GET    /<bucket>/<key>                  # Download (Range supported)
+DELETE /<bucket>/<key>                  # Delete object
+HEAD   /<bucket>/<key>                  # Object metadata
+POST   /<bucket>/<key>?select           # SQL query (SelectObjectContent)
+
+# Object Configuration
+GET|PUT|DELETE /<bucket>/<key>?tagging    # Object tags
+GET|PUT        /<bucket>/<key>?acl        # Object ACL
+GET|PUT        /<bucket>/<key>?retention  # Object retention
+GET|PUT        /<bucket>/<key>?legal-hold # Legal hold
+
+# Multipart Upload
+POST   /<bucket>/<key>?uploads          # Initiate
+PUT    /<bucket>/<key>?uploadId=X&partNumber=N  # Upload part
+POST   /<bucket>/<key>?uploadId=X       # Complete
+DELETE /<bucket>/<key>?uploadId=X       # Abort
+GET    /<bucket>/<key>?uploadId=X       # List parts
+
+# Copy (via x-amz-copy-source header)
+PUT    /<bucket>/<key>                  # CopyObject
+PUT    /<bucket>/<key>?uploadId&partNumber # UploadPartCopy
+
+# Admin API
+GET|PUT /admin/site                     # Local site config
+GET     /admin/sites                    # List peers
+POST    /admin/sites                    # Register peer
+GET|PUT|DELETE /admin/sites/<id>        # Manage peer
+GET     /admin/sites/<id>/health        # Peer health
+GET     /admin/topology                 # Cluster topology
+GET|POST|PUT|DELETE /admin/website-domains  # Domain mappings
+
+# KMS API
+GET|POST /kms/keys                      # List / Create keys
+GET|DELETE /kms/keys/<id>               # Get / Delete key
+POST   /kms/keys/<id>/enable            # Enable key
+POST   /kms/keys/<id>/disable           # Disable key
+POST   /kms/keys/<id>/rotate            # Rotate key
+POST   /kms/encrypt                     # Encrypt data
+POST   /kms/decrypt                     # Decrypt data
+POST   /kms/generate-data-key           # Generate data key
+POST   /kms/generate-random             # Generate random bytes
    +
    +