diff --git a/docs.md b/docs.md index e2d89fa..e582c76 100644 --- a/docs.md +++ b/docs.md @@ -1507,16 +1507,723 @@ The suite covers bucket CRUD, presigned downloads, bucket policy enforcement, an ## 14. API Matrix ``` +# Service Endpoints +GET /myfsio/health # Health check + +# Bucket Operations GET / # List buckets PUT / # Create bucket DELETE / # Remove bucket -GET / # List objects -PUT // # Upload object -GET // # Download object -DELETE // # Delete object -GET /?policy # Fetch policy -PUT /?policy # Upsert policy -DELETE /?policy # Delete policy +GET / # List objects (supports ?list-type=2) +HEAD / # Check bucket exists +POST / # POST object upload (HTML form) +POST /?delete # Bulk delete objects + +# Bucket Configuration +GET /?policy # Fetch bucket policy +PUT /?policy # Upsert bucket policy +DELETE /?policy # Delete bucket policy GET /?quota # Get bucket quota PUT /?quota # Set bucket quota (admin only) +GET /?versioning # Get versioning status +PUT /?versioning # Enable/disable versioning +GET /?lifecycle # Get lifecycle rules +PUT /?lifecycle # Set lifecycle rules +DELETE /?lifecycle # Delete lifecycle rules +GET /?cors # Get CORS configuration +PUT /?cors # Set CORS configuration +DELETE /?cors # Delete CORS configuration +GET /?encryption # Get encryption configuration +PUT /?encryption # Set default encryption +DELETE /?encryption # Delete encryption configuration +GET /?acl # Get bucket ACL +PUT /?acl # Set bucket ACL +GET /?tagging # Get bucket tags +PUT /?tagging # Set bucket tags +DELETE /?tagging # Delete bucket tags +GET /?replication # Get replication configuration +PUT /?replication # Set replication rules +DELETE /?replication # Delete replication configuration +GET /?logging # Get access logging configuration +PUT /?logging # Set access logging +GET /?notification # Get event notifications +PUT /?notification # Set event notifications (webhooks) +GET /?object-lock # Get object lock configuration +PUT /?object-lock # Set object lock configuration +GET /?uploads # List active multipart uploads +GET /?versions # List object versions +GET /?location # Get bucket location/region + +# Object Operations +PUT // # Upload object +GET // # Download object (supports Range header) +DELETE // # Delete object +HEAD // # Get object metadata +POST // # POST upload with policy +POST //?select # SelectObjectContent (SQL query) + +# Object Configuration +GET //?tagging # Get object tags +PUT //?tagging # Set object tags +DELETE //?tagging # Delete object tags +GET //?acl # Get object ACL +PUT //?acl # Set object ACL +PUT //?retention # Set object retention +GET //?retention # Get object retention +PUT //?legal-hold # Set legal hold +GET //?legal-hold # Get legal hold status + +# Multipart Upload +POST //?uploads # Initiate multipart upload +PUT //?uploadId=X&partNumber=N # Upload part +PUT //?uploadId=X&partNumber=N (with x-amz-copy-source) # UploadPartCopy +POST //?uploadId=X # Complete multipart upload +DELETE //?uploadId=X # Abort multipart upload +GET //?uploadId=X # List parts + +# Copy Operations +PUT // (with x-amz-copy-source header) # CopyObject + +# Admin API +GET /admin/site # Get local site info +PUT /admin/site # Update local site +GET /admin/sites # List peer sites +POST /admin/sites # Register peer site +GET /admin/sites/ # Get peer site +PUT /admin/sites/ # Update peer site +DELETE /admin/sites/ # Unregister peer site +GET /admin/sites//health # Check peer health +GET /admin/topology # Get cluster topology + +# KMS API +GET /kms/keys # List KMS keys +POST /kms/keys # Create KMS key +GET /kms/keys/ # Get key details +DELETE /kms/keys/ # Schedule key deletion +POST /kms/keys//enable # Enable key +POST /kms/keys//disable # Disable key +POST /kms/keys//rotate # Rotate key material +POST /kms/encrypt # Encrypt data +POST /kms/decrypt # Decrypt data +POST /kms/generate-data-key # Generate data key +POST /kms/generate-random # Generate random bytes ``` + +## 15. Health Check Endpoint + +The API exposes a simple health check endpoint for monitoring and load balancer integration: + +```bash +# Check API health +curl http://localhost:5000/myfsio/health + +# Response +{"status": "ok", "version": "0.1.7"} +``` + +The response includes: +- `status`: Always `"ok"` when the server is running +- `version`: Current application version from `app/version.py` + +Use this endpoint for: +- Load balancer health checks +- Kubernetes liveness/readiness probes +- Monitoring system integration (Prometheus, Datadog, etc.) + +## 16. Object Lock & Retention + +Object Lock prevents objects from being deleted or overwritten for a specified retention period. MyFSIO supports both GOVERNANCE and COMPLIANCE modes. + +### Retention Modes + +| Mode | Description | +|------|-------------| +| **GOVERNANCE** | Objects can't be deleted by normal users, but users with `s3:BypassGovernanceRetention` permission can override | +| **COMPLIANCE** | Objects can't be deleted or overwritten by anyone, including root, until the retention period expires | + +### Enabling Object Lock + +Object Lock must be enabled when creating a bucket: + +```bash +# Create bucket with Object Lock enabled +curl -X PUT "http://localhost:5000/my-bucket" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-bucket-object-lock-enabled: true" + +# Set default retention configuration +curl -X PUT "http://localhost:5000/my-bucket?object-lock" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "ObjectLockEnabled": "Enabled", + "Rule": { + "DefaultRetention": { + "Mode": "GOVERNANCE", + "Days": 30 + } + } + }' +``` + +### Per-Object Retention + +Set retention on individual objects: + +```bash +# Set object retention +curl -X PUT "http://localhost:5000/my-bucket/important.pdf?retention" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "Mode": "COMPLIANCE", + "RetainUntilDate": "2025-12-31T23:59:59Z" + }' + +# Get object retention +curl "http://localhost:5000/my-bucket/important.pdf?retention" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +``` + +### Legal Hold + +Legal hold provides indefinite protection independent of retention settings: + +```bash +# Enable legal hold +curl -X PUT "http://localhost:5000/my-bucket/document.pdf?legal-hold" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{"Status": "ON"}' + +# Disable legal hold +curl -X PUT "http://localhost:5000/my-bucket/document.pdf?legal-hold" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{"Status": "OFF"}' + +# Check legal hold status +curl "http://localhost:5000/my-bucket/document.pdf?legal-hold" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +``` + +## 17. Access Logging + +Enable S3-style access logging to track all requests to your buckets. + +### Configuration + +```bash +# Enable access logging +curl -X PUT "http://localhost:5000/my-bucket?logging" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "LoggingEnabled": { + "TargetBucket": "log-bucket", + "TargetPrefix": "logs/my-bucket/" + } + }' + +# Get logging configuration +curl "http://localhost:5000/my-bucket?logging" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Disable logging (empty configuration) +curl -X PUT "http://localhost:5000/my-bucket?logging" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{}' +``` + +### Log Format + +Access logs are written in S3-compatible format with fields including: +- Timestamp, bucket, key +- Operation (REST.GET.OBJECT, REST.PUT.OBJECT, etc.) +- Request ID, requester, source IP +- HTTP status, error code, bytes sent +- Total time, turn-around time +- Referrer, User-Agent + +## 18. Bucket Notifications & Webhooks + +Configure event notifications to trigger webhooks when objects are created or deleted. + +### Supported Events + +| Event Type | Description | +|-----------|-------------| +| `s3:ObjectCreated:*` | Any object creation (PUT, POST, COPY, multipart) | +| `s3:ObjectCreated:Put` | Object created via PUT | +| `s3:ObjectCreated:Post` | Object created via POST | +| `s3:ObjectCreated:Copy` | Object created via COPY | +| `s3:ObjectCreated:CompleteMultipartUpload` | Multipart upload completed | +| `s3:ObjectRemoved:*` | Any object deletion | +| `s3:ObjectRemoved:Delete` | Object deleted | +| `s3:ObjectRemoved:DeleteMarkerCreated` | Delete marker created (versioned bucket) | + +### Configuration + +```bash +# Set notification configuration +curl -X PUT "http://localhost:5000/my-bucket?notification" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "TopicConfigurations": [ + { + "Id": "upload-notify", + "TopicArn": "https://webhook.example.com/s3-events", + "Events": ["s3:ObjectCreated:*"], + "Filter": { + "Key": { + "FilterRules": [ + {"Name": "prefix", "Value": "uploads/"}, + {"Name": "suffix", "Value": ".jpg"} + ] + } + } + } + ] + }' + +# Get notification configuration +curl "http://localhost:5000/my-bucket?notification" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +``` + +### Webhook Payload + +The webhook receives a JSON payload similar to AWS S3 event notifications: + +```json +{ + "Records": [ + { + "eventVersion": "2.1", + "eventSource": "myfsio:s3", + "eventTime": "2024-01-15T10:30:00.000Z", + "eventName": "ObjectCreated:Put", + "s3": { + "bucket": {"name": "my-bucket"}, + "object": { + "key": "uploads/photo.jpg", + "size": 102400, + "eTag": "abc123..." + } + } + } + ] +} +``` + +### Security Notes + +- Webhook URLs are validated to prevent SSRF attacks +- Internal/private IP ranges are blocked by default +- Use HTTPS endpoints in production + +## 19. SelectObjectContent (SQL Queries) + +Query CSV, JSON, or Parquet files directly using SQL without downloading the entire object. Requires DuckDB to be installed. + +### Prerequisites + +```bash +pip install duckdb +``` + +### Usage + +```bash +# Query a CSV file +curl -X POST "http://localhost:5000/my-bucket/data.csv?select" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "Expression": "SELECT name, age FROM s3object WHERE age > 25", + "ExpressionType": "SQL", + "InputSerialization": { + "CSV": { + "FileHeaderInfo": "USE", + "FieldDelimiter": "," + } + }, + "OutputSerialization": { + "JSON": {} + } + }' + +# Query a JSON file +curl -X POST "http://localhost:5000/my-bucket/data.json?select" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "Expression": "SELECT * FROM s3object s WHERE s.status = '\"active'\"", + "ExpressionType": "SQL", + "InputSerialization": {"JSON": {"Type": "LINES"}}, + "OutputSerialization": {"JSON": {}} + }' +``` + +### Supported Input Formats + +| Format | Options | +|--------|---------| +| **CSV** | `FileHeaderInfo` (USE, IGNORE, NONE), `FieldDelimiter`, `QuoteCharacter`, `RecordDelimiter` | +| **JSON** | `Type` (DOCUMENT, LINES) | +| **Parquet** | Automatic schema detection | + +### Output Formats + +- **JSON**: Returns results as JSON records +- **CSV**: Returns results as CSV + +## 20. PostObject (HTML Form Upload) + +Upload objects using HTML forms with policy-based authorization. Useful for browser-based direct uploads. + +### Form Fields + +| Field | Required | Description | +|-------|----------|-------------| +| `key` | Yes | Object key (can include `${filename}` placeholder) | +| `file` | Yes | The file to upload | +| `policy` | No | Base64-encoded policy document | +| `x-amz-signature` | No | Policy signature | +| `x-amz-credential` | No | Credential scope | +| `x-amz-algorithm` | No | Signing algorithm (AWS4-HMAC-SHA256) | +| `x-amz-date` | No | Request timestamp | +| `Content-Type` | No | MIME type of the file | +| `x-amz-meta-*` | No | Custom metadata | + +### Example HTML Form + +```html +
+ + + + + +
+``` + +### With Policy (Signed Upload) + +For authenticated uploads, include a policy document: + +```bash +# Generate policy and signature using boto3 or similar +# Then include in form: +# - policy: base64(policy_document) +# - x-amz-signature: HMAC-SHA256(policy, signing_key) +# - x-amz-credential: access_key/date/region/s3/aws4_request +# - x-amz-algorithm: AWS4-HMAC-SHA256 +# - x-amz-date: YYYYMMDDTHHMMSSZ +``` + +## 21. Advanced S3 Operations + +### CopyObject + +Copy objects within or between buckets: + +```bash +# Copy within same bucket +curl -X PUT "http://localhost:5000/my-bucket/copy-of-file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-copy-source: /my-bucket/original-file.txt" + +# Copy to different bucket +curl -X PUT "http://localhost:5000/other-bucket/file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-copy-source: /my-bucket/original-file.txt" + +# Copy with metadata replacement +curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-copy-source: /my-bucket/file.txt" \ + -H "x-amz-metadata-directive: REPLACE" \ + -H "x-amz-meta-newkey: newvalue" +``` + +### UploadPartCopy + +Copy data from an existing object into a multipart upload part: + +```bash +# Initiate multipart upload +UPLOAD_ID=$(curl -X POST "http://localhost:5000/my-bucket/large-file.bin?uploads" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." | jq -r '.UploadId') + +# Copy bytes 0-10485759 from source as part 1 +curl -X PUT "http://localhost:5000/my-bucket/large-file.bin?uploadId=$UPLOAD_ID&partNumber=1" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-copy-source: /source-bucket/source-file.bin" \ + -H "x-amz-copy-source-range: bytes=0-10485759" + +# Copy bytes 10485760-20971519 as part 2 +curl -X PUT "http://localhost:5000/my-bucket/large-file.bin?uploadId=$UPLOAD_ID&partNumber=2" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-copy-source: /source-bucket/source-file.bin" \ + -H "x-amz-copy-source-range: bytes=10485760-20971519" +``` + +### Range Requests + +Download partial content using the Range header: + +```bash +# Get first 1000 bytes +curl "http://localhost:5000/my-bucket/large-file.bin" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "Range: bytes=0-999" + +# Get bytes 1000-1999 +curl "http://localhost:5000/my-bucket/large-file.bin" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "Range: bytes=1000-1999" + +# Get last 500 bytes +curl "http://localhost:5000/my-bucket/large-file.bin" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "Range: bytes=-500" + +# Get from byte 5000 to end +curl "http://localhost:5000/my-bucket/large-file.bin" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "Range: bytes=5000-" +``` + +Range responses include: +- HTTP 206 Partial Content status +- `Content-Range` header showing the byte range +- `Accept-Ranges: bytes` header + +### Conditional Requests + +Use conditional headers for cache validation: + +```bash +# Only download if modified since +curl "http://localhost:5000/my-bucket/file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "If-Modified-Since: Wed, 15 Jan 2025 10:00:00 GMT" + +# Only download if ETag doesn't match (changed) +curl "http://localhost:5000/my-bucket/file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "If-None-Match: \"abc123...\"" + +# Only download if ETag matches +curl "http://localhost:5000/my-bucket/file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "If-Match: \"abc123...\"" +``` + +## 22. Access Control Lists (ACLs) + +ACLs provide legacy-style permission management for buckets and objects. + +### Canned ACLs + +| ACL | Description | +|-----|-------------| +| `private` | Owner gets FULL_CONTROL (default) | +| `public-read` | Owner FULL_CONTROL, public READ | +| `public-read-write` | Owner FULL_CONTROL, public READ and WRITE | +| `authenticated-read` | Owner FULL_CONTROL, authenticated users READ | + +### Setting ACLs + +```bash +# Set bucket ACL using canned ACL +curl -X PUT "http://localhost:5000/my-bucket?acl" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-acl: public-read" + +# Set object ACL +curl -X PUT "http://localhost:5000/my-bucket/file.txt?acl" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-acl: private" + +# Set ACL during upload +curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-acl: public-read" \ + --data-binary @file.txt + +# Get bucket ACL +curl "http://localhost:5000/my-bucket?acl" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Get object ACL +curl "http://localhost:5000/my-bucket/file.txt?acl" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +``` + +### ACL vs Bucket Policies + +- **ACLs**: Simple, limited options, legacy approach +- **Bucket Policies**: Powerful, flexible, recommended for new deployments + +For most use cases, prefer bucket policies over ACLs. + +## 23. Object & Bucket Tagging + +Add metadata tags to buckets and objects for organization, cost allocation, or lifecycle rule filtering. + +### Bucket Tagging + +```bash +# Set bucket tags +curl -X PUT "http://localhost:5000/my-bucket?tagging" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "TagSet": [ + {"Key": "Environment", "Value": "Production"}, + {"Key": "Team", "Value": "Engineering"} + ] + }' + +# Get bucket tags +curl "http://localhost:5000/my-bucket?tagging" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Delete bucket tags +curl -X DELETE "http://localhost:5000/my-bucket?tagging" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +``` + +### Object Tagging + +```bash +# Set object tags +curl -X PUT "http://localhost:5000/my-bucket/file.txt?tagging" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "TagSet": [ + {"Key": "Classification", "Value": "Confidential"}, + {"Key": "Owner", "Value": "john@example.com"} + ] + }' + +# Get object tags +curl "http://localhost:5000/my-bucket/file.txt?tagging" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Delete object tags +curl -X DELETE "http://localhost:5000/my-bucket/file.txt?tagging" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Set tags during upload +curl -X PUT "http://localhost:5000/my-bucket/file.txt" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -H "x-amz-tagging: Environment=Staging&Team=QA" \ + --data-binary @file.txt +``` + +### Tagging Limits + +- Maximum 50 tags per object (configurable via `OBJECT_TAG_LIMIT`) +- Tag key: 1-128 Unicode characters +- Tag value: 0-256 Unicode characters + +### Use Cases + +- **Lifecycle Rules**: Filter objects for expiration by tag +- **Access Control**: Use tag conditions in bucket policies +- **Cost Tracking**: Group objects by project or department +- **Automation**: Trigger actions based on object tags + +## 24. CORS Configuration + +Configure Cross-Origin Resource Sharing for browser-based applications. + +### Setting CORS Rules + +```bash +# Set CORS configuration +curl -X PUT "http://localhost:5000/my-bucket?cors" \ + -H "Content-Type: application/json" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." \ + -d '{ + "CORSRules": [ + { + "AllowedOrigins": ["https://example.com", "https://app.example.com"], + "AllowedMethods": ["GET", "PUT", "POST", "DELETE"], + "AllowedHeaders": ["*"], + "ExposeHeaders": ["ETag", "x-amz-meta-*"], + "MaxAgeSeconds": 3600 + } + ] + }' + +# Get CORS configuration +curl "http://localhost:5000/my-bucket?cors" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Delete CORS configuration +curl -X DELETE "http://localhost:5000/my-bucket?cors" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +``` + +### CORS Rule Fields + +| Field | Description | +|-------|-------------| +| `AllowedOrigins` | Origins allowed to access the bucket (required) | +| `AllowedMethods` | HTTP methods allowed (GET, PUT, POST, DELETE, HEAD) | +| `AllowedHeaders` | Request headers allowed in preflight | +| `ExposeHeaders` | Response headers visible to browser | +| `MaxAgeSeconds` | How long browser can cache preflight response | + +## 25. List Objects API v2 + +MyFSIO supports both ListBucketResult v1 and v2 APIs. + +### Using v2 API + +```bash +# List with v2 (supports continuation tokens) +curl "http://localhost:5000/my-bucket?list-type=2" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# With prefix and delimiter (folder-like listing) +curl "http://localhost:5000/my-bucket?list-type=2&prefix=photos/&delimiter=/" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Pagination with continuation token +curl "http://localhost:5000/my-bucket?list-type=2&max-keys=100&continuation-token=TOKEN" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." + +# Start after specific key +curl "http://localhost:5000/my-bucket?list-type=2&start-after=photos/2024/" \ + -H "X-Access-Key: ..." -H "X-Secret-Key: ..." +``` + +### v1 vs v2 Differences + +| Feature | v1 | v2 | +|---------|----|----| +| Pagination | `marker` | `continuation-token` | +| Start position | `marker` | `start-after` | +| Fetch owner info | Always included | Use `fetch-owner=true` | +| Max keys | 1000 | 1000 | + +### Query Parameters + +| Parameter | Description | +|-----------|-------------| +| `list-type` | Set to `2` for v2 API | +| `prefix` | Filter objects by key prefix | +| `delimiter` | Group objects (typically `/`) | +| `max-keys` | Maximum results (1-1000, default 1000) | +| `continuation-token` | Token from previous response | +| `start-after` | Start listing after this key | +| `fetch-owner` | Include owner info in response | +| `encoding-type` | Set to `url` for URL-encoded keys diff --git a/templates/docs.html b/templates/docs.html index a0c5b41..2f713d7 100644 --- a/templates/docs.html +++ b/templates/docs.html @@ -43,6 +43,14 @@
  • Metrics History
  • Operation Metrics
  • Troubleshooting
    • +
  • Health Check
    • +
  • Object Lock & Retention
    • +
  • Access Logging
    • +
  • Notifications & Webhooks
    • +
  • SelectObjectContent
    • +
  • Advanced Operations
    • +
  • Access Control Lists
    • +
  • Object & Bucket Tagging
    • @@ -1694,6 +1702,403 @@ curl "{{ api_base | replace('/api', '/ui') }}/metrics/operations/history?hours=6 +
    +
    +
    + 17 +

    Health Check Endpoint

    +
    +

    The API exposes a health check endpoint for monitoring and load balancer integration.

    + +
    # Check API health
+curl {{ api_base }}/myfsio/health
+
+# Response
+{"status": "ok", "version": "0.1.7"}
    + +

    Use this endpoint for:

    +
      +
    • Load balancer health checks
      • +
    • Kubernetes liveness/readiness probes
      • +
    • Monitoring system integration (Prometheus, Datadog, etc.)
      • +
    +
    +
    +
    +
    +
    + 18 +

    Object Lock & Retention

    +
    +

    Object Lock prevents objects from being deleted or overwritten for a specified retention period.

    + +

    Retention Modes

    +
    + + + + + + + + + + + + + + + + + +
    ModeDescription
    GOVERNANCEObjects can't be deleted by normal users, but admins with bypass permission can override
    COMPLIANCEObjects can't be deleted or overwritten by anyone until the retention period expires
    +
    + +

    API Usage

    +
    # Set object retention
+curl -X PUT "{{ api_base }}/<bucket>/<key>?retention" \
+  -H "Content-Type: application/json" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '{"Mode": "GOVERNANCE", "RetainUntilDate": "2025-12-31T23:59:59Z"}'
+
+# Enable legal hold (indefinite protection)
+curl -X PUT "{{ api_base }}/<bucket>/<key>?legal-hold" \
+  -H "Content-Type: application/json" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '{"Status": "ON"}'
+
+# Get legal hold status
+curl "{{ api_base }}/<bucket>/<key>?legal-hold" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
    + +
    +
    + + + + +
    + Legal Hold: Provides indefinite protection independent of retention settings. Use for litigation holds or regulatory requirements. +
    +
    +
    +
    +
    +
    +
    +
    + 19 +

    Access Logging

    +
    +

    Enable S3-style access logging to track all requests to your buckets for audit and analysis.

    + +
    # Enable access logging
+curl -X PUT "{{ api_base }}/<bucket>?logging" \
+  -H "Content-Type: application/json" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '{
+    "LoggingEnabled": {
+      "TargetBucket": "log-bucket",
+      "TargetPrefix": "logs/my-bucket/"
+    }
+  }'
+
+# Get logging configuration
+curl "{{ api_base }}/<bucket>?logging" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
    + +

    Log Contents

    +

    Logs include: timestamp, bucket, key, operation type, request ID, requester, source IP, HTTP status, error codes, bytes transferred, timing, referrer, and User-Agent.

    +
    +
    +
    +
    +
    + 20 +

    Notifications & Webhooks

    +
    +

    Configure event notifications to trigger webhooks when objects are created or deleted.

    + +

    Supported Events

    +
    + + + + + + + + + + + + + + + + + +
    Event TypeDescription
    s3:ObjectCreated:*Any object creation (PUT, POST, COPY, multipart)
    s3:ObjectRemoved:*Any object deletion
    +
    + +
    # Set notification configuration
+curl -X PUT "{{ api_base }}/<bucket>?notification" \
+  -H "Content-Type: application/json" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '{
+    "TopicConfigurations": [{
+      "Id": "upload-notify",
+      "TopicArn": "https://webhook.example.com/s3-events",
+      "Events": ["s3:ObjectCreated:*"],
+      "Filter": {
+        "Key": {
+          "FilterRules": [
+            {"Name": "prefix", "Value": "uploads/"},
+            {"Name": "suffix", "Value": ".jpg"}
+          ]
+        }
+      }
+    }]
+  }'
    + +
    +
    + + + + +
    + Security: Webhook URLs are validated to prevent SSRF attacks. Internal/private IP ranges are blocked. +
    +
    +
    +
    +
    +
    +
    +
    + 21 +

    SelectObjectContent (SQL)

    +
    +

    Query CSV, JSON, or Parquet files directly using SQL without downloading the entire object.

    + +
    + Prerequisite: Requires DuckDB to be installed (pip install duckdb) +
    + +
    # Query a CSV file
+curl -X POST "{{ api_base }}/<bucket>/data.csv?select" \
+  -H "Content-Type: application/json" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '{
+    "Expression": "SELECT name, age FROM s3object WHERE age > 25",
+    "ExpressionType": "SQL",
+    "InputSerialization": {
+      "CSV": {"FileHeaderInfo": "USE", "FieldDelimiter": ","}
+    },
+    "OutputSerialization": {"JSON": {}}
+  }'
    + +

    Supported Formats

    +
    +
    +
    + CSV
    Headers, delimiters +
    +
    +
    +
    + JSON
    Document or lines +
    +
    +
    +
    + Parquet
    Auto schema +
    +
    +
    +
    +
    +
    +
    +
    + 22 +

    Advanced S3 Operations

    +
    +

    Copy objects, upload part copies, and use range requests for partial downloads.

    + +

    CopyObject

    +
    # Copy within same bucket
+curl -X PUT "{{ api_base }}/<bucket>/copy-of-file.txt" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-copy-source: /<bucket>/original-file.txt"
+
+# Copy with metadata replacement
+curl -X PUT "{{ api_base }}/<bucket>/file.txt" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-copy-source: /<bucket>/file.txt" \
+  -H "x-amz-metadata-directive: REPLACE" \
+  -H "x-amz-meta-newkey: newvalue"
    + +

    UploadPartCopy

    +

    Copy data from an existing object into a multipart upload part:

    +
    # Copy bytes 0-10485759 from source as part 1
+curl -X PUT "{{ api_base }}/<bucket>/<key>?uploadId=X&partNumber=1" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-copy-source: /source-bucket/source-file.bin" \
+  -H "x-amz-copy-source-range: bytes=0-10485759"
    + +

    Range Requests

    +
    # Get first 1000 bytes
+curl "{{ api_base }}/<bucket>/<key>" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "Range: bytes=0-999"
+
+# Get last 500 bytes
+curl "{{ api_base }}/<bucket>/<key>" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "Range: bytes=-500"
    + +

    Conditional Requests

    +
    + + + + + + + + + + + + + + + + + + + + + +
    HeaderBehavior
    If-Modified-SinceOnly download if changed after date
    If-None-MatchOnly download if ETag differs
    If-MatchOnly download if ETag matches
    +
    +
    +
    +
    +
    +
    + 23 +

    Access Control Lists (ACLs)

    +
    +

    ACLs provide legacy-style permission management for buckets and objects.

    + +

    Canned ACLs

    +
    + + + + + + + + + + + + + + + + + + + + + + + + + +
    ACLDescription
    privateOwner gets FULL_CONTROL (default)
    public-readOwner FULL_CONTROL, public READ
    public-read-writeOwner FULL_CONTROL, public READ and WRITE
    authenticated-readOwner FULL_CONTROL, authenticated users READ
    +
    + +
    # Set bucket ACL
+curl -X PUT "{{ api_base }}/<bucket>?acl" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-acl: public-read"
+
+# Set object ACL during upload
+curl -X PUT "{{ api_base }}/<bucket>/<key>" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-acl: private" \
+  --data-binary @file.txt
    + +
    + Recommendation: For most use cases, prefer bucket policies over ACLs for more flexible access control. +
    +
    +
    +
    +
    +
    + 24 +

    Object & Bucket Tagging

    +
    +

    Add metadata tags to buckets and objects for organization, cost allocation, or lifecycle rule filtering.

    + +

    Object Tagging

    +
    # Set object tags
+curl -X PUT "{{ api_base }}/<bucket>/<key>?tagging" \
+  -H "Content-Type: application/json" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '{
+    "TagSet": [
+      {"Key": "Classification", "Value": "Confidential"},
+      {"Key": "Owner", "Value": "john@example.com"}
+    ]
+  }'
+
+# Get object tags
+curl "{{ api_base }}/<bucket>/<key>?tagging" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>"
+
+# Set tags during upload
+curl -X PUT "{{ api_base }}/<bucket>/<key>" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -H "x-amz-tagging: Environment=Staging&Team=QA" \
+  --data-binary @file.txt
    + +

    Bucket Tagging

    +
    # Set bucket tags
+curl -X PUT "{{ api_base }}/<bucket>?tagging" \
+  -H "Content-Type: application/json" \
+  -H "X-Access-Key: <key>" -H "X-Secret-Key: <secret>" \
+  -d '{
+    "TagSet": [
+      {"Key": "Environment", "Value": "Production"},
+      {"Key": "Team", "Value": "Engineering"}
+    ]
+  }'
    + +

    Use Cases

    +
    +
    +
      +
    • Filter objects for lifecycle expiration by tag
      • +
    • Use tag conditions in bucket policies
      • +
    +
    +
    +
      +
    • Group objects by project or department
      • +
    • Trigger automation based on object tags
      • +
    +
    +
    +
    +