kqjy/MyFSIO
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 42 Wiki Activity

42 Releases 42 Tags

RSS Feed
  • v0.5.6 d76efcb48f
    Compare

    MyFSIO v0.5.6 Beta Pre-Release

    kqjy released this 2026-07-20 07:59:17 +00:00 | 8 commits to main since this release

    Storage Engine Hardening

    • Streaming checksum validation on PutObject, UploadPart, and POST uploads — checksums are now verified as bytes stream to disk instead of buffering up to 5 GiB in RAM.
    • Atomic write commits: conditional writes (If-Match / If-None-Match), object-lock retention, and bucket quotas are now enforced atomically at the storage commit, eliminating TOCTOU races (including from internal writers like replication and site sync).
    • Encrypt-before-commit SSE PUT: ciphertext is fully written before install in a single write, and full, range, and multipart SSE-C GETs now use chunked streaming decryption.
    • Durability: object data, segment files, and parent directories are fsynced before the write is acknowledged.
    • Hard-link version snapshots for zero-copy versioned overwrites.
    • Range-windowed segment opens: narrow range reads of large multipart objects (e.g. 10,000 parts) now open ~2 segment files instead of all of them.

    Metadata Layout: Per-Object Sidecars

    • METADATA_LAYOUT=sidecar (new default): object metadata is stored as one sidecar file per object instead of a per-directory _index.json, making metadata updates O(1). Readers resolve sidecar → index → legacy order forever, so existing data keeps working. Note: older binaries cannot read sidecar metadata.
    • Fail-closed corruption handling replaces the previous behavior where a corrupt _index.json was treated as an empty map (silently orphaning objects): corrupt metadata now returns 422, blocks read-modify-write updates, and blanks affected listings instead of destroying state.
    • --migrate-meta one-shot CLI converts existing aggregate indexes to sidecars in bulk (run with the server stopped; includes a no-rollback warning and startup notice).
    • Integrity scanner honors sidecar precedence.

    S3 API Correctness

    • New IncompleteBody error code with declared-length enforcement across PutObject, UploadPart, SSE-C, and checksum paths — short aws-chunked bodies were previously stored truncated with a 200 response. Transport errors during body reads are mapped correctly.

    Replication Reliability

    • Bounded replication queue with dedicated worker tasks (REPLICATION_QUEUE_CAPACITY, REPLICATION_CONCURRENCY), reconciliation, and overflow journaling — queue overflow is persisted for healer retry instead of dropped.

    HDD Admission Control (opt-in)

    • HDD_READ_CONCURRENCY / HDD_WRITE_CONCURRENCY cap concurrent S3 object data reads/writes (recommended 2 for HDD storage); requests that wait longer than DISK_QUEUE_TIMEOUT_SECONDS for a disk permit receive 503 SlowDown.
    • Disk-pressure metrics and a new UI card. Admin/UI requests, HEAD requests, and metadata operations are unaffected.

    Metrics & Observability

    • Persist-backed S3 error-code summary (1h / 6h / 24h windows) that survives snapshot rollover and server restarts, with real x-amz-error-code recording and per-bucket error attribution.
    • Recent-errors drill-down with request IDs, plus an error-codes-over-time chart with zero-filled gaps.
    • Atomic + quarantined metrics persistence, flush on shutdown, and a single snapshot at startup.
    • Cached storage walk for total stored bytes (METRICS_STORAGE_REFRESH_MINUTES, minimum 5) and no metrics writes while idle.

    Web UI

    • Vendored Bootstrap 5.3.2 for fully offline deployments.
    • Uptime now measures from server boot with hour/minute granularity.
    • Compact bucket list view and an opaque sticky object-table header.

    Documentation & Cleanup

    • All docs synced to code: rate limits, quota/XML APIs, GC + integrity tables, metrics endpoints, segments layout, and peer-credential scope.
    • Metrics interval env vars are floored to safe minimums.
    • Removed dead classify_endpoint code.
    • 11 new end-to-end hardening tests.

    Important Upgrade Notes

    • The default metadata layout is now sidecar. Older binaries cannot read sidecar metadata — do not downgrade after new writes occur. Set METADATA_LAYOUT=index to retain the legacy layout, or run --migrate-meta (server stopped) to convert existing data in bulk; migration is one-way.
    • HDD admission control is disabled by default (HDD_READ_CONCURRENCY=0 / HDD_WRITE_CONCURRENCY=0); set both to 2 on HDD-backed deployments.
    Downloads