kqjy/MyFSIO
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 42 Wiki Activity

42 Releases 42 Tags

RSS Feed
  • v0.5.8 1fbb143a3e
    Compare

    MyFSIO v0.5.8 Beta Pre-Release

    kqjy released this 2026-08-12 06:24:11 +00:00 | 0 commits to main since this release

    Security

    Authorization & Authentication

    • Fixed P0 bucket-name path escape: full S3 bucket-name syntax plus canonical containment is now enforced inside the storage backend (require_bucket, bucket_exists, create_bucket, multipart paths, versioning checks) — a percent-encoded absolute path in the bucket position can no longer replace STORAGE_ROOT.
    • Fixed P0 query-selector precedence mismatch at both bucket and object level: one shared BucketSubresource/ObjectSubresource parser is used by the auth middleware and every dispatcher. Multiple selectors are rejected with InvalidArgument at authorization time; non-dispatchable selectors return MethodNotAllowed instead of falling through. Closes an authenticated list-to-policy escalation, its unauthenticated variant on public-read buckets, and read-only principals writing/deleting via ?attributes/?select/?uploads/?retention.
    • The auth middleware percent-decodes bucket, key, and copy-source so it authorizes the same strings the handlers act on.
    • Every key in DeleteObjects is re-authorized individually — a prefix-scoped principal can no longer delete outside its prefix.
    • Presigned requests must sign every x-amz-* header (except content-sha256/date/decoded-content-length), closing copy-source, ACL, SSE, and governance-bypass header injection by URL bearers.
    • Bucket policies fail closed on unsupported Condition/NotPrincipal/NotAction/NotResource elements at evaluation, and reject them at both write paths.
    • is_admin now requires an unrestricted policy prefix — a prefix-scoped wildcard policy no longer short-circuits every authorization check.
    • Bucket-policy Resource keys match case-sensitively; bucket and Action matching stay case-insensitive.
    • An unreadable .bucket.json is treated as fail-closed across policy evaluation, object write/delete, and all config writes (previously acted as an empty config).
    • x-amz-bypass-governance-retention is honored only for admins or principals granted the new bypass_governance IAM action (or s3:BypassGovernanceRetention in a bucket policy), evaluated per key on bulk delete.
    • New IAM actions: system:gc_read/gc_run/integrity_read/integrity_run replace admin-only gating of GC and integrity on both the admin API and UI; ownershipControls/publicAccessBlock gain dedicated ownership_controls/public_access_block actions; action_matches generalized from hardcoded iam:* to any namespace:*.
    • Added missing bucket authorization to 11 UI handlers.

    Path Traversal & Input Validation

    • Multipart uploadId/segment ids are validated as 32-hex before any filesystem join — abort_multipart previously remove_dir_all'd caller-controlled paths, capable of destroying .myfsio.sys/config in one request.
    • Reserved __/x-amz- metadata keys are rejected at all five ingress points, including CopyObject, UI multipart init, and peer pull (x-amz-meta-__segments__ could reach recursive segment deletion and redirect segment reads).
    • UI restore version_id is validated to a single path component.
    • PutBucketLogging TargetBucket and website-domain mappings reject invalid bucket names; ListBuckets and --rebuild-listing skip invalid-named directories with a warning instead of failing.

    Secrets & Encryption

    • First-run and reset admin credentials are written encrypted, with a one-time plaintext-to-encrypted migration at startup; env-provided secrets are no longer echoed to stdout.
    • SECRET_KEY auto-generation persists to .myfsio.sys/config/.secret; the literal value dev-secret-key is rejected from both env and file.
    • iam.json, KMS master keys, kms_keys.json, .connections_key, connections.json, and .secret are all written owner-only through shared atomic helpers (myfsio_common::fs_util).
    • The server exits non-zero when ENCRYPTION_ENABLED/KMS_ENABLED init fails, and fails closed on bucket-default encryption instead of silently storing plaintext.
    • IAM disable-user/create-key/delete-key are routed through load_config/save_config so revocation works against an encrypted iam.json.
    • IAM mutations are serialized behind a mutation lock and persisted atomically (tmp + fsync + rename).

    Web UI

    • Fixed XSS via data-carrying inline onclick handlers — replaced with delegated data-* listeners (verified in-browser).
    • Anonymous requests use ephemeral sessions that never enter the session store; capacity eviction prefers unauthenticated sessions.
    • GET/POST /login is rate limited per IP (RATE_LIMIT_UI_LOGIN, default 20/minute) with a styled 429 page and Retry-After.

    S3 API

    • Object lock: PutBucketObjectLockConfiguration validates the XML and requires versioning (409 InvalidBucketState); DefaultRetention is now actually applied at every user-facing object-creation path (explicit headers win; replication/peer-pull/restore untouched). Retention and legal-hold updates are check-and-set under the per-object stripe lock via new update_object_retention/update_object_legal_hold trait methods — concurrent requests can no longer shorten COMPLIANCE retention, and legal-hold writes no longer clobber concurrent metadata. Same-mode extension needs no bypass; shortening, removal, and mode changes keep the bypass rules.
    • Static website hosting streams through the new handlers/object_read data plane: SSE-S3/SSE-KMS assets that were served as ciphertext are decrypted, HEAD reports plaintext Content-Length, 206 ranges use plaintext offsets, SSE-C assets return 403, and no path buffers whole objects in memory.
    • Ranged S3 GETs validate SSE-C keys like the whole-object path (400/403 instead of 500).
    • Unknown encoding-type values are rejected; max-keys is capped at 1000.
    • Date-based lifecycle rules are a no-op until the date passes, instead of using the configured date as an age cutoff.
    • Request-body limits on every config/XML/JSON ingress (1 MiB config XML and Select, 8 MiB CompleteMultipartUpload/DeleteObjects, 1–2 MiB JSON, bounded relay-outbound, 1 MiB non-file multipart form fields) with a new MaxMessageLengthExceeded error code.

    Storage & Durability

    • A failed PUT commit rolls back the archived version and metadata sidecar; the archived-null purge runs after the rename, so a failed commit can never destroy the prior null version.
    • Fixed quarantine GC protection reading the wrong metadata root, multipart corruption false positives in the integrity scanner, and issue-count reporting; removed the legacy heal path.
    • GC skips a bucket's segment sweep whenever the reference scan hit a read error, and reports the skip — an incomplete scan can no longer delete live segment data.

    Background Services & Reliability

    • Site sync aborts the cycle on any remote-listing failure instead of treating NoSuchBucket/404 as an empty bucket (sync_deletions could previously erase every locally synchronized object), and records last_error/last_error_at per bucket, surfaced on the Sites page.
    • The process exits non-zero when either listener task dies, so systemd restarts it.

    Web UI

    • UI part upload streams to disk with a 5 GiB cap instead of buffering each part twice in memory.
    • Failed uploads surface the real S3 error reason in the toast and upload dialog instead of a bare failure count.
    • Fixed dead cluster-page JavaScript, copy/move destination bucket default, audio preview stage height, and assorted polish.
    • Each System dashboard card names the specific missing permission when access is denied.

    Installer

    • install.sh/uninstall.sh canonicalize and refuse system paths, guard foreign data directories behind --adopt-data-dir, track installer-created users in a manifest so uninstall preserves pre-existing accounts, and re-installs preserve myfsio.env (--overwrite-env backs up then regenerates).

    Added

    • handlers/object_read.rs (shared streaming object-read data plane), myfsio_common::fs_util (atomic owner-only file helpers), login_rate_limited.html.
    • IAM actions: bypass_governance, system:gc_read/gc_run/integrity_read/integrity_run, ownership_controls, public_access_block.
    • Error codes: S3ErrorCode::MaxMessageLengthExceeded, S3ErrorCode::InvalidBucketState, StorageError::InvalidArgument.
    • Config: RATE_LIMIT_UI_LOGIN (default 20 per minute).
    • Validation helpers: is_valid_multipart_id, is_reserved_metadata_key, is_reserved_user_metadata_key, is_safe_path_segment, bucket_name_rejection; BucketConfig::unreadable.

    Removed

    • Dead has_query_key helper and the legacy integrity heal path.
    Downloads
  • v0.5.7 424d2f99c4
    Compare

    MyFSIO v0.5.7 Beta Pre-Release

    kqjy released this 2026-07-24 13:56:51 +00:00 | 6 commits to main since this release

    Breaking changes

    • Streaming SigV4 is now enforced by default. STRICT_STREAMING_SIGV4 defaults to true (was false). The server now validates per-chunk signature chains, the final zero-length chunk, and signed trailers on STREAMING-AWS4-HMAC-SHA256-PAYLOAD* uploads, and rejects requests that fail validation or declare streaming without a signing context — previously these were accepted unvalidated (fail-open passthrough). If you must keep the old behavior for a non-conforming client, set STRICT_STREAMING_SIGV4=false; this is a warn-and-passthrough escape hatch, and checksum trailers are still verified either way.
    • Unknown bucket sub-resources now return 501 NotImplemented. GET/DELETE against an unrecognized bucket sub-resource no longer falls through to a bucket listing.

    Added

    • Full upload-checksum verification: crc32c, sha1, and crc64nvme are now verified alongside md5/sha256/crc32 on every upload path, and the aws-chunked trailer checksum section is parsed and verified (it was previously discarded).
    • x-amz-checksum-* response headers on GET/HEAD are now gated behind x-amz-checksum-mode: ENABLED.
    • Durable per-bucket replication pending-ledger (.myfsio.sys/buckets/<bucket>/replication/pending.snapshot.json + pending.journal.jsonl) with crash-safe PENDING → COMPLETED → ack ordering, startup replay with zero sidecar scan, and a one-time seeding migration for pre-ledger buckets.
    • REPLICATION_FULL_RECONCILE_INTERVAL_HOURS (default 0 = off) for an optional periodic full replication consistency pass.
    • Persistent per-bucket counters (live objects/bytes, version count/bytes, delete-marker count) in the listing snapshot, giving O(1) quota checks and bucket stats.

    Changed

    • Listing-index compaction moved off the PUT/DELETE request path to a background worker (generation-numbered journals, seal-under-lock / snapshot-outside-lock, drains before shutdown).
    • Object metadata read cache is now a real LRU; OBJECT_CACHE_MAX_SIZE default raised 100 → 1024.
    • UI: presign expiry gains preset + custom-seconds inputs with inline min/max validation that blocks out-of-range values (previously clamped silently); create-bucket dialog autofocuses and adds an enable-versioning checkbox; AJAX 401 now redirects to /login?next=; a single shared binary-unit byte formatter replaces 8+ duplicates; metrics "As of" is server-formatted in DISPLAY_TIMEZONE; the quota card splits into Objects / Versions / Total stored; error toasts are titled "Error"; object preview has a stable layout with %PDF- magic-byte sniffing and a styled "Preview unavailable" fallback.
    • UI bucket / dashboard / cluster storage walks are parallelized.

    Fixed

    • Quota undercounting on versioned and versioning-suspended overwrites; quota fallback PUTs now bypass the stats cache instead of projecting from stale totals.
    Downloads
  • v0.5.6 d76efcb48f
    Compare

    MyFSIO v0.5.6 Beta Pre-Release

    kqjy released this 2026-07-20 07:59:17 +00:00 | 8 commits to main since this release

    Storage Engine Hardening

    • Streaming checksum validation on PutObject, UploadPart, and POST uploads — checksums are now verified as bytes stream to disk instead of buffering up to 5 GiB in RAM.
    • Atomic write commits: conditional writes (If-Match / If-None-Match), object-lock retention, and bucket quotas are now enforced atomically at the storage commit, eliminating TOCTOU races (including from internal writers like replication and site sync).
    • Encrypt-before-commit SSE PUT: ciphertext is fully written before install in a single write, and full, range, and multipart SSE-C GETs now use chunked streaming decryption.
    • Durability: object data, segment files, and parent directories are fsynced before the write is acknowledged.
    • Hard-link version snapshots for zero-copy versioned overwrites.
    • Range-windowed segment opens: narrow range reads of large multipart objects (e.g. 10,000 parts) now open ~2 segment files instead of all of them.

    Metadata Layout: Per-Object Sidecars

    • METADATA_LAYOUT=sidecar (new default): object metadata is stored as one sidecar file per object instead of a per-directory _index.json, making metadata updates O(1). Readers resolve sidecar → index → legacy order forever, so existing data keeps working. Note: older binaries cannot read sidecar metadata.
    • Fail-closed corruption handling replaces the previous behavior where a corrupt _index.json was treated as an empty map (silently orphaning objects): corrupt metadata now returns 422, blocks read-modify-write updates, and blanks affected listings instead of destroying state.
    • --migrate-meta one-shot CLI converts existing aggregate indexes to sidecars in bulk (run with the server stopped; includes a no-rollback warning and startup notice).
    • Integrity scanner honors sidecar precedence.

    S3 API Correctness

    • New IncompleteBody error code with declared-length enforcement across PutObject, UploadPart, SSE-C, and checksum paths — short aws-chunked bodies were previously stored truncated with a 200 response. Transport errors during body reads are mapped correctly.

    Replication Reliability

    • Bounded replication queue with dedicated worker tasks (REPLICATION_QUEUE_CAPACITY, REPLICATION_CONCURRENCY), reconciliation, and overflow journaling — queue overflow is persisted for healer retry instead of dropped.

    HDD Admission Control (opt-in)

    • HDD_READ_CONCURRENCY / HDD_WRITE_CONCURRENCY cap concurrent S3 object data reads/writes (recommended 2 for HDD storage); requests that wait longer than DISK_QUEUE_TIMEOUT_SECONDS for a disk permit receive 503 SlowDown.
    • Disk-pressure metrics and a new UI card. Admin/UI requests, HEAD requests, and metadata operations are unaffected.

    Metrics & Observability

    • Persist-backed S3 error-code summary (1h / 6h / 24h windows) that survives snapshot rollover and server restarts, with real x-amz-error-code recording and per-bucket error attribution.
    • Recent-errors drill-down with request IDs, plus an error-codes-over-time chart with zero-filled gaps.
    • Atomic + quarantined metrics persistence, flush on shutdown, and a single snapshot at startup.
    • Cached storage walk for total stored bytes (METRICS_STORAGE_REFRESH_MINUTES, minimum 5) and no metrics writes while idle.

    Web UI

    • Vendored Bootstrap 5.3.2 for fully offline deployments.
    • Uptime now measures from server boot with hour/minute granularity.
    • Compact bucket list view and an opaque sticky object-table header.

    Documentation & Cleanup

    • All docs synced to code: rate limits, quota/XML APIs, GC + integrity tables, metrics endpoints, segments layout, and peer-credential scope.
    • Metrics interval env vars are floored to safe minimums.
    • Removed dead classify_endpoint code.
    • 11 new end-to-end hardening tests.

    Important Upgrade Notes

    • The default metadata layout is now sidecar. Older binaries cannot read sidecar metadata — do not downgrade after new writes occur. Set METADATA_LAYOUT=index to retain the legacy layout, or run --migrate-meta (server stopped) to convert existing data in bulk; migration is one-way.
    • HDD admission control is disabled by default (HDD_READ_CONCURRENCY=0 / HDD_WRITE_CONCURRENCY=0); set both to 2 on HDD-backed deployments.
    Downloads
  • v0.5.5 a22988d556
    Compare

    MyFSIO v0.5.5 Beta Pre-Release

    kqjy released this 2026-07-02 07:38:11 +00:00 | 14 commits to main since this release

    Highlights

    This release introduces a new completed-multipart storage layout that makes CompleteMultipartUpload an O(metadata) operation on any filesystem, alongside a round of UI security hardening (XSS fixes), observability corrections, and object-browser reliability fixes.

    New Features

    Segment-based multipart object layout (MULTIPART_OBJECT_LAYOUT=segments, now the default)

    • Completed multipart objects are stored as immutable per-part segment files with a sparse stub at the object's key path, instead of concatenating all parts into a single file.
    • CompleteMultipartUpload becomes an O(metadata) operation — no full-object copy — so large multipart completes are fast on any filesystem.
    • Enables zero-copy versioned overwrites.
    • All read paths are layout-aware: GET, Range GET, partNumber GET, server-side copy, replication, and S3 Select all transparently handle both the new segments layout and the legacy concat layout.
    • CompleteMultipartUpload now returns x-amz-version-id.
    • Garbage collection sweeps orphaned segment directories (GC_SEGMENT_MAX_AGE_HOURS, default 24).

    Compatibility note: Only new completes use the segments layout; existing objects are untouched and readers handle both layouts indefinitely. Set MULTIPART_OBJECT_LAYOUT=concat to retain the legacy single-file assembly. Older server binaries cannot read segments-layout objects — upgrade all nodes before enabling.

    Security

    • Fixed UI cross-site scripting (XSS) vulnerabilities:
      • Corrected a no-op escapeHtml helper that was failing to escape output.
      • Closed a JSON-island <script> breakout in embedded page data.
      • Swept the remaining json_encode | safe template patterns.
    • Vendored Chart.js locally instead of loading it from a third-party origin.

    Improvements

    • Cluster resilience: Cluster overview auto-refresh now recovers after peer outages via a shared PollingManager, and offline-peer cards recover instead of staying stuck.
    • Scan pollers back off and retry instead of giving up with stale failure counters.
    • Static assets: ETags are memoized with vendor caching for cheaper conditional responses.
    • Object browser: Object timestamp display is now unified across the browser UI, and folder navigation syncs to a ?prefix= URL for deep links and working back/forward navigation.
    • Chart.js load failures are now surfaced to the user instead of failing silently.

    Bug Fixes

    • GC freed-bytes accounting: GC now emits an accurate total_bytes_freed and counts quarantined files, fixing a persistent undercount.
    • Object browser virtualizer: Fixed folder-checkbox selection desyncing on scroll, and the virtual list now measures actual row height instead of hardcoding 53px.
    • Removed dead bucket-detail modules and hardened poller guards and the Connections edit/refresh flow.

    Configuration

    Variable Default Description
    MULTIPART_OBJECT_LAYOUT segments Completed MPU storage layout: segments (immutable part segments + sparse stub) or concat (legacy single-file assembly). Affects new completes only.
    GC_SEGMENT_MAX_AGE_HOURS 24 Age before orphaned multipart segment directories are GC-swept.
    Downloads
  • v0.5.4 ec1fe1b1f7
    Compare

    MyFSIO v0.5.4 Beta Pre-Release

    kqjy released this 2026-06-08 07:57:32 +00:00 | 20 commits to main since this release

    Server-side encryption — SSE-C for multipart uploads

    • Per-part SSE-C for multipart uploads, with the customer key never stored. Previously the only SSE-C multipart path stored the base64 customer key in the upload manifest; that is gone. Initiate now generates a random per-object data key (ODK), wraps it under the customer key, and persists only the wrapped ODK + the customer-key MD5 in the manifest (gated behind an __mpu_sse_c__ marker, so SSE-S3 / SSE-KMS / single-PUT SSE-C / unencrypted paths are byte-for-byte unchanged).
    • Each UploadPart validates the supplied key against the stored MD5, unwraps the ODK, and encrypts the part to a self-describing block carrying a fresh per-part random salt (nonce = HKDF(ODK, part_number ‖ salt)), so re-uploading a part can never reuse an AES-GCM nonce.
    • GET / HEAD, Range, and partNumber reads decrypt per-part by re-deriving the nonce from the stored salt and mapping plaintext offsets via per-part plaintext sizes. Missing key → 400 InvalidRequest; wrong key → 403 AccessDenied (AEAD failure never surfaces as 500).
    • UploadPartCopy into an SSE-C multipart upload is rejected with 501 NotImplemented (use UploadPart), and the multipart minimum-part-size check runs against plaintext size before publish so a failed completion can't destroy a pre-existing object or leave a stray delete marker.

    S3 API correctness & conformance

    • Every error response now flows through a single error path that auto-generates a request id and emits x-amz-request-id / x-amz-id-2 headers alongside the XML body.
    • Per-object <Owner> in ListObjects / ListObjectsV2 / ListObjectVersions: owner is parsed from the object's stored __acl__ and DisplayName is resolved via a per-request IAM map. Legacy objects with no ACL fall back to a canonical myfsio owner (not the requester); V1 now always emits Owner + StorageClass per spec.
    • Added PutBucketOwnershipControls and PutPublicAccessBlock as no-op stubs (GET/PUT/DELETE) so boto3 ≥ 1.32 CreateBucket fixtures stop erroring before the test body runs.
    • Object keys that collide with an existing directory path are now storable: shadowed keys are written to a <dir>/.__myfsio_keydata__ marker, with crash-safe conversion of blocking files via the system temp dir. Listing, HEAD/GET/DELETE, and the integrity scanner all resolve through the marker.
    • Leading-slash object keys are normalized on the object data path. A key sent with a leading slash (e.g. //prefix/x on the wire) was rejected with 400 InvalidObjectKey on HEAD/GET/PUT/DELETE even though the SigV4 layer authorized it under the clean key — so clients keying not-found detection off 404/NoSuchKey saw a hard error instead of a missing object. The five object handlers now strip leading slashes to match the auth layer's key derivation, and the list-prefix handling from v0.5.3 was reworked to share the same normalization (internal double-slashes and .. traversal handling unchanged).
    • 416 Range Not Satisfiable now returns a Content-Range header; UploadPart validates the part number is within 1..=10000.
    • S3 Select: sets memory_limit / max_memory and fixes a UTF-8 chunk-boundary panic.

    Replication

    • Resumable, partial-failure-tolerant multipart replication. Failed attempts now persist the upload_id plus a source-identity fingerprint (size, etag, part size); a retry validates the fingerprint, calls ListParts to discover already-uploaded parts, and resumes the same MPU instead of restarting from byte zero (part ETags preserved verbatim). Stale ids — NoSuchUpload, changed source content/tuning, or permanent CompleteMultipartUpload errors — are aborted server-side and cleared so no retry ever reuses a dead id.
    • Per-part failures within a pass are collected and retried in a second in-MPU pass; a permanent error or task panic immediately aborts sibling parts and the MPU, so a missing/panicked part can never slip into CompleteMultipartUpload.
    • Progress-aware stall detection: each part body streams through a progress reader paired with a tokio::select! watchdog. REPLICATION_PART_STALL_TIMEOUT_SECONDS (default 300s) now bounds both "no first byte" and "mid-transfer stall" cases, replacing the coarse Smithy read-timeout for stuck-upload detection on HDD/WAN paths.
    • Failure-store writes now hold the cache lock across the read-modify-write and persist atomically (temp + rename); an RAII in-flight guard prevents a panicking replicate task from leaking in-flight counters or wedging a bucket's batch run; NoSuchBucket is classified via the structured SDK error code.

    Integrity & garbage collection

    • HDD-friendly scanning. INTEGRITY_HEAL_CONCURRENCY now defaults to 1 (serial heal — parallel hashing thrashes a single spindle), and a new INTEGRITY_SCAN_PACING_MS (default 0) applies a per-object pacing delay every ~100 objects across all scan phases to relieve disk-head pressure.
    • SSE-encrypted objects are skipped in the corruption scan. Comparing a plaintext ETag against ciphertext produced a false positive that, combined with auto-heal, was quarantining and poisoning every encrypted object.
    • Corruption mismatches are now re-verified under the object write lock before quarantine, with a grace-period guard so versions mid-archive aren't quarantined; the in-progress guard is hardened against the never-polled-task window and objects_scanned is de-duplicated.
    • Stuck "scan in progress" banner fixed. Integrity/GC run state leaked when an HTTP request was cancelled mid-run, leaving the banner stuck forever. Runs now execute on a detached task with an RAII guard that always clears the running/started-at state, regardless of client cancellation.

    Security & auth

    • Bucket-policy matching is now gated by request method. A read grant (s3:GetBucketCors / Lifecycle / Replication / …) no longer authorizes the corresponding PUT/DELETE. Allow-only semantics, IAM, and wildcards are unchanged.
    • SSE-C customer-key MD5 is persisted and constant-time verified on GET/HEAD.
    • DeleteObjects honors x-amz-bypass-governance-retention.
    • Buffered upload paths (checksummed PUT, POST form) are now size-bounded to prevent OOM; tag key/value are XML-escaped in GetBucketTagging / GetObjectTagging; the replication-UI failure endpoints validate the bucket name (path-traversal / cross-bucket tampering).
    • The streaming-SigV4 acceptance warning is demoted to once per process instead of one line per upload.

    Web UI

    • The /ui/buckets page gains an access filter (All / Private / Public / Custom) alongside the existing name search, driven by each bucket's resolved policy type; filter state is applied on load. (Bucket access is now modeled as a typed enum server-side.)
    Downloads
  • v0.5.3 f90cadd04e
    Compare

    MyFSIO v0.5.3 Beta Pre-Release

    kqjy released this 2026-05-05 10:51:35 +00:00 | 30 commits to main since this release

    S3 API correctness & conformance

    • CompleteMultipartUpload parser: reset current_tag on End event so pretty-printed XML parses; Parts missing ETag / PartNumber are rejected with explicit errors instead of being silently accepted.
    • PUT object ACL now actually parses the request body, rejects body + x-amz-acl combinations, and validates strict ACL XML (unique children, known permissions, single-identifier Grantee with xsi:type Group / CanonicalUser); supplying an owner ID that doesn't match the existing owner returns 403. New MalformedACLError.
    • Object ACL now records the authenticated principal as owner instead of a fabricated value; ACL display-name handling fixed.
    • Canned-ACL validation tightened.
    • GET object-lock returns 404 (ObjectLockConfigurationNotFoundError) when no configuration is set, instead of fabricating a Disabled config.
    • parse_tagging_xml / parse_delete_objects now survive pretty-printed XML (accumulate text across chunks, clear current_tag on EndEvent, skip whitespace-only text).
    • parse_range: a suffix length larger than the object now returns the full object instead of underflowing.
    • aws-chunked detection refined; Content-Encoding: aws-chunked is stripped from stored metadata regardless of transport.
    • ListObjects(V2): leading-slash prefix is now allowed; a trailing-slash key is emitted as Contents when the prefix targets it; subdir dir-markers are no longer double-emitted as both Contents and CommonPrefix.
    • General S3 listing fixes.
    • SelectObject: DuckDB json extension is now loaded before parsing JSON SELECT inputs.

    Replication

    • Per-connection transfer tuning: RemoteConnection gains an optional TransferTuning (profile + numeric overrides) covering part size, multipart concurrency, read buffer, and in-MPU retries, with ssd_lan / ssd_wan / hdd_lan / hdd_wan profiles. Replication uses the resolved tuning, retries transient part failures in-place with exponential backoff, and lowers SDK transport retry to 1 when in-MPU retries > 1. Legacy connections without tuning resolve to the prior 8 MiB / 4 streams defaults.
    • UI Connections create / edit modals expose an advanced "Transfer tuning" section.

    Quota & integrity

    • Integrity scanner now issues per-type so phantom / orphan storms can no longer fill the FIFO cap before the corruption phase pushes — corrupted-object auto-heal was being silently skipped.

    Admin / IAM / auth

    • Domains nav hidden from non-admins; /ui/buckets filters the list by IAM, bucket policy, and ACL so a user with a blank policy no longer sees every bucket.

    Rate limiting

    • Rate-limiter design fixed.
    Downloads
  • v0.5.2 cbe2c9cb19
    Compare

    MyFSIO v0.5.2 Beta Pre-Release

    kqjy released this 2026-05-03 07:33:01 +00:00 | 37 commits to main since this release

    S3 API correctness & conformance

    • Admin and KMS routes moved under the /myfsio/ namespace; only the myfsio bucket name is reserved (rejected at create_bucket). Fixes S3 error codes, LocationConstraint validation, and CORS preflight XML.
    • Atomic encrypt-before-publish CopyObject: snapshot-bound source metadata; atomic SSE-S3 plaintext ETag; finalize_put_sync now reserves __etag__ / __size__ / __last_modified__ / __version_id__ from caller metadata (fixes UI copy/move metadata leak).
    • IsLatest mis-flag fix; versioned retention / legal-hold; parse_range underflow; version_id validation; bucket_detail template crash fixed.
    • General S3 API compliance pass on response headers, conditional writes, replication, and security.

    Security

    • SigV4 host enforcement.
    • SelectObject SQL sandbox; SSRF guard with DNS-aware filtering and a custom AWS SDK HttpClient; CSRF for streaming requests; aws-chunked buffer cap; session DoS cap; IAM blank-policy rejection; restore-key dispatch hardening; create_bucket race fix.
    • UI session cookie: persistent Max-Age from SESSION_LIFETIME_DAYS with sliding expiry; SameSite reverted Strict → Lax so persistent cookies survive top-level navigations (bookmarks, new tabs, inbound links) while CSRF tokens still guard mutations.

    Replication, site sync & integrity

    • Large-object replication fixed: multipart upload, per-part timeout, background healer, status metadata.
    • Bidirectional-status no longer misclassifies auth-denied-but-reachable peers as REMOTE_UNREACHABLE (redundant peer-health pre-check removed).
    • Local site endpoint auto-syncs with the runtime bind address.

    Admin / IAM / auth

    • Bucket admin mutations gated on admin: POST-only bucket routes moved into the admin sub-router; mixed-method create / acl / cors / lifecycle handlers gained inline ensure_admin checks.
    • UI list authorisation split so a bucket-ARN policy Deny still binds while IAM prefix scoping is honoured.
    • UI bucket-encryption SSE-default shape (Rules[0].ApplyServerSideEncryptionByDefault.SSEAlgorithm) parsed correctly so existing UI-configured defaults still apply at PutObject.
    • Embedded peer-credential form submissions routed back to Sites.

    UI

    • GC and integrity-history initial render respect DISPLAY_TIMEZONE — no more zone flicker after refresh.
    • Optimistic retry-card always cleared on POST return; resume kind-conflict is peeked before re-enabling.
    Downloads
  • v0.5.1 d434c95e7f
    Compare

    MyFSIO v0.5.1 Beta Pre-Release

    kqjy released this 2026-04-28 07:43:36 +00:00 | 52 commits to main since this release

    Added

    • Cluster feature — multi-site peer registry, inbound-access-key trust, and UI for managing peers.
    • Object search endpoint — server-side search with hidden internal metadata keys.
    • Embedded UI assets — Tera templates and static files baked into the binary; deploys as a single file.
    • Real auto-heal — corrupted objects are now fetched from a verified peer with atomic swap; falls back to poison when no peer is available.
    • peer_inbound_access_key exposed in /ui/sites peers JSON.

    Fixed

    S3 read-path response headers

    • Last-Modified and x-amz-meta-* on Range 206 responses.
    • x-amz-server-side-encryption on HEAD and ?partNumber= responses.
    • 304 Not Modified now carries ETag, Last-Modified, version-id, and cache headers.
    • If-Match/If-Unmodified-Since and If-None-Match/If-Modified-Since treated as RFC-9110 pairs on GET and CopyObject.
    • Same headers added to website-hosting HEAD/200/206 responses.

    Replication

    • Loop regression fixed: replicated incoming writes are detected via the authenticated principal against the peer_inbound_access_key set, replacing a forgeable User-Agent check.
    • Retry preflight now uses HeadBucket on the actual target bucket.
    • Pause/resume made idempotent.

    Integrity & object recovery

    • Auto-heal data-loss bug fixed: returns 422 ObjectCorrupted, locks the heal swap, and verifies multipart peer body.
    • Poisoned objects can now be recovered via PUT/DELETE/DeleteObjects while preserving object-lock.
    • Race-free GetObject/HeadObject ?partNumber=N with correct zero-length-part response.

    S3 semantics

    • DeleteObject(VersionId='null') now permanently deletes the null version instead of creating a delete-marker.

    Cluster / peer-site

    • Peer-site edit 422 fixed; IAM admin definition aligned across runtime, UI, and JS.
    • Legacy full-access policies auto-migrated (gated on iam:* to avoid promoting bucketadmin).
    • Empty endpoints rejected on peer-site update.
    • Peer actions dropdown no longer overflows; peer 403 response body surfaced to the user.

    UI / search

    • Search auto-pagination no longer loops on failure.
    • CSRF tokens accepted in JSON body.

    Docs

    • General docs refresh covering cluster setup and IAM admin alignment.
    Downloads
  • v0.5.0 b4e2e15936
    Compare

    MyFSIO v0.5.0 Beta Pre-Release

    kqjy released this 2026-04-27 07:00:27 +00:00 | 57 commits to main since this release

    Full migration and transition from Python to Rust update.

    Core change

    • One Rust binary, no more Python runtime.
    • Tokio + Axum 0.8, async I/O end-to-end.
    • Workspace crates: myfsio-common, myfsio-auth, myfsio-crypto, myfsio-storage, myfsio-xml, myfsio-server.
    • Rust-only Dockerfile; run.py, gunicorn/waitress, requirements.txt, and the python/ tree are gone.

    UI changes

    • Same URLs, same templates — sessions and bookmarks keep working.
    • Faster page loads; bucket listing optimized for 10K–100K objects.
    • New Cluster page for peer-site management.
    • New object search in the bucket browser.
    • Static website hosting: proper 404 + error handling.
    • Folder selection now exposes the delete button.
    • More-actions dropdown uses Popper (no row-select bug, no overflow).
    • GC and integrity scan tables auto-refresh and respect DISPLAY_TIMEZONE.
    • Internal metadata keys hidden in object panels.
    • CSRF accepted in JSON bodies; login token edge cases fixed.

    S3 API

    • SigV4 / SHA256 hot paths fixed and hardened.
    • Versioning conformance: live VersionId, delete markers, Suspended, ListVersions pagination, DeleteObject(VersionId='null') hard-deletes correctly.
    • DeleteObjects parallelized; per-op rate limits restored.
    • Multipart: race-free ?partNumber=N, zero-length-part response, peer-body verification.
    • CopyObject now streams.
    • Range GET on SSE-encrypted objects (partial decryption).
    • Per-bucket CORS, canned ACL/SSE rejection, checksum attrs, XML round-trip.
    • Path-safety and error-code conformance hardened.
    • Presigned URLs: key/user status enforced, no X-Forwarded-Host trust.

    Cluster / replication / integrity

    • Real auto-heal: peer-fetch corrupted objects with verified swap, poison-fallback if no peer, returns 422 ObjectCorrupted.
    • Bi-directional site sync (SITE_SYNC_ENABLED).
    • Peer-site editor rejects empty endpoints; peer 403 bodies surfaced in UI.
    • Integrity scanner uses intra-bucket cursor tracking for progressive coverage.

    IAM

    • Admin definition aligned across runtime, UI, and JS.
    • Legacy full-access policies auto-migrate (gated on iam:* to avoid promoting bucket-admin users).
    • --reset-cred backs up iam.json (keeps 5 newest).

    Upgrade

    • Existing data/ directories work unchanged.
    • Replace your Python entrypoint with myfsio-server serve.
    • Build: cargo build --release -p myfsio-server --bin myfsio-server.

    Notes (important):

    • Some environmental variable will need changing, such as APP_HOST to HOST. Please refer to the documentation for more details.
    Downloads
  • v0.4.2 ae11c654f9
    Compare

    MyFSIO v0.4.2 Beta Pre-Release

    kqjy released this 2026-04-01 08:37:12 +00:00 | 95 commits to main since this release

    Features

    Version Flag

    • Added --version option to run.py

    ETag Self-Healing

    • Added self-heal for missing ETags
    • Hardened ETag index persistence

    Rust Extension Staleness Detection

    • Added robust myfsio_core staleness detection with automatic Python fallback
    • Documented Rust extension build process in README

    Performance

    Reduced Per-Request Overhead

    • Pre-compiled SigV4 regex patterns
    • Added in-memory etag index cache
    • Increased GET response chunks to 1MB
    • Made metadata cache size configurable
    • Skipped fsync for rebuildable caches

    Integrity Scanner Optimizations

    • Added intra-bucket cursor tracking for progressive full coverage
    • Implemented early batch exit to avoid unnecessary work
    • Switched to lazy sorted walk for lower memory usage
    • Added cursor-aware index reads

    Security

    Presigned URL Hardening

    • Enforced key and user status checks in SigV4 presigned paths
    • Removed duplicate verification logic
    • Removed trust of X-Forwarded-Host header

    Bug Fixes

    Object List Dropdown

    • Fixed more-actions dropdown triggering row selection on object list

    Chores

    Dependencies

    • Updated requirements.txt
    Downloads