kqjy/MyFSIO
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 42 Wiki Activity

42 Releases 42 Tags

RSS Feed
  • v0.5.4 ec1fe1b1f7
    Compare

    MyFSIO v0.5.4 Beta Pre-Release

    kqjy released this 2026-06-08 07:57:32 +00:00 | 20 commits to main since this release

    Server-side encryption — SSE-C for multipart uploads

    • Per-part SSE-C for multipart uploads, with the customer key never stored. Previously the only SSE-C multipart path stored the base64 customer key in the upload manifest; that is gone. Initiate now generates a random per-object data key (ODK), wraps it under the customer key, and persists only the wrapped ODK + the customer-key MD5 in the manifest (gated behind an __mpu_sse_c__ marker, so SSE-S3 / SSE-KMS / single-PUT SSE-C / unencrypted paths are byte-for-byte unchanged).
    • Each UploadPart validates the supplied key against the stored MD5, unwraps the ODK, and encrypts the part to a self-describing block carrying a fresh per-part random salt (nonce = HKDF(ODK, part_number ‖ salt)), so re-uploading a part can never reuse an AES-GCM nonce.
    • GET / HEAD, Range, and partNumber reads decrypt per-part by re-deriving the nonce from the stored salt and mapping plaintext offsets via per-part plaintext sizes. Missing key → 400 InvalidRequest; wrong key → 403 AccessDenied (AEAD failure never surfaces as 500).
    • UploadPartCopy into an SSE-C multipart upload is rejected with 501 NotImplemented (use UploadPart), and the multipart minimum-part-size check runs against plaintext size before publish so a failed completion can't destroy a pre-existing object or leave a stray delete marker.

    S3 API correctness & conformance

    • Every error response now flows through a single error path that auto-generates a request id and emits x-amz-request-id / x-amz-id-2 headers alongside the XML body.
    • Per-object <Owner> in ListObjects / ListObjectsV2 / ListObjectVersions: owner is parsed from the object's stored __acl__ and DisplayName is resolved via a per-request IAM map. Legacy objects with no ACL fall back to a canonical myfsio owner (not the requester); V1 now always emits Owner + StorageClass per spec.
    • Added PutBucketOwnershipControls and PutPublicAccessBlock as no-op stubs (GET/PUT/DELETE) so boto3 ≥ 1.32 CreateBucket fixtures stop erroring before the test body runs.
    • Object keys that collide with an existing directory path are now storable: shadowed keys are written to a <dir>/.__myfsio_keydata__ marker, with crash-safe conversion of blocking files via the system temp dir. Listing, HEAD/GET/DELETE, and the integrity scanner all resolve through the marker.
    • Leading-slash object keys are normalized on the object data path. A key sent with a leading slash (e.g. //prefix/x on the wire) was rejected with 400 InvalidObjectKey on HEAD/GET/PUT/DELETE even though the SigV4 layer authorized it under the clean key — so clients keying not-found detection off 404/NoSuchKey saw a hard error instead of a missing object. The five object handlers now strip leading slashes to match the auth layer's key derivation, and the list-prefix handling from v0.5.3 was reworked to share the same normalization (internal double-slashes and .. traversal handling unchanged).
    • 416 Range Not Satisfiable now returns a Content-Range header; UploadPart validates the part number is within 1..=10000.
    • S3 Select: sets memory_limit / max_memory and fixes a UTF-8 chunk-boundary panic.

    Replication

    • Resumable, partial-failure-tolerant multipart replication. Failed attempts now persist the upload_id plus a source-identity fingerprint (size, etag, part size); a retry validates the fingerprint, calls ListParts to discover already-uploaded parts, and resumes the same MPU instead of restarting from byte zero (part ETags preserved verbatim). Stale ids — NoSuchUpload, changed source content/tuning, or permanent CompleteMultipartUpload errors — are aborted server-side and cleared so no retry ever reuses a dead id.
    • Per-part failures within a pass are collected and retried in a second in-MPU pass; a permanent error or task panic immediately aborts sibling parts and the MPU, so a missing/panicked part can never slip into CompleteMultipartUpload.
    • Progress-aware stall detection: each part body streams through a progress reader paired with a tokio::select! watchdog. REPLICATION_PART_STALL_TIMEOUT_SECONDS (default 300s) now bounds both "no first byte" and "mid-transfer stall" cases, replacing the coarse Smithy read-timeout for stuck-upload detection on HDD/WAN paths.
    • Failure-store writes now hold the cache lock across the read-modify-write and persist atomically (temp + rename); an RAII in-flight guard prevents a panicking replicate task from leaking in-flight counters or wedging a bucket's batch run; NoSuchBucket is classified via the structured SDK error code.

    Integrity & garbage collection

    • HDD-friendly scanning. INTEGRITY_HEAL_CONCURRENCY now defaults to 1 (serial heal — parallel hashing thrashes a single spindle), and a new INTEGRITY_SCAN_PACING_MS (default 0) applies a per-object pacing delay every ~100 objects across all scan phases to relieve disk-head pressure.
    • SSE-encrypted objects are skipped in the corruption scan. Comparing a plaintext ETag against ciphertext produced a false positive that, combined with auto-heal, was quarantining and poisoning every encrypted object.
    • Corruption mismatches are now re-verified under the object write lock before quarantine, with a grace-period guard so versions mid-archive aren't quarantined; the in-progress guard is hardened against the never-polled-task window and objects_scanned is de-duplicated.
    • Stuck "scan in progress" banner fixed. Integrity/GC run state leaked when an HTTP request was cancelled mid-run, leaving the banner stuck forever. Runs now execute on a detached task with an RAII guard that always clears the running/started-at state, regardless of client cancellation.

    Security & auth

    • Bucket-policy matching is now gated by request method. A read grant (s3:GetBucketCors / Lifecycle / Replication / …) no longer authorizes the corresponding PUT/DELETE. Allow-only semantics, IAM, and wildcards are unchanged.
    • SSE-C customer-key MD5 is persisted and constant-time verified on GET/HEAD.
    • DeleteObjects honors x-amz-bypass-governance-retention.
    • Buffered upload paths (checksummed PUT, POST form) are now size-bounded to prevent OOM; tag key/value are XML-escaped in GetBucketTagging / GetObjectTagging; the replication-UI failure endpoints validate the bucket name (path-traversal / cross-bucket tampering).
    • The streaming-SigV4 acceptance warning is demoted to once per process instead of one line per upload.

    Web UI

    • The /ui/buckets page gains an access filter (All / Private / Public / Custom) alongside the existing name search, driven by each bucket's resolved policy type; filter state is applied on load. (Bucket access is now modeled as a typed enum server-side.)
    Downloads