Block a user
4225182b8e
Harden authorization, object lock, storage commits, background services, secrets at rest, website serving, request limits, sessions, and the installer. x-amz-bypass-governance-retention is now honored only for admins or principals granted the new bypass_governance IAM action (or s3:BypassGovernanceRetention in a bucket policy), evaluated per key on bulk delete, so callers with plain delete access can no longer strip GOVERNANCE retention; is_admin requires an unrestricted policy prefix so a prefix-scoped wildcard policy no longer short-circuits every authorization check; bucket-policy Resource keys match case-sensitively while bucket and Action matching stay case-insensitive; and Date-based lifecycle rules are a no-op until the date passes instead of using the configured date as an age cutoff. PutBucketObjectLockConfiguration validates the XML and requires versioning (409 InvalidBucketState), and its DefaultRetention is actually applied at every user-facing object-creation path with explicit headers winning and replication/peer-pull/restore paths untouched. Retention and legal-hold updates are check-and-set under the per-object stripe lock via new update_object_retention/update_object_legal_hold trait methods, so concurrent requests can no longer shorten COMPLIANCE retention and legal-hold writes no longer clobber concurrently written metadata; same-mode extension of either mode needs no bypass while shortening, removal and mode changes keep the bypass rules. A failed PUT commit now rolls back the archived version and metadata sidecar, and the archived-null purge runs after the rename so a failed commit can never destroy the prior null version. Site sync aborts the cycle on any remote-listing failure instead of treating NoSuchBucket/404 as an empty bucket (sync_deletions could erase every locally synchronized object) and records last_error/last_error_at per bucket so failing cycles are visible on the Sites page; GC skips a bucket's segment sweep whenever the reference scan hit a read error and reports the skip instead of deleting live segment data on an incomplete scan; and the process exits non-zero when either listener task dies so systemd restarts it. IAM mutations are serialized behind a mutation lock and persisted atomically (tmp+fsync+rename), first-run and reset admin credentials are written encrypted with a one-time plaintext-to-encrypted migration at startup and env-provided secrets no longer echoed to stdout, SECRET_KEY auto-generation to .myfsio.sys/config/.secret is implemented with dev-secret-key rejected from both env and file, and iam.json, KMS master keys, kms_keys.json, .connections_key, connections.json and .secret are all written owner-only through shared atomic helpers. Static website hosting streams through the extracted handlers/object_read data plane: SSE-S3/SSE-KMS assets that were served as ciphertext are decrypted, HEAD reports plaintext Content-Length, 206 ranges are computed from plaintext offsets, SSE-C assets return 403, and no path buffers whole objects in memory; ranged S3 GETs validate SSE-C keys like the whole-object path (400/403 instead of 500). Every config/XML/JSON body ingress is capped (1 MiB config XML and Select, 8 MiB CompleteMultipartUpload/DeleteObjects, 1-2 MiB JSON, bounded relay-outbound, 1 MiB non-file multipart form fields) with a new MaxMessageLengthExceeded error, and the UI part upload streams to disk with a 5 GiB cap instead of buffering each part twice in memory. Anonymous requests use ephemeral sessions that never enter the session store, capacity eviction prefers unauthenticated sessions, GET/POST /login is rate limited per IP (RATE_LIMIT_UI_LOGIN, default 20 per minute) with a styled 429 page and Retry-After, and failed UI uploads surface the real S3 reason in the toast and the upload dialog instead of a bare failure count. install.sh and uninstall.sh canonicalize and refuse system paths, guard foreign data directories behind --adopt-data-dir, track installer-created users in a manifest so uninstall preserves pre-existing accounts, and re-installs preserve myfsio.env (--overwrite-env backs up then regenerates). Adds handlers/object_read.rs, myfsio_common::fs_util, login_rate_limited.html, the bypass_governance IAM action, S3ErrorCode::{MaxMessageLengthExceeded,InvalidBucketState}, StorageError::InvalidArgument and RATE_LIMIT_UI_LOGIN; ~90 new regression tests.
f1c2ff4b8b
Reject unknown encoding-type and cap max-keys at 1000; fix dead cluster-page JS, copy/move destination default, audio preview stage height, and UI polish
d54370a026
Keep hardware-RAID SMART alive across reboots with an ExecStartPre ioctl-node fixup and classify probe failures by opening the nodes directly, seed pending backup status from backup init so fresh installs show scheduled repos, treat storage nodes as first-class destinations in backup onboarding and list existing repositories even when the server has no backend, and single-quote install snippets; bump to 0.4.4
42c5bb3b9c
Fix three P0 authorization defects: bucket-name path escape, and query-selector precedence mismatch at both the bucket and object level. Enforce full S3 bucket-name syntax plus canonical containment inside the storage backend (require_bucket/bucket_exists/create_bucket/multipart_upload_dir/list_multipart_uploads/is_versioning_enabled/get_versioning_status) instead of only in create_bucket, so a percent-encoded absolute path in the bucket position can no longer replace STORAGE_ROOT; percent-decode bucket, key and copy-source in the auth middleware so it authorizes the same strings the handler acts on; guard the service-layer joins the UI reaches without the storage backend (lifecycle history, replication ledger, archived/restore paths). Replace the divergent query scans with one BucketSubresource and one ObjectSubresource parser shared by the middleware and every dispatcher: multiple selectors are rejected with InvalidArgument at authorization time, non-dispatchable selectors return MethodNotAllowed instead of falling through, and a non-dispatchable pair authorizes as the method default so it is never weaker than the request. Closes the authenticated list-to-policy escalation, its unauthenticated variant on public-read buckets, and a read-only principal overwriting or deleting objects via PUT/DELETE ?attributes|?select|?uploads and DELETE ?retention. ?ownershipControls/?publicAccessBlock gain dedicated ownership_controls/public_access_block IAM actions; PutBucketLogging TargetBucket and website-domain mappings reject invalid names directly; replication validates only the source bucket strictly (a remote target may carry a name this server would not issue); ListBuckets and --rebuild-listing skip invalid-named directories with a warning instead of trapping or failing. Removes now-dead has_query_key; adds validation::bucket_name_rejection and 7 regression tests.
5238eadb79
Fix path traversal, authorization, and fail-open defects across storage, auth, and UI: validate multipart uploadId/segment ids as 32-hex before any filesystem join (abort_multipart previously remove_dir_all'd caller-controlled paths, destroying .myfsio.sys/config in one request); reject reserved __/x-amz- metadata keys at all five ingress points incl. CopyObject, UI multipart init, and peer pull (x-amz-meta-__segments__ reached recursive segment deletion and could redirect segment reads); validate UI restore version_id to a single path component; re-authorize every key in DeleteObjects (a prefix-scoped principal could delete outside its prefix); fail closed on bucket-policy Condition/NotPrincipal/NotAction/NotResource at evaluation and reject them at both write paths; require presigned requests to sign every x-amz-* header except content-sha256/date/decoded-content-length, closing copy-source, ACL, SSE, and governance-bypass injection by URL bearers; exit non-zero when ENCRYPTION_ENABLED/KMS_ENABLED init fails and fail closed on bucket-default encryption instead of silently storing plaintext; route IAM disable-user/create-key/delete-key through load_config/save_config so revocation works against an encrypted iam.json; treat an unreadable .bucket.json as fail-closed across policy eval, object write/delete, and all config writes rather than as an empty config; replace data-carrying inline onclick handlers with delegated data-* listeners (verified in-browser: a key containing quote-paren no longer executes on Copy/Move/Retry/Dismiss); and add missing bucket authorization to 11 UI handlers. Replace admin-only gating of GC and integrity with new system:gc_read/gc_run/integrity_read/integrity_run IAM actions on both the admin API and the UI, moving the seven /ui/system routes out of the admin-only router group so the per-handler check applies and each dashboard card names its missing permission; generalize action_matches from a hardcoded iam:* to any namespace:*. Adds validation::{is_valid_multipart_id,is_reserved_metadata_key,is_reserved_user_metadata_key,is_safe_path_segment} and BucketConfig::unreadable; 10 new tests, all 632 green; docs.md synced
e035e5671b
Full checksum + streaming-SigV4 validation, accurate quota + O(1) bucket counters, background listing compaction, durable replication ledger, LRU meta cache, and UI polish. Verify crc32c/sha1/crc64nvme (crc-fast+sha1) alongside md5/sha256/crc32 on every upload path and parse+verify the aws-chunked TRAILER section (was discarded); gate x-amz-checksum-* on GET/HEAD behind x-amz-checksum-mode: ENABLED; implement real per-chunk streaming-SigV4 signature-chain + final-zero-chunk + signed-trailer validation, now default on (STRICT_STREAMING_SIGV4=true; false is a warn-passthrough escape hatch, checksum trailers still verified; strict mode rejects streaming-declared requests with no signing context). Fix quota undercounting for versioned/suspended overwrites and route quota-fallback PUTs around the stats cache; unknown bucket-subresource GET/DELETE now 501 instead of returning a listing. Move listing-index compaction off the PUT/DELETE request path to a backend background worker with generation-numbered journals (journal.<gen>.jsonl), seal-under-lock/snapshot-outside-lock, and drain-before-shutdown; add persistent per-bucket counters (live objects/bytes, version count/bytes, delete-marker count) to the snapshot for O(1) quota + bucket_stats and parallelize UI buckets/dashboard/cluster walks with buffer_unordered(4). Add durable per-bucket replication pending-ledger (.myfsio.sys/buckets/<b>/replication/pending.{snapshot.json,journal.jsonl}) with append->PENDING->remote-success->COMPLETED->ack ordering, startup replay with zero sidecar scan, one-time seeding migration, and optional REPLICATION_FULL_RECONCILE_INTERVAL_HOURS (default 0=off). Replace the 100-entry DashMap meta-read cache with a real lru LruCache (OBJECT_CACHE_MAX_SIZE default 100->1024). UI: presign preset + custom-seconds with inline min/max validation that blocks out-of-range (was silently clamping), create-bucket autofocus + enable-versioning checkbox, AJAX 401 -> /login?next=, single shared binary-unit byte formatter (ui_format.rs; killed 8+ duplicates), server-formatted metrics 'As of' in DISPLAY_TIMEZONE, quota card split into Objects/Versions/Total-stored, 'Error'-titled error toasts, and stable object-preview layout with %PDF- magic-byte sniff + styled 'Preview unavailable' fallback. Adds services/replication_ledger.rs and ui_format.rs; README + docs.md synced
e035e5671b
Full checksum + streaming-SigV4 validation, accurate quota + O(1) bucket counters, background listing compaction, durable replication ledger, LRU meta cache, and UI polish. Verify crc32c/sha1/crc64nvme (crc-fast+sha1) alongside md5/sha256/crc32 on every upload path and parse+verify the aws-chunked TRAILER section (was discarded); gate x-amz-checksum-* on GET/HEAD behind x-amz-checksum-mode: ENABLED; implement real per-chunk streaming-SigV4 signature-chain + final-zero-chunk + signed-trailer validation, now default on (STRICT_STREAMING_SIGV4=true; false is a warn-passthrough escape hatch, checksum trailers still verified; strict mode rejects streaming-declared requests with no signing context). Fix quota undercounting for versioned/suspended overwrites and route quota-fallback PUTs around the stats cache; unknown bucket-subresource GET/DELETE now 501 instead of returning a listing. Move listing-index compaction off the PUT/DELETE request path to a backend background worker with generation-numbered journals (journal.<gen>.jsonl), seal-under-lock/snapshot-outside-lock, and drain-before-shutdown; add persistent per-bucket counters (live objects/bytes, version count/bytes, delete-marker count) to the snapshot for O(1) quota + bucket_stats and parallelize UI buckets/dashboard/cluster walks with buffer_unordered(4). Add durable per-bucket replication pending-ledger (.myfsio.sys/buckets/<b>/replication/pending.{snapshot.json,journal.jsonl}) with append->PENDING->remote-success->COMPLETED->ack ordering, startup replay with zero sidecar scan, one-time seeding migration, and optional REPLICATION_FULL_RECONCILE_INTERVAL_HOURS (default 0=off). Replace the 100-entry DashMap meta-read cache with a real lru LruCache (OBJECT_CACHE_MAX_SIZE default 100->1024). UI: presign preset + custom-seconds with inline min/max validation that blocks out-of-range (was silently clamping), create-bucket autofocus + enable-versioning checkbox, AJAX 401 -> /login?next=, single shared binary-unit byte formatter (ui_format.rs; killed 8+ duplicates), server-formatted metrics 'As of' in DISPLAY_TIMEZONE, quota card split into Objects/Versions/Total-stored, 'Error'-titled error toasts, and stable object-preview layout with %PDF- magic-byte sniff + styled 'Preview unavailable' fallback. Adds services/replication_ledger.rs and ui_format.rs; README + docs.md synced
6f63a8594d
Carry backup repository facts through failed runs, add editable repo quotas, and drill into SMART counter history; bump to 0.4.2
f5d5f08399
Add persistent per-bucket listing index + harden auth/storage/UI: LISTING_INDEX_ENABLED (default on) serves flat ListObjectsV2 O(page) from an ordered index persisted as checksummed snapshot.json + append-only journal.jsonl under .myfsio.sys/buckets/<bucket>/listing/, built lazily once and updated incrementally at storage-commit hooks (put/copy/multipart-complete/deletes/versioned ops incl. internal writers) with dirty-mark->rebuild escape hatch, compaction, and legacy-walk fallback on any error; sidecars stay authoritative (index is derived/disposable), --rebuild-listing one-shot CLI; SigV4 + presigned expiry fail closed on malformed timestamps, IAM expires_at fails closed on unparseable values, access log redacts X-Amz-Signature/security-token (presigned URLs were replayable from logs), relay request/response bodies bounded to 8 MiB; atomic per-bucket config read-modify-write via mutate_bucket_config (fixes lost-update race across concurrent subresource writes, 24 handlers refactored), finalize_put_sync rejects corrupt metadata before overwrite and writes+fsyncs sidecar before publishing bytes, directory fsync failures propagate; UI: inline create-bucket validation (was silent no-op), row-actions Delete un-clipped via body-reparented dropdown, lifecycle/CORS rule deletes confirm, keyboard-accessible bucket cards, danger/info toast colors, secret-toggle aria-label, capped system history, upload toast pluralization; 9 new storage tests, all 488 tests green; docs + UI docs synced
c980f29aad
Harden storage engine: stream checksum validation on PUT/UploadPart/POST (no 5GiB RAM buffering), enforce conditional writes + object-lock retention + bucket quotas atomically at the storage commit (fixes TOCTOU races incl. internal writers), encrypt-before-commit SSE PUT with single-write ciphertext install + chunked streaming decryption for full/range/MPU-SSE-C GETs, fsync data/segments/dirs before ack, bounded replication queue with workers + reconciliation + overflow journaling, hard-link version snapshots, range-windowed segment opens (narrow reads of 10k-part objects open ~2 files), opt-in HDD admission control (HDD_READ/WRITE_CONCURRENCY, 503 SlowDown on queue timeout) with disk-pressure metrics + UI card; docs + 11 e2e hardening tests
3b24108579
Replace aggregate _index.json metadata with per-object sidecar files (METADATA_LAYOUT=sidecar): O(1) metadata writes, sidecar>index>legacy read order forever, fail-closed corruption handling (422 + RMW write guard + listing blanking) replacing the corrupt-index empty-map destruction bug, --migrate-meta one-shot CLI with no-rollback warning and startup notice, integrity scanner sidecar precedence; add S3 IncompleteBody error code with declared-length enforcement across PutObject/UploadPart/SSE-C/checksum paths (short aws-chunked bodies previously stored truncated with 200) incl. transport-error mapping; docs + UI docs synced
15c2ddcf51
Sync all docs to code (rate limits, quota/XML APIs, GC+integrity tables, metrics endpoints, segments layout, peer-cred scope), vendor Bootstrap 5.3.2 for offline deployments, fix uptime epoch to server boot with h/m granularity, compact bucket list view, opaque sticky object-table header, floor metrics interval env vars, and delete dead classify_endpoint
v1.5.0: PWA offline support, universal search, timezone-aware reviews, and Anki import