Optimize bucket listing for 10K-100K objects
- Shallow listing: read per-directory _index.json once for eTags instead of N serial .meta.json reads. Validate prefix for path traversal and verify normalized target stays within bucket root. - Recursive listing: cache full per-directory index during the walk so each _index.json is parsed at most once per call. - Per-bucket listing cache with 5s TTL and per-bucket rebuild mutex. Invalidated on put/delete/copy/metadata/tags/multipart-complete. Pagination uses partition_point for O(log n) start lookup. - UI stream endpoint now actually streams via mpsc + Body::from_stream instead of buffering into a Vec<String>. Cancels producer on client disconnect. - UI JSON endpoint honors delimiter=/ and returns common_prefixes. - run_blocking wrapper dispatches sync filesystem work via block_in_place on multi-threaded runtimes, falls back to inline on current-thread runtimes (unit tests).
This commit is contained in:
@@ -43,6 +43,7 @@ thiserror = "2"
|
||||
chrono = { version = "0.4", features = ["serde"] }
|
||||
base64 = "0.22"
|
||||
tokio-util = { version = "0.7", features = ["io"] }
|
||||
tokio-stream = "0.1"
|
||||
futures = "0.3"
|
||||
dashmap = "6"
|
||||
crc32fast = "1"
|
||||
|
||||
Reference in New Issue
Block a user