Compare commits
2 Commits
9ec5797919
...
217af6d1c6
| Author | SHA1 | Date | |
|---|---|---|---|
| 217af6d1c6 | |||
| 51d54b42ac |
@@ -3,7 +3,7 @@
|
|||||||
logs
|
logs
|
||||||
data
|
data
|
||||||
tmp
|
tmp
|
||||||
myfsio-engine/target
|
target
|
||||||
myfsio-engine/tests
|
crates/*/tests
|
||||||
Dockerfile
|
Dockerfile
|
||||||
.dockerignore
|
.dockerignore
|
||||||
6
.gitignore
vendored
6
.gitignore
vendored
@@ -26,12 +26,8 @@ dist/
|
|||||||
*.egg-info/
|
*.egg-info/
|
||||||
.eggs/
|
.eggs/
|
||||||
|
|
||||||
# Rust / maturin build artifacts
|
|
||||||
python/myfsio_core/target/
|
|
||||||
python/myfsio_core/Cargo.lock
|
|
||||||
|
|
||||||
# Rust engine build artifacts
|
# Rust engine build artifacts
|
||||||
rust/myfsio-engine/target/
|
target/
|
||||||
|
|
||||||
# Local runtime artifacts
|
# Local runtime artifacts
|
||||||
logs/
|
logs/
|
||||||
|
|||||||
14
rust/myfsio-engine/Cargo.lock → Cargo.lock
generated
14
rust/myfsio-engine/Cargo.lock → Cargo.lock
generated
@@ -2542,6 +2542,15 @@ version = "0.1.2"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
|
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "matchers"
|
||||||
|
version = "0.2.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9"
|
||||||
|
dependencies = [
|
||||||
|
"regex-automata",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "matchit"
|
name = "matchit"
|
||||||
version = "0.8.4"
|
version = "0.8.4"
|
||||||
@@ -2724,6 +2733,7 @@ dependencies = [
|
|||||||
"roxmltree",
|
"roxmltree",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
|
"serde_urlencoded",
|
||||||
"sha2 0.10.9",
|
"sha2 0.10.9",
|
||||||
"subtle",
|
"subtle",
|
||||||
"sysinfo",
|
"sysinfo",
|
||||||
@@ -4333,10 +4343,14 @@ version = "0.3.23"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319"
|
checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
"matchers",
|
||||||
"nu-ansi-term",
|
"nu-ansi-term",
|
||||||
|
"once_cell",
|
||||||
|
"regex-automata",
|
||||||
"sharded-slab",
|
"sharded-slab",
|
||||||
"smallvec",
|
"smallvec",
|
||||||
"thread_local",
|
"thread_local",
|
||||||
|
"tracing",
|
||||||
"tracing-core",
|
"tracing-core",
|
||||||
"tracing-log",
|
"tracing-log",
|
||||||
]
|
]
|
||||||
@@ -38,7 +38,7 @@ percent-encoding = "2"
|
|||||||
regex = "1"
|
regex = "1"
|
||||||
unicode-normalization = "0.1"
|
unicode-normalization = "0.1"
|
||||||
tracing = "0.1"
|
tracing = "0.1"
|
||||||
tracing-subscriber = "0.3"
|
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||||
thiserror = "2"
|
thiserror = "2"
|
||||||
chrono = { version = "0.4", features = ["serde"] }
|
chrono = { version = "0.4", features = ["serde"] }
|
||||||
base64 = "0.22"
|
base64 = "0.22"
|
||||||
@@ -6,10 +6,10 @@ RUN apt-get update \
|
|||||||
&& apt-get install -y --no-install-recommends build-essential pkg-config libssl-dev \
|
&& apt-get install -y --no-install-recommends build-essential pkg-config libssl-dev \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
COPY myfsio-engine ./myfsio-engine
|
COPY Cargo.toml Cargo.lock ./
|
||||||
|
COPY crates ./crates
|
||||||
|
|
||||||
RUN cd myfsio-engine \
|
RUN cargo build --release --bin myfsio-server \
|
||||||
&& cargo build --release --bin myfsio-server \
|
|
||||||
&& strip target/release/myfsio-server
|
&& strip target/release/myfsio-server
|
||||||
|
|
||||||
|
|
||||||
@@ -24,9 +24,9 @@ RUN apt-get update \
|
|||||||
&& useradd -m -u 1000 myfsio \
|
&& useradd -m -u 1000 myfsio \
|
||||||
&& chown -R myfsio:myfsio /app
|
&& chown -R myfsio:myfsio /app
|
||||||
|
|
||||||
COPY --from=builder /build/myfsio-engine/target/release/myfsio-server /usr/local/bin/myfsio-server
|
COPY --from=builder /build/target/release/myfsio-server /usr/local/bin/myfsio-server
|
||||||
COPY --from=builder /build/myfsio-engine/crates/myfsio-server/templates /app/templates
|
COPY --from=builder /build/crates/myfsio-server/templates /app/templates
|
||||||
COPY --from=builder /build/myfsio-engine/crates/myfsio-server/static /app/static
|
COPY --from=builder /build/crates/myfsio-server/static /app/static
|
||||||
COPY docker-entrypoint.sh /app/docker-entrypoint.sh
|
COPY docker-entrypoint.sh /app/docker-entrypoint.sh
|
||||||
|
|
||||||
RUN chmod +x /app/docker-entrypoint.sh \
|
RUN chmod +x /app/docker-entrypoint.sh \
|
||||||
21
README.md
21
README.md
@@ -1,8 +1,6 @@
|
|||||||
# MyFSIO
|
# MyFSIO
|
||||||
|
|
||||||
MyFSIO is an S3-compatible object storage server with a Rust runtime and a filesystem-backed storage engine. The active server lives under `rust/myfsio-engine` and serves both the S3 API and the built-in web UI from a single process.
|
MyFSIO is an S3-compatible object storage server with a Rust runtime and a filesystem-backed storage engine. The repository root is the Cargo workspace; the server serves both the S3 API and the built-in web UI from a single process.
|
||||||
|
|
||||||
The `python/` implementation is deprecated as of 2026-04-21. It remains in the repository for migration reference and legacy tests, but new development and supported runtime usage should target the Rust server.
|
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
@@ -29,7 +27,6 @@ If you want API-only mode, set `UI_ENABLED=false`. There is no separate "UI-only
|
|||||||
From the repository root:
|
From the repository root:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo run -p myfsio-server --
|
cargo run -p myfsio-server --
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -60,14 +57,13 @@ UI_ENABLED=false cargo run -p myfsio-server --
|
|||||||
## Building a Binary
|
## Building a Binary
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo build --release -p myfsio-server
|
cargo build --release -p myfsio-server
|
||||||
```
|
```
|
||||||
|
|
||||||
Binary locations:
|
Binary locations:
|
||||||
|
|
||||||
- Linux/macOS: `rust/myfsio-engine/target/release/myfsio-server`
|
- Linux/macOS: `target/release/myfsio-server`
|
||||||
- Windows: `rust/myfsio-engine/target/release/myfsio-server.exe`
|
- Windows: `target/release/myfsio-server.exe`
|
||||||
|
|
||||||
Run the built binary directly:
|
Run the built binary directly:
|
||||||
|
|
||||||
@@ -166,10 +162,10 @@ data/
|
|||||||
|
|
||||||
## Docker
|
## Docker
|
||||||
|
|
||||||
Build the Rust image from the `rust/` directory:
|
Build the Rust image from the repository root:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker build -t myfsio ./rust
|
docker build -t myfsio .
|
||||||
docker run --rm -p 5000:5000 -p 5100:5100 -v "${PWD}/data:/app/data" myfsio
|
docker run --rm -p 5000:5000 -p 5100:5100 -v "${PWD}/data:/app/data" myfsio
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -180,11 +176,9 @@ If the instance sits behind a reverse proxy, set `API_BASE_URL` to the public S3
|
|||||||
The repository includes `scripts/install.sh` for systemd-style Linux installs. Build the Rust binary first, then pass it to the installer:
|
The repository includes `scripts/install.sh` for systemd-style Linux installs. Build the Rust binary first, then pass it to the installer:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo build --release -p myfsio-server
|
cargo build --release -p myfsio-server
|
||||||
|
|
||||||
cd ../..
|
sudo ./scripts/install.sh --binary ./target/release/myfsio-server
|
||||||
sudo ./scripts/install.sh --binary ./rust/myfsio-engine/target/release/myfsio-server
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The installer copies the binary into `/opt/myfsio/myfsio`, writes `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` unit.
|
The installer copies the binary into `/opt/myfsio/myfsio`, writes `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` unit.
|
||||||
@@ -194,7 +188,6 @@ The installer copies the binary into `/opt/myfsio/myfsio`, writes `/opt/myfsio/m
|
|||||||
Run the Rust test suite from the workspace:
|
Run the Rust test suite from the workspace:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo test
|
cargo test
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -209,4 +202,4 @@ cargo test
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The `version` field comes from the Rust crate version in `rust/myfsio-engine/crates/myfsio-server/Cargo.toml`.
|
The `version` field comes from the Rust crate version in `crates/myfsio-server/Cargo.toml`.
|
||||||
|
|||||||
@@ -82,11 +82,35 @@ impl EncryptionMetadata {
|
|||||||
pub struct EncryptionService {
|
pub struct EncryptionService {
|
||||||
master_key: [u8; 32],
|
master_key: [u8; 32],
|
||||||
kms: Option<std::sync::Arc<KmsService>>,
|
kms: Option<std::sync::Arc<KmsService>>,
|
||||||
|
config: EncryptionConfig,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy)]
|
||||||
|
pub struct EncryptionConfig {
|
||||||
|
pub chunk_size: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for EncryptionConfig {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self { chunk_size: 65_536 }
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl EncryptionService {
|
impl EncryptionService {
|
||||||
pub fn new(master_key: [u8; 32], kms: Option<std::sync::Arc<KmsService>>) -> Self {
|
pub fn new(master_key: [u8; 32], kms: Option<std::sync::Arc<KmsService>>) -> Self {
|
||||||
Self { master_key, kms }
|
Self::with_config(master_key, kms, EncryptionConfig::default())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn with_config(
|
||||||
|
master_key: [u8; 32],
|
||||||
|
kms: Option<std::sync::Arc<KmsService>>,
|
||||||
|
config: EncryptionConfig,
|
||||||
|
) -> Self {
|
||||||
|
Self {
|
||||||
|
master_key,
|
||||||
|
kms,
|
||||||
|
config,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn generate_data_key(&self) -> ([u8; 32], [u8; 12]) {
|
pub fn generate_data_key(&self) -> ([u8; 32], [u8; 12]) {
|
||||||
@@ -192,7 +216,10 @@ impl EncryptionService {
|
|||||||
let op = output_path.to_owned();
|
let op = output_path.to_owned();
|
||||||
let ak = actual_key;
|
let ak = actual_key;
|
||||||
let n = nonce;
|
let n = nonce;
|
||||||
tokio::task::spawn_blocking(move || encrypt_stream_chunked(&ip, &op, &ak, &n, None))
|
let chunk_size = self.config.chunk_size;
|
||||||
|
tokio::task::spawn_blocking(move || {
|
||||||
|
encrypt_stream_chunked(&ip, &op, &ak, &n, Some(chunk_size))
|
||||||
|
})
|
||||||
.await
|
.await
|
||||||
.map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??;
|
.map_err(|e| CryptoError::Io(std::io::Error::new(std::io::ErrorKind::Other, e)))??;
|
||||||
|
|
||||||
@@ -19,6 +19,7 @@ hyper = { workspace = true }
|
|||||||
bytes = { workspace = true }
|
bytes = { workspace = true }
|
||||||
serde = { workspace = true }
|
serde = { workspace = true }
|
||||||
serde_json = { workspace = true }
|
serde_json = { workspace = true }
|
||||||
|
serde_urlencoded = "0.7"
|
||||||
tracing = { workspace = true }
|
tracing = { workspace = true }
|
||||||
tracing-subscriber = { workspace = true }
|
tracing-subscriber = { workspace = true }
|
||||||
tokio-util = { workspace = true }
|
tokio-util = { workspace = true }
|
||||||
564
crates/myfsio-server/src/config.rs
Normal file
564
crates/myfsio-server/src/config.rs
Normal file
@@ -0,0 +1,564 @@
|
|||||||
|
use std::net::SocketAddr;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub struct RateLimitSetting {
|
||||||
|
pub max_requests: u32,
|
||||||
|
pub window_seconds: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RateLimitSetting {
|
||||||
|
pub const fn new(max_requests: u32, window_seconds: u64) -> Self {
|
||||||
|
Self {
|
||||||
|
max_requests,
|
||||||
|
window_seconds,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct ServerConfig {
|
||||||
|
pub bind_addr: SocketAddr,
|
||||||
|
pub ui_bind_addr: SocketAddr,
|
||||||
|
pub storage_root: PathBuf,
|
||||||
|
pub region: String,
|
||||||
|
pub iam_config_path: PathBuf,
|
||||||
|
pub sigv4_timestamp_tolerance_secs: u64,
|
||||||
|
pub presigned_url_min_expiry: u64,
|
||||||
|
pub presigned_url_max_expiry: u64,
|
||||||
|
pub secret_key: Option<String>,
|
||||||
|
pub encryption_enabled: bool,
|
||||||
|
pub encryption_chunk_size_bytes: usize,
|
||||||
|
pub kms_enabled: bool,
|
||||||
|
pub kms_generate_data_key_min_bytes: usize,
|
||||||
|
pub kms_generate_data_key_max_bytes: usize,
|
||||||
|
pub gc_enabled: bool,
|
||||||
|
pub gc_interval_hours: f64,
|
||||||
|
pub gc_temp_file_max_age_hours: f64,
|
||||||
|
pub gc_multipart_max_age_days: u64,
|
||||||
|
pub gc_lock_file_max_age_hours: f64,
|
||||||
|
pub gc_dry_run: bool,
|
||||||
|
pub integrity_enabled: bool,
|
||||||
|
pub metrics_enabled: bool,
|
||||||
|
pub metrics_history_enabled: bool,
|
||||||
|
pub metrics_interval_minutes: u64,
|
||||||
|
pub metrics_retention_hours: u64,
|
||||||
|
pub metrics_history_interval_minutes: u64,
|
||||||
|
pub metrics_history_retention_hours: u64,
|
||||||
|
pub lifecycle_enabled: bool,
|
||||||
|
pub lifecycle_max_history_per_bucket: usize,
|
||||||
|
pub website_hosting_enabled: bool,
|
||||||
|
pub object_key_max_length_bytes: usize,
|
||||||
|
pub object_tag_limit: usize,
|
||||||
|
pub object_cache_max_size: usize,
|
||||||
|
pub bucket_config_cache_ttl_seconds: f64,
|
||||||
|
pub replication_connect_timeout_secs: u64,
|
||||||
|
pub replication_read_timeout_secs: u64,
|
||||||
|
pub replication_max_retries: u32,
|
||||||
|
pub replication_streaming_threshold_bytes: u64,
|
||||||
|
pub replication_max_failures_per_bucket: usize,
|
||||||
|
pub site_sync_enabled: bool,
|
||||||
|
pub site_sync_interval_secs: u64,
|
||||||
|
pub site_sync_batch_size: usize,
|
||||||
|
pub site_sync_connect_timeout_secs: u64,
|
||||||
|
pub site_sync_read_timeout_secs: u64,
|
||||||
|
pub site_sync_max_retries: u32,
|
||||||
|
pub site_sync_clock_skew_tolerance: f64,
|
||||||
|
pub site_id: Option<String>,
|
||||||
|
pub site_endpoint: Option<String>,
|
||||||
|
pub site_region: String,
|
||||||
|
pub site_priority: i32,
|
||||||
|
pub api_base_url: String,
|
||||||
|
pub num_trusted_proxies: usize,
|
||||||
|
pub allowed_redirect_hosts: Vec<String>,
|
||||||
|
pub allow_internal_endpoints: bool,
|
||||||
|
pub cors_origins: Vec<String>,
|
||||||
|
pub cors_methods: Vec<String>,
|
||||||
|
pub cors_allow_headers: Vec<String>,
|
||||||
|
pub cors_expose_headers: Vec<String>,
|
||||||
|
pub session_lifetime_days: u64,
|
||||||
|
pub log_level: String,
|
||||||
|
pub multipart_min_part_size: u64,
|
||||||
|
pub bulk_delete_max_keys: usize,
|
||||||
|
pub stream_chunk_size: usize,
|
||||||
|
pub ratelimit_default: RateLimitSetting,
|
||||||
|
pub ratelimit_admin: RateLimitSetting,
|
||||||
|
pub ratelimit_storage_uri: String,
|
||||||
|
pub ui_enabled: bool,
|
||||||
|
pub templates_dir: PathBuf,
|
||||||
|
pub static_dir: PathBuf,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ServerConfig {
|
||||||
|
pub fn from_env() -> Self {
|
||||||
|
let host = std::env::var("HOST").unwrap_or_else(|_| "127.0.0.1".to_string());
|
||||||
|
let port: u16 = std::env::var("PORT")
|
||||||
|
.unwrap_or_else(|_| "5000".to_string())
|
||||||
|
.parse()
|
||||||
|
.unwrap_or(5000);
|
||||||
|
let host_ip: std::net::IpAddr = host.parse().unwrap();
|
||||||
|
let bind_addr = SocketAddr::new(host_ip, port);
|
||||||
|
let ui_port: u16 = std::env::var("UI_PORT")
|
||||||
|
.unwrap_or_else(|_| "5100".to_string())
|
||||||
|
.parse()
|
||||||
|
.unwrap_or(5100);
|
||||||
|
let storage_root = std::env::var("STORAGE_ROOT").unwrap_or_else(|_| "./data".to_string());
|
||||||
|
let region = std::env::var("AWS_REGION").unwrap_or_else(|_| "us-east-1".to_string());
|
||||||
|
|
||||||
|
let storage_path = PathBuf::from(&storage_root);
|
||||||
|
let iam_config_path = std::env::var("IAM_CONFIG")
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| {
|
||||||
|
storage_path
|
||||||
|
.join(".myfsio.sys")
|
||||||
|
.join("config")
|
||||||
|
.join("iam.json")
|
||||||
|
});
|
||||||
|
|
||||||
|
let sigv4_timestamp_tolerance_secs: u64 =
|
||||||
|
std::env::var("SIGV4_TIMESTAMP_TOLERANCE_SECONDS")
|
||||||
|
.unwrap_or_else(|_| "900".to_string())
|
||||||
|
.parse()
|
||||||
|
.unwrap_or(900);
|
||||||
|
|
||||||
|
let presigned_url_min_expiry: u64 = std::env::var("PRESIGNED_URL_MIN_EXPIRY_SECONDS")
|
||||||
|
.unwrap_or_else(|_| "1".to_string())
|
||||||
|
.parse()
|
||||||
|
.unwrap_or(1);
|
||||||
|
|
||||||
|
let presigned_url_max_expiry: u64 = std::env::var("PRESIGNED_URL_MAX_EXPIRY_SECONDS")
|
||||||
|
.unwrap_or_else(|_| "604800".to_string())
|
||||||
|
.parse()
|
||||||
|
.unwrap_or(604800);
|
||||||
|
|
||||||
|
let secret_key = {
|
||||||
|
let env_key = std::env::var("SECRET_KEY").ok();
|
||||||
|
match env_key {
|
||||||
|
Some(k) if !k.is_empty() && k != "dev-secret-key" => Some(k),
|
||||||
|
_ => {
|
||||||
|
let secret_file = storage_path
|
||||||
|
.join(".myfsio.sys")
|
||||||
|
.join("config")
|
||||||
|
.join(".secret");
|
||||||
|
std::fs::read_to_string(&secret_file)
|
||||||
|
.ok()
|
||||||
|
.map(|s| s.trim().to_string())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let encryption_enabled = parse_bool_env("ENCRYPTION_ENABLED", false);
|
||||||
|
let encryption_chunk_size_bytes = parse_usize_env("ENCRYPTION_CHUNK_SIZE_BYTES", 65_536);
|
||||||
|
|
||||||
|
let kms_enabled = parse_bool_env("KMS_ENABLED", false);
|
||||||
|
let kms_generate_data_key_min_bytes = parse_usize_env("KMS_GENERATE_DATA_KEY_MIN_BYTES", 1);
|
||||||
|
let kms_generate_data_key_max_bytes =
|
||||||
|
parse_usize_env("KMS_GENERATE_DATA_KEY_MAX_BYTES", 1024);
|
||||||
|
|
||||||
|
let gc_enabled = parse_bool_env("GC_ENABLED", false);
|
||||||
|
let gc_interval_hours = parse_f64_env("GC_INTERVAL_HOURS", 6.0);
|
||||||
|
let gc_temp_file_max_age_hours = parse_f64_env("GC_TEMP_FILE_MAX_AGE_HOURS", 24.0);
|
||||||
|
let gc_multipart_max_age_days = parse_u64_env("GC_MULTIPART_MAX_AGE_DAYS", 7);
|
||||||
|
let gc_lock_file_max_age_hours = parse_f64_env("GC_LOCK_FILE_MAX_AGE_HOURS", 1.0);
|
||||||
|
let gc_dry_run = parse_bool_env("GC_DRY_RUN", false);
|
||||||
|
|
||||||
|
let integrity_enabled = parse_bool_env("INTEGRITY_ENABLED", false);
|
||||||
|
|
||||||
|
let metrics_enabled = parse_bool_env("OPERATION_METRICS_ENABLED", false);
|
||||||
|
|
||||||
|
let metrics_history_enabled = parse_bool_env("METRICS_HISTORY_ENABLED", false);
|
||||||
|
|
||||||
|
let metrics_interval_minutes = parse_u64_env("OPERATION_METRICS_INTERVAL_MINUTES", 5);
|
||||||
|
let metrics_retention_hours = parse_u64_env("OPERATION_METRICS_RETENTION_HOURS", 24);
|
||||||
|
let metrics_history_interval_minutes = parse_u64_env("METRICS_HISTORY_INTERVAL_MINUTES", 5);
|
||||||
|
let metrics_history_retention_hours = parse_u64_env("METRICS_HISTORY_RETENTION_HOURS", 24);
|
||||||
|
|
||||||
|
let lifecycle_enabled = parse_bool_env("LIFECYCLE_ENABLED", false);
|
||||||
|
let lifecycle_max_history_per_bucket =
|
||||||
|
parse_usize_env("LIFECYCLE_MAX_HISTORY_PER_BUCKET", 50);
|
||||||
|
|
||||||
|
let website_hosting_enabled = parse_bool_env("WEBSITE_HOSTING_ENABLED", false);
|
||||||
|
let object_key_max_length_bytes = parse_usize_env("OBJECT_KEY_MAX_LENGTH_BYTES", 1024);
|
||||||
|
let object_tag_limit = parse_usize_env("OBJECT_TAG_LIMIT", 50);
|
||||||
|
let object_cache_max_size = parse_usize_env("OBJECT_CACHE_MAX_SIZE", 100);
|
||||||
|
let bucket_config_cache_ttl_seconds =
|
||||||
|
parse_f64_env("BUCKET_CONFIG_CACHE_TTL_SECONDS", 30.0);
|
||||||
|
|
||||||
|
let replication_connect_timeout_secs =
|
||||||
|
parse_u64_env("REPLICATION_CONNECT_TIMEOUT_SECONDS", 5);
|
||||||
|
let replication_read_timeout_secs = parse_u64_env("REPLICATION_READ_TIMEOUT_SECONDS", 30);
|
||||||
|
let replication_max_retries = parse_u64_env("REPLICATION_MAX_RETRIES", 2) as u32;
|
||||||
|
let replication_streaming_threshold_bytes =
|
||||||
|
parse_u64_env("REPLICATION_STREAMING_THRESHOLD_BYTES", 10_485_760);
|
||||||
|
let replication_max_failures_per_bucket =
|
||||||
|
parse_u64_env("REPLICATION_MAX_FAILURES_PER_BUCKET", 50) as usize;
|
||||||
|
|
||||||
|
let site_sync_enabled = parse_bool_env("SITE_SYNC_ENABLED", false);
|
||||||
|
let site_sync_interval_secs = parse_u64_env("SITE_SYNC_INTERVAL_SECONDS", 60);
|
||||||
|
let site_sync_batch_size = parse_u64_env("SITE_SYNC_BATCH_SIZE", 100) as usize;
|
||||||
|
let site_sync_connect_timeout_secs = parse_u64_env("SITE_SYNC_CONNECT_TIMEOUT_SECONDS", 10);
|
||||||
|
let site_sync_read_timeout_secs = parse_u64_env("SITE_SYNC_READ_TIMEOUT_SECONDS", 120);
|
||||||
|
let site_sync_max_retries = parse_u64_env("SITE_SYNC_MAX_RETRIES", 2) as u32;
|
||||||
|
let site_sync_clock_skew_tolerance: f64 =
|
||||||
|
std::env::var("SITE_SYNC_CLOCK_SKEW_TOLERANCE_SECONDS")
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or(1.0);
|
||||||
|
|
||||||
|
let site_id = parse_optional_string_env("SITE_ID");
|
||||||
|
let site_endpoint = parse_optional_string_env("SITE_ENDPOINT");
|
||||||
|
let site_region = std::env::var("SITE_REGION").unwrap_or_else(|_| region.clone());
|
||||||
|
let site_priority = parse_i32_env("SITE_PRIORITY", 100);
|
||||||
|
let api_base_url = std::env::var("API_BASE_URL")
|
||||||
|
.unwrap_or_else(|_| format!("http://{}", bind_addr))
|
||||||
|
.trim_end_matches('/')
|
||||||
|
.to_string();
|
||||||
|
let num_trusted_proxies = parse_usize_env("NUM_TRUSTED_PROXIES", 0);
|
||||||
|
let allowed_redirect_hosts = parse_list_env("ALLOWED_REDIRECT_HOSTS", "");
|
||||||
|
let allow_internal_endpoints = parse_bool_env("ALLOW_INTERNAL_ENDPOINTS", false);
|
||||||
|
let cors_origins = parse_list_env("CORS_ORIGINS", "*");
|
||||||
|
let cors_methods = parse_list_env("CORS_METHODS", "GET,PUT,POST,DELETE,OPTIONS,HEAD");
|
||||||
|
let cors_allow_headers = parse_list_env("CORS_ALLOW_HEADERS", "*");
|
||||||
|
let cors_expose_headers = parse_list_env("CORS_EXPOSE_HEADERS", "*");
|
||||||
|
let session_lifetime_days = parse_u64_env("SESSION_LIFETIME_DAYS", 1);
|
||||||
|
let log_level = std::env::var("LOG_LEVEL").unwrap_or_else(|_| "INFO".to_string());
|
||||||
|
let multipart_min_part_size = parse_u64_env("MULTIPART_MIN_PART_SIZE", 5_242_880);
|
||||||
|
let bulk_delete_max_keys = parse_usize_env("BULK_DELETE_MAX_KEYS", 1000);
|
||||||
|
let stream_chunk_size = parse_usize_env("STREAM_CHUNK_SIZE", 1_048_576);
|
||||||
|
let ratelimit_default =
|
||||||
|
parse_rate_limit_env("RATE_LIMIT_DEFAULT", RateLimitSetting::new(200, 60));
|
||||||
|
let ratelimit_admin =
|
||||||
|
parse_rate_limit_env("RATE_LIMIT_ADMIN", RateLimitSetting::new(60, 60));
|
||||||
|
let ratelimit_storage_uri =
|
||||||
|
std::env::var("RATE_LIMIT_STORAGE_URI").unwrap_or_else(|_| "memory://".to_string());
|
||||||
|
|
||||||
|
let ui_enabled = parse_bool_env("UI_ENABLED", true);
|
||||||
|
let templates_dir = std::env::var("TEMPLATES_DIR")
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| default_templates_dir());
|
||||||
|
let static_dir = std::env::var("STATIC_DIR")
|
||||||
|
.map(PathBuf::from)
|
||||||
|
.unwrap_or_else(|_| default_static_dir());
|
||||||
|
|
||||||
|
Self {
|
||||||
|
bind_addr,
|
||||||
|
ui_bind_addr: SocketAddr::new(host_ip, ui_port),
|
||||||
|
storage_root: storage_path,
|
||||||
|
region,
|
||||||
|
iam_config_path,
|
||||||
|
sigv4_timestamp_tolerance_secs,
|
||||||
|
presigned_url_min_expiry,
|
||||||
|
presigned_url_max_expiry,
|
||||||
|
secret_key,
|
||||||
|
encryption_enabled,
|
||||||
|
encryption_chunk_size_bytes,
|
||||||
|
kms_enabled,
|
||||||
|
kms_generate_data_key_min_bytes,
|
||||||
|
kms_generate_data_key_max_bytes,
|
||||||
|
gc_enabled,
|
||||||
|
gc_interval_hours,
|
||||||
|
gc_temp_file_max_age_hours,
|
||||||
|
gc_multipart_max_age_days,
|
||||||
|
gc_lock_file_max_age_hours,
|
||||||
|
gc_dry_run,
|
||||||
|
integrity_enabled,
|
||||||
|
metrics_enabled,
|
||||||
|
metrics_history_enabled,
|
||||||
|
metrics_interval_minutes,
|
||||||
|
metrics_retention_hours,
|
||||||
|
metrics_history_interval_minutes,
|
||||||
|
metrics_history_retention_hours,
|
||||||
|
lifecycle_enabled,
|
||||||
|
lifecycle_max_history_per_bucket,
|
||||||
|
website_hosting_enabled,
|
||||||
|
object_key_max_length_bytes,
|
||||||
|
object_tag_limit,
|
||||||
|
object_cache_max_size,
|
||||||
|
bucket_config_cache_ttl_seconds,
|
||||||
|
replication_connect_timeout_secs,
|
||||||
|
replication_read_timeout_secs,
|
||||||
|
replication_max_retries,
|
||||||
|
replication_streaming_threshold_bytes,
|
||||||
|
replication_max_failures_per_bucket,
|
||||||
|
site_sync_enabled,
|
||||||
|
site_sync_interval_secs,
|
||||||
|
site_sync_batch_size,
|
||||||
|
site_sync_connect_timeout_secs,
|
||||||
|
site_sync_read_timeout_secs,
|
||||||
|
site_sync_max_retries,
|
||||||
|
site_sync_clock_skew_tolerance,
|
||||||
|
site_id,
|
||||||
|
site_endpoint,
|
||||||
|
site_region,
|
||||||
|
site_priority,
|
||||||
|
api_base_url,
|
||||||
|
num_trusted_proxies,
|
||||||
|
allowed_redirect_hosts,
|
||||||
|
allow_internal_endpoints,
|
||||||
|
cors_origins,
|
||||||
|
cors_methods,
|
||||||
|
cors_allow_headers,
|
||||||
|
cors_expose_headers,
|
||||||
|
session_lifetime_days,
|
||||||
|
log_level,
|
||||||
|
multipart_min_part_size,
|
||||||
|
bulk_delete_max_keys,
|
||||||
|
stream_chunk_size,
|
||||||
|
ratelimit_default,
|
||||||
|
ratelimit_admin,
|
||||||
|
ratelimit_storage_uri,
|
||||||
|
ui_enabled,
|
||||||
|
templates_dir,
|
||||||
|
static_dir,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for ServerConfig {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
bind_addr: "127.0.0.1:5000".parse().unwrap(),
|
||||||
|
ui_bind_addr: "127.0.0.1:5100".parse().unwrap(),
|
||||||
|
storage_root: PathBuf::from("./data"),
|
||||||
|
region: "us-east-1".to_string(),
|
||||||
|
iam_config_path: PathBuf::from("./data/.myfsio.sys/config/iam.json"),
|
||||||
|
sigv4_timestamp_tolerance_secs: 900,
|
||||||
|
presigned_url_min_expiry: 1,
|
||||||
|
presigned_url_max_expiry: 604_800,
|
||||||
|
secret_key: None,
|
||||||
|
encryption_enabled: false,
|
||||||
|
encryption_chunk_size_bytes: 65_536,
|
||||||
|
kms_enabled: false,
|
||||||
|
kms_generate_data_key_min_bytes: 1,
|
||||||
|
kms_generate_data_key_max_bytes: 1024,
|
||||||
|
gc_enabled: false,
|
||||||
|
gc_interval_hours: 6.0,
|
||||||
|
gc_temp_file_max_age_hours: 24.0,
|
||||||
|
gc_multipart_max_age_days: 7,
|
||||||
|
gc_lock_file_max_age_hours: 1.0,
|
||||||
|
gc_dry_run: false,
|
||||||
|
integrity_enabled: false,
|
||||||
|
metrics_enabled: false,
|
||||||
|
metrics_history_enabled: false,
|
||||||
|
metrics_interval_minutes: 5,
|
||||||
|
metrics_retention_hours: 24,
|
||||||
|
metrics_history_interval_minutes: 5,
|
||||||
|
metrics_history_retention_hours: 24,
|
||||||
|
lifecycle_enabled: false,
|
||||||
|
lifecycle_max_history_per_bucket: 50,
|
||||||
|
website_hosting_enabled: false,
|
||||||
|
object_key_max_length_bytes: 1024,
|
||||||
|
object_tag_limit: 50,
|
||||||
|
object_cache_max_size: 100,
|
||||||
|
bucket_config_cache_ttl_seconds: 30.0,
|
||||||
|
replication_connect_timeout_secs: 5,
|
||||||
|
replication_read_timeout_secs: 30,
|
||||||
|
replication_max_retries: 2,
|
||||||
|
replication_streaming_threshold_bytes: 10_485_760,
|
||||||
|
replication_max_failures_per_bucket: 50,
|
||||||
|
site_sync_enabled: false,
|
||||||
|
site_sync_interval_secs: 60,
|
||||||
|
site_sync_batch_size: 100,
|
||||||
|
site_sync_connect_timeout_secs: 10,
|
||||||
|
site_sync_read_timeout_secs: 120,
|
||||||
|
site_sync_max_retries: 2,
|
||||||
|
site_sync_clock_skew_tolerance: 1.0,
|
||||||
|
site_id: None,
|
||||||
|
site_endpoint: None,
|
||||||
|
site_region: "us-east-1".to_string(),
|
||||||
|
site_priority: 100,
|
||||||
|
api_base_url: "http://127.0.0.1:5000".to_string(),
|
||||||
|
num_trusted_proxies: 0,
|
||||||
|
allowed_redirect_hosts: Vec::new(),
|
||||||
|
allow_internal_endpoints: false,
|
||||||
|
cors_origins: vec!["*".to_string()],
|
||||||
|
cors_methods: vec![
|
||||||
|
"GET".to_string(),
|
||||||
|
"PUT".to_string(),
|
||||||
|
"POST".to_string(),
|
||||||
|
"DELETE".to_string(),
|
||||||
|
"OPTIONS".to_string(),
|
||||||
|
"HEAD".to_string(),
|
||||||
|
],
|
||||||
|
cors_allow_headers: vec!["*".to_string()],
|
||||||
|
cors_expose_headers: vec!["*".to_string()],
|
||||||
|
session_lifetime_days: 1,
|
||||||
|
log_level: "INFO".to_string(),
|
||||||
|
multipart_min_part_size: 5_242_880,
|
||||||
|
bulk_delete_max_keys: 1000,
|
||||||
|
stream_chunk_size: 1_048_576,
|
||||||
|
ratelimit_default: RateLimitSetting::new(200, 60),
|
||||||
|
ratelimit_admin: RateLimitSetting::new(60, 60),
|
||||||
|
ratelimit_storage_uri: "memory://".to_string(),
|
||||||
|
ui_enabled: true,
|
||||||
|
templates_dir: default_templates_dir(),
|
||||||
|
static_dir: default_static_dir(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_templates_dir() -> PathBuf {
|
||||||
|
let manifest_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||||
|
manifest_dir.join("templates")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn default_static_dir() -> PathBuf {
|
||||||
|
let manifest_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||||
|
for candidate in [
|
||||||
|
manifest_dir.join("static"),
|
||||||
|
manifest_dir.join("..").join("..").join("..").join("static"),
|
||||||
|
] {
|
||||||
|
if candidate.exists() {
|
||||||
|
return candidate;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
manifest_dir.join("static")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_u64_env(key: &str, default: u64) -> u64 {
|
||||||
|
std::env::var(key)
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_usize_env(key: &str, default: usize) -> usize {
|
||||||
|
std::env::var(key)
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_i32_env(key: &str, default: i32) -> i32 {
|
||||||
|
std::env::var(key)
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_f64_env(key: &str, default: f64) -> f64 {
|
||||||
|
std::env::var(key)
|
||||||
|
.ok()
|
||||||
|
.and_then(|s| s.parse().ok())
|
||||||
|
.unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_bool_env(key: &str, default: bool) -> bool {
|
||||||
|
std::env::var(key)
|
||||||
|
.ok()
|
||||||
|
.map(|value| {
|
||||||
|
matches!(
|
||||||
|
value.trim().to_ascii_lowercase().as_str(),
|
||||||
|
"1" | "true" | "yes" | "on"
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_optional_string_env(key: &str) -> Option<String> {
|
||||||
|
std::env::var(key)
|
||||||
|
.ok()
|
||||||
|
.map(|value| value.trim().to_string())
|
||||||
|
.filter(|value| !value.is_empty())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_list_env(key: &str, default: &str) -> Vec<String> {
|
||||||
|
std::env::var(key)
|
||||||
|
.unwrap_or_else(|_| default.to_string())
|
||||||
|
.split(',')
|
||||||
|
.map(|value| value.trim().to_string())
|
||||||
|
.filter(|value| !value.is_empty())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn parse_rate_limit(value: &str) -> Option<RateLimitSetting> {
|
||||||
|
let parts = value.split_whitespace().collect::<Vec<_>>();
|
||||||
|
if parts.len() != 3 || !parts[1].eq_ignore_ascii_case("per") {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let max_requests = parts[0].parse::<u32>().ok()?;
|
||||||
|
if max_requests == 0 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let window_seconds = match parts[2].to_ascii_lowercase().as_str() {
|
||||||
|
"second" | "seconds" => 1,
|
||||||
|
"minute" | "minutes" => 60,
|
||||||
|
"hour" | "hours" => 3600,
|
||||||
|
"day" | "days" => 86_400,
|
||||||
|
_ => return None,
|
||||||
|
};
|
||||||
|
Some(RateLimitSetting::new(max_requests, window_seconds))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_rate_limit_env(key: &str, default: RateLimitSetting) -> RateLimitSetting {
|
||||||
|
std::env::var(key)
|
||||||
|
.ok()
|
||||||
|
.and_then(|value| parse_rate_limit(&value))
|
||||||
|
.unwrap_or(default)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use std::sync::{Mutex, OnceLock};
|
||||||
|
|
||||||
|
fn env_lock() -> &'static Mutex<()> {
|
||||||
|
static LOCK: OnceLock<Mutex<()>> = OnceLock::new();
|
||||||
|
LOCK.get_or_init(|| Mutex::new(()))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parses_rate_limit_text() {
|
||||||
|
assert_eq!(
|
||||||
|
parse_rate_limit("200 per minute"),
|
||||||
|
Some(RateLimitSetting::new(200, 60))
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
parse_rate_limit("3 per hours"),
|
||||||
|
Some(RateLimitSetting::new(3, 3600))
|
||||||
|
);
|
||||||
|
assert_eq!(parse_rate_limit("0 per minute"), None);
|
||||||
|
assert_eq!(parse_rate_limit("bad"), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn env_defaults_and_invalid_values_fall_back() {
|
||||||
|
let _guard = env_lock().lock().unwrap();
|
||||||
|
std::env::remove_var("OBJECT_KEY_MAX_LENGTH_BYTES");
|
||||||
|
std::env::set_var("OBJECT_TAG_LIMIT", "not-a-number");
|
||||||
|
std::env::set_var("RATE_LIMIT_DEFAULT", "invalid");
|
||||||
|
|
||||||
|
let config = ServerConfig::from_env();
|
||||||
|
|
||||||
|
assert_eq!(config.object_key_max_length_bytes, 1024);
|
||||||
|
assert_eq!(config.object_tag_limit, 50);
|
||||||
|
assert_eq!(config.ratelimit_default, RateLimitSetting::new(200, 60));
|
||||||
|
|
||||||
|
std::env::remove_var("OBJECT_TAG_LIMIT");
|
||||||
|
std::env::remove_var("RATE_LIMIT_DEFAULT");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn env_overrides_new_values() {
|
||||||
|
let _guard = env_lock().lock().unwrap();
|
||||||
|
std::env::set_var("OBJECT_KEY_MAX_LENGTH_BYTES", "2048");
|
||||||
|
std::env::set_var("GC_DRY_RUN", "true");
|
||||||
|
std::env::set_var("RATE_LIMIT_ADMIN", "7 per second");
|
||||||
|
std::env::set_var("HOST", "127.0.0.1");
|
||||||
|
std::env::set_var("PORT", "5501");
|
||||||
|
std::env::remove_var("API_BASE_URL");
|
||||||
|
|
||||||
|
let config = ServerConfig::from_env();
|
||||||
|
|
||||||
|
assert_eq!(config.object_key_max_length_bytes, 2048);
|
||||||
|
assert!(config.gc_dry_run);
|
||||||
|
assert_eq!(config.ratelimit_admin, RateLimitSetting::new(7, 1));
|
||||||
|
assert_eq!(config.api_base_url, "http://127.0.0.1:5501");
|
||||||
|
|
||||||
|
std::env::remove_var("OBJECT_KEY_MAX_LENGTH_BYTES");
|
||||||
|
std::env::remove_var("GC_DRY_RUN");
|
||||||
|
std::env::remove_var("RATE_LIMIT_ADMIN");
|
||||||
|
std::env::remove_var("HOST");
|
||||||
|
std::env::remove_var("PORT");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1172,6 +1172,16 @@ pub async fn put_object_tagging(state: &AppState, bucket: &str, key: &str, body:
|
|||||||
|
|
||||||
let xml_str = String::from_utf8_lossy(&body_bytes);
|
let xml_str = String::from_utf8_lossy(&body_bytes);
|
||||||
let tags = parse_tagging_xml(&xml_str);
|
let tags = parse_tagging_xml(&xml_str);
|
||||||
|
if tags.len() > state.config.object_tag_limit {
|
||||||
|
return xml_response(
|
||||||
|
StatusCode::BAD_REQUEST,
|
||||||
|
S3Error::new(
|
||||||
|
S3ErrorCode::InvalidTag,
|
||||||
|
format!("Maximum {} tags allowed", state.config.object_tag_limit),
|
||||||
|
)
|
||||||
|
.to_xml(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
match state.storage.set_object_tags(bucket, key, &tags).await {
|
match state.storage.set_object_tags(bucket, key, &tags).await {
|
||||||
Ok(()) => StatusCode::OK.into_response(),
|
Ok(()) => StatusCode::OK.into_response(),
|
||||||
@@ -294,8 +294,17 @@ async fn generate_data_key_inner(state: AppState, body: Body, include_plaintext:
|
|||||||
.and_then(|v| v.as_u64())
|
.and_then(|v| v.as_u64())
|
||||||
.unwrap_or(32) as usize;
|
.unwrap_or(32) as usize;
|
||||||
|
|
||||||
if !(1..=1024).contains(&num_bytes) {
|
if num_bytes < state.config.kms_generate_data_key_min_bytes
|
||||||
return json_err(StatusCode::BAD_REQUEST, "NumberOfBytes must be 1-1024");
|
|| num_bytes > state.config.kms_generate_data_key_max_bytes
|
||||||
|
{
|
||||||
|
return json_err(
|
||||||
|
StatusCode::BAD_REQUEST,
|
||||||
|
&format!(
|
||||||
|
"NumberOfBytes must be {}-{}",
|
||||||
|
state.config.kms_generate_data_key_min_bytes,
|
||||||
|
state.config.kms_generate_data_key_max_bytes
|
||||||
|
),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
match kms.generate_data_key(key_id, num_bytes).await {
|
match kms.generate_data_key(key_id, num_bytes).await {
|
||||||
@@ -389,8 +398,17 @@ pub async fn generate_random(State(state): State<AppState>, body: Body) -> Respo
|
|||||||
.and_then(|v| v.as_u64())
|
.and_then(|v| v.as_u64())
|
||||||
.unwrap_or(32) as usize;
|
.unwrap_or(32) as usize;
|
||||||
|
|
||||||
if !(1..=1024).contains(&num_bytes) {
|
if num_bytes < state.config.kms_generate_data_key_min_bytes
|
||||||
return json_err(StatusCode::BAD_REQUEST, "NumberOfBytes must be 1-1024");
|
|| num_bytes > state.config.kms_generate_data_key_max_bytes
|
||||||
|
{
|
||||||
|
return json_err(
|
||||||
|
StatusCode::BAD_REQUEST,
|
||||||
|
&format!(
|
||||||
|
"NumberOfBytes must be {}-{}",
|
||||||
|
state.config.kms_generate_data_key_min_bytes,
|
||||||
|
state.config.kms_generate_data_key_max_bytes
|
||||||
|
),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut bytes = vec![0u8; num_bytes];
|
let mut bytes = vec![0u8; num_bytes];
|
||||||
@@ -1122,6 +1122,14 @@ pub async fn put_object(
|
|||||||
Ok(tags) => tags,
|
Ok(tags) => tags,
|
||||||
Err(response) => return response,
|
Err(response) => return response,
|
||||||
};
|
};
|
||||||
|
if let Some(ref tags) = tags {
|
||||||
|
if tags.len() > state.config.object_tag_limit {
|
||||||
|
return s3_error_response(S3Error::new(
|
||||||
|
S3ErrorCode::InvalidTag,
|
||||||
|
format!("Maximum {} tags allowed", state.config.object_tag_limit),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
let aws_chunked = is_aws_chunked(&headers);
|
let aws_chunked = is_aws_chunked(&headers);
|
||||||
let boxed: myfsio_storage::traits::AsyncReadStream = if has_upload_checksum(&headers) {
|
let boxed: myfsio_storage::traits::AsyncReadStream = if has_upload_checksum(&headers) {
|
||||||
@@ -2860,6 +2868,7 @@ mod tests {
|
|||||||
ui_enabled: false,
|
ui_enabled: false,
|
||||||
templates_dir: manifest_dir.join("templates"),
|
templates_dir: manifest_dir.join("templates"),
|
||||||
static_dir: manifest_dir.join("static"),
|
static_dir: manifest_dir.join("static"),
|
||||||
|
..ServerConfig::default()
|
||||||
};
|
};
|
||||||
(AppState::new(config), tmp)
|
(AppState::new(config), tmp)
|
||||||
}
|
}
|
||||||
@@ -66,7 +66,7 @@ pub async fn login_submit(
|
|||||||
let next = form
|
let next = form
|
||||||
.next
|
.next
|
||||||
.as_deref()
|
.as_deref()
|
||||||
.filter(|n| n.starts_with("/ui/") || *n == "/ui")
|
.filter(|n| is_allowed_redirect(n, &state.config.allowed_redirect_hosts))
|
||||||
.unwrap_or("/ui/buckets")
|
.unwrap_or("/ui/buckets")
|
||||||
.to_string();
|
.to_string();
|
||||||
Redirect::to(&next).into_response()
|
Redirect::to(&next).into_response()
|
||||||
@@ -80,6 +80,32 @@ pub async fn login_submit(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn is_allowed_redirect(target: &str, allowed_hosts: &[String]) -> bool {
|
||||||
|
if target == "/ui" || target.starts_with("/ui/") {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
let Some(rest) = target
|
||||||
|
.strip_prefix("https://")
|
||||||
|
.or_else(|| target.strip_prefix("http://"))
|
||||||
|
else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
let host = rest
|
||||||
|
.split('/')
|
||||||
|
.next()
|
||||||
|
.unwrap_or_default()
|
||||||
|
.split('@')
|
||||||
|
.last()
|
||||||
|
.unwrap_or_default()
|
||||||
|
.split(':')
|
||||||
|
.next()
|
||||||
|
.unwrap_or_default()
|
||||||
|
.to_ascii_lowercase();
|
||||||
|
allowed_hosts
|
||||||
|
.iter()
|
||||||
|
.any(|allowed| allowed.eq_ignore_ascii_case(&host))
|
||||||
|
}
|
||||||
|
|
||||||
pub async fn logout(Extension(session): Extension<SessionHandle>) -> Response {
|
pub async fn logout(Extension(session): Extension<SessionHandle>) -> Response {
|
||||||
session.write(|s| {
|
session.write(|s| {
|
||||||
s.user_id = None;
|
s.user_id = None;
|
||||||
@@ -49,6 +49,8 @@ const AWS_QUERY_ENCODE_SET: &AsciiSet = &NON_ALPHANUMERIC
|
|||||||
.remove(b'.')
|
.remove(b'.')
|
||||||
.remove(b'~');
|
.remove(b'~');
|
||||||
|
|
||||||
|
const UI_OBJECT_BROWSER_MAX_KEYS: usize = 5000;
|
||||||
|
|
||||||
fn url_templates_for(bucket: &str) -> Value {
|
fn url_templates_for(bucket: &str) -> Value {
|
||||||
json!({
|
json!({
|
||||||
"download": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/download", bucket),
|
"download": format!("/ui/buckets/{}/objects/KEY_PLACEHOLDER/download", bucket),
|
||||||
@@ -185,10 +187,7 @@ fn safe_attachment_filename(key: &str) -> String {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn parse_api_base(state: &AppState) -> String {
|
fn parse_api_base(state: &AppState) -> String {
|
||||||
std::env::var("API_BASE_URL")
|
state.config.api_base_url.trim_end_matches('/').to_string()
|
||||||
.unwrap_or_else(|_| format!("http://{}", state.config.bind_addr))
|
|
||||||
.trim_end_matches('/')
|
|
||||||
.to_string()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn aws_query_encode(value: &str) -> String {
|
fn aws_query_encode(value: &str) -> String {
|
||||||
@@ -1022,11 +1021,13 @@ pub async fn stream_bucket_objects(
|
|||||||
let prefix = q.prefix.clone().unwrap_or_default();
|
let prefix = q.prefix.clone().unwrap_or_default();
|
||||||
|
|
||||||
if use_delimiter {
|
if use_delimiter {
|
||||||
|
let mut token: Option<String> = None;
|
||||||
|
loop {
|
||||||
let params = myfsio_common::types::ShallowListParams {
|
let params = myfsio_common::types::ShallowListParams {
|
||||||
prefix: prefix.clone(),
|
prefix: prefix.clone(),
|
||||||
delimiter: "/".to_string(),
|
delimiter: "/".to_string(),
|
||||||
max_keys: 5000,
|
max_keys: UI_OBJECT_BROWSER_MAX_KEYS,
|
||||||
continuation_token: None,
|
continuation_token: token.clone(),
|
||||||
};
|
};
|
||||||
match state
|
match state
|
||||||
.storage
|
.storage
|
||||||
@@ -1052,8 +1053,16 @@ pub async fn stream_bucket_objects(
|
|||||||
.to_string(),
|
.to_string(),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
if !res.is_truncated || res.next_continuation_token.is_none() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
token = res.next_continuation_token;
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
lines.push(json!({ "type": "error", "error": e.to_string() }).to_string());
|
||||||
|
break;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Err(e) => lines.push(json!({ "type": "error", "error": e.to_string() }).to_string()),
|
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
let mut token: Option<String> = None;
|
let mut token: Option<String> = None;
|
||||||
@@ -1123,7 +1132,7 @@ pub async fn list_bucket_folders(
|
|||||||
let params = myfsio_common::types::ShallowListParams {
|
let params = myfsio_common::types::ShallowListParams {
|
||||||
prefix: prefix.clone(),
|
prefix: prefix.clone(),
|
||||||
delimiter: "/".to_string(),
|
delimiter: "/".to_string(),
|
||||||
max_keys: 5000,
|
max_keys: UI_OBJECT_BROWSER_MAX_KEYS,
|
||||||
continuation_token: None,
|
continuation_token: None,
|
||||||
};
|
};
|
||||||
match state
|
match state
|
||||||
@@ -2408,8 +2417,11 @@ async fn update_object_tags(state: &AppState, bucket: &str, key: &str, body: Bod
|
|||||||
Err(response) => return response,
|
Err(response) => return response,
|
||||||
};
|
};
|
||||||
|
|
||||||
if payload.tags.len() > 50 {
|
if payload.tags.len() > state.config.object_tag_limit {
|
||||||
return json_error(StatusCode::BAD_REQUEST, "Maximum 50 tags allowed");
|
return json_error(
|
||||||
|
StatusCode::BAD_REQUEST,
|
||||||
|
format!("Maximum {} tags allowed", state.config.object_tag_limit),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
let tags = payload
|
let tags = payload
|
||||||
@@ -2841,6 +2853,15 @@ pub async fn bulk_delete_objects(
|
|||||||
"No objects found under the selected folders",
|
"No objects found under the selected folders",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
if keys.len() > state.config.bulk_delete_max_keys {
|
||||||
|
return json_error(
|
||||||
|
StatusCode::BAD_REQUEST,
|
||||||
|
format!(
|
||||||
|
"Bulk delete supports at most {} keys",
|
||||||
|
state.config.bulk_delete_max_keys
|
||||||
|
),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
let mut deleted = Vec::new();
|
let mut deleted = Vec::new();
|
||||||
let mut errors = Vec::new();
|
let mut errors = Vec::new();
|
||||||
@@ -1,8 +1,10 @@
|
|||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
use axum::body::Body;
|
||||||
use axum::extract::{Extension, Form, Path, Query, State};
|
use axum::extract::{Extension, Form, Path, Query, State};
|
||||||
use axum::http::{header, HeaderMap, StatusCode};
|
use axum::http::{header, HeaderMap, StatusCode};
|
||||||
use axum::response::{IntoResponse, Redirect, Response};
|
use axum::response::{IntoResponse, Redirect, Response};
|
||||||
|
use http_body_util::BodyExt;
|
||||||
use serde_json::{json, Value};
|
use serde_json::{json, Value};
|
||||||
use tera::Context;
|
use tera::Context;
|
||||||
|
|
||||||
@@ -203,6 +205,59 @@ fn wants_json(headers: &HeaderMap) -> bool {
|
|||||||
.unwrap_or(false)
|
.unwrap_or(false)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn parse_form_any(
|
||||||
|
headers: &HeaderMap,
|
||||||
|
body: Body,
|
||||||
|
) -> Result<HashMap<String, String>, String> {
|
||||||
|
let content_type = headers
|
||||||
|
.get(header::CONTENT_TYPE)
|
||||||
|
.and_then(|v| v.to_str().ok())
|
||||||
|
.unwrap_or("")
|
||||||
|
.to_string();
|
||||||
|
let is_multipart = content_type
|
||||||
|
.to_ascii_lowercase()
|
||||||
|
.starts_with("multipart/form-data");
|
||||||
|
|
||||||
|
let bytes = body
|
||||||
|
.collect()
|
||||||
|
.await
|
||||||
|
.map_err(|e| format!("Failed to read request body: {}", e))?
|
||||||
|
.to_bytes();
|
||||||
|
|
||||||
|
if is_multipart {
|
||||||
|
let boundary = multer::parse_boundary(&content_type)
|
||||||
|
.map_err(|_| "Missing multipart boundary".to_string())?;
|
||||||
|
let stream = futures::stream::once(async move {
|
||||||
|
Ok::<_, std::io::Error>(bytes)
|
||||||
|
});
|
||||||
|
let mut multipart = multer::Multipart::new(stream, boundary);
|
||||||
|
let mut out = HashMap::new();
|
||||||
|
while let Some(field) = multipart
|
||||||
|
.next_field()
|
||||||
|
.await
|
||||||
|
.map_err(|e| format!("Malformed multipart body: {}", e))?
|
||||||
|
{
|
||||||
|
let name = match field.name() {
|
||||||
|
Some(name) => name.to_string(),
|
||||||
|
None => continue,
|
||||||
|
};
|
||||||
|
if field.file_name().is_some() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let value = field
|
||||||
|
.text()
|
||||||
|
.await
|
||||||
|
.map_err(|e| format!("Invalid multipart field '{}': {}", name, e))?;
|
||||||
|
out.insert(name, value);
|
||||||
|
}
|
||||||
|
Ok(out)
|
||||||
|
} else {
|
||||||
|
let parsed: Vec<(String, String)> = serde_urlencoded::from_bytes(&bytes)
|
||||||
|
.map_err(|e| format!("Invalid form body: {}", e))?;
|
||||||
|
Ok(parsed.into_iter().collect())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn bucket_tab_redirect(bucket_name: &str, tab: &str) -> Response {
|
fn bucket_tab_redirect(bucket_name: &str, tab: &str) -> Response {
|
||||||
Redirect::to(&format!("/ui/buckets/{}?tab={}", bucket_name, tab)).into_response()
|
Redirect::to(&format!("/ui/buckets/{}?tab={}", bucket_name, tab)).into_response()
|
||||||
}
|
}
|
||||||
@@ -231,10 +286,7 @@ fn default_public_policy(bucket_name: &str) -> String {
|
|||||||
}
|
}
|
||||||
|
|
||||||
fn parse_api_base(state: &AppState) -> (String, String) {
|
fn parse_api_base(state: &AppState) -> (String, String) {
|
||||||
let api_base = std::env::var("API_BASE_URL")
|
let api_base = state.config.api_base_url.trim_end_matches('/').to_string();
|
||||||
.unwrap_or_else(|_| format!("http://{}", state.config.bind_addr))
|
|
||||||
.trim_end_matches('/')
|
|
||||||
.to_string();
|
|
||||||
let api_host = api_base
|
let api_host = api_base
|
||||||
.split("://")
|
.split("://")
|
||||||
.nth(1)
|
.nth(1)
|
||||||
@@ -1173,16 +1225,13 @@ pub async fn sites_dashboard(
|
|||||||
ctx.insert("connections", &conns);
|
ctx.insert("connections", &conns);
|
||||||
ctx.insert(
|
ctx.insert(
|
||||||
"config_site_id",
|
"config_site_id",
|
||||||
&std::env::var("SITE_ID").unwrap_or_default(),
|
&state.config.site_id.clone().unwrap_or_default(),
|
||||||
);
|
);
|
||||||
ctx.insert(
|
ctx.insert(
|
||||||
"config_site_endpoint",
|
"config_site_endpoint",
|
||||||
&std::env::var("SITE_ENDPOINT").unwrap_or_default(),
|
&state.config.site_endpoint.clone().unwrap_or_default(),
|
||||||
);
|
|
||||||
ctx.insert(
|
|
||||||
"config_site_region",
|
|
||||||
&std::env::var("SITE_REGION").unwrap_or_else(|_| state.config.region.clone()),
|
|
||||||
);
|
);
|
||||||
|
ctx.insert("config_site_region", &state.config.site_region);
|
||||||
ctx.insert("topology", &json!({"sites": [], "connections": []}));
|
ctx.insert("topology", &json!({"sites": [], "connections": []}));
|
||||||
render(&state, "sites.html", &ctx)
|
render(&state, "sites.html", &ctx)
|
||||||
}
|
}
|
||||||
@@ -2119,9 +2168,29 @@ pub async fn create_bucket(
|
|||||||
State(state): State<AppState>,
|
State(state): State<AppState>,
|
||||||
Extension(session): Extension<SessionHandle>,
|
Extension(session): Extension<SessionHandle>,
|
||||||
headers: HeaderMap,
|
headers: HeaderMap,
|
||||||
axum::extract::Form(form): axum::extract::Form<CreateBucketForm>,
|
body: Body,
|
||||||
) -> Response {
|
) -> Response {
|
||||||
let wants_json = wants_json(&headers);
|
let wants_json = wants_json(&headers);
|
||||||
|
let form = match parse_form_any(&headers, body).await {
|
||||||
|
Ok(fields) => CreateBucketForm {
|
||||||
|
bucket_name: fields
|
||||||
|
.get("bucket_name")
|
||||||
|
.cloned()
|
||||||
|
.unwrap_or_default(),
|
||||||
|
csrf_token: fields.get("csrf_token").cloned().unwrap_or_default(),
|
||||||
|
},
|
||||||
|
Err(message) => {
|
||||||
|
if wants_json {
|
||||||
|
return (
|
||||||
|
StatusCode::BAD_REQUEST,
|
||||||
|
axum::Json(json!({ "error": message })),
|
||||||
|
)
|
||||||
|
.into_response();
|
||||||
|
}
|
||||||
|
session.write(|s| s.push_flash("danger", message));
|
||||||
|
return Redirect::to("/ui/buckets").into_response();
|
||||||
|
}
|
||||||
|
};
|
||||||
let bucket_name = form.bucket_name.trim().to_string();
|
let bucket_name = form.bucket_name.trim().to_string();
|
||||||
|
|
||||||
if bucket_name.is_empty() {
|
if bucket_name.is_empty() {
|
||||||
@@ -333,7 +333,16 @@ pub fn create_ui_router(state: state::AppState) -> Router {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub fn create_router(state: state::AppState) -> Router {
|
pub fn create_router(state: state::AppState) -> Router {
|
||||||
let mut router = Router::new()
|
let default_rate_limit = middleware::RateLimitLayerState::new(
|
||||||
|
state.config.ratelimit_default,
|
||||||
|
state.config.num_trusted_proxies,
|
||||||
|
);
|
||||||
|
let admin_rate_limit = middleware::RateLimitLayerState::new(
|
||||||
|
state.config.ratelimit_admin,
|
||||||
|
state.config.num_trusted_proxies,
|
||||||
|
);
|
||||||
|
|
||||||
|
let mut api_router = Router::new()
|
||||||
.route("/myfsio/health", axum::routing::get(handlers::health_check))
|
.route("/myfsio/health", axum::routing::get(handlers::health_check))
|
||||||
.route("/", axum::routing::get(handlers::list_buckets))
|
.route("/", axum::routing::get(handlers::list_buckets))
|
||||||
.route(
|
.route(
|
||||||
@@ -362,7 +371,7 @@ pub fn create_router(state: state::AppState) -> Router {
|
|||||||
);
|
);
|
||||||
|
|
||||||
if state.config.kms_enabled {
|
if state.config.kms_enabled {
|
||||||
router = router
|
api_router = api_router
|
||||||
.route(
|
.route(
|
||||||
"/kms/keys",
|
"/kms/keys",
|
||||||
axum::routing::get(handlers::kms::list_keys).post(handlers::kms::create_key),
|
axum::routing::get(handlers::kms::list_keys).post(handlers::kms::create_key),
|
||||||
@@ -415,7 +424,17 @@ pub fn create_router(state: state::AppState) -> Router {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
router = router
|
api_router = api_router
|
||||||
|
.layer(axum::middleware::from_fn_with_state(
|
||||||
|
state.clone(),
|
||||||
|
middleware::auth_layer,
|
||||||
|
))
|
||||||
|
.layer(axum::middleware::from_fn_with_state(
|
||||||
|
default_rate_limit,
|
||||||
|
middleware::rate_limit_layer,
|
||||||
|
));
|
||||||
|
|
||||||
|
let admin_router = Router::new()
|
||||||
.route(
|
.route(
|
||||||
"/admin/site",
|
"/admin/site",
|
||||||
axum::routing::get(handlers::admin::get_local_site)
|
axum::routing::get(handlers::admin::get_local_site)
|
||||||
@@ -546,14 +565,81 @@ pub fn create_router(state: state::AppState) -> Router {
|
|||||||
.route(
|
.route(
|
||||||
"/admin/integrity/history",
|
"/admin/integrity/history",
|
||||||
axum::routing::get(handlers::admin::integrity_history),
|
axum::routing::get(handlers::admin::integrity_history),
|
||||||
);
|
)
|
||||||
|
|
||||||
router
|
|
||||||
.layer(axum::middleware::from_fn_with_state(
|
.layer(axum::middleware::from_fn_with_state(
|
||||||
state.clone(),
|
state.clone(),
|
||||||
middleware::auth_layer,
|
middleware::auth_layer,
|
||||||
))
|
))
|
||||||
|
.layer(axum::middleware::from_fn_with_state(
|
||||||
|
admin_rate_limit,
|
||||||
|
middleware::rate_limit_layer,
|
||||||
|
));
|
||||||
|
|
||||||
|
api_router
|
||||||
|
.merge(admin_router)
|
||||||
.layer(axum::middleware::from_fn(middleware::server_header))
|
.layer(axum::middleware::from_fn(middleware::server_header))
|
||||||
|
.layer(cors_layer(&state.config))
|
||||||
.layer(tower_http::compression::CompressionLayer::new())
|
.layer(tower_http::compression::CompressionLayer::new())
|
||||||
.with_state(state)
|
.with_state(state)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn cors_layer(config: &config::ServerConfig) -> tower_http::cors::CorsLayer {
|
||||||
|
use axum::http::{HeaderName, HeaderValue, Method};
|
||||||
|
use tower_http::cors::{Any, CorsLayer};
|
||||||
|
|
||||||
|
let mut layer = CorsLayer::new();
|
||||||
|
|
||||||
|
if config.cors_origins.iter().any(|origin| origin == "*") {
|
||||||
|
layer = layer.allow_origin(Any);
|
||||||
|
} else {
|
||||||
|
let origins = config
|
||||||
|
.cors_origins
|
||||||
|
.iter()
|
||||||
|
.filter_map(|origin| HeaderValue::from_str(origin).ok())
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
if !origins.is_empty() {
|
||||||
|
layer = layer.allow_origin(origins);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let methods = config
|
||||||
|
.cors_methods
|
||||||
|
.iter()
|
||||||
|
.filter_map(|method| method.parse::<Method>().ok())
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
if !methods.is_empty() {
|
||||||
|
layer = layer.allow_methods(methods);
|
||||||
|
}
|
||||||
|
|
||||||
|
if config.cors_allow_headers.iter().any(|header| header == "*") {
|
||||||
|
layer = layer.allow_headers(Any);
|
||||||
|
} else {
|
||||||
|
let headers = config
|
||||||
|
.cors_allow_headers
|
||||||
|
.iter()
|
||||||
|
.filter_map(|header| header.parse::<HeaderName>().ok())
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
if !headers.is_empty() {
|
||||||
|
layer = layer.allow_headers(headers);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if config
|
||||||
|
.cors_expose_headers
|
||||||
|
.iter()
|
||||||
|
.any(|header| header == "*")
|
||||||
|
{
|
||||||
|
layer = layer.expose_headers(Any);
|
||||||
|
} else {
|
||||||
|
let headers = config
|
||||||
|
.cors_expose_headers
|
||||||
|
.iter()
|
||||||
|
.filter_map(|header| header.parse::<HeaderName>().ok())
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
if !headers.is_empty() {
|
||||||
|
layer = layer.expose_headers(headers);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
layer
|
||||||
|
}
|
||||||
@@ -28,10 +28,19 @@ enum Command {
|
|||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
async fn main() {
|
async fn main() {
|
||||||
load_env_files();
|
load_env_files();
|
||||||
tracing_subscriber::fmt::init();
|
init_tracing();
|
||||||
|
|
||||||
let cli = Cli::parse();
|
let cli = Cli::parse();
|
||||||
let config = ServerConfig::from_env();
|
let config = ServerConfig::from_env();
|
||||||
|
if !config
|
||||||
|
.ratelimit_storage_uri
|
||||||
|
.eq_ignore_ascii_case("memory://")
|
||||||
|
{
|
||||||
|
tracing::warn!(
|
||||||
|
"RATE_LIMIT_STORAGE_URI={} is not supported yet; using in-memory rate limits",
|
||||||
|
config.ratelimit_storage_uri
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
if cli.reset_cred {
|
if cli.reset_cred {
|
||||||
reset_admin_credentials(&config);
|
reset_admin_credentials(&config);
|
||||||
@@ -114,7 +123,10 @@ async fn main() {
|
|||||||
std::sync::Arc::new(myfsio_server::services::lifecycle::LifecycleService::new(
|
std::sync::Arc::new(myfsio_server::services::lifecycle::LifecycleService::new(
|
||||||
state.storage.clone(),
|
state.storage.clone(),
|
||||||
config.storage_root.clone(),
|
config.storage_root.clone(),
|
||||||
myfsio_server::services::lifecycle::LifecycleConfig::default(),
|
myfsio_server::services::lifecycle::LifecycleConfig {
|
||||||
|
interval_seconds: 3600,
|
||||||
|
max_history_per_bucket: config.lifecycle_max_history_per_bucket,
|
||||||
|
},
|
||||||
));
|
));
|
||||||
bg_handles.push(lifecycle.start_background());
|
bg_handles.push(lifecycle.start_background());
|
||||||
tracing::info!("Lifecycle manager background service started");
|
tracing::info!("Lifecycle manager background service started");
|
||||||
@@ -178,7 +190,10 @@ async fn main() {
|
|||||||
let shutdown = shutdown_signal_shared();
|
let shutdown = shutdown_signal_shared();
|
||||||
let api_shutdown = shutdown.clone();
|
let api_shutdown = shutdown.clone();
|
||||||
let api_task = tokio::spawn(async move {
|
let api_task = tokio::spawn(async move {
|
||||||
axum::serve(api_listener, api_app)
|
axum::serve(
|
||||||
|
api_listener,
|
||||||
|
api_app.into_make_service_with_connect_info::<std::net::SocketAddr>(),
|
||||||
|
)
|
||||||
.with_graceful_shutdown(async move {
|
.with_graceful_shutdown(async move {
|
||||||
api_shutdown.notified().await;
|
api_shutdown.notified().await;
|
||||||
})
|
})
|
||||||
@@ -228,15 +243,43 @@ fn print_config_summary(config: &ServerConfig) {
|
|||||||
println!("IAM config: {}", config.iam_config_path.display());
|
println!("IAM config: {}", config.iam_config_path.display());
|
||||||
println!("Region: {}", config.region);
|
println!("Region: {}", config.region);
|
||||||
println!("Encryption enabled: {}", config.encryption_enabled);
|
println!("Encryption enabled: {}", config.encryption_enabled);
|
||||||
|
println!(
|
||||||
|
"Encryption chunk size: {} bytes",
|
||||||
|
config.encryption_chunk_size_bytes
|
||||||
|
);
|
||||||
println!("KMS enabled: {}", config.kms_enabled);
|
println!("KMS enabled: {}", config.kms_enabled);
|
||||||
|
println!(
|
||||||
|
"KMS data key bounds: {}-{} bytes",
|
||||||
|
config.kms_generate_data_key_min_bytes, config.kms_generate_data_key_max_bytes
|
||||||
|
);
|
||||||
println!("GC enabled: {}", config.gc_enabled);
|
println!("GC enabled: {}", config.gc_enabled);
|
||||||
|
println!(
|
||||||
|
"GC interval: {} hours, dry run: {}",
|
||||||
|
config.gc_interval_hours, config.gc_dry_run
|
||||||
|
);
|
||||||
println!("Integrity enabled: {}", config.integrity_enabled);
|
println!("Integrity enabled: {}", config.integrity_enabled);
|
||||||
println!("Lifecycle enabled: {}", config.lifecycle_enabled);
|
println!("Lifecycle enabled: {}", config.lifecycle_enabled);
|
||||||
|
println!(
|
||||||
|
"Lifecycle history limit: {}",
|
||||||
|
config.lifecycle_max_history_per_bucket
|
||||||
|
);
|
||||||
println!(
|
println!(
|
||||||
"Website hosting enabled: {}",
|
"Website hosting enabled: {}",
|
||||||
config.website_hosting_enabled
|
config.website_hosting_enabled
|
||||||
);
|
);
|
||||||
println!("Site sync enabled: {}", config.site_sync_enabled);
|
println!("Site sync enabled: {}", config.site_sync_enabled);
|
||||||
|
println!("API base URL: {}", config.api_base_url);
|
||||||
|
println!(
|
||||||
|
"Object key max: {} bytes, tag limit: {}",
|
||||||
|
config.object_key_max_length_bytes, config.object_tag_limit
|
||||||
|
);
|
||||||
|
println!(
|
||||||
|
"Rate limits: default {} per {}s, admin {} per {}s",
|
||||||
|
config.ratelimit_default.max_requests,
|
||||||
|
config.ratelimit_default.window_seconds,
|
||||||
|
config.ratelimit_admin.max_requests,
|
||||||
|
config.ratelimit_admin.window_seconds
|
||||||
|
);
|
||||||
println!(
|
println!(
|
||||||
"Metrics history enabled: {}",
|
"Metrics history enabled: {}",
|
||||||
config.metrics_history_enabled
|
config.metrics_history_enabled
|
||||||
@@ -256,6 +299,32 @@ fn validate_config(config: &ServerConfig) -> Vec<String> {
|
|||||||
if config.presigned_url_min_expiry > config.presigned_url_max_expiry {
|
if config.presigned_url_min_expiry > config.presigned_url_max_expiry {
|
||||||
issues.push("CRITICAL: PRESIGNED_URL_MIN_EXPIRY_SECONDS cannot exceed PRESIGNED_URL_MAX_EXPIRY_SECONDS.".to_string());
|
issues.push("CRITICAL: PRESIGNED_URL_MIN_EXPIRY_SECONDS cannot exceed PRESIGNED_URL_MAX_EXPIRY_SECONDS.".to_string());
|
||||||
}
|
}
|
||||||
|
if config.encryption_chunk_size_bytes == 0 {
|
||||||
|
issues.push("CRITICAL: ENCRYPTION_CHUNK_SIZE_BYTES must be greater than zero.".to_string());
|
||||||
|
}
|
||||||
|
if config.kms_generate_data_key_min_bytes == 0 {
|
||||||
|
issues.push(
|
||||||
|
"CRITICAL: KMS_GENERATE_DATA_KEY_MIN_BYTES must be greater than zero.".to_string(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if config.kms_generate_data_key_min_bytes > config.kms_generate_data_key_max_bytes {
|
||||||
|
issues.push("CRITICAL: KMS_GENERATE_DATA_KEY_MIN_BYTES cannot exceed KMS_GENERATE_DATA_KEY_MAX_BYTES.".to_string());
|
||||||
|
}
|
||||||
|
if config.gc_interval_hours <= 0.0 {
|
||||||
|
issues.push("CRITICAL: GC_INTERVAL_HOURS must be greater than zero.".to_string());
|
||||||
|
}
|
||||||
|
if config.bucket_config_cache_ttl_seconds < 0.0 {
|
||||||
|
issues.push("CRITICAL: BUCKET_CONFIG_CACHE_TTL_SECONDS cannot be negative.".to_string());
|
||||||
|
}
|
||||||
|
if !config
|
||||||
|
.ratelimit_storage_uri
|
||||||
|
.eq_ignore_ascii_case("memory://")
|
||||||
|
{
|
||||||
|
issues.push(format!(
|
||||||
|
"WARNING: RATE_LIMIT_STORAGE_URI={} is not supported yet; using in-memory limits.",
|
||||||
|
config.ratelimit_storage_uri
|
||||||
|
));
|
||||||
|
}
|
||||||
if let Err(err) = std::fs::create_dir_all(&config.storage_root) {
|
if let Err(err) = std::fs::create_dir_all(&config.storage_root) {
|
||||||
issues.push(format!(
|
issues.push(format!(
|
||||||
"CRITICAL: Cannot create storage root {}: {}",
|
"CRITICAL: Cannot create storage root {}: {}",
|
||||||
@@ -286,6 +355,17 @@ fn validate_config(config: &ServerConfig) -> Vec<String> {
|
|||||||
issues
|
issues
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn init_tracing() {
|
||||||
|
use tracing_subscriber::EnvFilter;
|
||||||
|
|
||||||
|
let filter = EnvFilter::try_from_env("RUST_LOG")
|
||||||
|
.or_else(|_| {
|
||||||
|
EnvFilter::try_new(std::env::var("LOG_LEVEL").unwrap_or_else(|_| "INFO".to_string()))
|
||||||
|
})
|
||||||
|
.unwrap_or_else(|_| EnvFilter::new("INFO"));
|
||||||
|
tracing_subscriber::fmt().with_env_filter(filter).init();
|
||||||
|
}
|
||||||
|
|
||||||
fn shutdown_signal_shared() -> std::sync::Arc<tokio::sync::Notify> {
|
fn shutdown_signal_shared() -> std::sync::Arc<tokio::sync::Notify> {
|
||||||
std::sync::Arc::new(tokio::sync::Notify::new())
|
std::sync::Arc::new(tokio::sync::Notify::new())
|
||||||
}
|
}
|
||||||
@@ -419,8 +499,49 @@ fn reset_admin_credentials(config: &ServerConfig) {
|
|||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
}
|
}
|
||||||
println!("Backed up existing IAM config to {}", backup.display());
|
println!("Backed up existing IAM config to {}", backup.display());
|
||||||
|
prune_iam_backups(&config.iam_config_path, 5);
|
||||||
}
|
}
|
||||||
|
|
||||||
ensure_iam_bootstrap(config);
|
ensure_iam_bootstrap(config);
|
||||||
println!("Admin credentials reset.");
|
println!("Admin credentials reset.");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn prune_iam_backups(iam_path: &std::path::Path, keep: usize) {
|
||||||
|
let parent = match iam_path.parent() {
|
||||||
|
Some(p) => p,
|
||||||
|
None => return,
|
||||||
|
};
|
||||||
|
let stem = match iam_path.file_stem().and_then(|s| s.to_str()) {
|
||||||
|
Some(s) => s,
|
||||||
|
None => return,
|
||||||
|
};
|
||||||
|
let prefix = format!("{}.bak-", stem);
|
||||||
|
|
||||||
|
let entries = match std::fs::read_dir(parent) {
|
||||||
|
Ok(entries) => entries,
|
||||||
|
Err(_) => return,
|
||||||
|
};
|
||||||
|
let mut backups: Vec<(i64, std::path::PathBuf)> = entries
|
||||||
|
.filter_map(|e| e.ok())
|
||||||
|
.filter_map(|e| {
|
||||||
|
let path = e.path();
|
||||||
|
let name = path.file_name()?.to_str()?;
|
||||||
|
let rest = name.strip_prefix(&prefix)?;
|
||||||
|
let ts: i64 = rest.parse().ok()?;
|
||||||
|
Some((ts, path))
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
backups.sort_by(|a, b| b.0.cmp(&a.0));
|
||||||
|
|
||||||
|
for (_, path) in backups.into_iter().skip(keep) {
|
||||||
|
if let Err(err) = std::fs::remove_file(&path) {
|
||||||
|
eprintln!(
|
||||||
|
"Failed to remove old IAM backup {}: {}",
|
||||||
|
path.display(),
|
||||||
|
err
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
println!("Pruned old IAM backup {}", path.display());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,7 +1,9 @@
|
|||||||
mod auth;
|
mod auth;
|
||||||
|
pub mod ratelimit;
|
||||||
pub mod session;
|
pub mod session;
|
||||||
|
|
||||||
pub use auth::auth_layer;
|
pub use auth::auth_layer;
|
||||||
|
pub use ratelimit::{rate_limit_layer, RateLimitLayerState};
|
||||||
pub use session::{csrf_layer, session_layer, SessionHandle, SessionLayerState};
|
pub use session::{csrf_layer, session_layer, SessionHandle, SessionLayerState};
|
||||||
|
|
||||||
use axum::extract::{Request, State};
|
use axum::extract::{Request, State};
|
||||||
241
crates/myfsio-server/src/middleware/ratelimit.rs
Normal file
241
crates/myfsio-server/src/middleware/ratelimit.rs
Normal file
@@ -0,0 +1,241 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
use std::net::SocketAddr;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
|
use axum::extract::{ConnectInfo, Request, State};
|
||||||
|
use axum::http::{header, StatusCode};
|
||||||
|
use axum::middleware::Next;
|
||||||
|
use axum::response::{IntoResponse, Response};
|
||||||
|
use parking_lot::Mutex;
|
||||||
|
|
||||||
|
use crate::config::RateLimitSetting;
|
||||||
|
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct RateLimitLayerState {
|
||||||
|
limiter: Arc<FixedWindowLimiter>,
|
||||||
|
num_trusted_proxies: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RateLimitLayerState {
|
||||||
|
pub fn new(setting: RateLimitSetting, num_trusted_proxies: usize) -> Self {
|
||||||
|
Self {
|
||||||
|
limiter: Arc::new(FixedWindowLimiter::new(setting)),
|
||||||
|
num_trusted_proxies,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug)]
|
||||||
|
struct FixedWindowLimiter {
|
||||||
|
setting: RateLimitSetting,
|
||||||
|
state: Mutex<LimiterState>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug)]
|
||||||
|
struct LimiterState {
|
||||||
|
entries: HashMap<String, LimitEntry>,
|
||||||
|
last_sweep: Instant,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy)]
|
||||||
|
struct LimitEntry {
|
||||||
|
window_started: Instant,
|
||||||
|
count: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
const SWEEP_MIN_INTERVAL: Duration = Duration::from_secs(60);
|
||||||
|
const SWEEP_ENTRY_THRESHOLD: usize = 1024;
|
||||||
|
|
||||||
|
impl FixedWindowLimiter {
|
||||||
|
fn new(setting: RateLimitSetting) -> Self {
|
||||||
|
Self {
|
||||||
|
setting,
|
||||||
|
state: Mutex::new(LimiterState {
|
||||||
|
entries: HashMap::new(),
|
||||||
|
last_sweep: Instant::now(),
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn check(&self, key: &str) -> Result<(), u64> {
|
||||||
|
let now = Instant::now();
|
||||||
|
let window = Duration::from_secs(self.setting.window_seconds.max(1));
|
||||||
|
let mut state = self.state.lock();
|
||||||
|
|
||||||
|
if state.entries.len() >= SWEEP_ENTRY_THRESHOLD
|
||||||
|
&& now.duration_since(state.last_sweep) >= SWEEP_MIN_INTERVAL
|
||||||
|
{
|
||||||
|
state
|
||||||
|
.entries
|
||||||
|
.retain(|_, entry| now.duration_since(entry.window_started) < window);
|
||||||
|
state.last_sweep = now;
|
||||||
|
}
|
||||||
|
|
||||||
|
let entry = state.entries.entry(key.to_string()).or_insert(LimitEntry {
|
||||||
|
window_started: now,
|
||||||
|
count: 0,
|
||||||
|
});
|
||||||
|
|
||||||
|
if now.duration_since(entry.window_started) >= window {
|
||||||
|
entry.window_started = now;
|
||||||
|
entry.count = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if entry.count >= self.setting.max_requests {
|
||||||
|
let elapsed = now.duration_since(entry.window_started);
|
||||||
|
let retry_after = window.saturating_sub(elapsed).as_secs().max(1);
|
||||||
|
return Err(retry_after);
|
||||||
|
}
|
||||||
|
|
||||||
|
entry.count += 1;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn rate_limit_layer(
|
||||||
|
State(state): State<RateLimitLayerState>,
|
||||||
|
req: Request,
|
||||||
|
next: Next,
|
||||||
|
) -> Response {
|
||||||
|
let key = rate_limit_key(&req, state.num_trusted_proxies);
|
||||||
|
match state.limiter.check(&key) {
|
||||||
|
Ok(()) => next.run(req).await,
|
||||||
|
Err(retry_after) => too_many_requests(retry_after),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn too_many_requests(retry_after: u64) -> Response {
|
||||||
|
(
|
||||||
|
StatusCode::TOO_MANY_REQUESTS,
|
||||||
|
[
|
||||||
|
(header::CONTENT_TYPE, "application/xml".to_string()),
|
||||||
|
(header::RETRY_AFTER, retry_after.to_string()),
|
||||||
|
],
|
||||||
|
myfsio_xml::response::rate_limit_exceeded_xml(),
|
||||||
|
)
|
||||||
|
.into_response()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn rate_limit_key(req: &Request, num_trusted_proxies: usize) -> String {
|
||||||
|
format!("ip:{}", client_ip(req, num_trusted_proxies))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn client_ip(req: &Request, num_trusted_proxies: usize) -> String {
|
||||||
|
if num_trusted_proxies > 0 {
|
||||||
|
if let Some(value) = req
|
||||||
|
.headers()
|
||||||
|
.get("x-forwarded-for")
|
||||||
|
.and_then(|v| v.to_str().ok())
|
||||||
|
{
|
||||||
|
let parts = value
|
||||||
|
.split(',')
|
||||||
|
.map(|part| part.trim())
|
||||||
|
.filter(|part| !part.is_empty())
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
if parts.len() > num_trusted_proxies {
|
||||||
|
let index = parts.len() - num_trusted_proxies - 1;
|
||||||
|
return parts[index].to_string();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(value) = req.headers().get("x-real-ip").and_then(|v| v.to_str().ok()) {
|
||||||
|
if !value.trim().is_empty() {
|
||||||
|
return value.trim().to_string();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
req.extensions()
|
||||||
|
.get::<ConnectInfo<SocketAddr>>()
|
||||||
|
.map(|ConnectInfo(addr)| addr.ip().to_string())
|
||||||
|
.unwrap_or_else(|| "unknown".to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use axum::body::Body;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn honors_trusted_proxy_count_for_forwarded_for() {
|
||||||
|
let req = Request::builder()
|
||||||
|
.header("x-forwarded-for", "198.51.100.1, 10.0.0.1, 10.0.0.2")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(rate_limit_key(&req, 2), "ip:198.51.100.1");
|
||||||
|
assert_eq!(rate_limit_key(&req, 1), "ip:10.0.0.1");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn falls_back_to_connect_info_when_forwarded_for_has_too_few_hops() {
|
||||||
|
let mut req = Request::builder()
|
||||||
|
.header("x-forwarded-for", "198.51.100.1")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap();
|
||||||
|
req.extensions_mut()
|
||||||
|
.insert(ConnectInfo(SocketAddr::from(([203, 0, 113, 9], 443))));
|
||||||
|
|
||||||
|
assert_eq!(rate_limit_key(&req, 2), "ip:203.0.113.9");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn ignores_forwarded_headers_when_no_proxies_are_trusted() {
|
||||||
|
let mut req = Request::builder()
|
||||||
|
.header("x-forwarded-for", "198.51.100.1")
|
||||||
|
.header("x-real-ip", "198.51.100.2")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap();
|
||||||
|
req.extensions_mut()
|
||||||
|
.insert(ConnectInfo(SocketAddr::from(([203, 0, 113, 9], 443))));
|
||||||
|
|
||||||
|
assert_eq!(rate_limit_key(&req, 0), "ip:203.0.113.9");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn uses_connect_info_for_direct_clients() {
|
||||||
|
let mut req = Request::builder().body(Body::empty()).unwrap();
|
||||||
|
req.extensions_mut()
|
||||||
|
.insert(ConnectInfo(SocketAddr::from(([203, 0, 113, 10], 443))));
|
||||||
|
|
||||||
|
assert_eq!(rate_limit_key(&req, 0), "ip:203.0.113.10");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn fixed_window_rejects_after_quota() {
|
||||||
|
let limiter = FixedWindowLimiter::new(RateLimitSetting::new(2, 60));
|
||||||
|
assert!(limiter.check("k").is_ok());
|
||||||
|
assert!(limiter.check("k").is_ok());
|
||||||
|
assert!(limiter.check("k").is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sweep_removes_expired_entries() {
|
||||||
|
let limiter = FixedWindowLimiter::new(RateLimitSetting::new(10, 1));
|
||||||
|
let far_past = Instant::now() - (SWEEP_MIN_INTERVAL + Duration::from_secs(5));
|
||||||
|
{
|
||||||
|
let mut state = limiter.state.lock();
|
||||||
|
for i in 0..(SWEEP_ENTRY_THRESHOLD + 1024) {
|
||||||
|
state.entries.insert(
|
||||||
|
format!("stale-{}", i),
|
||||||
|
LimitEntry {
|
||||||
|
window_started: far_past,
|
||||||
|
count: 5,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
state.last_sweep = far_past;
|
||||||
|
}
|
||||||
|
let seeded = limiter.state.lock().entries.len();
|
||||||
|
assert_eq!(seeded, SWEEP_ENTRY_THRESHOLD + 1024);
|
||||||
|
|
||||||
|
assert!(limiter.check("fresh").is_ok());
|
||||||
|
|
||||||
|
let remaining = limiter.state.lock().entries.len();
|
||||||
|
assert_eq!(
|
||||||
|
remaining, 1,
|
||||||
|
"expected sweep to leave only the fresh entry, got {}",
|
||||||
|
remaining
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -24,6 +24,35 @@ impl Default for GcConfig {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn dry_run_reports_but_does_not_delete_temp_files() {
|
||||||
|
let tmp = tempfile::tempdir().unwrap();
|
||||||
|
let tmp_dir = tmp.path().join(".myfsio.sys").join("tmp");
|
||||||
|
std::fs::create_dir_all(&tmp_dir).unwrap();
|
||||||
|
let file_path = tmp_dir.join("stale.tmp");
|
||||||
|
std::fs::write(&file_path, b"temporary").unwrap();
|
||||||
|
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
|
||||||
|
|
||||||
|
let service = GcService::new(
|
||||||
|
tmp.path().to_path_buf(),
|
||||||
|
GcConfig {
|
||||||
|
temp_file_max_age_hours: 0.0,
|
||||||
|
dry_run: true,
|
||||||
|
..GcConfig::default()
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
let result = service.run_now(false).await.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(result["temp_files_deleted"], 1);
|
||||||
|
assert!(file_path.exists());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
pub struct GcService {
|
pub struct GcService {
|
||||||
storage_root: PathBuf,
|
storage_root: PathBuf,
|
||||||
config: GcConfig,
|
config: GcConfig,
|
||||||
@@ -15,9 +15,9 @@ use crate::session::SessionStore;
|
|||||||
use crate::stores::connections::ConnectionStore;
|
use crate::stores::connections::ConnectionStore;
|
||||||
use crate::templates::TemplateEngine;
|
use crate::templates::TemplateEngine;
|
||||||
use myfsio_auth::iam::IamService;
|
use myfsio_auth::iam::IamService;
|
||||||
use myfsio_crypto::encryption::EncryptionService;
|
use myfsio_crypto::encryption::{EncryptionConfig, EncryptionService};
|
||||||
use myfsio_crypto::kms::KmsService;
|
use myfsio_crypto::kms::KmsService;
|
||||||
use myfsio_storage::fs_backend::FsStorageBackend;
|
use myfsio_storage::fs_backend::{FsStorageBackend, FsStorageBackendConfig};
|
||||||
|
|
||||||
#[derive(Clone)]
|
#[derive(Clone)]
|
||||||
pub struct AppState {
|
pub struct AppState {
|
||||||
@@ -42,7 +42,16 @@ pub struct AppState {
|
|||||||
|
|
||||||
impl AppState {
|
impl AppState {
|
||||||
pub fn new(config: ServerConfig) -> Self {
|
pub fn new(config: ServerConfig) -> Self {
|
||||||
let storage = Arc::new(FsStorageBackend::new(config.storage_root.clone()));
|
let storage = Arc::new(FsStorageBackend::new_with_config(
|
||||||
|
config.storage_root.clone(),
|
||||||
|
FsStorageBackendConfig {
|
||||||
|
object_key_max_length_bytes: config.object_key_max_length_bytes,
|
||||||
|
object_cache_max_size: config.object_cache_max_size,
|
||||||
|
bucket_config_cache_ttl: Duration::from_secs_f64(
|
||||||
|
config.bucket_config_cache_ttl_seconds,
|
||||||
|
),
|
||||||
|
},
|
||||||
|
));
|
||||||
let iam = Arc::new(IamService::new_with_secret(
|
let iam = Arc::new(IamService::new_with_secret(
|
||||||
config.iam_config_path.clone(),
|
config.iam_config_path.clone(),
|
||||||
config.secret_key.clone(),
|
config.secret_key.clone(),
|
||||||
@@ -51,7 +60,13 @@ impl AppState {
|
|||||||
let gc = if config.gc_enabled {
|
let gc = if config.gc_enabled {
|
||||||
Some(Arc::new(GcService::new(
|
Some(Arc::new(GcService::new(
|
||||||
config.storage_root.clone(),
|
config.storage_root.clone(),
|
||||||
crate::services::gc::GcConfig::default(),
|
crate::services::gc::GcConfig {
|
||||||
|
interval_hours: config.gc_interval_hours,
|
||||||
|
temp_file_max_age_hours: config.gc_temp_file_max_age_hours,
|
||||||
|
multipart_max_age_days: config.gc_multipart_max_age_days,
|
||||||
|
lock_file_max_age_hours: config.gc_lock_file_max_age_hours,
|
||||||
|
dry_run: config.gc_dry_run,
|
||||||
|
},
|
||||||
)))
|
)))
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
@@ -92,7 +107,22 @@ impl AppState {
|
|||||||
None
|
None
|
||||||
};
|
};
|
||||||
|
|
||||||
let site_registry = Some(Arc::new(SiteRegistry::new(&config.storage_root)));
|
let site_registry = {
|
||||||
|
let registry = SiteRegistry::new(&config.storage_root);
|
||||||
|
if let (Some(site_id), Some(endpoint)) =
|
||||||
|
(config.site_id.as_deref(), config.site_endpoint.as_deref())
|
||||||
|
{
|
||||||
|
registry.set_local_site(crate::services::site_registry::SiteInfo {
|
||||||
|
site_id: site_id.to_string(),
|
||||||
|
endpoint: endpoint.to_string(),
|
||||||
|
region: config.site_region.clone(),
|
||||||
|
priority: config.site_priority,
|
||||||
|
display_name: site_id.to_string(),
|
||||||
|
created_at: Some(chrono::Utc::now().to_rfc3339()),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Some(Arc::new(registry))
|
||||||
|
};
|
||||||
|
|
||||||
let website_domains = if config.website_hosting_enabled {
|
let website_domains = if config.website_hosting_enabled {
|
||||||
Some(Arc::new(WebsiteDomainStore::new(&config.storage_root)))
|
Some(Arc::new(WebsiteDomainStore::new(&config.storage_root)))
|
||||||
@@ -132,6 +162,7 @@ impl AppState {
|
|||||||
|
|
||||||
let templates = init_templates(&config.templates_dir);
|
let templates = init_templates(&config.templates_dir);
|
||||||
let access_logging = Arc::new(AccessLoggingService::new(&config.storage_root));
|
let access_logging = Arc::new(AccessLoggingService::new(&config.storage_root));
|
||||||
|
let session_ttl = Duration::from_secs(config.session_lifetime_days.saturating_mul(86_400));
|
||||||
Self {
|
Self {
|
||||||
config,
|
config,
|
||||||
storage,
|
storage,
|
||||||
@@ -148,7 +179,7 @@ impl AppState {
|
|||||||
replication,
|
replication,
|
||||||
site_sync,
|
site_sync,
|
||||||
templates,
|
templates,
|
||||||
sessions: Arc::new(SessionStore::new(Duration::from_secs(60 * 60 * 12))),
|
sessions: Arc::new(SessionStore::new(session_ttl)),
|
||||||
access_logging,
|
access_logging,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -172,7 +203,13 @@ impl AppState {
|
|||||||
|
|
||||||
let encryption = if config.encryption_enabled {
|
let encryption = if config.encryption_enabled {
|
||||||
match myfsio_crypto::kms::load_or_create_master_key(&keys_dir).await {
|
match myfsio_crypto::kms::load_or_create_master_key(&keys_dir).await {
|
||||||
Ok(master_key) => Some(Arc::new(EncryptionService::new(master_key, kms.clone()))),
|
Ok(master_key) => Some(Arc::new(EncryptionService::with_config(
|
||||||
|
master_key,
|
||||||
|
kms.clone(),
|
||||||
|
EncryptionConfig {
|
||||||
|
chunk_size: config.encryption_chunk_size_bytes,
|
||||||
|
},
|
||||||
|
))),
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::error!("Failed to initialize encryption: {}", e);
|
tracing::error!("Failed to initialize encryption: {}", e);
|
||||||
None
|
None
|
||||||
|
Before Width: | Height: | Size: 200 KiB After Width: | Height: | Size: 200 KiB |
|
Before Width: | Height: | Size: 872 KiB After Width: | Height: | Size: 872 KiB |
@@ -73,16 +73,13 @@
|
|||||||
</div>
|
</div>
|
||||||
<p class="text-muted">Build or run the Rust server and launch the API plus web UI from a single process.</p>
|
<p class="text-muted">Build or run the Rust server and launch the API plus web UI from a single process.</p>
|
||||||
<div class="alert alert-light border small mb-3">
|
<div class="alert alert-light border small mb-3">
|
||||||
Runtime note: MyFSIO now runs from the Rust server in <code>rust/myfsio-engine</code>. For the verified runtime configuration list, use the repository <code>docs.md</code>.
|
Runtime note: the repository root is the Cargo workspace. For the verified runtime configuration list, use the repository <code>docs.md</code>.
|
||||||
</div>
|
</div>
|
||||||
<ol class="docs-steps">
|
<ol class="docs-steps">
|
||||||
<li>Install a current Rust toolchain.</li>
|
<li>Install a current Rust toolchain.</li>
|
||||||
<li>Change into <code>rust/myfsio-engine</code>.</li>
|
<li>From the repository root, start the server with <code>cargo run -p myfsio-server --</code>.</li>
|
||||||
<li>Start the server with <code>cargo run -p myfsio-server --</code>.</li>
|
|
||||||
</ol>
|
</ol>
|
||||||
<pre class="mb-3"><code class="language-bash">cd rust/myfsio-engine
|
<pre class="mb-3"><code class="language-bash"># Run API + UI
|
||||||
|
|
||||||
# Run API + UI
|
|
||||||
cargo run -p myfsio-server --
|
cargo run -p myfsio-server --
|
||||||
|
|
||||||
# Show resolved configuration
|
# Show resolved configuration
|
||||||
@@ -112,7 +109,7 @@ cargo build --release -p myfsio-server
|
|||||||
<tbody>
|
<tbody>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>API_BASE_URL</code></td>
|
<td><code>API_BASE_URL</code></td>
|
||||||
<td><code>http://127.0.0.1:5000</code></td>
|
<td>Derived from <code>HOST</code>/<code>PORT</code></td>
|
||||||
<td>Internal S3 API URL used by the web UI proxy. Also used for presigned URL generation. Set to your public URL if running behind a reverse proxy.</td>
|
<td>Internal S3 API URL used by the web UI proxy. Also used for presigned URL generation. Set to your public URL if running behind a reverse proxy.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
@@ -184,33 +181,18 @@ cargo build --release -p myfsio-server
|
|||||||
<tr>
|
<tr>
|
||||||
<td><code>RATE_LIMIT_DEFAULT</code></td>
|
<td><code>RATE_LIMIT_DEFAULT</code></td>
|
||||||
<td><code>200 per minute</code></td>
|
<td><code>200 per minute</code></td>
|
||||||
<td>Default API rate limit.</td>
|
<td>Default rate limit for S3 and KMS API endpoints.</td>
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><code>RATE_LIMIT_LIST_BUCKETS</code></td>
|
|
||||||
<td><code>60 per minute</code></td>
|
|
||||||
<td>Rate limit for listing buckets.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><code>RATE_LIMIT_BUCKET_OPS</code></td>
|
|
||||||
<td><code>120 per minute</code></td>
|
|
||||||
<td>Rate limit for bucket operations.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><code>RATE_LIMIT_OBJECT_OPS</code></td>
|
|
||||||
<td><code>240 per minute</code></td>
|
|
||||||
<td>Rate limit for object operations.</td>
|
|
||||||
</tr>
|
|
||||||
<tr>
|
|
||||||
<td><code>RATE_LIMIT_HEAD_OPS</code></td>
|
|
||||||
<td><code>100 per minute</code></td>
|
|
||||||
<td>Rate limit for HEAD requests.</td>
|
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>RATE_LIMIT_ADMIN</code></td>
|
<td><code>RATE_LIMIT_ADMIN</code></td>
|
||||||
<td><code>60 per minute</code></td>
|
<td><code>60 per minute</code></td>
|
||||||
<td>Rate limit for admin API endpoints (<code>/admin/*</code>).</td>
|
<td>Rate limit for admin API endpoints (<code>/admin/*</code>).</td>
|
||||||
</tr>
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>RATE_LIMIT_STORAGE_URI</code></td>
|
||||||
|
<td><code>memory://</code></td>
|
||||||
|
<td>Rate limit storage backend. Only in-memory storage is currently supported.</td>
|
||||||
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>ADMIN_ACCESS_KEY</code></td>
|
<td><code>ADMIN_ACCESS_KEY</code></td>
|
||||||
<td>(none)</td>
|
<td>(none)</td>
|
||||||
@@ -377,8 +359,8 @@ cargo build --release -p myfsio-server
|
|||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>NUM_TRUSTED_PROXIES</code></td>
|
<td><code>NUM_TRUSTED_PROXIES</code></td>
|
||||||
<td><code>1</code></td>
|
<td><code>0</code></td>
|
||||||
<td>Number of trusted reverse proxies for <code>X-Forwarded-*</code> headers.</td>
|
<td>Number of trusted reverse proxies for <code>X-Forwarded-*</code> headers. Forwarded IP headers are ignored when this is <code>0</code>.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>ALLOWED_REDIRECT_HOSTS</code></td>
|
<td><code>ALLOWED_REDIRECT_HOSTS</code></td>
|
||||||
@@ -2078,7 +2060,7 @@ curl "{{ api_base | replace(from="/api", to="/ui") }}/metrics/operations/history
|
|||||||
<tr>
|
<tr>
|
||||||
<td>Large folder uploads hitting rate limits (429)</td>
|
<td>Large folder uploads hitting rate limits (429)</td>
|
||||||
<td><code>RATE_LIMIT_DEFAULT</code> exceeded (200/min)</td>
|
<td><code>RATE_LIMIT_DEFAULT</code> exceeded (200/min)</td>
|
||||||
<td>Increase rate limit in env config, use Redis backend (<code>RATE_LIMIT_STORAGE_URI=redis://host:port</code>) for distributed setups, or upload in smaller batches.</td>
|
<td>Increase <code>RATE_LIMIT_DEFAULT</code> in env config or upload in smaller batches. Distributed rate-limit storage is not supported yet.</td>
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
@@ -133,7 +133,7 @@
|
|||||||
{% endif %}
|
{% endif %}
|
||||||
<div class="row g-3">
|
<div class="row g-3">
|
||||||
{% for user in users %}
|
{% for user in users %}
|
||||||
<div class="col-md-6 col-xl-4 iam-user-item" data-user-id="{{ user.user_id }}" data-access-key="{{ user.access_key }}" data-display-name="{{ user.display_name|lower }}" data-access-key-filter="{{ user.access_key|lower }}">
|
<div class="col-md-6 col-xl-4 iam-user-item" data-user-id="{{ user.user_id }}" data-access-key="{{ user.access_key }}" data-display-name="{{ user.display_name|lower }}" data-access-key-filter="{{ user.access_key|lower }}" data-update-url="{{ url_for(endpoint="ui.update_iam_user", user_id=user.user_id) }}">
|
||||||
<div class="card h-100 iam-user-card{% if user.is_admin %} iam-admin-card{% else %}{% endif %}">
|
<div class="card h-100 iam-user-card{% if user.is_admin %} iam-admin-card{% else %}{% endif %}">
|
||||||
<div class="card-body">
|
<div class="card-body">
|
||||||
<div class="d-flex align-items-start justify-content-between mb-3">
|
<div class="d-flex align-items-start justify-content-between mb-3">
|
||||||
@@ -53,6 +53,7 @@ fn test_app_with_iam(iam_json: serde_json::Value) -> (axum::Router, tempfile::Te
|
|||||||
ui_enabled: false,
|
ui_enabled: false,
|
||||||
templates_dir: std::path::PathBuf::from("templates"),
|
templates_dir: std::path::PathBuf::from("templates"),
|
||||||
static_dir: std::path::PathBuf::from("static"),
|
static_dir: std::path::PathBuf::from("static"),
|
||||||
|
..myfsio_server::config::ServerConfig::default()
|
||||||
};
|
};
|
||||||
let state = myfsio_server::state::AppState::new(config);
|
let state = myfsio_server::state::AppState::new(config);
|
||||||
let app = myfsio_server::create_router(state);
|
let app = myfsio_server::create_router(state);
|
||||||
@@ -80,6 +81,120 @@ fn test_app() -> (axum::Router, tempfile::TempDir) {
|
|||||||
}))
|
}))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn test_app_with_rate_limits(
|
||||||
|
default: myfsio_server::config::RateLimitSetting,
|
||||||
|
admin: myfsio_server::config::RateLimitSetting,
|
||||||
|
) -> (axum::Router, tempfile::TempDir) {
|
||||||
|
let tmp = tempfile::TempDir::new().unwrap();
|
||||||
|
let iam_path = tmp.path().join(".myfsio.sys").join("config");
|
||||||
|
std::fs::create_dir_all(&iam_path).unwrap();
|
||||||
|
std::fs::write(
|
||||||
|
iam_path.join("iam.json"),
|
||||||
|
serde_json::json!({
|
||||||
|
"version": 2,
|
||||||
|
"users": [{
|
||||||
|
"user_id": "u-test1234",
|
||||||
|
"display_name": "admin",
|
||||||
|
"enabled": true,
|
||||||
|
"access_keys": [{
|
||||||
|
"access_key": TEST_ACCESS_KEY,
|
||||||
|
"secret_key": TEST_SECRET_KEY,
|
||||||
|
"status": "active"
|
||||||
|
}],
|
||||||
|
"policies": [{
|
||||||
|
"bucket": "*",
|
||||||
|
"actions": ["*"],
|
||||||
|
"prefix": "*"
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
})
|
||||||
|
.to_string(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let config = myfsio_server::config::ServerConfig {
|
||||||
|
bind_addr: "127.0.0.1:0".parse().unwrap(),
|
||||||
|
ui_bind_addr: "127.0.0.1:0".parse().unwrap(),
|
||||||
|
storage_root: tmp.path().to_path_buf(),
|
||||||
|
iam_config_path: iam_path.join("iam.json"),
|
||||||
|
ratelimit_default: default,
|
||||||
|
ratelimit_admin: admin,
|
||||||
|
ui_enabled: false,
|
||||||
|
..myfsio_server::config::ServerConfig::default()
|
||||||
|
};
|
||||||
|
let state = myfsio_server::state::AppState::new(config);
|
||||||
|
let app = myfsio_server::create_router(state);
|
||||||
|
(app, tmp)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn rate_limit_default_and_admin_are_independent() {
|
||||||
|
let (app, _tmp) = test_app_with_rate_limits(
|
||||||
|
myfsio_server::config::RateLimitSetting::new(1, 60),
|
||||||
|
myfsio_server::config::RateLimitSetting::new(2, 60),
|
||||||
|
);
|
||||||
|
|
||||||
|
let first = app
|
||||||
|
.clone()
|
||||||
|
.oneshot(
|
||||||
|
Request::builder()
|
||||||
|
.uri("/myfsio/health")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(first.status(), StatusCode::OK);
|
||||||
|
|
||||||
|
let second = app
|
||||||
|
.clone()
|
||||||
|
.oneshot(
|
||||||
|
Request::builder()
|
||||||
|
.uri("/myfsio/health")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(second.status(), StatusCode::TOO_MANY_REQUESTS);
|
||||||
|
assert!(second.headers().contains_key("retry-after"));
|
||||||
|
|
||||||
|
let admin_first = app
|
||||||
|
.clone()
|
||||||
|
.oneshot(
|
||||||
|
Request::builder()
|
||||||
|
.uri("/admin/gc/status")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(admin_first.status(), StatusCode::FORBIDDEN);
|
||||||
|
|
||||||
|
let admin_second = app
|
||||||
|
.clone()
|
||||||
|
.oneshot(
|
||||||
|
Request::builder()
|
||||||
|
.uri("/admin/gc/status")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(admin_second.status(), StatusCode::FORBIDDEN);
|
||||||
|
|
||||||
|
let admin_third = app
|
||||||
|
.oneshot(
|
||||||
|
Request::builder()
|
||||||
|
.uri("/admin/gc/status")
|
||||||
|
.body(Body::empty())
|
||||||
|
.unwrap(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(admin_third.status(), StatusCode::TOO_MANY_REQUESTS);
|
||||||
|
}
|
||||||
|
|
||||||
fn test_ui_state() -> (myfsio_server::state::AppState, tempfile::TempDir) {
|
fn test_ui_state() -> (myfsio_server::state::AppState, tempfile::TempDir) {
|
||||||
let tmp = tempfile::TempDir::new().unwrap();
|
let tmp = tempfile::TempDir::new().unwrap();
|
||||||
let iam_path = tmp.path().join(".myfsio.sys").join("config");
|
let iam_path = tmp.path().join(".myfsio.sys").join("config");
|
||||||
@@ -147,6 +262,7 @@ fn test_ui_state() -> (myfsio_server::state::AppState, tempfile::TempDir) {
|
|||||||
ui_enabled: true,
|
ui_enabled: true,
|
||||||
templates_dir: manifest_dir.join("templates"),
|
templates_dir: manifest_dir.join("templates"),
|
||||||
static_dir: manifest_dir.join("static"),
|
static_dir: manifest_dir.join("static"),
|
||||||
|
..myfsio_server::config::ServerConfig::default()
|
||||||
};
|
};
|
||||||
(myfsio_server::state::AppState::new(config), tmp)
|
(myfsio_server::state::AppState::new(config), tmp)
|
||||||
}
|
}
|
||||||
@@ -303,6 +419,7 @@ fn test_website_state() -> (myfsio_server::state::AppState, tempfile::TempDir) {
|
|||||||
ui_enabled: false,
|
ui_enabled: false,
|
||||||
templates_dir: std::path::PathBuf::from("templates"),
|
templates_dir: std::path::PathBuf::from("templates"),
|
||||||
static_dir: std::path::PathBuf::from("static"),
|
static_dir: std::path::PathBuf::from("static"),
|
||||||
|
..myfsio_server::config::ServerConfig::default()
|
||||||
};
|
};
|
||||||
(myfsio_server::state::AppState::new(config), tmp)
|
(myfsio_server::state::AppState::new(config), tmp)
|
||||||
}
|
}
|
||||||
@@ -1082,7 +1199,7 @@ async fn test_ui_metrics_history_endpoint_reads_system_history() {
|
|||||||
config_root.join("metrics_history.json"),
|
config_root.join("metrics_history.json"),
|
||||||
serde_json::json!({
|
serde_json::json!({
|
||||||
"history": [{
|
"history": [{
|
||||||
"timestamp": "2026-04-20T00:00:00Z",
|
"timestamp": chrono::Utc::now().to_rfc3339(),
|
||||||
"cpu_percent": 12.5,
|
"cpu_percent": 12.5,
|
||||||
"memory_percent": 33.3,
|
"memory_percent": 33.3,
|
||||||
"disk_percent": 44.4,
|
"disk_percent": 44.4,
|
||||||
@@ -1131,6 +1248,7 @@ async fn test_ui_metrics_history_endpoint_reads_system_history() {
|
|||||||
ui_enabled: true,
|
ui_enabled: true,
|
||||||
templates_dir: manifest_dir.join("templates"),
|
templates_dir: manifest_dir.join("templates"),
|
||||||
static_dir: manifest_dir.join("static"),
|
static_dir: manifest_dir.join("static"),
|
||||||
|
..myfsio_server::config::ServerConfig::default()
|
||||||
};
|
};
|
||||||
let state = myfsio_server::state::AppState::new(config);
|
let state = myfsio_server::state::AppState::new(config);
|
||||||
let (session_id, _csrf) = authenticated_ui_session(&state);
|
let (session_id, _csrf) = authenticated_ui_session(&state);
|
||||||
@@ -3851,6 +3969,7 @@ async fn test_non_admin_authorization_enforced() {
|
|||||||
ui_enabled: false,
|
ui_enabled: false,
|
||||||
templates_dir: std::path::PathBuf::from("templates"),
|
templates_dir: std::path::PathBuf::from("templates"),
|
||||||
static_dir: std::path::PathBuf::from("static"),
|
static_dir: std::path::PathBuf::from("static"),
|
||||||
|
..myfsio_server::config::ServerConfig::default()
|
||||||
};
|
};
|
||||||
let state = myfsio_server::state::AppState::new(config);
|
let state = myfsio_server::state::AppState::new(config);
|
||||||
state.storage.create_bucket("authz-bucket").await.unwrap();
|
state.storage.create_bucket("authz-bucket").await.unwrap();
|
||||||
@@ -3932,6 +4051,7 @@ async fn test_app_encrypted() -> (axum::Router, tempfile::TempDir) {
|
|||||||
ui_enabled: false,
|
ui_enabled: false,
|
||||||
templates_dir: std::path::PathBuf::from("templates"),
|
templates_dir: std::path::PathBuf::from("templates"),
|
||||||
static_dir: std::path::PathBuf::from("static"),
|
static_dir: std::path::PathBuf::from("static"),
|
||||||
|
..myfsio_server::config::ServerConfig::default()
|
||||||
};
|
};
|
||||||
let state = myfsio_server::state::AppState::new_with_encryption(config).await;
|
let state = myfsio_server::state::AppState::new_with_encryption(config).await;
|
||||||
let app = myfsio_server::create_router(state);
|
let app = myfsio_server::create_router(state);
|
||||||
@@ -19,6 +19,7 @@ use uuid::Uuid;
|
|||||||
pub struct FsStorageBackend {
|
pub struct FsStorageBackend {
|
||||||
root: PathBuf,
|
root: PathBuf,
|
||||||
object_key_max_length_bytes: usize,
|
object_key_max_length_bytes: usize,
|
||||||
|
object_cache_max_size: usize,
|
||||||
bucket_config_cache: DashMap<String, (BucketConfig, Instant)>,
|
bucket_config_cache: DashMap<String, (BucketConfig, Instant)>,
|
||||||
bucket_config_cache_ttl: std::time::Duration,
|
bucket_config_cache_ttl: std::time::Duration,
|
||||||
meta_read_cache: DashMap<(String, String), Option<HashMap<String, Value>>>,
|
meta_read_cache: DashMap<(String, String), Option<HashMap<String, Value>>>,
|
||||||
@@ -27,13 +28,35 @@ pub struct FsStorageBackend {
|
|||||||
stats_cache_ttl: std::time::Duration,
|
stats_cache_ttl: std::time::Duration,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct FsStorageBackendConfig {
|
||||||
|
pub object_key_max_length_bytes: usize,
|
||||||
|
pub object_cache_max_size: usize,
|
||||||
|
pub bucket_config_cache_ttl: std::time::Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for FsStorageBackendConfig {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
object_key_max_length_bytes: DEFAULT_OBJECT_KEY_MAX_BYTES,
|
||||||
|
object_cache_max_size: 100,
|
||||||
|
bucket_config_cache_ttl: std::time::Duration::from_secs(30),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl FsStorageBackend {
|
impl FsStorageBackend {
|
||||||
pub fn new(root: PathBuf) -> Self {
|
pub fn new(root: PathBuf) -> Self {
|
||||||
|
Self::new_with_config(root, FsStorageBackendConfig::default())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn new_with_config(root: PathBuf, config: FsStorageBackendConfig) -> Self {
|
||||||
let backend = Self {
|
let backend = Self {
|
||||||
root,
|
root,
|
||||||
object_key_max_length_bytes: DEFAULT_OBJECT_KEY_MAX_BYTES,
|
object_key_max_length_bytes: config.object_key_max_length_bytes,
|
||||||
|
object_cache_max_size: config.object_cache_max_size,
|
||||||
bucket_config_cache: DashMap::new(),
|
bucket_config_cache: DashMap::new(),
|
||||||
bucket_config_cache_ttl: std::time::Duration::from_secs(30),
|
bucket_config_cache_ttl: config.bucket_config_cache_ttl,
|
||||||
meta_read_cache: DashMap::new(),
|
meta_read_cache: DashMap::new(),
|
||||||
meta_index_locks: DashMap::new(),
|
meta_index_locks: DashMap::new(),
|
||||||
stats_cache: DashMap::new(),
|
stats_cache: DashMap::new(),
|
||||||
@@ -142,6 +165,27 @@ impl FsStorageBackend {
|
|||||||
.clone()
|
.clone()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn prune_meta_read_cache(&self) {
|
||||||
|
if self.object_cache_max_size == 0 {
|
||||||
|
self.meta_read_cache.clear();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let len = self.meta_read_cache.len();
|
||||||
|
if len <= self.object_cache_max_size {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let excess = len - self.object_cache_max_size;
|
||||||
|
let keys = self
|
||||||
|
.meta_read_cache
|
||||||
|
.iter()
|
||||||
|
.take(excess)
|
||||||
|
.map(|entry| entry.key().clone())
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
for key in keys {
|
||||||
|
self.meta_read_cache.remove(&key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn bucket_config_path(&self, bucket_name: &str) -> PathBuf {
|
fn bucket_config_path(&self, bucket_name: &str) -> PathBuf {
|
||||||
self.system_bucket_root(bucket_name)
|
self.system_bucket_root(bucket_name)
|
||||||
.join(BUCKET_CONFIG_FILE)
|
.join(BUCKET_CONFIG_FILE)
|
||||||
@@ -229,6 +273,7 @@ impl FsStorageBackend {
|
|||||||
};
|
};
|
||||||
|
|
||||||
self.meta_read_cache.insert(cache_key, result.clone());
|
self.meta_read_cache.insert(cache_key, result.clone());
|
||||||
|
self.prune_meta_read_cache();
|
||||||
result
|
result
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -8,6 +8,12 @@ pub fn format_s3_datetime(dt: &DateTime<Utc>) -> String {
|
|||||||
dt.format("%Y-%m-%dT%H:%M:%S%.3fZ").to_string()
|
dt.format("%Y-%m-%dT%H:%M:%S%.3fZ").to_string()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub fn rate_limit_exceeded_xml() -> String {
|
||||||
|
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
|
||||||
|
<Error><Code>SlowDown</Code><Message>Rate limit exceeded</Message></Error>"
|
||||||
|
.to_string()
|
||||||
|
}
|
||||||
|
|
||||||
pub fn list_buckets_xml(owner_id: &str, owner_name: &str, buckets: &[BucketMeta]) -> String {
|
pub fn list_buckets_xml(owner_id: &str, owner_name: &str, buckets: &[BucketMeta]) -> String {
|
||||||
let mut writer = Writer::new(Cursor::new(Vec::new()));
|
let mut writer = Writer::new(Cursor::new(Vec::new()));
|
||||||
|
|
||||||
34
docs.md
34
docs.md
@@ -1,26 +1,19 @@
|
|||||||
# MyFSIO Rust Operations Guide
|
# MyFSIO Rust Operations Guide
|
||||||
|
|
||||||
This document describes the current Rust server in `rust/myfsio-engine`. It replaces the older Python-oriented runbook.
|
This document describes the MyFSIO Rust server. The repository root is the Cargo workspace.
|
||||||
|
|
||||||
The `python/` implementation is deprecated as of 2026-04-21. It is retained for migration reference and legacy validation only; production usage and new development should use the Rust server.
|
## 1. Overview
|
||||||
|
|
||||||
## 1. What Changed
|
|
||||||
|
|
||||||
The active runtime is now Rust:
|
|
||||||
|
|
||||||
- One process serves both the S3 API and the web UI.
|
- One process serves both the S3 API and the web UI.
|
||||||
- The server entrypoint is `myfsio-server`.
|
- The server entrypoint is `myfsio-server`.
|
||||||
- The main development workflow is `cargo run -p myfsio-server --`.
|
- The main development workflow is `cargo run -p myfsio-server --`.
|
||||||
- API-only mode is controlled with `UI_ENABLED=false`.
|
- API-only mode is controlled with `UI_ENABLED=false`.
|
||||||
|
|
||||||
The deprecated `python/` directory may still contain older implementation code, templates, and tests, but it is not required to run the current server.
|
|
||||||
|
|
||||||
## 2. Quick Start
|
## 2. Quick Start
|
||||||
|
|
||||||
From the repository root:
|
From the repository root:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo run -p myfsio-server --
|
cargo run -p myfsio-server --
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -34,10 +27,9 @@ On first startup, MyFSIO bootstraps an admin user in `data/.myfsio.sys/config/ia
|
|||||||
|
|
||||||
### Windows
|
### Windows
|
||||||
|
|
||||||
From PowerShell:
|
From PowerShell at the repository root:
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
cd rust\myfsio-engine
|
|
||||||
cargo run -p myfsio-server --
|
cargo run -p myfsio-server --
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -52,7 +44,6 @@ There is no separate UI-only mode in the Rust server.
|
|||||||
## 3. Build and Run a Binary
|
## 3. Build and Run a Binary
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo build --release -p myfsio-server
|
cargo build --release -p myfsio-server
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -104,7 +95,7 @@ That makes local development and systemd installs behave consistently.
|
|||||||
|
|
||||||
## 6. Verified Configuration Reference
|
## 6. Verified Configuration Reference
|
||||||
|
|
||||||
These values are taken from `rust/myfsio-engine/crates/myfsio-server/src/config.rs`.
|
These values are taken from `crates/myfsio-server/src/config.rs`.
|
||||||
|
|
||||||
### Network and runtime
|
### Network and runtime
|
||||||
|
|
||||||
@@ -311,10 +302,10 @@ Notes:
|
|||||||
|
|
||||||
## 10. Docker
|
## 10. Docker
|
||||||
|
|
||||||
Build the Rust image from the `rust/` directory:
|
Build the Rust image from the repository root:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker build -t myfsio ./rust
|
docker build -t myfsio .
|
||||||
docker run --rm \
|
docker run --rm \
|
||||||
-p 5000:5000 \
|
-p 5000:5000 \
|
||||||
-p 5100:5100 \
|
-p 5100:5100 \
|
||||||
@@ -337,11 +328,9 @@ If you want generated links and presigned URLs to use an external hostname, set
|
|||||||
The repository includes `scripts/install.sh`. For the Rust server, build the binary first and pass the path explicitly:
|
The repository includes `scripts/install.sh`. For the Rust server, build the binary first and pass the path explicitly:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo build --release -p myfsio-server
|
cargo build --release -p myfsio-server
|
||||||
|
|
||||||
cd ../..
|
sudo ./scripts/install.sh --binary ./target/release/myfsio-server
|
||||||
sudo ./scripts/install.sh --binary ./rust/myfsio-engine/target/release/myfsio-server
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The installer copies that binary to `/opt/myfsio/myfsio`, creates `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` systemd unit.
|
The installer copies that binary to `/opt/myfsio/myfsio`, creates `/opt/myfsio/myfsio.env`, and can register a `myfsio.service` systemd unit.
|
||||||
@@ -396,14 +385,13 @@ The command:
|
|||||||
Run the Rust test suite:
|
Run the Rust test suite:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd rust/myfsio-engine
|
|
||||||
cargo test
|
cargo test
|
||||||
```
|
```
|
||||||
|
|
||||||
If you are validating documentation changes for the UI, the most relevant coverage lives under:
|
If you are validating documentation changes for the UI, the most relevant coverage lives under:
|
||||||
|
|
||||||
- `rust/myfsio-engine/crates/myfsio-server/tests`
|
- `crates/myfsio-server/tests`
|
||||||
- `rust/myfsio-engine/crates/myfsio-storage/src`
|
- `crates/myfsio-storage/src`
|
||||||
|
|
||||||
## 15. API Notes
|
## 15. API Notes
|
||||||
|
|
||||||
@@ -417,5 +405,5 @@ The Rust server exposes:
|
|||||||
|
|
||||||
For a route-level view, inspect:
|
For a route-level view, inspect:
|
||||||
|
|
||||||
- `rust/myfsio-engine/crates/myfsio-server/src/lib.rs`
|
- `crates/myfsio-server/src/lib.rs`
|
||||||
- `rust/myfsio-engine/crates/myfsio-server/src/handlers/`
|
- `crates/myfsio-server/src/handlers/`
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user