kqjy/MyFSIO
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 33 Wiki Activity

Add bidirectional mode option to replication panel UI

Browse Source
This commit is contained in:
kqjy
2026-01-25 12:35:14 +08:00
parent 23ea164215
commit 87c7f1bc7d
4 changed files with 299 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
app/ui.py
View File

@@ -470,6 +470,7 @@ def bucket_detail(bucket_name: str):
kms_enabled = current_app.config.get("KMS_ENABLED", False)
encryption_enabled = current_app.config.get("ENCRYPTION_ENABLED", False)
lifecycle_enabled = current_app.config.get("LIFECYCLE_ENABLED", False)
site_sync_enabled = current_app.config.get("SITE_SYNC_ENABLED", False)
can_manage_encryption = can_manage_versioning
bucket_quota = storage.get_bucket_quota(bucket_name)
@@ -522,6 +523,7 @@ def bucket_detail(bucket_name: str):
bucket_quota=bucket_quota,
bucket_stats=bucket_stats,
can_manage_quota=can_manage_quota,
site_sync_enabled=site_sync_enabled,
)

119
docs.md
View File

@@ -1248,12 +1248,22 @@ Replication uses a two-tier permission system:
This separation allows administrators to pre-configure where data should replicate, while allowing authorized users to toggle replication on/off without accessing connection credentials.
### Replication Modes
| Mode | Behavior |
|------|----------|
| `new_only` | Only replicate new/modified objects (default) |
| `all` | Sync all existing objects when rule is enabled |
| `bidirectional` | Two-way sync with Last-Write-Wins conflict resolution |
### Architecture
- **Source Instance**: The MyFSIO instance where you upload files. It runs the replication worker.
- **Target Instance**: Another MyFSIO instance (or any S3-compatible service like AWS S3, MinIO) that receives the copies.
Replication is **asynchronous** (happens in the background) and **one-way** (Source -> Target).
For `new_only` and `all` modes, replication is **asynchronous** (happens in the background) and **one-way** (Source -> Target).
For `bidirectional` mode, replication is **two-way** with automatic conflict resolution.
### Setup Guide
@@ -1355,16 +1365,117 @@ When paused, new objects uploaded to the source will not replicate until replica
> **Note:** Only admins can create new replication rules, change the target connection/bucket, or delete rules entirely.
### Bidirectional Replication (Active-Active)
### Bidirectional Site Replication
To set up two-way replication (Server A ↔ Server B):
For true two-way synchronization with automatic conflict resolution, use the `bidirectional` replication mode. This enables a background sync worker that periodically pulls changes from the remote site.
> **Important:** Both sites must be configured to sync with each other. Each site pushes its changes and pulls from the other. You must set up connections and replication rules on both ends.
#### Step 1: Enable Site Sync on Both Sites
Set these environment variables on **both** Site A and Site B:
```bash
SITE_SYNC_ENABLED=true
SITE_SYNC_INTERVAL_SECONDS=60 # How often to pull changes (default: 60)
SITE_SYNC_BATCH_SIZE=100 # Max objects per sync cycle (default: 100)
```
#### Step 2: Create IAM Users for Cross-Site Access
On each site, create an IAM user that the other site will use to connect:
| Site | Create User For | Required Permissions |
|------|-----------------|---------------------|
| Site A | Site B to connect | `read`, `write`, `list`, `delete` on target bucket |
| Site B | Site A to connect | `read`, `write`, `list`, `delete` on target bucket |
Example policy for the replication user:
```json
[{"bucket": "my-bucket", "actions": ["read", "write", "list", "delete"]}]
```
#### Step 3: Create Connections
On each site, add a connection pointing to the other:
**On Site A:**
- Go to **Connections** and add a connection to Site B
- Endpoint: `https://site-b.example.com`
- Credentials: Site B's IAM user (created in Step 2)
**On Site B:**
- Go to **Connections** and add a connection to Site A
- Endpoint: `https://site-a.example.com`
- Credentials: Site A's IAM user (created in Step 2)
#### Step 4: Enable Bidirectional Replication
On each site, go to the bucket's **Replication** tab and enable with mode `bidirectional`:
**On Site A:**
- Source bucket: `my-bucket`
- Target connection: Site B connection
- Target bucket: `my-bucket`
- Mode: **Bidirectional sync**
**On Site B:**
- Source bucket: `my-bucket`
- Target connection: Site A connection
- Target bucket: `my-bucket`
- Mode: **Bidirectional sync**
#### How It Works
- **PUSH**: Local changes replicate to remote immediately on write/delete
- **PULL**: Background worker fetches remote changes every `SITE_SYNC_INTERVAL_SECONDS`
- **Loop Prevention**: `S3ReplicationAgent` and `SiteSyncAgent` User-Agents prevent infinite sync loops
#### Conflict Resolution (Last-Write-Wins)
When the same object exists on both sites, the system uses Last-Write-Wins (LWW) based on `last_modified` timestamps:
- **Remote newer**: Pull the remote version
- **Local newer**: Keep the local version
- **Same timestamp**: Use ETag as tiebreaker (higher ETag wins)
A 1-second clock skew tolerance prevents false conflicts from minor time differences.
#### Deletion Synchronization
When `sync_deletions=true` (default), remote deletions propagate locally only if:
1. The object was previously synced FROM remote (tracked in sync state)
2. The local version hasn't been modified since last sync
This prevents accidental deletion of local-only objects.
#### Sync State Storage
Sync state is stored at: `data/.myfsio.sys/buckets/<bucket>/site_sync_state.json`
```json
{
"synced_objects": {
"path/to/file.txt": {
"last_synced_at": 1706100000.0,
"remote_etag": "abc123",
"source": "remote"
}
},
"last_full_sync": 1706100000.0
}
```
### Legacy Bidirectional Setup (Manual)
For simpler use cases without the site sync worker, you can manually configure two one-way rules:
1. Follow the steps above to replicate **A → B**.
2. Repeat the process on Server B to replicate **B → A**:
- Create a connection on Server B pointing to Server A.
- Enable replication on the target bucket on Server B.
**Loop Prevention**: The system automatically detects replication traffic using a custom User-Agent (`S3ReplicationAgent`). This prevents infinite loops where an object replicated from A to B is immediately replicated back to A.
**Loop Prevention**: The system automatically detects replication traffic using custom User-Agents (`S3ReplicationAgent` and `SiteSyncAgent`). This prevents infinite loops where an object replicated from A to B is immediately replicated back to A.
**Deletes**: Deleting an object on one server will propagate the deletion to the other server.

25
templates/bucket_detail.html
View File

@@ -1065,8 +1065,10 @@
<path d="M16 8A8 8 0 1 1 0 8a8 8 0 0 1 16 0zm-3.97-3.03a.75.75 0 0 0-1.08.022L7.477 9.417 5.384 7.323a.75.75 0 0 0-1.06 1.06L6.97 11.03a.75.75 0 0 0 1.079-.02l3.992-4.99a.75.75 0 0 0-.01-1.05z"/>
</svg>
<div>
<strong>Replication Active</strong> —
{% if replication_rule.mode == 'all' %}
<strong>Replication Active</strong> —
{% if replication_rule.mode == 'bidirectional' %}
Bi-directional sync enabled with LWW conflict resolution.
{% elif replication_rule.mode == 'all' %}
All objects (existing + new) are being replicated.
{% else %}
New uploads to this bucket are automatically replicated.
@@ -1159,7 +1161,7 @@
</div>
<div class="text-muted small text-uppercase">Mode</div>
<div class="fw-semibold small">
{% if replication_rule.mode == 'all' %}All Objects{% else %}New Only{% endif %}
{% if replication_rule.mode == 'bidirectional' %}Bidirectional{% elif replication_rule.mode == 'all' %}All Objects{% else %}New Only{% endif %}
</div>
</div>
</div>
@@ -1310,7 +1312,9 @@
<div>
<strong>Replication Paused</strong>
<p class="mb-1">Replication is configured but currently paused. New uploads will not be replicated until resumed.</p>
{% if replication_rule.mode == 'all' %}
{% if replication_rule.mode == 'bidirectional' %}
<p class="mb-0 small text-dark"><strong>Tip:</strong> When you resume, bi-directional sync will continue and any missed changes will be reconciled using LWW conflict resolution.</p>
{% elif replication_rule.mode == 'all' %}
<p class="mb-0 small text-dark"><strong>Tip:</strong> When you resume, any objects uploaded while paused will be automatically synced to the target.</p>
{% else %}
<p class="mb-0 small text-dark"><strong>Note:</strong> Objects uploaded while paused will not be synced (mode: new_only). Consider switching to "All Objects" mode if you need to sync missed uploads.</p>
@@ -1435,17 +1439,26 @@
<div class="text-muted small">Only replicate objects uploaded after enabling replication. Existing objects will not be copied.</div>
</label>
</div>
<div class="form-check p-3 m-0">
<div class="form-check p-3 border-bottom m-0">
<input class="form-check-input" type="radio" name="replication_mode" id="mode_all" value="all">
<label class="form-check-label w-100" for="mode_all">
<span class="fw-medium">All objects (existing + new)</span>
<div class="text-muted small">Replicate all existing objects immediately, plus all future uploads. <span class="text-warning">This may take time for large buckets.</span></div>
</label>
</div>
{% if site_sync_enabled %}
<div class="form-check p-3 m-0">
<input class="form-check-input" type="radio" name="replication_mode" id="mode_bidirectional" value="bidirectional">
<label class="form-check-label w-100" for="mode_bidirectional">
<span class="fw-medium">Bidirectional sync</span>
<div class="text-muted small">Two-way sync with Last-Write-Wins conflict resolution.</div>
</label>
</div>
{% endif %}
</div>
</div>
</div>
<button type="submit" class="btn btn-primary">
<svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" fill="currentColor" class="me-1" viewBox="0 0 16 16">
<path fill-rule="evenodd" d="M8 3a5 5 0 1 0 4.546 2.914.5.5 0 0 1 .908-.417A6 6 0 1 1 8 2v1z"/>

178
templates/docs.html
View File

@@ -34,7 +34,7 @@
<li><a href="#automation">Automation / CLI</a></li>
<li><a href="#api">REST endpoints</a></li>
<li><a href="#examples">API Examples</a></li>
<li><a href="#replication">Site Replication</a></li>
<li><a href="#replication">Site Replication &amp; Sync</a></li>
<li><a href="#versioning">Object Versioning</a></li>
<li><a href="#quotas">Bucket Quotas</a></li>
<li><a href="#encryption">Encryption</a></li>
@@ -224,6 +224,24 @@ python run.py --mode ui
<td><code>5</code></td>
<td>Interval between history snapshots.</td>
</tr>
<tr class="table-secondary">
<td colspan="3" class="fw-semibold">Site Sync Settings (Bidirectional Replication)</td>
</tr>
<tr>
<td><code>SITE_SYNC_ENABLED</code></td>
<td><code>false</code></td>
<td>Enable bi-directional site sync background worker.</td>
</tr>
<tr>
<td><code>SITE_SYNC_INTERVAL_SECONDS</code></td>
<td><code>60</code></td>
<td>Interval between sync cycles (seconds).</td>
</tr>
<tr>
<td><code>SITE_SYNC_BATCH_SIZE</code></td>
<td><code>100</code></td>
<td>Max objects to pull per sync cycle.</td>
</tr>
</tbody>
</table>
</div>
@@ -574,9 +592,9 @@ s3.complete_multipart_upload(
<div class="card-body">
<div class="d-flex align-items-center gap-2 mb-3">
<span class="docs-section-kicker">08</span>
<h2 class="h4 mb-0">Site Replication</h2>
<h2 class="h4 mb-0">Site Replication &amp; Sync</h2>
</div>
<p class="text-muted">Automatically copy new objects to another MyFSIO instance or S3-compatible service for backup or disaster recovery.</p>
<p class="text-muted">Replicate objects to another MyFSIO instance or S3-compatible service. Supports one-way replication for backup and bi-directional sync for geo-distributed deployments.</p>
<h3 class="h6 text-uppercase text-muted mt-4">Setup Guide</h3>
<ol class="docs-steps mb-3">
@@ -635,17 +653,147 @@ except Exception as e:
</div>
</div>
<h3 class="h6 text-uppercase text-muted mt-4">Bidirectional Replication (Active-Active)</h3>
<p class="small text-muted">To set up two-way replication (Server A ↔ Server B):</p>
<ol class="docs-steps mb-3">
<li>Follow the steps above to replicate <strong>A → B</strong>.</li>
<li>Repeat the process on Server B to replicate <strong>B → A</strong> (create a connection to A, enable rule).</li>
</ol>
<p class="small text-muted mb-3">
<strong>Loop Prevention:</strong> The system automatically detects replication traffic using a custom User-Agent (<code>S3ReplicationAgent</code>). This prevents infinite loops where an object replicated from A to B is immediately replicated back to A.
<br>
<strong>Deletes:</strong> Deleting an object on one server will propagate the deletion to the other server.
</p>
<h3 class="h6 text-uppercase text-muted mt-4">Replication Modes</h3>
<div class="table-responsive mb-3">
<table class="table table-sm table-bordered small">
<thead class="table-light">
<tr>
<th>Mode</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>new_only</code></td>
<td>Only replicate new/modified objects (default, one-way)</td>
</tr>
<tr>
<td><code>all</code></td>
<td>Sync all existing objects when rule is enabled (one-way)</td>
</tr>
<tr>
<td><strong><code>bidirectional</code></strong></td>
<td>Two-way sync with Last-Write-Wins conflict resolution</td>
</tr>
</tbody>
</table>
</div>
<h3 class="h6 text-uppercase text-muted mt-4">Bidirectional Site Replication</h3>
<p class="small text-muted">For true two-way synchronization with automatic conflict resolution, use the <code>bidirectional</code> mode. Both sites must be configured to sync with each other.</p>
<div class="alert alert-info border small mb-3">
<div class="d-flex gap-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentColor" class="bi bi-info-circle text-info mt-1 flex-shrink-0" viewBox="0 0 16 16">
<path d="M8 15A7 7 0 1 1 8 1a7 7 0 0 1 0 14zm0 1A8 8 0 1 0 8 0a8 8 0 0 0 0 16z"/>
<path d="m8.93 6.588-2.29.287-.082.38.45.083c.294.07.352.176.288.469l-.738 3.468c-.194.897.105 1.319.808 1.319.545 0 1.178-.252 1.465-.598l.088-.416c-.2.176-.492.246-.686.246-.275 0-.375-.193-.304-.533L8.93 6.588zM9 4.5a1 1 0 1 1-2 0 1 1 0 0 1 2 0z"/>
</svg>
<div>
<strong>Both sites need configuration.</strong> Each site pushes its changes and pulls from the other. You must set up connections and replication rules on both ends.
</div>
</div>
</div>
<h4 class="h6 mt-4 mb-2">Step 1: Enable Site Sync on Both Sites</h4>
<p class="small text-muted">Set these environment variables on <strong>both</strong> Site A and Site B:</p>
<pre class="mb-3"><code class="language-bash">SITE_SYNC_ENABLED=true
SITE_SYNC_INTERVAL_SECONDS=60 # How often to pull changes
SITE_SYNC_BATCH_SIZE=100 # Max objects per sync cycle</code></pre>
<h4 class="h6 mt-4 mb-2">Step 2: Create IAM Users for Cross-Site Access</h4>
<p class="small text-muted">On each site, create an IAM user that the other site will use to connect:</p>
<div class="table-responsive mb-3">
<table class="table table-sm table-bordered small">
<thead class="table-light">
<tr>
<th>Site</th>
<th>Create User For</th>
<th>Required Permissions</th>
</tr>
</thead>
<tbody>
<tr>
<td>Site A</td>
<td>Site B to connect</td>
<td><code>read</code>, <code>write</code>, <code>list</code>, <code>delete</code> on target bucket</td>
</tr>
<tr>
<td>Site B</td>
<td>Site A to connect</td>
<td><code>read</code>, <code>write</code>, <code>list</code>, <code>delete</code> on target bucket</td>
</tr>
</tbody>
</table>
</div>
<h4 class="h6 mt-4 mb-2">Step 3: Create Connections</h4>
<p class="small text-muted">On each site, add a connection pointing to the other:</p>
<div class="row g-3 mb-3">
<div class="col-md-6">
<div class="card border h-100">
<div class="card-header bg-light py-2"><strong class="small">On Site A</strong></div>
<div class="card-body small">
<p class="mb-1">Go to <strong>Connections</strong> and add:</p>
<ul class="mb-0 ps-3">
<li>Endpoint: <code>https://site-b.example.com</code></li>
<li>Credentials: Site B's IAM user</li>
</ul>
</div>
</div>
</div>
<div class="col-md-6">
<div class="card border h-100">
<div class="card-header bg-light py-2"><strong class="small">On Site B</strong></div>
<div class="card-body small">
<p class="mb-1">Go to <strong>Connections</strong> and add:</p>
<ul class="mb-0 ps-3">
<li>Endpoint: <code>https://site-a.example.com</code></li>
<li>Credentials: Site A's IAM user</li>
</ul>
</div>
</div>
</div>
</div>
<h4 class="h6 mt-4 mb-2">Step 4: Enable Bidirectional Replication</h4>
<p class="small text-muted">On each site, go to the bucket's <strong>Replication</strong> tab and enable with mode <code>bidirectional</code>:</p>
<div class="row g-3 mb-3">
<div class="col-md-6">
<div class="card border h-100">
<div class="card-header bg-light py-2"><strong class="small">On Site A</strong></div>
<div class="card-body small">
<ul class="mb-0 ps-3">
<li>Source bucket: <code>my-bucket</code></li>
<li>Target: Site B connection</li>
<li>Target bucket: <code>my-bucket</code></li>
<li>Mode: <strong>Bidirectional sync</strong></li>
</ul>
</div>
</div>
</div>
<div class="col-md-6">
<div class="card border h-100">
<div class="card-header bg-light py-2"><strong class="small">On Site B</strong></div>
<div class="card-body small">
<ul class="mb-0 ps-3">
<li>Source bucket: <code>my-bucket</code></li>
<li>Target: Site A connection</li>
<li>Target bucket: <code>my-bucket</code></li>
<li>Mode: <strong>Bidirectional sync</strong></li>
</ul>
</div>
</div>
</div>
</div>
<h4 class="h6 mt-4 mb-2">How It Works</h4>
<ul class="small text-muted mb-3">
<li><strong>PUSH:</strong> Local changes replicate to remote immediately on write/delete</li>
<li><strong>PULL:</strong> Background worker fetches remote changes every <code>SITE_SYNC_INTERVAL_SECONDS</code></li>
<li><strong>Conflict Resolution:</strong> Last-Write-Wins based on <code>last_modified</code> timestamps (1-second clock skew tolerance)</li>
<li><strong>Deletion Sync:</strong> Remote deletions propagate locally only for objects originally synced from remote</li>
<li><strong>Loop Prevention:</strong> <code>S3ReplicationAgent</code> and <code>SiteSyncAgent</code> User-Agents prevent infinite sync loops</li>
</ul>
<h3 class="h6 text-uppercase text-muted mt-4">Error Handling &amp; Rate Limits</h3>
<p class="small text-muted mb-3">The replication system handles transient failures automatically:</p>
@@ -1265,7 +1413,7 @@ curl "{{ api_base | replace('/api', '/ui') }}/metrics/operations/history?hours=6
<li><a href="#automation">Automation / CLI</a></li>
<li><a href="#api">REST endpoints</a></li>
<li><a href="#examples">API Examples</a></li>
<li><a href="#replication">Site Replication</a></li>
<li><a href="#replication">Site Replication &amp; Sync</a></li>
<li><a href="#versioning">Object Versioning</a></li>
<li><a href="#quotas">Bucket Quotas</a></li>
<li><a href="#encryption">Encryption</a></li>