Improve and add two-way replication functionality; Update docs
This commit is contained in:
@@ -1043,6 +1043,7 @@ def bucket_handler(bucket_name: str) -> Response:
|
|||||||
try:
|
try:
|
||||||
storage.delete_bucket(bucket_name)
|
storage.delete_bucket(bucket_name)
|
||||||
_bucket_policies().delete_policy(bucket_name)
|
_bucket_policies().delete_policy(bucket_name)
|
||||||
|
_replication_manager().delete_rule(bucket_name)
|
||||||
except StorageError as exc:
|
except StorageError as exc:
|
||||||
code = "BucketNotEmpty" if "not empty" in str(exc) else "NoSuchBucket"
|
code = "BucketNotEmpty" if "not empty" in str(exc) else "NoSuchBucket"
|
||||||
status = 409 if code == "BucketNotEmpty" else 404
|
status = 409 if code == "BucketNotEmpty" else 404
|
||||||
|
|||||||
@@ -500,6 +500,7 @@ def delete_bucket(bucket_name: str):
|
|||||||
_authorize_ui(principal, bucket_name, "delete")
|
_authorize_ui(principal, bucket_name, "delete")
|
||||||
_storage().delete_bucket(bucket_name)
|
_storage().delete_bucket(bucket_name)
|
||||||
_bucket_policies().delete_policy(bucket_name)
|
_bucket_policies().delete_policy(bucket_name)
|
||||||
|
_replication_manager().delete_rule(bucket_name)
|
||||||
flash(f"Bucket '{bucket_name}' removed", "success")
|
flash(f"Bucket '{bucket_name}' removed", "success")
|
||||||
except (StorageError, IamError) as exc:
|
except (StorageError, IamError) as exc:
|
||||||
flash(_friendly_error_message(exc), "danger")
|
flash(_friendly_error_message(exc), "danger")
|
||||||
|
|||||||
25
docs.md
25
docs.md
@@ -77,12 +77,20 @@ The repo now tracks a human-friendly release string inside `app/version.py` (see
|
|||||||
| `SECRET_KEY` | `dev-secret-key` | Flask session key for UI auth. |
|
| `SECRET_KEY` | `dev-secret-key` | Flask session key for UI auth. |
|
||||||
| `IAM_CONFIG` | `<repo>/data/.myfsio.sys/config/iam.json` | Stores users, secrets, and inline policies. |
|
| `IAM_CONFIG` | `<repo>/data/.myfsio.sys/config/iam.json` | Stores users, secrets, and inline policies. |
|
||||||
| `BUCKET_POLICY_PATH` | `<repo>/data/.myfsio.sys/config/bucket_policies.json` | Bucket policy store (auto hot-reload). |
|
| `BUCKET_POLICY_PATH` | `<repo>/data/.myfsio.sys/config/bucket_policies.json` | Bucket policy store (auto hot-reload). |
|
||||||
| `API_BASE_URL` | `http://127.0.0.1:5000` | Used by the UI to hit API endpoints (presign/policy). |
|
| `API_BASE_URL` | `None` | Used by the UI to hit API endpoints (presign/policy). If unset, the UI will auto-detect the host or use `X-Forwarded-*` headers. |
|
||||||
| `AWS_REGION` | `us-east-1` | Region embedded in SigV4 credential scope. |
|
| `AWS_REGION` | `us-east-1` | Region embedded in SigV4 credential scope. |
|
||||||
| `AWS_SERVICE` | `s3` | Service string for SigV4. |
|
| `AWS_SERVICE` | `s3` | Service string for SigV4. |
|
||||||
|
|
||||||
Set env vars (or pass overrides to `create_app`) to point the servers at custom paths.
|
Set env vars (or pass overrides to `create_app`) to point the servers at custom paths.
|
||||||
|
|
||||||
|
### Proxy Configuration
|
||||||
|
|
||||||
|
If running behind a reverse proxy (e.g., Nginx, Cloudflare, or a tunnel), ensure the proxy sets the standard forwarding headers:
|
||||||
|
- `X-Forwarded-Host`
|
||||||
|
- `X-Forwarded-Proto`
|
||||||
|
|
||||||
|
The application automatically trusts these headers to generate correct presigned URLs (e.g., `https://s3.example.com/...` instead of `http://127.0.0.1:5000/...`). Alternatively, you can explicitly set `API_BASE_URL` to your public endpoint.
|
||||||
|
|
||||||
## 4. Authentication & IAM
|
## 4. Authentication & IAM
|
||||||
|
|
||||||
1. On first boot, `data/.myfsio.sys/config/iam.json` is seeded with `localadmin / localadmin` that has wildcard access.
|
1. On first boot, `data/.myfsio.sys/config/iam.json` is seeded with `localadmin / localadmin` that has wildcard access.
|
||||||
@@ -262,6 +270,21 @@ Now, configure the primary instance to replicate to the target.
|
|||||||
aws --endpoint-url http://target-server:5002 s3 ls s3://backup-bucket
|
aws --endpoint-url http://target-server:5002 s3 ls s3://backup-bucket
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Bidirectional Replication (Active-Active)
|
||||||
|
|
||||||
|
To set up two-way replication (Server A ↔ Server B):
|
||||||
|
|
||||||
|
1. Follow the steps above to replicate **A → B**.
|
||||||
|
2. Repeat the process on Server B to replicate **B → A**:
|
||||||
|
- Create a connection on Server B pointing to Server A.
|
||||||
|
- Enable replication on the target bucket on Server B.
|
||||||
|
|
||||||
|
**Loop Prevention**: The system automatically detects replication traffic using a custom User-Agent (`S3ReplicationAgent`). This prevents infinite loops where an object replicated from A to B is immediately replicated back to A.
|
||||||
|
|
||||||
|
**Deletes**: Deleting an object on one server will propagate the deletion to the other server.
|
||||||
|
|
||||||
|
**Note**: Deleting a bucket will automatically remove its associated replication configuration.
|
||||||
|
|
||||||
## 7. Running Tests
|
## 7. Running Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -290,6 +290,18 @@ s3.complete_multipart_upload(
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<h3 class="h6 text-uppercase text-muted mt-4">Bidirectional Replication (Active-Active)</h3>
|
||||||
|
<p class="small text-muted">To set up two-way replication (Server A ↔ Server B):</p>
|
||||||
|
<ol class="docs-steps mb-3">
|
||||||
|
<li>Follow the steps above to replicate <strong>A → B</strong>.</li>
|
||||||
|
<li>Repeat the process on Server B to replicate <strong>B → A</strong> (create a connection to A, enable rule).</li>
|
||||||
|
</ol>
|
||||||
|
<p class="small text-muted mb-0">
|
||||||
|
<strong>Loop Prevention:</strong> The system automatically detects replication traffic using a custom User-Agent (<code>S3ReplicationAgent</code>). This prevents infinite loops where an object replicated from A to B is immediately replicated back to A.
|
||||||
|
<br>
|
||||||
|
<strong>Deletes:</strong> Deleting an object on one server will propagate the deletion to the other server.
|
||||||
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</article>
|
</article>
|
||||||
<article id="troubleshooting" class="card shadow-sm docs-section">
|
<article id="troubleshooting" class="card shadow-sm docs-section">
|
||||||
@@ -330,8 +342,8 @@ s3.complete_multipart_upload(
|
|||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>Requests hit the wrong host</td>
|
<td>Requests hit the wrong host</td>
|
||||||
<td><code>API_BASE_URL</code> not updated after tunneling/forwarding</td>
|
<td>Proxy headers missing or <code>API_BASE_URL</code> incorrect</td>
|
||||||
<td>Set <code>API_BASE_URL</code> in your shell or <code>.env</code> to match the published host.</td>
|
<td>Ensure your proxy sends <code>X-Forwarded-Host</code>/<code>Proto</code> headers, or explicitly set <code>API_BASE_URL</code> to your public domain.</td>
|
||||||
</tr>
|
</tr>
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
|
|||||||
Reference in New Issue
Block a user