| author | |
| committer | |
| log | 796fc67fb8d5f978f496b727f34dc28347747940 |
| tree | 60f7beb19fa7dd72e9c06a7d9c29688c21775c8f |
| parent | 8bfd74a99a223fc46af93692a4a5827d1fa98930 |
| signature | Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU |
Add admin-managed folder grants for Snow accounts and standalone API accounts, with scoped permissions, one-time credentials, and session revocation. Preserve native public links and selected-file shares through explicit retarget and offline move actions.
Persist management state through deployment and staging, and add a bounded, reversible cutover for the six requested shared folders. Prefer Zenith staging previews for service verification.
Assisted-by: gpt-6.1-sol26 files changed, 2321 insertions(+), 50 deletions(-)
AGENTS.md+13-8| ... | ... | @@ -7,14 +7,15 @@ availability and existing data as part of correctness. |
| 7 | 7 | |
| 8 | 8 | ## Access and authorization |
| 9 | 9 | |
| 10 | - Local edits and rehearsal VM changes are always implied by a task. Carry | |
| 11 | them through staging and verification without asking for permission again. | |
| 10 | - Local edits and staging previews on Zenith are always implied by a task. | |
| 11 | Carry them through verification without asking for permission again. Use | |
| 12 | disposable preview data for writes and keep active production services intact. | |
| 12 | 13 | - Production changes require a request to take the work to prod. A request to |
| 13 | 14 | fix, stage, test, or commit something does not authorize production deployment. |
| 14 | 15 | Once prod is requested, complete the normal deployment workflow. |
| 15 | 16 | - Read-only infrastructure inspection over SSH is allowed for the task. Confirm |
| 16 | the target host before a remote mutation; deployment defaults point at the | |
| 17 | rehearsal VM, but inherited environment variables can select production. | |
| 17 | the target host before a remote mutation. Set both deployment host and port | |
| 18 | explicitly; the CLI's legacy defaults still point at the rehearsal VM. | |
| 18 | 19 | - Private file contents require authorization per source. Infrastructure access, |
| 19 | 20 | a mounted share, or permission to list a tree does not authorize opening its |
| 20 | 21 | contents. Keep private material and derived model output inside its store. |
| ... | ... | @@ -35,11 +36,13 @@ availability and existing data as part of correctness. |
| 35 | 36 | | Surface | Connection / role | Configuration | |
| 36 | 37 | | --- | --- | --- | |
| 37 | 38 | | Mac checkout | `/Users/clo/dev/infra-2`; edit and upload here | This working copy | |
| 38 | | Rehearsal NixOS VM | `root@127.0.0.1`, SSH port `2222` | `nixos/vm.nix`, `studio.test`, demo pool | | |
| 39 | | Optional rehearsal NixOS VM | `root@127.0.0.1`, SSH port `2222` | `nixos/vm.nix`, `studio.test`, demo pool | | |
| 39 | 40 | | Zenith production | `root@10.0.0.1`, SSH port `22` | `nixos/zenith.nix`, `paperclover.net`, physical pool | |
| 40 | 41 | |
| 41 | 42 | `tools/deploy.py` reads `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT`; defaults |
| 42 | are the rehearsal connection above. Set both explicitly for production. | |
| 43 | are the rehearsal connection above. Set both explicitly for Zenith staging | |
| 44 | and production. Prefer staging previews on Zenith for service and dashboard | |
| 45 | verification; use the VM when host or guest isolation is required. | |
| 43 | 46 | Host settings and dataset names have their canonical home in `nixos/zenith.nix` |
| 44 | 47 | and `config/site.pkl`; inspect them instead of copying values into new tooling. |
| 45 | 48 | |
| ... | ... | @@ -99,7 +102,7 @@ Mac rehearsal routing. Do not hand-edit DNS, trust, or host files to bypass them |
| 99 | 102 | 3. Run the relevant local checks, then stage the working files: |
| 100 | 103 | |
| 101 | 104 | ```sh |
| 102 | STUDIO_DEPLOY_HOST=root@127.0.0.1 STUDIO_DEPLOY_PORT=2222 python3 tools/deploy.py stage <service> | |
| 105 | STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py stage <service> | |
| 103 | 106 | ``` |
| 104 | 107 | |
| 105 | 108 | The command prints `stage`, `release`, `host`, and `dataset`. Use that host; |
| ... | ... | @@ -135,7 +138,8 @@ read `tools/data-restore.md` before proposing it. |
| 135 | 138 | |
| 136 | 139 | - Run checks that cover the change. Inspect a test's entry point first: some |
| 137 | 140 | `tools/*-test.py` scripts mutate host services or require Linux/real guests. |
| 138 | Run those in rehearsal, not on production by default. | |
| 141 | Run those against isolated staging fixtures. Tests that alter host services, | |
| 142 | storage, or guests require a disposable VM or explicit production scope. | |
| 139 | 143 | - Dashboard: `cd dashboard && npm run check && npm run build`; run Rust tests |
| 140 | 144 | with `cargo test --manifest-path dashboard/Cargo.toml`. Host tests include |
| 141 | 145 | `python3 tools/dashboard-host-test.py` and `python3 tools/dashboard-vms-test.py`. |
| ... | ... | @@ -169,6 +173,7 @@ read `tools/data-restore.md` before proposing it. |
| 169 | 173 | [tools/personal-cutover.md](tools/personal-cutover.md), and the service's |
| 170 | 174 | `tools/*-migration.md` runbook. |
| 171 | 175 | - Provider behavior: [tools/service-dependencies.md](tools/service-dependencies.md). |
| 176 | - Copyparty administration and folder moves: [tools/copyparty.md](tools/copyparty.md). | |
| 172 | 177 | - Local inference: [tools/local-ai.md](tools/local-ai.md). |
| 173 | 178 | |
| 174 | 179 | Runbooks contain dated rehearsal evidence and migration history. Check current |
dashboard/src/auth.rs+39-7| ... | ... | @@ -571,6 +571,23 @@ impl Store { |
| 571 | 571 | } |
| 572 | 572 | } |
| 573 | 573 | |
| 574 | fn file_groups(user: &Value) -> Option<String> { | |
| 575 | if user["enabled"] != true | |
| 576 | || !array(&user["requiredActions"]).is_empty() | |
| 577 | || guest::is_guest(user) | |
| 578 | || user["kind"] == "ai" | |
| 579 | { | |
| 580 | return None; | |
| 581 | } | |
| 582 | let id = uuid::Uuid::parse_str(string(&user["id"])).ok()?; | |
| 583 | let mut groups = array(&user["groups"]) | |
| 584 | .iter() | |
| 585 | .map(|g| string(&g["name"]).to_owned()) | |
| 586 | .collect::<Vec<_>>(); | |
| 587 | groups.push(format!("snow-user-{id}")); | |
| 588 | Some(groups.join(",")) | |
| 589 | } | |
| 590 | ||
| 574 | 591 | pub async fn route(State(app): State<Arc<App>>, request: Request) -> Result<Response> { |
| 575 | 592 | if request.uri().path() == "/auth/shale/page" { |
| 576 | 593 | return shale_page::proxy(app, request).await; |
| ... | ... | @@ -591,14 +608,9 @@ pub async fn route(State(app): State<Arc<App>>, request: Request) -> Result<Resp |
| 591 | 608 | let headers = request.headers().clone(); |
| 592 | 609 | if path == "/auth/file/check" && method == Method::GET { |
| 593 | 610 | let user = auth.session(&headers, "file")?; |
| 594 | if user.is_null() || !array(&user["requiredActions"]).is_empty() { | |
| 611 | let Some(groups) = file_groups(&user) else { | |
| 595 | 612 | return Ok(StatusCode::UNAUTHORIZED.into_response()); |
| 596 | } | |
| 597 | let groups = array(&user["groups"]) | |
| 598 | .iter() | |
| 599 | .map(|g| string(&g["name"])) | |
| 600 | .collect::<Vec<_>>() | |
| 601 | .join(","); | |
| 613 | }; | |
| 602 | 614 | return Ok(( |
| 603 | 615 | StatusCode::NO_CONTENT, |
| 604 | 616 | [ |
| ... | ... | @@ -1284,3 +1296,23 @@ fn passkeys(db: &Connection, id: &str) -> Result<Vec<Passkey>> { |
| 1284 | 1296 | }) |
| 1285 | 1297 | .collect() |
| 1286 | 1298 | } |
| 1299 | ||
| 1300 | #[cfg(test)] | |
| 1301 | mod tests { | |
| 1302 | use super::*; | |
| 1303 | ||
| 1304 | #[test] | |
| 1305 | fn folder_identity_tracks_the_account_id_and_requires_an_enabled_regular_account() { | |
| 1306 | let mut account = json!({"id":"12345678-1234-1234-1234-123456789abc","username":"snow","enabled":true,"requiredActions":[],"groups":[{"name":"media"}]}); | |
| 1307 | assert_eq!(file_groups(&account).unwrap(), "media,snow-user-12345678-1234-1234-1234-123456789abc"); | |
| 1308 | account["username"] = json!("different"); | |
| 1309 | assert_eq!(file_groups(&account).unwrap(), "media,snow-user-12345678-1234-1234-1234-123456789abc"); | |
| 1310 | account["id"] = json!("12345678-1234-1234-1234-123456789abd"); | |
| 1311 | assert!(file_groups(&account).unwrap().ends_with("snow-user-12345678-1234-1234-1234-123456789abd")); | |
| 1312 | for (field, value) in [("enabled", json!(false)), ("kind", json!("guest")), ("kind", json!("ai")), ("requiredActions", json!(["UPDATE_PASSWORD"])), ("id", json!("x,infra-admin"))] { | |
| 1313 | let mut blocked = account.clone(); | |
| 1314 | blocked[field] = value; | |
| 1315 | assert!(file_groups(&blocked).is_none()); | |
| 1316 | } | |
| 1317 | } | |
| 1318 | } |
dashboard/src/host.rs+2| ... | ... | @@ -34,6 +34,8 @@ pub async fn open(request: Value) -> Result<(Value, tokio::net::UnixStream)> { |
| 34 | 34 | Some("vm.create" | "vm.media" | "vm.preset") |
| 35 | 35 | ) { |
| 36 | 36 | 910 |
| 37 | } else if request["operation"] == "copyparty.manage" { | |
| 38 | 310 | |
| 37 | 39 | } else { |
| 38 | 40 | 70 |
| 39 | 41 | }; |
dashboard/src/main.rs+3-1| ... | ... | @@ -15,6 +15,7 @@ mod oidc; |
| 15 | 15 | mod relay; |
| 16 | 16 | mod shale; |
| 17 | 17 | mod shale_page; |
| 18 | mod shares; | |
| 18 | 19 | mod storage; |
| 19 | 20 | mod telemetry; |
| 20 | 21 | mod users; |
| ... | ... | @@ -269,7 +270,7 @@ async fn api(State(app): State<Arc<App>>, request: Request) -> Result<Response> |
| 269 | 270 | "services" if parts.len() == 1 => Some("metrics"), |
| 270 | 271 | "storage" if parts.len() == 1 => Some("metrics"), |
| 271 | 272 | "ai" | "mcp" => Some("ai"), |
| 272 | "services" | "traces" | "storage" | "users" | "deploys" | "paper-clover" => { | |
| 273 | "services" | "traces" | "storage" | "users" | "shares" | "deploys" | "paper-clover" => { | |
| 273 | 274 | Some("admin") |
| 274 | 275 | } |
| 275 | 276 | "media" | "seedbox" | "youtube" => Some("media"), |
| ... | ... | @@ -322,6 +323,7 @@ async fn api(State(app): State<Arc<App>>, request: Request) -> Result<Response> |
| 322 | 323 | "ai" => ai::route(app, &method, &parts[1..], &me, &headers).await, |
| 323 | 324 | "mcp" => mcp::manage(app, &method, &parts[1..], &me, value, &headers).await, |
| 324 | 325 | "users" => users::route(app, &method, &parts[1..], &me, value).await, |
| 326 | "shares" => shares::route(app, &method, &parts[1..], &query, &me, value).await, | |
| 325 | 327 | "vms" | "seedbox" | "paper-clover" => { |
| 326 | 328 | apps::route(app, &method, &parts, &query, &me, value).await |
| 327 | 329 | } |
dashboard/src/shares.rs created+92| ... | ... | @@ -0,0 +1,92 @@ |
| 1 | use crate::*; | |
| 2 | ||
| 3 | fn operation(method: &Method, parts: &[&str], mut body: Value) -> Result<(&'static str, Value)> { | |
| 4 | let action = match parts { | |
| 5 | [] if method == Method::GET => return Ok(("state", json!({}))), | |
| 6 | ["folders"] if method == Method::GET => return Ok(("folders.list", body)), | |
| 7 | ["accounts"] if method == Method::POST => "account.create", | |
| 8 | ["accounts", _] if method == Method::PATCH => "account.update", | |
| 9 | ["accounts", _] if method == Method::DELETE => "account.delete", | |
| 10 | ["accounts", _, "rotate"] if method == Method::POST => "account.rotate", | |
| 11 | ["grants"] if method == Method::POST => "grant.save", | |
| 12 | ["grants", _] if method == Method::PATCH => "grant.save", | |
| 13 | ["grants", _] if method == Method::DELETE => "grant.delete", | |
| 14 | ["links"] if method == Method::POST => "link.create", | |
| 15 | ["links", _] if method == Method::PATCH => "link.update", | |
| 16 | ["links", _] if method == Method::DELETE => "link.delete", | |
| 17 | _ => return Err(Error::new(404, "No share action here.")), | |
| 18 | }; | |
| 19 | if body.is_null() { | |
| 20 | body = json!({}); | |
| 21 | } | |
| 22 | let fields = body | |
| 23 | .as_object_mut() | |
| 24 | .ok_or_else(|| Error::new(400, "Reload shares and try again."))?; | |
| 25 | if let Some(id) = parts.get(1) { | |
| 26 | let key = if parts[0] == "links" { "key" } else { "id" }; | |
| 27 | if fields.contains_key(key) { | |
| 28 | return Err(Error::new(400, "Reload shares before saving this change.")); | |
| 29 | } | |
| 30 | fields.insert(key.into(), json!(id)); | |
| 31 | } else if parts == ["grants"] { | |
| 32 | fields.insert("id".into(), Value::Null); | |
| 33 | } | |
| 34 | Ok((action, body)) | |
| 35 | } | |
| 36 | ||
| 37 | pub async fn route( | |
| 38 | app: Arc<App>, | |
| 39 | method: &Method, | |
| 40 | parts: &[&str], | |
| 41 | query: &HashMap<String, String>, | |
| 42 | me: &Value, | |
| 43 | body: Value, | |
| 44 | ) -> Result<Response> { | |
| 45 | need(me, "admin")?; | |
| 46 | let body = if parts == ["folders"] && method == Method::GET { | |
| 47 | json!({"path":query.get("path").map(String::as_str).unwrap_or_default()}) | |
| 48 | } else { | |
| 49 | body | |
| 50 | }; | |
| 51 | let (action, payload) = operation(method, parts, body)?; | |
| 52 | let users = { | |
| 53 | let db = app.auth.db.lock().unwrap(); | |
| 54 | let mut statement = db.prepare("SELECT id FROM users ORDER BY username")?; | |
| 55 | statement | |
| 56 | .query_map([], |row| row.get::<_, String>(0))? | |
| 57 | .map(|id| { | |
| 58 | let id = id?; | |
| 59 | let profile = auth::user(&db, &id)?; | |
| 60 | let groups = array(&profile["groups"]).iter().map(|group| group["name"].clone()).collect::<Vec<_>>(); | |
| 61 | Ok(json!({"id":id,"username":profile["username"],"enabled":profile["enabled"],"kind":profile["kind"],"requiredActions":profile["requiredActions"],"groups":groups})) | |
| 62 | }) | |
| 63 | .collect::<Result<Vec<_>>>()? | |
| 64 | }; | |
| 65 | let mut value = host::call(json!({ | |
| 66 | "operation":"copyparty.manage", "action":action, "payload":payload, "users":users, | |
| 67 | })) | |
| 68 | .await?; | |
| 69 | if value.is_object() { | |
| 70 | value["filesUrl"] = json!(env( | |
| 71 | "STUDIO_FILES_URL", | |
| 72 | &format!("https://file.{}", env("STUDIO_DOMAIN", "studio.test")), | |
| 73 | )); | |
| 74 | } | |
| 75 | Ok(Document::new(value).response()) | |
| 76 | } | |
| 77 | ||
| 78 | #[cfg(test)] | |
| 79 | mod tests { | |
| 80 | use super::*; | |
| 81 | ||
| 82 | #[test] | |
| 83 | fn route_identity_cannot_be_overridden_by_the_body() { | |
| 84 | assert!(operation(&Method::PATCH, &["links", "kept"], json!({"key":"other"})).is_err()); | |
| 85 | assert!(operation(&Method::PATCH, &["accounts", "kept"], json!({"id":"other"})).is_err()); | |
| 86 | let (action, payload) = | |
| 87 | operation(&Method::DELETE, &["links", "kept"], Value::Null).unwrap(); | |
| 88 | assert_eq!(action, "link.delete"); | |
| 89 | assert_eq!(payload, json!({"key":"kept"})); | |
| 90 | assert!(operation(&Method::GET, &["accounts", "kept", "rotate"], Value::Null).is_err()); | |
| 91 | } | |
| 92 | } |
dashboard/src/users.rs+15-2| ... | ... | @@ -35,8 +35,8 @@ fn profile(body: &Value, full: bool) -> Result<Value> { |
| 35 | 35 | let value = match key { |
| 36 | 36 | "username" => { |
| 37 | 37 | let v = string(value).trim().to_lowercase(); |
| 38 | if v.starts_with("ai-") { | |
| 39 | return Err(Error::new(400, "Usernames beginning with ai- are reserved for model accounts.")); | |
| 38 | if v.starts_with("ai-") || v.starts_with("api-") { | |
| 39 | return Err(Error::new(400, "Choose a username without the ai- or api- prefix. Those are reserved for service accounts.")); | |
| 40 | 40 | } |
| 41 | 41 | if v.len() > 254 || !username_pattern.is_match(&v) { |
| 42 | 42 | return Err(Error::new( |
| ... | ... | @@ -635,3 +635,16 @@ pub async fn account( |
| 635 | 635 | Document::new(value).response() |
| 636 | 636 | }) |
| 637 | 637 | } |
| 638 | ||
| 639 | #[cfg(test)] | |
| 640 | mod tests { | |
| 641 | use super::*; | |
| 642 | ||
| 643 | #[test] | |
| 644 | fn people_cannot_claim_service_account_usernames() { | |
| 645 | for username in ["api-account", "API-account", "ai-account"] { | |
| 646 | assert!(profile(&json!({"username":username}), false).is_err()); | |
| 647 | } | |
| 648 | assert!(profile(&json!({"username":"person"}), false).is_ok()); | |
| 649 | } | |
| 650 | } |
dashboard/web/api.contract.ts+19| ... | ... | @@ -9,6 +9,7 @@ import type { Domain, History, NewDomain, VMLibrary } from "./types/vms.ts"; |
| 9 | 9 | import type { Me, ServiceSummary, ServiceDetail, ServiceDefinition, HostInfo, Issue, Series, LogLine, TraceSummary, Trace } from "./types/model.ts"; |
| 10 | 10 | import type { MapNode } from "./types/storage.index.ts"; |
| 11 | 11 | import type { ProgressNode, PaperCloverStats } from "./types/paperClover.ts"; |
| 12 | import type { AccountSecret, FolderGrant, LinkInput, ShareState } from "./types/shares.ts"; | |
| 12 | 13 | |
| 13 | 14 | type Endpoint<Output, Input = {}, Status extends number = 200, Format extends string = "json"> = { |
| 14 | 15 | input: Input; output: Output; status: Status; outputFormat: Format; |
| ... | ... | @@ -54,6 +55,24 @@ type Explorer = { |
| 54 | 55 | type ExplorerRoutes<Prefix extends string> = { [P in keyof Explorer as `${Prefix}${P}`]: Explorer[P] }; |
| 55 | 56 | |
| 56 | 57 | export type Api = Hono<{}, { |
| 58 | "/shares": { $get: Endpoint<ShareState>; }; | |
| 59 | "/shares/folders": { $get: Endpoint<{ path: string; directories: { name: string; path: string }[] }, { query: { path: string } }>; }; | |
| 60 | "/shares/accounts": { $post: Endpoint<AccountSecret, { json: { name: string } }>; }; | |
| 61 | "/shares/accounts/:id": { | |
| 62 | $patch: Endpoint<ShareState, { param: { id: string }; json: { name: string } }>; | |
| 63 | $delete: Endpoint<ShareState, { param: { id: string } }>; | |
| 64 | }; | |
| 65 | "/shares/accounts/:id/rotate": { $post: Endpoint<AccountSecret, { param: { id: string } }>; }; | |
| 66 | "/shares/grants": { $post: Endpoint<ShareState, { json: Omit<FolderGrant, "id"> }>; }; | |
| 67 | "/shares/grants/:id": { | |
| 68 | $patch: Endpoint<ShareState, { param: { id: string }; json: Omit<FolderGrant, "id"> }>; | |
| 69 | $delete: Endpoint<ShareState, { param: { id: string } }>; | |
| 70 | }; | |
| 71 | "/shares/links": { $post: Endpoint<ShareState, { json: LinkInput & { createFolder: boolean } }>; }; | |
| 72 | "/shares/links/:key": { | |
| 73 | $patch: Endpoint<ShareState, { param: { key: string }; json: LinkInput & { mode: "retarget" | "move" } }>; | |
| 74 | $delete: Endpoint<ShareState, { param: { key: string } }>; | |
| 75 | }; | |
| 57 | 76 | "/ai": { $get: Endpoint<{ endpoint: string; model: string; context: number }>; }; |
| 58 | 77 | "/ai/key": { $post: Endpoint<{ key: string }>; }; |
| 59 | 78 | "/mcp": { $get: Endpoint<Connections>; }; |
dashboard/web/components/Dialog.tsx+2| ... | ... | @@ -20,6 +20,7 @@ interface DialogOptions { |
| 20 | 20 | canConfirm?: () => boolean; |
| 21 | 21 | /** Takes focus once the dialog closes, instead of the element that opened it. */ |
| 22 | 22 | returnFocus?: HTMLElement; |
| 23 | onClose?: () => void; | |
| 23 | 24 | /** |
| 24 | 25 | * Gets the body's named fields. The dialog stays open with a spinner until this settles; a rejection shakes it |
| 25 | 26 | * and shows the reason. |
| ... | ... | @@ -96,6 +97,7 @@ function open(options: DialogOptions, field?: TextDialogOptions) { |
| 96 | 97 | dispose(); |
| 97 | 98 | host.remove(); |
| 98 | 99 | options.returnFocus?.focus(); |
| 100 | options.onClose?.(); | |
| 99 | 101 | }} |
| 100 | 102 | onPointerDown={(event) => (pressedOutside = event.target === dialog)} |
| 101 | 103 | onClick={(event) => pressedOutside && event.target === dialog && close()}> |
dashboard/web/components/Sidebar.tsx+2| ... | ... | @@ -14,6 +14,7 @@ import SquarePlay from "lucide-solid/icons/square-play"; |
| 14 | 14 | import UserRound from "lucide-solid/icons/user-round"; |
| 15 | 15 | import Plug from "lucide-solid/icons/plug"; |
| 16 | 16 | import Users from "lucide-solid/icons/users"; |
| 17 | import FolderHeart from "lucide-solid/icons/folder-heart"; | |
| 17 | 18 | import { createSignal, For, type JSX, onCleanup, Show } from "solid-js"; |
| 18 | 19 | import { type Health, type Me, type Section, VIEW_AS } from "../types/model.ts"; |
| 19 | 20 | import { queries } from "../api.ts"; |
| ... | ... | @@ -59,6 +60,7 @@ const AFTER: Page[] = [ |
| 59 | 60 | }, |
| 60 | 61 | { href: "/paper-clover", label: "paper clover", icon: PaperCloverMark, section: "admin" }, |
| 61 | 62 | { href: "/users", label: "users", icon: Users, section: "admin" }, |
| 63 | { href: "/shares", label: "shares", icon: FolderHeart, section: "admin" }, | |
| 62 | 64 | { href: "/deploys", label: "deploys", icon: Rocket, section: "admin" }, |
| 63 | 65 | { href: "/vms", label: "vms", icon: Monitor, section: "vms" }, |
| 64 | 66 | ]; |
dashboard/web/main.tsx+3| ... | ... | @@ -16,6 +16,7 @@ import { Seedbox } from "./pages/Seedbox.tsx"; |
| 16 | 16 | import { YouTube } from "./pages/YouTube.tsx"; |
| 17 | 17 | import { PaperClover } from "./pages/PaperClover.tsx"; |
| 18 | 18 | import { directory, Users } from "./pages/Users.tsx"; |
| 19 | import { shares, Shares } from "./pages/Shares.tsx"; | |
| 19 | 20 | import { Deploys } from "./pages/Deploys.tsx"; |
| 20 | 21 | import { Deploy } from "./pages/Deploy.tsx"; |
| 21 | 22 | import { VMs } from "./pages/VMs.tsx"; |
| ... | ... | @@ -83,6 +84,8 @@ render(() => ( |
| 83 | 84 | <Route path="/youtube" component={YouTube} /> |
| 84 | 85 | <Route path="/paper-clover" component={PaperClover} preload={preload(queries.paperClover, queries.paperCloverActivity)} /> |
| 85 | 86 | <Route path="/users/:name?" component={Users} preload={preload(directory)} /> |
| 87 | <Route path="/shares/accounts/:id" component={Shares} preload={preload(shares)} /> | |
| 88 | <Route path="/shares/:key?" component={Shares} preload={preload(shares)} /> | |
| 86 | 89 | <Route path="/deploys" component={Deploys} preload={preload(queries.deploys, queries.services)} /> |
| 87 | 90 | <Route path="/deploys/:id/:tab?" component={Deploy} preload={preload(queries.deploys, queries.services)} /> |
| 88 | 91 | <Route path="/vms" component={VMs} preload={preload(queries.vms)} /> |
dashboard/web/pages/Shares.css created+45| ... | ... | @@ -0,0 +1,45 @@ |
| 1 | .shares-toolbar { display: flex; align-items: center; flex-wrap: wrap; gap: 12px; margin-bottom: 12px; } | |
| 2 | .shares-toolbar .search-box { margin-left: auto; width: 240px; } | |
| 3 | .shares-toolbar .count { opacity: 0.7; } | |
| 4 | .shares-table { overflow: auto; } | |
| 5 | .shares-table th { white-space: nowrap; } | |
| 6 | .share-name { color: var(--accent); font-weight: 600; } | |
| 7 | .share-name:hover { text-decoration: underline; } | |
| 8 | .share-path { font: 12px/1.5 var(--mono); overflow-wrap: anywhere; } | |
| 9 | .shares-table .share-path { min-width: 170px; } | |
| 10 | .share-detail { display: block; color: var(--muted); font-size: 12px; margin-top: 2px; } | |
| 11 | .share-expiry { white-space: nowrap; } | |
| 12 | .share-account-folder { display: flex; flex-wrap: wrap; gap: 4px 12px; margin-block: 3px; } | |
| 13 | .share-detail-page .page-head { flex-wrap: wrap; } | |
| 14 | .share-detail-page .page-head h1 { overflow-wrap: anywhere; } | |
| 15 | .share-form { max-width: 800px; } | |
| 16 | .share-form > fieldset { display: grid; gap: 18px; margin: 0; padding: 0; border: 0; min-width: 0; } | |
| 17 | .share-form .field { display: grid; gap: 5px; min-width: 0; } | |
| 18 | .share-folder-field { display: flex; gap: 6px; } | |
| 19 | .share-current-folder { display: flex; align-items: center; flex-wrap: wrap; gap: 8px 12px; } | |
| 20 | .share-current-folder .share-path { flex: 1; } | |
| 21 | .dialog .share-form { display: grid; gap: 18px; } | |
| 22 | .share-folder-field input { flex: 1; min-width: 0; font-family: var(--mono); font-size: 12px; } | |
| 23 | .share-rights, .share-path-mode { border: 0; margin: 0; padding: 0; min-width: 0; } | |
| 24 | .share-rights legend, .share-path-mode legend { margin-bottom: 7px; padding: 0; } | |
| 25 | .share-rights > div { display: flex; flex-wrap: wrap; gap: 8px 18px; } | |
| 26 | .share-path-mode { display: grid; gap: 8px; } | |
| 27 | .share-path-mode label { display: grid; grid-template-columns: 18px 1fr; column-gap: 5px; align-items: center; } | |
| 28 | .share-path-mode input { grid-column: 1; grid-row: 1; margin: 0; accent-color: var(--accent); } | |
| 29 | .share-path-mode .hint { grid-column: 2; margin: 0; } | |
| 30 | .share-options { display: grid; grid-template-columns: 1fr 1fr; gap: 18px; } | |
| 31 | .share-options button { justify-self: start; } | |
| 32 | .share-selection { margin: 0; padding-left: 20px; overflow-wrap: anywhere; } | |
| 33 | .share-columns { display: grid; grid-template-columns: minmax(250px, 1fr) minmax(320px, 1.5fr); gap: 8px; align-items: start; } | |
| 34 | .share-columns .card-title { margin: 0 0 12px; align-items: center; color: var(--text); } | |
| 35 | .share-grant { display: flex; align-items: center; gap: 6px; padding-block: 7px; } | |
| 36 | .share-grant > div { display: grid; flex: 1; min-width: 0; gap: 2px; } | |
| 37 | .share-grant .muted { font-size: 12px; } | |
| 38 | .share-grants p { font-size: 13px; } | |
| 39 | .share-secret dd { overflow-wrap: anywhere; } | |
| 40 | .share-picker .field { display: grid; gap: 5px; } | |
| 41 | .share-picker-list { max-height: 300px; min-height: 100px; overflow: auto; margin-top: 10px; } | |
| 42 | .share-picker-list button { display: flex; align-items: center; gap: 8px; text-align: left; border: 0; background: none; width: 100%; padding: 7px 8px; cursor: pointer; border-radius: 4px; overflow-wrap: anywhere; } | |
| 43 | .share-picker-list button:hover { background: var(--hover); } | |
| 44 | @media (max-width: 1000px) { .share-columns { grid-template-columns: 1fr; } } | |
| 45 | @media (max-width: 600px) { .share-options { grid-template-columns: 1fr; } .shares-toolbar .search-box { margin-left: 0; width: 100%; } } |
dashboard/web/pages/Shares.tsx created+318| ... | ... | @@ -0,0 +1,318 @@ |
| 1 | import { A, useNavigate, useParams, useSearchParams } from "@solidjs/router"; | |
| 2 | import ArrowLeft from "lucide-solid/icons/arrow-left"; | |
| 3 | import Folder from "lucide-solid/icons/folder"; | |
| 4 | import Plus from "lucide-solid/icons/plus"; | |
| 5 | import Search from "lucide-solid/icons/search"; | |
| 6 | import { createMemo, createResource, createSignal, For, onMount, Show } from "solid-js"; | |
| 7 | import { parseResponse } from "hono/client"; | |
| 8 | import { api, query, reason } from "../api.ts"; | |
| 9 | import { Checkbox } from "../components/Checkbox.tsx"; | |
| 10 | import { Copy } from "../components/Copy.tsx"; | |
| 11 | import { showConfirmDialog, showTextDialog } from "../components/Dialog.tsx"; | |
| 12 | import { ListPage } from "../components/ListPage.tsx"; | |
| 13 | import { lastGood, Loaded, SkeletonRows } from "../components/Loaded.tsx"; | |
| 14 | import { TabBar } from "../components/TabBar.tsx"; | |
| 15 | import type { AccountSecret, FileAccount, FolderGrant, ShareState, SharedLink } from "../types/shares.ts"; | |
| 16 | import { datetime, plural } from "../format.ts"; | |
| 17 | import "./Shares.css"; | |
| 18 | ||
| 19 | export const shares = query("shares", () => parseResponse(api.shares.$get())); | |
| 20 | ||
| 21 | const RIGHTS = [["r", "read"], ["w", "upload"], ["m", "move / rename"], ["d", "delete"], [".", "hidden files"], ["a", "Copyparty admin"]] as const; | |
| 22 | const permissions = (value: string) => RIGHTS.filter(([key]) => value.includes(key)).map(([, label]) => label).join(", "); | |
| 23 | const macPath = (path: string, root: string) => path === root || path.startsWith(root + "/") ? "/Volumes/clover" + path.slice(root.length) : path; | |
| 24 | const linkUrl = (state: ShareState, key: string) => state.filesUrl ? `${state.filesUrl.replace(/\/$/, "")}/shr/${encodeURIComponent(key)}` : null; | |
| 25 | ||
| 26 | function Rights(props: { value: string; onChange: (value: string) => void; link?: boolean }) { | |
| 27 | return <fieldset class="share-rights"><legend>permissions</legend><div> | |
| 28 | <For each={props.link ? RIGHTS.slice(0, 2) : RIGHTS}>{([key, label]) => | |
| 29 | <Checkbox checked={props.value.includes(key)} onChange={(checked) => props.onChange(checked ? props.value + key : props.value.replace(key, ""))}>{label}</Checkbox> | |
| 30 | }</For> | |
| 31 | </div></fieldset>; | |
| 32 | } | |
| 33 | ||
| 34 | function FolderField(props: { value: string; root: string; create?: boolean; onChange: (value: string, existing?: boolean) => void }) { | |
| 35 | const browse = () => { | |
| 36 | const [path, setPath] = createSignal(props.create && !props.value.endsWith("/") ? props.value.replace(/\/[^/]+$/, "") || props.root : props.value || props.root); | |
| 37 | showConfirmDialog({ | |
| 38 | title: "Choose folder", confirmLabel: "use folder", | |
| 39 | body: () => { | |
| 40 | const [folders, { refetch }] = createResource(path, (path) => parseResponse(api.shares.folders.$get({ query: { path } }))); | |
| 41 | return <div class="share-picker"> | |
| 42 | <label class="field">folder<input class="search" value={path()} onChange={(e) => setPath(e.currentTarget.value)} /></label> | |
| 43 | <button type="button" class="button small" disabled={path() === props.root || path() === "/Volumes/clover"} | |
| 44 | onClick={() => setPath(path().replace(/\/[^/]+\/?$/, "") || props.root)}>parent folder</button> | |
| 45 | <Loaded data={folders} what="folders" retry={refetch} skeleton={<SkeletonRows count={4} />}> | |
| 46 | {(data) => <div class="share-picker-list"><For each={data().directories} fallback={<p class="muted">No folders here</p>}> | |
| 47 | {(folder) => <button type="button" onClick={() => setPath(macPath(folder.path, props.root))}><Folder size={16} aria-hidden="true" />{folder.name}</button>} | |
| 48 | </For></div>} | |
| 49 | </Loaded> | |
| 50 | </div>; | |
| 51 | }, | |
| 52 | onConfirm: () => props.onChange(macPath(path(), props.root), true), | |
| 53 | }); | |
| 54 | }; | |
| 55 | return <div class="field">folder<div class="share-folder-field"> | |
| 56 | <input aria-label="Folder" class="search" value={props.value} required autocomplete="off" spellcheck={false} | |
| 57 | onInput={(event) => props.onChange(event.currentTarget.value)} /> | |
| 58 | <button type="button" class="button" onClick={browse}>browse</button> | |
| 59 | </div></div>; | |
| 60 | } | |
| 61 | ||
| 62 | function showSecret(result: AccountSecret, returnFocus?: HTMLElement) { | |
| 63 | showConfirmDialog({ | |
| 64 | title: "Save account password", confirmLabel: "done", | |
| 65 | returnFocus, | |
| 66 | description: "This password is shown once. Save it before closing.", | |
| 67 | body: () => <dl class="kv share-secret"><dt>username</dt><dd><Copy value={result.account.username} /></dd><dt>password</dt><dd><Copy value={result.password} /></dd></dl>, | |
| 68 | onConfirm: () => {}, | |
| 69 | }); | |
| 70 | } | |
| 71 | ||
| 72 | function editGrant(props: { subject: string; state: ShareState; grant?: FolderGrant; saved: (state: ShareState) => void }) { | |
| 73 | const [path, setPath] = createSignal(macPath(props.grant?.path ?? props.state.defaultRoot, props.state.root)); | |
| 74 | const [rights, setRights] = createSignal(props.grant?.permissions ?? "r"); | |
| 75 | showConfirmDialog({ | |
| 76 | title: props.grant ? "Edit folder access" : "Add folder access", confirmLabel: "save access", | |
| 77 | returnFocus: document.querySelector<HTMLElement>(".folder-access .card-title button") ?? undefined, | |
| 78 | canConfirm: () => !!path().trim() && !!rights(), | |
| 79 | body: () => <div class="share-form"> | |
| 80 | <FolderField value={path()} root={props.state.root} onChange={setPath} /> | |
| 81 | <Rights value={rights()} onChange={setRights} /> | |
| 82 | <p class="hint">Folder permissions add to existing access.</p> | |
| 83 | </div>, | |
| 84 | onConfirm: async () => { | |
| 85 | const json = { subject: props.subject, path: path(), permissions: rights() }; | |
| 86 | const state = await parseResponse(props.grant | |
| 87 | ? api.shares.grants[":id"].$patch({ param: { id: props.grant.id }, json }) | |
| 88 | : api.shares.grants.$post({ json })); | |
| 89 | props.saved(state); | |
| 90 | }, | |
| 91 | }); | |
| 92 | } | |
| 93 | ||
| 94 | export function FolderAccess(props: { subject: string; enabled?: boolean }) { | |
| 95 | const [state, { mutate, refetch }] = shares.use(); | |
| 96 | const ready = lastGood(state); | |
| 97 | return <section class="card folder-access"> | |
| 98 | <h2 class="card-title">folder access<span class="spacer" /> | |
| 99 | <button class="button small" disabled={!ready() || props.enabled === false} onClick={() => editGrant({ subject: props.subject, state: ready()!, saved: mutate })}>add folder</button> | |
| 100 | </h2> | |
| 101 | <Loaded data={state} what="folder access" retry={refetch} skeleton={<SkeletonRows count={2} />}> | |
| 102 | {(data) => <GrantList subject={props.subject} state={data()} saved={mutate} enabled={props.enabled} />} | |
| 103 | </Loaded> | |
| 104 | </section>; | |
| 105 | } | |
| 106 | ||
| 107 | function GrantList(props: { subject: string; state: ShareState; saved: (state: ShareState) => void; enabled?: boolean }) { | |
| 108 | const grants = () => props.state.grants.filter((grant) => grant.subject === props.subject); | |
| 109 | const remove = (grant: FolderGrant) => showConfirmDialog({ | |
| 110 | title: "Remove folder access?", description: macPath(grant.path, props.state.root), confirmLabel: "remove access", destructive: true, | |
| 111 | returnFocus: document.querySelector<HTMLElement>(".folder-access .card-title button") ?? undefined, | |
| 112 | onConfirm: async () => props.saved(await parseResponse(api.shares.grants[":id"].$delete({ param: { id: grant.id } }))), | |
| 113 | }); | |
| 114 | return <div class="share-grants"> | |
| 115 | <For each={grants()} fallback={<p class="muted">No folder permissions assigned here</p>}>{(grant) => <div class="share-grant"> | |
| 116 | <div><span class="share-path">{macPath(grant.path, props.state.root)}</span><span class="muted">{permissions(grant.permissions)}</span></div> | |
| 117 | <button class="button small" aria-label={`Edit access to ${macPath(grant.path, props.state.root)}`} disabled={props.enabled === false} onClick={() => editGrant({ subject: props.subject, state: props.state, grant, saved: props.saved })}>edit</button> | |
| 118 | <button class="button small danger" aria-label={`Remove access to ${macPath(grant.path, props.state.root)}`} onClick={() => remove(grant)}>remove</button> | |
| 119 | </div>}</For> | |
| 120 | <Show when={grants().length}><p class="hint">Folder permissions add to existing access.</p></Show> | |
| 121 | </div>; | |
| 122 | } | |
| 123 | ||
| 124 | export function Shares() { | |
| 125 | const params = useParams<{ key?: string; id?: string }>(); | |
| 126 | const [search, setSearch] = useSearchParams<{ tab?: string; q?: string }>(); | |
| 127 | const [state, { refetch, mutate }] = shares.use(); | |
| 128 | const navigate = useNavigate(); | |
| 129 | const accounts = () => search.tab === "accounts"; | |
| 130 | const createAccount = () => { | |
| 131 | let credentials: AccountSecret | undefined; | |
| 132 | showTextDialog({ | |
| 133 | title: "New API account", label: "name", placeholder: "T3Code", confirmLabel: "create account", | |
| 134 | onConfirm: async (name) => { | |
| 135 | const result = await parseResponse(api.shares.accounts.$post({ json: { name } })); | |
| 136 | mutate(result); | |
| 137 | navigate(`/shares/accounts/${result.account.id}`); | |
| 138 | credentials = result; | |
| 139 | }, | |
| 140 | onClose: () => { if (credentials) showSecret(credentials, document.querySelector<HTMLElement>(".share-detail-page h1") ?? undefined); }, | |
| 141 | }); | |
| 142 | }; | |
| 143 | return <Loaded data={state} what="shares" retry={refetch} skeleton={<div class="page"><SkeletonRows count={6} /></div>}> | |
| 144 | {(data) => <Show when={params.id} keyed fallback={<Show when={params.key} keyed fallback={ | |
| 145 | <ListPage id="shares" class="shares-page" flush head={<div class="page-head"><h1>shares</h1><span class="spacer" /> | |
| 146 | <Show when={accounts()} fallback={<A class="button primary" href="/shares/new"><Plus size={14} aria-hidden="true" />new share</A>}> | |
| 147 | <button class="button primary" onClick={createAccount}><Plus size={14} aria-hidden="true" />new account</button> | |
| 148 | </Show> | |
| 149 | </div>}> | |
| 150 | <div class="shares-toolbar"> | |
| 151 | <TabBar label="Share manager"><button aria-pressed={!accounts()} onClick={() => setSearch({ tab: undefined, q: undefined })}>shared links<span class="count">{data().links.length}</span></button> | |
| 152 | <button aria-pressed={accounts()} onClick={() => setSearch({ tab: "accounts", q: undefined })}>API accounts<span class="count">{data().accounts.length}</span></button> | |
| 153 | </TabBar> | |
| 154 | <label class="search-box"><Search size={14} aria-hidden="true" /><input type="search" aria-label="Search shares" placeholder="name or folder" value={search.q ?? ""} onInput={(e) => setSearch({ q: e.currentTarget.value || undefined }, { replace: true })} /></label> | |
| 155 | </div> | |
| 156 | <div class="fill shares-table"><Show when={accounts()} fallback={<LinkList state={data()} filter={search.q ?? ""} />}> | |
| 157 | <AccountList state={data()} filter={search.q ?? ""} create={createAccount} /> | |
| 158 | </Show></div> | |
| 159 | </ListPage> | |
| 160 | }>{(key) => <LinkPage key={key} state={data()} saved={mutate} />}</Show>}> | |
| 161 | {(id) => <Show when={data().accounts.find((account) => account.id === id)} fallback={<div class="page empty">This account no longer exists. <A href="/shares?tab=accounts">back to accounts</A></div>}> | |
| 162 | {(account) => <AccountPage account={account()} state={data()} saved={mutate} />} | |
| 163 | </Show>} | |
| 164 | </Show>} | |
| 165 | </Loaded>; | |
| 166 | } | |
| 167 | ||
| 168 | function LinkList(props: { state: ShareState; filter: string }) { | |
| 169 | const links = () => props.state.links.filter((link) => `${link.name} ${link.path} ${link.key}`.toLowerCase().includes(props.filter.toLowerCase())); | |
| 170 | return <Show when={links().length} fallback={<div class="empty">{props.filter ? "No shares match" : "No shared links yet"} <A class="button small" href="/shares/new">new share</A></div>}> | |
| 171 | <table class="data"><thead><tr><th>share</th><th>folder</th><th>access</th><th>expires</th><th>public link</th></tr></thead><tbody> | |
| 172 | <For each={links()}>{(link) => <tr> | |
| 173 | <td><A class="share-name" href={`/shares/${encodeURIComponent(link.key)}`}>{link.name || link.key}</A> | |
| 174 | <Show when={link.files.length}><span class="share-detail">{plural(link.files.length, "selected file")}</span></Show> | |
| 175 | </td> | |
| 176 | <td class="share-path">{macPath(link.path, props.state.root)}</td> | |
| 177 | <td>{permissions(link.permissions)}<Show when={link.hasPassword}><span class="share-detail">password protected</span></Show></td> | |
| 178 | <td class="share-expiry">{link.expires ? datetime(link.expires) : "never"}</td> | |
| 179 | <td><Show when={linkUrl(props.state, link.key)}>{(url) => <Copy value={url()} label="link">copy link</Copy>}</Show></td> | |
| 180 | </tr>}</For> | |
| 181 | </tbody></table> | |
| 182 | </Show>; | |
| 183 | } | |
| 184 | ||
| 185 | function AccountList(props: { state: ShareState; filter: string; create: () => void }) { | |
| 186 | const accounts = () => props.state.accounts.filter((account) => `${account.name} ${account.username}`.toLowerCase().includes(props.filter.toLowerCase())); | |
| 187 | return <Show when={accounts().length} fallback={<div class="empty">{props.filter ? "No accounts match" : "No API accounts yet"} <button class="button small" onClick={props.create}>new account</button></div>}> | |
| 188 | <table class="data"><thead><tr><th>account</th><th>username</th><th>folders</th></tr></thead><tbody> | |
| 189 | <For each={accounts()}>{(account) => <tr><td><A class="share-name" href={`/shares/accounts/${account.id}`}>{account.name}</A></td><td><Copy value={account.username} /></td> | |
| 190 | <td><For each={props.state.grants.filter((grant) => grant.subject === account.subject)} fallback={<span class="muted">no folders assigned</span>}>{(grant) => <span class="share-account-folder"><span class="share-path">{macPath(grant.path, props.state.root)}</span><span class="muted">{permissions(grant.permissions)}</span></span>}</For></td> | |
| 191 | </tr>}</For> | |
| 192 | </tbody></table> | |
| 193 | </Show>; | |
| 194 | } | |
| 195 | ||
| 196 | function AccountPage(props: { account: FileAccount; state: ShareState; saved: (state: ShareState) => void }) { | |
| 197 | const navigate = useNavigate(); | |
| 198 | let heading!: HTMLHeadingElement; | |
| 199 | onMount(() => heading.focus()); | |
| 200 | const rename = () => showTextDialog({ title: "Rename account", label: "name", initialValue: props.account.name, confirmLabel: "save name", | |
| 201 | onConfirm: async (name) => props.saved(await parseResponse(api.shares.accounts[":id"].$patch({ param: { id: props.account.id }, json: { name } }))), | |
| 202 | }); | |
| 203 | const rotate = () => { | |
| 204 | let credentials: AccountSecret | undefined; | |
| 205 | showConfirmDialog({ title: "Replace account password?", description: "The old password and open sessions stop working.", confirmLabel: "replace password", destructive: true, | |
| 206 | onConfirm: async () => { | |
| 207 | const result = await parseResponse(api.shares.accounts[":id"].rotate.$post({ param: { id: props.account.id } })); | |
| 208 | props.saved(result); credentials = result; | |
| 209 | }, | |
| 210 | onClose: () => { if (credentials) showSecret(credentials); }, | |
| 211 | }); | |
| 212 | }; | |
| 213 | const remove = () => showConfirmDialog({ title: `Delete ${props.account.name}?`, description: "This account loses access to its folders. The files stay in place.", confirmLabel: "delete account", destructive: true, | |
| 214 | onConfirm: async () => { | |
| 215 | props.saved(await parseResponse(api.shares.accounts[":id"].$delete({ param: { id: props.account.id } }))); | |
| 216 | navigate("/shares?tab=accounts"); | |
| 217 | }, | |
| 218 | }); | |
| 219 | return <div class="page share-detail-page"><div class="page-head"><A class="button icon-only" href="/shares?tab=accounts" aria-label="Back to API accounts"><ArrowLeft size={16} /></A> | |
| 220 | <h1 ref={heading} tabindex="-1">{props.account.name}</h1><span class="sub">API account</span><span class="spacer" /><button class="button danger" onClick={remove}>delete account</button></div> | |
| 221 | <div class="share-columns"><section class="card"><h2 class="card-title">connection<span class="spacer" /><button class="button small" onClick={rename}>rename</button></h2> | |
| 222 | <dl class="kv"><dt>username</dt><dd><Copy value={props.account.username} /></dd><dt>server</dt><dd><Show when={props.state.filesUrl} fallback="unavailable">{(url) => <Copy value={url()} />}</Show></dd></dl> | |
| 223 | <p class="hint">Use this account with HTTP, WebDAV, or an AI client.</p><button class="button small" onClick={rotate}>replace password</button> | |
| 224 | </section><section class="card folder-access"><h2 class="card-title">folder access<span class="spacer" /><button class="button small" onClick={() => editGrant({ subject: props.account.subject, state: props.state, saved: props.saved })}>add folder</button></h2> | |
| 225 | <GrantList subject={props.account.subject} state={props.state} saved={props.saved} /> | |
| 226 | </section></div> | |
| 227 | </div>; | |
| 228 | } | |
| 229 | ||
| 230 | function LinkPage(props: { key: string; state: ShareState; saved: (state: ShareState) => void }) { | |
| 231 | const navigate = useNavigate(); | |
| 232 | const original = () => props.state.links.find((link) => link.key === props.key); | |
| 233 | const fresh = props.key === "new"; | |
| 234 | return <Show when={fresh || original()} fallback={<div class="page empty">This share no longer exists. <A href="/shares">back to shares</A></div>}> | |
| 235 | <LinkForm original={original()} fresh={fresh} state={props.state} saved={(state, key) => { props.saved(state); if (fresh) navigate(`/shares/${encodeURIComponent(key)}`); }} /> | |
| 236 | </Show>; | |
| 237 | } | |
| 238 | ||
| 239 | function LinkForm(props: { original?: SharedLink; fresh: boolean; state: ShareState; saved: (state: ShareState, key: string) => void }) { | |
| 240 | const navigate = useNavigate(); | |
| 241 | let heading!: HTMLHeadingElement; | |
| 242 | onMount(() => heading.focus()); | |
| 243 | const [name, setName] = createSignal(props.original?.name ?? ""); | |
| 244 | const [customPath, setCustomPath] = createSignal(""); | |
| 245 | const path = () => customPath() || `${macPath(props.state.defaultRoot, props.state.root)}/${name().trim()}`; | |
| 246 | const [rights, setRights] = createSignal(props.original?.permissions ?? "r"); | |
| 247 | const [createFolder, setCreateFolder] = createSignal(true); | |
| 248 | const [password, setPassword] = createSignal<string | null>(null); | |
| 249 | const dateValue = (seconds: number) => { | |
| 250 | const date = new Date(seconds * 1000); | |
| 251 | return new Date(date.getTime() - date.getTimezoneOffset() * 60_000).toISOString().slice(0, 16); | |
| 252 | }; | |
| 253 | const [expires, setExpires] = createSignal(props.original?.expires ? dateValue(props.original.expires) : ""); | |
| 254 | const [pending, setPending] = createSignal(false); | |
| 255 | const [error, setError] = createSignal(""); | |
| 256 | const dirty = createMemo(() => props.fresh || name() !== props.original?.name || rights() !== props.original?.permissions || password() !== null || expires() !== (props.original?.expires ? dateValue(props.original.expires) : "")); | |
| 257 | const save = async () => { | |
| 258 | if (pending() || !dirty()) return; | |
| 259 | setPending(true); setError(""); | |
| 260 | try { | |
| 261 | const json = { name: name().trim(), path: props.original?.path ?? path(), permissions: rights(), expires: expires() ? new Date(expires()).getTime() / 1000 : 0, password: password() }; | |
| 262 | const result = await parseResponse(props.original | |
| 263 | ? api.shares.links[":key"].$patch({ param: { key: props.original.key }, json: { ...json, mode: "retarget" } }) | |
| 264 | : api.shares.links.$post({ json: { ...json, createFolder: createFolder() } })); | |
| 265 | const key = props.original?.key ?? result.links.find((link) => !props.state.links.some((old) => old.key === link.key))?.key; | |
| 266 | if (key) props.saved(result, key); | |
| 267 | setPassword(null); | |
| 268 | } catch (failure) { setError(reason(failure)); } | |
| 269 | finally { setPending(false); } | |
| 270 | }; | |
| 271 | const changeFolder = () => { | |
| 272 | const link = props.original!; | |
| 273 | const [destination, setDestination] = createSignal(macPath(link.path, props.state.root)); | |
| 274 | const [mode, setMode] = createSignal<"retarget" | "move">("retarget"); | |
| 275 | showConfirmDialog({ | |
| 276 | title: "Change folder", | |
| 277 | get confirmLabel() { return mode() === "move" ? "move folder" : "use folder"; }, | |
| 278 | canConfirm: () => !!destination().trim() && destination() !== macPath(link.path, props.state.root), | |
| 279 | body: () => <div class="share-form"> | |
| 280 | <FolderField value={destination()} root={props.state.root} onChange={setDestination} /> | |
| 281 | <fieldset class="share-path-mode"><legend>change folder</legend> | |
| 282 | <label><input type="radio" name="path-mode" checked={mode() === "retarget"} onChange={() => setMode("retarget")} />use another folder<span class="hint">Share the files already at the new path</span></label> | |
| 283 | <label><input type="radio" name="path-mode" checked={mode() === "move"} onChange={() => setMode("move")} />move files<span class="hint">Move the current folder to the new path</span></label> | |
| 284 | </fieldset> | |
| 285 | <p class="hint">Public share links stay the same. Other shares inside a moved folder follow it.</p> | |
| 286 | </div>, | |
| 287 | onConfirm: async () => props.saved(await parseResponse(api.shares.links[":key"].$patch({ param: { key: link.key }, json: { | |
| 288 | name: link.name, path: destination(), permissions: link.permissions, expires: link.expires, password: null, mode: mode(), | |
| 289 | } })), link.key), | |
| 290 | }); | |
| 291 | }; | |
| 292 | const remove = () => showConfirmDialog({ title: `Delete ${props.original?.name || "share"}?`, description: "The public link stops working. The files stay in place.", confirmLabel: "delete share", destructive: true, | |
| 293 | onConfirm: async () => { const state = await parseResponse(api.shares.links[":key"].$delete({ param: { key: props.original!.key } })); props.saved(state, props.original!.key); navigate("/shares"); }, | |
| 294 | }); | |
| 295 | return <div class="page share-detail-page"><div class="page-head"><A class="button icon-only" href="/shares" aria-label="Back to shares"><ArrowLeft size={16} /></A> | |
| 296 | <h1 ref={heading} tabindex="-1">{props.fresh ? "new share" : props.original?.name || props.original?.key}</h1><span class="spacer" /> | |
| 297 | <Show when={!props.fresh}><button class="button danger" disabled={pending()} onClick={remove}>delete share</button></Show> | |
| 298 | <button form="share-form" type="submit" class="button primary" disabled={!dirty() || pending()} aria-busy={pending()}>{props.fresh ? "create share" : "save changes"}</button></div> | |
| 299 | <form id="share-form" class="share-form card" onSubmit={(event) => { event.preventDefault(); void save(); }}> | |
| 300 | <fieldset disabled={pending()}> | |
| 301 | <label class="field">name<input class="search" required value={name()} placeholder="Name - Description" autocomplete="off" onInput={(event) => setName(event.currentTarget.value)} /></label> | |
| 302 | <Show when={props.fresh} fallback={<div class="field">folder<div class="share-current-folder"><span class="share-path">{macPath(props.original!.path, props.state.root)}</span><button type="button" class="button" onClick={changeFolder}>change folder</button></div></div>}> | |
| 303 | <FolderField value={path()} root={props.state.root} create={createFolder()} onChange={(value, existing) => { setCustomPath(value); if (existing) setCreateFolder(false); }} /> | |
| 304 | <Checkbox checked={createFolder()} onChange={setCreateFolder}>create this folder</Checkbox> | |
| 305 | </Show> | |
| 306 | <Show when={props.original?.files.length}><div class="field">selected files<ul class="share-selection"><For each={props.original?.files}>{(file) => <li>{file}</li>}</For></ul><span class="hint">This share keeps its file selection when its folder changes</span></div></Show> | |
| 307 | <Rights value={rights()} onChange={setRights} link /> | |
| 308 | <div class="share-options"><label class="field">expires<input type="datetime-local" class="search" value={expires()} onInput={(event) => setExpires(event.currentTarget.value)} /><span class="hint">Leave empty for no expiry</span></label> | |
| 309 | <label class="field">password<input class="search" type="password" autocomplete="new-password" value={password() ?? ""} placeholder={props.original?.hasPassword ? "Keep current password" : "No password"} onInput={(event) => setPassword(event.currentTarget.value)} /> | |
| 310 | <Show when={props.original?.hasPassword}><button type="button" class="button small" onClick={() => setPassword("")}>remove password</button></Show> | |
| 311 | <Show when={props.original?.hasPassword && password() === ""}><span class="hint">Password will be removed when saved</span></Show> | |
| 312 | </label></div> | |
| 313 | <Show when={props.original && linkUrl(props.state, props.original.key)}>{(url) => <div class="field">public link<Copy value={url()} /></div>}</Show> | |
| 314 | </fieldset> | |
| 315 | <Show when={error()}><p class="error" role="alert">{error()}</p></Show> | |
| 316 | </form> | |
| 317 | </div>; | |
| 318 | } |
dashboard/web/pages/Users.tsx+2| ... | ... | @@ -23,6 +23,7 @@ import { toast } from "../components/Toast.tsx"; |
| 23 | 23 | import { ago, count, date, datetime, plural } from "../format.ts"; |
| 24 | 24 | import "./Users.css"; |
| 25 | 25 | import { CommitIdentity } from "./Account.tsx"; |
| 26 | import { FolderAccess } from "./Shares.tsx"; | |
| 26 | 27 | |
| 27 | 28 | const STEPS: [string, string][] = [["SETUP", "finish setup"], ["UPDATE_PASSWORD", "new password"], ["UPDATE_PROFILE", "check profile"]]; |
| 28 | 29 | |
| ... | ... | @@ -592,6 +593,7 @@ function Profile(props: { user: User; data: Data; refetch: () => unknown }) { |
| 592 | 593 | </Show> |
| 593 | 594 | </Show> |
| 594 | 595 | </section> |
| 596 | <Show when={props.user.kind !== "guest"}><FolderAccess subject={`user:${props.user.id}`} enabled={props.user.enabled} /></Show> | |
| 595 | 597 | <section class="card"> |
| 596 | 598 | <h2 class="card-title"> |
| 597 | 599 | sign-in |
dashboard/web/types/shares.ts created+44| ... | ... | @@ -0,0 +1,44 @@ |
| 1 | export interface FileAccount { | |
| 2 | id: string; | |
| 3 | name: string; | |
| 4 | username: string; | |
| 5 | subject: string; | |
| 6 | } | |
| 7 | ||
| 8 | export interface FolderGrant { | |
| 9 | id: string; | |
| 10 | subject: string; | |
| 11 | path: string; | |
| 12 | permissions: string; | |
| 13 | } | |
| 14 | ||
| 15 | export interface SharedLink { | |
| 16 | key: string; | |
| 17 | name: string; | |
| 18 | path: string; | |
| 19 | permissions: string; | |
| 20 | owner: string; | |
| 21 | created: number; | |
| 22 | expires: number; | |
| 23 | hasPassword: boolean; | |
| 24 | files: string[]; | |
| 25 | } | |
| 26 | ||
| 27 | export interface ShareState { | |
| 28 | root: string; | |
| 29 | defaultRoot: string; | |
| 30 | filesUrl: string | null; | |
| 31 | accounts: FileAccount[]; | |
| 32 | grants: FolderGrant[]; | |
| 33 | links: SharedLink[]; | |
| 34 | } | |
| 35 | ||
| 36 | export interface LinkInput { | |
| 37 | name: string; | |
| 38 | path: string; | |
| 39 | permissions: string; | |
| 40 | expires: number; | |
| 41 | password: string | null; | |
| 42 | } | |
| 43 | ||
| 44 | export type AccountSecret = ShareState & { account: FileAccount; password: string }; |
nixos/configuration.nix+3-3| ... | ... | @@ -6,7 +6,7 @@ let |
| 6 | 6 | proxyToken = "/var/lib/studio/dashboard-proxy.token"; |
| 7 | 7 | hostTools = lib.fileset.toSource { |
| 8 | 8 | root = ../.; |
| 9 | fileset = lib.fileset.unions [ ../tools/dashboard-host.py ../tools/shale-labels.py ../config/shale.json ../dashboard/src/host-protocol.json ../tools/vms.py ../tools/vm-ssh.py ../tools/dashboard-run.py ../tools/release.py ../guest/protocol.json ]; | |
| 9 | fileset = lib.fileset.unions [ ../tools/dashboard-host.py ../service/copyparty/manage.py ../service/copyparty/copyparty.conf ../tools/shale-labels.py ../config/shale.json ../dashboard/src/host-protocol.json ../tools/vms.py ../tools/vm-ssh.py ../tools/dashboard-run.py ../tools/release.py ../guest/protocol.json ]; | |
| 10 | 10 | }; |
| 11 | 11 | nativePkl = pkgs.callPackage ./pkl.nix { }; |
| 12 | 12 | screenPython = pkgs.python3.withPackages (packages: [ packages.pygobject3 packages.gst-python ]); |
| ... | ... | @@ -203,11 +203,11 @@ in |
| 203 | 203 | RuntimeDirectoryPreserve = "yes"; |
| 204 | 204 | UMask = "0077"; |
| 205 | 205 | ProtectSystem = "strict"; |
| 206 | ReadWritePaths = [ "/srv/vm" "-/srv/prod/shale/data" "/var/lib/studio/dashboard" ] ++ lib.optionals (config.networking.hostName != "clover-demo") [ "/srv/clover/Media/vm" ]; | |
| 206 | ReadWritePaths = [ "/srv/vm" "-/srv/prod/shale/data" "-/srv/prod/copyparty/cfg" "/srv/staging" "/srv/clover" "/var/lib/studio/dashboard" ] ++ lib.optionals (config.networking.hostName != "clover-demo") [ "/srv/clover/Media/vm" ]; | |
| 207 | 207 | ProtectHome = true; |
| 208 | 208 | PrivateTmp = true; |
| 209 | 209 | NoNewPrivileges = true; |
| 210 | CapabilityBoundingSet = [ "CAP_SYS_ADMIN" "CAP_DAC_READ_SEARCH" "CAP_DAC_OVERRIDE" ]; | |
| 210 | CapabilityBoundingSet = [ "CAP_SYS_ADMIN" "CAP_DAC_READ_SEARCH" "CAP_DAC_OVERRIDE" "CAP_CHOWN" ]; | |
| 211 | 211 | MemoryMax = "256M"; |
| 212 | 212 | TasksMax = 64; |
| 213 | 213 | DevicePolicy = "closed"; |
service/copyparty/copyparty.conf+1| ... | ... | @@ -9,6 +9,7 @@ |
| 9 | 9 | name: clover's nas |
| 10 | 10 | stats, nos-dup |
| 11 | 11 | dav-auth |
| 12 | xvol | |
| 12 | 13 | cookie-cmax: 65536 |
| 13 | 14 | ah-alg: argon2 |
| 14 | 15 | xbu: c,/hooks/reject-apple-cruft.py |
service/copyparty/manage.py created+697| ... | ... | @@ -0,0 +1,697 @@ |
| 1 | import copy | |
| 2 | import ctypes | |
| 3 | import errno | |
| 4 | import fcntl | |
| 5 | import importlib | |
| 6 | import json | |
| 7 | import os | |
| 8 | from pathlib import Path | |
| 9 | import re | |
| 10 | import secrets | |
| 11 | import sqlite3 | |
| 12 | import subprocess | |
| 13 | import time | |
| 14 | import urllib.error | |
| 15 | import urllib.request | |
| 16 | import uuid | |
| 17 | ||
| 18 | runs = importlib.import_module('dashboard-run') | |
| 19 | Error = runs.Error | |
| 20 | PODMAN = ['podman', '--remote', '--url', 'unix:///run/podman/podman.sock'] | |
| 21 | ||
| 22 | ACTIONS = { | |
| 23 | 'state': set(), 'folders.list': {'path'}, | |
| 24 | 'account.create': {'name'}, 'account.update': {'id', 'name'}, | |
| 25 | 'account.rotate': {'id'}, 'account.delete': {'id'}, | |
| 26 | 'grant.save': {'id', 'subject', 'path', 'permissions'}, 'grant.delete': {'id'}, | |
| 27 | 'link.create': {'name', 'path', 'permissions', 'expires', 'password', 'createFolder'}, | |
| 28 | 'link.update': {'key', 'name', 'path', 'permissions', 'expires', 'password', 'mode'}, | |
| 29 | 'link.delete': {'key'}, | |
| 30 | } | |
| 31 | ||
| 32 | ||
| 33 | def identifier(value): | |
| 34 | if not isinstance(value, str): | |
| 35 | raise Error(400, 'Choose an account or folder permission from the list.') | |
| 36 | try: | |
| 37 | if str(uuid.UUID(value)) == value: | |
| 38 | return value | |
| 39 | except ValueError: | |
| 40 | pass | |
| 41 | raise Error(400, 'Choose an account or folder permission from the list.') | |
| 42 | ||
| 43 | ||
| 44 | def label(value): | |
| 45 | if (not isinstance(value, str) or not value.strip() or len(value) > 200 | |
| 46 | or any(ord(c) < 32 or c in '/\\' for c in value) | |
| 47 | or value.strip() in {'.', '..'}): | |
| 48 | raise Error(400, 'Enter a name under 200 characters without slashes.') | |
| 49 | return value.strip() | |
| 50 | ||
| 51 | ||
| 52 | def permissions(value, choices): | |
| 53 | if not isinstance(value, str) or not value or set(value) - set(choices): | |
| 54 | raise Error(400, 'Choose at least one supported permission.') | |
| 55 | return ''.join(c for c in choices if c in value) | |
| 56 | ||
| 57 | ||
| 58 | def atomic(path, data, owner=None): | |
| 59 | pending = path.with_name('.' + path.name + '.pending') | |
| 60 | fd = os.open(pending, os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NOFOLLOW, 0o600) | |
| 61 | try: | |
| 62 | os.fchmod(fd, 0o600) | |
| 63 | with os.fdopen(fd, 'wb') as target: | |
| 64 | target.write(data) | |
| 65 | target.flush() | |
| 66 | os.fsync(target.fileno()) | |
| 67 | if owner is not None: | |
| 68 | os.chown(pending, *owner) | |
| 69 | os.replace(pending, path) | |
| 70 | finally: | |
| 71 | pending.unlink(missing_ok=True) | |
| 72 | ||
| 73 | ||
| 74 | def volumes(base): | |
| 75 | sections = re.split(r'(?m)^(?=\[/)', base) | |
| 76 | result = [] | |
| 77 | for section in sections[1:]: | |
| 78 | lines = [line.strip() for line in section.splitlines() if line.strip()] | |
| 79 | match = re.fullmatch(r'\[(/[^\r\n]*)\]', lines[0]) | |
| 80 | if not match or len(lines) < 2: | |
| 81 | raise Error(503, 'Copyparty configuration needs repair before editing shares.') | |
| 82 | entry = {'url': match[1].rstrip('/') or '/', 'path': lines[1], 'accs': [], 'flags': []} | |
| 83 | target = None | |
| 84 | for line in lines[2:]: | |
| 85 | if line == 'accs:': | |
| 86 | target = entry['accs'] | |
| 87 | elif line == 'flags:': | |
| 88 | target = entry['flags'] | |
| 89 | elif target is not None: | |
| 90 | target.append(line) | |
| 91 | else: | |
| 92 | raise Error(503, 'Copyparty configuration needs repair before editing shares.') | |
| 93 | result.append(entry) | |
| 94 | return sections[0], result | |
| 95 | ||
| 96 | ||
| 97 | def render(base, state, hashes): | |
| 98 | global_config, entries = volumes(base) | |
| 99 | accounts = state['accounts'] | |
| 100 | if accounts: | |
| 101 | global_config += '\n[accounts]\n' + ''.join( | |
| 102 | ' api-' + account['id'] + ': ' + hashes['file_api_' + account['id'].replace('-', '')] + '\n' | |
| 103 | for account in accounts | |
| 104 | ) | |
| 105 | api_names = ['api-' + account['id'] for account in accounts] | |
| 106 | base_entries = copy.deepcopy(entries) | |
| 107 | paths = {grant['path'] for grant in state['grants']} | |
| 108 | for entry in base_entries: | |
| 109 | if entry['url'] == '/clover' or entry['url'].startswith('/clover/'): | |
| 110 | host = str(Path(state['root']) / entry['url'][len('/clover'):].lstrip('/')) | |
| 111 | if any(host == grant['path'] or host.startswith(grant['path'].rstrip('/') + '/') for grant in state['grants']): | |
| 112 | paths.add(host) | |
| 113 | for path in sorted(paths): | |
| 114 | relative = Path(path).relative_to(state['root']) | |
| 115 | url = '/clover' + ('/' + relative.as_posix() if relative.parts else '') | |
| 116 | parent = max((entry for entry in base_entries if url == entry['url'] or url.startswith(entry['url'].rstrip('/') + '/')), | |
| 117 | key=lambda entry: len(entry['url'])) | |
| 118 | exact = next((entry for entry in entries if entry['url'] == url), None) | |
| 119 | entry = exact if exact is not None else { | |
| 120 | 'url': url, 'path': '/w/clover' + ('/' + relative.as_posix() if relative.parts else ''), | |
| 121 | 'accs': list(parent['accs']), 'flags': list(parent['flags']), | |
| 122 | } | |
| 123 | for grant in state['grants']: | |
| 124 | if path == grant['path'] or path.startswith(grant['path'].rstrip('/') + '/'): | |
| 125 | kind, subject = grant['subject'].split(':', 1) | |
| 126 | principal = '@snow-user-' + subject if kind == 'user' else 'api-' + subject | |
| 127 | entry['accs'].append(grant['permissions'] + ': ' + principal) | |
| 128 | if 'xvol' not in entry['flags']: | |
| 129 | entry['flags'].append('xvol') | |
| 130 | if exact is None: | |
| 131 | entries.append(entry) | |
| 132 | rendered = global_config | |
| 133 | for entry in entries: | |
| 134 | allowed = {} | |
| 135 | if entry['url'] == '/clover' or entry['url'].startswith('/clover/'): | |
| 136 | host = str(Path(state['root']) / entry['url'][len('/clover'):].lstrip('/')) | |
| 137 | for grant in state['grants']: | |
| 138 | if grant['subject'].startswith('api:') and (host == grant['path'] or host.startswith(grant['path'].rstrip('/') + '/')): | |
| 139 | for perm in grant['permissions']: | |
| 140 | allowed.setdefault(perm, set()).add('api-' + grant['subject'][4:]) | |
| 141 | access = [] | |
| 142 | for line in entry['accs']: | |
| 143 | perms, names = line.split(':', 1) | |
| 144 | names = [name.strip() for name in names.split(',')] | |
| 145 | if api_names and ('*' in names or '@acct' in names): | |
| 146 | perms = perms.replace('A', 'rwmda.') | |
| 147 | for perm in dict.fromkeys(perms): | |
| 148 | denied = [name for name in api_names if name not in allowed.get(perm, set())] | |
| 149 | access.append(perm + ': ' + ','.join(names + ['-' + name for name in denied])) | |
| 150 | else: | |
| 151 | access.append(line) | |
| 152 | entry['accs'] = access | |
| 153 | rendered += '\n[' + entry['url'] + ']\n ' + entry['path'] + '\n accs:\n' | |
| 154 | rendered += ''.join(' ' + line + '\n' for line in entry['accs']) | |
| 155 | if entry['flags']: | |
| 156 | rendered += ' flags:\n' + ''.join(' ' + line + '\n' for line in entry['flags']) | |
| 157 | return rendered | |
| 158 | ||
| 159 | ||
| 160 | def configuration(cfg, state, items, source=None): | |
| 161 | source = source or Path(__file__).with_name('copyparty.conf') | |
| 162 | base = source.read_text() | |
| 163 | proof = items.get('idp_header', '') | |
| 164 | if not re.fullmatch(r'[0-9a-f]{32}', proof): | |
| 165 | raise Error(503, 'Copyparty sign-in configuration is unavailable. Redeploy the service.') | |
| 166 | capacity = os.statvfs(cfg) | |
| 167 | reserve = max(1, min(16, capacity.f_blocks * capacity.f_frsize // (4 * 10**9))) | |
| 168 | base = base.replace('STUDIO_IDP_HEADER', 'X-Studio-Idp-' + proof).replace('STUDIO_MIN_FREE_GB', str(reserve)) | |
| 169 | return render(base, state, items).encode() | |
| 170 | ||
| 171 | ||
| 172 | def remaining(deadline, maximum=10): | |
| 173 | timeout = min(maximum, deadline - time.monotonic()) | |
| 174 | if timeout <= 0: | |
| 175 | raise Error(504, 'Copyparty took too long to respond. Check the service before retrying.') | |
| 176 | return timeout | |
| 177 | ||
| 178 | ||
| 179 | def rename(source, destination): | |
| 180 | # RENAME_NOREPLACE keeps a concurrently created destination from being overwritten. | |
| 181 | libc = ctypes.CDLL(None, use_errno=True) | |
| 182 | if hasattr(libc, 'renameat2'): | |
| 183 | result = libc.renameat2(-100, os.fsencode(source), -100, os.fsencode(destination), 1) | |
| 184 | elif hasattr(libc, 'renamex_np'): | |
| 185 | result = libc.renamex_np(os.fsencode(source), os.fsencode(destination), 4) | |
| 186 | else: | |
| 187 | raise Error(503, 'This host cannot safely move folders. Retarget the share instead.') | |
| 188 | if result: | |
| 189 | code = ctypes.get_errno() | |
| 190 | if code == errno.EEXIST: | |
| 191 | raise Error(409, 'The destination already exists. Choose a new folder name.') | |
| 192 | if code == errno.EXDEV: | |
| 193 | raise Error(409, 'Choose a destination on the same filesystem to move this folder.') | |
| 194 | raise OSError(code, os.strerror(code)) | |
| 195 | ||
| 196 | ||
| 197 | class Manager: | |
| 198 | def __init__(self, users): | |
| 199 | root = Path(os.environ.get('STUDIO_ROOT', '/srv')) | |
| 200 | self.root = Path(os.environ.get('STUDIO_CLOVER_ROOT', str(root / 'clover'))).resolve() | |
| 201 | self.cfg = Path(os.environ.get('STUDIO_COPYPARTY_CFG', str(root / 'prod/copyparty/cfg'))) | |
| 202 | self.native = self.cfg / 'copyparty' | |
| 203 | self.service = os.environ.get('STUDIO_COPYPARTY_SERVICE', 'copyparty') | |
| 204 | self.users = users | |
| 205 | self.path = self.cfg / 'manager.json' | |
| 206 | self.changed = set() | |
| 207 | self.undo = [] | |
| 208 | self.secret_before = {} | |
| 209 | self.container_id = None | |
| 210 | self.db = None | |
| 211 | self.job = None | |
| 212 | self.stopped = False | |
| 213 | self.offline_at = None | |
| 214 | self.deadline = None | |
| 215 | ||
| 216 | def path_for(self, value, missing=False): | |
| 217 | if not isinstance(value, str) or len(value) > 4096 or any(ord(c) < 32 or c in '#[]\\' for c in value) or ' ' in value: | |
| 218 | raise Error(400, 'Choose a folder without control characters, brackets, or repeated spaces.') | |
| 219 | if value == '/Volumes/clover' or value.startswith('/Volumes/clover/'): | |
| 220 | value = str(self.root) + value[len('/Volumes/clover'):] | |
| 221 | path = Path(value) | |
| 222 | if not path.is_absolute() or '..' in path.parts or not path.is_relative_to(self.root): | |
| 223 | raise Error(400, 'Choose a folder inside the Clover file store.') | |
| 224 | current = self.root | |
| 225 | try: | |
| 226 | for part in path.relative_to(self.root).parts: | |
| 227 | current /= part | |
| 228 | if current.is_symlink(): | |
| 229 | raise Error(400, 'Choose a folder directly instead of a symbolic link.') | |
| 230 | if not missing and not path.is_dir(): | |
| 231 | raise Error(404, 'That folder no longer exists. Choose another folder.') | |
| 232 | if missing and not path.parent.is_dir(): | |
| 233 | raise Error(404, 'The parent folder does not exist. Choose an existing parent.') | |
| 234 | if path.exists() and not path.is_dir(): | |
| 235 | raise Error(400, 'Choose a folder instead of a file.') | |
| 236 | except OSError: | |
| 237 | raise Error(400, 'That folder cannot be opened. Choose another folder.') from None | |
| 238 | return path | |
| 239 | ||
| 240 | def container(self): | |
| 241 | if self.container_id: | |
| 242 | return self.container_id | |
| 243 | result = subprocess.run([*PODMAN, 'ps', '--format', '{{.ID}} {{.Image}}'], capture_output=True, text=True, timeout=10, check=True) | |
| 244 | for row in result.stdout.splitlines(): | |
| 245 | fields = row.split() | |
| 246 | if len(fields) != 2 or '/copyparty/' not in fields[1]: | |
| 247 | continue | |
| 248 | result = subprocess.run([*PODMAN, 'inspect', '--format', '{{json .Mounts}}', fields[0]], | |
| 249 | capture_output=True, text=True, timeout=10, check=True) | |
| 250 | mounts = {mount['Destination']: mount['Source'] for mount in json.loads(result.stdout)} | |
| 251 | if mounts.get('/cfg') == str(self.cfg): | |
| 252 | if mounts.get('/w/clover') != str(self.root): | |
| 253 | raise Error(503, 'The share manager points at a different file store. Check its host configuration.') | |
| 254 | self.container_id = fields[0] | |
| 255 | return self.container_id | |
| 256 | raise Error(503, 'Copyparty is offline. Start it before editing shares.') | |
| 257 | ||
| 258 | def hash(self, password): | |
| 259 | if not isinstance(password, str) or len(password) > 255 or any(c in password for c in '\r\n\0'): | |
| 260 | raise Error(400, 'Use a single-line password under 256 characters.') | |
| 261 | if not password: | |
| 262 | return '' | |
| 263 | script = ('import argparse,sys; from pathlib import Path; from copyparty.pwhash import PWHash; ' | |
| 264 | 'args=argparse.Namespace(ah_alg="argon2",ah_salt=Path("/cfg/copyparty/ah-salt.txt").read_text().strip()); ' | |
| 265 | 'print(PWHash(args).hash(sys.stdin.read()))') | |
| 266 | result = subprocess.run([*PODMAN, 'exec', '-i', self.container(), 'python3', '-c', script], | |
| 267 | input=password, capture_output=True, text=True, timeout=25) | |
| 268 | value = result.stdout.strip() | |
| 269 | if result.returncode or not re.fullmatch(r'\+[A-Za-z0-9_-]{32}', value): | |
| 270 | raise Error(502, 'The password could not be secured. Check Copyparty and try again.') | |
| 271 | return value | |
| 272 | ||
| 273 | def reload(self, items, timeout=30): | |
| 274 | deadline = time.monotonic() + timeout | |
| 275 | instances = runs.nomad('service/' + self.service, timeout=remaining(deadline)) or [] | |
| 276 | if len(instances) != 1: | |
| 277 | raise Error(503, 'Copyparty is offline or changing allocations. Try again shortly.') | |
| 278 | address, port = instances[0]['Address'], instances[0]['Port'] | |
| 279 | if not re.fullmatch(r'[0-9a-fA-F:.]+', address) or type(port) is not int or not 1 <= port <= 65535: | |
| 280 | raise Error(503, 'Copyparty address is unavailable. Redeploy its service.') | |
| 281 | address = '[' + address + ']' if ':' in address else address | |
| 282 | request = urllib.request.Request('http://' + address + ':' + str(port) + '/?reload=cfg', headers={ | |
| 283 | 'User-Name': 'snow', 'User-Groups': '', 'X-Studio-Idp-' + items['idp_header']: '1', | |
| 284 | }) | |
| 285 | try: | |
| 286 | with urllib.request.urlopen(request, timeout=remaining(deadline, 30)) as response: | |
| 287 | body = response.read(262144) | |
| 288 | if response.status != 200 or b'config has been reloaded' not in body: | |
| 289 | raise Error(502, 'Copyparty did not confirm the update. Try again after checking the service.') | |
| 290 | except (urllib.error.URLError, TimeoutError): | |
| 291 | raise Error(502, 'Copyparty could not apply the update. Check the service and try again.') from None | |
| 292 | ||
| 293 | def stop(self): | |
| 294 | if self.offline_at is None: | |
| 295 | self.offline_at = time.monotonic() | |
| 296 | if self.deadline is None: | |
| 297 | self.deadline = self.offline_at + 160 | |
| 298 | deadline = min(self.deadline, time.monotonic() + 40) | |
| 299 | if self.job is None: | |
| 300 | self.job = runs.nomad('job/' + self.service, timeout=remaining(deadline)) | |
| 301 | if self.job is None: | |
| 302 | raise Error(503, 'Copyparty is offline. Start it before moving folders.') | |
| 303 | self.job['Stop'] = False | |
| 304 | runs.nomad('job/' + self.service + '?purge=false', 'DELETE', timeout=remaining(deadline)) | |
| 305 | self.stopped = True | |
| 306 | while time.monotonic() < deadline: | |
| 307 | allocations = runs.nomad('job/' + self.service + '/allocations', timeout=remaining(deadline)) or [] | |
| 308 | if all(entry.get('ClientStatus') in {'complete', 'failed', 'lost'} for entry in allocations): | |
| 309 | self.container_id = None | |
| 310 | return | |
| 311 | time.sleep(0.5) | |
| 312 | raise Error(502, 'Copyparty did not stop in time. Check the service before retrying the move.') | |
| 313 | ||
| 314 | def start(self): | |
| 315 | if self.job is None: | |
| 316 | raise Error(503, 'Copyparty restart settings are unavailable. Redeploy its service.') | |
| 317 | deadline = min(self.deadline, time.monotonic() + 70) | |
| 318 | runs.nomad('jobs', 'POST', {'Job': self.job}, timeout=remaining(deadline)) | |
| 319 | items = getattr(self, 'items', None) | |
| 320 | if items is None: | |
| 321 | variable = runs.nomad('var/nomad/jobs/' + self.service, timeout=remaining(deadline)) | |
| 322 | if variable is None: | |
| 323 | raise Error(503, 'Copyparty secrets are unavailable. Redeploy its service.') | |
| 324 | items = variable['Items'] | |
| 325 | while time.monotonic() < deadline: | |
| 326 | try: | |
| 327 | self.reload(items, timeout=remaining(deadline, 30)) | |
| 328 | self.stopped = False | |
| 329 | return | |
| 330 | except Error as error: | |
| 331 | if error.status not in {502, 503}: | |
| 332 | raise | |
| 333 | time.sleep(0.5) | |
| 334 | raise Error(502, 'Copyparty could not restart in time. Check its service logs before retrying.') | |
| 335 | ||
| 336 | def virtual(self, path): | |
| 337 | return 'clover' + ('/' + path.relative_to(self.root).as_posix() if path != self.root else '') | |
| 338 | ||
| 339 | def real(self, virtual): | |
| 340 | if virtual == 'clover' or virtual.startswith('clover/'): | |
| 341 | return self.root / virtual[len('clover'):].lstrip('/') | |
| 342 | if virtual == 'media' or virtual.startswith('media/'): | |
| 343 | media = Path(os.environ.get('STUDIO_MEDIA_ROOT', str(self.root / 'Media'))) | |
| 344 | return media / virtual[len('media'):].lstrip('/') | |
| 345 | return None | |
| 346 | ||
| 347 | def state(self): | |
| 348 | links = [] | |
| 349 | for key, password, path, perms, count, owner, created, expires in self.db.execute('SELECT * FROM sh ORDER BY t0 DESC,k'): | |
| 350 | real = self.real(path) | |
| 351 | links.append({'key': key, 'name': self.data['names'].get(key, real.name if real is not None else Path(path).name or key), | |
| 352 | 'path': str(real) if real is not None else '/' + path, | |
| 353 | 'permissions': perms, 'owner': owner, 'created': created, 'expires': expires, | |
| 354 | 'hasPassword': bool(password), 'files': [row[0] for row in self.db.execute('SELECT vp FROM sf WHERE k=? ORDER BY vp', (key,))] if count else []}) | |
| 355 | return {'root': str(self.root), 'defaultRoot': str(self.root / 'Share'), | |
| 356 | 'accounts': [{**account, 'subject': 'api:' + account['id'], 'username': 'api-' + account['id']} for account in self.data['accounts']], | |
| 357 | 'grants': self.data['grants'], 'links': links} | |
| 358 | ||
| 359 | def secret(self, key, value): | |
| 360 | before = self.items.get(key) | |
| 361 | if value is None: | |
| 362 | self.items.pop(key, None) | |
| 363 | else: | |
| 364 | self.items[key] = value | |
| 365 | self.secret_before[key] = before | |
| 366 | ||
| 367 | def account(self, value): | |
| 368 | value = identifier(value) | |
| 369 | account = next((entry for entry in self.data['accounts'] if entry['id'] == value), None) | |
| 370 | if account is None: | |
| 371 | raise Error(404, 'That API account no longer exists. Reload accounts.') | |
| 372 | return account | |
| 373 | ||
| 374 | def grant(self, payload): | |
| 375 | subject = payload['subject'] | |
| 376 | if not isinstance(subject, str) or ':' not in subject: | |
| 377 | raise Error(400, 'Choose a Snow Sign On user or API account.') | |
| 378 | kind, value = subject.split(':', 1) | |
| 379 | identifier(value) | |
| 380 | if kind == 'api': | |
| 381 | self.account(value) | |
| 382 | elif kind == 'user': | |
| 383 | if not any(user['id'] == value and user.get('enabled') is True and user.get('kind') in {None, 'user'} for user in self.users): | |
| 384 | raise Error(400, 'Choose an enabled Snow Sign On user.') | |
| 385 | else: | |
| 386 | raise Error(400, 'Choose a Snow Sign On user or API account.') | |
| 387 | path = str(self.path_for(payload['path'])) | |
| 388 | perms = permissions(payload['permissions'], 'rwmda.g') | |
| 389 | entry = {'id': identifier(payload['id']) if payload['id'] is not None else str(uuid.uuid4()), | |
| 390 | 'subject': subject, 'path': path, 'permissions': perms} | |
| 391 | found = next((grant for grant in self.data['grants'] if grant['id'] == entry['id']), None) | |
| 392 | if payload['id'] is not None and found is None: | |
| 393 | raise Error(404, 'That folder permission no longer exists. Reload permissions.') | |
| 394 | if any(grant['id'] != entry['id'] and grant['subject'] == subject and grant['path'] == path for grant in self.data['grants']): | |
| 395 | raise Error(409, 'This account already has permissions for that folder. Edit its existing permission.') | |
| 396 | self.data['grants'] = [grant for grant in self.data['grants'] if grant['id'] != entry['id']] + [entry] | |
| 397 | ||
| 398 | def owner_permissions(self, owner, path, required): | |
| 399 | _, entries = volumes(configuration(self.cfg, self.data, self.items).decode()) | |
| 400 | url = '/' + self.virtual(path) | |
| 401 | entry = max((entry for entry in entries if url == entry['url'] or url.startswith(entry['url'].rstrip('/') + '/')), | |
| 402 | key=lambda entry: len(entry['url'])) | |
| 403 | groups = {'acct'} | |
| 404 | user = next((user for user in self.users if user.get('username') == owner), None) | |
| 405 | if user is not None: | |
| 406 | if user.get('enabled') is not True or user.get('kind') not in {None, 'user'} or user.get('requiredActions'): | |
| 407 | raise Error(409, 'The share owner cannot sign in. Enable their account before editing this share.') | |
| 408 | groups.update(user.get('groups', [])) | |
| 409 | groups.add('snow-user-' + identifier(user['id'])) | |
| 410 | effective = set() | |
| 411 | excluded_permissions = set() | |
| 412 | for rule in entry['accs']: | |
| 413 | perms, principals = rule.split(':', 1) | |
| 414 | names = {name.strip() for name in principals.split(',')} | |
| 415 | selected = owner in names or '*' in names or any('@' + group in names for group in groups) | |
| 416 | excluded = '-' + owner in names or any('-@' + group in names for group in groups) | |
| 417 | expanded = perms.replace('A', 'rwmda.') | |
| 418 | if selected: | |
| 419 | effective.update(expanded) | |
| 420 | if excluded: | |
| 421 | excluded_permissions.update(expanded) | |
| 422 | if set(required) - (effective - excluded_permissions): | |
| 423 | raise Error(409, 'The share owner needs these permissions at the destination. Grant them access before changing this share.') | |
| 424 | ||
| 425 | def link(self, action, payload): | |
| 426 | key = payload['key'] if action == 'link.update' else secrets.token_urlsafe(12) | |
| 427 | if not isinstance(key, str) or not re.fullmatch(r'[A-Za-z0-9_-]{1,200}', key): | |
| 428 | raise Error(400, 'Choose a share from the list.') | |
| 429 | old = self.db.execute('SELECT * FROM sh WHERE k=?', (key,)).fetchone() | |
| 430 | if action == 'link.update' and old is None: | |
| 431 | raise Error(404, 'That share no longer exists. Reload shares.') | |
| 432 | name = label(payload['name']) | |
| 433 | perms = permissions(payload['permissions'], 'rwmd') | |
| 434 | expires = payload['expires'] | |
| 435 | if type(expires) is not int or expires < 0 or expires > 2**53: | |
| 436 | raise Error(400, 'Choose an expiration date or keep the share indefinitely.') | |
| 437 | password = payload['password'] | |
| 438 | if password is not None and not isinstance(password, str): | |
| 439 | raise Error(400, 'Enter a password or leave the existing password unchanged.') | |
| 440 | if password is None: | |
| 441 | password = old[1] if old is not None else '' | |
| 442 | else: | |
| 443 | password = self.hash(password) | |
| 444 | create = action == 'link.create' and payload['createFolder'] is True | |
| 445 | if action == 'link.create' and type(payload['createFolder']) is not bool: | |
| 446 | raise Error(400, 'Choose an existing folder or create a new one.') | |
| 447 | destination = payload['path'] | |
| 448 | share_root = self.root / 'Share' | |
| 449 | if create and (not destination or Path(destination).parent == share_root or str(destination).startswith('/Volumes/clover/Share/')): | |
| 450 | self.path_for(str(share_root), missing=True) | |
| 451 | if not share_root.exists(): | |
| 452 | share_root.mkdir(mode=0o775) | |
| 453 | share_root.chmod(0o775) | |
| 454 | owner = self.root.stat() | |
| 455 | os.chown(share_root, owner.st_uid, owner.st_gid) | |
| 456 | self.undo.append(lambda: share_root.rmdir()) | |
| 457 | if not destination: | |
| 458 | destination = str(share_root / name) | |
| 459 | destination = self.path_for(destination, missing=create or action == 'link.update' and payload['mode'] == 'move') | |
| 460 | files = [row[0] for row in self.db.execute('SELECT vp FROM sf WHERE k=?', (key,))] if old is not None and old[4] else [] | |
| 461 | mode = payload.get('mode', 'retarget') | |
| 462 | if mode not in {'retarget', 'move'}: | |
| 463 | raise Error(400, 'Choose whether to move the folder or use another folder.') | |
| 464 | if create: | |
| 465 | try: | |
| 466 | destination.mkdir(mode=0o775) | |
| 467 | destination.chmod(0o775) | |
| 468 | except FileExistsError: | |
| 469 | raise Error(409, 'That folder already exists. Choose it as an existing folder or change its name.') from None | |
| 470 | owner = destination.parent.stat() | |
| 471 | os.chown(destination, owner.st_uid, owner.st_gid) | |
| 472 | self.undo.append(lambda: destination.rmdir()) | |
| 473 | if old is not None and mode == 'move': | |
| 474 | source = self.real(old[2]) | |
| 475 | if source is None: | |
| 476 | raise Error(400, 'Only folders inside the Clover file store can be moved here.') | |
| 477 | source = self.path_for(str(source)) | |
| 478 | if source != destination: | |
| 479 | for grant in self.data['grants']: | |
| 480 | if Path(grant['path']).is_relative_to(source): | |
| 481 | grant['path'] = str(destination / Path(grant['path']).relative_to(source)) | |
| 482 | self.owner_permissions(old[5], destination, perms) | |
| 483 | self.move(source, destination) | |
| 484 | for filename in files: | |
| 485 | if (not isinstance(filename, str) or filename in {'.', '..'} or '/' in filename or '\\' in filename | |
| 486 | or not (destination / filename).is_file() or (destination / filename).is_symlink()): | |
| 487 | raise Error(409, 'The destination must contain every selected file. Choose a matching folder.') | |
| 488 | if old is not None: | |
| 489 | self.owner_permissions(old[5], destination, perms) | |
| 490 | self.changed.add(key) | |
| 491 | self.db.execute('UPDATE sh SET pw=?,vp=?,pr=?,t1=? WHERE k=?', (password, self.virtual(destination), perms, expires, key)) | |
| 492 | else: | |
| 493 | self.changed.add(key) | |
| 494 | self.db.execute('INSERT INTO sh VALUES(?,?,?,?,?,?,?,?)', (key, password, self.virtual(destination), perms, 0, 'snow', int(time.time()), expires)) | |
| 495 | self.data['names'][key] = name | |
| 496 | ||
| 497 | def move(self, source, destination): | |
| 498 | if source == self.root or source == self.root / 'Share' or destination.is_relative_to(source) or source.is_relative_to(destination): | |
| 499 | raise Error(400, 'Choose separate folders below the Clover file store.') | |
| 500 | if destination.exists(): | |
| 501 | raise Error(409, 'The destination already exists. Choose a new folder name.') | |
| 502 | if source.stat().st_dev != destination.parent.stat().st_dev: | |
| 503 | raise Error(409, 'Choose a destination on the same filesystem to move this folder.') | |
| 504 | mounts = Path('/proc/self/mountinfo') | |
| 505 | if mounts.exists(): | |
| 506 | for row in mounts.read_text().splitlines(): | |
| 507 | mount = Path(re.sub(r'\\([0-7]{3})', lambda match: chr(int(match[1], 8)), row.split()[4])) | |
| 508 | if mount.is_relative_to(source): | |
| 509 | raise Error(409, 'A mounted filesystem cannot be moved. Choose a folder without mounts inside it.') | |
| 510 | elif os.path.ismount(source): | |
| 511 | raise Error(409, 'A mounted filesystem cannot be moved. Choose a folder inside it.') | |
| 512 | rows = self.db.execute('SELECT k,vp,st FROM sh').fetchall() | |
| 513 | for key, virtual, count in rows: | |
| 514 | real = self.real(virtual) | |
| 515 | if real is not None and source != real and source.is_relative_to(real): | |
| 516 | selected = {row[0] for row in self.db.execute('SELECT vp FROM sf WHERE k=?', (key,))} if count else None | |
| 517 | if selected is None or source.relative_to(real).parts[0] in selected: | |
| 518 | raise Error(409, 'Another share includes this folder through its parent. Retarget that share before moving the folder.') | |
| 519 | self.stop() | |
| 520 | for key, virtual, _ in rows: | |
| 521 | real = self.real(virtual) | |
| 522 | if real is not None and real.is_relative_to(source): | |
| 523 | self.changed.add(key) | |
| 524 | self.db.execute('UPDATE sh SET vp=? WHERE k=?', (self.virtual(destination / real.relative_to(source)), key)) | |
| 525 | for grant in self.data['grants']: | |
| 526 | if Path(grant['path']).is_relative_to(source): | |
| 527 | grant['path'] = str(destination / Path(grant['path']).relative_to(source)) | |
| 528 | rename(source, destination) | |
| 529 | self.undo.append(lambda: rename(destination, source)) | |
| 530 | ||
| 531 | def dispatch(self, action, payload): | |
| 532 | extra = {} | |
| 533 | if action.startswith('account.'): | |
| 534 | account = {'id': str(uuid.uuid4()), 'name': label(payload['name'])} if action == 'account.create' else self.account(payload['id']) | |
| 535 | value = account['id'] | |
| 536 | if action == 'account.create': | |
| 537 | if any(user.get('username') == 'api-' + value for user in self.users): | |
| 538 | raise Error(409, 'That API username already belongs to a Snow Sign On user. Try creating the account again.') | |
| 539 | self.data['accounts'].append(account) | |
| 540 | if action in {'account.update', 'account.create'}: | |
| 541 | account['name'] = label(payload['name']) | |
| 542 | if action in {'account.create', 'account.rotate'}: | |
| 543 | password = secrets.token_urlsafe(32) | |
| 544 | self.secret('file_api_' + value.replace('-', ''), self.hash(password)) | |
| 545 | extra = {'account': {**account, 'subject': 'api:' + value, 'username': 'api-' + value}, 'password': password} | |
| 546 | if action in {'account.delete', 'account.rotate'}: | |
| 547 | session_path = self.native / 'sessions.db' | |
| 548 | if session_path.exists(): | |
| 549 | with sqlite3.connect(session_path) as sessions: | |
| 550 | sessions.execute('DELETE FROM us WHERE un=?', ('api-' + value,)) | |
| 551 | if action == 'account.delete': | |
| 552 | self.data['accounts'].remove(account) | |
| 553 | self.data['grants'] = [grant for grant in self.data['grants'] if grant['subject'] != 'api:' + value] | |
| 554 | self.secret('file_api_' + value.replace('-', ''), None) | |
| 555 | elif action == 'grant.save': | |
| 556 | self.grant(payload) | |
| 557 | elif action == 'grant.delete': | |
| 558 | value = identifier(payload['id']) | |
| 559 | if not any(grant['id'] == value for grant in self.data['grants']): | |
| 560 | raise Error(404, 'That folder permission no longer exists. Reload permissions.') | |
| 561 | self.data['grants'] = [grant for grant in self.data['grants'] if grant['id'] != value] | |
| 562 | elif action in {'link.create', 'link.update'}: | |
| 563 | self.link(action, payload) | |
| 564 | elif action == 'link.delete': | |
| 565 | key = payload['key'] | |
| 566 | if not isinstance(key, str) or not self.db.execute('SELECT 1 FROM sh WHERE k=?', (key,)).fetchone(): | |
| 567 | raise Error(404, 'That share no longer exists. Reload shares.') | |
| 568 | self.changed.add(key) | |
| 569 | self.db.execute('DELETE FROM sh WHERE k=?', (key,)) | |
| 570 | self.db.execute('DELETE FROM sf WHERE k=?', (key,)) | |
| 571 | self.data['names'].pop(key, None) | |
| 572 | return extra | |
| 573 | ||
| 574 | def apply(self, action, payload): | |
| 575 | self.data = json.loads(runs.read(self.path, missing='{"accounts":[],"grants":[],"names":{}}')) | |
| 576 | origin = Path(self.data.get('root', os.environ.get('STUDIO_SOURCE_CLOVER_ROOT', '/srv/clover'))) | |
| 577 | if 'root' not in self.data and self.data.get('grants'): | |
| 578 | origin = Path(os.environ.get('STUDIO_SOURCE_CLOVER_ROOT', str(self.root))) | |
| 579 | if origin != self.root: | |
| 580 | for grant in self.data['grants']: | |
| 581 | grant['path'] = str(self.root / Path(grant['path']).relative_to(origin)) | |
| 582 | self.data['root'] = str(self.root) | |
| 583 | if not isinstance(self.data.get('accounts'), list) or not isinstance(self.data.get('grants'), list) or not isinstance(self.data.get('names'), dict): | |
| 584 | raise Error(503, 'Share settings need repair before editing. Check the host service.') | |
| 585 | database = self.native / 'shares.db' | |
| 586 | if not database.is_file(): | |
| 587 | raise Error(503, 'Copyparty shares are not ready. Start Copyparty and try again.') | |
| 588 | self.db = sqlite3.connect(database.as_uri() + '?mode=rw', uri=True, timeout=5) | |
| 589 | self.db.row_factory = None | |
| 590 | try: | |
| 591 | version = self.db.execute("SELECT v FROM kv WHERE k='sver'").fetchone() | |
| 592 | if version != (2,) or [row[1] for row in self.db.execute('PRAGMA table_info(sh)')] != ['k', 'pw', 'vp', 'pr', 'st', 'un', 't0', 't1']: | |
| 593 | raise Error(503, 'This Copyparty share format is unsupported. Update Snowglobe before editing shares.') | |
| 594 | if action == 'state': | |
| 595 | return self.state() | |
| 596 | if action == 'folders.list': | |
| 597 | path = self.path_for(payload['path']) | |
| 598 | with os.scandir(path) as entries: | |
| 599 | directories = sorted(({'name': entry.name, 'path': str(path / entry.name)} for entry in entries | |
| 600 | if not entry.name.startswith('.') and entry.is_dir(follow_symlinks=False)), key=lambda entry: entry['name'].casefold()) | |
| 601 | return {'path': str(path), 'directories': directories} | |
| 602 | self.container() | |
| 603 | variable_path = 'nomad/jobs/' + self.service | |
| 604 | variable = runs.nomad('var/' + variable_path) | |
| 605 | if variable is None: | |
| 606 | raise Error(503, 'Copyparty secrets are unavailable. Redeploy its service.') | |
| 607 | self.items = dict(variable['Items']) | |
| 608 | original_items = dict(self.items) | |
| 609 | config_path = self.cfg / 'copyparty.conf' | |
| 610 | old_config = config_path.read_bytes() | |
| 611 | metadata = config_path.stat() | |
| 612 | owner = (metadata.st_uid, metadata.st_gid) | |
| 613 | self.db.execute('BEGIN IMMEDIATE') | |
| 614 | original_rows = {row[0]: row for row in self.db.execute('SELECT * FROM sh')} | |
| 615 | original_files = {key: self.db.execute('SELECT * FROM sf WHERE k=?', (key,)).fetchall() for key in original_rows} | |
| 616 | committed = False | |
| 617 | wrote_files = False | |
| 618 | old_manager = self.path.read_bytes() if self.path.exists() else None | |
| 619 | changed_secrets = False | |
| 620 | after_rows = {} | |
| 621 | try: | |
| 622 | extra = self.dispatch(action, payload) | |
| 623 | rendered = configuration(self.cfg, self.data, self.items) | |
| 624 | if self.secret_before: | |
| 625 | runs.nomad('var/' + variable_path + '?cas=' + str(variable['ModifyIndex']), 'PUT', | |
| 626 | {'Namespace': 'default', 'Path': variable_path, 'Items': self.items}) | |
| 627 | changed_secrets = True | |
| 628 | atomic(config_path, rendered, owner) | |
| 629 | wrote_files = True | |
| 630 | atomic(self.path, json.dumps(self.data, ensure_ascii=False).encode()) | |
| 631 | after_rows = {key: self.db.execute('SELECT * FROM sh WHERE k=?', (key,)).fetchone() for key in self.changed} | |
| 632 | self.db.commit() | |
| 633 | committed = True | |
| 634 | if self.job is not None: | |
| 635 | self.start() | |
| 636 | else: | |
| 637 | self.reload(self.items) | |
| 638 | return {**self.state(), **extra} | |
| 639 | except Exception: | |
| 640 | if self.job is not None: | |
| 641 | self.deadline = self.offline_at + 280 | |
| 642 | self.stop() | |
| 643 | self.db.rollback() | |
| 644 | if committed and self.changed: | |
| 645 | self.db.execute('BEGIN IMMEDIATE') | |
| 646 | for key in self.changed: | |
| 647 | current = self.db.execute('SELECT * FROM sh WHERE k=?', (key,)).fetchone() | |
| 648 | if current != after_rows[key]: | |
| 649 | raise Error(502, 'The update could not finish and a share changed during recovery. Check Copyparty before retrying.') from None | |
| 650 | self.db.execute('DELETE FROM sh WHERE k=?', (key,)) | |
| 651 | self.db.execute('DELETE FROM sf WHERE k=?', (key,)) | |
| 652 | if key in original_rows: | |
| 653 | self.db.execute('INSERT INTO sh VALUES(?,?,?,?,?,?,?,?)', original_rows[key]) | |
| 654 | self.db.executemany('INSERT INTO sf VALUES(?,?)', original_files[key]) | |
| 655 | self.db.commit() | |
| 656 | for undo in reversed(self.undo): | |
| 657 | undo() | |
| 658 | if wrote_files: | |
| 659 | atomic(config_path, old_config, owner) | |
| 660 | if old_manager is None: | |
| 661 | self.path.unlink(missing_ok=True) | |
| 662 | else: | |
| 663 | atomic(self.path, old_manager) | |
| 664 | if changed_secrets: | |
| 665 | latest = runs.nomad('var/' + variable_path) | |
| 666 | items = dict(latest['Items']) | |
| 667 | for key, value in self.secret_before.items(): | |
| 668 | if items.get(key) != self.items.get(key): | |
| 669 | raise Error(502, 'The API credential changed during recovery. Check Copyparty before retrying.') from None | |
| 670 | if value is None: | |
| 671 | items.pop(key, None) | |
| 672 | else: | |
| 673 | items[key] = value | |
| 674 | runs.nomad('var/' + variable_path + '?cas=' + str(latest['ModifyIndex']), 'PUT', | |
| 675 | {'Namespace': 'default', 'Path': variable_path, 'Items': items}) | |
| 676 | if self.job is not None: | |
| 677 | self.items = original_items | |
| 678 | self.start() | |
| 679 | elif committed: | |
| 680 | self.reload(original_items) | |
| 681 | raise | |
| 682 | finally: | |
| 683 | self.db.close() | |
| 684 | ||
| 685 | ||
| 686 | def handle(request): | |
| 687 | action, payload, users = request.get('action'), request.get('payload'), request.get('users') | |
| 688 | if (not isinstance(action, str) or action not in ACTIONS or not isinstance(payload, dict) | |
| 689 | or set(payload) != ACTIONS[action] or not isinstance(users, list)): | |
| 690 | raise Error(400, 'Use the fields required by this share action.') | |
| 691 | manager = Manager(users) | |
| 692 | if not manager.cfg.is_dir(): | |
| 693 | raise Error(503, 'Copyparty settings are unavailable. Deploy its service before editing shares.') | |
| 694 | lock = os.open(manager.cfg / '.manager.lock', os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW, 0o600) | |
| 695 | with os.fdopen(lock, 'w') as locked: | |
| 696 | fcntl.flock(locked, fcntl.LOCK_EX) | |
| 697 | return manager.apply(action, payload) |
service/copyparty/prepare.py+35-25| ... | ... | @@ -2,35 +2,45 @@ |
| 2 | 2 | import json |
| 3 | 3 | import os |
| 4 | 4 | from pathlib import Path |
| 5 | import re | |
| 6 | 5 | import subprocess |
| 7 | 6 | import sys |
| 8 | 7 | |
| 8 | root = Path(__file__).resolve().parents[2] | |
| 9 | sys.path.insert(0, str(root / 'tools')) | |
| 10 | import manage as copyparty | |
| 11 | ||
| 9 | 12 | |
| 10 | 13 | data = json.load(sys.stdin) |
| 11 | root = Path(data["hostRoot"]) / "cfg" | |
| 12 | service = Path(data["hostRoot"]).name | |
| 13 | secret = json.loads(subprocess.check_output( | |
| 14 | ["nomad", "var", "get", "-out=json", f"nomad/jobs/{service}"], | |
| 14 | cfg = Path(data['hostRoot']) / 'cfg' | |
| 15 | service = Path(data['hostRoot']).name | |
| 16 | variable = json.loads(subprocess.check_output( | |
| 17 | ['nomad', 'var', 'get', '-out=json', 'nomad/jobs/' + service], | |
| 15 | 18 | text=True, stderr=subprocess.DEVNULL, |
| 16 | ))["Items"]["idp_header"] | |
| 17 | if not re.fullmatch(r"[0-9a-f]{32}", secret): | |
| 18 | raise ValueError("invalid Copyparty identity secret") | |
| 19 | config = Path(__file__).with_name("copyparty.conf").read_text() | |
| 20 | if config.count("STUDIO_IDP_HEADER") != 1: | |
| 21 | raise ValueError("expected one Copyparty identity header marker") | |
| 22 | config = config.replace("STUDIO_IDP_HEADER", "X-Studio-Idp-" + secret) | |
| 23 | if config.count("STUDIO_MIN_FREE_GB") != 1: | |
| 24 | raise ValueError("expected one Copyparty free-space marker") | |
| 25 | capacity = os.statvfs(data["hostRoot"]) | |
| 26 | reserve = max(1, min(16, capacity.f_blocks * capacity.f_frsize // (4 * 10**9))) | |
| 27 | config = config.replace("STUDIO_MIN_FREE_GB", str(reserve)) | |
| 28 | target = root / "copyparty.conf" | |
| 29 | if not target.exists() or target.read_text() != config: | |
| 30 | pending = root / ".copyparty.conf.pending" | |
| 31 | pending.write_text(config) | |
| 32 | os.chmod(pending, 0o600) | |
| 33 | os.chown(pending, data["uid"], data["uid"]) | |
| 34 | os.replace(pending, target) | |
| 19 | )) | |
| 20 | items = variable['Items'] | |
| 21 | state_path = cfg / 'manager.json' | |
| 22 | state = json.loads(state_path.read_text()) if state_path.exists() else {'accounts': [], 'grants': [], 'names': {}} | |
| 23 | canonical_root = Path(os.environ.get('STUDIO_CLOVER_ROOT', str(Path(os.environ.get('STUDIO_ROOT', '/srv')) / 'clover'))) | |
| 24 | origin = Path(state.get('root', str(canonical_root))) | |
| 25 | if origin != canonical_root: | |
| 26 | for grant in state['grants']: | |
| 27 | grant['path'] = str(canonical_root / Path(grant['path']).relative_to(origin)) | |
| 28 | state['root'] = str(canonical_root) | |
| 29 | missing = ['file_api_' + account['id'].replace('-', '') for account in state['accounts'] | |
| 30 | if 'file_api_' + account['id'].replace('-', '') not in items] | |
| 31 | if missing and service != 'copyparty': | |
| 32 | source = copyparty.runs.nomad('var/nomad/jobs/copyparty') | |
| 33 | for key in missing: | |
| 34 | if source is None or key not in source['Items']: | |
| 35 | raise RuntimeError('Copyparty API credentials are missing from the source Nomad variables') | |
| 36 | items[key] = source['Items'][key] | |
| 37 | copyparty.runs.nomad('var/nomad/jobs/' + service + '?cas=' + str(variable['ModifyIndex']), 'PUT', | |
| 38 | {'Namespace': 'default', 'Path': 'nomad/jobs/' + service, 'Items': items}) | |
| 39 | elif missing: | |
| 40 | raise RuntimeError('Copyparty API credentials are missing; restore its Nomad variables before starting it') | |
| 41 | rendered = copyparty.configuration(cfg, state, items, Path(__file__).with_name('copyparty.conf')) | |
| 42 | target = cfg / 'copyparty.conf' | |
| 43 | if not target.exists() or target.read_bytes() != rendered: | |
| 44 | copyparty.atomic(target, rendered, (data['uid'], data['uid'])) | |
| 35 | 45 | os.chmod(target, 0o600) |
| 36 | os.chown(target, data["uid"], data["uid"]) | |
| 46 | os.chown(target, data['uid'], data['uid']) |
tools/copyparty-cutover-test.py created+299| ... | ... | @@ -0,0 +1,299 @@ |
| 1 | #!/usr/bin/env python3 | |
| 2 | import importlib.util | |
| 3 | import json | |
| 4 | import os | |
| 5 | from pathlib import Path | |
| 6 | import sqlite3 | |
| 7 | import sys | |
| 8 | import tempfile | |
| 9 | import unittest | |
| 10 | from unittest.mock import patch | |
| 11 | ||
| 12 | spec = importlib.util.spec_from_file_location('cutover', Path(__file__).with_name('copyparty-cutover.py')) | |
| 13 | cutover = importlib.util.module_from_spec(spec) | |
| 14 | spec.loader.exec_module(cutover) | |
| 15 | copyparty = cutover.copyparty | |
| 16 | public_statuses = cutover.public_statuses | |
| 17 | ||
| 18 | ||
| 19 | class Cutover(unittest.TestCase): | |
| 20 | def setUp(self): | |
| 21 | self.temporary = tempfile.TemporaryDirectory(prefix='copyparty-cutover-') | |
| 22 | self.addCleanup(self.temporary.cleanup) | |
| 23 | self.base = Path(self.temporary.name) | |
| 24 | self.root = self.base / 'clover' | |
| 25 | self.cfg = self.base / 'cfg' | |
| 26 | (self.root / 'Share').mkdir(parents=True) | |
| 27 | (self.cfg / 'copyparty').mkdir(parents=True) | |
| 28 | self.environment = patch.dict(os.environ, {'STUDIO_CLOVER_ROOT': str(self.root), 'STUDIO_COPYPARTY_CFG': str(self.cfg)}) | |
| 29 | self.environment.start() | |
| 30 | self.addCleanup(self.environment.stop) | |
| 31 | self.manager = copyparty.Manager([]) | |
| 32 | self.manager.job = {'ID': 'copyparty', 'Stop': False, 'Meta': {'studio_hostname': 'copyparty.example.invalid'}} | |
| 33 | self.events = [] | |
| 34 | edge = patch.object(cutover, 'public_statuses', side_effect=lambda host, keys, deadline: {key: 200 for key in keys}) | |
| 35 | edge.start() | |
| 36 | self.addCleanup(edge.stop) | |
| 37 | ||
| 38 | def stop(): | |
| 39 | self.events.append('stop') | |
| 40 | if self.manager.offline_at is None: | |
| 41 | self.manager.offline_at = cutover.time.monotonic() | |
| 42 | self.manager.stopped = True | |
| 43 | ||
| 44 | def start(): | |
| 45 | self.events.append('start') | |
| 46 | self.manager.stopped = False | |
| 47 | ||
| 48 | self.manager.stop = stop | |
| 49 | self.manager.start = start | |
| 50 | for source, _ in cutover.MOVES: | |
| 51 | path = self.root / source | |
| 52 | (path / 'nested').mkdir(parents=True) | |
| 53 | (path / 'selected.txt').write_text('Disposable fixture') | |
| 54 | (path / 'nested' / 'child.txt').write_text('Disposable child') | |
| 55 | (self.cfg / 'copyparty.conf').write_text('Fixture configuration') | |
| 56 | self.database = self.cfg / 'copyparty/shares.db' | |
| 57 | with sqlite3.connect(self.database) as db: | |
| 58 | db.executescript("CREATE TABLE kv(k text,v int); INSERT INTO kv VALUES('sver',2); CREATE TABLE sh(k text,pw text,vp text,pr text,st int,un text,t0 int,t1 int); CREATE TABLE sf(k text,vp text);") | |
| 59 | db.executemany('INSERT INTO sh VALUES(?,?,?,?,?,?,?,?)', [ | |
| 60 | ('folder-key', 'opaque-hash', 'clover/Documents/Public/Snowbound', 'rw', 0, 'snow', 10, 20), | |
| 61 | ('nested-key', '', 'clover/Documents/Public/Snowbound/nested', 'r', 0, 'snow', 30, 0), | |
| 62 | ('selected-key', 'another-hash', 'clover/Documents/EvilTakeout', 'r', 1, 'snow', 40, 50), | |
| 63 | ('unaffected-key', '', 'clover/Other', 'r', 0, 'snow', 60, 0), | |
| 64 | ]) | |
| 65 | db.execute('INSERT INTO sf VALUES(?,?)', ('selected-key', 'selected.txt')) | |
| 66 | with sqlite3.connect(self.database) as db: | |
| 67 | self.original = cutover.rows(db) | |
| 68 | self.before_content = {str(path.relative_to(self.root)): path.read_bytes() for path in self.root.rglob('*.txt')} | |
| 69 | ||
| 70 | def test_move_preserves_keys_selections_passwords_inodes_and_content(self): | |
| 71 | expected = cutover.plan(self.manager) | |
| 72 | result = cutover.apply(self.manager, expected, self.base / 'backups') | |
| 73 | self.assertEqual(result['result'], 'moved') | |
| 74 | self.assertEqual(self.events, ['stop', 'start']) | |
| 75 | with sqlite3.connect(self.database) as db: | |
| 76 | actual = cutover.rows(db) | |
| 77 | self.assertEqual(actual, cutover.remapped(self.manager, self.original, cutover.mappings(self.manager))) | |
| 78 | self.assertEqual(actual['sf'], self.original['sf']) | |
| 79 | for detail in expected['moves']: | |
| 80 | source, destination = Path(detail['source']), Path(detail['destination']) | |
| 81 | self.assertFalse(source.exists()) | |
| 82 | self.assertEqual(destination.stat().st_ino, detail['inode']) | |
| 83 | for old_path, contents in self.before_content.items(): | |
| 84 | old = self.root / old_path | |
| 85 | if old.is_relative_to(source): | |
| 86 | self.assertEqual((destination / old.relative_to(source)).read_bytes(), contents) | |
| 87 | backup = Path(result['backup']) | |
| 88 | for child in backup.iterdir(): | |
| 89 | self.assertEqual(child.stat().st_mode & 0o777, 0o600) | |
| 90 | self.assertEqual(backup.stat().st_mode & 0o777, 0o700) | |
| 91 | ||
| 92 | def test_public_probe_sends_bounded_verified_head_and_preserves_redirect_status(self): | |
| 93 | response = cutover.subprocess.CompletedProcess([], 0, stdout='302', stderr='') | |
| 94 | with patch.object(cutover.subprocess, 'run', return_value=response) as run: | |
| 95 | actual = public_statuses('copyparty.example.invalid', ['selected file'], cutover.time.monotonic() + 1) | |
| 96 | self.assertEqual(actual, {'selected file': 302}) | |
| 97 | arguments = run.call_args.args[0] | |
| 98 | self.assertEqual(arguments[:2], ['curl', '--disable']) | |
| 99 | self.assertIn('--head', arguments) | |
| 100 | self.assertEqual(arguments[arguments.index('--output') + 1], '/dev/null') | |
| 101 | self.assertEqual(arguments[arguments.index('--write-out') + 1], '%{http_code}') | |
| 102 | self.assertEqual(arguments[arguments.index('--noproxy') + 1], '*') | |
| 103 | self.assertEqual(arguments[-2:], ['--', 'https://copyparty.example.invalid/shr/selected%20file/']) | |
| 104 | self.assertLessEqual(run.call_args.kwargs['timeout'], 1) | |
| 105 | self.assertEqual(float(arguments[arguments.index('--max-time') + 1]), run.call_args.kwargs['timeout']) | |
| 106 | for flag in ['--insecure', '-k', '--location', '-L', '--include', '-i']: | |
| 107 | self.assertNotIn(flag, arguments) | |
| 108 | ||
| 109 | def test_public_probe_treats_network_tls_and_process_timeout_as_unavailable(self): | |
| 110 | for response in [ | |
| 111 | cutover.subprocess.CompletedProcess([], 35, stdout='000', stderr='TLS problem'), | |
| 112 | cutover.subprocess.CompletedProcess([], 28, stdout='200', stderr='Timeout'), | |
| 113 | cutover.subprocess.CompletedProcess([], 0, stdout='000', stderr=''), | |
| 114 | cutover.subprocess.CompletedProcess([], 0, stdout='200 headers', stderr=''), | |
| 115 | ]: | |
| 116 | with patch.object(cutover.subprocess, 'run', return_value=response): | |
| 117 | self.assertEqual(public_statuses('host', ['key'], cutover.time.monotonic() + 1), {'key': None}) | |
| 118 | with patch.object(cutover.subprocess, 'run', side_effect=cutover.subprocess.TimeoutExpired('curl', 0.1)): | |
| 119 | self.assertEqual(public_statuses('host', ['key'], cutover.time.monotonic() + 0.1), {'key': None}) | |
| 120 | with patch.object(cutover.subprocess, 'run') as run: | |
| 121 | self.assertEqual(public_statuses('host', ['key'], cutover.time.monotonic() - 1), {'key': None}) | |
| 122 | run.assert_not_called() | |
| 123 | ||
| 124 | def test_stalled_probe_children_cannot_hold_the_executor_past_its_budget(self): | |
| 125 | run = cutover.subprocess.run | |
| 126 | def stalled(arguments, **options): | |
| 127 | return run([sys.executable, '-c', 'import time; time.sleep(10)'], **options) | |
| 128 | started = cutover.time.monotonic() | |
| 129 | with patch.object(cutover.subprocess, 'run', side_effect=stalled): | |
| 130 | actual = public_statuses('host', ['one', 'two'], started + 0.15) | |
| 131 | self.assertEqual(actual, {'one': None, 'two': None}) | |
| 132 | self.assertLess(cutover.time.monotonic() - started, 1.5) | |
| 133 | ||
| 134 | def test_public_poll_recovers_and_timeout_obeys_its_budget(self): | |
| 135 | clock = [0.0] | |
| 136 | def sleep(delay): | |
| 137 | clock[0] += delay | |
| 138 | with patch.object(cutover.time, 'monotonic', side_effect=lambda: clock[0]), patch.object(cutover.time, 'sleep', side_effect=sleep): | |
| 139 | with patch.object(cutover, 'public_statuses', side_effect=[{'key': 503}, {'key': 200}]): | |
| 140 | cutover.wait_public('host', {'key': 200}, 1) | |
| 141 | self.assertEqual(clock[0], 0.5) | |
| 142 | with patch.object(cutover, 'public_statuses', return_value={'key': None}), self.assertRaises(TimeoutError): | |
| 143 | cutover.wait_public('host', {'key': 200}, 1) | |
| 144 | self.assertEqual(clock[0], 1) | |
| 145 | ||
| 146 | def test_edge_timeout_rolls_back_and_records_recovery_evidence(self): | |
| 147 | expected = cutover.plan(self.manager) | |
| 148 | with patch.object(cutover, 'wait_public', side_effect=[TimeoutError('injected edge timeout'), None]): | |
| 149 | with self.assertRaisesRegex(TimeoutError, 'edge timeout'): | |
| 150 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 151 | self.assertEqual(cutover.plan(self.manager), expected) | |
| 152 | self.assertEqual(self.events, ['stop', 'start', 'stop', 'start']) | |
| 153 | evidence = json.loads(next((self.base / 'backups').glob('*/result.json')).read_text()) | |
| 154 | self.assertEqual(evidence['result'], 'rolled-back') | |
| 155 | self.assertEqual(evidence['publicStatuses'], {key: 200 for key in expected['affectedKeys']}) | |
| 156 | ||
| 157 | def test_existing_grants_follow_moves_and_configuration_is_rerendered(self): | |
| 158 | source, destination = cutover.mappings(self.manager)[0] | |
| 159 | state = {'accounts': [], 'grants': [{'id': 'fixture', 'subject': 'user:fixture', 'path': str(source), 'permissions': 'r'}], 'names': {}} | |
| 160 | (self.cfg / 'manager.json').write_text(json.dumps(state)) | |
| 161 | expected = cutover.plan(self.manager) | |
| 162 | with patch.object(copyparty.runs, 'nomad', return_value={'Items': {}}), patch.object(copyparty, 'configuration', return_value=b'Updated configuration') as render: | |
| 163 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 164 | saved = json.loads((self.cfg / 'manager.json').read_text()) | |
| 165 | self.assertEqual(saved['grants'][0]['path'], str(destination)) | |
| 166 | self.assertEqual(render.call_args.args[1]['grants'], saved['grants']) | |
| 167 | self.assertEqual((self.cfg / 'copyparty.conf').read_bytes(), b'Updated configuration') | |
| 168 | ||
| 169 | def test_failed_restart_restores_files_and_database_before_restart(self): | |
| 170 | def start(): | |
| 171 | self.events.append('start') | |
| 172 | if self.events.count('start') == 1: | |
| 173 | raise RuntimeError('injected startup failure') | |
| 174 | self.manager.stopped = False | |
| 175 | self.manager.start = start | |
| 176 | expected = cutover.plan(self.manager) | |
| 177 | with self.assertRaisesRegex(RuntimeError, 'injected startup failure'): | |
| 178 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 179 | self.assertEqual(self.events, ['stop', 'start', 'stop', 'start']) | |
| 180 | self.assertEqual(cutover.plan(self.manager), expected) | |
| 181 | with sqlite3.connect(self.database) as db: | |
| 182 | self.assertEqual(cutover.rows(db), self.original) | |
| 183 | for path, content in self.before_content.items(): | |
| 184 | self.assertEqual((self.root / path).read_bytes(), content) | |
| 185 | ||
| 186 | def test_rollback_uses_actual_helper_stop_and_start_with_frozen_nomad_job(self): | |
| 187 | self.manager.stop = copyparty.Manager.stop.__get__(self.manager) | |
| 188 | self.manager.start = copyparty.Manager.start.__get__(self.manager) | |
| 189 | self.manager.job = None | |
| 190 | requests = [] | |
| 191 | def nomad(path, method='GET', body=None, **_): | |
| 192 | requests.append((path, method, body)) | |
| 193 | if path == 'job/copyparty' and method == 'GET': | |
| 194 | return {'ID': 'copyparty', 'Stop': False, 'Meta': {'immutable': 'fixture', 'studio_hostname': 'copyparty.example.invalid'}} | |
| 195 | if path.endswith('/allocations'): | |
| 196 | return [{'ClientStatus': 'complete'}] | |
| 197 | if path.startswith('var/'): | |
| 198 | return {'Items': {'idp_header': 'fixture'}} | |
| 199 | return {} | |
| 200 | expected = cutover.plan(self.manager) | |
| 201 | with patch.object(copyparty.runs, 'nomad', side_effect=nomad), patch.object(self.manager, 'reload', side_effect=[copyparty.Error(400, 'injected reload rejection'), None]): | |
| 202 | with self.assertRaisesRegex(copyparty.Error, 'injected reload rejection'): | |
| 203 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 204 | self.assertEqual(cutover.plan(self.manager), expected) | |
| 205 | registrations = [body for path, method, body in requests if path == 'jobs' and method == 'POST'] | |
| 206 | self.assertEqual(registrations, [{'Job': self.manager.job}, {'Job': self.manager.job}]) | |
| 207 | self.assertEqual(len([path for path, method, _ in requests if method == 'DELETE']), 2) | |
| 208 | self.assertFalse(self.manager.stopped) | |
| 209 | ||
| 210 | def test_partial_rename_failure_rolls_back_before_start(self): | |
| 211 | native_rename = copyparty.rename | |
| 212 | calls = 0 | |
| 213 | def rename(source, destination): | |
| 214 | nonlocal calls | |
| 215 | calls += 1 | |
| 216 | if calls == 3: | |
| 217 | raise RuntimeError('injected third rename failure') | |
| 218 | self.assertTrue(self.manager.stopped) | |
| 219 | native_rename(source, destination) | |
| 220 | expected = cutover.plan(self.manager) | |
| 221 | with patch.object(copyparty, 'rename', side_effect=rename), self.assertRaisesRegex(RuntimeError, 'third rename'): | |
| 222 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 223 | self.assertEqual(cutover.plan(self.manager), expected) | |
| 224 | self.assertEqual(self.events, ['stop', 'start']) | |
| 225 | ||
| 226 | def test_stop_timeout_still_restarts_the_frozen_job(self): | |
| 227 | def stop(): | |
| 228 | self.manager.offline_at = cutover.time.monotonic() | |
| 229 | self.manager.stopped = True | |
| 230 | raise RuntimeError('injected stop timeout') | |
| 231 | self.manager.stop = stop | |
| 232 | expected = cutover.plan(self.manager) | |
| 233 | with self.assertRaisesRegex(RuntimeError, 'stop timeout'): | |
| 234 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 235 | self.assertEqual(self.events, ['start']) | |
| 236 | self.assertEqual(cutover.plan(self.manager), expected) | |
| 237 | ||
| 238 | def test_ambiguous_stop_response_reregisters_even_without_stopped_flag(self): | |
| 239 | def stop(): | |
| 240 | self.manager.offline_at = cutover.time.monotonic() | |
| 241 | self.manager.stopped = False | |
| 242 | raise TimeoutError('lost Nomad stop response') | |
| 243 | self.manager.stop = stop | |
| 244 | expected = cutover.plan(self.manager) | |
| 245 | with self.assertRaisesRegex(TimeoutError, 'lost Nomad stop response'): | |
| 246 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 247 | self.assertEqual(self.events, ['start']) | |
| 248 | self.assertEqual(cutover.plan(self.manager), expected) | |
| 249 | ||
| 250 | def test_verification_budget_reserves_time_for_restart(self): | |
| 251 | expected = cutover.plan(self.manager) | |
| 252 | original_tree = cutover.tree | |
| 253 | def timed_tree(path, deadline=None): | |
| 254 | if deadline is not None: | |
| 255 | raise TimeoutError('injected verification deadline') | |
| 256 | return original_tree(path) | |
| 257 | with patch.object(cutover, 'tree', side_effect=timed_tree), self.assertRaisesRegex(TimeoutError, 'verification deadline'): | |
| 258 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 259 | self.assertEqual(self.events, ['stop', 'start']) | |
| 260 | self.assertEqual(cutover.plan(self.manager), expected) | |
| 261 | ||
| 262 | def test_source_and_database_drift_stop_before_mutation(self): | |
| 263 | expected = cutover.plan(self.manager) | |
| 264 | (self.root / cutover.MOVES[0][0] / 'changed.txt').write_text('Concurrent disposable write') | |
| 265 | with self.assertRaisesRegex(ValueError, 'plan changed'): | |
| 266 | cutover.apply(self.manager, expected, self.base / 'backups') | |
| 267 | self.assertEqual(self.events, []) | |
| 268 | ||
| 269 | def test_parent_shares_and_parent_selections_require_explicit_url_preservation(self): | |
| 270 | for selected in [0, 1]: | |
| 271 | with sqlite3.connect(self.database) as db: | |
| 272 | db.execute('INSERT OR REPLACE INTO sh VALUES(?,?,?,?,?,?,?,?)', ('parent', '', 'clover/Documents/Public', 'r', selected, 'snow', 1, 0)) | |
| 273 | if selected: | |
| 274 | db.execute('INSERT INTO sf VALUES(?,?)', ('parent', 'Snowbound')) | |
| 275 | with self.assertRaisesRegex(ValueError, 'parent of a moved folder'): | |
| 276 | cutover.plan(self.manager) | |
| 277 | with sqlite3.connect(self.database) as db: | |
| 278 | db.execute("DELETE FROM sh WHERE k='parent'") | |
| 279 | db.execute("DELETE FROM sf WHERE k='parent'") | |
| 280 | ||
| 281 | def test_symlinks_mounts_and_existing_destinations_reject_before_downtime(self): | |
| 282 | source, destination = cutover.mappings(self.manager)[0] | |
| 283 | destination.mkdir() | |
| 284 | with self.assertRaises(ValueError): | |
| 285 | cutover.plan(self.manager) | |
| 286 | destination.rmdir() | |
| 287 | link = source / 'link' | |
| 288 | link.symlink_to('selected.txt') | |
| 289 | with self.assertRaisesRegex(ValueError, 'symbolic link'): | |
| 290 | cutover.plan(self.manager) | |
| 291 | link.unlink() | |
| 292 | with patch.object(cutover.os.path, 'ismount', side_effect=lambda path: Path(path) == source): | |
| 293 | with self.assertRaises(ValueError): | |
| 294 | cutover.plan(self.manager) | |
| 295 | self.assertEqual(self.events, []) | |
| 296 | ||
| 297 | ||
| 298 | if __name__ == '__main__': | |
| 299 | unittest.main() |
tools/copyparty-cutover.py created+279| ... | ... | @@ -0,0 +1,279 @@ |
| 1 | #!/usr/bin/env python3 | |
| 2 | import argparse | |
| 3 | import concurrent.futures | |
| 4 | import copy | |
| 5 | import hashlib | |
| 6 | import importlib.util | |
| 7 | import json | |
| 8 | import os | |
| 9 | import re | |
| 10 | from pathlib import Path | |
| 11 | import sqlite3 | |
| 12 | import socket | |
| 13 | import subprocess | |
| 14 | import stat | |
| 15 | import time | |
| 16 | import uuid | |
| 17 | from urllib.parse import quote | |
| 18 | ||
| 19 | ||
| 20 | spec = importlib.util.spec_from_file_location("copyparty", Path(__file__).resolve().parent.parent / "service/copyparty/manage.py") | |
| 21 | copyparty = importlib.util.module_from_spec(spec) | |
| 22 | spec.loader.exec_module(copyparty) | |
| 23 | ||
| 24 | ||
| 25 | MOVES = ( | |
| 26 | ('Documents/Public/Snowbound', 'Clover - Snowbound'), | |
| 27 | ('Documents/Public/Meadow', 'Clover - Meadow Chat'), | |
| 28 | ('Documents/Public/Bill Wurtz Music', 'Xavier - Bill Wurtz Music'), | |
| 29 | ('Documents/HOTEWIG Reanimated', 'Collab - HOTEWIG Reanimated'), | |
| 30 | ('Documents/EvilTakeout', 'Lino - Evil Inc Data Takeout'), | |
| 31 | ('Documents/Windows7', 'Clover - Windows 7'), | |
| 32 | ) | |
| 33 | ||
| 34 | ||
| 35 | def digest(value): | |
| 36 | return hashlib.sha256(json.dumps(value, ensure_ascii=False, separators=(',', ':')).encode()).hexdigest() | |
| 37 | ||
| 38 | ||
| 39 | def rows(database): | |
| 40 | if database.execute("SELECT v FROM kv WHERE k='sver'").fetchone() != (2,): | |
| 41 | raise ValueError('Unsupported Copyparty share database version.') | |
| 42 | return { | |
| 43 | 'sh': database.execute('SELECT * FROM sh ORDER BY k').fetchall(), | |
| 44 | 'sf': database.execute('SELECT * FROM sf ORDER BY k,vp').fetchall(), | |
| 45 | } | |
| 46 | ||
| 47 | ||
| 48 | def mappings(manager): | |
| 49 | return [(manager.root / source, manager.root / 'Share' / name) for source, name in MOVES] | |
| 50 | ||
| 51 | ||
| 52 | def remapped(manager, before, moves): | |
| 53 | after = copy.deepcopy(before) | |
| 54 | for index, row in enumerate(before['sh']): | |
| 55 | source_path = manager.real(row[2].strip('/')) | |
| 56 | for source, destination in moves: | |
| 57 | if source_path is not None and source_path.is_relative_to(source): | |
| 58 | changed = list(row) | |
| 59 | changed[2] = manager.virtual(destination / source_path.relative_to(source)) | |
| 60 | after['sh'][index] = tuple(changed) | |
| 61 | break | |
| 62 | if (source_path is not None and source.is_relative_to(source_path)) or row[2].strip('/') == '': | |
| 63 | raise ValueError('A share includes a parent of a moved folder. Preserve its URL paths before cutover: ' + row[0]) | |
| 64 | return after | |
| 65 | ||
| 66 | ||
| 67 | def tree(path, deadline=None): | |
| 68 | def unreadable(error): | |
| 69 | raise error | |
| 70 | ||
| 71 | result = [] | |
| 72 | for directory, dirs, files in os.walk(path, followlinks=False, onerror=unreadable): | |
| 73 | if deadline is not None and time.monotonic() >= deadline: | |
| 74 | raise TimeoutError("Folder verification exceeded the cutover budget.") | |
| 75 | base = Path(directory) | |
| 76 | if base != path and os.path.ismount(base): | |
| 77 | raise ValueError('A source contains a mounted filesystem: ' + str(base)) | |
| 78 | for entry in [base] + [base / name for name in files] + [base / name for name in dirs if (base / name).is_symlink()]: | |
| 79 | if deadline is not None and time.monotonic() >= deadline: | |
| 80 | raise TimeoutError("Folder verification exceeded the cutover budget.") | |
| 81 | info = entry.lstat() | |
| 82 | if stat.S_ISLNK(info.st_mode): | |
| 83 | raise ValueError('Review this source symbolic link before cutover: ' + str(entry)) | |
| 84 | result.append((str(entry.relative_to(path)), info.st_dev, info.st_ino, info.st_mode, info.st_size if not stat.S_ISDIR(info.st_mode) else 0, info.st_mtime_ns)) | |
| 85 | dirs[:] = [name for name in dirs if not (base / name).is_symlink()] | |
| 86 | return sorted(result) | |
| 87 | ||
| 88 | ||
| 89 | def plan(manager, deadline=None): | |
| 90 | moves = mappings(manager) | |
| 91 | details = [] | |
| 92 | for source, destination in moves: | |
| 93 | manager.path_for(str(source)) | |
| 94 | manager.path_for(str(destination), missing=True) | |
| 95 | if source.is_symlink() or os.path.ismount(source) or os.path.lexists(destination): | |
| 96 | raise ValueError('A source is mounted or a destination already exists: ' + str(source)) | |
| 97 | source_info, parent_info = source.stat(), destination.parent.stat() | |
| 98 | if source_info.st_dev != parent_info.st_dev: | |
| 99 | raise ValueError('Source and destination must be on the same filesystem: ' + str(source)) | |
| 100 | manifest = tree(source, deadline) | |
| 101 | details.append({'source': str(source), 'destination': str(destination), 'device': source_info.st_dev, | |
| 102 | 'inode': source_info.st_ino, 'entries': len(manifest), 'tree': digest(manifest)}) | |
| 103 | with sqlite3.connect((manager.native / 'shares.db').as_uri() + '?mode=ro', uri=True) as database: | |
| 104 | before = rows(database) | |
| 105 | after = remapped(manager, before, moves) | |
| 106 | return {'host': socket.gethostname(), 'service': manager.service, 'configRoot': str(manager.cfg), 'moves': details, 'shares': len(before['sh']), 'selections': len(before['sf']), | |
| 107 | 'affectedKeys': [old[0] for old, new in zip(before['sh'], after['sh']) if old != new], | |
| 108 | 'sharesBefore': digest(before), 'sharesAfter': digest(after), | |
| 109 | 'config': digest((manager.cfg / 'copyparty.conf').read_text()), | |
| 110 | 'manager': digest((manager.cfg / 'manager.json').read_text() if (manager.cfg / 'manager.json').exists() else None)} | |
| 111 | ||
| 112 | ||
| 113 | def write_database(database, data): | |
| 114 | current = rows(database) | |
| 115 | if current['sf'] != data['sf'] or len(current['sh']) != len(data['sh']) or any( | |
| 116 | old[:2] + old[3:] != new[:2] + new[3:] | |
| 117 | for old, new in zip(current['sh'], data['sh']) | |
| 118 | ): | |
| 119 | raise ValueError('Share keys, selections, passwords, or permissions changed during cutover.') | |
| 120 | with database: | |
| 121 | database.executemany('UPDATE sh SET vp=? WHERE k=?', [(row[2], row[0]) for row in data['sh']]) | |
| 122 | if rows(database) != data: | |
| 123 | raise ValueError('Copyparty share paths changed while the database was being written.') | |
| 124 | ||
| 125 | ||
| 126 | def public_statuses(host, keys, deadline): | |
| 127 | def head(key): | |
| 128 | timeout = min(3, deadline - time.monotonic()) | |
| 129 | if timeout <= 0: | |
| 130 | return key, None | |
| 131 | try: | |
| 132 | response = subprocess.run( | |
| 133 | ['curl', '--disable', '--noproxy', '*', '--silent', '--show-error', | |
| 134 | '--head', '--output', '/dev/null', '--write-out', '%{http_code}', | |
| 135 | '--max-time', str(timeout), '--', 'https://' + host + '/shr/' + quote(key, safe='') + '/'], | |
| 136 | capture_output=True, text=True, timeout=timeout, | |
| 137 | ) | |
| 138 | status = response.stdout.strip() | |
| 139 | return key, int(status) if response.returncode == 0 and re.fullmatch(r'[1-5][0-9]{2}', status) else None | |
| 140 | except (OSError, subprocess.TimeoutExpired): | |
| 141 | return key, None | |
| 142 | ||
| 143 | with concurrent.futures.ThreadPoolExecutor(max_workers=4) as workers: | |
| 144 | return dict(workers.map(head, keys)) | |
| 145 | ||
| 146 | ||
| 147 | def wait_public(host, expected, deadline): | |
| 148 | while time.monotonic() < deadline: | |
| 149 | if public_statuses(host, expected, deadline) == expected: | |
| 150 | return | |
| 151 | time.sleep(min(0.5, max(0, deadline - time.monotonic()))) | |
| 152 | raise TimeoutError('Public share links did not recover before the cutover deadline.') | |
| 153 | ||
| 154 | ||
| 155 | def apply(manager, expected, backup_root): | |
| 156 | if plan(manager) != expected: | |
| 157 | raise ValueError('The cutover plan changed. Review a new plan before applying.') | |
| 158 | if manager.job is None: | |
| 159 | manager.job = copyparty.runs.nomad('job/' + manager.service) | |
| 160 | host = (manager.job or {}).get('Meta', {}).get('studio_hostname', '') | |
| 161 | if not isinstance(host, str) or not re.fullmatch(r'[a-zA-Z0-9.-]+', host): | |
| 162 | raise ValueError('Copyparty has no public hostname. Check its deployed service route.') | |
| 163 | manager.job['Stop'] = False | |
| 164 | baseline = public_statuses(host, expected['affectedKeys'], time.monotonic() + 15) | |
| 165 | if any(status is None for status in baseline.values()): | |
| 166 | raise ValueError('Public shares are unavailable. Restore their current routes before cutover.') | |
| 167 | backup_root.mkdir(parents=True, exist_ok=True, mode=0o700) | |
| 168 | metadata = backup_root.lstat() | |
| 169 | if not stat.S_ISDIR(metadata.st_mode) or metadata.st_uid != os.geteuid() or metadata.st_mode & 0o077: | |
| 170 | raise ValueError('Cutover backups require an owner-only directory.') | |
| 171 | backup = backup_root / (time.strftime('%Y%m%dT%H%M%SZ', time.gmtime()) + '-' + uuid.uuid4().hex[:8]) | |
| 172 | backup.mkdir(mode=0o700) | |
| 173 | database_path = manager.native / 'shares.db' | |
| 174 | saved_files = {} | |
| 175 | for path in (manager.cfg / 'copyparty.conf', manager.cfg / 'manager.json'): | |
| 176 | saved_files[path] = path.read_bytes() if path.exists() else None | |
| 177 | if saved_files[path] is not None: | |
| 178 | copyparty.atomic(backup / path.name, saved_files[path]) | |
| 179 | copyparty.atomic(backup / 'plan.json', json.dumps(expected, ensure_ascii=False).encode()) | |
| 180 | moved = [] | |
| 181 | before = None | |
| 182 | database = None | |
| 183 | manager.deadline = time.monotonic() + 285 | |
| 184 | mutation_deadline = manager.deadline - 205 | |
| 185 | restart_attempted = False | |
| 186 | try: | |
| 187 | manager.stop() | |
| 188 | copyparty.atomic(backup / 'nomad-job.json', json.dumps(manager.job).encode()) | |
| 189 | if plan(manager, mutation_deadline) != expected: | |
| 190 | raise ValueError('The source files or share rows changed before Copyparty stopped.') | |
| 191 | database = sqlite3.connect(database_path) | |
| 192 | before = rows(database) | |
| 193 | with sqlite3.connect(backup / 'shares.db') as target: | |
| 194 | database.backup(target) | |
| 195 | (backup / 'shares.db').chmod(0o600) | |
| 196 | after = remapped(manager, before, mappings(manager)) | |
| 197 | data_path = manager.cfg / 'manager.json' | |
| 198 | data = json.loads(saved_files[data_path]) if saved_files[data_path] is not None else None | |
| 199 | for source, destination in mappings(manager): | |
| 200 | copyparty.rename(source, destination) | |
| 201 | moved.append((source, destination)) | |
| 202 | write_database(database, after) | |
| 203 | if data is not None: | |
| 204 | for grant in data.get('grants', []): | |
| 205 | original = Path(grant['path']) | |
| 206 | for source, destination in mappings(manager): | |
| 207 | if original.is_relative_to(source): | |
| 208 | grant['path'] = str(destination / original.relative_to(source)) | |
| 209 | break | |
| 210 | copyparty.atomic(data_path, json.dumps(data, ensure_ascii=False).encode()) | |
| 211 | variable = copyparty.runs.nomad('var/nomad/jobs/' + manager.service) | |
| 212 | rendered = copyparty.configuration(manager.cfg, {**data, 'root': str(manager.root)}, variable['Items']) | |
| 213 | info = (manager.cfg / 'copyparty.conf').stat() | |
| 214 | copyparty.atomic(manager.cfg / 'copyparty.conf', rendered, (info.st_uid, info.st_gid)) | |
| 215 | for detail in expected['moves']: | |
| 216 | destination = Path(detail['destination']) | |
| 217 | if digest(tree(destination, mutation_deadline)) != detail['tree']: | |
| 218 | raise ValueError('Moved folder metadata changed: ' + str(destination)) | |
| 219 | if digest(rows(database)) != expected['sharesAfter']: | |
| 220 | raise ValueError('Share keys, selections, passwords, or permissions changed during cutover.') | |
| 221 | if time.monotonic() >= mutation_deadline: | |
| 222 | raise TimeoutError("Folder updates exceeded the cutover budget.") | |
| 223 | restart_attempted = True | |
| 224 | manager.start() | |
| 225 | wait_public(host, baseline, manager.offline_at + 150) | |
| 226 | result = {'result': 'moved', 'moves': len(moved), 'affectedKeys': expected['affectedKeys'], 'backup': str(backup), | |
| 227 | 'downtimeSeconds': time.monotonic() - manager.offline_at, 'publicStatuses': baseline} | |
| 228 | copyparty.atomic(backup / 'result.json', json.dumps(result).encode()) | |
| 229 | return result | |
| 230 | except BaseException as failure: | |
| 231 | if moved or before is not None: | |
| 232 | if restart_attempted: | |
| 233 | manager.stop() | |
| 234 | if database is None: | |
| 235 | database = sqlite3.connect(database_path) | |
| 236 | if before is not None: | |
| 237 | write_database(database, before) | |
| 238 | for source, destination in reversed(moved): | |
| 239 | copyparty.rename(destination, source) | |
| 240 | for path, content in saved_files.items(): | |
| 241 | if content is None: | |
| 242 | path.unlink(missing_ok=True) | |
| 243 | else: | |
| 244 | owner = path.stat() | |
| 245 | copyparty.atomic(path, content, (owner.st_uid, owner.st_gid)) | |
| 246 | if manager.job is not None: | |
| 247 | manager.start() | |
| 248 | wait_public(host, baseline, manager.deadline) | |
| 249 | evidence = {'result': 'rolled-back', 'reason': str(failure), 'downtimeSeconds': time.monotonic() - manager.offline_at, | |
| 250 | 'publicStatuses': baseline} | |
| 251 | copyparty.atomic(backup / 'result.json', json.dumps(evidence).encode()) | |
| 252 | raise | |
| 253 | finally: | |
| 254 | if database is not None: | |
| 255 | database.close() | |
| 256 | ||
| 257 | ||
| 258 | def main(): | |
| 259 | parser = argparse.ArgumentParser(description='Review or apply the six Clover share-folder moves.') | |
| 260 | parser.add_argument('--apply', action='store_true') | |
| 261 | parser.add_argument('--plan', type=Path, help='Save the reviewed plan, or read it when applying.') | |
| 262 | args = parser.parse_args() | |
| 263 | manager = copyparty.Manager([]) | |
| 264 | if args.apply: | |
| 265 | if args.plan is None: | |
| 266 | parser.error('--apply requires --plan from the reviewed preflight') | |
| 267 | lock = os.open(manager.cfg / '.manager.lock', os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW, 0o600) | |
| 268 | with os.fdopen(lock, 'w') as locked: | |
| 269 | copyparty.fcntl.flock(locked, copyparty.fcntl.LOCK_EX) | |
| 270 | print(json.dumps(apply(manager, json.loads(args.plan.read_text()), Path('/var/lib/studio/copyparty-cutover')))) | |
| 271 | else: | |
| 272 | value = plan(manager) | |
| 273 | if args.plan is not None: | |
| 274 | copyparty.atomic(args.plan, json.dumps(value, ensure_ascii=False, indent=2).encode()) | |
| 275 | print(json.dumps(value, ensure_ascii=False)) | |
| 276 | ||
| 277 | ||
| 278 | if __name__ == '__main__': | |
| 279 | main() |
tools/copyparty-test.py created+323| ... | ... | @@ -0,0 +1,323 @@ |
| 1 | #!/usr/bin/env python3 | |
| 2 | import copy | |
| 3 | import importlib.util | |
| 4 | import json | |
| 5 | import os | |
| 6 | from pathlib import Path | |
| 7 | import sqlite3 | |
| 8 | import tempfile | |
| 9 | import unittest | |
| 10 | from unittest.mock import patch | |
| 11 | import uuid | |
| 12 | ||
| 13 | spec = importlib.util.spec_from_file_location('shares', Path(__file__).resolve().parent.parent / 'service/copyparty/manage.py') | |
| 14 | shares = importlib.util.module_from_spec(spec) | |
| 15 | spec.loader.exec_module(shares) | |
| 16 | ||
| 17 | ||
| 18 | class Shares(unittest.TestCase): | |
| 19 | def setUp(self): | |
| 20 | self.temporary = tempfile.TemporaryDirectory() | |
| 21 | self.addCleanup(self.temporary.cleanup) | |
| 22 | self.root = Path(self.temporary.name) / 'clover' | |
| 23 | self.cfg = Path(self.temporary.name) / 'cfg' | |
| 24 | self.root.mkdir() | |
| 25 | self.root = self.root.resolve() | |
| 26 | (self.cfg / 'copyparty').mkdir(parents=True) | |
| 27 | (self.cfg / 'copyparty.conf').write_text('original configuration\n') | |
| 28 | self.database = self.cfg / 'copyparty/shares.db' | |
| 29 | with sqlite3.connect(self.database) as db: | |
| 30 | db.executescript("CREATE TABLE kv(k text,v int); INSERT INTO kv VALUES('sver',2); CREATE TABLE sh(k text,pw text,vp text,pr text,st int,un text,t0 int,t1 int); CREATE TABLE sf(k text,vp text);") | |
| 31 | self.subject = str(uuid.uuid4()) | |
| 32 | self.users = [{'id': self.subject, 'username': 'test-user', 'enabled': True, 'kind': None, 'requiredActions': []}] | |
| 33 | self.variable = {'ModifyIndex': 1, 'Items': {'idp_header': 'a' * 32, 'unrelated': 'retained'}} | |
| 34 | self.environment = patch.dict(os.environ, {'STUDIO_CLOVER_ROOT': str(self.root), 'STUDIO_COPYPARTY_CFG': str(self.cfg)}) | |
| 35 | self.environment.start() | |
| 36 | self.addCleanup(self.environment.stop) | |
| 37 | self.nomad = patch.object(shares.runs, 'nomad', side_effect=self.request) | |
| 38 | self.nomad.start() | |
| 39 | self.addCleanup(self.nomad.stop) | |
| 40 | self.reloads = [] | |
| 41 | self.reload = patch.object(shares.Manager, 'reload', autospec=True, side_effect=lambda manager, items, **kwargs: self.reloads.append(copy.deepcopy(items))) | |
| 42 | self.reload_mock = self.reload.start() | |
| 43 | self.addCleanup(self.reload.stop) | |
| 44 | self.hash = patch.object(shares.Manager, 'hash', side_effect=lambda password: '+' + 'a' * 32 if password else '') | |
| 45 | self.hash.start() | |
| 46 | self.container = patch.object(shares.Manager, 'container', return_value='disposable') | |
| 47 | self.container.start() | |
| 48 | self.addCleanup(self.container.stop) | |
| 49 | self.addCleanup(self.hash.stop) | |
| 50 | ||
| 51 | def request(self, path, method='GET', body=None, timeout=10): | |
| 52 | self.assertTrue(path.startswith('var/nomad/jobs/copyparty')) | |
| 53 | if method == 'GET': | |
| 54 | return copy.deepcopy(self.variable) | |
| 55 | self.assertEqual(method, 'PUT') | |
| 56 | self.assertEqual(path.split('?cas=')[1], str(self.variable['ModifyIndex'])) | |
| 57 | self.variable = {'Items': copy.deepcopy(body['Items']), 'ModifyIndex': self.variable['ModifyIndex'] + 1} | |
| 58 | return copy.deepcopy(self.variable) | |
| 59 | ||
| 60 | def call(self, action, **payload): | |
| 61 | return shares.handle({'action': action, 'payload': payload, 'users': self.users}) | |
| 62 | ||
| 63 | def link(self, key='public', path='clover/source', count=0, files=(), owner='snow'): | |
| 64 | with sqlite3.connect(self.database) as db: | |
| 65 | db.execute('INSERT INTO sh VALUES(?,?,?,?,?,?,?,?)', (key, '', path, 'rw', count, owner, 100, 0)) | |
| 66 | db.executemany('INSERT INTO sf VALUES(?,?)', [(key, file) for file in files]) | |
| 67 | ||
| 68 | def update(self, path, mode='retarget', key='public'): | |
| 69 | return self.call('link.update', key=key, name='Clover - Example', path=str(path), permissions='rw', expires=0, password=None, mode=mode) | |
| 70 | ||
| 71 | def offline(self): | |
| 72 | def stop(manager): | |
| 73 | manager.job = {'ID': 'copyparty'} | |
| 74 | manager.offline_at = shares.time.monotonic() | |
| 75 | manager.stopped = True | |
| 76 | def start(manager): | |
| 77 | manager.stopped = False | |
| 78 | return patch.object(shares.Manager, 'stop', autospec=True, side_effect=stop), patch.object(shares.Manager, 'start', autospec=True, side_effect=start) | |
| 79 | ||
| 80 | def test_account_password_is_once_only_and_nomad_preserves_other_secrets(self): | |
| 81 | created = self.call('account.create', name='Local AI') | |
| 82 | self.assertTrue(created['password']) | |
| 83 | identifier = created['account']['id'] | |
| 84 | key = 'file_api_' + identifier.replace('-', '') | |
| 85 | self.assertEqual(self.variable['Items'][key], '+' + 'a' * 32) | |
| 86 | self.assertEqual(self.variable['Items']['unrelated'], 'retained') | |
| 87 | state = self.call('state') | |
| 88 | self.assertNotIn('password', state) | |
| 89 | self.assertNotIn(created['password'], (self.cfg / 'manager.json').read_text()) | |
| 90 | self.assertNotIn(created['password'], (self.cfg / 'copyparty.conf').read_text()) | |
| 91 | self.assertIn('w: @acct,-api-' + identifier, (self.cfg / 'copyparty.conf').read_text()) | |
| 92 | self.call('account.delete', id=identifier) | |
| 93 | self.assertNotIn(key, self.variable['Items']) | |
| 94 | ||
| 95 | def test_rotation_revokes_existing_copyparty_session(self): | |
| 96 | created = self.call('account.create', name='Local AI') | |
| 97 | identifier = created['account']['id'] | |
| 98 | session_path = self.cfg / 'copyparty/sessions.db' | |
| 99 | with sqlite3.connect(session_path) as db: | |
| 100 | db.executescript('CREATE TABLE us(un text,si text,t0 int);') | |
| 101 | db.execute('INSERT INTO us VALUES(?,?,?)', ('api-' + identifier, 'old-cookie', 100)) | |
| 102 | db.execute('INSERT INTO us VALUES(?,?,?)', ('snow', 'retained-cookie', 100)) | |
| 103 | self.call('account.rotate', id=identifier) | |
| 104 | with sqlite3.connect(session_path) as db: | |
| 105 | self.assertEqual(db.execute('SELECT un,si FROM us').fetchall(), [('snow', 'retained-cookie')]) | |
| 106 | ||
| 107 | def test_human_grant_uses_uuid_group_and_preserves_ancestor_access(self): | |
| 108 | folder = self.root / 'work' | |
| 109 | child = folder / 'child' | |
| 110 | child.mkdir(parents=True) | |
| 111 | first = self.call('grant.save', id=None, subject='user:' + self.subject, path=str(folder), permissions='rw') | |
| 112 | self.call('grant.save', id=None, subject='user:' + self.subject, path=str(child), permissions='r') | |
| 113 | config = (self.cfg / 'copyparty.conf').read_text() | |
| 114 | self.assertIn('rw: @snow-user-' + self.subject, config) | |
| 115 | self.assertNotIn('test-user', config) | |
| 116 | self.assertIn('xvol', config) | |
| 117 | child_config = config.split('[/clover/work/child]')[1] | |
| 118 | self.assertIn('A: snow', child_config) | |
| 119 | self.assertIn('rw: @snow-user-' + self.subject, child_config) | |
| 120 | self.call('grant.delete', id=first['grants'][0]['id']) | |
| 121 | config = (self.cfg / 'copyparty.conf').read_text() | |
| 122 | self.assertNotIn('[/clover/work]\n', config) | |
| 123 | self.assertNotIn('rw: @snow-user-' + self.subject, config) | |
| 124 | ||
| 125 | def test_exact_base_mount_is_rendered_once(self): | |
| 126 | (self.root / 'Public').mkdir() | |
| 127 | self.call('grant.save', id=None, subject='user:' + self.subject, path=str(self.root / 'Public'), permissions='d') | |
| 128 | config = (self.cfg / 'copyparty.conf').read_text() | |
| 129 | self.assertEqual(config.count('[/clover/Public]'), 1) | |
| 130 | self.assertIn('r.: *', config) | |
| 131 | self.assertIn('d: @snow-user-' + self.subject, config) | |
| 132 | ||
| 133 | def test_untrusted_boundary_and_symlink_escape_reject_before_nomad(self): | |
| 134 | folder = self.root / 'work' | |
| 135 | folder.mkdir() | |
| 136 | (self.root / 'escape').symlink_to(Path(self.temporary.name)) | |
| 137 | cases = [dict(id=None, subject='user:' + self.subject, path=str(self.root / 'escape'), permissions='r'), | |
| 138 | dict(id=None, subject='user:' + self.subject, path=str(self.root / '../outside'), permissions='r'), | |
| 139 | dict(id=None, subject='user:' + self.subject, path=str(folder), permissions='x'), | |
| 140 | dict(id=None, subject='user:' + str(uuid.uuid4()), path=str(folder), permissions='r')] | |
| 141 | for payload in cases: | |
| 142 | with self.subTest(payload=payload), self.assertRaises(shares.Error): | |
| 143 | self.call('grant.save', **payload) | |
| 144 | with self.assertRaises(shares.Error): | |
| 145 | self.call('state', extra=True) | |
| 146 | for user in [dict(self.users[0], enabled=False), dict(self.users[0], kind='ai'), dict(self.users[0], kind='guest')]: | |
| 147 | self.users = [user] | |
| 148 | with self.assertRaises(shares.Error): | |
| 149 | self.call('grant.save', id=None, subject='user:' + self.subject, path=str(folder), permissions='r') | |
| 150 | ||
| 151 | def test_pending_invited_user_can_receive_permissions_before_setup(self): | |
| 152 | folder = self.root / 'invited-work' | |
| 153 | folder.mkdir() | |
| 154 | self.users[0]['requiredActions'] = ['SETUP', 'UPDATE_PASSWORD'] | |
| 155 | state = self.call('grant.save', id=None, subject='user:' + self.subject, path=str(folder), permissions='rw') | |
| 156 | self.assertEqual(state['grants'][0]['subject'], 'user:' + self.subject) | |
| 157 | self.assertEqual(state['grants'][0]['path'], str(folder)) | |
| 158 | self.assertIn('rw: @snow-user-' + self.subject, (self.cfg / 'copyparty.conf').read_text()) | |
| 159 | ||
| 160 | def test_retarget_preserves_selected_files_and_rejects_missing_selection(self): | |
| 161 | destination = self.root / 'destination' | |
| 162 | destination.mkdir() | |
| 163 | (destination / 'kept.txt').write_text('disposable') | |
| 164 | self.link(count=1, files=['kept.txt']) | |
| 165 | self.update(destination) | |
| 166 | state = self.call('state') | |
| 167 | self.assertEqual(state['links'][0]['files'], ['kept.txt']) | |
| 168 | self.assertEqual(state['links'][0]['path'], str(destination)) | |
| 169 | missing = self.root / 'missing' | |
| 170 | missing.mkdir() | |
| 171 | with self.assertRaises(shares.Error): | |
| 172 | self.update(missing) | |
| 173 | self.assertEqual(self.call('state')['links'][0]['path'], str(destination)) | |
| 174 | ||
| 175 | def test_default_folder_create_and_link_delete_keep_data(self): | |
| 176 | result = self.call('link.create', name='Clover - Example', path='', permissions='rw', expires=0, password='', createFolder=True) | |
| 177 | path = self.root / 'Share/Clover - Example' | |
| 178 | self.assertTrue(path.is_dir()) | |
| 179 | self.call('link.delete', key=result['links'][0]['key']) | |
| 180 | self.assertTrue(path.is_dir()) | |
| 181 | self.assertEqual(self.call('state')['links'], []) | |
| 182 | ||
| 183 | def test_move_preserves_key_selection_nested_links_and_grants(self): | |
| 184 | source = self.root / 'source' | |
| 185 | (source / 'nested').mkdir(parents=True) | |
| 186 | (source / 'kept.txt').write_text('disposable') | |
| 187 | destination = self.root / 'destination' | |
| 188 | self.link(count=1, files=['kept.txt']) | |
| 189 | self.link(key='nested', path='clover/source/nested') | |
| 190 | self.call('grant.save', id=None, subject='user:' + self.subject, path=str(source / 'nested'), permissions='rw') | |
| 191 | stop, start = self.offline() | |
| 192 | with stop as stopped, start as started: | |
| 193 | self.update(destination, mode='move') | |
| 194 | stopped.assert_called_once() | |
| 195 | started.assert_called_once() | |
| 196 | self.assertFalse(source.exists()) | |
| 197 | self.assertEqual((destination / 'kept.txt').read_text(), 'disposable') | |
| 198 | state = self.call('state') | |
| 199 | self.assertEqual({link['key']: link['path'] for link in state['links']}, {'public': str(destination), 'nested': str(destination / 'nested')}) | |
| 200 | self.assertEqual(state['grants'][0]['path'], str(destination / 'nested')) | |
| 201 | self.assertEqual(next(link for link in state['links'] if link['key'] == 'public')['files'], ['kept.txt']) | |
| 202 | ||
| 203 | def test_move_rejects_existing_destination_and_ancestor_selection(self): | |
| 204 | source = self.root / 'source' | |
| 205 | source.mkdir() | |
| 206 | destination = self.root / 'destination' | |
| 207 | destination.mkdir() | |
| 208 | self.link() | |
| 209 | with self.assertRaises(shares.Error): | |
| 210 | self.update(destination, mode='move') | |
| 211 | destination.rmdir() | |
| 212 | self.link(key='ancestor', path='clover', count=0) | |
| 213 | with self.assertRaises(shares.Error): | |
| 214 | self.update(destination, mode='move') | |
| 215 | self.assertTrue(source.is_dir()) | |
| 216 | self.assertFalse(destination.exists()) | |
| 217 | ||
| 218 | def test_reload_failure_restores_config_state_database_and_secret(self): | |
| 219 | original = (self.cfg / 'copyparty.conf').read_bytes() | |
| 220 | calls = 0 | |
| 221 | def reload(manager, items): | |
| 222 | nonlocal calls | |
| 223 | calls += 1 | |
| 224 | if calls == 1: | |
| 225 | raise shares.Error(502, 'test rejection') | |
| 226 | self.reload_mock.side_effect = reload | |
| 227 | with self.assertRaises(shares.Error): | |
| 228 | self.call('account.create', name='Local AI') | |
| 229 | self.reload_mock.side_effect = lambda manager, items, **kwargs: self.reloads.append(copy.deepcopy(items)) | |
| 230 | self.assertEqual((self.cfg / 'copyparty.conf').read_bytes(), original) | |
| 231 | self.assertEqual(self.call('state')['accounts'], []) | |
| 232 | self.assertEqual(self.variable['Items'], {'idp_header': 'a' * 32, 'unrelated': 'retained'}) | |
| 233 | self.assertEqual(calls, 2) | |
| 234 | destination = self.root / 'target' | |
| 235 | destination.mkdir() | |
| 236 | self.link() | |
| 237 | self.reload_mock.side_effect = [shares.Error(502, 'test rejection'), None] | |
| 238 | with self.assertRaises(shares.Error): | |
| 239 | self.update(destination) | |
| 240 | self.assertEqual(self.call('state')['links'][0]['path'], str(self.root / 'source')) | |
| 241 | ||
| 242 | def test_failed_offline_restart_restores_folder_and_mappings(self): | |
| 243 | source = self.root / 'source' | |
| 244 | source.mkdir() | |
| 245 | destination = self.root / 'destination' | |
| 246 | self.link() | |
| 247 | stop, start = self.offline() | |
| 248 | with stop as stopped, patch.object(shares.Manager, 'start', autospec=True, side_effect=[shares.Error(502, 'test restart'), None]) as started: | |
| 249 | with self.assertRaises(shares.Error): | |
| 250 | self.update(destination, mode='move') | |
| 251 | self.assertEqual(stopped.call_count, 2) | |
| 252 | self.assertEqual(started.call_count, 2) | |
| 253 | self.assertTrue(source.is_dir()) | |
| 254 | self.assertFalse(destination.exists()) | |
| 255 | self.assertEqual(self.call('state')['links'][0]['path'], str(source)) | |
| 256 | ||
| 257 | def test_new_folders_work_under_host_private_umask(self): | |
| 258 | previous = os.umask(0o077) | |
| 259 | try: | |
| 260 | self.call('link.create', name='Clover - Restricted', path=str(self.root / 'Share/Clover - Restricted'), | |
| 261 | permissions='r', expires=0, password='', createFolder=True) | |
| 262 | finally: | |
| 263 | os.umask(previous) | |
| 264 | self.assertEqual((self.root / 'Share').stat().st_mode & 0o777, 0o775) | |
| 265 | self.assertEqual((self.root / 'Share/Clover - Restricted').stat().st_mode & 0o777, 0o775) | |
| 266 | ||
| 267 | def test_api_scope_excludes_wildcards_but_explicit_child_grant_works(self): | |
| 268 | account = self.call('account.create', name='Scoped')['account'] | |
| 269 | (self.root / 'Public').mkdir() | |
| 270 | self.call('grant.save', id=None, subject=account['subject'], path=str(self.root), permissions='r') | |
| 271 | config = (self.cfg / 'copyparty.conf').read_text() | |
| 272 | public = config.split('[/clover/Public]')[1].split('[/logs]')[0] | |
| 273 | self.assertIn('r: *\n', public) | |
| 274 | self.assertIn('w: @acct,-' + account['username'], public) | |
| 275 | self.assertIn('r: ' + account['username'], public) | |
| 276 | logs = config.split('[/logs]')[1] | |
| 277 | self.assertIn('g: *,-' + account['username'], logs) | |
| 278 | manager = shares.Manager(self.users) | |
| 279 | manager.data = json.loads((self.cfg / 'manager.json').read_text()) | |
| 280 | manager.items = self.variable['Items'] | |
| 281 | manager.owner_permissions(account['username'], self.root / 'Public', 'r') | |
| 282 | with self.assertRaises(shares.Error): | |
| 283 | manager.owner_permissions(account['username'], self.root / 'Public', 'w') | |
| 284 | ||
| 285 | def test_admin_can_retarget_other_owner_with_destination_permissions(self): | |
| 286 | destination = self.root / 'other' | |
| 287 | destination.mkdir() | |
| 288 | self.call('grant.save', id=None, subject='user:' + self.subject, path=str(destination), permissions='rw') | |
| 289 | self.link(owner='test-user') | |
| 290 | self.update(destination) | |
| 291 | self.assertEqual(self.call('state')['links'][0]['owner'], 'test-user') | |
| 292 | denied = self.root / 'denied' | |
| 293 | denied.mkdir() | |
| 294 | with self.assertRaises(shares.Error): | |
| 295 | self.update(denied) | |
| 296 | self.assertEqual(self.call('state')['links'][0]['path'], str(destination)) | |
| 297 | ||
| 298 | def test_stage_rebases_grants_without_changing_native_virtual_sources(self): | |
| 299 | folder = self.root / 'work' | |
| 300 | folder.mkdir() | |
| 301 | self.call('grant.save', id=None, subject='user:' + self.subject, path=str(folder), permissions='r') | |
| 302 | self.link() | |
| 303 | clone = self.root.parent / 'clover-clone' | |
| 304 | clone.mkdir() | |
| 305 | (clone / 'work').mkdir() | |
| 306 | with patch.dict(os.environ, {'STUDIO_CLOVER_ROOT': str(clone)}): | |
| 307 | state = self.call('state') | |
| 308 | self.assertEqual(state['grants'][0]['path'], str(clone / 'work')) | |
| 309 | self.assertEqual(state['links'][0]['path'], str(clone / 'source')) | |
| 310 | ||
| 311 | def test_no_clobber_rename_keeps_both_folders(self): | |
| 312 | source = self.root / 'source' | |
| 313 | destination = self.root / 'destination' | |
| 314 | source.mkdir() | |
| 315 | destination.mkdir() | |
| 316 | with self.assertRaises(shares.Error): | |
| 317 | shares.rename(source, destination) | |
| 318 | self.assertTrue(source.is_dir()) | |
| 319 | self.assertTrue(destination.is_dir()) | |
| 320 | ||
| 321 | ||
| 322 | if __name__ == '__main__': | |
| 323 | unittest.main() |
tools/copyparty.md created+49| ... | ... | @@ -0,0 +1,49 @@ |
| 1 | # Copyparty administration and folder cutover | |
| 2 | ||
| 3 | Snowglobe's admin Shares page manages native public links and separate API | |
| 4 | accounts. A person's page assigns folder permissions. Existing Copyparty | |
| 5 | permissions remain additive; changing a folder is an explicit retarget or move | |
| 6 | action. New shared folders default to `Share/Name - Description` under Clover. | |
| 7 | ||
| 8 | Copyparty owns its native `cfg/copyparty/shares.db` and selected-file records. | |
| 9 | `cfg/manager.json` stores API account labels, UUID-based folder grants, and link | |
| 10 | labels. API password hashes live in the service's Nomad variable under | |
| 11 | `file_api_<UUID without hyphens>`. Restore these together; restoring only the | |
| 12 | config file cannot recover managed accounts. `prepare.py` regenerates config | |
| 13 | from the canonical service policy and managed state on every deployment. | |
| 14 | ||
| 15 | ## Six-folder production move | |
| 16 | ||
| 17 | `copyparty-cutover.py` contains the requested six source/destination pairs. | |
| 18 | It requires a reviewed plan, absent destinations, and same-filesystem renames. | |
| 19 | The plan binds the host, service, roots, source metadata, and complete native | |
| 20 | share projection. No file contents are read. Public HEAD responses are captured | |
| 21 | before stopping the service and verified after restart; existing keys, | |
| 22 | passwords, permissions, expirations, and selected-file records must agree. | |
| 23 | ||
| 24 | Run the script from the reviewed immutable release on the confirmed target: | |
| 25 | ||
| 26 | ```sh | |
| 27 | umask 077 | |
| 28 | python3 tools/copyparty-cutover.py --plan /root/copyparty-cutover-plan.json | |
| 29 | python3 tools/copyparty-cutover.py --apply --plan /root/copyparty-cutover-plan.json | |
| 30 | ``` | |
| 31 | ||
| 32 | Copyparty remains stopped through all folder and database changes. A mismatch | |
| 33 | restores the old paths and records before restarting the frozen original job. | |
| 34 | Mutation verification has an 80-second cutoff; public readiness has a | |
| 35 | 150-second forward deadline, leaving recovery time within the 285-second | |
| 36 | overall budget. Restricted backups and result evidence live under | |
| 37 | `/var/lib/studio/copyparty-cutover`. Code rollback does not undo this file move. | |
| 38 | ||
| 39 | ## 2026-10-16 staging evidence | |
| 40 | ||
| 41 | Zenith preview `copyparty-preview-9909de81` verified API account scope, | |
| 42 | permission changes, password/session revocation, signed UUID identity grants, | |
| 43 | symlink confinement, selected-file retention, and nested public links. A | |
| 44 | disposable offline move completed in 6.94 seconds. The six requested clone | |
| 45 | folders were then renamed with unchanged inode/metadata manifests: all 20 | |
| 46 | share records and four selected-file rows matched the expected path-only | |
| 47 | projection, and all nine affected public URLs returned their original 200 | |
| 48 | responses within a 78.2-second upper bound. Production source metadata and | |
| 49 | share records remained unchanged during rehearsal. |
tools/dashboard-auth-test.py+11-1| ... | ... | @@ -119,6 +119,7 @@ def main(): |
| 119 | 119 | assert picture_headers['content-type'] == 'image/png' |
| 120 | 120 | request('/api/account/pictures/missing', status=404) |
| 121 | 121 | request('/api/me', status=401, extra={'User-Name': name, 'User-Groups': 'infra-admin'}) |
| 122 | request('/api/shares', status=401, extra={'User-Name': name, 'User-Groups': 'infra-admin'}) | |
| 122 | 123 | request('/auth/status', status=403, extra={'Studio-Proxy-Token': 'wrong'}) |
| 123 | 124 | csrf = request('/auth/status', cookies=cookies)['csrf'] |
| 124 | 125 | csrf_headers = request('/auth/status', cookies=cookies, include_headers=True) |
| ... | ... | @@ -142,6 +143,11 @@ def main(): |
| 142 | 143 | assert 'Max-Age=2592000' in fields['set-cookie'] |
| 143 | 144 | assert 'admin' in request('/api/me', cookies=cookies)['sections'] |
| 144 | 145 | request('/api/users', 'POST', {}, cookies, 403, {'Origin': 'https://evil.example'}) |
| 146 | request('/api/shares/accounts', 'POST', {'name': 'Agent'}, cookies, 403, {'Origin': 'https://evil.example'}) | |
| 147 | preview_cookies = {**cookies, 'view-as': 'media-manage'} | |
| 148 | request('/api/shares', cookies=preview_cookies, status=403) | |
| 149 | request('/api/shares/links', 'POST', {}, preview_cookies, 403) | |
| 150 | request('/api/users', 'POST', {'profile': {'username': 'API-agent', 'email': '', 'firstName': '', 'lastName': ''}, 'groups': [], 'setup': {'kind': 'invite'}}, cookies, 400) | |
| 145 | 151 | assert request('/auth/password', 'POST', {**login, 'next': '//evil.example'}, cookies)['next'] == '/' |
| 146 | 152 | # Import's password format must verify with the original Keycloak parameters. |
| 147 | 153 | with sqlite3.connect(data / 'accounts.sqlite') as db: |
| ... | ... | @@ -196,7 +202,9 @@ def main(): |
| 196 | 202 | finished = request('/auth/file/callback?' + callback.split('?', 1)[1], cookies=file_cookies, status=302, host=file) |
| 197 | 203 | assert finished['location'] == file + '/clover/Public/' |
| 198 | 204 | request('/auth/file/callback?' + callback.split('?', 1)[1], cookies=file_cookies, status=403, host=file) |
| 199 | request('/auth/file/check', cookies=file_cookies, status=204, host=file) | |
| 205 | file_identity = request('/auth/file/check', cookies=file_cookies, status=204, host=file) | |
| 206 | assert file_identity['x-auth-request-preferred-username'] == name | |
| 207 | assert 'snow-user-' + actor in file_identity['x-auth-request-groups'].split(',') | |
| 200 | 208 | request('/auth/file/check', cookies=cookies, status=401, host=file) |
| 201 | 209 | request('/api/me', cookies=file_cookies, status=401) |
| 202 | 210 | request('/auth/file/sign-in?rd=https://evil.example/', status=400, host=file) |
| ... | ... | @@ -206,6 +214,8 @@ def main(): |
| 206 | 214 | request('/auth/setup', 'POST', {'csrf': new_csrf, 'setup': setup, 'email': 'new@example.invalid', 'password': 'another-password'}, newcomer) |
| 207 | 215 | request('/auth/setup', 'POST', {'csrf': new_csrf, 'setup': setup, 'email': 'new@example.invalid', 'password': 'another-password'}, newcomer, 410) |
| 208 | 216 | request('/api/users', cookies=newcomer, status=403) |
| 217 | request('/api/shares', cookies=newcomer, status=403) | |
| 218 | request('/api/shares/grants', 'POST', {}, newcomer, 403) | |
| 209 | 219 | key = {'publicKey': Path('config/admin.pub').read_text().strip(), 'label': 'identity fixture'} |
| 210 | 220 | request('/api/account/signing-keys', 'POST', key, newcomer, 204) |
| 211 | 221 | pending_key = request('/api/account', cookies=newcomer)['commitIdentity']['signingKeys'][0] |
tools/dashboard-host-test.py+14| ... | ... | @@ -78,6 +78,20 @@ class Boundary(unittest.TestCase): |
| 78 | 78 | self.host.handle(request) |
| 79 | 79 | command.assert_not_called() |
| 80 | 80 | |
| 81 | def test_share_management_requires_only_its_declared_context(self): | |
| 82 | request = {"operation": "copyparty.manage", "action": "state", "payload": {}, "users": []} | |
| 83 | with patch.object(host_module.copyparty, "handle", return_value={"links": []}) as handle: | |
| 84 | self.assertEqual(self.host.handle(request), {"links": []}) | |
| 85 | handle.assert_called_once_with(request) | |
| 86 | for changed in [ | |
| 87 | {key: value for key, value in request.items() if key != "users"}, | |
| 88 | {**request, "root": "/srv/private"}, | |
| 89 | {**request, "env": {"NOMAD_TOKEN": "test"}}, | |
| 90 | ]: | |
| 91 | with self.assertRaises(host_module.Rejected): | |
| 92 | self.host.handle(changed) | |
| 93 | self.assertEqual(handle.call_count, 1) | |
| 94 | ||
| 81 | 95 | def test_reclaim_is_always_a_dry_run(self): |
| 82 | 96 | with patch.object(host_module, "command", return_value="reclaim\t1024") as command: |
| 83 | 97 | self.host.handle({"operation": "storage.reclaim", "dataset": "studio-demo/a", |
tools/dashboard-host.py+9-1| ... | ... | @@ -2,7 +2,7 @@ |
| 2 | 2 | import concurrent.futures |
| 3 | 3 | import contextlib |
| 4 | 4 | import fcntl |
| 5 | import importlib | |
| 5 | import importlib.util | |
| 6 | 6 | import json |
| 7 | 7 | import math |
| 8 | 8 | import os |
| ... | ... | @@ -21,6 +21,11 @@ import time |
| 21 | 21 | import vms |
| 22 | 22 | import tty |
| 23 | 23 | |
| 24 | spec = importlib.util.spec_from_file_location("copyparty", Path(__file__).resolve().parent.parent / "service/copyparty/manage.py") | |
| 25 | copyparty = importlib.util.module_from_spec(spec) | |
| 26 | spec.loader.exec_module(copyparty) | |
| 27 | ||
| 28 | ||
| 24 | 29 | dashboard_runs = importlib.import_module("dashboard-run") |
| 25 | 30 | shale_labels = importlib.import_module("shale-labels") |
| 26 | 31 | |
| ... | ... | @@ -90,6 +95,7 @@ class Host: |
| 90 | 95 | "shale.write": {"grant", "repository", "action", "arguments"}, |
| 91 | 96 | "host.sample": set(), |
| 92 | 97 | "host.usage": {"refresh"}, |
| 98 | "copyparty.manage": {"action", "payload", "users"}, | |
| 93 | 99 | **dashboard_runs.FIELDS, |
| 94 | 100 | **{"vm." + action: set() if fields is None else {"payload"} for action, fields in vms.ACTION_FIELDS.items()}, |
| 95 | 101 | } |
| ... | ... | @@ -97,6 +103,8 @@ class Host: |
| 97 | 103 | raise Rejected("Choose a supported host operation.") |
| 98 | 104 | if set(request) != parameters[operation] | {"operation"}: |
| 99 | 105 | raise Rejected("Use only the fields required by this host operation.") |
| 106 | if operation == "copyparty.manage": | |
| 107 | return copyparty.handle(request) | |
| 100 | 108 | if operation == "shale.metadata": |
| 101 | 109 | return shale_labels.metadata(request) |
| 102 | 110 | if operation == "shale.write": |
tools/dashboard-run.py+2-2| ... | ... | @@ -74,7 +74,7 @@ def service(target, key=None): |
| 74 | 74 | raise Error(400, "Choose a secret from this service's list.") |
| 75 | 75 | |
| 76 | 76 | |
| 77 | def nomad(path, method="GET", body=None): | |
| 77 | def nomad(path, method="GET", body=None, timeout=10): | |
| 78 | 78 | token = read(release.STATE / "nomad.token", missing="").strip() |
| 79 | 79 | if not token: |
| 80 | 80 | raise Error(503, "Nomad credentials are unavailable. Check the host service configuration.") |
| ... | ... | @@ -82,7 +82,7 @@ def nomad(path, method="GET", body=None): |
| 82 | 82 | data=json.dumps(body).encode() if body is not None else None, |
| 83 | 83 | headers={"X-Nomad-Token": token, "Content-Type": "application/json"}) |
| 84 | 84 | try: |
| 85 | with urllib.request.urlopen(request, timeout=10) as response: | |
| 85 | with urllib.request.urlopen(request, timeout=timeout) as response: | |
| 86 | 86 | data = response.read(MAX_LOG + 1) |
| 87 | 87 | except urllib.error.HTTPError as error: |
| 88 | 88 | if error.code == 404 and method == "GET": |