| 1 | # gcp-storage-uploads |
| 2 | |
| 3 | An upload pipeline on Cloud Run. A public API stores files in Cloud |
| 4 | Storage; a private indexer picks each upload up from the bucket's |
| 5 | notification, hashes it, and records its metadata in Firestore. |
| 6 | |
| 7 | ``` |
| 8 | PUT /files/:name ──> Api ──> gs://…/uploads/:name |
| 9 | │ OBJECT_FINALIZE |
| 10 | v |
| 11 | Pub/Sub push ──> Indexer ──> Firestore files/:name |
| 12 | GET /files ─────────> Api <────────────────────────── (size, type, sha256) |
| 13 | ``` |
| 14 | |
| 15 | - `src/resources.ts` — the `Uploads` bucket and the `Files` Firestore |
| 16 | database (named, `FIRESTORE_NATIVE`). |
| 17 | - `src/Api.ts` — public `GCP.Function`: |
| 18 | - `PUT /files/:name` stores the body at `uploads/:name` and returns `202`. |
| 19 | - `GET /files/:name` returns the stored bytes. |
| 20 | - `GET /files` lists the indexed metadata. |
| 21 | - `DELETE /files/:name` deletes the object and its metadata. |
| 22 | - `src/Indexer.ts` — private `GCP.Function` consuming `OBJECT_FINALIZE` |
| 23 | events for `uploads/`. It reads the exact object generation the event |
| 24 | names, computes its sha256, and writes `files/:name`. Uploads never wait |
| 25 | on the hash. |
| 26 | |
| 27 | ## Bindings and event sources |
| 28 | |
| 29 | | Host | Binding / event source | IAM granted to the host's service account | |
| 30 | | ------- | ----------------------------------------- | -------------------------------------------------------------------- | |
| 31 | | Api | `GCP.Storage.ReadBucket(bucket)` | `roles/storage.objectViewer` on the bucket | |
| 32 | | Api | `GCP.Storage.WriteBucket(bucket)` | `roles/storage.objectUser` on the bucket | |
| 33 | | Api | `GCP.Firestore.ReadWriteDatabase(files)` | `roles/datastore.user` on the project, conditioned on this database | |
| 34 | | Indexer | `GCP.Storage.ReadBucket(bucket)` | `roles/storage.objectViewer` on the bucket | |
| 35 | | Indexer | `GCP.Firestore.WriteDatabase(files)` | `roles/datastore.user` on the project, conditioned on this database | |
| 36 | | Indexer | `GCP.Storage.consumeBucketEvents(bucket)` | `roles/run.invoker` on the Indexer itself (for the push OIDC token) | |
| 37 | |
| 38 | `consumeBucketEvents` is provided by `GCP.Storage.BucketEventSourceLive` |
| 39 | with `GCP.Run.TopicEventSource` for delivery. Together they create a |
| 40 | Pub/Sub topic, a `JSON_API_V1` bucket notification filtered to |
| 41 | `OBJECT_FINALIZE` under `uploads/` (granting the Cloud Storage service |
| 42 | agent `roles/pubsub.publisher` on the topic), and a push subscription that |
| 43 | calls the Indexer with an OIDC token for its own service account. |
| 44 | |
| 45 | Firestore databases have no resource-level IAM policy, so the Firestore |
| 46 | roles are granted on the project under an IAM Condition that names the |
| 47 | `Files` database. |
| 48 | |
| 49 | ## Deploy |
| 50 | |
| 51 | Requires Docker (the services are built from `main`). Credentials come from your alchemy profile: run `alchemy profile` once and pick GCP (*Service account JSON* for a key file, or *Stored* for an access token or key kept in `~/.alchemy/credentials`, plus a default region), then deploy with `--profile <name>`. |
| 52 | |
| 53 | ```sh |
| 54 | pnpm deploy --profile <name> |
| 55 | ``` |
| 56 | |
| 57 | ```sh |
| 58 | curl -X PUT --data-binary @photo.jpg -H 'content-type: image/jpeg' "$url/files/photo.jpg" |
| 59 | curl "$url/files" # metadata appears once the indexer has run |
| 60 | curl -o out.jpg "$url/files/photo.jpg" |
| 61 | curl -X DELETE "$url/files/photo.jpg" |
| 62 | ``` |
| 63 | |
| 64 | On a fresh deploy the project-level Firestore grants can take a few |
| 65 | minutes to propagate. Until then the indexer's writes fail and Pub/Sub |
| 66 | redelivers them, so metadata shows up late rather than never. |
| 67 | |
| 68 | ## Test |
| 69 | |
| 70 | ```sh |
| 71 | ALCHEMY_PROFILE=<name> bun test |
| 72 | ``` |
| 73 | |
| 74 | Deploys the stack, uploads a text file and a random binary, reads both |
| 75 | back, waits for the indexer's records, checks each sha256 against the |
| 76 | uploaded bytes and each record against the live Firestore document and |
| 77 | object generation, deletes a file, then destroys the stack and checks the |
| 78 | bucket, database, and services are gone. |
| 79 | |
| 80 | ## Destroy |
| 81 | |
| 82 | ```sh |
| 83 | pnpm destroy --profile <name> |
| 84 | ``` |
| 85 | |
| 86 | The bucket uses `forceDestroy: true`, so destroy removes its objects too. |
| 87 | Drop it for real user data. |