feat(attachments): import workspace bundle with rehydrate + UUID remap (TASK-885) (#306)

* feat(attachments): import workspace bundle with attachment rehydrate + UUID remap (TASK-885)

POST /workspaces/import now accepts a tar.gz bundle (Content-Type:
application/gzip) and rebuilds the workspace + attachments + items in
one round trip. JSON imports still work — content-type dispatch in
handleImportWorkspace routes the request.

Three-phase flow:
1. Walk the tar, capture pad-export.json + manifest.json + every
   attachment blob into memory.
2. Run the existing ImportWorkspace path to create the workspace +
   collections + items + comments + links + versions. New IDs are
   generated; item.slug is preserved (the existing remap path
   doesn't re-slugify).
3. For each manifest entry, rehydrate the blob through the storage
   backend (re-validate MIME + re-hash defensively, don't trust the
   manifest), insert a fresh attachments row. Build an oldID→newID
   map keyed on attachment uuid.
4. Walk every imported item's content + fields, replace
   "pad-attachment:OLD" with "pad-attachment:NEW" in one
   transactional pass. Refresh FTS afterward (direct UPDATE bypasses
   triggers).

Phase 2 errors per-attachment are logged and skipped — the workspace
keeps importing rather than rolling back. The import handler returns
the new workspace and the operator can inspect logs for any
attachment that didn't make it.

CLI:
- pad workspace export now defaults to --bundle (.tar.gz) since
  pad import handles bundles. --json reverts to legacy items-only.
- pad import auto-detects format by file extension (.tar.gz / .tgz
  → application/gzip). Other extensions go through the legacy JSON
  path.
- New Client.PostRawWithContentType for explicit-content-type POSTs.

Tests:
- TestImportBundle_RoundTrip: upload → embed in markdown → export
  source → import to FRESH server → verify attachment list has 1
  row with new UUID → item content rewritten to new UUID and old
  UUID is gone → download new blob matches original bytes.
- TestImportBundle_LegacyJSONStillWorks: JSON content-type still
  hits the legacy path.
- TestImportBundle_RejectsBadGzip: garbage gzip body returns 400.

Parent: PLAN-866. With TASK-884 + TASK-885 merged, the round-trip
acceptance criterion (export → import → images intact) is met.

* fix(attachments): stream import end-to-end per Codex (round 1)

Two memory regressions Codex caught on PR #306:

P1 (server). importBundle was buffering every blob into a
map[string][]byte during a first pass, then iterating the manifest
on a second pass. A 2 GiB bundle full of 25 MiB attachments would
pin ~2 GiB of heap. Reworked to single-pass streaming:

  pad-export.json → import workspace + build slug→id map
  attachments/manifest.json → index entries by tar path
  attachments/<uuid>.<ext> → look up entry, rehydrate now

The export bundler always writes pad-export.json + manifest.json
BEFORE any blob (deterministic order from
handlers_export_bundle.go), so this works without buffering. Bundles
that violate the ordering — a third-party tool that writes blobs
first — return 400 with a clear error. Memory footprint now bounded
by the largest single blob (≤25 MiB) regardless of bundle size.

Stale blobs without a manifest entry are skipped (their bytes
io.Copy'd to io.Discard so the tar reader stays in sync). Unknown
top-level entries (forward-compat for future bundle additions) are
also consumed and ignored rather than left dangling.

P2 (CLI). pad import used os.ReadFile, buffering the entire bundle
client-side before posting. Switched to os.Open + a new
Client.PostStreamWithContentType helper that streams the body
directly into the request — together with the server-side fix,
import is end-to-end streaming.

Tests:
- TestImportBundle_RejectsOutOfOrderTar: hand-crafted bundle with
  a blob before pad-export.json returns 400 with "ordering" in
  the message.
- existing TestImportBundle_RoundTrip / LegacyJSONStillWorks /
  RejectsBadGzip continue to pass under the new streaming flow.

* fix(cli): give streaming endpoints a 1h timeout per Codex (round 2)

Codex P1 round 2: PostStreamWithContentType + RawStream were both
using the shared 10s-timeout httpClient. The default works fine for
normal API calls but kills a multi-GiB bundle import or export over
anything slower than a local network — Client.Timeout fires
mid-stream with "Client.Timeout exceeded".

Added a dedicated streamClient on Client with a 1h timeout, used by
both RawStream (export bundle download) and
PostStreamWithContentType (import bundle upload). 1h is generous
enough for ~100 MB/s uplinks shipping a 350 GiB bundle and still
caps a hung connection eventually.

The 10s default stays in place for every other call — short timeouts
are the right SLA for normal API requests and protect the CLI from
hanging on a wedged server.

* fix(attachments): make import bundle cap configurable per Codex (round 3)

Codex P1: the 2 GiB import cap was hard-coded with a comment promising
operator override "later" — but no setter existed, so workspaces over
2 GiB stream out fine on export and fail on re-import.

Added Server.SetImportBundleMaxBytes wired from the
PAD_IMPORT_BUNDLE_MAX_BYTES env var in cmd/pad/main.go. Mirrors the
existing PAD_ATTACHMENT_MAX_BYTES pattern. Default stays at 2 GiB so
the typical workspace works without configuration; operators with
larger exports can raise it without recompiling.

The per-blob cap (importBlobMaxBytes = 25 MiB) is intentionally kept
constant — it bounds in-flight memory regardless of total bundle
size, and a 25 MiB-per-blob ceiling matches the upload handler's
default, so a bundle can never smuggle larger blobs than the upload
endpoint accepts.

* fix(attachments): scale per-blob import cap with PAD_ATTACHMENT_MAX_BYTES per Codex (round 4)

Codex P1 round 4: importBlobMaxBytes was hard-coded at 25 MiB but
the upload handler's per-file cap is configurable via
PAD_ATTACHMENT_MAX_BYTES. An operator who raised the upload cap to
allow 50 MiB attachments could export a workspace successfully
(WorkspaceAttachmentsForExport doesn't gate on size) but the
re-import would reject every blob over 25 MiB.

Replaced the const with effectiveBlobMaxBytes() which reads
s.attachmentMaxBytes (or falls back to defaultAttachmentMaxBytes).
The pad-export.json cap also scales with this value (4×) so a
content-heavy workspace doesn't trip its own JSON ceiling on a
server with raised attachment limits.

Error message on a too-large blob now points the operator at
PAD_ATTACHMENT_MAX_BYTES so they know which knob to turn rather
than digging through code to find the cap.

* fix(attachments): independent metadata cap for bundle import per Codex (round 5)

Codex P2 round 5: tying pad-export.json + manifest.json caps to
PAD_ATTACHMENT_MAX_BYTES regressed deployments that LOWER the
attachment cap. A 1 MiB attachment cap would force metadata to fit
in 4 MiB / 1 MiB respectively — but metadata size scales with
workspace item count, not attachment blob sizes, so a tight upload
limit shouldn't gate it.

Added importMetadataMaxBytes = 100 MiB constant for both metadata
files. effectiveBlobMaxBytes() still drives the per-blob cap which
genuinely tracks attachment-upload policy.
This commit is contained in:
xarmian
2026-04-29 18:56:43 -04:00
committed by GitHub
parent a0336e0248
commit 134f55045d
7 changed files with 958 additions and 22 deletions
+55 -18
View File
@@ -377,6 +377,19 @@ func serveCmd() *cobra.Command {
srv.SetAttachments(attachReg, attachMax)
slog.Info("Attachment storage wired", "backend", "fs", "dir", attachDir)
// Workspace bundle import cap. Default is 2 GiB inside
// internal/server; PAD_IMPORT_BUNDLE_MAX_BYTES lets
// operators with larger exports raise the ceiling without
// recompiling (Codex review on PR #306 round 3).
if v := os.Getenv("PAD_IMPORT_BUNDLE_MAX_BYTES"); v != "" {
if n, perr := strconv.ParseInt(v, 10, 64); perr == nil && n > 0 {
srv.SetImportBundleMaxBytes(n)
slog.Info("Import bundle cap overridden", "max_bytes", n)
} else {
slog.Warn("PAD_IMPORT_BUNDLE_MAX_BYTES ignored — not a positive integer", "value", v)
}
}
// Wire the image processor used for thumbnail derivation
// (TASK-878) and the editor's rotate/crop tools (TASK-879/880).
// The default build picks the pure-Go backend (no cgo);
@@ -4526,24 +4539,23 @@ Examples:
func exportCmd() *cobra.Command {
var outputFile string
var bundle bool
var jsonOnly bool
cmd := &cobra.Command{
Use: "export",
Short: "Export workspace to JSON or a self-contained tar.gz bundle",
Long: `Export the current workspace (collections, items, comments, versions)
to a portable JSON file. Pass --bundle to include attachment blobs in
a tar.gz bundle:
Short: "Export workspace as a self-contained tar.gz bundle",
Long: `Export the current workspace (collections, items, comments, versions,
and attachments) to a portable tar.gz bundle:
pad-export.json workspace metadata + items + collections + ...
attachments/manifest.json uuid {filename, mime, size, content_hash}
attachments/<uuid>.<ext> original attachment blobs
The default stays JSON for now so existing pad import flows keep
working. The bundle becomes the default once pad import learns to
unpack tar.gz (planned for TASK-885).`,
Pass --json to emit the legacy items-only JSON file (no attachments).
Both formats can be re-imported via 'pad workspace import'.`,
RunE: func(cmd *cobra.Command, args []string) error {
client, _ := getClient()
ws := getWorkspace()
bundle := !jsonOnly
path := "/workspaces/" + ws + "/export"
defaultExt := ".json"
@@ -4614,7 +4626,7 @@ unpack tar.gz (planned for TASK-885).`,
},
}
cmd.Flags().StringVarP(&outputFile, "output", "o", "", "output file path (default: stdout)")
cmd.Flags().BoolVar(&bundle, "bundle", false, "emit a tar.gz bundle including attachment blobs (TASK-885 will make this the default once pad import handles bundles)")
cmd.Flags().BoolVar(&jsonOnly, "json", false, "emit legacy items-only JSON (no attachments)")
return cmd
}
@@ -4624,31 +4636,56 @@ func importCmd() *cobra.Command {
var nameFlag string
cmd := &cobra.Command{
Use: "import <file>",
Short: "Import workspace from JSON export",
Long: `Import a workspace from a previously exported JSON file. Creates a new workspace with regenerated IDs.`,
Args: cobra.ExactArgs(1),
Short: "Import workspace from JSON export or tar.gz bundle",
Long: `Import a workspace from a previously exported file. Creates a new
workspace with regenerated IDs.
Accepts both formats produced by 'pad workspace export':
- .json (legacy, items only)
- .tar.gz (new bundle, includes attachment blobs)
Format is detected by file extension. Override workspace name with --name.`,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
client, _ := getClient()
filePath := args[0]
data, err := os.ReadFile(filePath)
if err != nil {
return fmt.Errorf("read file: %w", err)
}
path := "/workspaces/import"
if nameFlag != "" {
path += "?name=" + nameFlag
}
// Detect bundle by extension. .tar.gz / .tgz route through
// the gzip-stream import path; everything else goes the
// legacy JSON route. We don't sniff magic bytes — extension
// is the explicit signal the user gave us.
contentType := "application/json"
low := strings.ToLower(filePath)
if strings.HasSuffix(low, ".tar.gz") || strings.HasSuffix(low, ".tgz") {
contentType = "application/gzip"
}
// Stream the file rather than reading it all in. A multi-
// GiB bundle would otherwise pin that much memory client-
// side before the first byte hits the wire — defeats the
// server's streaming import (Codex P2 on PR #306 round 1).
f, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("open file: %w", err)
}
defer f.Close()
var ws models.Workspace
if err := client.PostRaw(path, data, &ws); err != nil {
if err := client.PostStreamWithContentType(path, f, contentType, &ws); err != nil {
return fmt.Errorf("import: %w", err)
}
fmt.Printf("Imported workspace %q (slug: %s)\n", ws.Name, ws.Slug)
fmt.Printf(" Collections: imported\n")
fmt.Printf(" Items, comments, links, versions: imported\n")
if contentType == "application/gzip" {
fmt.Printf(" Attachments: rehydrated from bundle\n")
}
fmt.Printf(" All IDs regenerated\n")
return nil
},
+48 -4
View File
@@ -18,8 +18,14 @@ import (
type Client struct {
baseURL string
httpClient *http.Client
authToken string // session or API token, sent as Authorization: Bearer
agentName string // optional agent name, sent as X-Pad-Agent header
// streamClient has a much longer timeout than httpClient and is
// used by RawStream / PostStreamWithContentType for endpoints
// that can transfer multi-GiB payloads (workspace export
// bundles). Sharing the default 10s timeout would kill those
// transfers mid-flight on anything but a local network.
streamClient *http.Client
authToken string // session or API token, sent as Authorization: Bearer
agentName string // optional agent name, sent as X-Pad-Agent header
}
func NewClient(host string, port int) *Client {
@@ -34,6 +40,17 @@ func NewClientFromURL(baseURL string) *Client {
httpClient: &http.Client{
Timeout: 10 * time.Second,
},
// Long-running transfer client for streaming endpoints
// (workspace export bundles in/out, future S3 downloads).
// 10s on the default client is the right SLA for normal API
// calls but kills a multi-GiB bundle upload mid-stream over
// anything but a fast local link. 1 hour is generous enough
// for ~100 MB/s uplinks shipping a 350 GiB bundle and still
// caps a hung connection eventually. (Codex review on PR
// #306 round 2.)
streamClient: &http.Client{
Timeout: 1 * time.Hour,
},
}
// Auto-load credentials if available
@@ -417,7 +434,7 @@ func (c *Client) RawStream(path string, w io.Writer) (int64, *http.Response, err
if err != nil {
return 0, nil, err
}
resp, err := c.httpClient.Do(req)
resp, err := c.streamClient.Do(req)
if err != nil {
return 0, nil, fmt.Errorf("request failed: %w", err)
}
@@ -431,11 +448,19 @@ func (c *Client) RawStream(path string, w io.Writer) (int64, *http.Response, err
// PostRaw sends raw bytes to the API and decodes the JSON response.
func (c *Client) PostRaw(path string, data []byte, result interface{}) error {
return c.PostRawWithContentType(path, data, "application/json", result)
}
// PostRawWithContentType is the explicit-content-type variant of
// PostRaw. Used by the bundle import path to send a tar.gz as
// application/gzip so the server's content-type dispatch routes the
// request to the bundle handler instead of the JSON decoder.
func (c *Client) PostRawWithContentType(path string, data []byte, contentType string, result interface{}) error {
req, err := c.newRequest("POST", path, bytes.NewReader(data))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Content-Type", contentType)
resp, err := c.httpClient.Do(req)
if err != nil {
return fmt.Errorf("request failed: %w", err)
@@ -444,6 +469,25 @@ func (c *Client) PostRaw(path string, data []byte, result interface{}) error {
return c.handleResponse(resp, result)
}
// PostStreamWithContentType POSTs a streaming body (typically an
// *os.File for a multi-GiB bundle import) without buffering the full
// payload in memory client-side. Mirrors the server's streaming
// import path — together they keep import memory bounded by the
// largest single blob (~25 MiB) rather than the full bundle size.
func (c *Client) PostStreamWithContentType(path string, body io.Reader, contentType string, result interface{}) error {
req, err := c.newRequest("POST", path, body)
if err != nil {
return err
}
req.Header.Set("Content-Type", contentType)
resp, err := c.streamClient.Do(req)
if err != nil {
return fmt.Errorf("request failed: %w", err)
}
defer resp.Body.Close()
return c.handleResponse(resp, result)
}
// --- Auth API ---
// LoginResponse is the response from POST /auth/login.
+437
View File
@@ -0,0 +1,437 @@
package server
import (
"archive/tar"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"strings"
"github.com/PerpetualSoftware/pad/internal/attachments"
"github.com/PerpetualSoftware/pad/internal/models"
)
// defaultImportBundleMaxBytes caps an uploaded bundle. Mirrors the
// upload handler's defaultAttachmentMaxBytes scaling: a workspace
// export can contain thousands of attachments, so this is much
// higher than any single-file upload limit. The cap exists primarily
// to bound the temp-file footprint on the import host. Operators
// running larger workspaces should override via
// Server.SetImportBundleMaxBytes (wired from PAD_IMPORT_BUNDLE_MAX_BYTES
// in cmd/pad/main.go).
const defaultImportBundleMaxBytes int64 = 2 << 30 // 2 GiB
// importMetadataMaxBytes is the size ceiling for the small JSON
// payloads inside a bundle (pad-export.json + attachments/manifest.json).
// Independent of the per-blob cap so a deployment that LOWERS
// PAD_ATTACHMENT_MAX_BYTES (e.g. to 1 MiB for a tightly-controlled
// host) doesn't inadvertently reject metadata for a workspace
// nobody intended to gate on attachment-blob limits. (Codex P2 on
// PR #306 round 5.)
//
// 100 MiB comfortably holds a workspace with many thousands of
// items + version history. Bumped via the import bundle cap, not
// per-attachment cap, since metadata size scales with the export's
// item count, not attachment sizes.
const importMetadataMaxBytes int64 = 100 << 20 // 100 MiB
// effectiveBlobMaxBytes returns the per-blob ceiling for bundle
// import — matches whatever the upload handler accepts so an
// operator who raised PAD_ATTACHMENT_MAX_BYTES on the source can
// re-import the resulting export on a destination configured the
// same way. Codex flagged the hard-coded 25 MiB cap on PR #306
// round 4: a workspace with attachments uploaded under a larger
// cap would round-trip the export but fail the re-import.
func (s *Server) effectiveBlobMaxBytes() int64 {
if s.attachmentMaxBytes > 0 {
return s.attachmentMaxBytes
}
return defaultAttachmentMaxBytes
}
// handleImportWorkspaceBundle accepts a tar.gz bundle produced by
// the export endpoint and rebuilds the workspace including all
// attachment blobs.
//
// The handler does the JSON / bundle dispatch; the actual work is
// done by importBundle which is unit-testable without the http stack.
//
// Bundle layout (matches handlers_export_bundle.go):
//
// pad-export.json
// attachments/manifest.json
// attachments/<uuid>.<ext>
//
// Two-phase flow:
// 1. Parse pad-export.json, run the existing ImportWorkspace path to
// create the workspace + items. Returns an item-ID map (old → new).
// 2. For each manifest entry, find the matching tar entry, rehydrate
// the blob through the storage backend (re-validate MIME + hash),
// and insert an attachment row pointed at the remapped item.
// Build an attachment-ID map (old → new) as we go.
// 3. Scan all imported items' content + fields for pad-attachment:OLD
// references and rewrite to pad-attachment:NEW.
//
// Errors before phase 2 begins return a clean HTTP error. Errors mid-
// rehydrate are logged with attachment_id context; the workspace is
// kept (it has live items) and the partial attachment state is left
// for the operator to inspect. Orphan GC will eventually reclaim any
// blob whose row insertion failed — the upload-handler's "blob may be
// orphan on disk" comment applies here too.
func (s *Server) handleImportWorkspaceBundle(w http.ResponseWriter, r *http.Request) {
if s.attachments == nil {
writeError(w, http.StatusServiceUnavailable, "attachments_disabled",
"Attachment storage is not configured on this server")
return
}
// Bound the request body BEFORE the gzip reader spools any of it.
maxBytes := s.importBundleMaxBytes
if maxBytes <= 0 {
maxBytes = defaultImportBundleMaxBytes
}
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
gz, err := gzip.NewReader(r.Body)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_bundle",
"Could not read gzip stream: "+err.Error())
return
}
defer gz.Close()
newName := r.URL.Query().Get("name")
userID := currentUserID(r)
ws, err := s.importBundle(r.Context(), gz, newName, userID)
if err != nil {
// Errors from importBundle are already shaped with status hints —
// surface as 400 unless the underlying error wraps an http hint.
var statusErr *importStatusError
if errors.As(err, &statusErr) {
writeError(w, statusErr.status, statusErr.code, statusErr.message)
return
}
writeError(w, http.StatusBadRequest, "import_failed", err.Error())
return
}
// Mirror the JSON-import path's owner-attachment so the workspace
// shows up under the importer's account.
if userID != "" {
_ = s.store.AddWorkspaceMember(ws.ID, userID, "owner")
}
writeJSON(w, http.StatusCreated, ws)
}
// importBundle reads a tar (already gzip-decompressed) from r and
// orchestrates the two-phase import. Returns the new workspace.
//
// Single-pass streaming: the export bundler always writes
// pad-export.json + attachments/manifest.json BEFORE any blob, so
// we can run ImportWorkspace + parse the manifest as soon as those
// two entries land, then stream-rehydrate each subsequent blob
// without ever holding the full bundle in memory. Bundles that
// violate the ordering — e.g. a third-party tool that put blobs
// first — are rejected with a clear error.
//
// Memory footprint: at most one blob (≤ effectiveBlobMaxBytes) held
// at a time during rehydration, plus the small JSON payloads at the
// front. A 2 GiB bundle with thousands of 25 MiB images now needs
// ~25 MiB peak rather than ~2 GiB. (Codex P1 on PR #306 round 1.)
//
// Split out from the handler so tests can drive it with a tar.Reader
// over an in-memory bundle and assert on the resulting state without
// a live HTTP server.
func (s *Server) importBundle(ctx context.Context, r io.Reader, newName, ownerID string) (*models.Workspace, error) {
tr := tar.NewReader(r)
blobCap := s.effectiveBlobMaxBytes()
var ws *models.Workspace
var manifestByPath map[string]*models.AttachmentManifestEntry
var oldItemIDToSlug, slugToNewID map[string]string
oldAttachToNew := map[string]string{}
exportSeen := false
manifestSeen := false
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return nil, fmt.Errorf("read tar entry: %w", err)
}
if hdr.Typeflag != tar.TypeReg && hdr.Typeflag != tar.TypeRegA { //nolint:staticcheck // TypeRegA accepted for older bundles
continue
}
switch {
case hdr.Name == "pad-export.json":
// pad-export.json can grow large for content-heavy
// workspaces (items + version history). Use the
// metadata-specific cap so deployments that lower
// PAD_ATTACHMENT_MAX_BYTES (e.g. to 1 MiB) don't
// inadvertently make metadata fail.
if hdr.Size > importMetadataMaxBytes {
return nil, fmt.Errorf("pad-export.json exceeds %d-byte cap (declared %d)", importMetadataMaxBytes, hdr.Size)
}
buf, err := readEntry(tr, hdr.Size)
if err != nil {
return nil, fmt.Errorf("read pad-export.json: %w", err)
}
var export models.WorkspaceExport
if err := json.Unmarshal(buf, &export); err != nil {
return nil, &importStatusError{
status: http.StatusBadRequest, code: "bad_bundle",
message: "Bundle pad-export.json could not be decoded: " + err.Error(),
}
}
ws, err = s.store.ImportWorkspace(&export, newName, ownerID)
if err != nil {
return nil, fmt.Errorf("import workspace: %w", err)
}
oldItemIDToSlug = make(map[string]string, len(export.Items))
for _, it := range export.Items {
oldItemIDToSlug[it.ID] = it.Slug
}
slugToNewID, err = s.store.WorkspaceItemSlugMap(ws.ID)
if err != nil {
return ws, fmt.Errorf("build slug→id map: %w", err)
}
exportSeen = true
case hdr.Name == "attachments/manifest.json":
if !exportSeen {
return nil, &importStatusError{
status: http.StatusBadRequest, code: "bad_bundle",
message: "Bundle ordering violation: manifest.json before pad-export.json",
}
}
// Manifest size scales with attachment count, not blob
// content, so use the metadata cap rather than the
// per-blob one (same rationale as pad-export.json above).
if hdr.Size > importMetadataMaxBytes {
return ws, fmt.Errorf("manifest.json exceeds %d-byte cap (declared %d)",
importMetadataMaxBytes, hdr.Size)
}
buf, err := readEntry(tr, hdr.Size)
if err != nil {
return ws, fmt.Errorf("read manifest.json: %w", err)
}
var manifest models.AttachmentManifest
if err := json.Unmarshal(buf, &manifest); err != nil {
return ws, fmt.Errorf("manifest decode: %w (workspace created but attachments not restored)", err)
}
if manifest.Version > exportBundleVersion {
return ws, fmt.Errorf("manifest version %d not supported by this server (max %d)",
manifest.Version, exportBundleVersion)
}
manifestByPath = make(map[string]*models.AttachmentManifestEntry, len(manifest.Entries))
for i := range manifest.Entries {
e := &manifest.Entries[i]
manifestByPath[bundleAttachmentPath(e.ID, e.Filename)] = e
}
manifestSeen = true
case strings.HasPrefix(hdr.Name, "attachments/"):
if !exportSeen {
return nil, &importStatusError{
status: http.StatusBadRequest, code: "bad_bundle",
message: "Bundle ordering violation: attachment blob before pad-export.json",
}
}
if !manifestSeen {
return ws, &importStatusError{
status: http.StatusBadRequest, code: "bad_bundle",
message: "Bundle ordering violation: attachment blob before manifest.json",
}
}
entry, ok := manifestByPath[hdr.Name]
if !ok {
// Blob has no manifest entry — could be a stale entry
// from a bundle the operator hand-edited. Skip the
// bytes (consume the tar slot) and move on.
if _, err := io.Copy(io.Discard, io.LimitReader(tr, hdr.Size)); err != nil {
return ws, fmt.Errorf("skip unmanifested blob %s: %w", hdr.Name, err)
}
continue
}
if hdr.Size > blobCap {
return ws, fmt.Errorf("blob %s exceeds %d-byte cap (declared %d) — raise PAD_ATTACHMENT_MAX_BYTES on this server to allow",
hdr.Name, blobCap, hdr.Size)
}
blob, err := readEntry(tr, hdr.Size)
if err != nil {
return ws, fmt.Errorf("read blob %s: %w", hdr.Name, err)
}
newAttID, err := s.rehydrateAttachment(ctx, ws.ID, entry, blob,
oldItemIDToSlug, slugToNewID, ownerID)
if err != nil {
slog.Warn("import: rehydrate failed",
"attachment_id", entry.ID, "error", err)
continue
}
oldAttachToNew[entry.ID] = newAttID
default:
// Unknown top-level entry — consume it so the tar reader
// stays in sync, then forward-compat ignore. Future
// bundle versions might add a CHANGELOG.md or schema
// migration script we don't recognize yet.
if _, err := io.Copy(io.Discard, io.LimitReader(tr, hdr.Size)); err != nil {
return ws, fmt.Errorf("skip unknown entry %s: %w", hdr.Name, err)
}
}
}
if !exportSeen {
return nil, &importStatusError{
status: http.StatusBadRequest, code: "bad_bundle",
message: "Bundle is missing pad-export.json",
}
}
// Phase 3: rewrite pad-attachment:OLD references in every imported
// item's content + fields to pad-attachment:NEW. Done via store
// helper so we get a single transactional pass and the FTS index
// is updated correctly.
if len(oldAttachToNew) > 0 {
if err := s.store.RemapAttachmentReferencesInWorkspace(ws.ID, oldAttachToNew); err != nil {
slog.Warn("import: attachment reference remap failed",
"workspace_id", ws.ID, "error", err)
// Non-fatal — items still exist with stale references.
// Operator can re-run a remap manually if needed.
}
}
// Drop the storage-usage cache — the imported attachments
// just bumped the workspace total.
s.storageInfoCache.invalidate(ws.ID)
return ws, nil
}
// readEntry reads exactly size bytes from a tar reader (the rest of
// the current entry) into a buffer, validating that the read length
// matches the header's declared Size. Tar entries are bounded by the
// caller; this helper just makes the read+verify pattern uniform.
func readEntry(tr *tar.Reader, size int64) ([]byte, error) {
buf, err := io.ReadAll(io.LimitReader(tr, size+1))
if err != nil {
return nil, err
}
if int64(len(buf)) != size {
return nil, fmt.Errorf("read %d bytes, header says %d", len(buf), size)
}
return buf, nil
}
// rehydrateAttachment runs the upload-handler's MIME validation,
// hash, and store.Put for one manifest entry, then inserts a fresh
// attachments row in the new workspace. Returns the new UUID. The
// new row points at the remapped item id when the original was
// attached to one; orphan attachments stay orphaned.
func (s *Server) rehydrateAttachment(
ctx context.Context,
workspaceID string,
entry *models.AttachmentManifestEntry,
blob []byte,
oldItemIDToSlug, slugToNewID map[string]string,
ownerID string,
) (string, error) {
// Defense in depth: re-validate the MIME against the allowlist on
// the first 512 bytes. Trusting the manifest's mime field would
// let a malicious bundle smuggle a blocked type past the upload
// gate. If the actual bytes don't sniff to the manifest's mime,
// trust the sniffed value (matches the upload handler's policy).
head := blob
if len(head) > 512 {
head = head[:512]
}
allowed, code, vErr := attachments.ValidateUpload(head, entry.Filename)
if vErr != nil {
return "", fmt.Errorf("mime validation (%s): %w", code, vErr)
}
// Hash the blob ourselves rather than trusting the manifest. A
// bundle could lie about content_hash; the storage layer's
// hash-verify guards us at write time, but hashing locally lets
// the dedupe path work even when the supplied hash is wrong.
sum := sha256.Sum256(blob)
hash := hex.EncodeToString(sum[:])
// Hand the bytes to the configured backend. Same path the upload
// handler uses; FSStore re-hashes defensively.
store, err := s.attachments.Resolve(attachments.FSPrefix + ":" + hash)
if err != nil {
return "", fmt.Errorf("resolve attachment store: %w", err)
}
storageKey, err := store.Put(ctx, hash, allowed.MIME, strings.NewReader(string(blob)))
if err != nil {
return "", fmt.Errorf("store.Put: %w", err)
}
// Translate the old item id (from the manifest) into the new id
// via item.slug, which ImportWorkspace preserves.
var newItemIDPtr *string
if entry.ItemID != "" {
if slug, ok := oldItemIDToSlug[entry.ItemID]; ok {
if newID, ok := slugToNewID[slug]; ok && newID != "" {
newItemIDPtr = &newID
}
}
}
uploadedBy := entry.UploadedBy
if uploadedBy == "" {
uploadedBy = ownerID
}
if uploadedBy == "" {
uploadedBy = "system"
}
att := &models.Attachment{
WorkspaceID: workspaceID,
ItemID: newItemIDPtr,
UploadedBy: uploadedBy,
StorageKey: storageKey,
ContentHash: hash,
MimeType: allowed.MIME,
SizeBytes: int64(len(blob)),
Filename: entry.Filename,
Width: entry.Width,
Height: entry.Height,
}
if err := s.store.CreateAttachment(att); err != nil {
return "", fmt.Errorf("create attachment row: %w", err)
}
// Re-derive thumbnails for image originals. Mirrors the upload
// handler — runs async via goAsync so the import handler doesn't
// stall on imaging work, and Server.Stop() waits for in-flight
// derivation before close.
if allowed.Category == attachments.CategoryImage && s.imageProcessor != nil {
original := att.ID
s.goAsync(func() { s.deriveThumbnails(original) })
}
return att.ID, nil
}
// importStatusError lets importBundle return errors with HTTP-status
// hints attached, so the handler doesn't have to repeat the
// classification. Keeps importBundle pure-Go-testable.
type importStatusError struct {
status int
code string
message string
}
func (e *importStatusError) Error() string { return e.message }
@@ -0,0 +1,261 @@
package server
import (
"archive/tar"
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/PerpetualSoftware/pad/internal/attachments"
"github.com/PerpetualSoftware/pad/internal/models"
)
// TestImportBundle_RoundTrip pins TASK-885's acceptance criterion:
// export from one workspace → import into a fresh server → items
// keep their pad-attachment:UUID references intact (rewritten to
// the new attachment ids), the blobs are reachable through the
// download endpoint, and the byte content matches the original
// upload.
//
// The most realistic test of a feature that touches three layers
// (export tar, import dispatch, attachment-reference remap). If any
// stage drops a UUID or fails to rewrite content, the final item
// won't render the image and this test catches it.
func TestImportBundle_RoundTrip(t *testing.T) {
// 1. Source workspace: upload an attachment, attach it to an
// item, embed the pad-attachment: reference in the item's
// markdown content.
src, srcSlug := testServerWithAttachments(t)
body := realPNG()
rr := doMultipartUpload(src, srcSlug, "logo.png", body)
if rr.Code != http.StatusCreated {
t.Fatalf("upload: %d %s", rr.Code, rr.Body.String())
}
var upload struct {
ID string `json:"id"`
URL string `json:"url"`
}
if err := json.Unmarshal(rr.Body.Bytes(), &upload); err != nil {
t.Fatalf("decode upload: %v", err)
}
// Create an item that references the attachment in markdown.
itemContent := fmt.Sprintf("Hello world\n\n![logo](pad-attachment:%s)\n", upload.ID)
rr = doRequest(src, "POST", "/api/v1/workspaces/"+srcSlug+"/collections/docs/items",
map[string]any{"title": "With Image", "content": itemContent})
if rr.Code != http.StatusCreated {
t.Fatalf("create item: %d %s", rr.Code, rr.Body.String())
}
// 2. Export the source workspace as a bundle.
rr = doRequest(src, "GET", "/api/v1/workspaces/"+srcSlug+"/export?format=tar", nil)
if rr.Code != http.StatusOK {
t.Fatalf("export: %d %s", rr.Code, rr.Body.String())
}
bundle := rr.Body.Bytes()
// 3. Spin up a fresh server (independent storage) and import the
// bundle into it. Using a separate server is what proves the
// UUID remap actually works — same-server import would
// accidentally pass even if the remap were broken.
dest, _ := testServerWithAttachments(t)
req := httptest.NewRequest("POST", "/api/v1/workspaces/import?name=Imported",
bytes.NewReader(bundle))
req.Header.Set("Content-Type", "application/gzip")
req.RemoteAddr = "127.0.0.1:1234"
rr = httptest.NewRecorder()
dest.ServeHTTP(rr, req)
if rr.Code != http.StatusCreated {
t.Fatalf("import: status=%d body=%s", rr.Code, rr.Body.String())
}
var newWS models.Workspace
if err := json.Unmarshal(rr.Body.Bytes(), &newWS); err != nil {
t.Fatalf("decode new ws: %v", err)
}
// 4. The destination workspace must have a new attachments table
// with the rehydrated row, and the imported item's content
// must reference the NEW attachment id (not the old one).
rr = doRequest(dest, "GET", "/api/v1/workspaces/"+newWS.Slug+"/attachments", nil)
if rr.Code != http.StatusOK {
t.Fatalf("list attachments: %d %s", rr.Code, rr.Body.String())
}
var attResp struct {
Attachments []struct {
ID string `json:"id"`
Filename string `json:"filename"`
SizeBytes int64 `json:"size_bytes"`
} `json:"attachments"`
}
if err := json.Unmarshal(rr.Body.Bytes(), &attResp); err != nil {
t.Fatalf("decode att list: %v", err)
}
if len(attResp.Attachments) != 1 {
t.Fatalf("imported attachments: got %d, want 1", len(attResp.Attachments))
}
newAttID := attResp.Attachments[0].ID
if newAttID == upload.ID {
t.Fatalf("attachment id was NOT remapped (got %s, original %s)", newAttID, upload.ID)
}
if attResp.Attachments[0].SizeBytes != int64(len(body)) {
t.Errorf("imported attachment size=%d, want %d",
attResp.Attachments[0].SizeBytes, len(body))
}
if attResp.Attachments[0].Filename != "logo.png" {
t.Errorf("imported attachment filename=%q, want logo.png",
attResp.Attachments[0].Filename)
}
// 5. Item content must reference the NEW attachment id (rewrite
// pass worked) and NOT the old one. Read the imported item
// via the docs collection's items endpoint.
rr = doRequest(dest, "GET", "/api/v1/workspaces/"+newWS.Slug+"/collections/docs/items", nil)
if rr.Code != http.StatusOK {
t.Fatalf("list items: %d %s", rr.Code, rr.Body.String())
}
var items []models.Item
if err := json.Unmarshal(rr.Body.Bytes(), &items); err != nil {
t.Fatalf("decode items: %v body=%s", err, rr.Body.String())
}
var imported *models.Item
for i := range items {
if items[i].Title == "With Image" {
imported = &items[i]
break
}
}
if imported == nil {
t.Fatalf("imported item not found; got %d items", len(items))
}
if !strings.Contains(imported.Content, "pad-attachment:"+newAttID) {
t.Errorf("imported content missing new attachment ref %s; content=%q",
newAttID, imported.Content)
}
if strings.Contains(imported.Content, "pad-attachment:"+upload.ID) {
t.Errorf("imported content still has stale old attachment ref %s; content=%q",
upload.ID, imported.Content)
}
// 6. Download the rehydrated blob and confirm bytes match the
// original upload. This is the strongest guarantee that the
// storage backend correctly received the bytes from the bundle.
rr = doRequest(dest, "GET",
"/api/v1/workspaces/"+newWS.Slug+"/attachments/"+newAttID, nil)
if rr.Code != http.StatusOK {
t.Fatalf("download imported blob: %d", rr.Code)
}
if !bytes.Equal(rr.Body.Bytes(), body) {
t.Errorf("imported blob differs from original upload (got %d bytes, want %d)",
rr.Body.Len(), len(body))
}
}
// TestImportBundle_LegacyJSONStillWorks confirms the JSON dispatch
// still works alongside the new bundle path. Hits the same endpoint
// with JSON content-type — must route to the existing handler.
func TestImportBundle_LegacyJSONStillWorks(t *testing.T) {
srv, slug := testServerWithAttachments(t)
if rr := doRequest(srv, "POST", "/api/v1/workspaces/"+slug+"/collections/docs/items",
map[string]any{"title": "Plain", "content": "no attachments"}); rr.Code != http.StatusCreated {
t.Fatalf("create item: %d", rr.Code)
}
rr := doRequest(srv, "GET", "/api/v1/workspaces/"+slug+"/export", nil)
if rr.Code != http.StatusOK {
t.Fatalf("export json: %d", rr.Code)
}
jsonExport := rr.Body.Bytes()
// Import into a fresh server using JSON content-type.
dest, _ := testServerWithAttachments(t)
req := httptest.NewRequest("POST", "/api/v1/workspaces/import?name=JsonImport",
bytes.NewReader(jsonExport))
req.Header.Set("Content-Type", "application/json")
req.RemoteAddr = "127.0.0.1:1234"
rr = httptest.NewRecorder()
dest.ServeHTTP(rr, req)
if rr.Code != http.StatusCreated {
t.Fatalf("legacy JSON import: status=%d body=%s", rr.Code, rr.Body.String())
}
}
// TestImportBundle_RejectsOutOfOrderTar pins the streaming-import
// invariant added in PR #306 round 1 (Codex P1): the bundle MUST
// place pad-export.json + manifest.json BEFORE any blob so the
// server can stream-rehydrate without buffering. Bundles that put
// blobs first would still work in the previous implementation but
// would force buffering of every blob — we now reject them up front
// with a clear error rather than silently buffering.
func TestImportBundle_RejectsOutOfOrderTar(t *testing.T) {
srv, _ := testServerWithAttachments(t)
// Hand-craft a tar.gz where a blob entry comes BEFORE
// pad-export.json. Strict-format violation.
var buf bytes.Buffer
gzw := gzip.NewWriter(&buf)
tw := tar.NewWriter(gzw)
if err := tw.WriteHeader(&tar.Header{Name: "attachments/abcd.png", Mode: 0o644, Size: 4}); err != nil {
t.Fatalf("write header: %v", err)
}
if _, err := tw.Write([]byte{0xde, 0xad, 0xbe, 0xef}); err != nil {
t.Fatalf("write blob: %v", err)
}
// Even if we'd write a manifest after, the blob coming first
// already violates the contract. Stop here.
tw.Close()
gzw.Close()
req := httptest.NewRequest("POST", "/api/v1/workspaces/import", bytes.NewReader(buf.Bytes()))
req.Header.Set("Content-Type", "application/gzip")
req.RemoteAddr = "127.0.0.1:1234"
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, req)
if rr.Code != http.StatusBadRequest {
t.Errorf("out-of-order bundle: status=%d, want 400; body=%s", rr.Code, rr.Body.String())
}
if !strings.Contains(rr.Body.String(), "ordering") {
t.Errorf("expected ordering-violation error, got body=%s", rr.Body.String())
}
}
// TestImportBundle_RejectsBadGzip pins the early-error path: a
// truncated/invalid gzip body must return 400, not 500. Catches
// regressions where the gzip-reader error gets swallowed and the
// handler proceeds with a half-decompressed stream.
func TestImportBundle_RejectsBadGzip(t *testing.T) {
srv := testServer(t)
// Wire the attachment registry so the dispatcher doesn't 503.
srv.SetAttachments(attachments.NewRegistry(), 0)
garbage := []byte("not a gzip stream")
req := httptest.NewRequest("POST", "/api/v1/workspaces/import",
bytes.NewReader(garbage))
req.Header.Set("Content-Type", "application/gzip")
req.RemoteAddr = "127.0.0.1:1234"
rr := httptest.NewRecorder()
srv.ServeHTTP(rr, req)
if rr.Code != http.StatusBadRequest {
t.Errorf("bad gzip: status=%d, want 400; body=%s", rr.Code, rr.Body.String())
}
}
// readBundleAsBytes is a tiny helper for callers that want the raw
// bundle to feed straight into another POST. Kept separate from
// readBundle (which extracts a map) so import-side tests don't have
// to re-encode the bundle.
func readBundleAsBytes(t *testing.T, body io.Reader) []byte {
t.Helper()
out, err := io.ReadAll(body)
if err != nil {
t.Fatalf("read bundle: %v", err)
}
return out
}
+20
View File
@@ -343,6 +343,26 @@ func (s *Server) handleExportWorkspace(w http.ResponseWriter, r *http.Request) {
}
func (s *Server) handleImportWorkspace(w http.ResponseWriter, r *http.Request) {
// Content-Type dispatch:
// application/gzip / application/x-gzip / application/x-tar
// → tar.gz bundle path (TASK-885) — handles attachments.
// anything else → JSON path (legacy items-only).
//
// We prefer Content-Type over file-magic sniffing so a misnamed
// upload fails fast with a clear error rather than silently going
// through the wrong code path. The CLI's pad import command sets
// the right header based on the file extension; web UI does the
// same when uploading a .tar.gz.
ct := strings.TrimSpace(r.Header.Get("Content-Type"))
if i := strings.IndexByte(ct, ';'); i >= 0 {
ct = ct[:i]
}
ct = strings.ToLower(strings.TrimSpace(ct))
if ct == "application/gzip" || ct == "application/x-gzip" || ct == "application/x-tar" {
s.handleImportWorkspaceBundle(w, r)
return
}
var data models.WorkspaceExport
// WorkspaceExport contains all collections, items, comments, and item
// versions for the workspace — even a modest project export blows past
+17
View File
@@ -79,6 +79,13 @@ type Server struct {
// guarding.
storageInfoCache *storageInfoCache
// importBundleMaxBytes caps a single workspace import bundle.
// 0 → defaultImportBundleMaxBytes (2 GiB). Set via
// SetImportBundleMaxBytes from cmd/pad/main.go using the
// PAD_IMPORT_BUNDLE_MAX_BYTES env var so operators with larger
// exports can opt in without recompiling.
importBundleMaxBytes int64
// bg tracks fire-and-forget goroutines spawned by request handlers
// (TouchUserActivity in middleware_auth, async email sends, etc.) so
// the server can drain them before shutdown / test cleanup. Without
@@ -279,6 +286,16 @@ func (s *Server) SetImageProcessor(p attachments.Processor) {
s.imageProcessor = p
}
// SetImportBundleMaxBytes overrides the default 2 GiB cap on a
// single workspace import bundle. Set to 0 to fall back to the
// default. Wired from PAD_IMPORT_BUNDLE_MAX_BYTES in cmd/pad/main.go
// so operators with workspaces over 2 GiB can opt in without
// recompiling. Larger caps trade memory headroom (one blob in
// flight at a time, ≤25 MiB) for a longer import wall-clock.
func (s *Server) SetImportBundleMaxBytes(n int64) {
s.importBundleMaxBytes = n
}
// SetSecureCookies enables the Secure flag on all cookies.
func (s *Server) SetSecureCookies(secure bool) {
s.secureCookies = secure
+120
View File
@@ -554,6 +554,126 @@ func (s *Store) SoftDeleteAttachment(id string) error {
return nil
}
// WorkspaceItemSlugMap returns slug → id for every live item in a
// workspace. Used by the bundle import path (TASK-885) to remap an
// old attachment's item_id (from the manifest) to the freshly-
// generated id, via item.slug which ImportWorkspace preserves.
//
// Soft-deleted items are excluded; the import path can't realistically
// recreate an attachment under a deleted parent without also
// resurrecting the parent, and the manifest's item_id only has
// meaning for live items at export time.
func (s *Store) WorkspaceItemSlugMap(workspaceID string) (map[string]string, error) {
rows, err := s.db.Query(s.q(`
SELECT id, slug FROM items WHERE workspace_id = ? AND deleted_at IS NULL
`), workspaceID)
if err != nil {
return nil, fmt.Errorf("workspace item slug map: %w", err)
}
defer rows.Close()
out := map[string]string{}
for rows.Next() {
var id, slug string
if err := rows.Scan(&id, &slug); err != nil {
return nil, fmt.Errorf("scan slug map: %w", err)
}
out[slug] = id
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate slug map: %w", err)
}
return out, nil
}
// RemapAttachmentReferencesInWorkspace rewrites every
// "pad-attachment:OLD" reference in items.content + items.fields
// to "pad-attachment:NEW" for every (old, new) pair in the map.
// Run after a bundle import has rehydrated attachments so item
// content points at the new attachment ids instead of the source
// workspace's ids.
//
// Implementation: a single transaction that loads every item's
// content+fields, runs strings.ReplaceAll for each pair, and
// writes back only when something changed. ReplaceAll is safe
// because attachment UUIDs don't appear as substrings of other
// UUIDs (RFC4122 hex, length 36, all unique by construction).
//
// FTS reindex via the existing rebuild helper happens AFTER the
// transaction commits — direct UPDATE bypasses the SQLite FTS
// triggers the same way ImportWorkspace's INSERTs do.
func (s *Store) RemapAttachmentReferencesInWorkspace(workspaceID string, oldToNew map[string]string) error {
if len(oldToNew) == 0 {
return nil
}
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("begin remap tx: %w", err)
}
defer tx.Rollback()
rows, err := tx.Query(s.q(`SELECT id, content, fields FROM items WHERE workspace_id = ? AND deleted_at IS NULL`), workspaceID)
if err != nil {
return fmt.Errorf("scan items for remap: %w", err)
}
type rowUpdate struct {
id string
content string
fields string
}
var updates []rowUpdate
for rows.Next() {
var id, content, fields string
if err := rows.Scan(&id, &content, &fields); err != nil {
rows.Close()
return fmt.Errorf("scan item: %w", err)
}
newContent := remapAttachmentRefs(content, oldToNew)
newFields := remapAttachmentRefs(fields, oldToNew)
if newContent != content || newFields != fields {
updates = append(updates, rowUpdate{id: id, content: newContent, fields: newFields})
}
}
if err := rows.Err(); err != nil {
rows.Close()
return fmt.Errorf("iterate items for remap: %w", err)
}
rows.Close()
for _, u := range updates {
if _, err := tx.Exec(s.q(`UPDATE items SET content = ?, fields = ? WHERE id = ?`),
u.content, u.fields, u.id); err != nil {
return fmt.Errorf("update item %s: %w", u.id, err)
}
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("commit remap: %w", err)
}
// Refresh FTS so search queries see the rewritten content.
s.rebuildFTSForWorkspace(workspaceID)
return nil
}
// remapAttachmentRefs replaces "pad-attachment:OLD" with
// "pad-attachment:NEW" for every (old, new) pair in the map. Pure
// string operation — kept private so callers go through
// RemapAttachmentReferencesInWorkspace which also handles the FTS
// reindex.
//
// The "pad-attachment:" prefix is part of the search key so we don't
// accidentally rewrite a UUID that happens to appear in unrelated
// content (e.g. an item title that mentions an attachment id by
// accident — unlikely but free to guard against).
func remapAttachmentRefs(s string, oldToNew map[string]string) string {
for old, fresh := range oldToNew {
if old == "" || fresh == "" || old == fresh {
continue
}
s = strings.ReplaceAll(s, "pad-attachment:"+old, "pad-attachment:"+fresh)
}
return s
}
// WorkspaceAttachmentsForExport returns every original (non-derived,
// non-deleted) attachment in the workspace so the export bundler
// can stream them into the tar. Derived rows (thumbnails) are