Files
pad/internal/models/export.go
T
xarmian a0336e0248 feat(attachments): bundle attachments + manifest in workspace export (TASK-884) (#305)
* feat(attachments): bundle attachments + manifest in workspace export (TASK-884)

GET /workspaces/{ws}/export?format=tar streams a gzip'd tar bundle:

  pad-export.json              # the existing WorkspaceExport JSON
  attachments/manifest.json    # uuid → {filename, mime, size, hash, ...}
  attachments/<uuid>.<ext>     # original blobs only — no thumbnails

Default (no ?format) keeps returning JSON so existing automation
hitting the endpoint without a query param continues to work
unchanged. The CLI's pad workspace export now opts into the bundle
by default; pass --json for the legacy items-only output.

Implementation:
- store.WorkspaceAttachmentsForExport returns originals only
  (parent_id IS NULL); thumbnails are re-derived on import via the
  existing pipeline so shipping them would double the bundle size.
- handleExportWorkspaceBundle streams chunks straight into the
  response writer rather than buffering — a workspace with multi-
  GB of attachments would otherwise pin that much memory.
- AttachmentManifest is versioned (separate from WorkspaceExport
  version) so the bundle layout can evolve independently.
- bundleAttachmentPath is exported (lowercase package fn) so the
  import path in TASK-885 can resolve manifest entries to tar
  entries without duplicating the filename logic.
- CLI gates against writing binary tar.gz to a TTY and appends the
  conventional extension when -o is passed without one.

Tests:
- TestExportBundle_RoundTrip: two uploads → bundle contains
  pad-export.json + manifest + 2 blobs whose bytes match the
  uploads + manifest decodes cleanly + WorkspaceExport decodes.
- TestExportBundle_HidesThumbnails: synthetic thumbnail row, the
  manifest excludes it.
- TestExportBundle_LegacyJSONStillWorks: no ?format param returns
  application/json with a decodable WorkspaceExport (backward
  compat regression guard).

Parent: PLAN-866. TASK-885 (import path + UUID remap) consumes the
manifest produced here.

* fix(attachments): stream export bundle + revert default to JSON per Codex (round 1)

Two findings from Codex on PR #305:

1. CLI buffered the entire response in memory via RawGet → io.ReadAll,
   defeating the server-side streaming design and risking OOM on a
   multi-GB bundle. Added Client.RawStream which copies the response
   body straight into an io.Writer; export now opens the target file
   and streams directly into it.

2. Default tar.gz output broke `pad export → pad import` round trip
   because the import handler still only accepts JSON. Reverted the
   CLI default to JSON; bundle is now opt-in via --bundle. The flag
   docstring notes that TASK-885 will flip the default once import
   handles bundles.

* fix(attachments): surface tar/gzip close errors and truncation per Codex (round 2)

Codex round 2 finding: deferred tw.Close() / gzw.Close() ignored
errors. If a backend returned fewer bytes than size_bytes claimed,
io.Copy returned nil, the tar writer's "missed N bytes" trip fired
at Close, and the handler still completed a 200 OK with a corrupt
bundle that gunzip would later refuse to decompress — silently from
the operator's perspective.

Two changes:
1. The deferred close now logs both tw.Close() and gzw.Close()
   errors with structured context, so a corruption-on-finalize
   trip shows up in the operator log.
2. streamAttachmentToTar checks the bytes-copied count against
   a.SizeBytes after io.Copy and returns a per-attachment error
   when they disagree. The error is logged with attachment_id +
   storage_key so an operator can correlate the corruption with
   the row to investigate.

Regression test: TestExportBundle_TruncatedBlobLogsError forces a
size_bytes/blob desync via direct UPDATE and asserts the resulting
bundle bytes don't decode cleanly. (HTTP status stays 200 because
headers are already on the wire by the time we detect the desync;
that's an inherent limitation of mid-stream errors, but the new
logs + close-error surfacing make the failure observable.)

* fix(attachments): X-Bundle-Status trailer for export-stream success per Codex (round 3)

Codex P1 round 3: even with the per-blob truncation log + tar/gzip
close-error logs, mid-stream failures looked successful to clients.
The CLI's RawStream finished without a transport error, the file
landed on disk, and "Exported workspace" printed regardless of
whether the bundle was actually complete.

Two complementary signals now mark a clean stream:

1. HTTP trailer X-Bundle-Status. The handler declares the trailer
   in the initial Trailer header and sets it to "ok" only after
   tw.Close() and gzw.Close() both return without error. CLI checks
   the trailer after streaming and discards the file + returns
   error if it's absent or non-"ok".

2. The handler skips the deferred clean close on the error path,
   leaving the gzip footer unwritten. A client that ignores the
   trailer (curl, third-party tooling) still sees a corrupt gzip
   stream that gunzip refuses to decompress.

CLI: pad workspace export --bundle now removes any partial output
file on failure rather than leaving a corrupt one behind.
Client.RawStream signature changed to return (bytes, *http.Response,
error) so callers can inspect resp.Trailer; the only caller is the
export command.

Tests: TestExportBundle_TruncatedBlobAbortsStream now asserts both
signals (trailer absent + gzip/tar can't fully decode), and
TestExportBundle_SuccessTrailer pins the happy-path trailer.
2026-04-29 18:30:43 -04:00

128 lines
4.9 KiB
Go

package models
// WorkspaceExport is the complete portable representation of a workspace.
type WorkspaceExport struct {
Version int `json:"version"` // Export format version (1)
ExportedAt string `json:"exported_at"`
Workspace WorkspaceExportMeta `json:"workspace"`
Collections []CollectionExport `json:"collections"`
Items []ItemExport `json:"items"`
Comments []CommentExport `json:"comments,omitempty"`
ItemLinks []ItemLinkExport `json:"item_links,omitempty"`
ItemVersions []ItemVersionExport `json:"item_versions,omitempty"`
}
// AttachmentManifestEntry describes one attachment blob in the
// tar-bundle export's attachments/manifest.json. The bundle layout is:
//
// pad-export.json # the WorkspaceExport above
// attachments/manifest.json # uuid → AttachmentManifestEntry
// attachments/<uuid>.<ext> # the actual blob bytes
//
// Thumbnails are NOT bundled — they're re-derived on import via the
// existing thumbnail pipeline. ParentID and Variant therefore stay
// nil/empty for every entry shipped in a bundle, but the fields are
// kept here for forward compatibility (e.g. if a future format
// version starts shipping pre-derived variants).
type AttachmentManifestEntry struct {
ID string `json:"id"` // attachment UUID (the original)
Filename string `json:"filename"` // user-facing filename
MIME string `json:"mime"` // canonical MIME from upload time
SizeBytes int64 `json:"size_bytes"` // bytes on disk (matches the blob)
ContentHash string `json:"content_hash"` // sha256 hex, the dedupe key
Width *int `json:"width,omitempty"`
Height *int `json:"height,omitempty"`
ItemID string `json:"item_id,omitempty"` // exporter's item UUID; remapped on import
ParentID string `json:"parent_id,omitempty"`
Variant string `json:"variant,omitempty"`
UploadedBy string `json:"uploaded_by"`
CreatedAt string `json:"created_at"`
}
// AttachmentManifest is the top-level shape of attachments/manifest.json
// inside an export bundle. Wraps a list of entries plus a small
// "schema" version so the import path can validate / migrate.
type AttachmentManifest struct {
Version int `json:"version"` // manifest version, 1
Entries []AttachmentManifestEntry `json:"entries"`
}
// WorkspaceExportMeta holds workspace metadata for export.
type WorkspaceExportMeta struct {
Name string `json:"name"`
Slug string `json:"slug"`
Description string `json:"description"`
Settings string `json:"settings"`
}
// CollectionExport holds a collection's data for export.
type CollectionExport struct {
ID string `json:"id"`
Name string `json:"name"`
Slug string `json:"slug"`
Icon string `json:"icon"`
Description string `json:"description"`
Schema string `json:"schema"`
Settings string `json:"settings"`
Prefix string `json:"prefix"`
SortOrder int `json:"sort_order"`
IsDefault bool `json:"is_default"`
IsSystem bool `json:"is_system"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// ItemExport holds an item's data for export.
type ItemExport struct {
ID string `json:"id"`
CollectionID string `json:"collection_id"`
Title string `json:"title"`
Slug string `json:"slug"`
Content string `json:"content"`
Fields string `json:"fields"`
Tags string `json:"tags"`
Pinned bool `json:"pinned"`
SortOrder int `json:"sort_order"`
ParentID string `json:"parent_id,omitempty"`
CreatedBy string `json:"created_by"`
LastModifiedBy string `json:"last_modified_by"`
Source string `json:"source"`
ItemNumber int `json:"item_number"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// CommentExport holds a comment's data for export.
type CommentExport struct {
ID string `json:"id"`
ItemID string `json:"item_id"`
Author string `json:"author"`
Body string `json:"body"`
CreatedBy string `json:"created_by"`
Source string `json:"source"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
// ItemLinkExport holds an item link's data for export.
type ItemLinkExport struct {
ID string `json:"id"`
SourceID string `json:"source_id"`
TargetID string `json:"target_id"`
LinkType string `json:"link_type"`
CreatedBy string `json:"created_by"`
CreatedAt string `json:"created_at"`
}
// ItemVersionExport holds an item version's data for export.
type ItemVersionExport struct {
ID string `json:"id"`
ItemID string `json:"item_id"`
Content string `json:"content"`
ChangeSummary string `json:"change_summary"`
CreatedBy string `json:"created_by"`
Source string `json:"source"`
IsDiff bool `json:"is_diff"`
CreatedAt string `json:"created_at"`
}