Files
pad/internal/store/dialect.go
T
xarmian 0fd5d0cdfb fix: green up Go (PostgreSQL) CI (BUG-842) (#275)
* fix(store): swap plainto_tsquery → websearch_to_tsquery for PG FTS (BUG-842)

`TestListItems_FTS_HyphenatedSearchTerm/task-five` has been failing on
every Go (PostgreSQL) CI run because `plainto_tsquery('english',
'task-five')` doesn't match the asciihword lexeme(s) the english parser
produces for an indexed `task-five-distinctive`. The result is that
every PG full-text search for hyphenated terms returns zero rows.

`websearch_to_tsquery` (Postgres 11+) is purpose-built for arbitrary
user input and tokenizes hyphenated terms the same way `to_tsvector`
does for the indexed document, so the query intersects the index
correctly. Swapped in three spots in the postgres dialect — FTSMatch,
FTSSnippet, FTSRank — and updated the caller-side comments that
referenced plainto_tsquery. SQLite path is unchanged: it goes through
items_fts MATCH with sanitizeFTSQuery, never through these methods.

* fix(server): drain background goroutines on Stop() (BUG-842)

`TestAdminBillingStats_SidecarSidecarError_DegradesToLocalOnly` (and
other server tests) have been flaking on the Go (PostgreSQL) CI runner
with `TempDir RemoveAll cleanup: directory not empty`. Root cause:
several request handlers spawned bare `go func() { ... }()` goroutines
that touched the SQLite WAL DB after the test function returned.
testServer's t.Cleanup closed the store but had no way to drain those
goroutines first, so a fire-and-forget WAL write could re-create the
`-wal`/`-shm` files between Close() and t.TempDir's RemoveAll.

Add a Server.bg sync.WaitGroup, a Server.goAsync helper that wraps a
WaitGroup-tracked goroutine, and a Server.Stop() that blocks until
every goAsync closure has finished. Convert the four known
fire-and-forget sites to goAsync:

- middleware_auth.go (TouchUserActivity)
- handlers_auth.go   (password reset email)
- handlers_cloud.go  (stripe_processed_events pruning)
- handlers_members.go (workspace invitation email)

Wire `srv.Stop()` into both testServer (server_test.go) and
newMetricsTestServer (metrics_auth_test.go) so cleanup order is
Stop → Close → TempDir RemoveAll. Add
TestServer_Stop_DrainsBackgroundGoroutines to pin the contract: a
goAsync goroutine must block Stop until it returns.

* fix(store): correct PG FTS hyphenation via OR-combined plainto_tsquery (BUG-842)

The previous attempt swapped plainto_tsquery → websearch_to_tsquery,
which was wrong: websearch_to_tsquery treats `-` as a NEGATION operator
(Google-style), so `task-five` becomes `task & !five` and the search
returns 0 rows for the same reason as before. This commit reverts the
swap and applies the actual fix.

PG's english parser indexes `task-five-distinctive` as
`{task-five-distinct, task, five, distinct}` — the asciihword AND its
parts. plainto_tsquery applied to the partial query `task-five`
produces `task-fiv & task & five`: the stemmed asciihword for the
PARTIAL query (`task-fiv`) is NOT in the vector, so the AND fails.

Replacing the hyphen with a space makes plainto emit `task & five`,
which DOES match — but doing that unconditionally breaks `BUG-842`-
style queries: PG indexes the `-842` suffix as a negative-number
lexeme, so `plainto_tsquery('BUG-842')` matches via `-842`, while
`plainto_tsquery('BUG 842')` searches for `842` and misses.

The fix ORs the two query variants together so the search vector is
matched against either the raw user query OR its hyphen-as-space form.
Both `task-five` (against `task-five-distinctive`) and `BUG-842`
(against `BUG-842 fix the cleanup race`) hit. Verified locally against
postgres:17-alpine via PAD_TEST_POSTGRES_URL — both 10x stress and
race-detector runs are green.

Surfaces:
  - dialect.go: FTSMatch / FTSSnippet / FTSRank now consume TWO
    placeholders each in the PG dialect.
  - items.go: listItemsFTS PG branch + SearchItems PG branch update
    args to pass (raw, sanitized) for every PG `?` placeholder.
  - search.go: SearchItems main / count / facets PG branches updated
    likewise. New sanitizePGFTSQuery helper alongside sanitizeFTSQuery.
  - documents.go: ListDocuments PG branch updated.

Tests:
  - TestListItems_FTS_HyphenatedSearchTerm extended with a `BUG-842`
    case to pin the OR-combined logic — naive hyphen-stripping would
    silently regress this.
  - New TestSanitizePGFTSQuery unit test.

* chore: gofmt 11 files with import-order issues (BUG-842 PR cleanup)

The Go (SQLite) CI job has been failing on `main` (and every PR built
against it) because golangci-lint flags 11 files whose third-party
imports are intermixed with internal imports — the import-grouping
rule that gofmt enforces. None of these were introduced by the
BUG-842 PR; they're pre-existing on main. The PR can't go green
without this cleanup, though, so it's bundled here.

Pure mechanical change — `gofmt -w <files>` only re-orders import
groups; no logic changes. Files touched:

  cmd/pad/configure.go
  cmd/pad/main.go
  internal/cli/format.go
  internal/server/handlers_admin_invitations.go
  internal/server/handlers_admin_users.go
  internal/server/handlers_grants.go
  internal/server/handlers_share_links.go
  internal/server/handlers_stars.go
  internal/server/middleware_auth.go
  internal/store/store.go
  internal/store/store_test.go

After this commit `gofmt -l ./cmd ./internal` returns clean.
2026-04-28 16:21:43 -04:00

298 lines
9.6 KiB
Go

package store
import (
"fmt"
"strings"
)
// DriverType identifies the database backend.
type DriverType string
const (
DriverSQLite DriverType = "sqlite"
DriverPostgres DriverType = "postgres"
)
// Dialect encapsulates SQL syntax differences between database backends.
// The Store calls dialect methods to generate backend-specific SQL fragments.
type Dialect interface {
// Driver returns the driver type.
Driver() DriverType
// Placeholder returns the nth parameter placeholder (1-indexed).
// SQLite: "?", PostgreSQL: "$1", "$2", etc.
Placeholder(n int) string
// Rebind converts a query with "?" placeholders to the dialect's format.
// For SQLite this is a no-op. For PostgreSQL, "?" becomes "$1", "$2", etc.
Rebind(query string) string
// JSONExtractText returns SQL to extract a text value from a JSON column.
// SQLite: json_extract(col, '$.key')
// PostgreSQL: col->>'key'
JSONExtractText(column, key string) string
// JSONExtractPath returns SQL to extract a value at a dotted path from a JSON column.
// SQLite: json_extract(col, '$.path.to.key')
// PostgreSQL: col #>> '{path,to,key}'
JSONExtractPath(column, path string) string
// JSONSet returns SQL to set a value at a path in a JSON column.
// SQLite: json_set(col, '$.key', ?)
// PostgreSQL: jsonb_set(col::jsonb, '{key}', ?::jsonb)
// Returns the SQL fragment and any extra placeholders used.
JSONSet(column, key string) string
// JSONRemove returns SQL to remove a key from a JSON column.
// SQLite: json_remove(col, '$.key')
// PostgreSQL: col::jsonb - 'key'
JSONRemove(column, key string) string
// Now returns the SQL expression for the current UTC timestamp.
// SQLite: datetime('now')
// PostgreSQL: NOW() AT TIME ZONE 'UTC'
Now() string
// NowRFC3339 returns the SQL expression for current UTC time in RFC3339 format.
// SQLite: strftime('%Y-%m-%dT%H:%M:%SZ', 'now')
// PostgreSQL: TO_CHAR(NOW() AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
NowRFC3339() string
// GroupConcat returns SQL for string aggregation with a separator.
// SQLite: GROUP_CONCAT(DISTINCT expr)
// PostgreSQL: STRING_AGG(DISTINCT expr, ',')
GroupConcat(expr string, distinct bool) string
// BoolToInt converts a Go bool to a query parameter value.
// SQLite: 0/1 (integers)
// PostgreSQL: true/false (native booleans)
BoolToInt(b bool) interface{}
// ILike returns the case-insensitive LIKE operator.
// SQLite: LIKE (case-insensitive by default)
// PostgreSQL: ILIKE
ILike() string
// Concat returns SQL to concatenate string expressions.
// SQLite: expr1 || expr2
// PostgreSQL: expr1 || expr2 (same, but useful as abstraction point)
Concat(exprs ...string) string
// FTSMatch returns the full-text search WHERE clause fragment.
// SQLite: "table MATCH ?" — consumes ONE arg.
// PostgreSQL: an OR-combined plainto_tsquery match that consumes TWO
// args — the raw user query AND its hyphen-sanitized form. See
// sanitizePGFTSQuery for the rationale (BUG-842).
FTSMatch(table, column string) string
// FTSSnippet returns SQL for highlighted search result snippets.
// SQLite: snippet(fts_table, col_idx, '<mark>', '</mark>', '...', 32)
// — consumes ONE arg.
// PostgreSQL: ts_headline backed by the same OR-combined tsquery as
// FTSMatch — consumes TWO args (raw, sanitized).
FTSSnippet(ftsTable string, colIndex int, sourceColumn string) string
// FTSRank returns the column/expression for full-text relevance ranking.
// SQLite: rank (built-in FTS5 column) — consumes ZERO args.
// PostgreSQL: ts_rank backed by the same OR-combined tsquery as
// FTSMatch — consumes TWO args (raw, sanitized).
FTSRank(table, column string) string
// JSONArrayContains returns SQL + the arg to check if a JSON array column
// contains a given text value.
// SQLite: "column LIKE ?" with arg `%"value"%`
// PostgreSQL: "column::jsonb @> ?::jsonb" with arg `["value"]`
JSONArrayContains(column, value string) (string, interface{})
}
// ---------- SQLite dialect ----------
type sqliteDialect struct{}
func (d *sqliteDialect) Driver() DriverType { return DriverSQLite }
func (d *sqliteDialect) Placeholder(_ int) string { return "?" }
func (d *sqliteDialect) Rebind(query string) string { return query }
func (d *sqliteDialect) JSONExtractText(column, key string) string {
return fmt.Sprintf("json_extract(%s, '$.%s')", column, key)
}
func (d *sqliteDialect) JSONExtractPath(column, path string) string {
return fmt.Sprintf("json_extract(%s, '$.%s')", column, path)
}
func (d *sqliteDialect) JSONSet(column, key string) string {
return fmt.Sprintf("json_set(%s, '$.%s', ?)", column, key)
}
func (d *sqliteDialect) JSONRemove(column, key string) string {
return fmt.Sprintf("json_remove(%s, '$.%s')", column, key)
}
func (d *sqliteDialect) Now() string {
return "datetime('now')"
}
func (d *sqliteDialect) NowRFC3339() string {
return "strftime('%Y-%m-%dT%H:%M:%SZ', 'now')"
}
func (d *sqliteDialect) GroupConcat(expr string, distinct bool) string {
if distinct {
return fmt.Sprintf("GROUP_CONCAT(DISTINCT %s)", expr)
}
return fmt.Sprintf("GROUP_CONCAT(%s)", expr)
}
func (d *sqliteDialect) BoolToInt(b bool) interface{} {
if b {
return 1
}
return 0
}
func (d *sqliteDialect) ILike() string { return "LIKE" }
func (d *sqliteDialect) Concat(exprs ...string) string {
return strings.Join(exprs, " || ")
}
func (d *sqliteDialect) FTSMatch(table, _ string) string {
return fmt.Sprintf("%s MATCH ?", table)
}
func (d *sqliteDialect) FTSSnippet(ftsTable string, colIndex int, _ string) string {
return fmt.Sprintf("snippet(%s, %d, '<mark>', '</mark>', '...', 32)", ftsTable, colIndex)
}
func (d *sqliteDialect) FTSRank(_, _ string) string {
return "rank"
}
func (d *sqliteDialect) JSONArrayContains(column, value string) (string, interface{}) {
return column + " LIKE ?", "%\"" + value + "\"%"
}
// ---------- PostgreSQL dialect ----------
type postgresDialect struct{}
func (d *postgresDialect) Driver() DriverType { return DriverPostgres }
func (d *postgresDialect) Placeholder(n int) string {
return fmt.Sprintf("$%d", n)
}
func (d *postgresDialect) Rebind(query string) string {
return rebindQuery(query)
}
func (d *postgresDialect) JSONExtractText(column, key string) string {
return fmt.Sprintf("%s->>'%s'", column, key)
}
func (d *postgresDialect) JSONExtractPath(column, path string) string {
parts := strings.Split(path, ".")
return fmt.Sprintf("%s #>> '{%s}'", column, strings.Join(parts, ","))
}
func (d *postgresDialect) JSONSet(column, key string) string {
return fmt.Sprintf("jsonb_set(COALESCE(%s, '{}')::jsonb, '{%s}', to_jsonb(?::text))", column, key)
}
func (d *postgresDialect) JSONRemove(column, key string) string {
return fmt.Sprintf("(%s::jsonb - '%s')", column, key)
}
func (d *postgresDialect) Now() string {
return "(NOW() AT TIME ZONE 'UTC')"
}
func (d *postgresDialect) NowRFC3339() string {
return "TO_CHAR(NOW() AT TIME ZONE 'UTC', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')"
}
func (d *postgresDialect) GroupConcat(expr string, distinct bool) string {
if distinct {
return fmt.Sprintf("STRING_AGG(DISTINCT %s, ',')", expr)
}
return fmt.Sprintf("STRING_AGG(%s, ',')", expr)
}
func (d *postgresDialect) BoolToInt(b bool) interface{} {
return b
}
func (d *postgresDialect) ILike() string { return "ILIKE" }
func (d *postgresDialect) Concat(exprs ...string) string {
return strings.Join(exprs, " || ")
}
// PG FTS uses an OR-combined plainto_tsquery to handle hyphenated user
// queries correctly. For an indexed title `task-five-distinctive`, the
// english parser writes `task-five-distinct, task, five, distinct`. A
// raw `plainto_tsquery('english', 'task-five')` produces
// `task-fiv & task & five` — the stemmed asciihword `task-fiv` is NOT in
// the vector, so the AND fails and the search returns 0 rows. Replacing
// the hyphen with a space (`'task five'`) makes plainto emit
// `task & five`, which DOES match. We can't unconditionally do that,
// though: titles like `BUG-842` are indexed as `bug, -842` (negative
// number lexeme) — `plainto_tsquery('BUG-842')` matches them via `-842`,
// but `plainto_tsquery('BUG 842')` would search for the lexeme `842` and
// miss. ORing the two query variants together covers both cases. See
// BUG-842 and sanitizePGFTSQuery.
//
// Each method below consumes TWO `?` placeholders (raw query, sanitized
// query). Callers pass them in args via sanitizePGFTSQueryArgs.
func (d *postgresDialect) FTSMatch(table, column string) string {
return fmt.Sprintf(
"%s.%s @@ (plainto_tsquery('english', ?) || plainto_tsquery('english', ?))",
table, column,
)
}
func (d *postgresDialect) FTSSnippet(_ string, _ int, sourceColumn string) string {
return fmt.Sprintf(
"ts_headline('english', %s, plainto_tsquery('english', ?) || plainto_tsquery('english', ?), 'StartSel=<mark>,StopSel=</mark>,MaxFragments=1,MaxWords=32')",
sourceColumn,
)
}
func (d *postgresDialect) FTSRank(table, column string) string {
return fmt.Sprintf(
"ts_rank(%s.%s, plainto_tsquery('english', ?) || plainto_tsquery('english', ?))",
table, column,
)
}
func (d *postgresDialect) JSONArrayContains(column, value string) (string, interface{}) {
return column + "::jsonb @> ?::jsonb", `["` + value + `"]`
}
// ---------- Helper ----------
// rebindQuery converts "?" placeholders to PostgreSQL's "$1", "$2", etc.
// Respects string literals (single quotes) and does not modify "?" inside them.
func rebindQuery(query string) string {
var buf strings.Builder
buf.Grow(len(query) + 16)
n := 0
inString := false
for i := 0; i < len(query); i++ {
ch := query[i]
if ch == '\'' {
inString = !inString
buf.WriteByte(ch)
} else if ch == '?' && !inString {
n++
fmt.Fprintf(&buf, "$%d", n)
} else {
buf.WriteByte(ch)
}
}
return buf.String()
}