garage/doc/drafts/garbage-collector.md

# [RFC] Garbage Collector Elimination

## Statement of problem and prior art

Currently, Garage's garbage collector has a few identified issues.
Namely, it can only be run if all nodes in a partition are currently online,
it may not be correct in front of a rebalancing (this is partially mitigated by a 24h delay added to tombstone deletion),
and it isn't resilient to a subset of nodes being restored from snapshots.
It's not clear if it is possible to implement a garbage collection process that can eliminate tombstones, but also support
a node rollback to a point in time where a key existed.

This problematic, perhaps unsurprisingly, maps very well to the general abstraction of CRDTs for sets, where a whole partition
would be a single CRDT. The semantic required in Garage demands reinsertion of deleted keys, which excludes the most simple
forms of sets such as G-Sets and 2P-Sets. As the goal is to not handle garbage collection of tombstones, a standard ORSet
is also unfitting. In fact it could be argued Garage already uses something akin to an ORSet with a garbage collector today.

There exists CRDTs supporting this feature-set, one of which is an OptORSet[^1].
It only needs a set-wide metadata proportional in the number of writers, and a per alive-key metadata proportional in the number
of writers to that key (but no metadata for dead keys).
These metadata are akin to DVVs[^2]. An element is considered new if its DVV comes causaly after the current DVV of the set.
Correctness of this algorithm depends however on having causal delivery, which isn't given in Garage, neither in general nor in
presence of snapshot restoration.

## Proposal

We devise a new kind of CRDT based on the core ideas of an OptORSet, but replacing each version inside its DVVs with a list of
range of observed updates, which we name a Seen Vector (SV). Such metadata can in the worst case grow linearly with the number of
insertion. In practice, assuming all elements are eventually known to every replica, the storage requirement of a SV is equivalent
to that of a standard DVV. Under causal delivery, a SV degenerates into a standard DVV.

To add an element to the set, a node increments its own version counter, and sends an update containing (element, replica, version).
On receiving such an update, a node checks if it has already seen this particular (replica, version), and if so ignore it.
If it hasn't seen that update, it saves the new element, and update its SV to include that new (replica, version).

TODO: describe formaly the algorithm


Algorithm 1, Seen Vector:
```
payload set S    -- S: set of triple (replica i, timestamp s, timestamp e)

initial ∅
query seen (replica i, timestamp c): boolean b
  let b = (∃s <= c,e > c: (i,s,e) ∈ S)

update increment ()
  prepare ()
    let r = myID()
    let t = e|∀s,s', ∄e' > e: (r,s,e) ∈ S, (r,s',e') ∈ S    -- t is the maximum end bound for this replica
  effect(r, t)
    if ¬seen (r, t) then
      if seen (r, t - 1) ∧ seen (r, t + 1) then
        let R = {∃s: (r, s, t) ∈ S}
        let R' = {∃e: (r, t + 1, e) ∈ S}
        let M = S \ R
        let M' = M \ R'
        S := M' ∪ {(r, s, e)}
      else if seen (r, t - 1)then
        let R = {∃s: (r, s, t) ∈ S}
        let M = S \ R
        S := M ∪ {(r, s, t+1)}
      else if seen (r, t + 1)then
        let R = {∃e: (r, t + 1, e) ∈ S}
        let M = S \ R
        E := E ∪ {(r, t, e)}
      else
        E := E ∪ {(r, t, t+1)}

merge (B)
  # TODO this is correct, but largelly suboptimal. We should perform the increment.effect subroutine for all elements of B instead
  S := S ∪ B.S
```

Algorithm 2, OptORSet with SV. This algorithm is largely copied and adapted from Figure 3 of [^1]
```
payload set E, SV sv  -- E: elements, set of triples (element e, timestamp c, replica i)
                      -- sv: SeenVector of received triples

initial ∅, ∅
query contains (element e) : boolean b
  let b = (∃c, i : (e, c, i) ∈ E)

query elements () : set S
  let S = {e|∃c, i : (e, c, i) ∈ E}

update add (element e)
  prepare (e)
    let r = myID() -- r = source replica
    let c = sv.increment.prepare().t
  effect (e, c, r)
    if ¬sv.seen(r, c) then
    let O = {(e, c′, r) ∈ E|c′ < c}
    sv.increment.effect(r, c)
    E := E ∪ {(e, c, r)} \ O

update remove (element e)
  prepare (e) -- Collect all unique triples containing e
    let R = {(e, c, i) ∈ E}
  effect (R) -- Remove triples observed at source
    pre causal delivery
    E := E \ R

merge (B)
  let M = (E ∩ B.E)
  let M ′ = {(e, c, i) ∈ E \ B.E| ¬B.sv.seen(i, c)}
  let M ′′ = {(e, c, i) ∈ B.E \ E| ¬sv.seen(i, c)}
  let U = M ∪ M ′ ∪ M ′′
  let O = {(e, c, i) ∈ U |∃(e, c′, i) ∈ U : c < c′}
  E := U \ O
  sv := sv.merge(B.sv)
```

## Storage evaluation

As stated, if all updates are received, the SV is similar in size to that of a standard DVV. It may be however that a node create and
immediately delete an element, creating holes in its sequence from the point of view of other replicas. These holes could be filled
through an interactive process where the replica observing holes asks the node to scan over the whole set, and for each version in these
hole, reply if no element has that exact version number. Holes caused by existing elements should be eventually fixed by an anti-entropy
process, so replying with these elements appears unnecessary.

The per-element storage requirement is proportional to the number of replicas having modified that element, even under non steady-state.
This happens because as we always exchange whole elements, we have causal delivery for individual keys.

## Replica version rollback

This scheme assumes the same node won't issue the same version twice, which isn't a given when a node might be rollback to a previous state.
The author proposes that on initialization, a replica asks all other replicas for the highest version number they know for it.
If all replicas reply with a number less than or equal to the current version, it is safe to reuse the currently known number.
If some replicas reply with a number higher, the node increase its version to that number.
If at least one replica doesn't reply, it can't make any assumption about its actual version number.
The node then increment a generation number, which is made part of its replica id, starts a new sequence from zero.

## Appendix: providing SV to the underlying elements

Some more complexe elements may want to have access to a version id and the Seen Vector to perform their own internal merge operations.
The author reckon this may help implementing S3 versioning, by giving a simple way for Objects to know if an ObjectVersion was yet
unknown or is known and already deleted.

## References
 	
[^1]: An optimized conflict-free replicated set, https://doi.org/10.48550/arXiv.1210.3368
[^2]: Dotted Version Vectors: Logical Clocks for Optimistic Replication, https://doi.org/10.48550/arXiv.1011.5808