gsadmin/garage
1
0
Fork 0
mirror of https://github.com/deuxfleurs-org/garage.git synced 2026-08-06 21:03:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
751 Commits 31 Branches 89 Tags
8033bdb0b4577133cff7d4d90a811ed8f3e13365
Commit Graph

189 Commits

Author SHA1 Message Date
Alex Auvolat 8033bdb0b4 More precisions in errors & small refactoring 2022-05-13 19:36:17 +02:00
Alex Auvolat 5a535788fc Json body for custom errors 2022-05-13 19:28:23 +02:00
Alex Auvolat ea325d78d3 More error refactoring 2022-05-13 19:18:51 +02:00
Alex Auvolat ec16d166f9 Separate error types for k2v and signature 2022-05-13 15:43:44 +02:00
Alex Auvolat 7a5d329e49 More error refactoring 2022-05-13 15:21:32 +02:00
Alex Auvolat f82b938033 Rename error::Error to s3::error::Error 2022-05-13 15:10:52 +02:00
Alex Auvolat 96b11524d5 Error refactoring 2022-05-13 15:04:53 +02:00
Alex Auvolat c0fb9fd0fe Common error type and admin error type that uses it 2022-05-13 14:30:30 +02:00
Alex Auvolat 983037d965 Possibility of different error types for different APIs 2022-05-13 13:51:34 +02:00
Alex Auvolat e4e1f8f0d6 Fix clippy 2022-05-12 17:11:45 +02:00
Alex Auvolat e7ddba53e3 Slightly more detailed error reporting from helper 2022-05-12 17:10:25 +02:00
Alex Auvolat ed76893581 Simplify 2022-05-12 11:21:23 +02:00
Alex Auvolat fc2f73ddb5 BucketAllowKey and BucketDenyKey 2022-05-12 11:19:41 +02:00
Alex Auvolat fe399a3265 DeleteBucket 2022-05-12 11:02:36 +02:00
Alex Auvolat de1a5b87b6 CreateBucket 2022-05-12 10:45:09 +02:00
Alex Auvolat 2b93a01d2b ListBucket and GetBucketInfo 2022-05-12 10:20:34 +02:00
Alex Auvolat 393b76ecba Implement CreateKey, DeleteKey and rudimentary UpdateKey 2022-05-11 11:40:26 +02:00
Alex Auvolat 5c00c9fb46 First key endpoints: ListKeys and GetKeyInfo 2022-05-11 11:10:28 +02:00
Alex Auvolat f97a7845e9 Add API access key admin endpoints 2022-05-11 10:27:40 +02:00
Alex Auvolat dd54d0b2b1 Refactor code for apply/revert, implement Update/Apply/RevertLayout 2022-05-10 13:25:10 +02:00
Alex Auvolat 01c4876fb4 Specify remaining cluster-related endpoints 2022-05-10 13:25:10 +02:00
Alex Auvolat ec03e3d16c Fmt & cleanup 2022-05-10 13:25:10 +02:00
Alex Auvolat 7a19daafbd Implement /status Admin endpoint 2022-05-10 13:25:10 +02:00
Alex Auvolat 633958c7b1 Refactor admin API to be in api/admin and use common code 2022-05-10 13:25:06 +02:00
Alex 5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex Auvolat c9ef3e461b fix clippy 2022-04-19 12:50:40 +02:00
Alex Auvolat c93008d333 Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex Auvolat e5341ca47b Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom a4f9f19ac3 remove size limitation in UploadPartCopy (#280) 
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
 2022-04-19 12:49:43 +02:00
Alex Auvolat 8f9cf3a5d1 fix a clippy lint 2022-03-28 15:48:55 +02:00
Alex Auvolat c1d9854d2c Move block manager to separate module 2022-03-23 10:25:15 +01:00
trinity-1686a 1eb7fdb08f add test framework for arbitraty S3 requests 
and implement some basic test with it
 2022-03-23 10:22:36 +01:00
KokaKiwi e934934f14 garage_api: Update streaming payload stream unit tests 2022-03-23 10:22:36 +01:00
KokaKiwi 98545a16dd garage_api: Handle streaming payload early in request handling 2022-03-23 10:22:36 +01:00
Alex Auvolat ba6b56ae68 Fix some new clippy lints 2022-03-14 12:27:49 +01:00
Alex Auvolat 203e8d2c34 Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex Auvolat d9a35359bf Add metrics to web endpoint 2022-03-14 10:53:50 +01:00
Alex Auvolat 2a5609b292 Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
Alex Auvolat f0d0cd9a20 Remove strum crate dependency; add protobuf nix dependency 2022-03-14 10:53:00 +01:00
Alex Auvolat bb04d94fa9 Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
Alex Auvolat 8c2fb0c066 Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
Alex Auvolat 8a5bbc3b0b More permissive OPTIONS on S3 API 2022-03-01 11:15:16 +01:00
Alex Auvolat 97f245f218 Add tracing output to signature calculation 2022-02-28 12:22:39 +01:00
Alex Auvolat 8129a98291 Process CORS earlier in pipeline 2022-02-28 12:22:39 +01:00
trinity-1686a f6f8b7f1ad Support for PostObject (#222) 
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-02-21 23:02:30 +01:00
Alex Auvolat c99f55c420 Add restriction on part ordering in CompleteMultipartUpload 2022-01-25 12:45:00 +01:00
Alex Auvolat acdf893362 Fix partnumber 2022-01-25 12:25:23 +01:00
Alex Auvolat 338b1b83ee Implement part_number for GetObject 2022-01-24 21:04:42 +01:00
Alex Auvolat 6dab836f3a Multipart improvements 
- support part_number for HeadObject
- add checks in complete_multipart_upload
 2022-01-24 21:04:40 +01:00
Alex Auvolat 513a6b15f9 Handle OPTIONS on website endpoint 2022-01-24 12:32:28 +01:00