ba6b56ae68
Fix some new clippy lints
2022-03-14 12:27:49 +01:00
203e8d2c34
Bump version to 0.7 because of incompatible Netapp
2022-03-14 10:54:24 +01:00
d9a35359bf
Add metrics to web endpoint
2022-03-14 10:53:50 +01:00
2a5609b292
Add metrics to API endpoint
2022-03-14 10:53:36 +01:00
f0d0cd9a20
Remove strum crate dependency; add protobuf nix dependency
2022-03-14 10:53:00 +01:00
bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing
2022-03-14 10:52:30 +01:00
8c2fb0c066
Add tracing integration with opentelemetry
2022-03-14 10:52:13 +01:00
8a5bbc3b0b
More permissive OPTIONS on S3 API
2022-03-01 11:15:16 +01:00
97f245f218
Add tracing output to signature calculation
2022-02-28 12:22:39 +01:00
8129a98291
Process CORS earlier in pipeline
2022-02-28 12:22:39 +01:00
f6f8b7f1ad
Support for PostObject ( #222 )
...
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html )
- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data
Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr >
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com >
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/222
Reviewed-by: Alex <alex@adnab.me >
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com >
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com >
2022-02-21 23:02:30 +01:00
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload
2022-01-25 12:45:00 +01:00
acdf893362
Fix partnumber
2022-01-25 12:25:23 +01:00
338b1b83ee
Implement part_number for GetObject
2022-01-24 21:04:42 +01:00
6dab836f3a
Multipart improvements
...
- support part_number for HeadObject
- add checks in complete_multipart_upload
2022-01-24 21:04:40 +01:00
513a6b15f9
Handle OPTIONS on website endpoint
2022-01-24 12:32:28 +01:00
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general
...
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
2022-01-24 11:58:00 +01:00
820924534a
use clamp instead of min(max())
2022-01-24 11:56:59 +01:00
440374524b
Implement ListParts
2022-01-21 10:42:30 +01:00
fe003d6fbc
Add ListPartsResult structure
2022-01-20 16:38:55 +01:00
e55fa38c99
Add date verification to presigned urls ( #196 )
...
fix #96
fix #162 by returning Forbidden instead Bad Request
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com >
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com >
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com >
2022-01-18 12:22:31 +01:00
178e35f868
refactor s3_router and api_server to make unused Endpoint parameters more obvious
2022-01-17 15:50:24 +01:00
fdcddbe168
Fix Multipart Upload with WinSCP ( #164 ) ( #193 )
...
Closes #164 .
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr >
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr >
2022-01-17 11:18:40 +01:00
b45dcc1925
Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD ( #64 ) ( #156 )
...
Closes #64 .
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr >
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr >
2022-01-17 10:55:31 +01:00
d4dd2e2640
Make use of website config, return error document on error
2022-01-13 14:25:19 +01:00
3ea8ca1b9e
Implement GetBucketWebsite
2022-01-13 14:23:52 +01:00
f7349f4005
Add quotes in returned etags
2022-01-13 14:03:33 +01:00
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures
2022-01-13 14:03:30 +01:00
6617a72220
Implement UploadPartCopy
2022-01-13 13:58:47 +01:00
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy ( fix #187 )
2022-01-13 13:56:55 +01:00
b4592a00fe
Implement ListMultipartUploads ( #171 )
...
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.
It took me some times as I wanted to propose the following things:
- Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156 ) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476 ).
- Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
- An IO closure to make unit tests possibles.
- Unit tests, to track regressions and document how to interact with the code
- Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.
Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.
Some points that we might want to discuss:
- ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
- Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
- Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).
Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr >
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/171
Co-authored-by: Quentin <quentin@dufour.io >
Co-committed-by: Quentin <quentin@dufour.io >
2022-01-12 19:04:55 +01:00
3586c7257c
Clarify error message
2022-01-10 14:10:04 +01:00
7ee11f0eb6
Fix unit tests
2022-01-05 17:34:48 +01:00
168a90dfb5
Fix some error codes
2022-01-05 17:07:36 +01:00
135858d067
Implement DeleteBucket
2022-01-05 16:28:19 +01:00
8395030e48
Implement CreateBucket
2022-01-05 15:56:48 +01:00
677ab60cc1
Small changes in key model and refactoring
2022-01-04 18:59:17 +01:00
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates
2022-01-04 12:53:14 +01:00
1bcd6fabbd
New buckets for 0.6.0: small changes
...
- Fix bucket delete
- fix merge of bucket creation date
- Replace deletable with option in aliases
Rationale: if two aliases point to conflicting bucket, resolving
by making an arbitrary choice risks making data accessible when it
shouldn't be. We'd rather resolve to deleting the alias until
someone puts it back.
2022-01-04 12:52:47 +01:00
2140cd7205
Remove website redirects
2022-01-04 12:52:46 +01:00
beeef4758e
Some movement of helper code and refactoring of error handling
2022-01-04 12:52:46 +01:00
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration
2022-01-04 12:47:28 +01:00
b76d0580a0
Fix forgotten flag
2022-01-04 12:47:28 +01:00
b1cfd16913
New buckets for 0.6.0: small fixes, including:
...
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
same
- fix small remarks by trinity
- don't have a separate website_access field
2022-01-04 12:46:41 +01:00
0bbb6673e7
Model changes
2022-01-04 12:45:52 +01:00
5b1117e582
New model for buckets
2022-01-04 12:45:46 +01:00
ca7b438f3f
less strict ListBuckets ( #178 )
...
fix #175
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com >
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com >
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com >
2021-12-15 15:05:36 +01:00
60d4459926
BucketWebsite ( #174 )
...
fix #77
this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though.
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com >
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/174
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com >
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com >
2021-12-15 10:41:39 +01:00
c4ac8835d3
add proper request router for s3 api ( #163 )
...
fix #161
Current request router was organically grown, and is getting messier and messier with each addition.
This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/161#issuecomment-1773 ) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3).
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com >
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/163
Reviewed-by: Alex <alex@adnab.me >
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com >
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com >
2021-12-06 15:17:47 +01:00
7f26ed55cd
Improved handling of HTTP ranges
...
- correct HTTP code when range syntax is invalid (fix #140 )
- when multiple ranges are given, simply ignore and send whole file
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com >
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/157
Reviewed-by: Alex <alex@adnab.me >
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com >
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com >
2021-11-29 11:52:42 +01:00
Alex Auvolat