Skip to content

feat: add S3/object storage bucket upload support

Ahlad Pataparla requested to merge feat/object-storage-bucket into develop

Summary

This merge request adds S3/object storage bucket upload support to Corpus CLI, allowing users to upload files directly from s3:// URIs.

Changes

S3/Object Storage Integration

  • Add aioboto3 and botocore dependencies for S3 integration
  • Implement S3 download producer for s3:// URI support
  • Add S3 credential management and interactive credential setup
  • Add ProgressTracker for upload progress monitoring
  • Add S3 buffer management and cleanup utilities
  • Update CLI to accept s3:// URIs in upload command
  • Add comprehensive test coverage for S3 upload functionality

Upload Fixes

  • Fix macOS buffer cleanup: delete .DS_Store before rmdir, use shutil.rmtree for final cleanup instead of os.rmdir

Testing

  • Added comprehensive test coverage for S3 upload functionality
  • Fixed mypy type errors and ruff linting issues
  • Updated existing tests to match new code paths

Merge request reports

Loading