Implement Duplicate Submission Detection and Marking

Description: We need to detect and mark duplicate submissions to maintain data quality and integrity. This feature should work across all supported media types (text, audio, video, image) and ensure that only unique content is retained in the database.

Requirements:

1. Duplicate Detection:

   • For text submissions, use content-based hashing (e.g., SHA-256) or similarity algorithms (e.g., cosine similarity for embeddings) to identify duplicates.
   • For audio/video/image submissions, use perceptual hashing (e.g., pHash, dHash, or aHash) to detect near-duplicates.
   • Configurable similarity threshold for each media type.

2. Marking as Invalid:

   • When a duplicate is detected, mark the submission as invalid in the database.
   • Optionally, notify the submitter (via email or in-app notification) that their submission was flagged as a duplicate.

3. Database Schema Updates:

   • Add a is_duplicate boolean field to the submissions table.
   • Add a duplicate_of foreign key field to reference the original submission, if applicable.

4. API Endpoints:

   • Extend the submission API to include duplicate checking before insertion.
   • Provide an admin endpoint to manually check and mark duplicates.

5. Performance Considerations:

   • Ensure the duplicate detection process is efficient, especially for large datasets.
   • Consider asynchronous processing for media files to avoid blocking the API.

6. Geolocation Awareness:

   • Optionally, consider geolocation data in duplicate detection (e.g., same media submitted from the same location within a short time window).

7. Review Workflow Integration:

   • Allow reviewers to manually override duplicate flags if necessary.

8. Documentation:

   • Update OpenAPI docs to reflect new fields and endpoints.

Acceptance Criteria:

• Duplicate detection is implemented for all media types.
• Detected duplicates are automatically marked as invalid.
• Admin users can review and override duplicate flags.
• API and database changes are documented and tested.