refactore : using the graphql for fetching the projects ,mr,issues except commits
Merge Request
Overview
This MR introduces a major performance optimization by migrating multiple data-fetching operations from the GitLab REST API to the GraphQL API. It also introduces caching for commit data, configures uvloop for faster asynchronous execution, and removes the deprecated batch option feature from the system.
What does this MR do and why?
- GraphQL Migration: Fetches projects, merge requests, and issues (in profiles, batch analytics, and the weekly performance tracker) via GraphQL. This drastically reduces the number of HTTP requests compared to REST paginated calls, preventing API rate limiting and improving overall sync speed.
-
Async & Concurrency Upgrades: Switched to using
uvloopfor optimized event loop performance. Implemented fanout concurrency strategies and disabled redundant pagination in specific areas. -
Commit Caching: Added a project-scoped caching mechanism in
commits.pyto prevent redundant commit fetches across different members in the same project, reducing API load. - Feature Pruning: Removed the "batch option" functionality from the UI and underlying database services to streamline batch management.
- Test Coverage: Added extensive unit tests for the new GraphQL client and the commit caching layer to maintain a robust test suite.
Changes Made
Technical Details
- Added new infrastructure modules:
graphql_client.pyandgraphql_queries.pyto define and execute GraphQL queries. - Refactored
users.py,projects.py,merge_requests.py,issues.py, andtimelogs.pyto construct and parse GraphQL responses instead of REST JSON. - Replaced traditional
asyncioevent loop withuvloopduring initialization to improve raw throughput. - Implemented
_raw_commits_cacheand asyncio locks incommits.pyto safely cache and reuse commit history payloads. - Modified
ui/admin.pyandui/batch.pyto remove the old batch option feature and updatedroster_service.pyto support the changes. - Added 124 lines of tests in
tests/test_commits.pyand 76 lines intests/test_graphql_client.py, updating over 30 test files to align with the new GraphQL schema format.
Type of Change
-
🐛 Bug fix (non-breaking change that fixes an issue) -
✨ New feature (non-breaking change that adds functionality) -
💥 Breaking change (fix or feature that would cause existing functionality to change) -
📝 Documentation update -
♻ ️ Refactor (no functional changes) -
⚡ Performance improvement -
🧪 Test update -
🔧 Configuration change -
🚨 Security fix -
🗑 ️ Deprecation (removing deprecated code)
Related Issues / References
Screenshots or Screen Recordings
How to Validate Locally
- Run the application with
uv run streamlit run app.py. - Navigate to the Batch Analytics or User Profile views. Observe that the loading times are significantly faster for MRs and Issues.
- Test deleting a batch from the Admin page to verify the batch option removal behaves correctly.
- Run the test suite using
uv run pytestto ensure all tests pass and coverage is maintained.
Testing Done
-
Unit tests added/updated -
API endpoint tests passing
Test Cases Covered:
| Scenario | Expected Result | Status |
|---|---|---|
| Fetching GitLab resources via GraphQL | Returns correctly parsed data matching legacy REST schema | |
| Multiple requests for the same project commits | API is only called once, subsequent requests hit cache | |
| Deleting a batch from the admin UI | Safely removes the batch and cascade-deletes associated members |
Test Commands Run:
# Run all tests with coverage
uv run pytest --cov=src --cov-report=term --cov-fail-under=70 tests/
# Run specifically the new GraphQL tests
uv run pytest tests/test_graphql_client.py -v
# Run the new Commits Cache tests
uv run pytest tests/test_commits.py -v
Edited by Bikkumalla Sai Krishna