Skip to content

refactore : using the graphql for fetching the projects ,mr,issues except commits

Bikkumalla Sai Krishna requested to merge glabflow_upgrade into dev

Merge Request

Overview

This MR introduces a major performance optimization by migrating multiple data-fetching operations from the GitLab REST API to the GraphQL API. It also introduces caching for commit data, configures uvloop for faster asynchronous execution, and removes the deprecated batch option feature from the system.

What does this MR do and why?

  • GraphQL Migration: Fetches projects, merge requests, and issues (in profiles, batch analytics, and the weekly performance tracker) via GraphQL. This drastically reduces the number of HTTP requests compared to REST paginated calls, preventing API rate limiting and improving overall sync speed.
  • Async & Concurrency Upgrades: Switched to using uvloop for optimized event loop performance. Implemented fanout concurrency strategies and disabled redundant pagination in specific areas.
  • Commit Caching: Added a project-scoped caching mechanism in commits.py to prevent redundant commit fetches across different members in the same project, reducing API load.
  • Feature Pruning: Removed the "batch option" functionality from the UI and underlying database services to streamline batch management.
  • Test Coverage: Added extensive unit tests for the new GraphQL client and the commit caching layer to maintain a robust test suite.

Changes Made

Technical Details

  • Added new infrastructure modules: graphql_client.py and graphql_queries.py to define and execute GraphQL queries.
  • Refactored users.py, projects.py, merge_requests.py, issues.py, and timelogs.py to construct and parse GraphQL responses instead of REST JSON.
  • Replaced traditional asyncio event loop with uvloop during initialization to improve raw throughput.
  • Implemented _raw_commits_cache and asyncio locks in commits.py to safely cache and reuse commit history payloads.
  • Modified ui/admin.py and ui/batch.py to remove the old batch option feature and updated roster_service.py to support the changes.
  • Added 124 lines of tests in tests/test_commits.py and 76 lines in tests/test_graphql_client.py, updating over 30 test files to align with the new GraphQL schema format.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 📝 Documentation update
  • ️ Refactor (no functional changes)
  • Performance improvement
  • 🧪 Test update
  • 🔧 Configuration change
  • 🚨 Security fix
  • 🗑️ Deprecation (removing deprecated code)

Related Issues / References

Screenshots or Screen Recordings

How to Validate Locally

  1. Run the application with uv run streamlit run app.py.
  2. Navigate to the Batch Analytics or User Profile views. Observe that the loading times are significantly faster for MRs and Issues.
  3. Test deleting a batch from the Admin page to verify the batch option removal behaves correctly.
  4. Run the test suite using uv run pytest to ensure all tests pass and coverage is maintained.

Testing Done

  • Unit tests added/updated
  • API endpoint tests passing

Test Cases Covered:

Scenario Expected Result Status
Fetching GitLab resources via GraphQL Returns correctly parsed data matching legacy REST schema
Multiple requests for the same project commits API is only called once, subsequent requests hit cache
Deleting a batch from the admin UI Safely removes the batch and cascade-deletes associated members

Test Commands Run:

# Run all tests with coverage
uv run pytest --cov=src --cov-report=term --cov-fail-under=70 tests/

# Run specifically the new GraphQL tests
uv run pytest tests/test_graphql_client.py -v

# Run the new Commits Cache tests
uv run pytest tests/test_commits.py -v
Edited by Bikkumalla Sai Krishna

Merge request reports

Loading