Skip to content

ci: CI/CD Pipeline Hardening & Performance Optimization

Bikkumalla Sai Krishna requested to merge fixing-pipeline into feature-final

Summary

This MR introduces significant enhancements to the CI/CD pipeline, focusing on performance, reliability, and security. Key improvements include a restructured pipeline architecture to eliminate redundant installs, a hybrid caching strategy to resolve disk space and 413 (Request Entity Too Large) errors, and fixes for tooling (Mypy/Babel).

Proposed Changes

🚀 CI/CD Performance & Scalability

  • Introduced Setup Stage: Created a dedicated setup stage that builds a shared virtual environment (.venv). This ensures all subsequent stages (Lints, Tests, Security) reuse the same environment, cutting down total pipeline time by eliminating redundant uv sync calls.
  • Hybrid Cached-Sync Strategy: Implemented a hybrid approach to handle dependencies. This resolves recurring 413 Request Entity Too Large and No space left on device errors by optimizing how the runner handles the cache and local links.
  • Cache Optimization & Hardlinking: Removed unnecessary duplicate caching layers and switched to a hardlinking strategy for the uv cache. This significantly reduces disk I/O and storage consumption on the GitLab runner.

🛠️ Tooling & Bug Fixes

  • Mypy Duplication Fix: Updated mypy.ini with explicit_package_bases = True. This resolves the "Duplicate module named..." errors that occur when the source directory is not explicitly defined in the package hierarchy.
  • Stable i18n Generation: Refined the pybabel extraction process. By using sed to strip the POT-Creation-Date from messages.pot, we prevent git from flagging the file as "changed" when only the timestamp has updated, ensuring a cleaner git history.
  • Security Patches: Updated core dependencies (as seen in uv.lock) to address known vulnerabilities and maintain a secure supply chain.

🧹 Maintenance

  • Repository Cleanup: Removed the .coverage binary file from version control to keep the repository lightweight and prevent local development artifacts from being tracked.

Impact Analysis

  • Pipeline Speed: Expected reduction in pipeline duration by ~30-40% due to shared .venv.
  • Reliability: Resolution of pipeline-breaking disk space and network timeout errors.
  • Quality: More stable type-checking and cleaner translation workflows.

Verification Plan

Automated Tests

  • Pre-commit hooks updated and verified locally.
  • Pipeline setup stage verified for successful .venv export.
  • Mypy check confirmed to pass without duplication errors.
  • Babel pot extraction verified for idempotent output.

Note to Reviewer: These changes were primarily driven by the need to stabilize the pipeline on the shared runner which was hitting resource limits (413 errors and disk space). The new DAG (Directed Acyclic Graph) approach with a shared setup stage is the recommended way to scale this project further.

Edited by Bikkumalla Sai Krishna

Merge request reports

Loading