ci: CI/CD Pipeline Hardening & Performance Optimization
Summary
This MR introduces significant enhancements to the CI/CD pipeline, focusing on performance, reliability, and security. Key improvements include a restructured pipeline architecture to eliminate redundant installs, a hybrid caching strategy to resolve disk space and 413 (Request Entity Too Large) errors, and fixes for tooling (Mypy/Babel).
Proposed Changes
🚀 CI/CD Performance & Scalability
-
Introduced Setup Stage: Created a dedicated
setupstage that builds a shared virtual environment (.venv). This ensures all subsequent stages (Lints, Tests, Security) reuse the same environment, cutting down total pipeline time by eliminating redundantuv synccalls. -
Hybrid Cached-Sync Strategy: Implemented a hybrid approach to handle dependencies. This resolves recurring
413 Request Entity Too LargeandNo space left on deviceerrors by optimizing how the runner handles the cache and local links. -
Cache Optimization & Hardlinking: Removed unnecessary duplicate caching layers and switched to a hardlinking strategy for the
uvcache. This significantly reduces disk I/O and storage consumption on the GitLab runner.
🛠 ️ Tooling & Bug Fixes
-
Mypy Duplication Fix: Updated
mypy.iniwithexplicit_package_bases = True. This resolves the "Duplicate module named..." errors that occur when the source directory is not explicitly defined in the package hierarchy. -
Stable i18n Generation: Refined the
pybabelextraction process. By usingsedto strip thePOT-Creation-Datefrommessages.pot, we prevent git from flagging the file as "changed" when only the timestamp has updated, ensuring a cleaner git history. -
Security Patches: Updated core dependencies (as seen in
uv.lock) to address known vulnerabilities and maintain a secure supply chain.
🧹 Maintenance
-
Repository Cleanup: Removed the
.coveragebinary file from version control to keep the repository lightweight and prevent local development artifacts from being tracked.
Impact Analysis
-
Pipeline Speed: Expected reduction in pipeline duration by ~30-40% due to shared
.venv. - Reliability: Resolution of pipeline-breaking disk space and network timeout errors.
- Quality: More stable type-checking and cleaner translation workflows.
Verification Plan
Automated Tests
-
Pre-commit hooks updated and verified locally. -
Pipeline setupstage verified for successful.venvexport. -
Mypy check confirmed to pass without duplication errors. -
Babel pot extraction verified for idempotent output.
Note to Reviewer: These changes were primarily driven by the need to stabilize the pipeline on the shared runner which was hitting resource limits (413 errors and disk space). The new DAG (Directed Acyclic Graph) approach with a shared setup stage is the recommended way to scale this project further.
Edited by Bikkumalla Sai Krishna