Skip to content

Fix/data fetching in camp analytics

Suma Pullaiahgari requested to merge fix/data-fetching into develop

Analytics Data Fetching Improvements

Overview

Refactored the analytics module to fix funnel capping logic, eliminate a fragile global cache, reduce redundant DB queries, and add a consolidated /api/v1/analytics/full endpoint for real-time polling.

Changes Made

Files Modified

File Change
app/services/analytics_service.py Major refactor: removed global cache, fixed funnel logic, added queue status constants, added get_full_camp_analytics()
app/schemas/analytics.py Added CampAnalyticsResponse schema
app/api/v1/routes/analytics_routes.py Added GET /analytics/full endpoint
tests/test_services/test_analytics_service.py Rewrote mock setup to align with actual query order, added funnel capping tests

1. Removed Global Cache (analytics_service.py)

Before: A module-level _cache dict stored Medicine and Doctor objects, shared across all analytics functions. It was cleared at the start of every top-level function (_clear_cache()), making it behave like a per-request local — but it remained in global scope, risking cross-request pollution under concurrent access.

After: Each function fetches the data it needs in a single bulk query and passes a local dict (medicines_dict, doctors_dict) to helper functions. This eliminates:

  • Global mutable state
  • Race conditions under concurrency
  • Redundant _preload_medicines / _preload_doctors indirection
  • The _get_medicine, _get_doctor, _build_doctor_prescription_detail helpers (unused after refactor)

2. Fixed Funnel Capping Logic

Root cause: The old capping chain was incomplete and used brittle per-stage comparisons with > 0 guards:

# Old (buggy)
if patients_consultation > patients_waiting and patients_waiting > 0:
    patients_consultation = patients_waiting
if patients_waiting > patients_kyp and patients_kyp > 0:
    patients_waiting = patients_kyp
if patients_kyp > patients_done_vitals and patients_done_vitals > 0:
    patients_kyp = patients_done_vitals

Problems:

  • Only three stages were capped, leaving the rest unchecked
  • > 0 guard skipped capping when preceding stage was 0, breaking the monotonic cascade
  • consultation_done was not even present in the cap chain

New approach — full monotonic cascade:

registered -> [vitals, doctor_assignment] -> kyp -> waiting ->
consultation_done -> prescribed -> verified -> counseling

Every stage is capped against its predecessor with a simple if val > prev: val = prev — no zero-guards, no gaps. This guarantees the funnel is monotonically non-increasing at every step.

3. Queue-Based Funnel Computation

Before: The funnel used ConsultationQueue → Patient → User → CampVisit join chain with individual == status checks, plus string fallback (or_ for both enum and string value).

After: A single base query joins ConsultationQueue.camp_visit_id → CampVisit.id directly (no Patient/User hop, avoiding cross-camp matches). Three cumulative status tuples (_QUEUE_KYP_STATUSES, _QUEUE_WAITING_STATUSES, _QUEUE_CONSULTATION_DONE_STATUSES) define "at or beyond" thresholds. Each stage reuses the same _queue_base_query with a single .filter(Status.in_(...)) — only one DB round-trip for the join, then filter-only for each stage.

4. Doctors Count Fix

Before: total_doctors was len(doctors_list), which only counted doctors who had at least one consultation in the system. This under-counted doctors registered for the camp.

After: total_doctors is now SELECT COUNT(DISTINCT user_id) FROM CampVisit WHERE camp_role = 'doctor' — accurately reflects all doctors registered for the camp.

5. New Consolidated Endpoint

GET /api/v1/analytics/full returns funnel, medicines, and doctors analytics in a single call, wrapped in a CampAnalyticsResponse:

{
  "funnel": { ... },
  "medicines": { ... },
  "doctors": { ... },
  "patients_attended": 42
}

The camp is resolved once and shared across all sub-analytics, avoiding three redundant _get_or_validate_camp calls. Designed for 30-second polling intervals.

6. CampAnalyticsResponse Schema (New)

class CampAnalyticsResponse(BaseModel):
    funnel: CampAnalytics
    medicines: MedicinesAnalytics
    doctors: DoctorsAnalytics
    patients_attended: int = 0

7. Test Infrastructure Fix

The _make_full_query_mock helper was missing the patients_pending_food query from its visit_vals iterator. The service makes 7 CampVisit queries, but the mock only supplied 6 values — shifting all downstream values by one position. This caused doctor_assignment to silently receive prescribed's mock value, which cascaded through the capping chain and incorrectly capped consultation_done at 4 instead of 5 in test_consultation_done_preserved_when_under_waiting.

Testing

  • All 50 analytics tests pass (1 pre-existing skip)
  • Funnel monotonicity explicitly verified in test_funnel_cascade_is_monotonically_non_increasing
  • Added test_all_zeros_when_no_patients, test_consultation_done_equals_waiting_when_all_waited_patients_done, test_no_camp_returns_zeroed_analytics, test_returns_camp_analytics_schema

Merge request reports