Lesson 1.3 Features, Labels, and Model Evaluation (accuracy, ROC-AUC, F1)

Outcomes (progressive ladder)

• Foundational: Define features and labels; build a tidy dataset for a nutrition task; read a confusion matrix.
• Intermediate: Choose and compute metrics—Accuracy, Precision/Recall, F1, ROC-AUC—and explain when each is appropriate.
• Advanced: Handle class imbalance, temporal leakage, and calibration; set operating thresholds using cost and decision logic; report subgroup (fairness) metrics and confidence intervals.

Concept Primer

Every supervised ML problem starts with two ingredients:

• Features (X): Measurable inputs describing a case—e.g., 14-day logging streak, average dinner time, step count, macronutrient ratios, device type, or image/text embeddings from a plate photo or meal note.
• Label/Target (y): The outcome you want to predict—e.g., “missed dinner log tomorrow” (0/1), “meal meets ≥25 g protein” (0/1), “estimated carbs in grams” (continuous).

In nutrition, the label is often a proxy for what we care about. Logged calories are not ground truth; under-reporting is common. Your evaluation must reflect this reality: use temporal validation (train on older weeks, test on newer) and prefer behavioral or clinical outcomes (A1c change, adherence over 4 weeks) when available.

Model evaluation is about two questions:

1. Discrimination: Can the model separate positives from negatives? (ROC-AUC, PR-AUC)
2. Decisions: At a chosen threshold, how well do alerts or nudges perform? (Precision, Recall, F1, Specificity, Accuracy, costs)

Feature Engineering in Nutrition (practical menu)

Behavioral features

• Streak length, time-of-day entropy, weekend vs. weekday indicator, push-open response rate.
  Nutritional features
• Daily macro % split, fiber grams per 1,000 kcal, sodium rolling average, diet diversity index.
  Contextual features
• Cuisine profile embedding, budget band, cultural/religious restrictions, device OS (camera quality proxy).
  Physio/clinical features
• Steps, HRV, sleep duration, BMI, HbA1c, meds (binary flags), fasting windows.
  Perception features (from DL)
• Plate segmentation counts, portion estimates (g), top-k food logits; confidence/entropy as inputs.

Guardrails: normalize units (g/mL), keep consistent windows (e.g., last 7 days), and timestamp everything.

Label Design & Hygiene

• Clear definition: “Missed dinner log” = no entry between 6–10 pm local time ± 30 min grace.
• Positive/negative windows: Avoid overlap with feature windows to prevent target leakage (e.g., don’t use 8–10 pm features to predict an event defined 6–10 pm).
• Noisy labels: For plate photos, use dual annotation and reconcile conflicts; store label quality flags.
• Class balance: Record prevalence (e.g., only 18% miss dinner logs). Metrics shift with prevalence—Accuracy may look high even for trivial models.

Splits & Validation (nutrition-aware)

• Temporal split (recommended): Train on older weeks, validate/test on newer—mirrors deployment.
• Group/Patient split: Ensure a single user’s data do not appear in both train and test.
• External validation: If you have multi-region data, hold out a region/cuisine as a stress test.
• Cross-validation variant: GroupKFold by user; avoid random KFold that leaks identity.

Confusion Matrix & Core Metrics (binary classification)

Suppose we predict “will miss dinner log tomorrow.” On the test set:

Calculations:

• Accuracy =(TP+TN)/(TP+FP+FN+TN)=(90+820)/1000=0.91= (TP+TN)/(TP+FP+FN+TN) = (90+820)/1000 = 0.91=(TP+TN)/(TP+FP+FN+TN)=(90+820)/1000=0.91
• Precision =TP/(TP+FP)=90/150=0.60= TP/(TP+FP) = 90/150 = 0.60=TP/(TP+FP)=90/150=0.60 (alert purity)
• Recall (Sensitivity) =TP/(TP+FN)=90/120=0.75= TP/(TP+FN) = 90/120 = 0.75=TP/(TP+FN)=90/120=0.75 (how many misses we caught)
• Specificity =TN/(TN+FP)=820/880≈0.93= TN/(TN+FP) = 820/880 ≈ 0.93=TN/(TN+FP)=820/880≈0.93
• F1 =2⋅(Precision⋅Recall)/(Precision+Recall)=2⋅(0.60⋅0.75)/1.35≈0.67= 2·(Precision·Recall)/(Precision+Recall) = 2·(0.60·0.75)/1.35 ≈ 0.67=2⋅(Precision⋅Recall)/(Precision+Recall)=2⋅(0.60⋅0.75)/1.35≈0.67

Interpretation: Despite 91% Accuracy, many positives are rare; F1 and PR-AUC better reflect usefulness for triggering nudges.

ROC-AUC vs. PR-AUC

• ROC-AUC measures ranking quality across thresholds using TPR (Recall) vs. FPR; stable when classes are balanced.
• PR-AUC focuses on Precision vs. Recall; more informative under class imbalance (typical in adherence, rare adverse events).
  Rule of thumb: Report both, lead with PR-AUC when positive prevalence < 30%.

Threshold Setting (turning scores into actions)

Model scores don’t act until you pick a threshold ttt.

Options

• Youden’s J (maximize Sensitivity + Specificity − 1): balanced screening.
• Max F1: balances Precision/Recall when false positives/negatives have similar costs.
• Cost-based threshold: minimize C=cFP⋅FP+cFN⋅FNC = c_{FP}·FP + c_{FN}·FNC=cFP⋅FP+cFN⋅FN; in coaching, repeated false positives may increase churn → set higher Precision.
• Operational constraints: cap daily nudges; pick ttt so expected alerts/user/day ≤ budget.

Example: If user annoyance is costly, require Precision ≥ 0.7 and then choose the highest Recall possible under that constraint.

Calibration (getting probabilities you can trust)

• Why: A score of 0.8 should mean “~80% of such cases are true positives.” This matters for scheduling interventions and consent messaging.
• How: Plot reliability curves; compute Brier score and Expected Calibration Error (ECE); apply Platt or isotonic scaling on validation data.
• Monitor: Calibration drifts with seasonality (festivals, holidays) and new cuisines.

Handling Imbalanced Data

• Data level: stratified sampling, class-balanced mini-batches.
• Loss level: class weights; focal loss for hard examples.
• Modeling: threshold-moving; anomaly detection as a precursor.
• Metrics: prioritize PR-AUC, Recall@Precision≥X, F1; always report prevalence.

Subgroup/Fairness Reporting (nutrition-specific)

Slice by: cuisine, language, device camera quality, age band, gender, region.
For each slice, report: prevalence, Precision, Recall, F1, PR-AUC, and Δ vs. overall.
If ΔF1 > 0.05 for any slice, trigger an action plan (data collection, reweighting, or separate thresholds).

Worked Mini-Example (step-through)

Task: Predict “meal meets ≥25 g protein.” Test set has 2,000 meals; prevalence = 22% (440 positives).
Model outputs probabilities. At threshold 0.6 we get: TP=280, FP=140, FN=160, TN=1,420.

• Accuracy = (280+1,420)/2,000 = 0.85
• Precision = 280/(280+140) = 0.67
• Recall = 280/(280+160) = 0.64
• F1 ≈ 0.65
• If raising threshold to 0.75 yields Precision=0.78, Recall=0.50 (F1=0.61), choose based on cost: is a false positive suggestion annoying (high cost) or acceptable?

Add calibration: If many 0.8-scored meals actually meet protein only 60% of the time, recalibrate before deployment.

Regression Sidebar (when the label is continuous)

For portion or calorie estimation:

• Report MAE (grams), RMSE, and MAPE (but beware tiny denominators).
• Add Prediction intervals (e.g., ±1 SD) to drive UI confirmation: “Looks like ~420 kcal (±90). Confirm?”

Operating Playbook (putting it together)

1. Define label precisely and avoid leakage (time windows!).
2. Engineer features with unit normalization and rolling windows.
3. Split data by time and user groups; hold out a cuisine/region for external test.
4. Train baseline (logistic or tree) before fancy models; record prevalence.
5. Evaluate with PR-AUC, ROC-AUC, F1 at multiple thresholds; add calibration and decision curves.
6. Select threshold with cost/UX constraints; simulate alert volume per user/day.
7. Report slices and CIs (bootstrap); document known failure cases.
8. Monitor in production: drift, calibration, alert fatigue, subgroup gaps; retrain cadence.

Pseudocode Snippets (conceptual)

Metric computation from confusion matrix

TP, FP, FN, TN = 90, 60, 30, 820
accuracy = (TP + TN) / (TP + FP + FN + TN)
precision = TP / (TP + FP) if (TP + FP) else 0.0
recall = TP / (TP + FN) if (TP + FN) else 0.0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0

Threshold search under precision constraint

# probs, y_true are arrays
for t in [i/100 for i in range(10, 91)]:
y_hat = (probs >= t).astype(int)
prec = precision_score(y_true, y_hat)
rec = recall_score(y_true, y_hat)
if prec >= 0.70:
select_threshold = t # keep the highest recall among these

Calibration check (conceptual)

# bin probabilities, compare predicted vs. observed frequency
bins = np.linspace(0,1,11)
observed, predicted = [], []
for lo, hi in zip(bins[:-1], bins[1:]):
mask = (probs >= lo) & (probs < hi)
predicted.append(probs[mask].mean())
observed.append(y_true[mask].mean())
ece = np.sum([abs(o - p) * np.mean(mask) for o,p,mask in zip(observed, predicted, [(probs>=lo)&(probs<hi) for lo,hi in zip(bins[:-1], bins[1:])])])

Practice Set (self-check)

1. Label spec: Write a one-paragraph label definition for “late-night snacking” that avoids leakage.
2. Metric choice: For a rare adverse event (prevalence 2%), justify why PR-AUC and Recall@Precision≥0.8 are superior to Accuracy.
3. Thresholding: Your model has PR-AUC 0.56. Choose an operating point that keeps nudges ≤ 0.3/day/user and Precision ≥ 0.7. Explain the trade-off.
4. Calibration: Sketch how you’d recalibrate after moving to a region with different cuisine; what plots and metrics will you show?
5. Fairness: Define three slices (e.g., vegetarian North Indian, South Indian, non-Indian cuisines). What’s your remediation plan if ΔF1 > 0.05 for one slice?

Glossary (targeted)

• Feature: Input variable describing a case.
• Label/Target: Outcome to predict.
• Leakage: Using future or post-event info in training features.
• ROC-AUC: Probability the model ranks a random positive above a random negative.
• PR-AUC: Area under Precision–Recall curve; sensitive to class imbalance.
• F1 Score: Harmonic mean of Precision and Recall.
• Calibration / ECE / Brier: Measures of probability reliability.
• Specificity: True negative rate.
• Prevalence: Fraction of positives in the data—impacts many metrics.

Takeaway

Great models in nutrition start with clean labels and trustworthy features. Evaluate beyond Accuracy: pair PR-AUC (ranking) with F1/Precision/Recall (decision), set thresholds that respect user experience and safety, and keep probabilities calibrated. Always check subgroups—cuisine, language, device—so your model performs fairly for everyone who uses your product.