Lesson 1.2 Supervised, Unsupervised & Reinforcement Learning

Outcomes (progressive ladder)

• Foundational: Define supervised, unsupervised, and reinforcement learning with nutrition-focused examples.
• Intermediate: Choose the right learning paradigm for tabular, text, image, and behavior data; pick fit-for-purpose metrics.
• Advanced: Design safe pipelines using semi/self-supervised pretraining, active learning, and offline off-policy evaluation for RL-like recommenders.

Concept Primer

You’ll see three big families in practice:

• Supervised Learning (SL): Learn a mapping from inputs xxx to labels yyy. We supply ground truth. Example: predict “will skip dinner logging tomorrow” from past behavior; estimate grams of carbohydrates from a parsed meal. Objective: minimize a loss such as cross-entropy or mean squared error.
• Unsupervised Learning (UL): Find structure in data without labels. Example: cluster weekly meal patterns to personalize nudges; learn embeddings of foods that show which items substitute for each other. Objective: density modeling, reconstruction, or contrastive criteria.
• Reinforcement Learning (RL): Learn actions aaa that maximize long-term reward RRR through interaction with an environment EEE. Example: decide when and how to send nudges so users log meals consistently over weeks. Objective: maximize expected return while handling exploration vs. exploitation and strict safety constraints.

Think of the nutrition app stack as: perception (often supervised + self-supervised) → pattern discovery (unsupervised) → decision/coach (contextual bandits or conservative RL).

Quick taxonomy table

Supervised Learning (nutrition-focused)

Formulation: Given (xi,yi)(x_i, y_i)(xi,yi), learn fθ(x)f_\theta(x)fθ(x) minimizing L(fθ(x),y)\mathcal{L}(f_\theta(x), y)L(fθ(x),y).

• Classification: predict class (e.g., “skip dinner log”: yes/no). Loss: cross-entropy; watch class imbalance.
• Regression: predict a value (e.g., grams of protein). Loss: MSE/MAE; consider quantile loss for asymmetric errors.
• Sequence labeling: tag tokens in meal text (food, quantity, unit). Loss: token-level cross-entropy; add constraints.

Nutrition examples

• Risk modeling: p(hyperglycemia∣meal, insulin, steps)p(\text{hyperglycemia} | \text{meal, insulin, steps})p(hyperglycemia∣meal, insulin, steps) with calibrated probabilities.
• Perception: CNN/Transformer to classify foods; regression head for portion size.
• Quality scoring: predict “meal meets target fiber” to trigger recipes.

Pitfalls & fixes

• Imbalance: use class weights, focal loss, or resampling; evaluate with PR-AUC.
• Leakage: ensure temporal splits (train before val/test time).
• Calibration drift: apply Platt/isotonic scaling and monitor ECE over time.

Unsupervised Learning (structure & sanity checks)

Clustering: Group similar users or weeks of meals.

• Example features: macro ratios, diversity index, average sodium, time-of-day logging entropy.
• Uses: cohort-specific coaching; A/B test stratification; bias checks across cuisines.

Dimensionality reduction / embeddings:

• Food2Vec from co-consumption; visualize cuisines on 2D projection to spot coverage gaps.
• Use embeddings to power nearest-neighbor substitutions (“no paneer? try tofu palak”).

Anomaly detection:

• Flag improbable entries (10,000 kcal lunch) or sudden sodium spikes; route to confirmation.
• Identify under-reporting: compare predicted intake from wearables vs. logged meals.

Pitfalls & fixes

• Clusters can be unstable; validate with silhouette/NMI plus domain review.
• Embeddings inherit data bias; enforce diversity in training batches.

Reinforcement Learning (behavioral coaching, safely)

Setup: State sts_tst (recent logging streak, time, hunger proxy, preferences), action ata_tat (send nudge type/time or stay silent), reward rtr_trt (log event, healthy swap, long-term retention). Goal: policy π(a∣s)\pi(a|s)π(a∣s) maximizing return E[∑γtrt]\mathbb{E}[\sum \gamma^t r_t]E[∑γtrt].

Pragmatic variants

• Contextual Bandits: One-step RL; pick the nudge now to maximize immediate reward (e.g., logging within 2 hours).
• Conservative Offline RL / Slate bandits: Learn from historical logs without new risky exploration; add safety constraints (never propose actions that could harm).
• Constrained RL: Optimize logging while capping notification frequency and respecting quiet hours.

Safety & evaluation

• Avoid online exploration on health-critical paths. Start with off-policy evaluation (OPE) on historical data:
  • IPS/DR estimators for policy value; check high variance.
  • Counterfactual logging: log propensities during A/B tests to improve OPE reliability.
• Use guardrails: action whitelists, clinical hard rules, budgeted nudges, and abstention when uncertainty is high.

Pitfalls & fixes

• Reward hacking: define composite rewards (short-term log + weekly adherence + user satisfaction).
• Non-stationarity: retrain on rolling windows; detect drift in context features.
• Credit assignment: use eligibility traces or learn uplift models to approximate long-term impact before full RL.

Choosing the paradigm (decision checklist)

1. Do you have reliable labels tied to outcomes? → Supervised.
2. Are you exploring structure, cleaning data, or compressing signals? → Unsupervised.
3. Are actions taken repeatedly over time with delayed outcomes? → RL/bandits, with offline evaluation first.
4. Mixed reality? Use hybrids: self-supervised pretraining on meal photos → fine-tuned supervised head; bandit on top of calibrated supervised scores; clusters to define RL contexts.

Guided Lab Walkthrough (conceptual, tool-agnostic)

Task: Build a minimal pipeline for “reduce missed dinner logs.”

1. Feature map (SL):
   • Inputs: last 14-day streak, hour-of-day histogram, weekday, prior response to nudges.
   • Target: yyy = missed dinner log (1/0).
   • Train a calibrated classifier; choose threshold by maximizing net benefit on a decision curve.
2. Structure discovery (UL):
   • Cluster users by logging time patterns (morning-centric, evening-centric, sporadic).
   • Validate clusters with silhouette score and manual inspection of 10 random members per cluster.
3. Policy layer (Bandit/RL-lite):
   • Actions: {no nudge, pre-dinner nudge, post-dinner recap}.
   • Reward: logged within 2 hours; penalty for sending a nudge that’s ignored (to avoid spam).
   • Start with epsilon-greedy contextual bandit offline-evaluated using IPS; cap nudges/day.
4. Safety & governance:
   • Quiet hours; never nudge within clinical fasting windows.
   • Log propensities and outcomes for future OPE; monitor slice performance by cuisine and language.

Patterns that scale from basics to advanced

• Semi-supervised learning: Pretrain on unlabeled meal photos (autoencoding/contrastive) → fine-tune on few labels.
• Active learning: Query users only when the model is uncertain about a meal photo or portion; label budget stays small.
• Curriculum learning: Start RL with a safe baseline policy (rules + supervised thresholds) then relax constraints gradually.
• Monotonic models: In supervised planners, enforce “more fiber ⇒ non-decreasing satiety score.”

Practical metrics cheat-sheet

• Classification (SL): PR-AUC, F1, Brier, ECE; report by cuisine/device slices.
• Regression (SL): MAE/RMSE; add MAPE for portion size but guard against division by tiny ground truths.
• Clustering (UL): Silhouette, Calinski-Harabasz; plus business outcomes (e.g., cluster-tailored nudges improve adherence).
• Bandits/RL: Off-policy value with IPS/DR; online regret once deployed; user-experience caps (nudges/day).

Pseudocode snippets

Supervised (calibrated classifier)

p = clf.predict_proba(X_val)[:,1]
p_cal = isotonic.fit_transform(p, y_val)
if p_cal > 0.7: send_nudge("pre-dinner")

Unsupervised (meal-pattern clustering)

features = weekly_meal_stats(user_logs) # macro ratios, variance, timing entropy
clusters = kmeans(features, k=4)
label_clusters(clusters, names=["AM-heavy","PM-heavy","balanced","sporadic"])

Contextual bandit (safe skeleton)

if quiet_hours or fasting_window: action = "no_nudge"
else:
a = argmax( Q(context, action) + explore_noise )
if budget_exceeded: a = "no_nudge"
execute(a); log(context, a, reward, propensity)

Practice set (self-check)

1. Pick one feature of your app and express it in all three paradigms (how would SL, UL, and RL each approach it?).
2. Design a labeling plan for supervised portion estimation with only 800 labeled photos—what can you pretrain with unlabeled images, and which 200 images would you label next via active learning?
3. Propose three cluster descriptors for weekly meal behavior and how each would change your coaching strategy.
4. Draft an off-policy evaluation plan for a new nudge policy using past logs (what propensities do you have, which estimator will you use, what safety thresholds?).
5. List two slice metrics you will monitor to ensure fairness across cuisines or languages; define your trigger for remediation.

Glossary (targeted)

• Off-policy evaluation (OPE): Estimating a new policy’s value using data collected under a different policy.
• Inverse Propensity Scoring (IPS): Weigh outcomes by the inverse probability of having taken the observed action.
• Doubly Robust (DR): Combines IPS with a learned outcome model for lower variance.
• Exploration vs. Exploitation: Trade-off between trying uncertain actions and choosing the current best.
• Contrastive Learning: Self-supervised method that pulls similar views together in embedding space, useful for pretraining on meal photos.
• Uplift Modeling: Predicts treatment effect of an intervention, not just the outcome.

Further reading (curated)

• Sutton & Barto — Reinforcement Learning: An Introduction (policy/value basics).
• Bishop — Pattern Recognition and Machine Learning (SL/UL foundations).
• Kallenberg et al. — Bandits and OPE tutorials (safe evaluation).
• Ribeiro et al. — Model interpretability surveys for health contexts.

Takeaway

Use supervised learning when you have trustworthy labels and clear targets; unsupervised learning to discover patterns, compress signals, and catch anomalies; and reinforcement learning (often via safe contextual bandits) to choose when and how to act over time. In nutrition, the strongest systems combine them: pretrain on unlabeled data, fine-tune supervised heads, cluster to tailor strategies, and wrap decisions in conservative, evaluated policies with hard safety rules.