Target & Lowe's (India): Data Scientist / Engineer

• Hierarchy: SKU × store × week, with reconciliation across levels (store → region; SKU → category).
• Features: lags, rolling means, seasonality, holidays and events, price and promotions, weather, stock-outs (to correct censored demand), new-store flags.
• Models: a global LightGBM across series (scales well), or deep models (TFT) for rich covariates; baselines are seasonal naïve and ETS.
• Validation: rolling-origin backtests; metrics WAPE and bias; error by velocity segment (fast vs slow movers).
• Ops: weekly retraining, monitoring, and planner overrides with a feedback loop.

• Code: modular packages, config files, unit tests; data validation (Great Expectations / pandera).
• Tracking and registry: MLflow for experiments; a model registry with stages and approvals.
• Pipelines: orchestrated training and batch scoring (Airflow, Kubeflow, Databricks Jobs); a feature store for consistency.
• Deployment: batch (scheduled) or real-time (a container + autoscaling); CI/CD with an evaluation gate.
• Monitoring: data and prediction drift, performance on delayed labels, latency and cost; automatic alerts and retraining triggers.

• Driver: builds the DAG from transformations, the Catalyst optimiser plans stages, and the scheduler assigns tasks to executors (JVMs on worker nodes) that process partitions in parallel.
• Narrow transformations (map, filter) stay within a partition; wide ones (groupBy, join) trigger a shuffle: data is redistributed across the network by key, which is expensive.
• Optimise: broadcast small tables, filter early, choose partition counts sensibly, handle skew (salting, AQE skew-join), cache reused DataFrames.

• Data lake: cheap object storage of raw files in any format (schema-on-read). Flexible, but it can become a "swamp" without governance.
• Warehouse: structured, schema-on-write, fast SQL, strong governance; more costly and less flexible for ML.
• Lakehouse: open table formats (Delta / Iceberg / Hudi) on a lake add ACID transactions, schema enforcement, time travel and fast SQL, so BI and ML work on one copy of the data.

• orders(customer_id) vs customers(customer_id): INNER (customers with orders), LEFT (all orders, plus customer info where it exists), RIGHT, FULL OUTER (reconciliation).
• Pitfall: duplicate keys on both sides multiply rows (a many-to-many explosion). Check key uniqueness before joining, and deduplicate or aggregate first.

def flatten(xs):
    for x in xs:
        if isinstance(x, list): yield from flatten(x)
        else: yield x
print(list(dict.fromkeys(flatten([1, [2, [3, 1]], 4, [2]]))))   # [1, 2, 3, 4]