If you live in a gated community, you’ve been there: You request a ride from your apartment complex, expect your driver to come to you as usual, and then — your driver’s car icon just stops right at the front gate. You watch helplessly as the ETA ticks up. A chat message comes in: “Hey, how do I get in?” You scramble to remember the gate code. They try it. It doesn’t work. You end up meeting them awkwardly on the sidewalk outside while your coffee gets cold — a pickup journey frustrating for both you and your driver.

It turns out you’re not alone: Gated community pickups can make up 25–30% of Lyft rides in selected markets. For a long time, our app offered no special guidance in these situations. Riders would drop their pin inside the gates (fair enough — that’s where they are), while drivers would pull up to a locked entrance with no way in, leaving both parties to sort things out over chat. The result was predictable: more cancellations, longer waits, and a lot of unnecessary stress for our customers.

The Lyft Mapping team decided it was time to fix this properly — not with a band-aid, but a new end-to-end experience. Here’s how we did it.

What Was Actually Going Wrong?

We looked through gated ride examples, zoomed into our metrics data, and found two root causes behind most of the friction.

The first was an inflexible selection of pickup spots. Our app would suggest pickup spots near a rider’s location — which, for riders inside a gated community, often means inside the gate. But our data told a different story: many riders actually preferred meeting their driver right outside the gate, knowing their driver couldn’t access the property. The app wasn’t giving them that option clearly.

The second was a communication black hole. Even riders who knew how to get their driver through the gate had no good way to pass along access instructions in advance. Instead, they’d wait until the driver was already idling at the entrance before firing off a text with the gate code, setting off a frantic last-minute back-and-forth.

Two problems, two different moments in the ride flow. This signaled that we needed to fix things both before and after a ride was requested.

Only on Lyft Maps

Lyft’s Mapping team sits in a unique position to solve this problem, with workstreams across map data, pickup spot recommendations, routing, and the rider and driver app experience.

This one needed all of the pieces to complete the map (pun intended).

Piece 1: Drawing Gated Communities on the Map

Before anything else, we needed a map that actually knew about gated communities — where they are, where their entrances are, and how many gates they have.

Our Map Data team built a gate area shape generation algorithm to do exactly this. Gated communities come in all shapes and sizes: some are small apartment complexes with one entrance, and some are large developments with multiple gates and even their own internal road networks. The algorithm had to handle all of it reliably and provided the foundation accurate enough to build our new pickup experience for riders within these gated communities.

How does the app even know I’m inside a gated community? When you open the app, we cross-reference your GPS location against the gate area shapes we generated. If you’re within a known gated community boundary, the app quietly switches into “gates mode” and adjusts what you see as your pickup spot selection process. No extra steps on your end.

Is my community in the system? We’ve built our gate coverage from sources like OpenStreetMap and driver feedback, and we’re continuously monitoring for gaps and adding new communities during every map update — so coverage grows over time without you having to do a thing. That said, no map is perfect, and local knowledge is invaluable. If you don’t see the new experience or encounter issues at your gate, reaching out to Lyft support helps us prioritize the communities that need attention most.

Piece 2: Giving Riders a Smarter Selection of Choices

Now, when you open the Lyft app from inside a gated community, you’ll see pickup spots at your complex entrances (labeled “I’ll walk outside the gate”), right alongside the usual spots near your current location (labeled “Pick me up inside gate”). You can decide freely whether you prefer to wait locally (maybe it’s raining outside) or want to meet your driver outside the gate (get some steps in, right?).

The team looked at historical ride patterns to surface spots that real riders have actually used near gates — so we’re not just pointing you toward the gate in theory, we’re pointing you to where riders in your community actually go.

We also designed the new UI to feel familiar, borrowing patterns from our existing Venues pickup flow so our riders wouldn’t have to learn something from scratch.

Before shipping, we ran a controlled experiment to make sure we weren’t accidentally making things worse — specifically, that the new flow wouldn’t add enough friction to make riders bail on requesting a ride altogether. Good news: no meaningful drop-off. Even better news: we saw actual pickups happening at the spots that riders chose initially, meaning that they no longer have to walk around to find their drivers.

Piece 3: Routing Drivers to the Right Gate

Recommending gate-aware pickup spots to a rider is only half the job. The driver also needs to be routed there correctly.

Normally, a pickup route is simple — get the driver from point A (their location) to point B (the rider’s pickup spot). For gated communities, our Routing team introduced a new “detour”: the gate itself becomes an intermediate stop, an arbitrary, “invisible” stop that the driver passes through on the way to the rider. We use our map data to direct drivers to the right gate, accounting for road direction and the most logical approach path.

This seemingly small routing tweak creates something valuable: a precise moment in the driver’s journey where we can surface gate instructions at exactly the right time.

Pieces 4: Actually Getting the Driver Through the Gate

Now that we get drivers to the gate, we still need to get them through the gate to meet those who asked to be picked up inside, and we really can’t let our customers resort to back-and-forth communications.

Our Map Experience team brought this to life: After a rider is matched with their driver, they’re prompted to share access information. Rather than presenting a blank text field — which puts all the pressure on the rider to know exactly what to say, right after they’ve just requested a ride — our Design team drew inspiration from the intercom panels riders use every day: a familiar numpad for gate codes, and a short list of plain-language options for everything else. The goal was to make instruction sharing effortless.

We leveraged historical data from gate instructions that riders had previously shared to identify the most common access scenarios. Our Content team worked to make each option feel conversational and specific enough to be useful, without being so wordy that riders would skip past them. The free-text fallback is still there for the edge cases, but in practice, most riders find what they need in the list.

You might be thinking: I usually text my driver the gate code — what’s actually different here? The difference is timing and reliability. When you text, you’re reacting: the driver might be busy driving, or is already at the gate, already stopped, already waiting.

We also paid special attention to how we should show the shared instructions to drivers. They are likely navigating with eyes on the road — so a paragraph of gate instructions on their screen at the wrong moment is unhelpful and more importantly, unsafe. We designed the gate instructions to surface as a small, but obvious banner in the navigation screen at precisely the moment the driver approaches the gates, timed to the routing waypoint we added in Piece 3. The information is brief and scannable — just enough to get through the gate. No extra steps or unnecessary calls or texts.

We also know that sharing a gate code feels more sensitive than sharing a pickup pin — it’s access to where you live, and your privacy is important to us. So we built in privacy controls that let you view or delete your instructions at any time after sharing. Gate codes are never stored between trips, and only your matched driver ever sees them on their navigation screen on their way to pick you up. To further ensure your gate codes stay protected, we implemented safeguards to prevent screenshots.

Does It Actually Work?

We asked riders directly through our optional feedback survey. About 95% responded positively — and their comments said it better than any metric could:

“Driver successfully picked me up OUTSIDE the gate, as requested.”
“Picked me up where I asked.”

We also see data reflecting our goals: riders and drivers are not giving up on a ride just because there’s a gate in the way. Drivers are not looping around a community entrance trying to figure out how to get in. Riders are not wandering the sidewalk looking for a car that’s stuck on the other side of a barrier. Phones are not fired off with chats and calls because our riders and drivers can’t find each other. Those are the moments we were trying to eliminate — and the numbers reflect it.

We observe lower cancellation rates from both riders and drivers, when riders share gate instructions.

In addition, riders are walking less to their final pickup locations from their requested pins, waiting less, and don’t need to reach out to their drivers as much. We also see drivers needing to change course in the final stretch less. Each of these represents a ride that went smoothly instead of sideways.

Now, that same ride from your gated community looks a little different: your driver’s icon moves smoothly past the gate with the instruction you share with a simple tap, and you get into your ride stress-free, with your coffee warm. The results gave us confidence that the approach works — and got us thinking about where else we could take it.

More Gates, and The Bigger Picture: Bringing the Real World to the Map

For gated communities specifically, we’re just getting started. We want to provide riders with the option to save non-sensitive gate instructions for future trips, so riders who regularly get picked up from the same community don’t have to re-enter the same information every time. Your gate intercom instruction isn’t going to change — we shouldn’t make you tell us every time. We will also work on smarter gate selection for communities with multiple entrances, choosing the best gate based on where both you and your driver are at that moment, not just the nearest one on the map.

But the more we worked on this project, the more we realized gates are just one example of a much bigger problem. The real world is full of physical constraints that affect where a driver can go and where a rider can safely and conveniently be picked up — and we see a big opportunity of leveraging our Mapping stack to improve customers’ pickup experiences across many of these situations.

Consider road closures. A street gets blocked for construction, a parade, or a marathon, and suddenly the route our app plotted doesn’t work anymore. The driver gets rerouted at the last second, the pickup pin is now on the wrong side of a barrier, and rider and driver end up circling each other trying to figure out where to actually meet.

Or consider unsafe road segments — a pickup spot that looks perfectly fine on a map but sits on a high-speed tunnel exit way with an accident-prone history. The rider waits on the curb, and the driver pulls over in a spot, attempting to avoid moving traffic. Neither of them had any idea there was a safer option just half a block away.

What the gated community project gave us is a repeatable playbook for all of these situations: encode the real-world constraint into the map, surface that context in pickup spot recommendations, thread it through routing, and deliver the right information at the right moment and place in the app. Whether the obstacle is a gate, a closed road, or an unsafe stretch of curb, the same approach applies.

Gated communities were a specific problem. But the approach behind solving them isn’t specific at all. Every time the real world gets in the way of a safe and convenient pickup, there’s an opportunity to make the map smarter — and the ride a little less stressful for everyone involved.

Lyft is hiring! If you’re passionate about maps, visit Lyft Careers to see our openings.

How We Built a Smarter Pickup Experience for Gated Communities was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Lyft, understanding how riders go through our user experience is fundamental to operating a healthy marketplace. Specifically, it is important to have a robust model determining if a rider will actually request a ride after entering a destination and viewing a price and ETA. Accurately predicting this decision, that we call conversion, informs countless decisions across our platform. Whether it is to better balance supply and demand, improve user experiences, optimize recommendations and advertisement, understand long-term engagement, decide how to distribute coupons… rider conversion prediction is a central challenge for the Lyft business.

However, predicting human behavior at scale is incredibly complex, and the exact same person might well open the app just to check current availability or actually to request a ride after viewing our prices. The contexts under which riders make their conversion decisions are extremely diverse and almost unique to each session. A user’s intent changes based on where they are and where they want to go, what time it is, their previous interactions with the platform, current supply-demand market conditions, to cite a few.

When we try to model this using standard machine learning approaches, we run into a significant challenge: data sparsity.

The Challenge of High Cardinality and Sparsity

To accurately predict conversion, we need to slice our data very thinly across many categorical features. Imagine trying to predict the conversion probability for a business traveler leaving the suburbs of Detroit at 4:00 AM on a Tuesday to catch their flight at the airport 30 minutes away. While Lyft has vast amounts of data overall, the amount of data available for that specific intersection of contexts often reveals to be very tiny. Maybe we only have ten examples in history.

If we use standard techniques like Gradient Boosted Trees (e.g., LGBM, XGBoost), we encounter severe overfitting. A standard model looking at 10 examples in the training data where 8 converted might confidently predict an 80% conversion probability. But that’s likely statistical noise. The next 10 examples might show only 2 conversions.

One could argue that much more complex models, such as Deep Neural Networks or even Large Language Models in a few-shot fashion, could be able to handle this sparsity if properly tuned and calibrated; but this comes at the cost of greatly increased inference time. Indeed, predicting session conversion is very often needed live, while the rider interacts with our systems, and we want to be able to confidently predict their conversion as soon as we know their destination. And of course, no one wants to see a loading wheel rotating for hours, so when you think about the user experience, we need a model that can handle an extremely high-cardinality set of different situations, provide robust and accurate predictions even when data is scarce, and serve those predictions with ultra-low latency.

Overcome Sparsity with Bayesian Trees

To solve this, our team developed a modeling framework to predict sessions’ conversions in real-time. At its core, it is a highly optimized, hierarchical lookup structure designed to handle sparse categorical data by leveraging Bayesian theory. It incorporates priors (knowledge gained from broader contexts) to smooth predictions in specific, data-rare contexts.

The Hierarchical Tree Structure

We organize the training data into a tree structure based on a hierarchy of partition keys. These keys define the context. While the exact keys we use depend on the specific application, imagine a hierarchy like this:

• Root: All the sessions
• Level 1 Split (e.g., Spatial Context): City Region
• Level 2 Split (e.g., Temporal Context): Time of Day / Day of Week
• Level 3 Split (e.g., Congestion Context): Current local supply-demand balance
• …

As we move down the tree, the context becomes more and more specific, and the data available at each node becomes sparser and sparser.

The Secret: Bayesian Smoothing with Gaussian Priors

How do we deal with a leaf node that only has five data points? We definitely cannot trust only the 5 data points and train an independent model, but also we cannot have a global model that trains on the whole data, as it would miss the specifics of this leaf node that guide rider conversion decision-making.

Our conversion model uses the statistical properties of the parent node to inform the prediction at the child node. This is achieved through Bayesian smoothing, specifically utilizing gaussian priors on model parameters. It allows data-sparse segments (e.g. a leaf node in the tree) to default to the robust average of their parent group.

Here is the intuition:

1. Model architecture: Each node in the tree is a copy of the same parametric model. It can be a logistic regression, a support vector machine, or even a shallow neural network. To make a prediction y from an input x, you rely on a parametric model f that depends on trainable parameters Θ such that y = f(x, Θ). Each node has its own version of Θ that we want to be as specific as possible to the segment the node represents.
2. Training: The tree is trained top-down, starting from the root of the tree, level by level, until the leaves, with less and less data as we dive deeper and deeper. For each individual node, its parent node has been trained on more data. Hence Θ_parent provides a strong belief about what Θ_child should look like when training the child node. As such, when fitting the child’s model on the child’s specific data, we add in the training a L2 penalization for diverging too far from the parent’s, that is ||Θ_parent — Θ_child||². This is known as a Gaussian Bayesian prior, centered at the parent’s mean.

The choice of the regularization strength λ is critical and should depend on the data size available between the parent’s node and its children, such that the model automatically balances parent’s trust with specification to child’s data. If a leaf node has very little data, its prediction will be heavily pulled towards the parent’s stable mean. As the leaf node acquires more data, the model gradually trusts the local data more, moving the prediction away from the parent’s prior and towards the local raw average.

Ensuring Behavioral Consistency

Beyond handling sparsity, we often have domain knowledge about how conversion should behave relative to certain continuous variables. For instance, consider the rider’s historical conversion rate: intuitively, as this increases, the predicted conversion probability for the current session should also increase. All else being equal, we should logically predict the conversion of a rider with a 90% historical rate higher than the one for another rider with a 40% historical rate. The model should enforce a monotonic relationship between historical conversion and current session’s conversion.

However, standard ML models can sometimes learn erratic shapes that violate this intuitive logic due to noise in the training data, ending up with non monotonic relationships between some input and the output that shouldn’t be. With Bayesian trees, we can keep very simple parametric models at each node because the use case is already very specific (that is the whole point of Bayesian trees, right?). Using simpler models— e.g. a logistic regression cvr = σ(Θ . cvr_hist) — can give much more control on the model monotonicity (e.g. enforce Θ>0) and ensure appropriate behavior of the model, and as such better explainability. This guarantees that the model’s outputs are not only statistically robust but also directionally intuitive, reliable and interpretable.

Conclusion

Bayesian Conversion models represent a significant step forward in our ability to model rider conversion decision-making in highly dynamic, sparse environments. By combining a highly structured hierarchical approach with the robust statistical grounding of Bayesian smoothing, we can generate accurate, stable predictions where traditional models fail. This architecture allows us to be hyper-local and specific when the data supports it, while gracefully falling back to broader, stable trends when faced with the unknown. It’s a key piece of infrastructure that helps Lyft make smarter decisions in real-time.

Lyft is hiring! If you’re passionate about developing state of the art machine learning/optimization models or building the infrastructure that powers them, read more about them on our blog and join our team.

Predicting Rider Conversion in Sparse Data Environments with Bayesian Trees was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Amber Wang and Yoonji Kim at Lyft.

Background

Whenever you use the Lyft app, there is a complex balancing act happening behind the scenes. Various levers are used to keep the marketplace running smoothly; Base prices and coupons for riders affect demand, while driver pay and bonuses impact the level of available supply. Since every change to prices and payments impacts Lyft’s costs and revenue, they lead to key optimization problems, such as:

• How should we allocate budget between driver incentives and rider incentives?
• How do we invest resources to achieve x% rides growth, and how much does it cost in terms of short term profit?

These are the questions the Foundational Models team at Lyft tries to answer in a systematic way. A key ingredient is understanding the effects of different types of investments — for instance, what will happen if we increase the total budget for driver incentives by x%? What will happen if we increase the rider price of all rides by y%? It’s worth noting that the long term effects of such decisions tend to dominate the short term effects: we may earn more short term profit from a ride if we charge riders more and pay drivers less, but lose riders and drivers in the long run.

Estimating the long term effects of resource allocation decisions is challenging in a multi-sided marketplace such as Lyft. Because these decisions tend to be consequential, their effects go beyond first order effects on directly affected users. For example, if we increase driver incentive spending by x% in week 1, drivers will drive more in week 1 (short term effect), and may return to drive a bit more in the following weeks (direct long term effects). But this is not the full picture: in week 1, when there is a positive increase in driver hours as the result of more incentives, riders will enjoy better experiences (e.g. less surge pricing, shorter wait times) and may want to return to Lyft in the future. However, more driver hours in the market also mean the average driver is less busy, and that idleness may discourage them from driving in the future. An analog holds for any decision that boosts short term demand — riders are more likely to encounter bad experiences such as higher surge prices and longer wait times, whereas drivers may become busier and earn a bit more. At Lyft, we refer to such indirect effects as the “market-mediated effects.” Compared to the direct long term effects, these market-mediated long term effects are much harder to estimate.

Below is an illustration of the causal relationships we consider for the policy change “increase driver incentive spend.”

Summary of our solution

This article introduces the methodology framework we use to estimate the “market-mediated long term effects,” the more challenging component of the long-term effects. In short, we use a two-step approach that can be thought of as a surrogacy approach in its broad sense. Under the assumption that the “market-mediated” outcomes are fully mediated through negative user experiences, (1) we first estimate how our policy changes affect the distribution of core negative user experiences, then (2) estimate how negative user experiences affect users’ future behavior.

Both steps are done with observational causal inference, allowing fast and inexpensive updating. Verification is another major challenge. There is no single form of experiment that can provide a perfect verification; therefore, we have to combine multiple imperfect signals. Specifically, the first step methodology can be verified using a switch-back experiment, and the second step of estimation can in principle be verified using user-split experiments.

As the last step, we combine the direct long term effects (estimated separately) and the market-mediate long term effects and use region-split experiments to verify the overall long term effects. In general, region-split experiments in general suffer from poor pre-intervention fit and low power; we developed a forward selection algorithm to optimize experiment design by picking the treated and control regions.

Below, we describe our methodology in detail.

Step 1: from decisions to negative user experiences

We first start with a question: when we move a policy decision today (e.g., raise rider price water-level or change driver incentive budgets), how does the short-term user experience shift?

What we model: How does a policy change affect short-term user experiences? We focus on a set of negative user experiences that matter most downstream — such as long waits, high surge, and cancellations for riders, or lower hourly earnings, idleness and incentive earnings for drivers — and maintain the critical assumption that these are the only channels through which today’s decisions affect the future (by affecting today’s market). The goal of this step is to estimate the effect of our policy decisions on the distribution of these negative user experiences.

The challenge is that negative user experiences are highly cyclical and seasonal, varying with time‑of‑week, holidays, weather, and shifting supply/demand. A naive regression would blend these rhythms with the true effect of our decisions. Therefore, our approach:

• Residualizes predictable time-of-week patterns so we compare like-with-like, which mimics a “random shock” by capturing the deviation from this period’s normal status.
• Controls for remaining market information such as supply and demand so the coefficient of our policy decision reflects its incremental impact on negative user experience.

How we estimate (observational, residualized regressions): Conceptually, for any negative user experience metric (e.g., wait time), we fit a residualized model on deviations from the market’s own baseline:

Below is a visualization of a simulated example where we have one contextual variable. With residualized terms, we can capture how policy change would marginally change negative experience (green solid line).

The coefficient beta policy tells us how much negative user experience moves when a policy decision (like rider price water‑level or incentive spend) deviates from its usual level, after accounting for typical time‑of‑week rhythms and market conditions. Because we measure everything as “deviation from normal,” the effects read like elasticities around everyday operating conditions. For example, a higher‑than‑usual price suppresses demand and hence is associated with a predictable decrease in negative rider experience (e.g. high surge, long wait time), holding everything else constant. The output is a calibrated response function that turns a policy change into a forecasted shift in the distribution of negative user experience, not just its average, with uncertainty to reflect real‑world variability.

How we validate: To validate this mapping, we use switch‑back experiments that alternate policy settings across comparable time slots and compare the model’s predicted changes in negative user experience to the experimental lifts we observe. The experiment either verifies our model or informs its iteration (e.g., changing controls or introducing additional ones).

Step 2: from negative user experiences to future outcomes

The next question is: given a change in a certain negative user experience today, how do future outcomes move (e.g., future rides, retention, driver hours)? In the broader surrogacy framing, Step 1 captures short‑term shifts in experience, and Step 2 translates those shifts into long‑term behavior under the assumption that long‑term market-mediated effects are completely mediated by short‑term negative user experiences.

What we model: The long term effects of negative user experiences. We treat these negative experiences as exposures that vary naturally across users, times, and places, and estimate their impact on future outcomes — such as future rides — while controlling for confounders (time, location, and rider history).

How we estimate (observational, double‑robust): We use Augmented Inverse Probability Weighting (AIPW, Chernozhukov et.al, 2021), a doubly robust causal estimator combining (i) a propensity model for exposure (the likelihood of facing a given level of negative user experience, given context) and (ii) outcome models for future metrics, conditional on confounders. This yields average treatment effects for negative user experience. We summarize the mapping via a “surrogacy index” that quantifies how much short-term negative experiences will affect long‑term outcomes; this is the scaling we use to move from short‑term exposure to negative experiences to long‑term impact.

How we validate: We run user‑split experiments that perturb negative experiences and compare the model’s predicted changes in future outcomes to the experimental lifts, checking calibration (predicted vs. observed) for validation.

How it works together with Step 1: Step 1 converts a policy change into a shift in the distribution of negative user experiences; Step 2 converts that exposure shift into forecasted changes in future outcomes (e.g., future rides). Together, via the surrogacy index, they provide a causal link from today’s decision to long‑term business impact.

Step 3: verify overall LTE by region-split experiments

With the market‑mediated effect from Steps 1–2, we combine the direct long term effect (estimated separately) using a transparent formula grounded in market mechanics. This yields a single policy‑level forecast for long‑run rides and financials that reflects both mediated and direct channels.

To validate end‑to‑end predictions, we run region‑split experiments. We developed a forward‑selection algorithm, inspired by the forward difference‑in‑differences (FDiD, Li, 2024) approach, to choose treated and control regions: starting from a single treated region, we iteratively add treated regions that best improve pre‑period fit and expected power. The region-split experiments allow us to observe the overall long-term effects of a policy intervention to an entire market.

Below is a simulated example (using simulated data) of a region-split experiment. We find a set of treated regions where we can find another set of control regions that mimic the average behavior of treated regions. Once we inject an intervention shock on the treated regions (e.g., increase of incentives), we track the discrepancy between the average rides of treated and control regions.

Conclusion

We presented a framework to connect today’s resource decisions to long‑run marketplace impact. First, we translate policy changes into shifts in negative user experience using residualized modeling and verify those short‑term responses with switch‑back experiments. Second, we map negative user experience to future outcomes with doubly‑robust observational inference (AIPW) and a surrogacy index, then validate with user‑split experiments. Finally, we combine the market‑mediated and direct long-term effects and validate end‑to‑end predictions via region‑split experiments, using a forward‑selection design to choose treated and control regions.

What this enables:

• Scenario planning and policy evaluation.
• Budget allocation across levers (pricing, incentives) informed by the short-term profit v.s. long-term rides Pareto Frontier.
• Continuous calibration as markets evolve, grounded in experimentally-verified observational causal inference.

The result is a model‑based, experiment-verified causal engine: decisions move user experience, user experience moves future behavior, and the composition yields long‑term business impact. This enables fully-informed decisions on resource allocation.

Lyft is hiring! If you’re passionate about Data Science, visit Lyft Careers to see our openings.

Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Stefan Zier

For years, Lyft’s localization infrastructure relied exclusively on human translation. While this model usually ensured excellent quality, it was bound by multi-day turnarounds and costs that scaled linearly with every new language. For the few languages Lyft initially supported (Spanish, Portuguese, and French), these limits were acceptable.

However, Lyft’s expansion goals quickly outpaced what traditional workflows could support. Lyft’s recent Québec launch required compliance with Bill 96 (legislation mandating French-first user experiences) which demanded faster turnaround than multi-day cycles allowed. Simultaneously, the Lyft Urban Solutions (“LUS”: Bikes & Scooters) division sought to expand into European markets, requiring six new languages. The business need had changed as we now needed to move faster without sacrificing quality.

This post explores how we re-architected Lyft’s Translation Pipeline to leverage AI alongside linguist oversight and ultimately unlock new market launches. We will walk through context injection, decoupling content generation from evaluation, implementing guardrails, and treating prompts as version-controlled production code. The new pipeline reduces translation latency from days to minutes while maintaining the fidelity required for legal compliance and brand integrity.

Note: We will walk through our batch translation pipeline — used for 99% of app and web content — which targets a 30-minute SLA for 95% of translations. We also support real-time translation (e.g., ride chat) which uses a different architecture.*

How Translations Reach Hundreds of Services

Before diving into the LLM pipeline, it helps to understand how translations flow through Lyft’s infrastructure. This 2020 post explains the internationalization architecture initially built to move beyond one language/currency/country. Since then, the platform has grown to serve 11 locales across 150+ services.

At its core, the pipeline does two things in parallel: it uploads source strings to Smartling, our Translation Management System (TMS), for human oversight, and submits them to LLM workers for rapid draft generation. This dual-path architecture lets us accelerate turnaround while keeping the TMS as the system of record for quality control. The AI-translated strings ship immediately to unblock launches and linguists review them asynchronously. Once approved, the linguist-reviewed version replaces the early release.

Lyft’s localization pipeline operates in three phases:

1. Drafting (Ingest): Requesters submit source strings along with context — where the text appears in the UI and the intended tone. The Drafter uses this context to generate translation candidates.
2. Unblocking Launches (Early Release): The “Evaluator” runs various quality checks to select the optimal translation candidate. This version is distributed in minutes, unblocking product launches while linguist review takes place in parallel.
3. Finalization (Review): Professional linguists review the drafts within the TMS. Approved translations replace the early-release versions and are established as the definitive system of record. Flagged translations are corrected before being distributed.

Why LLMs?

We started with an evaluation of traditional Neural Machine Translation (NMT) providers. We found NMT was fast but translations often did not preserve Lyft-specific terminology or context. This pointed us toward LLMs, and recent research validated that direction.

LLMs have reached human-level translation quality for resource-rich languages. A 2024 study found GPT-4 produced error rates comparable to junior translators for major language pairs, though performance weakens for lower-resource languages. Industry adoption reflects this shift with over 70% of translations now machine-assisted and an overall growing trust in AI-provided translation.

Context handling is the key differentiator. Traditional NMT services process each string in isolation with no awareness of where or how it will be used. In our system, requesters provide context about each string — where it appears, its purpose, any relevant UI constraints. LLMs can incorporate this metadata directly to often produce more accurate translations.

LLMs can evaluate, not just translate. Multiple valid translations can exist for the same source string. Research established that LLMs are state-of-the-art evaluators of translation quality, with WMT25 confirming that large LLMs show strong system-level evaluation performance. This enabled the following architecture where one LLM translates, another evaluates and provides feedback, and the translator again refines based on the critique.

Building the Iterative Pipeline

A naive approach to machine translation is a single API call: send English text, receive translated text. This fails at scale for several reasons:

1. No nuance. Single-shot translations are often “correct enough” but rarely optimal for brand voice or regional idioms.
2. No quality signal. Without evaluation, there’s no way to know if a translation is acceptable before shipping it to users.
3. No recovery path. When a translation fails validation, the system has no mechanism to try again with corrective feedback.

We needed a system that could generate options, critique them, and iterate, a workflow that mirrors how human translators actually work. This led us to an architecture where multiple specialized processes translate and evaluate through structured handoffs.

The “Drafter” — Translation Generation

The job of the Drafter is primarily creative as it aims to produce diverse, high-quality translation candidates. We configure it to generate three distinct candidates for every source string.

Why three? We find a single translation often converges on the most likely phrasing, which may not be optimal for Lyft’s brand voice or the specific UI context. Multiple candidates increase the probability that at least one captures the right tone, handles edge cases correctly, and uses terminology naturally.

Model selection: For the Drafter, we use a fast, non-reasoning model here as translation is primarily a generative task where standard models already perform very well. Additionally, a faster model comes with lower cost and allows us to iterate.

Sample Prompt

DRAFTER_PROMPT = """
You are a professional translator for Lyft.
Translate into {language} for {country}.
Give {num_translations} translations of the following text.

GLOSSARY: {glossary}
PLACEHOLDERS (preserve exactly): {placeholders}

Text: {source_text}
"""

Sample Input/Output

Note: The LLM interactions return structured data via Pydantic schemas rather than free-form text. This ensures type safety, reliable parsing, and clear contracts between Drafter and Evaluator.

# Input 
source_text = "Your {vehicle_type} is arriving in {eta_minutes} minutes"
language = "French"
country = "Canada"

# Output (parsed)
DrafterOutput(
    candidates=[
        TranslationCandidate(text="Votre {vehicle_type} arrive dans {eta_minutes} minutes"),
        TranslationCandidate(text="Votre {vehicle_type} sera là dans {eta_minutes} minutes"),
        TranslationCandidate(text="Votre {vehicle_type} arrivera d'ici {eta_minutes} minutes"),
    ]
)

The “Evaluator” — Translation Evaluation

The Evaluator acts as a strict quality gate. It receives all candidates from the Drafter and scores each against a rubric, ultimately selecting the best one or rejecting them all.

Model selection: We use a reasoning-focused model for evaluation. Unlike generation, evaluation requires analytical comparison: checking source versus target for semantic drift, verifying terminology compliance, catching subtle tone mismatches. The deliberate reasoning process helps surface errors that a faster model might miss.

The Evaluator grades each candidate on four dimensions:

1. Accuracy & Clarity: Does the translation preserve the full meaning of the source? Is it unambiguous?
2. Fluency & Adaptation: Does it read naturally to a native speaker? Is it culturally appropriate for the target region?
3. Brand Alignment: Does it use official Lyft terminology? Are proper nouns, airport codes, and brand names preserved in English?
4. Technical Correctness: Is it free of spelling and grammar errors? Are all Lyft terms/phrases applied correctly?

Each candidate receives a grade: pass or revise. If any candidate passes, we ask the Evaluator to select the best one. If all fail, the Evaluator provides a detailed critique explaining why each failed.

Sample Output

EvaluatorOutput(
    evaluations=[
        CandidateEvaluation(candidate_index=0, grade=Grade.PASS, 
                           explanation="Accurate, natural phrasing."),
        CandidateEvaluation(candidate_index=1, grade=Grade.PASS,
                           explanation="Natural and conversational."),
        CandidateEvaluation(candidate_index=2, grade=Grade.REVISE,
                           explanation="'d'ici' implies uncertainty, inappropriate for ETA."),
    ],
    best_candidate_index=0,
)

Why separate Drafter and Evaluator?

Now that we’ve seen both components, it’s worth explaining why we separate them. The critique-and-refine pattern has several benefits:

1. Easier Evaluation: Spotting errors is simpler than perfect generation, so the Evaluator doesn’t need to be a flawless translator.
2. Context Preservation: The original translator retains the reasoning for its choices when refining based on feedback.
3. Bias Avoidance: Separating roles prevents the self-approval bias of a single model translating and evaluating its own work.
4. Flexibility/Cost: Different models can be used for each role (e.g., a fast drafting model and a more capable evaluator).

Retry, Reflection, and Self-Correction

The feedback loop between Drafter and Evaluator is a continuous mechanism to ensure if all candidates fail evaluation, the system doesn’t give up. It learns from the failure and tries again, up to three times.

We find this iterative refinement yields the largest gains in the first 1–2 cycles, so the three-attempt limit balances quality improvement against latency and cost.

When the Evaluator rejects all candidates, its critique is captured and injected into the Drafter’s next attempt. The prompt explicitly instructs the Drafter to address previous failures:

# This critique is prepended to the next Drafter prompt
critique_for_retry = """
All candidates failed glossary compliance. Key issues:
- "Ride" must be translated as "trajet" per Lyft Quebec glossary
- Do not use "course" which is European French
"""

The retry prompt explicitly instructs the Drafter to address previous failures: correct glossary usage, fix placeholders, adjust tone. We find this iterative refinement process results in a success rate of over 95% across most languages.

In cases of persistent disagreement between the Drafter and Evaluator, we limit retries to three attempts. If no acceptable translation emerges, product teams must wait for a linguist in our TMS to handle the string, or they can internally request an expedited translation for urgent launches.

Context Injection

LLMs don’t have access to Lyft’s terminology guidelines or know, for example, that {driver_name} is a variable in code that must be preserved. We inject this context through careful prompt engineering.

Terminology and reference data aim to reduce the chances of LLM hallucination, a common failure mode. For example, “Driver” might become “Conducteur” in French when Lyft’s official term is “Chauffeur.” To solve this, we maintain data sources of Lyft terms and phrases that the Drafter and Evaluator reference when completing their tasks:

• Glossary terms: Individual words and even phrases with their official translations (e.g., “Driver” → “Chauffeur” in French)
• Do-not-translate lists: Brand names, product names, and proper nouns that should remain in English

All of these assets are maintained in the TMS where linguists routinely review them and update the datasets. The Translation Pipeline then pulls them in periodically for use at translation time to augment prompts to the Drafter and Evaluator:

Deterministic Guardrails

LLMs are probabilistic as they may hallucinate variable placeholders, mangle URLs, or strip formatting. For production translation, we need deterministic validation. Without guardrails, we observed consistent hallucination patterns: placeholder translation was the most common failure mode (e.g., {driver_name} becoming {nom_du_conducteur}), followed by placeholder omission and format string corruption.

Our guardrail system operates in two phases: pre-translation extraction and post-translation validation. This layered approach ensures protected elements survive the LLM round-trip intact.

Pre-Translation: Extract and Tokenize

Before any content reaches the LLM, we extract elements that must be preserved exactly. This includes:

• Variables and format strings: Curly-brace variables like {driver_name} and {eta}, printf-style placeholders (%s, %1$s, %@)
• URLs and identifiers: Links, email addresses, and region codes requiring exact preservation
• Structural elements: HTML tags (with balance checking), escape sequences (ex. \n, \t, \r)

We use regex pattern matching to identify these elements and replace them with numbered tokens (__PH_0__, __PH_1__, etc.) that the LLM is less likely to accidentally mangle. The numbering also handles reordering as different languages have different grammatical structures, so {vehicle_type} might appear before {eta_minutes} in English but after it in German. The numbered tokens maintain the correct mapping regardless of where they end up in the translated string.

Sample Transformation

Original: "Hey {first_name}! Your Lyft arrives at {eta}.\nTrack: https://lyft.com/r/abc"

Masked: "Hey __PH_0__! Your Lyft arrives at __PH_1__.__PH_2__Track: __PH_3__"

Mapping:
    __PH_0__ → {first_name}
    __PH_1__ → {eta}
    __PH_2__ → \n
    __PH_3__ → https://lyft.com/r/abc

The mapping is injected into the prompt as human-readable context, giving the LLM explicit instructions about what to preserve.

Post-Translation: Validate and Restore

After the LLM returns a translation, we run deterministic validation before accepting it. Validation checks three conditions:

1. Presence: Every expected token appears exactly once
2. No hallucination: No unexpected tokens were introduced
3. Structure: Structured content, like HTML tags, are balanced (i.e. open tags have corresponding close tags)

When validation fails, we inject the specific errors into a retry prompt rather than giving up. The LLM knows exactly what went wrong (e.g., “Missing placeholder: __PH_2__ (\n)”) and can correct it on the next attempt. This creates a deterministic feedback loop that catches mistakes prompt instructions alone would miss.

Once validation passes, restoration is a simple token-for-original swap:

Translated: "Salut __PH_0__! Votre Lyft arrive à __PH_1__.__PH_2__Suivre: __PH_3__"

Restored: "Salut {first_name}! Votre Lyft arrive à {eta}.\nSuivre: https://lyft.com/r/abc"

Experimentation & Configuration

Initially in development, we treated prompts as configuration, tweaking ad hoc, testing manually, deploying with little review. We found, for example, a small wording change in the Evaluator’s prompt can cause a rise in false rejections. We had no way to identify the regression or roll back.

Now, prompts are version-controlled production code. Every prompt template lives in our Translation Pipeline alongside the code, subject to the same review process. Each prompt version includes a changelog documenting what changed and why. Prompt changes require testing to demonstrate the change improves (or at least doesn’t regress) translation quality on our evaluation suite.

To do this, we evaluate changes against ground truth translations in our TMS, flagging divergences from linguist-approved versions. These differences are reviewed before any prompt or model change receives production traffic.

Multi-Model Experimentation

We don’t commit to a single model provider. The pipeline abstracts the LLM layer, but experimentation happens through Lyft’s configuration infrastructure. This gives us:

• Traffic splitting: Roll out new models to 5%, 20%, 50% of requests
• Shadow mode: Run new models in parallel without affecting production
• Per-locale overrides: Different models/prompts for different markets

Instant rollback: Revert to previous configuration without a deploy

# Example config schema
drafter:
  model: <fast-generation-model>
  fallback: <fallback-model>
  prompt: <prompt-id>
  shadow:
    model: <candidate-model>
    prompt: <prompt-id>
    traffic_percent: 10

evaluator:
  model: <reasoning-model>
  fallback: <fallback-model>
  prompt: <prompt-id>
  reasoning_effort: medium

locale_overrides:
  <locale_code>:
    evaluator:
      reasoning_effort: high

We run experiments comparing model combinations across major model providers. This is how we determined that faster, cheaper models (GPT’s mini models, Claude Haiku) perform comparably to frontier models for initial generation while reasoning-focused models significantly outperform on catching subtle errors.

Locale-Specific Tuning

Not all locales need the same prompt. We discovered this when expanding to English variants (en-GB and en-CA). These locales require only orthographic changes like spelling (“color” → “colour”), punctuation, and occasional vocabulary swaps (“trunk” → “boot”).

Without locale-specific guidance, the LLM interprets “translate to British English” as license for entire rewrites. A simple “Try again” button became “Have another go” — technically valid British English, but a jarring tone shift that didn’t match our UI voice.

As shown in the config above, we use locale overrides that constrain the transformation scope. For example:

LOCALE_OVERRIDE_EN_GB = """
You are adapting American English text for British English speakers.

IMPORTANT: This is an ORTHOGRAPHIC adaptation, not a full translation.

Only change:
- Spelling (color → colour, center → centre, organize → organise)
- Punctuation conventions where required

DO NOT change:
- Tone or voice
- Sentence structure  
- Casual vs. formal register
- Idioms or expressions (unless they are specifically American and would confuse UK readers)

The goal is that a British reader sees familiar spelling, not that the text "sounds British."
"""

This constraint dramatically reduced over-adaptation errors for English variants while maintaining consistent brand voice across locales. We now apply similar scoping constraints for other closely-related language pairs where full translation would be overkill.

Conclusion

Through careful oversight, we see 95% of translations need no significant changes after linguists review. The remaining 5% usually represent genuinely difficult cases (regional idioms, legal disclaimers, brand voice decisions) where human oversight adds real value.

Re-architecting Lyft’s translation pipeline taught us that LLMs excel at translation when you design for their limitations. The iterative Drafter/Evaluator pattern mirrors how human translators actually work: generate options, critique them, refine. Deterministic guardrails catch what prompts alone cannot enforce. And treating prompts as production code (versioned, tested, reviewed) prevents the silent regressions that plague ad hoc LLM deployments.

Acknowledgements

Thank you to the following team members for making this possible: Janani Sundarrajan, Sebastiano Bea, Jiachen Jiang, Alex Hartwell, Yousra Saidani, Adriana Deneault, Yuantao Ji, Miles Krell, Alex Atencio.

Lyft is hiring! If you’re passionate about leveraging AI to enhance millions of user experiences, visit Lyft Careers to see our openings.

*The Lyft Terms of Service and other product-specific terms and conditions remain strictly human translated.

Scaling Localization with AI at Lyft was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Causal Frontier: Measurement Beyond Randomization

The gold standard for determining the causal impact of a policy or product change at a company like Lyft is the A/B test (randomized experiment). By randomly assigning users to a treatment or control group, A/B tests inherently eliminate bias, providing clean estimates of the Average Treatment Effect (ATE). However, many critical business questions and large-scale initiatives simply cannot be randomized. This forces scientists to move past traditional experimentation and leverage quasi-experimental methods.

We rely on non-randomized measurement in several key scenarios across Lyft:

• Partnerships and Policies: Assessing the incremental impact of a partnership (e.g., linking two company accounts) is often a non-randomized assignment. Since these collaborations require coordinated operational work across both companies and are typically announced or promoted broadly, this makes controlled randomization impractical.
• Long-Term Effect (LTE): Measuring effects that unfold over a long period, like the LTE of high prices on future rides, is typically handled by observational studies.
• Post-Launch Evaluation: Continuous monitoring of a policy after it has been fully rolled out requires a method that doesn’t involve costly holdout groups or degradation tests.
• Biased Data: In cases where pre-existing experimental data is found to have an imbalance, a quasi-experimental approach can potentially leverage the biased data instead of requiring a costly rerun.

Introducing Doubly Robust Models: Causal Inference Without Randomness

To address these non-randomized measurement needs, Lyft relies on various quasi-experiment estimators. In this blog we specifically focus on using the Augmented Inverse Propensity Weighting (AIPW) model. This model was first established at Lyft to measure the impact of a negative user experience on future topline metrics like rides and bookings; AIPW is a form of doubly robust estimation used to estimate the Average Treatment Effect (ATE) or Average Treatment Effect on Treated (ATT).

The doubly robust nature is what makes the AIPW model so powerful: the formula relies on fitting two separate models for a given set of confounders X and treatment D — the outcome model, g(D, X), and the propensity score model, e(X) — and the overall estimator consistently estimates the true ATE if at least one of these two models is correctly specified.

The Critical Need for Validation

Unlike A/B tests, quasi-experiments lack the fundamental guarantee of unbiasedness provided by randomization. This makes validation, diagnostics, and trust-building absolutely critical. If we cannot randomize the treatment assignment, we must rely on a large set of control variables (confounders) to adjust for pre-exposure differences between the treatment and control groups. This adjustment process makes the analysis highly susceptible to bias if not correctly executed and rigorously checked. The core validation pillars are centered on managing confounders and monitoring model diagnostics, as we’ll explore in the next section.

The Validation Engine: Confounders and Diagnostics

Since quasi-experiments lack the inherent randomization of A/B tests, we must prove the validity of our causal estimates through rigorous inputs and explicit model diagnostics. We build trust in every single result through Confounder Management and a Diagnostic Scorecard.

1. Rigorous Data and Confounder Inputs

The greatest threat to a quasi-experiment’s validity is selection bias — users with certain characters are more likely to be treated by the policy, making the control group not a proper counterfactual for the treatment group. The only way to correct for this is by meticulously identifying and controlling for these pre-existing differences, known as confounders. At Lyft, we have developed a quasi-experimentation platform that makes this process mandatory and customizable.

a) The Confounder Set Requirement

The analysis requires potentially hundreds of features/confounders to adequately reduce bias. The quasi-experimentation platform doesn’t allow users to skip this step; users are required to select or define one confounder set for AIPW analysis.

• Pre-defined sets: Users can choose from established sets on the platform that are pre-defined as SQL queries to pull the relevant data.
• Customization is key: Crucially, the platform exposes the underlying SQL query, allowing users to customize, modify, add, or remove variables within a set to perfectly match their specific use case.
• Preventing leakage: The system automatically ensures confounder data is gathered from before the user’s first exposure date to the treatment, preventing leaky covariates that would incorrectly attribute the treatment’s effect to a non-causal variable.

b) The Balancing Act: Correcting for Downsampling Bias

A core task in data preparation for AIPW is balancing the size of the treatment and control groups in the presence of imbalance data. When the one group is smaller than the other, the system randomly downsamples the larger group to achieve balance. However, this random downsampling introduces a new scientific challenge:

• Non-Representative Samples: Even if the original sample satisfies model assumptions across treatment groups, taking a random subset of the larger group may make that subset non-representative of the true population distribution with respect to the confounders.

To recover the true population-level ATE, we apply a correction in the AIPW estimates per Ballinari (2024), which involves two related concepts:

i. Propensity Score Correction: We must convert the sample-estimated propensity score, p_s(X), back into the true population propensity score, p(X), using a conversion formula that accounts for the downsampling ratio L. This ensures the model uses the real probability of being treated in the population, not just in the sample.

ii. Outcome Reweighting: After the propensity score correction, the efficient scores must be subjected to a weighted average based on the sampling ratio. Specifically, for every observation in the downsampled control group, we must account for the fact that it represents 1/L copies of the original population. This process involves uniformly rescaling the weights of each observation so they average to 1.

This reweighting of outcomes is a critical scientific refinement currently being implemented to debias the results and ensure the final ATE estimate accurately reflects the total impact on the original population.

2. Model Diagnostics and Assumptions

The output of every AIPW analysis on our quasi experimentation platform is a scorecard with two essential tabs: the Scores Tab and the Diagnostic Tab. The Diagnostic Tab is where we evaluate the model’s health to look for clues that its fundamental assumptions hold, providing visual proof of the estimation quality. Below are two of the tens of diagnostics we show users:

i. Checking Common Support (Propensity Overlap)

The AIPW model relies on the common support assumption (or overlapping assumption), meaning that for any given set of confounders, there must be a non-zero probability of being in either the treatment or control group (0<e(X)<1). If this assumption is violated, the inverse propensity score weights explode. This means that the outliers will receive an extreme weight and take over the overall effect.

The platform provides a histogram visualizing the Propensity Overlap between the control and treatment groups. Below are examples of sufficient and insufficient overlaps between treated and control groups

If the overlap is poor, the causal estimates are unreliable. The analysis is further refined by applying a user-defined trim level to discard extreme propensity score values and satisfy the common support assumption.

ii. Ensuring Covariate Balance

The primary goal of using confounders is to achieve a state where the treatment and control groups are statistically similar on all observed characteristics.

A key diagnostic graph illustrates the Covariate Balance before and after adjustment.

• Before Adjustment (Original): Shows the initial difference (bias) in each feature between the groups.
• After Adjustment (IPW/IPW Trimmed): Shows how well the model has reduced the difference. Successful adjustment moves the metrics close to the dotted zero line, confirming that the confounders are properly balanced.

Measuring Bias in AIPW

Since AIPW is a new addition to our causal inference platform, we needed to verify its performance. We decided to use randomized experiments as our “ground truth” to see how closely our observational AIPW estimates matched experimental results. We looked for an empirical setting where there was both experimental and observational data for the same intervention, time period, and regional markets. This allows us to compare experimental results directly against AIPW estimates derived from observational data, where we controlled for biases using a large set of confounders.

Empirical Setting: Weekly Ride Challenges

Weekly ride challenges are conditional bonuses given to drivers if they complete a certain number of rides within a week. Since these bonus payments are costly, Lyft uses an algorithm to target ride challenges efficiently given a fixed budget. Specifically, the algorithm allocates ride challenges to drivers who are expected to be most responsive. AIPW estimates the effect of ride challenges using observational data by comparing drivers who do versus do not receive the challenges (within the targeted group) while having similar propensity scores.

The experimental ground truth leverages an ongoing experiment to evaluate the effectiveness of ride challenges. In this holdout experiment, drivers are randomly assigned into a “control” group that will not receive a ride challenge. Remaining drivers in the “targeted” group may or may not receive a ride challenge depending on whether they are targeted by the algorithm (illustrated below). AIPW estimates the treatment effect on observational data by comparing drivers who do versus do not receive the challenge within the targeted group.

For the experimental ground truth, we cannot directly compare drivers in targeted versus control groups. This is because many drivers in the targeted group do not receive the ride challenge, so a direct comparison would yield unbiased estimates for “potentially” being exposed to ride challenges. This diluted effect would not be comparable with AIPW, which is comparing drivers with versus without the ride challenge.

To get around this issue, we used instrumental variables (IV) to estimate the Local Average Treatment Effect (LATE). Randomized assignment into targeted versus control groups is the “instrument” that influences whether the driver receives the ride challenge, but it does not perfectly determine it. LATE is the treatment effect of ride challenges for “compliers”: drivers who receive the challenge if assigned to the targeted group, and drivers in the control group who would have received the ride challenge if they had been assigned to the targeted group. Conceptually, compliers are “potential targets” who would be targeted by the algorithm for ride challenges, so whether they receive the challenge is solely determined by random assignment into targeted versus control groups. We compare LATE from experimental data with Average Treatment Effect for Treated units (ATET) from observational data, since they both correspond to the effect of ride challenges for drivers who are targeted by the algorithm.

What We Found

Through this careful “apples-to-apples” comparison, we found that AIPW estimates obtained from observational data understate ground truth magnitudes obtained from experimental data through IV estimates. In the first row of the table below, AIPW estimates that ride challenges increase driver hours by 11.1%, while the experimental estimate suggests a 13.3% effect. The AIPW estimate is 2.1 percentage points below the experimental estimate, or equivalently, 16.2% lower than the ground truth magnitude.

To understand the root cause, we explored whether this discrepancy is because the analysis sample for AIPW is different from the overall population. Since AIPW requires the same propensity score values to be observed in treated and control groups, the analysis sample trims out drivers whose propensity scores are too high or too low. Since this is a subset of the overall population, the discrepancy observed earlier may reflect differences in the populations they represent.

To test this hypothesis, we used the propensity score model to find a comparable subset in the experimental data with the same range of hypothetical propensity scores. The second row of the table shows that the discrepancy between observational AIPW and experimental IV is minimal when comparing within this subset. The estimated effect from AIPW is only -0.3 percentage points below the experimental ground truth (smaller by 2.57%). This large reduction in discrepancy shows that AIPW understates the ground truth primarily because the trimmed analysis sample is different from the overall population.

However, this doesn’t mean that AIPW will always understate the experimental ground truth. In fact, there is a well-known study at Facebook that actually found the opposite. Whether AIPW understates or overstates the true effect heavily depends on the empirical context. In our case, ride challenges are targeted to the most effective drivers, who tend to have higher propensity scores. It is difficult to find drivers who have similar propensity scores but did not receive the ride challenge, since most of them are targeted by the algorithm. As a result, the most effective drivers in the treated group end up being trimmed out of the analysis sample due to a lack of comparable drivers in the control group. Since AIPW estimates the treatment effect on an analysis sample that excludes the most effective drivers, it makes sense that AIPW understates the true effect on the overall population that includes such drivers.

Improving the AIPW Platform Through our Learnings

To summarize our findings, we initially found that the estimated effect from AIPW understates the experimental ground truth. A deep dive revealed two key drivers:

1. Hidden Confounders: Even with a rich feature set, hidden confounders can still bias estimates.
2. Propensity Trimming: Trimming users with extreme propensity scores (to satisfy the common support assumption) shifted our analysis sample so it was no longer representative of the target population.

This discrepancy disappeared once we subset the experimental data to match the analysis sample in AIPW. The “bias” was largely due to measuring the effect on a narrower segment of users who behaved differently than the general population.

Unobserved confounding and trimming on propensity scores can be more problematic in some applications than others. Since AIPW is not reliable when these issues are severe, we introduced two additional diagnostics to recognize when our estimates are vulnerable to these issues:

• Marginal Sensitivity Model (Zhao et al., 2018): This metric quantifies the robustness of estimates to hidden confounders, which bias our propensity scores. It measures the required discrepancy between true and estimated propensities to change our directional conclusions. More specifically, the discrepancy factor G quantifies the extent to which true versus observed odds ratios diverge due to unobserved confounding (1/G ≤ odds ratio ≤ G). This is used to infer bounds on true propensity scores given observed propensities e(X) and factor . We use this to construct worst-case bounds on treatment effects, and the diagnostic metric is the smallest value so that the worst case bounds touch zero. Higher values for this sensitivity metric indicate that the direction of true impact does not change even with large discrepancies in propensity scores due to unobserved confounding.
• Covariate Comparisons for Trimmed vs. Untrimmed: We plot observed characteristics of trimmed users against untrimmed users. If trimmed users look significantly different from untrimmed users, then our analysis sample would lack external validity to the overall population. Each point below reports the normalized difference between trimmed versus untrimmed users for each observed covariate, separately for treated and control groups. In the example below, users with certain characteristics are more likely to be trimmed in the treated group than they are in the control group. If users with certain characteristics are more likely to be trimmed, then the estimated ATE from the analysis sample may overstate or understate the true ATE in the overall population.

Conclusion: Trustworthy Causal Impact at Scale

Integrating AIPW into our platform has unlocked the ability to measure causal impact when A/B tests are infeasible. Since quasi-experimentation platform is a new capability at Lyft, we needed to rigorously validate the model to build stakeholders’ trust around the tool. Validation will continue to be essential as we onboard additional tools to the platform, which will help us deliver trustworthy insights for our most complex challenges at Lyft’s marketplace.

Lyft is hiring! If you’re passionate about experimentation and measurement, visit Lyft Careers to see our openings.

Trusting the Untestable: Validation and Diagnostics for the Doubly Robust Models was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article expands upon a presentation from the Feature Store Summit 2025, which can be viewed in full here. There is also another video available on the evolution of Lyft’s Feature Store from DE4AI 2024.

Introduction and Core Purpose

Lyft’s Feature Store stands as a core infrastructural pillar within its Data Platform organization, designed to optimize the management and deployment of Machine Learning (ML) features at massive scale. Its primary objective is to centralize feature engineering efforts, guaranteeing uniformity across diverse models and workflows that perform important data-driven decision making across the entire rideshare stack. By streamlining the entire lifecycle — from feature creation and storage to low-latency access and high-throughput processing — it facilitates effective offline and online model training and inference.

This post will provide a refreshed look (since 5 years ago) at the architectural evolution, practical applications, performance tuning, and significant improvements in developer experience we’ve performed over the past few years to improve efficiency, scalability, performance, and user accessibility. Ultimately, we aim to illustrate how the Feature Store empowers Lyft engineers to develop highly effective service components and ML models, a capability that is becoming vital for emerging AI and Large Language Model (LLM) applications.

Defining Our Audience and Impact

Before diving into the system’s architecture, it’s essential to understand the importance and breadth of our user base. Our product is mission-critical, serving diverse engineering functions across the company.

Understanding this subset of varied high-impact use cases (just five of 60+) is key to appreciating the necessary robustness and flexibility of the Feature Store design.

The Feature Store Architecture: A Platform of Platforms

Our system is best described as a platform of platforms. While the full architecture diagram is complex, we can break it down into three digestible components: Batch, Online, and Streaming features.

Batch Feature Ingestion and Serving

Batch features are the most widely used family of features within our platform. These features are defined from existing Hive data tables and represent a set of standardized data points that are calculated and refreshed on a set cadence, typically on a daily basis.

The ingestion process begins when customers define features using a Spark SQL query and a simple JSON file representing the dedicated configuration metadata.

A Python cron service reads these configurations and automatically generates an Astronomer-hosted Airflow Directed Acyclic Graph (DAG). Crucially, these generated DAGs are production-ready out-of-the-box. They handle:

1. Executing the Spark SQL query to compute the feature data
2. Storing the feature data to both the offline and online data paths
3. Running integrated data quality checks
4. Compatibility for feature discovery

The executed DAG generates a dataframe and delivers the results to two distinct paths:

• Offline Data Path: The feature data is stored in Hive tables for historical data analysis and machine learning model training.
• Online Data Path: The processed features are translated and sent to our low-latency online serving layer for real-time inference.

The Online Serving Layer

Our online serving layer, referred to as dsfeatures (short for “data science features”), is central to our feature serving capability. It is an optimized wrapper over various AWS data stores, providing a reliable and ultra-low-latency retrieval mechanism for real-time serving.

The core structure of dsfeatures is:

• Backing Store: DynamoDB is utilized as the primary, persistent source for features. It uses various metadata fields as the primary key with a GSI for GDPR deletion efficiency.
• Performance Cache: A ValKey write-through LRU cache is deployed on top of DynamoDB to facilitate ultra-low-latency retrievals by storing the most frequently-accessed (meta)data with a generous TTL.
• Embeddings: An OpenSearch integration is utilized specifically for serving embedding features, which require specialized indexing and retrieval capabilities.

Customer Interaction and Data Retrieval

The dsfeatures service centralizes how both internal DAGs and external customers interact with the feature data. From a customer’s perspective, data retrieval and management are straightforward, facilitated by our dedicated Software Development Kits (SDKs): go-lyft-features (Golang) and lyft-dsp-features (Python).

Services utilize these SDKs to make API calls directly to the dsfeatures service. The most common retrieval methods are Get or BatchGet calls, which the service handles and returns the requested data in a developer-friendly format.

Crucially, the SDK libraries expose full CRUD (Create, Read, Update, Delete) operations. This capability allows system components, such as our internal Airflow DAGs, to read and write features, and even lets customers manage real-time features ad-hoc by directly invoking these API calls against our data stores.

The Streaming Pipeline

While batch features are essential, we also rely on streaming features to ensure data recency for low-latency applications and customer demands.

Our streaming pipeline follows a robust, multi-stage architecture to process features in real time.

1. Ingestion: Streaming applications, developed primarily using Apache Flink, read analytic events from Kafka topics (or sometimes Kinesis streams).
2. Transformation: The Flink applications perform necessary initial transformations on the data. This includes manual metadata creation and proper value formatting.
3. Ingest Service: The feature payloads from customer applications are sunk to spfeaturesingest— our “Streaming Platform feature ingest” Flink application. It handles the (de)serialization of the payloads and subsequent interaction with dsfeatures via WRITE API call(s) to ensure the features are processed in the right format, guaranteeing availability for online retrieval by other services.

Regardless of the ingestion method (batch, streaming, or on-demand), the Feature Store maintains uniform metadata and strongly consistent reads. This is crucial for ensuring feature accuracy and availability across all consuming applications and services.

Prioritizing User Experience and Feature Governance

Understanding our architecture is only half the picture; the user experience is central to maximizing productivity. Our Feature Store primarily serves two frequent personas: Software Engineers (who drive service activity) and ML Modelers (who design features and models). Since developers can often embody both roles or work in mixed teams, we’ve designed our system to simplify interaction for everyone.

Ease of Use and Quick Iteration

We learned early on that our core personas are particularly proficient in SQL and place a high value on quick iteration. To facilitate this, our design centers on:

• Performant SparkSQL as the preferred processing engine and language for batch feature queries.
• Simple JSON configuration files to define feature behavior/metadata.

This approach ensures that developers can focus on their primary responsibilities without technical intricacies getting in their way. The Feature Store presents this user-friendly interface and APIs that simplify interaction, minimizing the learning curve and facilitating rapid adoption. Engineers can readily register, update, and retrieve features using well-documented APIs and well-supported examples.

Feature Governance and Metadata

Our configuration files include essential metadata such as ownership details, urgency tiering, run-to-run carryover/rollup logic, and explicit feature naming & data-typing. This metadata is crucial for more than just customer clarity; it is vital for our monitoring and observability systems, aiding in debugging and providing posterity of feature history (both metadata and values).

To support robust feature management, the Feature Store incorporates versioning and lineage tracking capabilities, encapsulated in our metadata:

• Versioning allows developers to monitor changes to features over time, ensuring the use of correct versions for their models/services. If the SQL or expected feature behavior undergoes business logic changes, a version bump is expected.
• Lineage tracking offers crucial insights into the origin and transformation of features, enhancing both transparency and accountability across the platform.

Accelerating the Feature Engineering Workflow

To complement our simple SQL/JSON foundation, we’ve integrated with Kyte to accelerate the development lifecycle. This homegrown solution is central to Airflow local development at Lyft — more about Kyte can be learned here.

We provide a custom Command Line Interface (CLI) within the Kyte environment that significantly improves the feature prototyping experience, allowing users to:

• Perform feature validation against their configurations.
• Test SQL runs for immediate feedback and investigable results.
• Execute DAG runs in a local environment.
• Confidently backfill previous dates against their DAGs.

Feature Discoverability

Once features are generating data, discoverability is the next crucial step. Our generated DAGs automatically tag feature metadata within Amundsen, Lyft’s central data discovery platform. This integration allows users to easily search for existing features, a critical step in preventing the duplication of efforts and reducing wasted engineering work.

By simplifying data discovery and feature engineering, we solidify the Feature Store’s crucial role in the ML Model Development lifecycle, ensuring a strong partnership with our Machine Learning Platform (MLP) team, which owns the remaining model-building steps.

Platform Evolution and Prioritization

While our current setup is robust, it has evolved significantly. We made deliberate, strong steps to prioritize efficiency and improved customer experience over the last few years.

Streamlining for Core Success

• Orchestration Migration: We completed two migration hops, ultimately transitioning our core orchestration platform from in-house Flyte to fully-managed Astronomer (tradeoffs discussed here). This offloads ETL platform stability issues to external engineers while allowing our Orchestration team to invest in higher priority internal initiatives.
• Niche Support: In the spirit of the Pareto principle, we removed compatibility for alternative query engines like HiveQL and Redshift. This allowed us to reduce support for minor niche use cases and invest more deeply in our core use cases that drive the most value for Lyft.
• Staging Environment: We unlocked a reliable staging capability for our platform, boosting confidence in the release of urgent or sensitive features through easier prototyping and E2E testing in non-production environments.
• Standardizing Access: We developed the long-requested Golang SDK and an offline-data Python SDK. These normalize customer activity against our data, making it more monitorable, accessible, and understandable.
• Data Contracts: We are actively implementing the organization’s “Data Contracts” initiative, which enforces explicit expectations regarding feature freshness, ownership, and quality. This is crucial for maintaining trust as data generation scales rapidly.

Investing in New Capabilities

• Streaming Abstraction: A RealtimeMLPipeline interface was designed for Flink applications, aiming to make generating streaming features a low-friction process and meeting growing internal demand.
• Embedding Support: We integrated OpenSearch into the entire Feature Store stack to support embedding features, which has generated significant interest and new use cases.
• Staffing Investment: Significant new staffing is being dedicated to the Stream Compute team, reflecting the rising importance and complexity of this class of features. See Lyft Careers for open roles!

Optimization: Transparency and Lean Retrieval

With a platform of this size and importance, efficiency is in continuous battle against egregious bloat. Our recent optimization strategy focused on two core areas: transparency and reliability.

Data Generation: Transparency

For the generation pipeline, we focused on transparency. Improved monitoring against failed DAGs and tasks, coupled with strong ownership tracking, has made debugging faulty features significantly easier. This, in turn, has provided the confidence to actively deprecate unused or incorrectly used features, ensuring a healthier, more maintainable feature space for the platform team while saving resources for others (f.e. less wasted Spark compute & Astronomer task scheduling resources).

Data Retrieval: Reliability

Our customers’ primary requests for retrieval improvements centered on two metrics: better latencies and higher success rates. Given that AWS datastores are our main source of unpredictable transient failures and high P999 tail latencies (latency is generally not part of their SLAs), our strategy was to focus on being as lean as possible to make this less likely & disruptive:

• Cache Modernization: We upgraded our cache technology and version, adopting ValKey as the latest solution over ElastiCache.
• Payload Optimization: We removed unnecessary fields from the retrieval code path and the cache payload to reduce data transfer size and processing time.
• Rightsizing: We right-sized our EKS pods in dsfeatures to minimize the aggregate number of necessary Redis connections, which historically resulted in networking issues.
• Policy Hardening: We improved retry and timeout policies both within customer services and our own SDKs to prevent premature network exits and degraded success rates.
• TTL Management: We increased the cache TTL (Time-To-Live) as much as possible for both feature values and the metadata used in retrieval decision-making, carefully balancing latency performance against storage cost.

Results: Unprecedented Growth and Performance

These focused adjustments and improvements led to remarkable trends across the platform.

• Latency Reduction: We cut the standard P95 latency experience during read operations by a full third. This had a tangible downstream effect, evidenced by increased customer SRs and a significant reduction in customer-support threads in our internal Slack channels, directly aiding on-call engineers.
• Batch Feature Growth: Our batch features, the largest family by volume, grew by over 12% year-over-year. This growth occurred despite our active feature deprecation efforts, suggesting a highly positive experience and deepening partnership from the teams that utilize our platform.
• Caller Growth: The number of distinct production service callers increased by almost 25% over the last year. Since each distinct caller represents a fundamentally unique use case — whether a separate service or a new facet within an existing one — it strongly validates the company’s increasing appetite for feature usage and the value it brings.
• Scale of Activity: Aggregate R/W activity on the platform increased by over a trillion in raw count, based on conservative extrapolation. This serves as a powerful reminder of the enormous and continuously growing scale at which our platform operates.

Considering the remarkable progress of the past year, the potential for future growth and impact is truly immense.

Conclusion

Lyft’s Feature Store is a testament to robust data infrastructure, propelling ML excellence and operational efficiency within the company. Its architecture effectively addresses the complexities inherent in managing and deploying ML features at scale. This strategic approach ensures Lyft remains at the forefront of data-driven decision-making.

More than just a data management tool, the Feature Store at Lyft serves as a vital catalyst for innovation and a key enabler of Lyft’s overarching mission: to serve & connect.

As the machine learning landscape continues its rapid evolution, the Feature Store will undoubtedly retain its critical role within Lyft’s data strategy, driving advancements and solidifying Lyft’s leadership in data-driven technology. We are truly excited to see what the future holds, and see how we can continue to serve & connect our customers.

Acknowledgements

I would like to thank Devon Mittow, Janice Lee, Yigal Kassel for all their direct contributions to the feature space. The charter’s achievements would not have been possible without them.

Further thanks to Maheep Myneni, Arda Kuyumcu, and Aniruddh Adkar for being incredible team members whose collaboration and support unlocked the confidence to embark on all these important projects & developments.

Finally, thanks to Prem Santosh Udaya Shankar, Rohit Menon, Arbaz Mirza, Konstantin Gizdarski, Yunhao Qing and Brian Balser whose technical and management support, past and present, led to this charter’s overall success and growth.

Further Reading

• Learn about our RTML architecture: Building Real-time Machine Learning Foundations at Lyft by Konstantin Gizdarski and Martin Liu
• Learn about a major RTML use case: Real-Time Spatial Temporal Forecasting @ Lyft by Rakesh Kumar and Josh Xi
• Learn about our ML platform architecture: LyftLearn Evolution: Rethinking ML Platform Architecture by Yaroslav Yatsiuk

Lyft is hiring! If you’re passionate about Infrastructure & Data Platform, visit Lyft Careers to see our openings.

Lyft’s Feature Store: Architecture, Optimization, and Evolution was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Intro

When working with Python, memory management often feels like a solved problem. The garbage collector quietly does its job, and unlike C or C++, we rarely think about malloc or free. This doesn’t mean that there are no memory leaks in Python. Reference cycles, unreleased resources like connection pooling, global caches, etc can slowly inflate your process’s memory footprint. You might not notice it at first, until your worker starts OOM-ing, latency creeps up, or container restarts become mysteriously frequent.

In this post, we’ll share the story of a real-world memory leak we encountered during a Python upgrade — how we discovered it, the tools and techniques we used to investigate, and the lessons we learned.

What happened after upgrading to Python 3.10?

Back in the summer of 2024, we had an initiative at Lyft to upgrade all of our Python services from v3.8 to 3.10 as v3.8 was scheduled to be EoL by the end of 2024. You can find more details on how our awesome Backend Foundations team at Lyft does Python upgrade across hundreds of repos at scale here. The upgrade involved two phases: the first phase was to upgrade all the dependencies to be Python 3.10 compatible, and the second phase was to upgrade the services to Python 3.10. The dependency upgrades went smoothly for all services and then the phase to upgrade all services to Python 3.10 rolled out. While all services were running Python 3.10 smoothly, there was one service for which the upgrade in the test environment caused a flurry of latency spikes, resulting in timeouts for downstream services.

After profiling the APIs with increased latency with stats, we found that the source of latency were repository queries to the dynamo tables. Specifically, we had pynamodb based repository queries which would spin up a bunch of greenlets to fetch data from multiple tables and combine the result which was showing increased timeouts. The individual queries themselves were fine; however, it was the thread join which took the longest time causing the worker to timeout (default = 30 seconds).

The other interesting thing we found was memory consumption slowly creeping up with time in all of the pods.

At this point, we weren’t sure if there was something up with gevent/greenlet causing the memory leak or the memory leak causing the latency since decreased memory availability can cause increasing page fetches from the disk. We first checked if the gevent monitoring thread detected any event loop blocks, which could potentially cause these timeouts. We then pivoted to find out the root cause of the memory leak. Fortunately, Lyft has an internal library which can help profile memory which is based on tracemalloc.

Memory profiling tool

The Lyft memory profiling tool is based on tracemalloc. To capture the memory trace for a given gunicorn process, we registered the worker process to listen to USR2 signal during the application initialization phase.

# app/__init__.py

MemoryProfiler().register_handlers()

# mem_profiler.py pseudo code

class MemoryProfiler: 
    def __init__(self) -> None:
        self._state_machine = self._profiling_state_machine()

     def register_handlers(self) -> None:
        # Register gunicorn worker to listen to USR2 to dump traces
        signal.signal(signal.SIGUSR2, self.handle_signal)  

    def handle_signal(self, signum: signal.Signals, frame: FrameType) -> None:
        next(self._state_machine)

    def _profiling_state_machine(self) -> Generator[None, None, None]:
        while True:
            try:
                self.start_tracing() # tracemalloc.start()
                self.memory_dump() # Create snapshot1
                yield
                self.memory_dump() # Create snaphot2,compare with snapshot1, and dump the difference in a file 
            finally:
                if tracemalloc.is_tracing():
                    tracemalloc.stop()

Let’s start the tracing!

Ok, now that we had the memory profiler setup, we are ready for some tracing to find the source of the leak. To start the tracing, we send USR2 signal to the gunicorn process in the K8s pod to start tracing and send the signal again after some time interval to capture the stack trace with highest memory usage.

ps aux

Now, we will send a USR2 signal to worker with pid 12

kill -USR2 12

Upon checking the process list again….

ps aux

… we observed that the gunicorn process we planned to trace got killed 🙁

It took several hours of debugging and a journey back to one of my favorite class to find the root of the issue — preload. To understand why preload caused the process to be killed, we first need to understand how gunicorn works.

Gunicorn

Gunicorn works on the pre-fork model. There is a leader process which forks a bunch of workers. There are two ways to fork the workers:

No Preload

When the leader process forks a worker, the worker has its own application code. This results in the worker process having a larger memory footprint than the leader.

smem -a - sort=pid -k

With Preload

Preload is a memory optimization based on the concept of copy-on-write. Essentially, the workers share the imports and application code with the leader and only modified pages are written to the worker’s memory.

smem -a - sort=pid -k

So how does preload play a role with USR2 signal killing the process?

If you remember, we registered the signal during the app initialization by calling register_handlers().

# app/__init__.py

MemoryProfiler().register_handlers()  


# mem_profiler.py

class MemoryProfiler:

    def register_handlers(self) -> None:
        # Register gunicorn worker to listen to USR2 to dump traces
        signal.signal(signal.SIGUSR2, self.handle_signal)

Since the app had preload=True, only the leader process was registering the USR2 signal to handle the tracing. The worker process did not register due to copy-on-write and that causes any kill -USR2 to actually kill the process!

Let’s start the tracing again (with no preload)!

Now that we have figured out that preload caused the process to be killed, we turn off the preload option and start the tracing again.

ps aux 

kill -USR2 12

The worker does not get killed!

We created a script which iterates through all the K8s pods and sends a USR2 signal to all the workers to start the tracing and resends the signal to stop the tracing after a certain time interval. The trace had a lot of false positives since it collects dumps which may not necessarily be the source of the leak, but have not been garbage collected yet.

Root causing

The most interesting (and common) memory dump trace after sifting through hundreds of them was the following:

If you remember the initial conclusion we had with the following graph, we knew that the increase in timeouts had something to do with pynamodb and gevent/greenlets since we saw thread joins taking a long time:

The stack trace combined with the graph above, narrowed down the issue to pynamo/botocore. After digging online, we found the following issue with urllib3 v1.26.16. Essentially, in a highly concurrent environment using gevent, connections were not being returned to the pool which caused the pool to sit at its max size and block further requests. This particular stack trace confirmed our suspicion:

The root cause of the issue was some incompatibility between weakref.finalize and gevent’s monkey patching causing non-deterministic deadlock which made the issue hard to reproduce. The immediate fix was to downgrade the urllib3 version to 1.26.15, after which the timeouts and the memory leak were gone!! The actual fix which ensures urllib3 connection pooling is cooperative was released in April 2025 and we have seen no issues upgrading both gevent to v25.4.1 as well as urllib3 to 1.26.16+.

It is unclear though why the Python version upgrade exposed the issue. In fact, urllib3 upgrade was not part of the dependency upgrade we had done to prepare for the Python 3.10 upgrade! We had actually been running Python 3.8 with urllib v1.26.16 for about a year without any problem. Ironically, we had upgraded to v1.26.16 specifically because it logged the total connections whenever connection pools were full.

Tip

1. If you run into memory leaks which are affecting the live production system, you can use gunicorn’s max-request settings which recycles the worker processes after N requests. This ensures your process or container does not run into OOM. While this helps mitigate the issue, it is critical to continue investigating the source of the memory leak.
2. Gevent monitoring thread has an option to print trace for greenlets which exceed a certain memory threshold. While I have personally never tried this, it could help find objects which are holding large amounts of memory, but not necessarily the source of a leak.

Closing Notes

There is no silver bullet to debugging memory leaks; it is a hard issue to debug them. There a few things you can look for eg. unbounded global caches, unreleased resources tied to database/network pooling, recently upgraded libraries, etc. If you check the actual gevent/urllib3 issue, none of them talked about memory leaks, only timeouts. We just happened to run into a memory leak and try to find the root cause of it 😀

Lyft is hiring! If you’re passionate about efficient database connection management, visit Lyft Careers to see our openings.

From Python3.8 to Python3.10: Our Journey Through a Memory Leak was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Yaroslav Yatsiuk

At Lyft, machine learning (ML) is the engine behind our most critical business functions — from dispatch and pricing optimization to fraud detection and support automation. Our ML infrastructure serves thousands of production models making hundreds of millions of real-time predictions per day, supported by thousands of daily training jobs that keep ML models fresh and accurate.

As our scale grew, we faced a classic engineering challenge: the very complexity that powered our platform was becoming a bottleneck to its future growth. We needed to answer a fundamental question: How could we evolve our platform to accelerate innovation for our users while simplifying its underlying architecture?

This post explores how we rethought LyftLearn’s architecture to solve this problem. We’ll walk through our transition from a fully Kubernetes-based system to a hybrid platform, combining the simplicity of managed compute on AWS SageMaker for offline workloads with the flexibility of Kubernetes for online model serving. Afterwards, we’ll share the key technical decisions and trade-offs that made this evolution possible.

LyftLearn Overview

LyftLearn is Lyft’s end-to-end machine learning platform, managing the complete ML lifecycle from model development to production serving. Built to support hundreds of data scientists and ML engineers, it handles the full spectrum of ML workloads at scale. The platform is composed of three integrated products:

LyftLearn Compute (Offline Stack) handles model development and training workloads. ML Modelers use JupyterLab environments to prototype models, then run training jobs, batch processing, and hyperparameter optimization at scale. These workloads are elastic and on-demand — they spin up when needed, process large datasets, and terminate when complete.

LyftLearn Serving (Online Stack) powers production inference, serving millions of predictions per minute with millisecond latency. It provides online model serving with real-time ML capabilities, automated deployment and promotion workflows, and online validation to ensure model quality before production traffic.

LyftLearn Observability monitors model health and detects degradation across the platform. It tracks performance drift, identifies anomalies, scores model health, and monitors model activity to ensure production models maintain quality as data and business conditions evolve.

While all three components work together to provide a unified ML platform, the offline and online stacks have fundamentally different operational characteristics. Offline workloads need elastic, cost-efficient compute that scales to zero between jobs. Online model serving requires always-on infrastructure with strict latency guarantees and tight operational control. These differences led us to adopt different infrastructure strategies for each — and it’s the evolution of our offline stack that transformed how we deliver LyftLearn Compute today.

The Original Architecture

The original offline stack (LyftLearn Compute) ran entirely on Kubernetes — every training job, batch prediction, hyperparameter optimization run, and JupyterLab notebook environment executed as a Kubernetes workload, orchestrated through a collection of custom-built services. We documented this architecture in detail in our 2021 blog post, LyftLearn: ML Model Training Infrastructure built on Kubernetes.

The following diagram shows a high-level view of the LyftLearn Compute 1.0 architecture:

To understand the operational complexity, let’s look at some of the key components and how they worked together:

LyftLearn Service served as the backend API, receiving requests from three primary sources: the LyftLearn UI for ad-hoc jobs, Airflow DAGs for scheduled training and batch prediction pipelines, and CI/CD pipelines that registered models along with their Docker images during deployments. It managed model configurations, job metadata, and coordinated with downstream services.

K8s Orchestration Service translated job requests into Kubernetes resources. When LyftLearn Service called it to create a training job, it would:

• insert the job record in LyftLearn DB (so watchers could track it)
• construct the Kubernetes Job specification with containers, resource requests, environment variables, sidecars, references to docker images in AWS Elastic Container Registry (ECR), and other K8s resources
• submit the job to the Kubernetes cluster

Background Watchers ran continuously to manage jobs lifecycle and infrastructure. We maintained multiple worker scripts handling different responsibilities:

• job status watcher (monitoring job state transitions and timing)
• container status watcher (tracking individual container states)
• ingress status watcher (managing notebook endpoint URLs)
• job cleanup watcher (removing completed jobs from Kubernetes)
• analytics event watcher (capturing usage events)
• additional scripts for EFS cleanup, spending tracking, and stats publishing

Creating any job meant assembling a complete set of Kubernetes resources: Pod specifications with init and sidecar containers for secrets and metrics, ConfigMaps for hyperparameters, Secrets for credentials, PersistentVolumeClaims for notebook storage, Services and Ingresses for network access, and role-based access control (RBAC) policies (ServiceAccounts, Roles, RoleBindings) for cluster permissions. In essence, we owned the entire operational lifecycle — from scheduling and retries to cleanup and low-level resource management.

What Worked Well

The Kubernetes-based architecture successfully powered production ML workloads for years and delivered some real technical advantages, including:

Unified Infrastructure Stack
ML workloads ran on the same Kubernetes infrastructure as Lyft’s production services, using the same networking stack, observability tooling, security patterns, and operational processes. This meant the platform team leveraged existing infrastructure expertise and tooling rather than maintaining separate systems for ML workloads.

Fast Job Startup
Jobs could launch as fast as 30–45 seconds on existing K8s cluster infrastructure. Unlike on-demand compute provisioning which requires waiting for instances to start and initialize, jobs scheduled immediately onto available nodes with cached images, making the approach particularly effective for frequently running training jobs and batch processing workflows.

Flexible Resource Specifications
Engineers could request any CPU/memory combination their workload needed. Memory-intensive preprocessing jobs could request 16 CPUs with 512GB RAM, while CPU-intensive training jobs used 64 CPUs with 128GB RAM. These ratios didn’t map cleanly to fixed AWS instance types, so this flexibility allowed precise resource allocation based on workload needs.

This architecture served hundreds of engineers running thousands of daily jobs that powered business-critical ML workflows. However, as Lyft’s scale grew, so did the operational complexity of managing such a system.

Challenges of a Growing Platform

We identified several key challenges that were consuming an increasing amount of our focus:

The Feature Tax
Every new capability we added to the platform, from distributed hyperparameter optimization using Katib/Vizier to distributed training with Kubeflow operators, required building, deploying, and maintaining a corresponding set of custom Kubernetes orchestration logic. While this approach gave us maximum control, it also meant that a significant portion of our development cycle was dedicated to building and managing the infrastructure for each new feature, rather than the feature itself.

Managing State in a Distributed System
To keep our platform’s database synchronized with the cluster state, we relied on background watcher scripts that continuously monitored Kubernetes events for job status changes, container updates, and ingress resource availability.

The eventually-consistent nature of Kubernetes created operational complexity. Training containers could succeed while Kubernetes marked jobs as failed due to sidecar issues. Event streams would timeout or arrive out of order. Container statuses could transition between states as different watchers processed conflicting events. We developed sophisticated synchronization checks and logic to handle these cases, but managing state consistency for thousands of daily jobs required considerable on-call attention and directly impacted our development velocity.

Kubernetes Cluster Management
A persistent challenge in managing a large-scale ML compute platform is optimizing resource utilization for heterogeneous workloads. ML jobs often have distinct phases with conflicting resource profiles: data processing tends to be memory-intensive, while model training is often CPU- or GPU-intensive. This created a complex optimization puzzle, making it challenging to maximize node utilization across the cluster.

As the platform grew, we also had to proactively manage resource contention during bursts of highly parallel workloads. Ensuring that the cluster autoscaler could provision capacity quickly enough to prevent job queuing for critical workflows required careful planning and continuous management.

The pattern was clear: as the platform scaled, so did the operational investment required to manage its low-level infrastructure. To continue innovating for our users, we needed to abstract away this underlying complexity and refocus our efforts on what mattered most: building new platform capabilities, optimizing ML workflows, and accelerating the entire ML development lifecycle.

The Journey to LyftLearn 2.0

The growing operational complexity of our Kubernetes stack was limiting our development velocity. This reality pushed us to explore how we could simplify operations while delivering more powerful capabilities to our users. We began evaluating managed solutions to abstract this infrastructure complexity, which led us to a deep evaluation of AWS SageMaker.

We evaluated SageMaker across both our online (LyftLearn Serving) and offline (LyftLearn Compute) stacks.

For LyftLearn Serving, adopting SageMaker would have required a fundamental re-architecture of our core workflows. Our model deployment, promotion, and serving solutions were deeply integrated with Lyft’s internal tooling. Observability relied on our standard monitoring infrastructure, not on AWS CloudWatch. Client services communicated via Envoy, not via SageMaker’s specific invocation and authentication patterns.

Our analysis confirmed that the existing Kubernetes-based stack was exceptionally reliable and efficient, performing well within our required latency requirements. We determined the right path forward was to retain our existing, battle-tested model serving infrastructure.

For LyftLearn Compute, the evaluation pointed in a different direction. This was where our greatest operational complexity lived: managing eventually-consistent job states, optimizing cluster capacity for heterogeneous workloads, and building custom Kubernetes orchestration for new ML capabilities.

SageMaker’s managed infrastructure would address these challenges directly. It offered out-of-the-box support for a variety of job types, which would allow us to stop building and maintaining low-level orchestration logic. Its native state management would eliminate the need for our custom watcher system, and its elastic compute model would handle capacity automatically, removing the need for complex cluster planning and autoscaling management.

While SageMaker’s per-instance costs were higher, the Total Cost of Ownership (TCO) was clearly lower. By eliminating idle compute, cluster administration overhead, and the constant infrastructure firefighting, the economics of a managed service made sense.

The evaluation led to a clear strategy: adopt SageMaker for LyftLearn Compute, where we had the greatest opportunity to reduce operational complexity, and retain Kubernetes for LyftLearn Serving, where our existing solution was already highly reliable and efficient.

The diagram below provides a high-level, conceptual view of how we wanted to transform the offline stack:

On the left, the original architecture: LyftLearn Service sent job requests to a K8s Orchestration Service, which constructs Kubernetes Job specifications and submits them to the Kubernetes API. This orchestration service was complex — it managed pod configurations, resource allocations, volumes, and all the low-level details of Kubernetes jobs. Background watchers continuously polled the Kubernetes API for events — job completions, container status changes, resource updates — and wrote those updates back to the LyftLearn database. The compute layer ran on Lyft-managed Kubernetes clusters.

On the right, the new architecture: Under the hood, this is a significantly simpler solution. LyftLearn Service interacts with a lean SageMaker Manager Service that only makes AWS SDK calls — it doesn’t manage any low-level infrastructure. We replaced the fleet of problematic background watchers with a single, reliable SQS consumer that processes status updates pushed from EventBridge. The heavy lifting of orchestration and state management is delegated to AWS. The goal was simplification without losing power.

It looks simple on a diagram, but making this transition without disrupting critical ML workflows and hundreds of users was a significant engineering challenge. The following sections detail some of the most difficult technical challenges we solved to make this migration possible.

Migration: Solving the Hard Problems

Our core principle for the migration was to replace the execution engine — Kubernetes to SageMaker — while keeping our ML workflows completely unchanged. The actual ML code — the Python scripts that train models, process data, and run inference — had to work identically on both platforms. No modifications to model training logic, no changes to data preprocessing, no updates to inference code.

Forcing hundreds of users across dozens of teams to rewrite their business-critical ML workflows was not an option. The cost of such a disruption in terms of lost productivity and engineering effort would have made the migration untenable, which meant the burden of compatibility was entirely on our platform. The requirement of zero code changes transformed the project into a complex systems engineering challenge for the ML Platform team, but it was a necessary one. The real task wasn’t just running a container on a different platform — it was ensuring environmental parity.

During the transition, we solved numerous challenges across the stack. Here are a few of the most complex ones we solved to make this possible.

Replicating the Kubernetes Runtime Environment
Our Kubernetes environment provided automatic credential injection via webhooks, metrics collection through sidecars, and configuration management via ConfigMaps. SageMaker offered none of these primitives. We built a compatibility layer into cross-platform base Docker images to replicate this behavior:

• Credentials: In Kubernetes, credentials from our internal secret management solution, Confidant , were automatically injected at pod creation. SageMaker has no equivalent mechanism. We built a custom solution, as part of the container entrypoint script, that fetches credentials at job startup and exposes them exactly as Kubernetes did, ensuring user code worked identically on both platforms
• Environment Variables: SageMaker constrains the number of environment variables passed via its API. Similar to our credential solution, we moved most environment setup to runtime, fetching additional configuration at job startup.
• Metrics: Kubernetes workloads sent StatsD metrics to sidecar containers. SageMaker has no sidecar support, so we reconfigured the runtime and networking to connect directly to our metrics aggregation gateway. The user-facing API remained unchanged.
• Hyperparameters: In Kubernetes, hyperparameters were stored in ConfigMaps and mounted as files. SageMaker’s API has much stricter size limits than K8s, making direct parameter passing impossible for our use cases. We developed a solution to upload hyperparameters to AWS S3 before each job and have SageMaker automatically download them to its standard input path. This overcame the API limitation while still using SageMaker’s native capabilities.

These represent only a subset of the environmental differences we systematically solved across the migration.

Building for the Hybrid Architecture
We developed new SageMaker-compatible base images to replace our old LyftLearn images. The critical design requirement was that these images must work across our entire hybrid platform: in SageMaker (for training and batch processing) and in Kubernetes (for serving). This meant the same Docker image that trained a model would also serve it, guaranteeing consistency. These base images serve as a foundation that teams extend with their own dependencies.

We built SageMaker-compatible base images with different capabilities to match our workload diversity. Here are some of the most important ones:

• LyftLearn image: For traditional ML workloads
• LyftLearn Distributed image: Adds Spark ecosystem integration for distributed processing
• LyftLearn DL image: Adds GPU support and libraries for deep learning workloads

The Spark-compatible images presented the biggest challenge. They needed to maintain full compatibility with our existing Spark infrastructure — custom wrappers, executor configurations, and JAR (Java Archive) dependencies. But they also had to run correctly in three distinct execution contexts: SageMaker Jobs, SageMaker Studio notebooks, and Model serving in K8s.

These images detect their execution environment at runtime and adapt. They automatically configure different environment variables, use different users and permissions, and set up Spark appropriately for each context, all while preserving an identical core runtime.

Matching Kubernetes Job Launch Times
In Kubernetes, notebooks, training, and processing jobs could start quickly because nodes were warm due to a significant percentage of cluster resources sitting idle. SageMaker provisions instances on-demand — no idle waste, but slower startup.

For JupyterLab notebooks, we adopted SOCI (Seekable Open Container Initiative) indexes. SOCI enables lazy loading: SageMaker fetches only the filesystem layers needed immediately rather than pulling entire multi-gigabyte images. This cut notebook startup times by 40–50%.

For training and batch processing jobs, SOCI wasn’t available. We optimized our Docker image sizes, which were sufficient for most of our workloads. However, this wasn’t enough for our most latency-sensitive workflows. Some models retrain every 15 minutes, making slower startup times unacceptable. For this subset of jobs, we adopted SageMaker’s warm pools, which keep instances alive between runs.

These optimizations gave us Kubernetes-like startup times with fully serverless infrastructure.

Cross-Cluster Networking for Spark
Many of our ML Platform users rely heavily on the interactive Spark experience in JupyterLab notebooks. In Kubernetes, this was simple, as the driver and executors ran in the same cluster. The new architecture, however, required the Spark driver to run in a SageMaker Studio notebook while the executors remained on our EKS K8s cluster.

This hybrid model presented a major networking challenge, as shown in the diagram below. Spark client mode requires bidirectional communication:

• The driver (in SageMaker) must call the EKS API Server Endpoint to request executor pods.
• The executor pods must be able to establish inbound connections directly back to the driver’s SageMaker Instance Elastic Network Interface (ENI).

The default SageMaker Studio networking blocked these critical inbound connections, breaking Spark’s communication model. This issue was a fundamental blocker that could jeopardize the entire migration. Without a solution for interactive Spark, we could not move our users to SageMaker Studio. To resolve this, we partnered closely with the AWS team. As a result of this collaboration, they introduced networking changes to the Studio Domains in our account that enabled the required inbound traffic from our EKS cluster. Despite the cross-cluster setup, Spark performance remained the same, and the interactive experience for ML Platform users was identical to the original Kubernetes environment.

LyftLearn 2.0: The Hybrid Architecture

As a result of this architectural transformation, we arrived at the hybrid architecture we planned: SageMaker for LyftLearn Compute and Kubernetes for LyftLearn Serving.

As the diagram illustrates, the two systems are fully decoupled, each operating as a purpose-built stack:

LyftLearn Serving runs on Kubernetes, powering a distributed architecture for real-time inference. Dozens of ML teams deploy their own model serving services — each containing their team’s models with custom prediction handlers and configurations — handling production predictions for specific use cases (pricing, fraud, dispatch, ETA, etc.). The Model Registry Service coordinates model deployments across these services. (We detailed this serving architecture in our 2023 blog post: Powering Millions of Real-Time Decisions with LyftLearn Serving.)

LyftLearn Compute runs on SageMaker, where the SageMaker Manager Service orchestrates training, batch processing, Hyperparameter Optimization (HPO), and JupyterLab notebooks through AWS SDK calls. EventBridge and SQS provide event-driven state management, replacing our background watchers.

Integration happens through the Model Registry and S3. Training jobs in SageMaker generate model binaries and save them to S3. The Model Registry tracks these artifacts, and model serving services pull them for deployment. Docker images flow from CI/CD through ECR to both platforms. The LyftLearn database maintains job metadata and model configurations across both stacks.

Each LyftLearn product operates independently while maintaining seamless end-to-end ML workflows.

Putting It All Together

We rolled out changes repository by repository, running both infrastructures in parallel. Our approach was systematic: build a comprehensive compatibility layer that made SageMaker feel like Kubernetes to ML code, validate each workflow type thoroughly, then migrate teams incrementally. Each repository required minimal changes — typically updating configuration files and workflow APIs — while the actual ML code remained untouched.

For our users, the migration was nearly invisible. But behind the scenes, the operational improvements were substantial. We reduced ML training and batch processing compute costs by eliminating idle cluster resources and moving to on-demand provisioning. System reliability improved significantly, with infrastructure-related incidents becoming rare occurrences. Most importantly, this stability and the serverless nature of the new compute freed our team to focus on building platform capabilities rather than managing low-level infrastructure components.

Key Lessons

Build versus buy is a pragmatic decision, not an ideology
We adopted SageMaker for training because managing custom batch compute infrastructure was consuming engineering capacity better spent on ML platform capabilities. We kept our serving infrastructure custom-built because it delivered the cost efficiency and control we needed. The decision wasn’t about preferring managed services or custom infrastructure — it was about choosing the right tool for each specific workload.

Abstract complexity from users.
The migration succeeded because we absorbed all the complexity. Users didn’t rewrite ML code or learn SageMaker APIs — they continued their work while we handled secrets management, networking, metrics collection, and environmental parity. The platform’s job is to evolve infrastructure while preserving velocity and avoiding disruptions, not to distribute migration work across hundreds of teams.

Invest in compatibility layers
The cross-platform base images were the foundation of the migration’s success. They enabled gradual, repository-by-repository migration with easy rollbacks. Most importantly, they guaranteed that the same Docker image for model training in SageMaker would serve it in Kubernetes, eliminating train-serve inconsistencies. The upfront investment in cross-platform compatibility paid dividends throughout the migration.

The best platform engineering isn’t about the technology stack you run — it’s about the complexity you hide and the velocity you unlock.

Acknowledgment

This platform evolution was a massive team effort. Special thanks to Vlad Yermakov, Herman Khivrenko, Nima Nasiri and Andy Rosales-Elias for their exceptional work making it a success.

We’re also grateful to Raj Bagwe and Vikram Sawant, our partners from AWS, for their invaluable support on this initiative.

Lyft is hiring! If you’re passionate about building AI/ML platforms and applications at scale, visit Lyft Careers to see our openings.

LyftLearn Evolution: Rethinking ML Platform Architecture was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

I joined Lyft in January of 2024, as a Data Scientist — Decisions, on the Rider Science Core Experience team. My journey at Lyft began with a starter project, which focussed on using the Rider Experience Score (RES) tool to measure long-term effects of various rider experiences at Lyft.

In this blog post, I will discuss my experience at Lyft as a new hire, focusing on this starter project.

What is RES?

Motivation

At Lyft, we aim to deliver seamless and reliable experiences for our riders. To continuously improve the platform, it’s important to understand which rider experiences most impact our riders, and how those experiences influence their decision to continue using Lyft over time (rider retention).

For example, we can imagine a hypothetical scenario where a rider experiences a lower than normal ETA (the estimated request to pickup time). Experiencing lower ETA can potentially be an improved experience for riders, motivating us to make a product change which drives a decrease in ETA. To justify the introduction of this product change, we first want to quantify how low ETA impacts long-term rider retention. Using an A/B test to evaluate the long-term rider retention impact of low ETA may be problematic, since an A/B test typically runs for 2–6 weeks, which may not be a long enough time period to accurately measure long-term effects. In other cases, A/B tests may not be possible, such as when introducing a new feature which is rolled out to all users.

RES is a tool for estimating how various user experiences (low ETA, early driver arrival, etc.) impact long-term rides taken, which is the Δ in Figure 1, without requiring an A/B test.

Challenges in Estimating the Effects of User Experiences

In order to explain RES, I will first discuss the challenges in estimating how user experiences impact rider retention.

The essence of this problem is estimating the causal effect of exposing riders to a particular experience on rider retention. For simplicity, we define the rider retention effect as the impact on the number of rides taken in the future 28 days. The true causal effect of a rider experiencing low ETA would be obtained by observing the future 28 day rides for a rider session in which low ETA is encountered, and comparing that to the future 28 day rides for the exact same rider session, except low ETA is not encountered (counterfactual). But, it’s impossible to observe counterfactuals; in reality, for a given session, we only observe the session where the rider experiences low ETA or not, but not both (the Fundamental Problem of Causal Inference).

A simple potential solution to overcome the challenge of not being able to observe the counterfactual outcome might be to look at historical observational data for riders that experienced low ETA, and compare their average ride retention to riders who didn’t experience low ETA, which is the average treatment effect (ATE) estimated by difference-in-means estimator.

However, this naive approach can lead us astray. Let’s see what happens when we apply this method. Figure 2 shows hypothetical results (not real data) that illustrate the problem we might encounter.

Based on the observations in Figure 2, we would conclude that low ETA actually has a negative effect on rider retention, which must be incorrect. To understand how we arrived at this incorrect conclusion, we segment our data into regions City A and City B, and also indicate on the x-axis whether or not the rider experienced low ETA (left of dashed line didn’t see low ETA, right did); results are shown in Figure 3.

We now see that within City B, and within City A, the effect of low ETA is showing a positive trend, as expected. We also see that City B is much less likely to be in the “treatment” group (right of dash, low ETA), and that City B has higher baseline rides in the next 28 days. We thus see that our groups for low and not low ETA are biased (selection bias), which results in an incorrect estimate of the causal effect of low ETA. The difference-in-means estimator works well for randomized experiments, but is biased for non-randomized cases; in the above example, region is correlated with both the treatment ‘low ETA’, as well as the outcome # of rides in the next 28 days (region is a confounder). This is an example of an issue that can arise when estimating causal effects from observational data.

The gold standard approach for estimating causal effects is randomized experimentation (A/B test). With randomization, we would end up with a roughly equal split of City A and City B across control and treatment groups, thus mitigating the bias discussed above. But, a limitation of an A/B test is that we typically can only run them for a fixed short period of time, which won’t allow measuring long-term effects on rider retention. We thus can benefit from methodology that can mitigate the bias in causal effects estimated from observational data.

RES Methodology

The RES tool employs causal inference methodology to mitigate the bias in causal effects measurements obtained with observational data.

There are various methods to control for confounding variables. Propensity score methods model the relationship between confounders X and treatments W (i.e. with an ML model), outcome methods model the relationship between confounders X and the outcome Y, and double ML methods model both the relationship between X and Y and W and X.

RES employs Augmented Inverse Propensity Score Weighting estimator (AIPW; more info on AIPW can be found here), which is an example of a double ML Method.

At a high level, AIPW consists of estimating potential outcome functions of treatment and control experiences (Direct Method), as well as adjusting estimation bias with propensity weighted residuals. AIPW has the desirable theoretical properties of Neyman Orthogonality (robust to ML estimation errors), and doubly robustness. We also apply a cross-fitting procedure (data split) to get Double-ML properties (unbiased estimation).

More precisely, we compute the treatment effect as follows:

In order to compute the treatment effect, we train three XGBoost or LightGBM models

For every observation i, using the above models (and cross-fitting), we predict the outcome μ₁*(xᵢ), μ₀*(xᵢ), and e*(xᵢ). Finally, we plug μ₁*(xᵢ), μ₀*(xᵢ), and e*(xᵢ) into the equation for AIPW to get an estimate for the ATE for low ETA. Though the relationships between outcomes, treatment, and confounders in the above low ETA example are relatively simple, using machine learning models for e*(xᵢ) and μ*(xᵢ) allows us to model complex non-linear relationships which may arise in other settings.

In the example above, AIPW helps us avoid making a wrong conclusion in two ways:

1. By modelling the relationship between region and 28-day rides (μ*(xᵢ)), we are able to isolate the effect of region on our outcome Y, and capture that City B riders are more likely to have more rides in 28 days than City A.
2. By modelling the relationship between region and low vs. normal ETA (e*(xᵢ)), we capture that City B riders are less likely to experience low ETA than City A riders. AIPW reweights observations inversely to their propensity scores, effectively upweighting points in the bottom-left and top-right of Figure 3. This reweighting reveals the underlying upward slope, allowing us to conclude that low ETA has a positive impact on 28-day rides.

The AIPW estimator combines both 1. and 2., which gives favorable statistical properties, such as treatment effect estimation being insensitive to errors in models e*(xᵢ) and μ*(xᵢ).

What I did as a new hire

The existing RES estimates needed an update, and there was also a need to add additional rider experiences to the RES pipeline. RES generally needed a refresh, and I was tasked with doing so.

I started by gaining more familiarity with AIPW. Reading through the internal causal inference lecture series at Lyft was extremely helpful. I also found Stefan Wager’s STATS 361 course notes very useful for learning about AIPW.

Next, I spent time learning about how to use LyftLearn, Lyft’s internal computing platform for Big Data and Machine Learning. I then became familiar with how to use the RES codebase. I analyzed the RES codebase to see if there were any opportunities to improve reliability and efficiency of the RES code. I identified some aspects of the RES code which could be improved, and then submitted a pull request with the corresponding changes. For example, there was a subtle issue which prevented model diagnostics from completing in certain cases, which I was able to fix.

With the unblocking of the RES pipeline, I sought to compute estimates of long-term effects of various Lyft user experiences. I began with identifying the most important experiences to add to the RES pipeline.

I reached out to other teams to see which experiences would be most important to their work. Examples of experiences that were identified are “Improved Match Time Prediction”, and “Improved ETA Reliability”.

After gathering a list of experiences, I prioritized and selected 23 based on discussions with my manager about their potential impact. This process provided great insight into the experiences that matter most to our riders, and where we could have the greatest positive impact on retention. I also refreshed estimates for the pre-existing reliability experiences (with newly added High Value Mode sub group analysis).

A major challenge I faced in computing estimates for my chosen experiences was selecting confounders. Internal RES documentation provides excellent guidance on selecting confounders, but significant trial and error was still required. For example, I computed RES estimates for Prime Time experience, where Prime Time refers to a multiplier on ride price during high demand periods, and I had included Neighborhood Supply as a confounder. The ROC AUC of our trained model was suspiciously high, which we realized was partly due to Neighborhood Supply being a leaky confounder for Prime Time. A leaky confounder is a covariate which contains information that is concurrent or subsequent to the experience; this is an issue, as it can cause some of the treatment effect to be attributed to the confounder, leading to biased estimates.

For each of the 23 experiences I worked on, I had to make sure to include key confounders, and also avoid including problematic confounders, which was very time consuming. But, this process provided a great opportunity to learn about various data sources that exist at Lyft, and to reflect on how key covariates may be related causally to Lyft rider experiences.

Conclusion

Though I faced many challenges in my starter project, I had excellent support from my colleagues at Lyft, and was able to successfully generate causal estimates for the 23 user experiences identified.

Navigating the various challenges I encountered was a very intellectually stimulating and fulfilling experience. The insights from RES play a crucial role in helping teams focus on the experiences that matter most to our riders; contributing to a workstream with such a high impact on improving rider experience has been very rewarding.

Getting started at Lyft has been an enriching journey. I’ve had the opportunity to apply real-world causal inference techniques and collaborate with amazing colleagues. I’m excited to continue contributing to impactful projects, and to see what the future holds.

Lyft is hiring! If you’re passionate about Data Science, visit Lyft Careers to see our openings.

My Starter Project on the Lyft Rider Data Science Team was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Lyft started adopting Kotlin into our Android codebase in 2018. Fast forward 7 years, and we are finally done! Lyft Rider, Driver and Lyft Urban Solutions apps are now fully Kotlin-based.

I joined Lyft in 2022, so this post will describe the efforts undertaken after that.

Our motivation included several points:

• Kotlin code is more concise, and oftentimes, way more concise than Java. In some cases, 10 lines of Java could be turned into a 1-liner in Kotlin.
• We get compile speed benefits by using the newest K2 compiler.
• All the new UIs in Lyft are written using Compose — the modern declarative UI framework approach which is the industry standard. All the existing UIs will eventually be migrated to Compose. That also means Kotlin was the only option, as Compose only supports Kotlin.
• We started adopting Coroutines — the structured concurrency framework which greatly simplifies writing asynchronous code. Coroutines are part of the Kotlin standard library, so that was an extra argument to adopt Kotlin faster.
• Our codebase is huge, so we often run automated migrations, which require adopting migration scripts for Java as well.
• Working entirely in Kotlin is a big plus for engineers considering Lyft.

Pre-migration

The first thing which needs to be done when undertaking a project this large is to know where we are standing. To achieve this, Lyft has an internal tool called Migration Tracker, which tracks all of the migrations in both Android and iOS codebases. Examples of migrations are:

• Migrating from RxJava to Coroutines
• Switching from our old UI approach to the new declarative one
• Eliminating uses of Java in favor of Kotlin

A daily cron job runs the Migration Tracker and updates an internal website, presenting graphs which help us ensure we meet the migration deadlines.

As of Feb 24, 2025, the Kotlin migration was 85% ready. That means we still needed to migrate about 1,000 files scattered across 20+ teams and 150+ Bazel modules.

Migration

Fellow developer Oleksii Zaiats built a tool which greatly sped up the migration process: the Migration Script. It leveraged Android Studio IDE Scripting, which is a powerful but rarely used tool, fitting perfectly for this kind of task.

Put simply, the script flow was as follows:

• For a given team, it found all modules owned by the team.
• For the given module, it ran the automatic migration mechanism for each Java file.
• It automatically fixed some common imperfections of Android Studio’s built-in Java to Kotlin converter.
• After all the files in the module were migrated to Kotlin, it created a git commit with all the changes, naming the git branch appropriately.
• The owning team was notified of the changes and reviews were requested.

This simple approach was not without its flaws, but it gave us a real productivity boost and allowed us to migrate a couple of modules per day.

Challenges and Caveats

The major pain point in the migration process was that the automatic migration tool was not perfect:

• In many cases, it uses nullable-types where non-nullable types are fine, resulting in code like Observable<Optional<List<Ride?>?>?>?
• It is unnecessarily verbose, using explicit types everywhere.
• It is not smart enough to convert an explicit for loop into a Kotlin one-liner using map or filter.
• It cannot automatically use lateinit var which is often needed when writing View-based UIs.

In addition, the structure of the legacy code itself presented additional complications. We had some very old code implementing a hand-written INullable interface, which was similar to Optional but not quite. The semantics of INullable required us to do extensive code review each time we touched one of these classes.

Last but not least, once we encountered a class which absolutely had to be written in Java! It was implementing an interface with a signature like so:

public void onTouchEvent(@NonNull Float x, @NonNull Float y)

However, on some devices, the API contract was broken, so x and y could in fact be null! In Java this was totally fine, but in Kotlin this resulted in a crash. Thankfully, we have rewritten the entire screen using another approach, not using this interface anymore.

Post-migration

After we were finally done, we needed to ensure that no one accidentally adds a Java file to our codebase. To achieve this, we have added a Lint check, integrated in our CI system, which checks every pull request and explicitly prohibits Java code.

Conclusion

After concluding this years-long effort, Lyft developers can skip the hassle of Java-Kotlin interop and concentrate on solving the problem that actually matters — providing our users with the world’s best transportation!

Lyft is hiring! If you’re passionate about working on a Kotlin-only app, visit Lyft Careers to see our openings.

Migrating Lyft’s Android Codebase to Kotlin was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.