Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Flows, Funnels, and Metrics, Oh My!

Flows, Funnels, and Metrics, Oh My!

A presentation about experimentation on the home onboarding flow at Airbnb by Theresa Johnson and Laura Kelly

Laura Kelly

April 18, 2017
Tweet

More Decks by Laura Kelly

Other Decks in Technology

Transcript

  1. Flows, Funnels and Metrics, Oh My! Home Onboarding @ Airbnb

    Theresa Johnson & Laura Kelly / April 18th, 2017 / Taking Flight
  2. Bios • Dr. Theresa Johnson is a data scientist on

    the Co-Hosting team. She completed a PhD in Aeronautics and Astronautics at Stanford University before joining Airbnb in 2016. • Laura Kelly is a software engineer on the Vacation Rentals team. She went to Washington University in Saint Louis and studied Cognitive Neuroscience. She joined Airbnb in 2015. • Theresa and Laura worked together to increase New Homes Metrics by 10%.
  3. 1. Growth Metrics 2. Creating a Home on Airbnb 3.

    Description 4. Location 5. Final Thoughts • New Homes Added • New Homes Published • Home Onboarding Funnel • New Homes Booked
  4. Curated Growth We are selective about which experiments to run,

    given our Minimum Detectable Effect. • Find problems using data • Size the impact potential • Define user problem & brainstorm solutions • Size the engineering effort • Prioritize ruthlessly • New Homes Booked • New Homes Published
  5. Experiments • Engineer builds the feature • Together determine experiment

    delivery conditions • Engineer builds experiment delivery • Data scientists code and track metrics
  6. 1. Growth Metrics 2. Creating a Home on Airbnb 3.

    Description 4. Location 5. Final Thoughts Advertise Settings Upsell Basic Information about your home Upsell with pictures, description & features Calendar, price, house rules Settings Calendar, price, house rules Advertise Basic Information about your home
  7. Logging Engineering worked closely with data science BEFORE iterating to

    log these events • Impressions • Clicks • Step completion • Prior progress
  8. Home Onboarding Funnel Jan Feb Mar Apr 30 May 10

    20 30 40 0 20 25 10 40 Only 30% of all users make it through the “Upsell” Step 18% of users drop off from the “Upsell” Step
  9. 1. Growth Metrics 2. Creating a Home on Airbnb 3.

    Description 4. Location 5. Final Thoughts The “highlights” based description hurt total reservations for new homes • Increased step completion but created “cookie cutter” descriptions Data Science challenges • Find more specific ‘tags’ that help hosts, but are appealing to guests • Model bookings vs. description length to find best word limit to enforce Engineering Challenges • Sunk cost of the description builder • Data scientists found “highlights” of landmarks near the home from guidebooks • Engineering found local highlights from google maps. • Use these to auto-create sentences to fill in the description.
  10. Home Onboarding Funnel Jan Feb Mar Apr 30 May 10

    20 30 40 0 20 25 10 40 Only 30% of all users make it through the “Upsell” Step 25% of users drop off from the “Advertising” Step
  11. 1. Growth Metrics 2. Creating a Home on Airbnb 3.

    Description 4. Location 5. Final Thoughts Location step accounted for nearly all of the drop-off in the ‘Advertise’ section. Data Science challenges • Deep dive to determine % users suffering from google map errors • Model conditional expected conversion on the new step placement E(Y|X=x) Engineering Challenges • Large amount of state related to user's step in the flow and what has been completed -- we had to design the flow so that steps could be moved or added • Listing creation back-end event tied to address step • Location step revealed personal information • Map dragging introduced complexity
  12. Location step experiment +2.4% New Homes Added Location Property type

    Bedrooms Bathrooms Amenities Property type Bedrooms Bathrooms Location step moves to later in the flow Location Amenities After Before
  13. 1. Growth Metrics 2. Creating a Home on Airbnb 3.

    Description 4. Location 5. Final Thoughts • Best practices for successful data science and engineering collaboration ◦ Communicate early and often in the project, way before you're ready to iterate ▪ Define expectations and success ◦ Trust each other and leverage complementary skills ▪ Be strategic ◦ Brainstorm together ▪ Use asynchronous tools to avoid meeting overload • Next Steps ◦ Mobile App New Home Onboarding