Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AB Testing tips for Mobile

AB Testing tips for Mobile

AB Testing tips for Mobile. Tips with Remote Config

Tomoaki Imai

July 13, 2016
Tweet

More Decks by Tomoaki Imai

Other Decks in Programming

Transcript

  1. twitter: @tomoaki_imai github: tomoima525 ࠓҪ ஐষ Android Engineer 2016.1 Mercari

    US ग़޲ ඒຯ͍͠ΧϨʔͷΑ͏ʹ ਓΛ޾ͤʹग़དྷΔΑ͏ʹͳΓ͍ͨ
  2. Things need to take care • ਖ਼͍͠ޮՌଌఆ ‣ ҙຯͷͳ͍ςετΛ΍͍ͬͯͳ͍͔ ‣

    ฼਺ʁάϧʔϐϯάʁ༗ҙࠩ? • ͍ͭͰ΋σϑΥϧτʹ໭ͤΔ ‣ Ξοϓσʔτͳ͘ରԠͰ͖Δ ‣ ιʔείʔυΛԚ͞ͳ͍
  3. History of AB Test AB Testಋೖ ໨త: ͞ΒͳΔػೳվળ Apptimize(AB Testing

    Tool) + Segment(τϥοΩϯά) + mixpanel(෼ੳ) 2014 Ն 2015 य़ AB TestͷΠϯϋ΢εԽ ݱࡏ τϥοΩϯάͷΠϯϋ΢εԽ (Pascal) 2015 Ն ৗ࣌ 10݅લޙ ͷAB TestΛ࣮ࢪ ෼ੳͷ Big Query΁ͷҠߦ
  4. Type of AB Test • UI Test ‣ ݟͨ໨ɺϨΠΞ΢τͷมߋ ‣

    ྫ) Ձ֨දࣔํ๏ͷมߋ • Flow Test ‣ ಛఆϑϩʔͷ௥Ճ/εΩοϓ ‣ ྫ)ೖྗ߲໨ͷ௥Ճ/εΩοϓ • New feature Test ‣ ৽ػೳͷදࣔ/ඇදࣔ ‣ ྫ) ύʔιφϥΠζػೳ
  5. AB Testing tool • In House + αʔόαΠυ͔Βॊೈͳλʔήοτઃఆ + Pascal,

    Big Queryͱ૊Έ߹Θ༷ͤͯʑͳ੾Γޱ͔Β෼ੳ AB Test Client ͸ ϥΠϒϥϦԽ https://github.com/mercari/siberi-android - Apptimize like ͳ Interface - ςετσʔλΛDBͰ؅ཧ - iOS΋ۙ೔ϦϦʔε༧ఆ
  6. "QQUJNJ[F *O)PVTF 3FNPUF $POpH ಋೖ༰қ͞ ̋ ˚ ˕ Ձ֨ ☓

    ̋ ̋ ઃఆͷ ॊೈੑ ☓ ̋ ˚ ֦ுੑ ☓ ˕ ˚ • MercariͰͷಋೖ༧ఆ͸ͳ͠ ‣ ৽نࣄۀͰ͸ಋೖ༰қ͞ɺՁ֨ͷ؍఺ͰקΊ͍ͨ ‣ ·ͩ଍Γͯͳ͍෦෼͕͋ΔͷͰ࢖͏࣌͸஫ҙ(ޙड़)
  7. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ ྫ) αϯϓϧ਺ - AB TestͷԾઆʹରͯ͠ద੾ͳαϯϓϧ਺͕ઃఆͰ͖͍ͯΔ͔ -

    R ͰܭࢉՄೳ QPXFSQSPQUFTU Q Q TJHMFWFM QPXFS ex) AB ςετͷ݁Ռߪೖ཰͕ 5% -> 6%ͱͳΔͱ͍͏Ծઆ ࢀߟ: http://oku.edu.mie-u.ac.jp/~okumura/stat/150922.html sig.level -> ༗ҙਫ४ (Ծઆ͕ਖ਼͍͠ͱޡͬͯ൑அ͢Δ֬཰) power -> ݕఆྗ (Ծઆ͕ਖ਼͍͠ͱ൑அ͢Δ֬཰) O -> 8100݅Ҏ্ͷαϯϓϧ͕ඞཁ
  8. Tips 2. VariantϕʔεͰςετ͢Δ • AB Testͷ࢓ํ ‣ Keyϕʔε ‣ Buttonͷ৭,

    ςΩετ,ͦΕͧΕʹ஋Λઃఆ͢Δ ‣ Variantϕʔε ‣ ςετ಺༰ΛέʔεผͰఆٛ͢Δ 7BSJBOU &YQFSJNFOU$POFOU  /PUJO5FTU EJTDPVOU DPMPSSFE UFYUlz  $POUSPM EJTDPVOU DPMPSSFE UFYUlz  &YQFSJNFOU EJTDPVOU DPMPSSFE  UFYUl4VNNFSDBNQBJHOz
  9. • 1ͭͷςετͷઃఆ߲໨͕૿͑ΔͱKeyϕʔε͸൥ࡶʹͳΔ => ࡉ͔͍ॲཧ͸Clientଆʹ೚ͤͨํ͕҆શ • αʔόଆ͸VariantΛৼΓ෼͚Δ͚ͩʹ͢Δͱɺςετ಺༰͕؅ ཧ͠΍͍͢ Siberi.runTest(Test.HOLIDAY_PROMO, content ->

    { switch(content.getVariant()){ case 0: break; // not test group case 1: break; // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } }); Tips 2. VariantϕʔεͰςετ͢Δ 5JQT ͸ςετ֎ͱ͢Δ JOU͕EFGBVMUͷͨΊ
  10. remoteConfig.fetch(len).addOnCompleteListener(this, new OnCompleteListener<Void>() { @Override public void onComplete(@NonNull Task<Void> task)

    { switch ((int)remoteConfig.getLong(HOLIDAY_PROMO)){ case 0: // not test group case 1: // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } } }); Tips 2. VariantϕʔεͰςετ͢Δ • RemoteConfig SFNPUF$POpHHFU-POH͸஋͕ औΕͳ͍৔߹͸͕ฦΔ
  11. • ςετ࣮ࢪޙͷ͋Δ͋Δ ‣ ͋ΔςετͰ͔͠࢖Θͳ͍ϑΟʔϧυม਺/ϝιουΛফ͠ ๨ΕΔ ‣ ͲͷςετͷͨΊͷϑΟʔϧυม਺/ϝιου͔൑அͣ͠Β͍ • @ForExperiment Annotation

    ‣ ςετίʔυʹ෇Ճ͢ΔAnnotation ‣ @ForExperiment(Test໊) ͱ͢Δ͜ͱͰՄಡੑΛ্͛Δ Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ
  12. Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ @ForExperiment(Experiments.TEST_002) TextView testTv; @ForExperiment(Experiments.TEST_002) private void changeText(String

    text){ testTv.setText(text); } public interface Experiments { String TEST_001 = "test_001_change_button_color"; String TEST_002 = "test_002_change_text"; } • @ForExperimentͰͲͷTest༻ͷม਺/ϝιουͳͷ͔໌ࣔతʹͳΔ • Testऴྃޙ͸ Experiments.java಺ͷStringΛফͤ͹ফ͠࿙Ε͕ແ͘ͳΔ @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.FIELD, ElementType.TYPE, ElementType.LOCAL_VARIABLE, ElementType.METHOD})
 public @interface ForExperiment {
 String value();
 }
  13. Wrap up • ࠓ͙͢ʹAB TestΛ࢝ΊΔͳΒ Firebase RemoteConfig ͸Φεεϝ(͚Ͳར༻ʹ͸·ͩ·ͩ஫ҙ) • ͪΐͬͱͨ͠഑ྀͰҙຯͷ͋ΔAB

    Test ͕ޮ཰ྑ࣮͘ ࢪͰ͖Δ • ద੾ͳλʔήοτઃఆ • Variant ϕʔεͰͷςετ • ಈ࡞ݕূ͠΍͍͢࢓૊Έ • AnnotationͰςετ༻ίʔυΛ؅ཧ