Ivory - Concepts

1. IVORY CONCEPTS http://github.com/ambiata/ivory © Ambiata 2014
2. IVORY A scalable and extensible data store for storing facts and extracting features © Ambiata 2014
3. Ivory Repository Ingest facts Extract features © Ambiata 2014
4. REPOSITORY • Storing and extracting data for a single class of entity, e.g.: • customer • account • asset © Ambiata 2014
5. DATA MODEL © Ambiata 2014
6. customer-1 balance 634 @ 2014-02-01 single “fact” Fact: Entity - Attribute - Value - Time The value of a feature (attribute) for a given entity known to be valid from a certain point in time. © Ambiata 2014
7. customer-1 balance 634 @ 2014-02-01 customer-2 customer-3 customer-4 469 @ 2014-02-01 276 @ 2014-04-01 1966 @ 2014-03-01 © Ambiata 2014 scalable
8. customer-2 customer-3 customer-4 customer-1 gender balance purchases zipcode 634 @ 2014-02-01 extensible 469 @ 2014-02-01 276 @ 2014-04-01 1966 @ 2014-03-01 ‘M’ @ 2012-01-01 3 @ 2014-03-27 ‘4670’ @ 2009-05-13 © Ambiata 2014
9. 736 @ 2014-01-01 3 @ 2014-02-19 184 @ 2014-02-01 312 @ 2014-03-01 customer-1 customer-2 customer-3 customer-4 gender balance purchases zipcode ‘M’ @ 2012-01-01 276 @ 2014-04-01 4 @ 2014-04-04 2 @ 2014-03-12 3 @ 2014-03-27 ‘2381’ @ 2004-08-19 ‘4670’ @ 2009-05-13 ‘F’ @ 2007-04-01 ‘3001’ @ 2011-09-14 1876 @ 2014-02-01 1966 @ 2014-03-01 634 @ 2014-02-01 Sparse 469 @ 2014-02-01 © Ambiata 2014
10. INGESTING FACTS © Ambiata 2014
11. • Facts are ingested in atomic units called factsets • Facts in a factset can span any set of: • entities • attributes • dates/times © Ambiata 2014
12. customer-1 balance 634 2014-02-01 customer-3 balance 184 2014-02-01 customer-4 purchases 4 2014-02-04 cusomter-2 balance 312 2014-03-01 customer-3 gender F 2007-04-01 customer-2 zipcode 3001 2011-03-14 © Ambiata 2014
13. ATTRIBUTE DICTIONARY © Ambiata 2014
14. • Any attribute that is ingested must be declared in the repository’s dictionary • Dictionary stores metadata for each attribute • Updated dictionaries can be imported into a repository at any time © Ambiata 2014
15. namespace name encoding type description demographics gender string categorical Gender demographics zipcode string categorical Post-code, zip-code accounts balance double numerical Balance of savings account accounts purchases int numerical Number of credit-card purchases © Ambiata 2014
16. EXTRACTING FEATURES © Ambiata 2014
17. © Ambiata 2014 0.00 3 3001 634.83 16 4670 15.12 2 - 33.56 2 - 98.34 12 3303 523.81 23 2046 1086.05 17 - 224.81 9 - 78.21 2 2134 126.48 4 - M - F M F - F F M - gender balance purchases zipcode 89340218 feature instance 48149407 18452274 07499337 62948721 93754723 00272446 13374497 31989993 46474236
18. SNAPSHOTS • Attribute values for entities at a point in time • Same time for all entities • Select latest attribute values with respect to that time • Typically used in preparing instances for scoring © Ambiata 2014
19. 736 @ 2014-01-01 6 @ 2014-02-19 184 @ 2014-02-01 312 @ 2014-03-01 customer-1 customer-2 customer-3 customer-4 gender balance purchases zipcode ‘M’ @ 2012-01-01 276 @ 2014-04-01 4 @ 2014-02-04 2 @ 2014-03-12 3 @ 2014-03-27 ‘2381’ @ 2004-08-19 ‘4670’ @ 2009-05-13 ‘F’ @ 2007-04-01 ‘3001’ @ 2011-09-14 1876 @ 2014-02-01 1966 @ 2014-03-01 634 @ 2014-02-01 snapshot @ 2014-03-01 469 @ 2014-02-01 © Ambiata 2014
20. customer-1 customer-2 customer-3 customer-4 gender balance purchases zipcode ‘M’ 312 4 ‘4670’ ‘F’ ‘3001’ 1966 634 469 © Ambiata 2014
21. • It is assumed snapshots run periodically - e.g. daily, weekly • Ivory exploits this assumption to improve the runtime of successive snapshots
22. CHORDS • Attribute values for entities at a point in time • Different times for different entities • Select latest attribute values with respect to the times • Typically used in preparing instances for training © Ambiata 2014
23. 736 @ 2014-01-01 6 @ 2014-02-19 184 @ 2014-02-01 312 @ 2014-03-01 customer-1 customer-2 customer-3 customer-4 gender balance purchases zipcode ‘M’ @ 2012-01-01 276 @ 2014-04-01 4 @ 2014-04-04 2 @ 2014-03-12 3 @ 2014-03-27 ‘2381’ @ 2004-08-19 ‘4670’ @ 2009-05-13 ‘F’ @ 2007-04-01 ‘3001’ @ 2011-09-14 1876 @ 2014-02-01 1966 @ 2014-03-01 634 @ 2014-02-01 469 @ 2014-02-01 customer2 @ 2014-03-01 customer4 @ 2014-01-01 © Ambiata 2014
24. customer-2 @ 2014-03-01 customer-4 @ 2014-01-01 gender balance purchases postcode ‘M’ 312 6 ‘4670’ ‘3001’ 1876 © Ambiata 2014
25. DERIVED FACTS © Ambiata 2014
26. 184 @ 2014-02-01 312 @ 2014-03-01 customer-2 balance max.balance.4M 276 @ 2014-04-01 ? © Ambiata 2014 Maximum balance over last 4 months can be derived from set of balance facts
27. Many facts can be derived from a time series of base facts © Ambiata 2014
28. base fact derived facts balance Maximum balance over the last month Mean balance over the last 2 months Balance gradient over the last 3 months purchase Number of purchases in the last 3 weeks Proportion of supermarket purchases in the last 2 weeks zipcode Number of times the zipcode has change in the last 5 years Longest period where the zipcode has not changed in the last 5 years © Ambiata 2014
29. VIRTUAL FEATURES © Ambiata 2014
30. • Ivory represents derived facts as virtual features • Virtual features are declared in the dictionary • Specify expressions against base facts • Are computed lazily when features extracted © Ambiata 2014
31. name source expression window max.balance.4M balance max 4 month mean.balance.6M balance mean 6 months num.purchases.3W purchase count 3 weeks changes.zipcode.5Y zipcode num_flips 5 years © Ambiata 2014
32. COMMITS © Ambiata 2014
33. Ivory Repository Ingest facts Extract features Import dictionary © Ambiata 2014
34. • A commit is recorded for any repository change: • factset ingestions • dictionary imports • The repository at a given commit is an immutable data store • Snapshot and chord can be done at a specific commit © Ambiata 2014
35. 1 2 3 4 5 0 create repository import dictionary ingest factset ingest factset import dictionary ingest factset snapshot snapshot chord © Ambiata 2014
36. KEY CONCEPTS © Ambiata 2014
37. • Repository • Commit • Dictionary • Factset • Base fact • Virtual feature • Snapshot • Chord © Ambiata 2014