What Reading 5 Papers can yield for your Business

From my Lambda Days talk.

Transcript

3. Answer to question of life, the Universe and everything
4. But what’s even the question?
5. Somehow
6. Somehow Somehow: numbers evolve around percentiles clustering evolves around words & spam filters regressions are about predicting page views combinations/permutations are not even mentioned
8. What reading 5 papers can yield for business
9. Minimal toolchain to yield info
10. Get fields (+ groups/permutations) ! Counts for categorical values ! Distributions for numerical values
11. Getting first impression of data
12. How many?
13. Did you see that one?
14. What’s the distribution?
15. How different are these?
16. Does it belong here?
17. Getting info
18. `someone` said “Measure everything” Let’s just say that you can efficiently measure locally, merge on stat machines percentiles are cool and everything, but mostly for latency
19. Same approaches are used in databases HFT high-scale data analysis ! So should work for pretty much anyone.
20. OLAP Principles
21. Consolidation
22. Time: Tumbling
26. 2-1 = 1, less than 5 buffer
27. 3-1 = 2, less than 5
30. ok, time to emit!
31. emitting values:
32. new buffer
33. etc etc etc
34. Time: Sliding
38. 2-1 = 1, less than 5 buffer
39. 3-1 = 2, less than 5
42. ok, time to emit!
43. emitting values:
46. etc etc etc
47. Grouping
49. Joining
50. (R ⋈ S) cartesian product
51. Observations (if group / reduce won’t do)
52. Count-min Sketch (how many?)
54. Count-min sketch Millions of IPs How many requests per IP? Millions Object Allocations How many Objects of type N were allocated? Efficient for large amount of “keys” Approximates minimum amount of occurrences
55. Hashing Non-cryptographic hash function (like Murmur3) We need N linear hashing fns so we take Random Seed
56. Update Given: K(rows) M (width) table(KxM) ! for i in (0..K) h = hash(item, seed) position = h % M table[i][position] += 1 end
59. Lookup Given: K(rows) M (width) table(KxM) ! for i in (0..K) h = hash(item, seed) position = h % M return table[i][position] end
60. Top-K Hold a CM-sketch Do an update on each event occurrence Do a lookup in CM-sketch Add looked up key + value to the top-list If top-list is over capacity, trim the smallest value
61. Streaming Histogram
62. Update Given: h={(p1 ,m1 ), . . . , (pB ,mB )} ! if p = pi for some i mi = mi + 1 else find min <-> between points merge (pi ,mi ) and (pi+1 ,mi+1 ) replace (pi ,mi ) with merged end
63. Update Given: h={(p1 ,m1 ), . . . , (pB ,mB )} ! if p = pi for some i mi = mi + 1 else mB+1 =p (expanding) if (B + 1 > max-size) find min <-> between points merge (pi ,mi ) and (pi+1 ,mi+1 ) replace (pi ,mi ) with merged end end
68. Histogram Initially written for streaming decision-trees Non-approximate, will give you exact stats Lightweight, since only has to keep N bins Very useful when you don’t know value bounds
72. HyperLogLog
73. Number of distinct items in X To use with any of the tricks from `consolidation` section Approximate, although error is known HyperLogLog
74. K-means
79. Operates on vectors (double) Assign each observation to cluster Take Eucledian distance measure, by minimising for each dimension ! Calculate new means to be centroids Repeat until converges Use results to yield a model K-means
80. Naive Bayes
81. Operates on vectors (double) Model = class, variance, cluster mean Find the cluster closest to the observation 100% streamable Naive bayes
82. Minimum amount of memory 100% commutative (Many) more algorithms available out there Data will show the way, no golden hammer Conclusions
83. Learn yourself some statistics probability theory Don’t wait for use case, find profit in data now Conclusions
84. @ifesdjeen