Leveraging Deep Analysis: With Data Frame

Everybody has hear about Pandas for Data Science, but have you seen Kotlin Data Frame?
Join us for an in-depth exploration of Kotlin DataFrame, a powerful library that seamlessly integrates with Kotlin's conciseness and static typing to elevate your data science pursuits

We will dive deep into the advanced features of the DataFrame and Kotlin’s type system, demonstrating how to leverage it to prevent type-related errors, ensure data integrity for large amount of data.

Whether you are a data scientist or a Kotlin developer seeking to expand your skills, this session will provide you with a starting point of knowledge and tools to start using Kotlin for data science, let's sit down cause this talk is gonna be wild!

Transcript

1. Leveraging Deep Analysis With Data Frame Dinorah Tovar Google Developer Expert Android @ddinorahtovar @ddinorahtovar @ddinorahtovar @ddinorahtovar
2. I’m not a data engineer, data scientist or BI engineer Disclaimer ✨ I just like Kotlin, a lot✨ @ddinorahtovar
3. @ddinorahtovar Data This a story about ✨ and how data can be exploited ✨
4. First: Are amazing! Maths @ddinorahtovar
5. @ddinorahtovar Math’s are coolest thing! Kotlin help us to solve many problems - but specially in maths! • A fast operational language • Mathematical expressions • Algebraic data types (sealed class) • We can do simple functions - from basic to complex
6. def main(): z = complex(2, 1) w = complex(0, 0) start = time.time() for i in range(0, 10000000): w += z w *= z w -= z w /= z ende = time.time() @ddinorahtovar This function takes Seconds 2.76
7. fun main() { val y = complex(2, 1) var x = complex(0, 0) val time = measureTimeMillis { repeat(10000000) { x += y x *= y x -= y x /= y } } } @ddinorahtovar This function takes Mseconds 258
8. @ddinorahtovar 9 times Kotlin is at least ✨ faster and more efficient ✨
9. Mathematical expressions are pretty much pseudocode @ddinorahtovar
10. 2x + 10 First degree equation fun calc(x: Int) : Int { return 2*x + 10 } @ddinorahtovar
11. 10 ∑ i=1 1 i + 1 Sum val sum = 0 for (i in 1 .. 10) { sum += 1/(i+1) } fun f() = (1 .. 10).map { i -> 1 / (i + 1) }.sum() @ddinorahtovar
12. 3 ∏ i=1 i Product fun f() = (1 .. 3).fold( 1L, Long :: times) @ddinorahtovar
13. {1,2,4,10} Average Max - Min fun f() { val seq = floatArrayOf( 1F, 2F, 4F, 10F ) val avg = seq.average() val max = seq.max() val min = seq.min() } @ddinorahtovar
14. ( 222 222 222 ) Multi- dimensional Vectors fun f() { val mx = arrayOf( arrayOf(floatArrayOf(2F, 2F, 2F)), arrayOf(floatArrayOf(2F, 2F, 2F)), arrayOf(floatArrayOf(2F, 2F, 2F)) ) val normArray = mx.map { column -> column.map { row -> row.map { element -> element / 255F } } } } @ddinorahtovar
15. Second: Works EVERYWHERE… Kotlin @ddinorahtovar Yes, Everywhere
16. Let’s talk about Jupyter But first @ddinorahtovar and Python (cause thats what the data scientist use)
17. @ddinorahtovar Live coding 🔴 There’s isn’t a better way to explain it that: ✨ Disclaimer: Live coding is hard so please laugh if I make a mistake… I also not an expert in Python✨
18. @ddinorahtovar Kotlin works everywhere Mobile Cross platform Server Side Android Native Web Development Data Science
19. @ddinorahtovar Kotlin Kernel Kotlin Notebook Sends Code Kotlin-Jupyter Notebook Other tools Kotlin Kernel Jupyter Notebook Kotlin compiler Execute code Return results Kernel Layer
20. @DataSchema interface BasicInfo { val name: String val age: Int } fun main (args: Array<String>) { someVariable.sortBy { age } } @ddinorahtovar Kotlin can Code in classes Execute
21. @ddinorahtovar But also in Making it even accesible via REPL Cells val plot = letsPlot(plotData) + geomBar(stat = Stat.identity, fill = "#a2d2ff") { x = “some_column" y = “some_other_column” } + labs( title = “A cool name for the chart", x = "some_column", y = “some_other_column" ) + theme(axisTextX = elementText(angle = 45, hjust = 1))
22. @ddinorahtovar Live coding 🔴 Quick: ✨ Disclaimer: Don’t forget to laugh ✨
23. Third: Data manipulation Data frame @ddinorahtovar
24. @ddinorahtovar Pandas Data frame is inspired in one of the BIG ones ✨ Generic, Polymorphic and Interoperable✨
25. @ddinorahtovar Generic any type of Objects Complex enum class TemperatureUnit { CELSIUS, FAHRENHEIT, KELVIN } data class SensorReading( val value: Double, val unit: TemperatureUnit, val timestamp: Long ) { override fun toString(): String { return "Reading(${value.format(2)} ${unit}, ts=$timestamp)" } }
26. @ddinorahtovar Polymorphic Objects Structural hierarchy @DataSchema interface BasicInfo { val name: String val age: Int } @DataSchema interface FullInfo : BasicInfo { val city: String val job: String? }
27. @ddinorahtovar Let’s add some To this objects Data val dfA = dataFrameOf("name", "age")( "Alice", 30, "Bob", 25, "Eve", 28 ).cast<BasicInfo>() val dfB = dataFrameOf( “name", “age", “city", “job" )( "Charlie", 35, "New York", "Engineer", "David", 40, "London", "Artist", "Frank", 22, "Paris", null ).cast<FullInfo>()
28. @ddinorahtovar Polymorphic This operations Simplify fun DataFrame<BasicInfo>.printNamesAndAges() { this.select { BasicInfo :: name and BasicInfo :: age }.print() } fun main (args: Array<String>) { // Processing dfA (conforms to BasicInfo) dfA.printNamesAndAges() // Processing dfB // (conforms to FullInfo, // which includes BasicInfo) dfB.printNamesAndAges() }
29. @ddinorahtovar But also From data class to Data frames Interoperable data class Author(val name: String, val country: String) data class Book( val title: String, val year: Int, val author: Author, val genres: List<String> ) val booksList = listOf( Book( "The Hitchhiker's Guide", 1979, Author("Douglas Adams", "UK"), listOf("Sci-Fi", "Comedy") ) )
30. @ddinorahtovar But also From data class to Data frames Interoperable val dataFrame = booksList.toDataFrame()
31. @ddinorahtovar Live coding 🔴 The fun part: ✨ Disclaimer: Don’t forget to laugh ✨
32. @ddinorahtovar
33. Leveraging Deep Analysis With Data Frame Dinorah Tovar Google Developer Expert Android @ddinorahtovar @ddinorahtovar @ddinorahtovar @ddinorahtovar