Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Joystick Controller using Compose Mu...

Building a Joystick Controller using Compose Multiplatform

Ready to play and learn with Compose Multiplatform? Let’s discover the Gesture APIs to craft our very own joystick.
This hands-on session is perfect for developers keen on improving their Compose skills across platforms. You'll get to mix a bit of trigonometry fun with the practical magic of Compose Multiplatform, applicable for both iOS and Android. Come along for an adventure in coding - it's going to be educational, practical, and most importantly, a lot of fun!

Renaud MATHIEU

April 29, 2024
Tweet

More Decks by Renaud MATHIEU

Other Decks in Programming

Transcript

  1. 🤌 Filippo Scognamiglio 🥖 Renaud Mathieu Building a Joystick using

    Compose Multiplatform Android Makers by droidcon 2024 ❤
  2. Understand gestures pointerInput fun Modifier.pointerInput( key1: Any?, block: suspend PointerInputScope.()

    -> Unit ): Modifier = this then SuspendPointerInputElement( key1 = key1, pointerInputHandler = block )
  3. Understand gestures detectDragGestures suspend fun PointerInputScope.detectDragGestures( onDragStart: (Offset) -> Unit

    = { }, onDragEnd: () -> Unit = { }, onDragCancel: () -> Unit = { }, onDrag: (change: PointerInputChange, dragAmount: Offset) -> Unit ): Unit
  4. Joystick NavKey, NavState @Composable pointerInput(x,y) enum class NavState { RELEASED,

    PRESSED } enum class NavKey { UNKNOWN, ARROW_LEFT, ARROW_UP, ARROW_RIGHT, ARROW_DOWN, ARROW_UPPER_LEFT, ARROW_LOWER_LEFT, ARROW_UPPER_RIGHT, ARROW_LOWER_RIGHT }
  5. Joystick @Composable @Composable fun JoyStick( modifier: Modifier = Modifier, size:

    Dp = 200.dp, dotSize: Dp = 110.dp, onValueChanged: (NavKey, NavState) -> Unit )
  6. 🧰 Tools A. Calculate the angle B. Find the direction

    from the angle C. Apply the offset to the dot
  7. x,y 0,0 ϑ A B C tan(ϑ) = BC /

    AC tan(ϑ) = y / x ϑ = atan(y / x) Calculate the angle
  8. theta = getTheta(x, y) private fun getTheta(x: Float, y: Float)

    = when { x < 0 -& y -= 0 -> (PI).toFloat() + atan(y / x) x < 0 -& y < 0 -> -(PI).toFloat() + atan(y / x) else -> atan(y / x) }
  9. Find the direction from the angle private fun getDirectionFromAngle(theta: Float):

    NavKey = when { theta -= 0 -& theta -= PI.div(6) -> { NavKey.ARROW_RIGHT } theta -= PI.div(6) -& theta -= PI.div(3) -> { NavKey.ARROW_LOWER_RIGHT } …
  10. Calculate the offset of the dot Calculate the radius then

    the offset • Polar to Cartesian private fun polarToCartesian( radius: Float, theta: Float ): Pair<Float, Float> = Pair(radius * cos(theta), radius * sin(theta))
  11. Box( modifier = Modifier .paint( painter = painterResource("joystick_background.xml"), alignment =

    Alignment.Center ) ) { Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot" ) }
  12. val localDensity = LocalDensity.current val centerX = with(localDensity) { ((size

    - dotSize) / 2).toPx() } val centerY = with(localDensity) { ((size - dotSize) / 2).toPx() } Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot", modifier = Modifier .offset { IntOffset( centerX.roundToInt(), centerY.roundToInt() ) } .size(dotSize) )
  13. val localDensity = LocalDensity.current val centerX = with(localDensity) { ((size

    - dotSize) / 2).toPx() } val centerY = with(localDensity) { ((size - dotSize) / 2).toPx() } Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot", modifier = Modifier .offset { IntOffset( centerX.roundToInt(), centerY.roundToInt() ) } .size(dotSize) )
  14. var navKey by remember { mutableStateOf(NavKey.UNKNOWN) } var navState by

    remember { mutableStateOf(NavState.RELEASED) }
  15. .pointerInput(Unit) { detectDragGestures( onDragStart = { navState = NavState.PRESSED },

    onDragEnd = { }, onDrag = { pointerInputChange: PointerInputChange, offset: Offset -> } ) }
  16. .pointerInput(Unit) { detectDragGestures( onDrag = { pointerInputChange: PointerInputChange, offset: Offset

    -> if (navState -= NavState.PRESSED) { -/ Nothing is pressed return@detectDragGestures } pointerInputChange.consume() val x = offsetX + offset.x - centerX val y = offsetY + offset.y - centerY theta = getTheta(x, y) val fourDimensionDirection: NavKey = getFourDimensionFromAngle(theta) if (navKey -= fourDimensionDirection) {
  17. theta = getTheta(x, y) val fourDimensionDirection: NavKey = getFourDimensionFromAngle(theta) if

    (navKey -= fourDimensionDirection) { if (navKey -= NavKey.UNKNOWN) { onValueChanged(navKey, NavState.RELEASED) } navKey = fourDimensionDirection -/ vibrate() if (navKey -= NavKey.UNKNOWN) { onValueChanged(navKey, NavState.PRESSED) } } } ) }
  18. radius = sqrt((x.pow(2)) + (y.pow(2))) offsetX += offset.x offsetY +=

    offset.y val t = polarToCartesian(radius, theta) positionX = t.first positionY = t.second
  19. val maxRadius = with(localDensity) { 30.dp.toPx() } if (radius >

    maxRadius) { polarToCartesian(maxRadius, theta) } else { polarToCartesian(radius, theta) }.apply { positionX = first positionY = second }
  20. Image( painter = painterResource("joystick_dot.xml"), contentDescription = "JoyStickDot", modifier = Modifier

    .offset { IntOffset( (positionX + centerX).roundToInt(), (positionY + centerY).roundToInt() ) }
  21. onDragEnd = { -/ Reset everybody offsetX = centerX offsetY

    = centerY radius = 0f theta = 0f positionX = 0f positionY = 0f -/ Release everybody navState = NavState.RELEASED if (navKey -= NavKey.UNKNOWN) { releaseAllNavKeys() navKey = NavKey.UNKNOWN } }
  22. Single gamepad state data class InputState( internal val digitalKeys: PersistentSet<Int>

    = persistentSetOf(), internal val directions: PersistentMap<Int, Offset> = persistentMapOf(), ) { fun setDigitalKey(keyId: Int, value: Boolean): InputState { … } fun getDigitalKey(digitalId: Int): Boolean { return digitalKeys.contains(digitalId) } fun setDirection(directionId: Int, offset: Offset): InputState { … } fun getDirection(directionId: Int, default: Offset = Offset.Unspecified): Offset { return directions.getOrElse(directionId) { default } } }
  23. Single input handling • All inputs are received a single

    compose function • Each Control compose function is associated to a Handler • The handler receives the pointer event if: ◦ It’s tracking the pointer ◦ The pointer is currently in its trigger area • Each handler updates the state and may request to track a pointer id data class Pointer(val pointerId: Long, val position: Offset) data class Result(val inputState: InputState, val dragGestureStart: Pointer? = null) fun handle(pointers: List<Pointer>, inputState: InputState, dragGestureStart: Pointer?): Result
  24. Finding the right Handler val gamepadPosition: MutableState<Offset> val trackedPointers =

    scope.getTrackedIds() val handlersAssociations: Map<Handler?, List<Pointer-> = event.changes .asSequence() .filter { it.pressed } .map { Pointer(it.id.value, it.position + gamepadPosition.value) } .groupBy { pointer /> if (pointer.pointerId in trackedPointers) { scope.getHandlerTracking(pointer.pointerId) } else { scope.getHandlerAtPosition(pointer.position) } }
  25. Updating the state scope.inputState.value = scope.getAllHandlers().fold(scope.inputState.value) { state, handler />

    val pointers = handlersAssociations.getOrElse(handler) { emptyList() } val relativePointers = pointers .map { Pointer(it.pointerId, it.position.relativeTo(handler.rect)) } val (updatedState, updatedTracked) = handler.handle(relativePointers, state, scope.getStartDragGestureForHandler(handler)) scope.setStartGestureForHandler(handler, updatedTracked) updatedState }
  26. Writing layouts - Scope class JamPadScope { internal val inputState

    = mutableStateOf(InputState()) private val handlers = mutableMapOf<String, Handler>() --. internal fun registerHandler(handler: Handler) { --. } }
  27. Analog Handling override fun handle(pointers: List<Pointer>, inputState: InputState, startDragGesture: Pointer?):

    Result { val currentDragGesture = pointers.firstOrNull { it.pointerId -= startDragGesture-.pointerId } return when { pointers.isEmpty() -> { Result(inputState.setDirection(id, Offset.Unspecified), null) } startDragGesture -= null -& currentDragGesture -= null -> { val deltaPosition = (currentDragGesture.position - startDragGesture.position) val offsetValue = deltaPosition.coerceIn(Offset(-1f, -1f), Offset(1f, 1f)) Result(inputState.setDirection(id, offsetValue), startDragGesture) } else -> { val firstPointer = pointers.first() Result(inputState.setDirection(id, Offset.Zero), firstPointer) } } }
  28. The Analog Control @Composable fun JamPadScope.ControlAnalog( modifier: Modifier = Modifier,

    id: Int, background: @Composable () -> Unit = { DefaultControlBackground() }, foreground: @Composable (Boolean) -> Unit = { DefaultButtonForeground(pressed = it, scale = 1f) }, ) { val position = inputState.value.getDirection(id, Offset.Zero) BoxWithConstraints( modifier = modifier .aspectRatio(1f) .onGloballyPositioned { registerHandler(AnalogPointerHandler(id, it.boundsInRoot())) }, contentAlignment = Alignment.Center, ) { Box(modifier = Modifier.fillMaxSize(0.75f)) { background() } Box(modifier = Modifier .fillMaxSize(0.50f) .offset(maxWidth * position.x * 0.25f, maxHeight * position.y * 0.25f), ) { foreground(inputState.value.getDirection(id) -= Offset.Unspecified) } } }
  29. Cross Handling enum class State(val position: Offset) { UP(Offset(0f, 1f)),

    DOWN(Offset(0f, -1f)), LEFT(Offset(-1f, 0f)), RIGHT(Offset(1f, 0f)), UP_LEFT(Offset(-1f, 1f)), UP_RIGHT(Offset(1f, 1f)), DOWN_LEFT(Offset(-1f, -1f)), DOWN_RIGHT(Offset(1f, -1f)), }
  30. override fun handle( pointers: List<Pointer>, inputState: InputState, startDragGesture: Pointer?, ):

    Result { val currentDragGesture = pointers.firstOrNull { it.pointerId -= startDragGesture-.pointerId } return when { pointers.isEmpty() -> { Result( inputState.setDirection(id, Offset.Unspecified), null ) } currentDragGesture -= null -> { Result( inputState.setDirection(id, findCloserState(currentDragGesture)), startDragGesture ) } else -> { val firstPointer = pointers.first() Result( inputState.setDirection(id, findCloserState(firstPointer)), firstPointer ) } } }
  31. The Cross Control @Composable fun JamPadScope.ControlCross( modifier: Modifier = Modifier,

    id: Int, background: @Composable () -> Unit = { DefaultControlBackground() }, foreground: @Composable (Offset) -> Unit = { DefaultCrossForeground(direction = it) }, ) { Box( modifier = modifier .aspectRatio(1f) .onGloballyPositioned { registerHandler(CrossPointerHandler(id, it.boundsInRoot())) }, ) { background() foreground(inputState.value.getDirection(id)) } }
  32. Face buttons We need to allow customization Depending on the

    number of buttons we lay them down in a circle. For each pointer in the circle we compute the closest button.
  33. Face buttons fun computeSizeOfItemsOnCircumference(itemsCount: Int): Float { val angle =

    sin(Constants.PI / maxOf(itemsCount, 2)) return (angle / (1 + angle)) }
  34. Composite buttons Some games require you to press two Face

    Buttons and the same time. We simulate this with composite buttons.
  35. Haptics Feeling a button press is very important because the

    brain needs to register the input happens. We can use Kotlin Multiplatform actual expect mechanism to implement the Android and iOS variants. When the input state changes we try can emit a press or release feedback.
  36. Haptics enum class HapticEffect { PRESS, RELEASE, } interface HapticGenerator

    { fun generate(type: HapticEffect) } @Composable expect fun rememberHapticGenerator(): HapticGenerator
  37. iOS Haptics object IosHapticGenerator : HapticGenerator { private val impactFeedbackGenerator

    = UIImpactFeedbackGenerator() override fun generate(type: HapticEffect) { when (type) { HapticEffect.PRESS -> impactFeedbackGenerator.impactOccurredWithIntensity(0.5) HapticEffect.RELEASE -> impactFeedbackGenerator.impactOccurredWithIntensity(0.3) } } }
  38. Android Haptics class AndroidHapticGenerator(applicationContext: Context) : HapticGenerator { … override

    fun generate(type: HapticEffect) { val effect = when (type) { HapticEffect.PRESS -> strongEffect HapticEffect.RELEASE -> weakEffect } vibrator.vibrate(effect) }
  39. A minimal gamepad JamPad( modifier = Modifier.fillMaxSize().aspectRatio(2f), onInputStateUpdated = {

    … } ) { // Inside JamPad scope… Row(modifier = Modifier.fillMaxSize()) { ControlCross( modifier = Modifier.weight(1f), id = 0 ) ControlFaceButtons( modifier = Modifier.weight(1f), ids = listOf(1, 2, 3) ) } }
  40. Introducing JamPadCompose A Virtual Gamepad Library for Kotlin Multiplatform https://github.com/piepacker/JamPadCompose

    • The library is still in its early stages of development • Contributions and feedback are highly welcome to improve the library and expand its capabilities.