Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DIY Your Touchpad Experience: Building Your Own Gestures

DIY Your Touchpad Experience: Building Your Own Gestures

iberianpig

May 11, 2023
Tweet

Other Decks in Programming

Transcript

  1. DIY Your Touchpad Experience
    Building Your Own Gestures
    Kohei Yamada

    View Slide

  2. About Me
    Kohei Yamada
    GitHub @iberianpig
    Twitter @nukumaro22
    自己紹介

    View Slide

  3. Do you like improving your environment?
    環境カイゼンは好きですか?
    > Many programmers, myself included, like to customize our environment to
    our liking. They are happy when it improves their efficiency, even if only a little.
    > You could call them a kind of
    "Environment Improvement Geek".
    > Matz
    私を含めて多くのプログラマは,自分の環境を自分の好みに合わせてカスタマイズすることを好みます。それによってほんのわずかでも自分の効率が向上すると喜びを感じるのです。
    ある種の
    「環境改善オタク」と呼んでもよいでしょう。
    [改訂新版]Emacs実践入門──思考を直感的にコード化し、開発を加速する より 本書に寄せて
    まつもと ゆきひろ

    View Slide

  4. My Development Environment
    Dell XPS 13 Ubuntu 22.04 / X11 Vim
    LAPTOP OS Editor
    私の開発環境

    View Slide

  5. Environment Improvement
    環境の改善
    Daily yak shaving builds plugins

    View Slide

  6. About Fusuma

    View Slide

  7. A Gem that enables touchpad gestures on Linux.
    About Fusuma
    Fusumaについて

    View Slide

  8. About Fusuma
    Fusumaについて
    🐧 On Linux touchpads
    ✋ Swipe
    👌 Pinch-to-zoom

    View Slide

  9. View Slide

  10. Fusuma Features
    Fusumaの特徴
    ● Easy installation with RubyGems
    ● Defining Gestures and Actions in YAML
    ● Extension of gesture recognition by Plugin system

    View Slide

  11. How to use Fusuma
    Fusumaの使い方
    Install libinput-tools
    1
    Add $USER to Input group
    2
    Customize ~/.config/fusuma/config.yml
    5
    Run $ fusuma from the CLI
    4
    Install Fusuma with $ gem i fusuma
    3

    View Slide

  12. Customize gestures and actions
    ジェスチャとアクションのカスタマイズ
    ● Swipe
    ● Pinch
    ● Rotate
    ● Hold

    ● command
    ● sendkey(plugin)
    ● wmctrl(plugin)
    Gesture
    Action

    View Slide

  13. DEMO
    ● 4-finger Swipe
    ○ Switch Workspace
    ● 3-finger swipe
    ○ Open/Close browser tab
    ○ Back/forward browser history
    ● 3-finger dragging
    デモ

    View Slide

  14. How I make Fusuma

    View Slide

  15. HW X server
    Input Event Flow in Linux
    LinuxでのInputイベントの流れ
    Kernel
    device evdev libevdev libinput xf86-input-libinput
    X client
    App
    input event Simulate keyboard
    Fusuma
    Input Event Flow

    View Slide

  16. Touchpad drivers on Linux
    ● Well-known touchpad drivers, around 2010
    ● only X.Org
    ● Many Configuration options
    ● Gestures were not supported
    ● Release of libinput 1.0 in 2015
    ● Used in Wayland and X.Org
    ● No configuration file
    ● Improvements to gesture support
    Synaptics
    2015
    libinput
    2010 2020
    Libinput
    Synaptics
    Linuxのタッチパッドドライバ

    View Slide

  17. Seeking multitouch Gestures
    ● Desire for Mac-like trackpad gestures
    ● Test different laptops' gesture behavior in
    electronics store
    Prototype on Synaptics
    Synapticsドライバでのプロトタイプ
    Released xSwipe in 2012
    ● Discovered logging capability with
    `$ synclient -m` for synaptics
    ● Read and parse logs in loop
    ● Detect direction and send shortcut
    ● Written in Perl
    Synaptics
    2015
    libinput
    2010 2020
    My first DIY

    View Slide

  18. Synaptics to Libinput
    SynapticsからLibinputへ
    Synaptics replaced by libinput
    ● synaptics driver in maintenance mode
    ● xSwipe only worked with Synaptics driver
    ● Frustration with inability to use gestures
    My First Gem
    Synaptics
    2015
    libinput
    2010 2020
    Released Fusuma in 2016
    ● Use `libinput debug-events`
    ● Rewrite xSwipe in Ruby for libinput
    ● `rake release`!
    🪦

    View Slide

  19. Gesture Recognition

    View Slide

  20. Gesture Recognition
    ジェスチャ認識
    Pipeline
    Input Buffer Detector Executor
    Libinput Gesture
    Swipe
    Pinch
    Rotate
    Hold
    Command
    Touchpad
    Keyboard
    1 4
    2 3

    View Slide

  21. Input
    libinput-tools CLI as input
    inputとしてのlibinput-tools
    `libinput debug-events`

    View Slide

  22. ● Execute libinput using Open3.popen3
    ● Output lines without buffering using
    `stdbuf -oL`
    ● convert a line to a event
    $stdout to event in real-time
    リアルタイムで標準出力をイベントに
    Input

    View Slide

  23. Buffering Events
    イベントを溜める
    ● Aggregate events
    ● Multiple buffers depending on input
    Buffer
    Input Buffer
    Libinput Gesture
    Detector
    Swipe
    Executor
    Command
    Keypress
    Gesture State Time fingers delta x delta y
    SWIPE BEGIN +37.492s 3 10.23 -1.14
    SWIPE UPDATE. +37.492s 3 10.54 -1.17
    SWIPE UPDATE. +37.499s 3 7.62 -1.76
    SWIPE UPDATE. +37.506s 3 10.25 -2.64

    View Slide

  24. Gesture detection
    ジェスチャーの検知
    Gesture State Time fingers delta x delta y
    SWIPE BEGIN +37.492s 3 10.23 -1.14
    SWIPE UPDATE. +37.492s 3 10.54 -1.17
    SWIPE UPDATE. +37.499s 3 7.62 -1.76
    SWIPE UPDATE. +37.506s 3 10.25 -2.64
    Detect a left swipe with three fingers
    => `swipe / 3 / right`
    Compare the average movement with threshold
    Detector
    Create Event with search index

    View Slide

  25. Detect a left swipe with three fingers
    Retrieving Action from Config
    設定ファイルからアクションを導出
    config.yml
    `swipe / 3 / right` => { command: ‘xdotool key alt+Left’ }
    Executor

    View Slide

  26. Executor
    Executing Actions
    アクションの実行
    ● Execute command assigned to a gesture
    ● Use Process.spawn to avoid slowing down
    the main loop
    ● Use Process.detach to avoid zombie
    Detect a left swipe with three fingers
    `swipe /3 / right` => { command: ‘xdotool key alt+Left’ }

    View Slide

  27. Pipeline for Gesture Recognition
    1. input:
    2. buffer:
    3. detector:
    4. executor:
    handle multiple events with IO.select
    Aggregate events
    Detect gestures from the events
    Execute actions assigned from events
    Structure of Pipeline lib/fusuma.rb
    ジェスチャー認識のパイプライン
    Input Buffer Detector Executor
    Libinput Gesture
    Swipe
    Pinch
    Rotate
    Hold
    Command
    1 4
    2 3

    View Slide

  28. Extend Pipeline
    with Plugins

    View Slide

  29. Extend Pipeline with Plugins
    Each tasks in pipeline are implemented with a plug-in system
    プラグインでパイプラインを拡張する
    ● Depends on result(event) previous tasks
    ● Unified interface for each task
    ○ e.g) Input#io, Executor#execute
    ● Pipeline just call methods along the interface
    ● Additional features using Gem
    ● Implement tasks that satisfy interface
    pipeline
    plugins
    Input Buffer Detector Executor
    Libinput
    Tap
    Gesture
    Timer
    Swipe
    Pinch
    Rotate
    Hold
    Command
    Timer
    App
    Tap
    App
    Tap
    App
    Keypress Keypress
    Sendkey
    Wmctrl

    View Slide

  30. Internal plugins(default)
    Internal plugins
    Each task in pipeline is a plugin
    内部プラグイン
    Input Buffer Detector Executor
    Libinput
    Timer
    Gesture
    Timer
    Swipe
    Pinch
    Rotate
    Hold
    Command

    View Slide

  31. External plugins
    外部プラグイン
    External plugins(gem)
    Detector
    Input
    Executor
    Buffer
    Executor
    Detector
    Buffer
    Input Detector
    Buffer
    Input Detector
    Buffer Executor
    Executor
    Executor
    Input Executor
    Input Detector
    Buffer
    Input Detector
    Buffer
    Experimental Features / Environment-specific / Requires additional tool

    View Slide

  32. ● Uses evdev to send key events as from
    keyboard device
    ○ evdev is a generic kernel interface used by
    both X11 and Wayland
    ○ Using libevdev requires ffi
    ■ need to install ruby-dev
    fusuma-plugin-sendkey
    evdevを利用したキーイベント送信プラグイン

    View Slide

  33. fusuma-plugin-appmatcher
    アプリ毎に設定を切り替えできるプラグイン
    ● Configure application-specific gestures and actions
    in config.yml
    ● Switches the config according to the active
    application
    ● Supports X11 and Gnome shell(Wayland)
    ○ X11: xprop -spy
    ○ Gnome: Gnome Extension and D-Bus

    View Slide

  34. Plugin mechanism

    View Slide

  35. Plugin mechanism
    Find Gem using conventions
    Register to plugin list by hooks
    Call registered plugins
    1
    2
    3
    プラグインのメカニズム

    View Slide

  36. Find Gem using conventions
    規約ベースでGemを探索する
    ● Gem name rule
    ○ "fusuma-plugin-*"
    ● Directory rule
    ○ "lib/fusuma/plugin/input/*.rb"
    ■ input/buffer/detector/executor
    Find plugins from gem
    conventions
    ▲ Referenced plugin mechanisms from fluentd, pry, ruboty, and textbringer.
    ● Seek plugins using Gem.find_latest_files
    ● Check dependencies using gemspec

    View Slide

  37. Register to plugin list by hooks
    フックでプラグインを登録
    ● Use inherited hook into class inheritance
    ● Registers relationship
    ○ superclass => subclass relationship
    ■ Base => [Input, Buffer, Detector, Executor]
    ■ Input => [LibinputCommandInput, ...]
    Example:
    ● Input < Base
    ● LibinputCommandInput < Input

    ● Base -> Input -> LibinputCommandInput
    Inherited hooks

    View Slide

  38. Call registered plugins
    登録したプラグインの呼び出し
    ● Get plugin classes from Manager
    ● Initialize plugins to start pipeline in main process
    Initialize registered plugins lib/fusuma.rb

    View Slide

  39. Plugin and Pipeline
    Pros / Cons
    Input Buffer Detector Executor
    Libinput
    Tap
    Gesture
    Timer
    Swipe
    Pinch
    Rotate
    Hold
    Command
    Timer
    App
    Tap
    App
    Tap
    App
    Keypress
    Keypress
    Sendkey
    Wmctrl

    View Slide

  40. Pros / Cons of plugin and pipeline
    Pros
    🙆 Event-based coding simplifies understanding and
    testing
    🙆 Easily decide where to place the source code
    Cons
    🙅 Processing can be redundant
    🙅 State management can be difficult
    Pipeline Plugin
    Pros
    🙆 Release of experimental features
    🙆 Pipeline no longer dependent on plug-ins
    Cons
    🙅 Code maintenance can be complicated
    🙅 Difficult to detect conflicts and interference between
    external plugins, resulting in bugs
    プラグインとパイプラインのメリット・デメリット

    View Slide

  41. my new gem…

    View Slide

  42. Limitations of Dragging gestures on Touchpad
    タッチパッドのドラッグ操作って難しい
    🙅 Tap and drag with one finger ends when the finger leaves the touchpad
    🙅 3-finger dragging is also difficult to operate due to the narrow range of movement
    🙅 Difficult to move and scroll the workspace while dragging operation is in progress

    View Slide

  43. Rekimoto, Jun. "ThumbSense: Automatic input mode sensing for touchpad-based
    interactions." _CHI'03 Extended Abstracts on Human Factors in Computing Systems_. 2003.
    https://www2.sonycsl.co.jp/person/rekimoto/papers/chi03tsense.pdf
    ThumbSense
    タッチパッドは修飾キー
    Only when the finger is on the touchpad
    ● Switch specific keys to left-click or right-click
    ● Replace with any command or keybinding

    View Slide

  44. fusuma-plugin-thumbsense
    タッチ中にキーリマップできるプラグイン
    Only when the finger is on the touchpad
    ● Convert specific keys to left-click or right-click
    ● Replace with any command or keybinding

    View Slide

  45. DEMO
    デモ
    ● Click
    ○ J/F keys remapped to LEFT Click while touching on Touchpad
    ● Drag
    ○ Scrolling and swiping can be used simultaneously
    ○ Works well with touchpad gestures.

    View Slide

  46. Implement ThumbSense as plugin
    プラグインとしてThumbSenseを実装
    ● Detect touch/release on touchpad
    ● Send request to Change Layer
    fusuma-plugin-thumbsense
    fusuma-plugin-remapper
    Input Detector
    Buffer Executor
    Input Detector
    Buffer Executor
    ● Remapper running in external process
    ● Remap a key to mouse button
    ● Create Virtual Device using uinput
    ● lnput plugin to redirect key events

    View Slide

  47. fusuma-plugin-thumbsense
    タッチ検知でレイヤーを変更するプラグイン
    ● When `touch / 1 / begin` detected,
    ● Detector write to pipe for changing layer
    ■ layer = { thumbsense: true }
    mapping = { j: 'BTN_LEFT', k: 'BTN_RIGHT' }
    fusuma-plugin-thumbsense
    1
    1
    2
    2

    View Slide

  48. fusuma-plugin-remapper
    外部から動的にキーリマップできるプラグイン
    ● When receiving a layer change,
    ○ merge the layer and update key mapping
    ● Map physical keyboard events to current mapping
    ○ Write mapped event to Virtual Device
    ● Redirect physical keyboard events to Fusuma Input
    ○ Remapper works as "RemapperInput".
    fusuma-plugin-remapper
    1
    1
    2
    2
    3
    3

    View Slide

  49. Advantages of separating Remapper
    リマッパーを分離したポイント
    ○ Remapper only converts keys, while Fusuma only detects events
    ○ As a side effect, Appmatcher + Remapper allows you to define
    customizable key remappers for specific app
    ○ Remapper can also call existing executor plugin such as
    command/wmctrl/sendkey

    View Slide


  50. Let's build your own tools that feel good in your hands using Ruby!

    View Slide