Process tons of jobs with Swift

Tokyo Server Side Swift meetup #8

Transcript

1. Process tons of jobs with Swift Tokyo Server Side Swift meetup 2017.06.27 @ainame
2. Self-intro • Satoshi Namai ◦ https://github.com/ainame ◦ https://twitter.com/ainame • iOS Engineer? • Love Ruby and Swift • Heavy user of sidekiq gem for tons of background jobs in previous job
3. Agenda • My motivation • About sidekiq gem • ainame/Lumpik • vs Crystal-lang • Redis client
4. My motivation • SSS(Server Side Swift) is very very very hard!! ◦ Can’t keep writing without enthusiasm ◦ It’s fun for me. • Learn not only Swift but also the System Programming ◦ Swift has compatibility for C-APIs, we can write applications with low-level APIs. • Be a Hacker ◦ Few people try ◦ This is an opportunity
5. SSS except Web • There are many Web app frameworks in Swift ◦ Kitura, Vapor, Zewo, Perfect, etc… ◦ https://github.com/swift-server/http • How about the background job processing? ◦ There’s nothing? ◦ Opportunity for me
6. Background job processing • works with Web app normaly ◦ Web app should respond for the client quickly • works for the time-consuming tasks ◦ UPDATE query ◦ send push/e-mail notifications ◦ video encoding/image processing etc...
7. Redis Web/Rails app Cron/Rails runner One-off scripts Worker Worker Worker Worker
10. Sidekiq • Fastest Ruby’s job system for general purpose ◦ works with Redis as a queue for jobs ◦ works with rails well • Why many rubyists choose sidekiq? ◦ less memory & low latency ◦ nice web admin ◦ Pro/Enterprise Support!! http://sidekiq.org/about
11. https://www.indiehackers.com/businesses/sidekiq
13. Inside Sidekiq • Sidekiq has 3 main modules ◦ Processor - fetches a job and executes a worker on dedicated thread ◦ Poller - poll and enqueue jobs to retry ◦ Heatbeat - send statictics metadata to Redis (lib/sidekiq/launcher.rb) • Definition: Processor/Worker/Job ◦ Job is data like arguments of function for Worker ◦ Worker is the PORO what you want to do ◦ Processor fetches a job and executes
14. Redis Poller Heatbeat Processor Processor Processor Processor Worker Worker Worker Worker Job Job Job Job Manager Manage retried jobs • re-enqueue retry jobs • adjust polling intervals by the number of processes Send process statuses for the admin • How many do processors work? • How many did jobs succeeded/failed? • What job dose processor work with now? Fetch
15. FIFO queue on the Redis light jobs id=5 id=7 id=27 …. id=58 heavy “abc” “swift” “fuga” …. “ios” > RPUSH “light” 58 > RPUSH “heavy” “ios” > BRLPOP “light” “heavy” 2 id=5 Fetch a job from two queues Append a job to light queue
16. ainame/Lumpik
17. ainame/Lumpik Sidekiq clone on the Swift • Fastest Swift’s job queue system for general purpose? • Type-safe arguments • Good concurrency with GCD • Compatibiity for sidekiq-web
20. Dependencies • vapor/redis・・・ Redis client • ainame/Swift-Daemon・・・Damonize a process • IBM-Swift/BlueSignals・・・Handle signals • SwiftBeaver/SwiftBeaver・・・Logger • kylef/Commander・・・Option parser • jpsim/Yams・・・YAML
21. ainame/Swift-Daemon • Daemonize a Swift process • How to detach a process from the shell 1. fork 2. setsid 3. dup2 • Learn from ruby/ruby
22. Invoke a specified worker from String Swift can’t instatiate any class from String → Generate Router class from source code with Sourcery
23. How to use ainame/Lumpik $ swift package init $ emacs Package.swift $ swift package resolve $ emacs xxxx/main.swift ... $ brew install sourcery $ sourcery --sources xxx --templates Templates --output xxx
24. Demo
25. ainame/Lumpik is a WIP • Not implemented yet several features ◦ dead queue ◦ scheduled queue • Targets for Swift 4 ◦ faster String APIs ◦ adopt Codable/JSONEncoder, but Swift doesn’t support it on the Linux currently • Cloud be faster than sidekiq.cr ◦ make faster Redis client in Swift
26. vs sidekiq.cr Original author, @mperham makes faster sidekiq on the Crystal-lang • Pros ◦ mperham/sidekiq.cr is really fast ◦ ruby like syntax ◦ very fast • Cons ◦ doesn’t work CPU intensive jobs well by multithreading
27. Benchmarks crystal-redis client is crasy fast... Redis: Crystal vs Ruby vs Node vs ... https://www.stefanwille.com/2015/05/redis-clients-crystal-vs-ruby-vs-c-vs-go/ https://github.com/stefanwille/redis-client-benchmarks https://github.com/ainame/vapor-redis-benchmark I have a plan to make more faster Redis client than crystal-redis but it’s difficult for several reasons...
28. What I want for Redis client in Swift • Prefer method signature to enum based signature for command definition ◦ redis.set(“key”, value) vs redis.command(.set, [“key”, value]) • Prefer real typed value to enum based value for response ◦ get(_ key: String) throws -> String vs get(_ key: String) -> ResponseValue • Pipelining / MULTI-EXEC transaction ◦ Pipelining is a basic technique to speedup • Using buffered IO based TCP client ◦ buffered IO is also basic technique to speedup
29. Recap • Sidekiq is one of the most successful OSS • Learn how to make good middleware with 写経 • Swift dosen’t suit for system programming now • We have to create fast and easy IO library like Ruby or Crystal
30. Enjoy Server Side Swift!