Share
Embed
Transcript
30 slides
-
Process tons of jobs with Swift Tokyo Server Side Swift
meetup 2017.06.27 @ainame -
Self-intro • Satoshi Namai ◦ https://github.com/ainame ◦ https://twitter.com/ainame • iOS
Engineer? • Love Ruby and Swift • Heavy user of sidekiq gem for tons of background jobs in previous job -
Agenda • My motivation • About sidekiq gem • ainame/Lumpik
• vs Crystal-lang • Redis client -
My motivation • SSS(Server Side Swift) is very very very
hard!! ◦ Can’t keep writing without enthusiasm ◦ It’s fun for me. • Learn not only Swift but also the System Programming ◦ Swift has compatibility for C-APIs, we can write applications with low-level APIs. • Be a Hacker ◦ Few people try ◦ This is an opportunity -
SSS except Web • There are many Web app frameworks
in Swift ◦ Kitura, Vapor, Zewo, Perfect, etc… ◦ https://github.com/swift-server/http • How about the background job processing? ◦ There’s nothing? ◦ Opportunity for me -
Background job processing • works with Web app normaly ◦
Web app should respond for the client quickly • works for the time-consuming tasks ◦ UPDATE query ◦ send push/e-mail notifications ◦ video encoding/image processing etc... -
Sidekiq • Fastest Ruby’s job system for general purpose ◦
works with Redis as a queue for jobs ◦ works with rails well • Why many rubyists choose sidekiq? ◦ less memory & low latency ◦ nice web admin ◦ Pro/Enterprise Support!! http://sidekiq.org/about -
Inside Sidekiq • Sidekiq has 3 main modules ◦ Processor
- fetches a job and executes a worker on dedicated thread ◦ Poller - poll and enqueue jobs to retry ◦ Heatbeat - send statictics metadata to Redis (lib/sidekiq/launcher.rb) • Definition: Processor/Worker/Job ◦ Job is data like arguments of function for Worker ◦ Worker is the PORO what you want to do ◦ Processor fetches a job and executes -
Redis Poller Heatbeat Processor Processor Processor Processor Worker Worker Worker
Worker Job Job Job Job Manager Manage retried jobs • re-enqueue retry jobs • adjust polling intervals by the number of processes Send process statuses for the admin • How many do processors work? • How many did jobs succeeded/failed? • What job dose processor work with now? Fetch -
FIFO queue on the Redis light jobs id=5 id=7 id=27
…. id=58 heavy “abc” “swift” “fuga” …. “ios” > RPUSH “light” 58 > RPUSH “heavy” “ios” > BRLPOP “light” “heavy” 2 id=5 Fetch a job from two queues Append a job to light queue -
ainame/Lumpik Sidekiq clone on the Swift • Fastest Swift’s job
queue system for general purpose? • Type-safe arguments • Good concurrency with GCD • Compatibiity for sidekiq-web -
Dependencies • vapor/redis・・・ Redis client • ainame/Swift-Daemon・・・Damonize a process •
IBM-Swift/BlueSignals・・・Handle signals • SwiftBeaver/SwiftBeaver・・・Logger • kylef/Commander・・・Option parser • jpsim/Yams・・・YAML -
ainame/Swift-Daemon • Daemonize a Swift process • How to detach
a process from the shell 1. fork 2. setsid 3. dup2 • Learn from ruby/ruby -
Invoke a specified worker from String Swift can’t instatiate any
class from String → Generate Router class from source code with Sourcery -
How to use ainame/Lumpik $ swift package init $ emacs
Package.swift $ swift package resolve $ emacs xxxx/main.swift ... $ brew install sourcery $ sourcery --sources xxx --templates Templates --output xxx -
ainame/Lumpik is a WIP • Not implemented yet several features
◦ dead queue ◦ scheduled queue • Targets for Swift 4 ◦ faster String APIs ◦ adopt Codable/JSONEncoder, but Swift doesn’t support it on the Linux currently • Cloud be faster than sidekiq.cr ◦ make faster Redis client in Swift -
vs sidekiq.cr Original author, @mperham makes faster sidekiq on the
Crystal-lang • Pros ◦ mperham/sidekiq.cr is really fast ◦ ruby like syntax ◦ very fast • Cons ◦ doesn’t work CPU intensive jobs well by multithreading -
Benchmarks crystal-redis client is crasy fast... Redis: Crystal vs Ruby
vs Node vs ... https://www.stefanwille.com/2015/05/redis-clients-crystal-vs-ruby-vs-c-vs-go/ https://github.com/stefanwille/redis-client-benchmarks https://github.com/ainame/vapor-redis-benchmark I have a plan to make more faster Redis client than crystal-redis but it’s difficult for several reasons... -
What I want for Redis client in Swift • Prefer
method signature to enum based signature for command definition ◦ redis.set(“key”, value) vs redis.command(.set, [“key”, value]) • Prefer real typed value to enum based value for response ◦ get(_ key: String) throws -> String vs get(_ key: String) -> ResponseValue • Pipelining / MULTI-EXEC transaction ◦ Pipelining is a basic technique to speedup • Using buffered IO based TCP client ◦ buffered IO is also basic technique to speedup -
Recap • Sidekiq is one of the most successful OSS
• Learn how to make good middleware with 写経 • Swift dosen’t suit for system programming now • We have to create fast and easy IO library like Ruby or Crystal