Introducing Scala to your Java/Ruby Shop : My experiences at IGN

With the move to SOA, IGN's API engineering became a traditional Java/Ruby shop. We introduced Scala to the team in August '11. As of April '12 we have our major services running on Scala, and we could not be happier. I'll talk about the strategic & tactical rationale, gradual introduction to the new language, cultivating experts and mentoring junior developers. We will also cover the common roadblocks the leaders may run into, addressing the concerns of engineers and stakeholders, and building an engineering culture that facilitates innovation and talent growth. Please note that this talk is not a Ruby/Java/Scala religious debate - we believe in using the best tool for the job; and what works for us may not work for every org (which is where the culture comes in). My team's journey from "Holy crap! This looks academic and scary!" to "Let me whip out a service by EOD today" in a matter of months is something I am very proud of, and would love to share it with fellow developer community.

Transcript

1. Introducing  Scala  to  your  Java/ Ruby  Shop   Manish  Pandit   Silicon  Valley  Code  Camp  ‘12   Oct  7th,  2012   My  experiences  at  IGN  
2. About  me     Manish  Pandit   Director  of  Engineering,  IGN   @lobster1234     linkedin.com/in/mpandit  
3. About  IGN   We  are  a  leading  online  media  and  services   company     obsessed  with  gaming  and  entertainment.     56MM  Monthly  Unique  Views   737MM  Monthly  Page  Views   20MM  Monthly  Video  Views    
4. My  ObjecXve   Is:   To  share  my  story  of  introducing  Scala  to  IGN,  and   (hopefully)  show  you  the  path  to  enlightenment.     Isn't:   To  incite  a  funcXonal  vs.  imperaXve,  or  Scala  vs.  ____   debate.    
5. The  IGN  Tech  Stack    
6. Best  tool  for  the  job   •  Front  End  :  JS,  CSS3,  HTML5,  backbone,   Coffeescript,  jQuery   •  Middle  Tier  :    ZF2  (PHP),  Rails   •  APIs  :  Scala,  some  Java   •  Persistence  :  MongoDB,  MySQL,  Redis   •  Search  :  ElasXcSearch   •  Caching  :  Memcached,  Varnish    
7. APIs  at  IGN  :  Numbers   •  ~5  Billion  requests  a  month   •  Average  Response  Xme  of  under   20ms  on  cache  misses   •  17  APIs  between  Social,  OAuth,   and  Content   •  Hardest  hit  APIs  doing  about   25K  RPM  at  peak  
8. The  Kitchen  Sink   •  CMS  –  JSPs  talking  to  Oracle  DB  and  some  Java   services   •  V1  API  :  Java  Services  using  Oracle  DB  with   Memcached   •  V2  API  :  Rails  based  API  layer  using  MongoDB  and  a   Java/Memcached  front  cache   •  Lots  of  duct  tape  like  messaging  and  cron  jobs  for   simple  tasks   •  Too  diverse  of  a  stack  in  the  API  infrastructure  
9. The  API  EvoluXon  :  V3  [2012]   •  A  fresh  look  at  the  APIs  and  IGN’s  direcXon   •  First  alempt  to     – Integrate  all  content  types   – Open  up  the  APIs  to  external  consumers   – Evolve  the  CMS,  tradiXonally  a  monolithic  system   – ElasXcSearch   – Varnish  Response  Caching   •  A  learning  opportunity  for  tradiXonal  Java   stack  engineers  
10. Roadblocks   •  Resistance  to  change   – Culture   – Willingness  and  commitment   •  Too  many  choices  –  making  the  right  pick   – Involve  every  one   – Do  not  decide  by  commilee,  decide  by  learning   – Time  box  and  measure  everything   – Be  accountable,  as  you’re  looked  upon  as  the   expert  
11. The  hunt  for…   •  A  runXme  as  fast,  if  not  faster  than  Java     •  Concise,  yet  expressive   •  Less  boilerplate   •  Smooth  learning  curve  and  ramp  up   •  Re-­‐use  ecosystem  of  libraries     •  Growing  adopXon  and  developer  velocity   •  Built  in  support  for  concurrency   •  Fast  and  fun  development   •  Next  level  of  programming  –  disrupt!  
12. Why  Scala  :  Strategic  Reasons   •  Evolve  the  talent  brand  along  side  evolving   the  plaporm   •  Establish  an  API  plaporm  by  replacing   “rewrite”  with  learning  instead  of  por*ng   •  Start  thinking  funcXonal   •  Align  with  the  leaders  in  the  APIs   •  Get  involved  with  a  technology  as  it  is  evolving   (vs.  established).  Helps  with  influence.    
13. Why  Scala  :  TacXcal  Reasons   •  Performance  Xed  to  cores  than  processor   speed,  i.e.  concurrency   •  Ideal  for  API  development  –  (relaXvely)  simple   to  handle  concurrency  and  immutability   •  JVM  based  =  Performance   •  Re-­‐use  Java  toolkits  and  libraries   •  Concise,  yet  expressive  code     •  Actor  model  makes  the  difficult  parts  easy  
14. Why  Scala  :  TacXcal  Reasons   •  Richer  CollecXons  API   •  Traits  enabling  Mix-­‐Ins  and  behavior  re-­‐use   •  StaXcally  Typed  w/Type  inference   •  FuncXonal,  but  not  alien  (it  can  be!)   •  REPL  
15. Yes,  Virginia,  Scala  is  hard  :  DPP   So,  how  can  you  figure  out  if  Scala  will  be  "easy"  or  "hard"  for  your   organizaXon:   •  Your  company  has  speakers  at  JavaOne,  OSCON,  Strangle  Loop,  QCon:   Scala  will  be  easy   •  Lunch-­‐Xme  discussions  involve  the  criteria  for  moving  from  a  developer  to   a  senior  developer:  Scala  will  be  hard   •  Your  developers  can  write  code  in  NotePad  if  they  have  to:  Easy   •  Your  developers  stare  blankly  or  say  3  "Hail  Marys"  when  they  hear  the   name  "Zed  Shaw":  Scala  ==  Hard   •  Developers  all  follow  Dean  Wampler  on  Twiler:  Scala  Easy   •  Your  developers  come  in  at  9:15  and  leave  before  6  and  don't  check  work   email  at  night:  Hard   hlp://blog.goodstuff.im/yes-­‐virginia-­‐scala-­‐is-­‐hard  
17. Picking  up  Scala  :  Week  1   •  Scala  for  Java  Developers  fast-­‐tracked  the   learning   •  Syntax  Familiarity   •  Java  without  semicolons?   def total(nums: List[Int]) :Int = { var sum:Int=0 for(num<-nums) { sum+=num } sum }
18. Picking  up  Scala  :  Week  4   •  Reduce  the  lines  of  code,  embrace   immutability,  use  the  funcXonal  paradigms   built  into  the  language     •  Unmystrify  the  implicits  magic   nums.foldLeft(0)((n,c) => n+c)
19. Picking  up  Scala  :  Today   •  Focus  on  immutability   •  Think  funcXonal  –  with  the  knowledge  gained   from  using  the  funcXonal  aspects  of  the   language,  build  control  abstracXons   •  Explore  and  implement  ScalaZ   •  Keep  learning,  and  applying   •  Move  all  the  Actor  processing  to  Akka    
20. Less  Boilerplate   public class HelloWorld{ public static void main(String... args){ System.out.println("Hello, World!"); } } object HelloWorld extends App{ println("Hello, World!") } scala> println("Hello, World!")
21. Expressive,  yet  concise   val x = if(n%2==0) "Even" else "Odd" val x = for(i <- 1 to 10) yield i*2 val x = for(i <- 1 to 10; if(i%2==0)) yield i*2 val y = 20 match{ case p if(p%2==0) => "Even" case _ => "Odd" } val largest = List(1,5,2,6).foldLeft(0)((a,b)=> if(a>b) a else b)
22. Less  pain  points   No  more  dreaded  null  or  throwing  unwanted  excepXons  :   Use  OpXon[T] def thisMayReturnNull[T](x:T):Option[T]={ if(true) Some(x) else None }   No  more  creaXng  classes  only  to  return  more  than  1  value  :   Use  Tuples def doublePair(x:Int)= (x,x*2)
23. Try-­‐catch  mess   try{ //Construct a URL which can throw a MalformedURLException //do something with a URL which can throw IOException //do something with string encoding which can throw // UnsupportedEncodingException } catch{ case mal:MalformedURLException => //something case ioe: IOException => //something case e: Exception => //something }
24. Type  Inference   scala> val x = "Sunday" x: java.lang.String = Sunday scala> def x(n:Int) = n*3 x: (n: Int)Int scala> 1 to 5 res2: scala.collection.immutable.Range.Inclusive = Range(1, 2, 3, 4, 5) scala> def x = if(true) Left("Yes!") else Right(new Exception("Didn’t work out!")) x: Product with Serializable with Either[java.lang.String,java.lang.Exception]
25. Dynamic  Mix-­‐ins   class Home(bedrooms:Int=3,bathrooms:Double=2.5) trait Garage{ def park(n:Int) = println("Parking " + n + "cars") } val someHome = new Home val someHomeWithGarage = new Home with Garage someHomeWithGarage.park(2)
26. Concurrency   •  Actors  make  concurrency  so  much  easier   – Messaging  vs.  Blocking   – Lightweight,  300-­‐500  bytes  per  instance   – Isolated  
27. Scala  API  Components   •  Scalatra   •  Li|-­‐MongoRecord  for  MongoDB     •  Casbah  for  MongoDB   •  PlayFramework  2   •  Actors  for  tasks  like  syndicaXon   •  sbt  and  Maven  for  Builds   •  MongoDB  for  persistence  
28. Other  API  Components   •  Varnish  Cache     •  ElasXcsearch   •  Yammer  Metrics  (ping,  healthcheck)   •  Swagger  (self-­‐documenXng  RESTful  APIs)   •  3Scale  for  Partner  APIs   •  IntelliJ  IDEA  and  Eclipse  with  ScalaIDE   •  PlayFramework  2.0  (New!)   •  ScalaTest  for  TesXng  
29. Challenges  with  Scala*   •  Steepness  of  the  Learning  curve  depends   enXrely  on  your  org  culture   •  Scala  is  what  you  make  of  it   •  hlp://scalaz.github.com/scalaz/scalaz-­‐2.9.1-­‐6.0.4/doc.sxr/scalaz/ BKTree.scala.html   •  Middle  Ground  between  Simplicity  and,   well..elegance    *  None  of  these  challenges  outweigh  the  benefits.  They’re  just  something  to   be  aware  of.  
30. Challenges  with  Scala   •  Slow  compilaXon  based  on  the  source   •  No  (binary)  compaXbility  of  dependencies  for   the  major  releases  (2.8,  2.9,  2.10(?))   •  Tooling  not  as  rich  as  Java     – Changing  (rapidly)  with  Typesafe  invesXng  in   ScalaIDE   – ScalaIDE  comes  with  a  Scala  Worksheet,  which  is   like  REPL  but  richer    
31. Lessons  Learned  :  Culture   •  Culture  of  Meetups,  Hack   Nights  and  Code  Jams   •  Being  able  to  connect  with   the  contributors  and   language  creators   •  Willingness  to  invest  in   learning,  evolving  vs.   maintaining  
32. Lessons  Learned  :  Team   •  Small  team  (~5  engineers)     •  Java  background  helped  a  lot   •  10%  Xme  set  aside  for  skill  building   •  Learn  and  share   •  It  is  a  commitment   •  CommunicaXon!  
33. Lessons  Learned  :  Material   •  Books  that  have  exercises,  so  you   can  quanXfy  your  learning.   •  Online  tutorials   •  StackOverflow     •  Coursera  :  A  course  on  FuncXonal   Programming  by  MarXn  Odersky  
34. Resources   •  Books   –  Programming  in  Scala  by  Odersky   –  Scala  for  the  ImpaXent  by  Horstmann   •  Interwebz   –  hlp://debasishg.blogspot.com   –  hlp://www.reddit.com/r/scala   –  hlp://stackoverflow.com/tags/scala/info   –  hlp://twiler.github.com/scala_school   –  hlps://www.coursera.org/course/progfun  
35. scala> case class Me(name:String="Manish Pandit",twitter:String="@lobster1234") defined class Me scala> val me = new Me me: Me = Me(Manish Pandit,@lobster1234) scala> println(“Questions!”) Questions!