Migrating from Julia 0.4 to 1.6

When Julia 1.0 was released several years ago, we were faced with the challenge of migrating over our large Julia 0.4 codebase while making sure our customers didn't notice a thing. This talk is a story about reassessing life choices, keeping customers happy, and a quick build, slow release, 5 year, 40 step migration from 0.4 to 1.6. We hope to never have to do this again!

We started our Julia journey with version 0.2 over 10 years ago, and upgraded fairly quickly up to 0.4.7 as new versions were released. Then we started getting more customers, and our APIs stabilised, and migrations became expensive, so we sat at 0.4.7, watching every quarter as various dependencies reached the end of their supported lives.

Given that migrations were expensive, we knew that whenever we migrated, it we'd have to make it count for many years to come. We started migrating our code to 1.0, deciding to skip all the advice about migrating to 0.7 first. While this was happening, Julia moved to 1.1, 1.2, 1.3, 1.4, and 1.5, so we decided to jump to 1.5 instead, and then Julia released 1.6.

Over the course of the migration we developed a series of 40 steps that allowed us to change the underlying library while requiring only minimal changes to our customers' code.

In this talk, we will quickly go over some of the more surprising changes we've had to make. We will not discuss plans for 1.10 or 1.11.

Transcript

1. Philip Tellis Principal RUM Distiller @ Akamai @bluesmoon Julia since v0.2
2. ‘ T A L K Migrating Julia 0.4 → 1.6
3. 2014 Added Julia to our Product - version 0.2 Later that year, migrated to 0.3 Even 0.3 is better than MatLab
4. 2015 Attended my first JuliaCon (@MIT) Migrated to 0.4 I thought, wow this is a piece of cake! (better than Python)
5. 2016-18 Codebase grew a lot, Customer base has grew as well (change averse), julia changes too fast (0.5, 0.6, 0.7, 1.0)
6. 2019 1.0 becomes LTS, so we decide to migrate straight from 0.4 to 1.0
7. This took 2 years!
8. LTS = 1.6
9. LTS = 1.10
10. Timeline • 2014: Added Advanced Analytics to our product with Julia 0.2 • 2015: Migrated to Julia 0.4 - This was easy • 2019: Started planning migration to Julia 1.0 (LTS release at the time) ◦ Julia had changed a lot in this time ◦ Our code base had also grown significantly ◦ Our customer base had also grown and had varying levels of change management • 2020-2023: Migrated from Julia 0.4 to Julia 1.0 1.6 (which is no longer LTS!!) • Now: planning our migration to 1.10 - It will be easier this time
11. 1. type is now struct and immutable by default 2. ismatch is replaced by occursin 3. ucfirst and lcfirst are replaced by uppercasefirst and lowercasefirst 4. find* returns nothing instead of 0 for the failure case 5. Void is now Nothing built-ins
12. 6. shift! is now called popfirst! and unshift! is called pushfirst! 7. replace(str, pat, sub) is now replace(str, pat => sub) 8. reduce, mapreduce, mapfoldl, and mapfoldr initial state is now a kwarg init 9. Number of digits for round is now a kwarg digits. 10. round needs to be broadcast for arrays built-ins
13. 11. tic, toc, and toq have been removed 12. Calling filter(f, ::Dict), signature of f changed from (k, v) to k => v 13. findin no longer exists. indexin may be used though not exactly, or, findall(in(needles), haystack) 14. indmin and indmax no longer exist, use findmin and findmax built-ins
14. 15. STDIN, STDOUT, and STDERR are now stdin, stdout, and stderr 16. map is no longer defined on Dicts 17. The keep kwarg of split is now called keepempty 18. readall has been replaced with read 19. findfirst, findnext, findprev, and findlast on strings, the arguments are swapped. built-ins
15. 20. Logging functions() are now @macros 21. fieldnames only works on types and not on objects 22. obj.(sym) doesn’t work, but getfield(obj, sym) does 23. Template syntax xyz {T <: Foo}(x::T) changed to xyz(x::T) where {T <: Foo} 24. Second argument to invoke changed from tuple() to Tuple{} built-ins
16. 25. sort on DataFrames takes cols as the second positional argument rather than a kwarg 26. eltypes(df) change to eltype.(eachcol(df)) 27. df[:colname] changed to df[!, :colname] 28. names returns strings rather than Symbols 29. NA::NAType replaced with missing::Missing DataFrames
17. 30. join split to semijoin, leftjoin, antijoin, rightjoin, outerjoin, innerjoin, crossjoin 31. by changed to groupby |> combine 32. names!(df, [new, names]) deprecated. Use rename! 33. Use CSV.write instead of writetable 34. isreadable no longer works on filenames, use isfile DataFrames
18. 35. So much better! @test_logs, @test_throws are awesome! 36. Julia 0.4 had JLD but 1.6+ has JLD2 which at the time could not read JLD files. Had to write a bash based migration script. Testing
19. The Migration • We first rewrote all our base packages to use more modern Julia practices • We then wrote compat shims where possible to help our customers migrate • The majority of the code could be fixed with a straight search & replace • For a short period of time we had a REQUIRES file as well as Project/Manifest.toml • The hardest part was migrating customer code because none of them had allocated any time for this migration. I had to do a lot of it myself.
20. 1. Varargs are now of type Vararg which is not <: of Type 2. Changes needed with @test_logs because log messages have different amounts of whitespace Going to 1.10
21. Thank You! Happy Migrating!