OCaml Platform 2017

An OCaml Workshop talk at ICFP 2017, updating on the state of the OCaml platform tooling.

Transcript

1. State of the
 OCaml Platform David Allsopp, Stephen Dolan,
 Louis Gesbert, Gemma Gordon,
 Anil Madhavapeddy (speaker),
 KC Sivaramakrishnan With contributions from Nicolas Assouad, Frédéric Bour, Daniel Buenzli, Romain Calascibetta, Amir Chaudhry, Enguerrand Decorne, Jeremie Dimino, Thomas Gazagnaire, Louis Gesbert, Armaël Guéneau, Daniel Hillerström, David Kaloper-Meršinjak, Théo Laurent, Qi Li, Jon Ludlam, Thomas Leonard, Maxime Lesourd, Hannes Mehnert, Richard Mortier, Olivier Nicole, Thomas Refis, David Scott, David Sheets, Mark Shinwell, Magnus Skjegstad, Liang Wang, Leo White, Jeremy Yallop,
 and members of the INRIA Galium team, OCamlPro and Jane Street. September 2017
2. tl;dr • OCaml Labs is 5 years old today! • This work is based on a huge collaborative effort • First some perspective on the past five years • Then a status update on 2017's developments • And finally a sneak preview of 2018 work
3. What is the Platform? BUILD TEST EXPLAIN PACKAGE CODE The OCaml Platform combines the OCaml compiler toolchain with a coherent set of tools for build, documentation, testing and IDE integration. The project is a collaborative effort across the OCaml community, tied together by the OCaml Labs group in Cambridge, and OCamlPro in Paris. The requirements of the Platform are guided by large industrial users such as Jane Street, Citrix, Docker, Facebook, Microsoft and LexiFi, as well as accrued feedback from the opam project.
4. BUILD TEST EXPLAIN PACKAGE CODE 2012 2011 2010 2009 <2008 G godi ocamldoc ounit omake ocamlfind mirage oasis The wilderness years 3 hour CUFP "tutorial" Every project an island odb
5. BUILD TEST EXPLAIN PACKAGE CODE 2012 2011 2010 2009 <2008 G 2017 2016 2015 2014 2013 opam 1.0 ocamlot godi ocamldoc ounit RWO merlin omake ocamlfind assemblage datakit-ci crowbar ocaml.org ocamllabs.io irmin ctypes git jbuilder odoc opam 2.0 mirage oasis topkg odig docker codoc OCaml Labs founded A surge of development AFP course Shift to smaller libs
6. BUILD TEST EXPLAIN PACKAGE CODE 2012 2011 2010 2009 <2008 G 2017 2016 2015 2014 2013 opam 1.0 ocamlot godi ocamldoc ounit RWO merlin omake ocamlfind assemblage datakit-ci crowbar ocaml.org ocamllabs.io irmin ctypes git jbuilder odoc opam 2.0 mirage oasis topkg odig docker codoc AFP course Disaggregation & coevolution cmt[i] files ocamlbuild break out components distribute via opam
7. BUILD TEST EXPLAIN PACKAGE CODE 2012 2011 2010 2009 <2008 G 2017 2016 2015 2014 2013 opam 1.0 ocamlot godi ocamldoc ounit RWO merlin omake ocamlfind assemblage datakit-ci crowbar ocaml.org ocamllabs.io irmin ctypes git jbuilder odoc opam 2.0 mirage oasis topkg odig docker codoc AFP course "One in one out" Sustainable ecosystem Consistent interfaces ppx
8. camlp4 -> PPX • PPX are extension points and annotations for OCaml. • Key PPX libraries now hosted in one place: • github.com/ocaml-ppx • more robust to OCaml compiler versions (ocaml-migrate- parsetree) • Getting rid of camlp4 is taking years. • Steady education process required for community. • Migration guides on discuss.ocaml.org would accelerate process. • e.g. pa_ulex -> sedlex CODE
9. Merlin 3.0 • Major Merlin 3.0 developer tool release • Scalable protocol to communicate with IDEs • Robust Windows support. • Now promoted to https://github.com/ocaml/merlin • Community now using it as a standard for IDEs • Visual Studio Code, Atom, Sublime Text • Facebook Reason syntax support also. • More sophisticated short paths algorithm than upstream. CODE
10. Documentation odoc: generate HTML for a group of libraries, with cross referencing. EXPLAIN http://docs.mirage.io $ opam install odoc odig $ opam install mirage # and anything $ odig odoc
11. Documentation odoc: generate HTML for a group of libraries, with cross referencing. Now working on docs.ocaml.org - can we build this at a large scale for every package? EXPLAIN http://docs.mirage.io
12. Documentation A few remaining issues to fully deprecate ocamldoc: porting complex sites like Lwt EXPLAIN https://ocsigen.github.io/lwt/manual-draft/Lwt.html
13. Documentation Ongoing refresh at dev.realworldocaml.org EXPLAIN
14. OPAM 2.0beta PACKAGE • OPAM is the source-based package manager for publishing OCaml code • Focus this year has been on stabilising the upcoming 2.0 • Crossed PR #10000 in opam-repository and PR #3000 in opam!
15. OPAM 2.0beta • OPAM is the source-based package manager for publishing OCaml code • Focus this year has been on stabilising the upcoming 2.0 • Over 7000 packages now managed PACKAGE
16. OPAM 2.0beta • OPAM is the source-based package manager for publishing OCaml code • Focus this year has been on stabilising the upcoming 2.0 • Over 600 individual contributors to the repository PACKAGE
17. OPAM 2.0beta • New features in recent betas: • more expressive package dependencies depends: [ "foo" {>= "3.0" & < "4.0~" & os = "linux"} ] depends: [ "ocaml" >= "4.05.0" ] PACKAGE
18. OPAM 2.0beta • New features in recent betas: • more expressive package dependencies depends: [ "foo" {>= "3.0" & < "4.0~" & os = "linux"} ] depends: [ "ocaml" >= "4.05.0" ] depends: [ ... "datakit-server" {>= "0.9.0"} "datakit-client" {with-test & >= "0.9.0"} "datakit-github" {with-test & >= "0.9.0"} "alcotest" {with-test & >= "0.7.0"} ] PACKAGE
19. OPAM 2.0beta • New features in recent betas: • more expressive package dependencies • computed versions to make multiple packages easier depends: [ ... "datakit-server" {>= version} "datakit-client" {with-test & >= version} "datakit-github" {with-test & >= version} "alcotest" {with-test & >= version} ] Simplifies the package repository by making it easier to have many small libraries PACKAGE
20. OPAM 2.0beta • New features in recent betas: • more expressive package dependencies • computed versions to make multiple packages easier • local switches for use per-project cd ~/src/project opam switch create ./ ... opam list PACKAGE
21. OPAM 2.0beta • New features in recent betas: • more expressive package dependencies • computed versions to make multiple packages easier • local switches for use per-project • Solver integrated as a library, no external solver needed! PACKAGE
22. OPAM 2.0beta • New features in recent betas: • more expressive package dependencies • computed versions to make multiple packages easier • local switches for use per-project • Solver integrated as a library, no external solver needed! • Windows support is being upstreamed! PACKAGE
23. OPAM 2.0beta • New features in recent betas: • more expressive package dependencies • computed versions to make multiple packages easier • local switches for use per-project • Solver integrated as a library, no external solver needed! • Windows support is being upstreamed! PACKAGE But how do we get a usable package repository? Need a simpler build system Luckily 2017 has been kind to build...
24. Jbuilder A build system specialised to real world OCaml code. Provide a description of your project, and it will be built! BUILD
25. Jbuilder A build system specialised to real world OCaml code. Provide a description of your project, and it will be built! • Compose multiple checkouts in subdirs and it can be built in one pass • Integrates with opam to locate external packages • Multiple workspaces to support different OCaml versions or build options (e.g. afl or flambda) • Declarative model encourages portable build rules, so it often "just works" on Windows. • Fast. Really fast. BUILD
26. OPAM 2.0 release Why isn't it released yet? I want these cool features!
27. Continuous Integration • 2017 has been a good year: big releases to opam, significantly improved CI to submitted packages • Automated infrastructure is very efficient vs engineering time, so scripting everything we can! • Currently expanding to meet new needs: benchmarking, fuzzing, and portability • Also need to automate the opam1 -> opam2 transition Please join the opam maintenance team! TEST
28. BUILD TEST EXPLAIN PACKAGE CODE 2012 2011 2010 2009 <2008 G 2017 2016 2015 2014 2013 opam 1.0 ocamlot godi ocamldoc ounit RWO merlin omake ocamlfind assemblage datakit-ci crowbar ocaml.org ocamllabs.io irmin ctypes git jbuilder odoc opam 2.0 mirage oasis topkg odig docker codoc AFP course "One in one out" Sustainable ecosystem Consistent interfaces ppx
29. jbuilder crowbar odoc opam2 merlin visual studio travis github odig topkg
30. jbuilder crowbar odoc opam2 merlin visual studio travis github odig topkg
31. BUILD TEST EXPLAIN PACKAGE CODE 2017 2016 2015 2014 2013 2022 2021 2020 2019 2018 merlin irmin ctypes git assemblage jbuilder ocamlot datakit-ci crowbar docker RWO ocaml.org ocamllabs.io odoc odig codoc opam 1.0 opam 2.0 topkg codename: bob Work in Progress
32. Some common cargo commands are (see all commands with --list): build Compile the current project check Analyze the current project and report errors clean Remove the target directory doc Build this project's and its dependencies' docs new Create a new cargo project init Create a new cargo project in an existing directory run Build and execute src/main.rs test Run the tests bench Run the benchmarks update Update dependencies listed in Cargo.lock search Search registry for crates publish Package and upload this project to the registry install Install a Rust binary an aside: cargo/rust Single binary, gives access to full Rust ecosystem Can we adopt same approach for Platform?
33. Some common bob commands are (see all commands with --list): build Compile the current project check Analyze the current project and report errors clean Remove the target directory doc Build this project's and its dependencies' docs new Create a new OCaml project init Create a new OCaml project in an existing directory run Build and execute src/main.ml test Run the tests fuzz Run the fuzzer compat Check compatibility across different operating systems bench Run the benchmarks update Update dependencies listed in opam.lock search Search registry for opam packages publish Package and upload this project to the opam registry install Install an OCaml binary codename: bob
34. bob • Single CLI tool for all OCaml platform activities. • Includes jbuilder, odoc, merlin, opam, alcotest, crowbar, topkg with a single CLI • Sensible defaults, with escape hatches for advanced users • Access to offline/online infrastructure: • dedicated continuous integration access • consistent OS support for Linux, Windows, macOS • search online community (odoc, opam, github) Will be released in 2018 as the first OCaml Platform release, with associated tools bundled
35. bob • Bob is not yet complete or even fully designed • We are starting an open design process this year, as we release opam 2.0 • opam maintenance window is around ~3-5 years, so want to get this revision right
36. Online Community ocaml.org opam.ocaml.org realworldocaml.org lists.ocaml.org discuss.ocaml.org ci.ocaml.io docs.ocaml.org github.com/ocaml Unify the design of these sites, with ocaml.org as the main public site Create searchable API for all online OCaml resources - used by bob search
37. What is the Platform? BUILD TEST EXPLAIN PACKAGE CODE The OCaml Platform combines the OCaml compiler toolchain with a coherent set of tools for build, documentation, testing and IDE integration. The project is a collaborative effort across the OCaml community, tied together by the OCaml Labs group in Cambridge, and OCamlPro in Paris, and the whole OCaml community. See you on discuss.ocaml.org :)