The Evolution of the CRuby Build System

Transcript

1. The Evolution of the CRuby Build System Yuta Saito (@kateinoigakukun) RubyKaigi 2025 1
2. About me • Yuta Saito / @kateinoigakukun • Software Engineer at Goodnotes • Committer of CRuby and Swift • Creator of ruby.wasm 2
3. Overview 1. Introduction 2. Measuring Build System 3. Walkthrough of Current Build Process 4. Challenges Today 5. Towards a Better Build System 6. Summary 3
4. 1. Introduction 4
5. 1.1. Motivation • Working on ruby.wasm for 3 years • Found many challenges in the build system • Want to make it better for everyone 5
6. 1.2. How do you usually build CRuby? Most users install Ruby through version managers: - rbenv - rvm - chruby - asdf 6
7. 1.3. Build from tarball All stable versions are released as tarballs on ruby-lang.org. $ curl "https://cache.ruby-lang.org/pub/ruby/3.4/ruby-3.4.2.tar.gz" | \ tar xz --strip-components=1 $ ./configure $ make $ make install On Windows $ ./win32/configure.bat $ nmake 7
8. 1.4. Another Way: Build from git source • Requires that ruby is available on the build machine. (baseruby) • ./autogen.sh is required to generate configure $ git clone https://github.com/ruby/ruby.git $ cd ruby $ ./autogen.sh $ ./configure $ make $ make install 8
9. 1.5. Difference between tarball and git source • Tarball must not depend on baseruby to avoid bootstrap problem • Some autogenerated files are included: • configure • tool/config.{sub,guess} • parse.c • *.rbinc • enc/trans/*.c • lib/prism/*.rb • gems/*.gem • revision.h • Changelog 9
10. 2. Measuring Build System 10
11. 2.1. How to analyze build? GNU Make allows us to choose the shell other than /bin/sh. # Record trace info in .trace-make/*.trace during build $ make SHELL="gem exec tracemake shell" # Emit aggregated trace info for chrome://tracing $ gem exec tracemake aggregate -o make-trace.json 11
13. 2.2. Data collection across Ruby versions Used all-ruby project by @akr • A collection of build scripts and patches • Allows building multiple Ruby versions with consistent settings • Applied a few modifications to collect build time data 13
14. Version Time (seconds) 0 20 40 60 80 0.49 0.65 1.0-971003 1.1a1 1.1b0 1.1b9 1.1b9_09 1.1b9_18 1.1b9_27 1.1c3 1.2 1.3.1-990224 1.3.3-990430 1.3.6 1.6.0 1.8.0 1.8.3- 1.8.5- 1.8.5-p113 1.8.6-p110 1.8.6-p388 1.8.7-p17 1.8.7-p299 1.8.7-p371 1.9.1- 1.9.1-p429 1.9.2-p180 1.9.3-p125 1.9.3-p429 2.0.0-rc1 2.0.0-p594 2.1.0 2.1.9 2.2.4 2.3.0 2.4.0- 2.4.5 2.5.2 2.6.0- 2.6.5 2.7.0-rc1 2.7.7 3.0.4 3.1.4 3.2.2 3.3.0-rc1 3.4.0- Times for './configure' Across Versions 14
15. version Lines 0 2000 4000 6000 0.49 0.65 1.0-971003 1.1a1 1.1b0 1.1b9 1.1b9_09 1.1b9_18 1.1b9_27 1.1c3 1.2 1.3.1-990224 1.3.3-990430 1.3.6 1.6.0 1.8.0 1.8.3-preview3 1.8.5-preview4 1.8.5-p113 1.8.6-p110 1.8.6-p388 1.8.7-p17 1.8.7-p299 1.8.7-p371 1.9.1-preview1 1.9.1-p429 1.9.2-p180 1.9.3-p125 1.9.3-p429 2.0.0-rc1 2.0.0-p594 2.1.0 2.1.9 2.2.4 2.3.0 2.4.0-preview1 2.4.5 2.5.2 2.6.0-preview2 2.6.5 2.7.0-rc1 2.7.7 3.0.4 3.1.4 3.2.2 3.3.0-rc1 3.4.0-preview1 LoC of ./configure.{in,ac} ./tool/m4 15
16. $ git shortlog -ns configure.ac tool/m4 | head -n10 301 Nobuyoshi Nakada 59 ๼෦ণฏ 36 Takashi Kokubun 24 Yusuke Endoh 24 Yuta Saito 20 Samuel Williams 18 Jeremy Evans 16 NARUSE, Yui 15 Masaki Matsushita 12 Alan Wu 16
17. Version Time (seconds) 0 100 200 300 400 0.49 0.65 1.0-971003 1.1a1 1.1b0 1.1b9 1.1b9_09 1.1b9_18 1.1b9_27 1.1c3 1.2 1.3.1-990224 1.3.3-990430 1.3.6 1.6.0 1.8.0 1.8.3-preview3 1.8.5-preview4 1.8.5-p113 1.8.6-p110 1.8.6-p388 1.8.7-p17 1.8.7-p299 1.8.7-p371 1.9.1-preview1 1.9.1-p429 1.9.2-p180 1.9.3-p125 1.9.3-p429 2.0.0-rc1 2.0.0-p594 2.1.0 2.1.9 2.2.4 2.3.0 2.4.0-preview1 2.4.5 2.5.2 2.6.0-preview2 2.6.5 2.7.0-rc1 2.7.7 3.0.4 3.1.4 3.2.2 3.3.0-rc1 3.4.0-preview1 Times for 'make -j1' Across Versions 17
18. 3. Walkthrough of Current Build Process 18
19. 3.1. Supported "make" • gmake (GNU Make) • The latest version is 4.4 • Need to support 3.81 because it's still the default on macOS1 • bmake (BSD make) • nmake (Windows) 1 My first commit to CRuby was about fixing old gmake build d1f0d1ca2ea4d7418b096ce71f68ce2bb3afd2c4 19
20. 3.2. ./configure Unix-y platforms ./configure configure.ac ./autogen.sh (autoreconf) ruby/config.h template/Makefile.in Makefile GNUmakefile (only for gmake) template/GNUmakefile.in common.mk uncommon.mk Windows Makefile configure.bat 20
21. 3.3. common.mk / uncommon.mk • common.mk is included as is by nmake • For non-nmake makes, pre-processed to strip nmake-specific path inference syntax # common.mk array.$(OBJEXT): {$(VPATH)}array.c # Generated uncommon.mk array.$(OBJEXT): array.c 21
22. 3.4. Makefile includes (for gmake) GNUmakefile Makefile uncommon.mk defs/gmake.mk yjit/yjit.mk 22
23. 3.5. Makefile includes (for nmake / Windows) Makefile win32/Makefile.sub win32/setup.mak common.mk 23
24. 3.6. Makefile (for other make) For makes other than gmake and nmake, ./configure appends the following after Makefile instantiated from template/Makefile.in: • common.mk • yjit/not_gmake.mk 24
25. 3.7. make 25
26. 3.8. Rubies during "make" Name Description $(BASERUBY) The Ruby from your system ./miniruby A subset of ruby compiled for the target. Extension libraries are unavailable. $(MINIRUBY) ./miniruby if not cross-compiling. $(BASERUBY) if cross-compiling. $(BOOTSTRAPRUBY) $(BASERUBY) if available, otherwise $ (MINIRUBY) $(XRUBY) Executable(?) ruby without including ./lib 26
27. 3.9. Uses of $(BASERUBY) • .rb -> .rbinc • erb templates • parse.y -> parse.c (by Lrama) {$(srcdir)}.rb.rbinc: $(ECHO) making $@ $(Q) $(BASERUBY) $(tooldir)/mk_builtin_loader.rb $< id.c: $(tooldir)/generic_erb.rb $(srcdir)/template/id.c.tmpl $(srcdir)/defs/id.def $(ECHO) generating $@ $(Q) $(BASERUBY) $(tooldir)/generic_erb.rb --output=$@ \ $(srcdir)/template/id.c.tmpl 27
28. 3.10. Uses of $(MINIRUBY) Configure extensions ext/json/exts.mk: FORCE $(Q)$(MINIRUBY) $(srcdir)/ext/extmk.rb --make='$(MAKE)' \ --command-output=$@ $(EXTMK_ARGS) --extstatic $(EXTSTATIC) \ -- configure $(@D) 28
29. 3.12. Build Stages with Recursive make root Makefile ext/*/Makefile ext/exts.mk ext/configure-ext.mk root Makefile Compile interpreter Configure extensions Compile extensions Compile each extension Link interpreter with extensions Developer make make -f ext/configure-ext.mk make -f ext/exts.mk make -f ext/*/Makefile make ruby EXTOBJS=<.o files> 29
30. 4. Challenges 30
31. 4.1 Many build variants • Platforms • In-tree build vs out-of-source build • Dynamic libruby vs static libruby • Availability of baseruby Handled in both ./configure and make stages 31
32. 4.2. Make-time dynamism Debugging build issue is hard due to make-time dynamism. • yes-${TARGET} / no-${TARGET} pattern • if directive on Makefile • etc 32
33. 4.2. Make-time dynamism yes-${TARGET} / no-${TARGET} pattern Conditionally enable specific targets test-basic: $(TEST_RUNNABLE)-test-basic no-test-basic: PHONY yes-test-basic: prog PHONY # Run basictest 33
34. 4.2. Make-time dynamism if directive on Makefile $(srcdir)/missing/des_tables.c: $(srcdir)/missing/crypt.c ifeq ($(if $(filter yes,$(CROSS_COMPILING)),,$(CC)),) touch $@ else @$(ECHO) building make_des_table $(CC) $(INCFLAGS) $(CPPFLAGS) -DDUMP $(EXE_LDFLAGS) $(XLDFLAGS) $(LIBS) -omake_des_table $(srcdir)/missing/crypt.c $(Q) $(MAKEDIRS) $(@D) $(Q) ./make_des_table > $@.new $(Q) mv $@.new $@ $(Q) $(RMALL) make_des_table* endif 34
35. 4.2. Make-time dynamism Debugging build issue is hard due to make-time dynamism. • yes-${TARGET} / no-${TARGET} pattern • if directive on Makefile • etc The dynamism comes from the limited expressivity of configure.ac 35
36. 4.3. Unintentional dependency on baseruby • When baseruby is available, it's used even for tarball builds • This creates subtle variations in the build graph depending on baseruby availability • These variations have caused post-release issues • Example: https://bugs.ruby-lang.org/issues/20687 • Tarball builds should never depend on baseruby • Need to be able to "validate" the build plan 36
37. 4.4. Long build time • Long ./configure • Long sequential shell script • 60% of time is spent for feature detection • Most of them can be parallelized • Long make • Long idle time 37
38. 4.4. Long build time Recursive make for extension libraries makes scheduling poor 38
39. 5. Towards a Better Build System 39
40. 5.1. How to make it simpler? • Make-time dynamism → Describe the dynamic build planning logic in Ruby instead of M4 • Unintentional dependency on baseruby → Validate the build plan 40
41. 5.2. How to improve build time? • Long ./configure → Parallelize by integrating it with the build system • Long make → Stop stage-wise build to minimize idle-time by expressive build language 41
42. New Build System (Work in progress) 42
43. Vision • Construct build graph by Ruby • Build system as a library • No Ruby dependency for end users 43
44. $ cat manifest.rb Rk2::BuildSystem.main do |b| cc = b.find_executable(%w[cc clang gcc], env: "CC") obj = b.output("example.o") src = b.root.join("example.c") b.add_task(Rk2::CCompile.new(obj, cc, src)) b.add_task(Rk2::CLink.new(b.output("example"), [obj])) end $ rk2 build manifest.rb example 44
45. Rk2::BuildSystem.main do |b| cflags = b.env("CFLAGS") cflags = cflags + ["-I", b.root, "-I", b.root.join("include")] cc = b.find_executable(%w[cc clang gcc], env: "CC") have_checks = HEADERS_TO_CHECK.map do |header| b.add_task(HaveHeader.new(header)) end config_h = b.erb(b.output("config.h"), <<~ERB, have_checks: have_checks) <% have_checks.each do |have_check| %> <= b.select(have_check, "#define HAVE_<%= have_check.upcase %> 1", "") %> <% end %> ERB c_objs = INTERPRETER_C_SOURCE_FILES.map do |src| src_path = b.root.join(src) out_path = b.output(src + ".o") b.add_task(CCompile.new(out_path, cc, src_path, cflags)) end miniruby = b.add_task(Rk2::CLink.new(b.output("miniruby"), c_objs)) b.aggregate(:default, [miniruby]) end 45
46. Key Features 1. Lowering to autotools-style $ rk2 portablize ./manifest.rb --target=autotools -o ./configure $ ./configure $ make 2. Build system as a library graph = Rk2.load_manifest("./manifest.rb") has_baseruby = graph.plan("all").live_nodes.any? do |node| node.is_a?(Rk2::GetEnv) && node.env_key == "BASERUBY" end 46
47. TODO A lot of things to do... • Name of the project • Documentation • Windows support in erb2sh and rk2 portablize • make-time options (e.g. make test-all TESTS=...) 47
48. 6. Summary • CRuby build system has a long history • Walkthrough of current build process • Review of challenges • Work in progress build system • Let's make CRuby build system better together! 48