Maintaining Fortran in Python in Perpetuity

Presented at SciPy 2022.

Details here: https://rgoswami.me/posts/scipycon-2022-meta

This talk covers recent advances to the f2py project, including preliminary derived type support, better documentation, and restructured design to enhance new contributor experiences. Additionally, we will discuss the structure and role of interoperability libraries and how modern Fortran intends to harmoniously and symbiotically evolve within the Python ecosystem in the context of Fortran-Lang, the np.distutils deprecation, newer compute devices and the ever changing landscape of high performance programming.

Transcript

1. MAINTAINING FORTRAN IN PYTHON IN PERPETUITY ROHIT GOSWAMI .and. MELISSA MENDONCA .and. RALF GOMMERS .and. THIRUMALAI SHAKTIVEL .and. PEARU PETERSON
2. BRIEF INTRODUCTION 3
3. HELLO! Find me here: Who? Rohit Goswami MInstP Doctoral Researcher, University of Iceland, Faculty of Physical Sciences Software Engineer, Quansight Labs, Austin TX https://rgoswami.me 4
4. LOGISTICS All contents are Slides are in presentations/scipy22/F2PYmaint hosted on GitHub Questions are welcome at the end, or interrupt me 5
5. PROGRAMMING LANGUAGES 7
6. MOTIVATION –> Through the magic of automated coding and standards “If a program or package (the words are used interchangeably) is to have a long life and to be of wide application in its field, it is essential for it to be easily moved from one machine to another. It used to be common to dismiss such movement with the statement, ‘There is no such thing as a machine- independent program.’ Nonetheless, a great many packages do now move from one machine to another”lyonUsingAnsFortran1980 8
7. LANGUAGE STANDARDS “The standard is the contract between the compiler writer and the application developer.”clermanModernFortranStyle2012 9
8. CHANGING STANDARDS character(10) BLAH*8 character*8 :: BLAH_ONE(10) character(8) :: BLAH_ONE(10) #!/usr/bin/env python print("Hello World") print "Hello World" 10
9. FORTRAN, C, PYTHON F2003 Introduced the ISO_C_BINDING F2008 C_PTR for void * and more F2018 Brought interop for exotic Fortran features which via C descriptors Interop Described in great detail on : fortran90.org 12
10. F2PY 14
11. HISTORY Developed by Pearu Peterson July 9, 1999 f2py.py –> Fortran to Python Interface Generator (FPIG) January 22, 2000 f2py2e –> Fortran to Python Interface Generator, 2nd edition. July 19, 2007 numpy.f2py –> f2py2e moved to NumPy project. This is current stable code of f2py. Used extensively for F77 , :) petersonF2PYToolConnecting2009 NumPy waltNumPyArrayStructure2011 SciPy virtanenSciPyFundamentalAlgorithms2020 MsSpec sebilleauMsSpec1MultipleScattering2011 15
12. DESIGN A best effort wrapper Specifications via .pyf or inline comments Not a compiler Can rewrite code :) 16
13. EXPLORATIONS IN F77 18
14. FIBONACCI C FILE: FIB1.F SUBROUTINE FIB(A,N) C CALCULATE FIRST N FIBONACCI NUMBERS INTEGER N REAL*8 A(N) DO I=1,N IF (I.EQ.1) THEN A(I) = 0.0D0 ELSEIF (I.EQ.2) THEN A(I) = 1.0D0 ELSE A(I) = A(I-1) + A(I-2) ENDIF ENDDO END C END FILE FIB1.F f2py -m fib -c fib1.f python -c "import fib; import numpy as np; a=np.zeros(7); fib.fib(a); print(a); exit();" 19
15. UP THE MAGICIAN’S SLEEVE Generated files mkdir blah f2py -m fib -c fib1.f --build-dir blah tree blah blah ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── blah │ │ └── src.macosx-10.9-x86_64-3.9 │ │ ├── fortranobject.o │ │ └── fortranobject.o.d │ ├── fibmodule.o │ └── fibmodule.o.d ├── fib1.o └── src.macosx-10.9-x86_64-3.9 ├── blah │ └── src.macosx-10.9-x86_64-3.9 │ ├── fortranobject.c │ └── fortranobject.h └── fibmodule.c 7 directories, 8 files 20
16. COMPLEXITY NumPy Distutils Which can then be built simply with: Not fun for non python projects wc -l fortranobject.c fortranobject.h fibmodule.c 1107 fortranobject.c 132 fortranobject.h 372 fibmodule.c 1611 total from numpy.distutils.core import Extension, setup fibby = Extension(name = 'fib', sources = ['fib1.f']) if __name__ == "__main__": setup(name = 'fib', ext_modules = [ fibby ]) python setup.py build ag -g .so # build/lib.macosx-10.9-x86_64-3.9/fib.cpython-39-darwin.so 21
17. (SOME) PLANNED FEATURES 23
18. MESON AND f2py project('test_builds', 'c', version : '0.1') add_languages('fortran') py_mod = import('python') py3 = py_mod.find_installation() py3_dep = py3.dependency() incnp = run_command(py3, ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'], check : true ).stdout().strip() inc_np = include_directories(incnp) py3.extension_module('fib1', 'fib1.f', 'fib1module.c', 'fortranobject.c', include_directories: inc_np dependencies : py3_dep, install : true) 24
19. COMMAND LINE INTERFACE The F2PY UX is rather distinctive -c flag acts as a compiler / build-tool Otherwise works to generate C wrappers and Signature files Rewrite ongoing [GSoC] Supports Namami Shankar argparse based design 25
20. DERIVED TYPES Soon to be a NEP : Tested by cmocka e.g. bind(c) types https://github.com/HaoZeke/f2py_derived_nep map to classes meson setup bbdir meson compile -C bbdir python -c "import bbdir.pycart as pycart; aak = pycart.pycart(1,10,2); print(aak); aak.unitstep(); print(aak)" pycart(x: 1.000000, y: 10.000000, z: 2.000000) Modifying derived type pycart(x: 2.000000, y: 11.000000, z: 3.000000) 26
21. MOVING BEYOND bind(c) Generate Fortran wrappers to manipulate types All allocations handled in Fortran Python only calls wrappers to aforementioned types 27
22. LOGICAL RESTRUCTURE Current flat hierarchy –> intimidating Don’t need much Fortran for front-end work codegen Creates wrappers csrcs fortranobject.{c,h} for builds frontend Parser, type inference stds Includes f77, f90, and pyf specs utils For testing and more tests For sanity (no term) WIP : https://github.com/numpy/numpy/pull/20481 28
23. MAINTENANCE 30
24. CONCEPTS Fortran 202X standard 668 pages Python-C API description 320 pages NumPy-C API description 103 pages C++ 2020 (N4849) standard 1815 pages C11 (N1570) standard 701 pages 31
25. TEST ISOLATION Write a test for each bug Typically already in the issue! class TestNegativeBounds(util.F2PyTest): # Check that negative bounds work correctly sources = [util.getpath("tests", "src", "negative_bounds", "issue_20853.f90")] @pytest.mark.slow def test_negbound(self): xvec = np.arange(12) xlow = -6 xhigh = 4 # Calculate the upper bound, # Keeping the 1 index in mind def ubound(xl, xh): return xh - xl + 1 rval = self.module.foo(is_=xlow, ie_=xhigh, arr=xvec[:ubound(xlow, xhigh)]) expval = np.arange(11, dtype = np.float32) assert np.allclose(rval, expval) 32
26. FRONT-END Covers (parser, code-gen) pdb and editable installs are good choices Tough otherwise (global variables) micromamba create -f environment.yml micromamba activate numpy-dev pip install -e . # EDITABLE MODE # Add a breakpoint anywhere breakpoint() # Profit f2py -m blah buggy.f90 e.g. –> Negative bounds https://github.com/numpy/numpy/pull/21256/files 33
27. BACK-END Covers (C wrappers, code-gen) gdb works well Sometimes with CPython extensions (see the sprint) Manipulate the generated C code Then move back to code-gen in Python e.g. –> Respect value https://github.com/numpy/numpy/pull/21807/files 34
28. SUPPORTING value ATTRIBUTES Fortran 2003 supports call-by-value f2py wrappers pass by reference Details Very well reported issue here 35
29. VERIFY BUG # Make wrappers f2py -m foo blah.f90 meson bbdir meson compile -C bbdir cd bbdir python -c "import foo; print( foo.fortfuncs.square(3) );" 170676880 static PyObject *f2py_rout_foo_fortfun const PyObject *capi_self, PyObject *capi_args, PyObject *capi_keywds, void (*f2py_func)(int*,int*)) { (*f2py_func)(&x,&y); // Passed by refe 36
30. FIXUP C WRAPPER # Make wrappers meson compile -C bbdir cd bbdir python -c "import foo; print( foo.fortfuncs.square(3) );" 9 static PyObject *f2py_rout_foo_fortfun const PyObject *capi_self, PyObject *capi_args, PyObject *capi_keywds, void (*f2py_func)(int,int*)) { (*f2py_func)(x,&y); 37
31. MODIFY PARSER value should be recognized by crackfortran Add a breakpoint and stare So it is already recognized.. name_match = re.compile(r'[A-Za-z][\w$]*').match breakpoint() for v in list(vars.keys()): >>> c # till fortfuncs is processed >>> pp vars {'x': {'attrspec': ['intent(in)', 'value'], 'typespec': 'integer'}, 'y': {'attrspec': ['intent(out)'], 'typespec': 'integer'}} 38
32. AUGMENT AUX FUNCTIONS Handle the attribute within aux.py Also conditionally modify signatures + call conventions def isattr_value(var): return 'value' in var.get('attrspec', []) def getcallprotoargument(rout, cb_map={}): ... if not isattr_value(var): ctype = ctype + '*' 39
33. GENERATE WRAPPERS This is similar to the handling of the existing intent(c) In rules.py 'callfortran': {l_or(isintent_c, isattr_value): '#varname#,', l_not(l_or(isintent_c, isattr_value)): '&#varname#,'}, 40
34. FINAL TASKS Add a test Solicit reviews Ask about a release note Profit! 41
35. CONCLUSIONS 43
36. ROAD-MAP Updating the test suite, using cmocka Rewriting the C wrappers for newer standards Build tool support [Namami Shankar] np.distutils is going the way of the dodo Implementing newer standards (90, 95, 2003, 2008, 2018, 2020Y) Automating guarantees Documentation and more interop with NumPy-C crackfortran works via dictionaries and strings.. Perhaps a more abstract semantic representation… Or concepts from G3 F2PY [Pearu] Handle parallelism (proposed) do concurrent and ufuncs 44
37. WHY DOES FORTRAN STAY IN PYTHON ANYWAY? To write efficient wrappers without being a language lawyer 45
38. HOW DOES FORTRAN STAY IN PYTHON ANYWAY? With all your help (Issues, users, developers, non-code contribs) 46
39. THE END 48
40. ACKNOWLEDGMENTS as my supervisor, as my co-supervisor at GSI Tech. ( , and ) NumPy Team (Matti, Ross, Sebastian, Chuck, Inessa, etc.) Family, pets, Groupmembers, audience Prof. Hannes Jónsson Prof. Birgir Hrafnkelsson Dr. Ondřej Čertík Quansight Labs Dr. Ralf Gommers Dr. Melissa Weber Mendonça Dr. Pearu Peterson 49
41. THANKS! 50