Share
Embed
Software Portability
Given at the Open Science Grid User School 2016
https://twiki.opensciencegrid.org/bin/view/Education/OSGUserSchool2016
Christina Koch
July 27, 2016
Transcript
36 slides
-
Backpacking with Code: Software Portability for DHTC Wednesday morning, 9:00
am Christina Koch ([email protected]) Research Computing Facilitator University of Wisconsin - Madison -
OSG User School 2016 Goals for this session • Understand
the basics of... - how software works - where software is installed - how software is accessed and run • ...and the implications for DHTC • Describe what it means to make software “portable” • Learn about and use two software portability techniques: - Run compiled code - Build installation and use it in jobs 2 -
Motivation running a piece of software is like cooking a
meal in a kitchen 3 -
OSG User School 2016 The problem Running software on your
own computer = cooking in your own kitchen 4 -
OSG User School 2016 The problem In your own kitchen:
• You have all the pots and pans you need • You know where everything is • You have access to all the cupboards On your own computer: • The software is installed, you know where it is, and you can access it. 5 -
OSG User School 2016 The problem 6 Running on a
shared computer = cooking in someone else’s kitchen. -
OSG User School 2016 The problem In someone else’s kitchen:
• You are guaranteed some things… • …but others may be missing • You don’t know where everything is • Some of the cupboards are locked On a shared computer: • Your software may be missing, un- findable, or inaccessible. 7 -
OSG User School 2016 The solution • Think like a
backpacker • Take your software with you - Install anywhere - Run anywhere • This is called making software portable 8 -
OSG User School 2016 Software • How do we make
software portable? • First we have to understand: - What software is and how it works - Where software lives - How we run it 9 -
OSG User School 2016 How software works • A software
program can be thought of as a list of instructions or tasks that can be run on an computer • A launched program that is running on your computer is managed by your computer’s operating system (OS) • The program may make requests (access this network via wireless, save to disk, use another processor) that are mediated by the OS • A single program may also depend on other programs besides the OS 10 -
OSG User School 2016 How software works* 11 Program (software,
code, executable, binary) Running Program (process, instance) Hardware (processors, memory, disk) Operating System runs own tasks makes requests launches to translates program’s request monitors running programs depends on *Not to scale -
OSG User School 2016 How software works Implications for DHTC:
• Software must be able to run on target operating system (usually Linux) • Request specific OS as job requirement • Know what else your software depends on 12 -
OSG User School 2016 Location, location, location • Where can
software be installed? 13 / bin usr lib programs home fred wilma bin local system locations local locations -
OSG User School 2016 Location, location, location • Who can
install the software? 14 / bin usr lib programs home fred wilma bin local Usually requires administrative privileges Owner of the directory -
OSG User School 2016 Location, location, location • Who can
access the software? 15 / bin usr lib programs home fred wilma bin local Anyone on the system The local user can control who has access -
OSG User School 2016 Location, location, location Implications for DHTC:
• Software MUST be able to install to a local location • Software must be installable without administrative privileges 16 -
OSG User School 2016 Location and running software Instead of
graphic interface… command line 17 • All DHTC jobs must use software that can be run from the command line. • To run a program on the command line, your computer needs to know where the program is located in your computer’s filesystem. -
OSG User School 2016 Common command line programs • Common
command line programs like `ls` and `pwd` are in a system location called `/bin` • Your computer knows their location because `/bin` is included in your `PATH` 18 • The PATH is a list of locations to look for programs -
OSG User School 2016 Other programs on command line Adding
their location to the PATH, then running Using an relative or absolute path to the software 19 • Other programs may be installed in locations not listed in the PATH. You can access them by: -
OSG User School 2016 Command line Implications for DHTC: •
Software must have ability to be run from the command line • Multiple commands are okay, as long as they can be executed in order within a job • There are different ways to “find” your software on the command line: relative path, absolute path, and PATH variable 20 -
OSG User School 2016 Portability requirements Based on the previous
slides, we now know that in order to make software portable for DHTC, the software: • Must work on target operating system (probably Linux) • Must be accessible to your job (placed or installed in job’s working directory) • Must be able to run without administrative privileges • Must be able to run from the command line, without any interactive input from you 21 -
OSG User School 2016 Returning to our scenario: In a
DHTC situation, we are: • Using someone else’s computer - Software may not be installed - The wrong version may be installed - We can’t find/run the installed software Therefore: • We need to bring along and install/run software ourselves 22 -
OSG User School 2016 Portability methods There are two primary
methods to make code portable: • Use a single compiled binary - Typically for code written in C, C++ and Fortran • “Install” with every job - Can’t be compiled into a single binary - Interpreted languages (Matlab, Python, R) 23 -
OSG User School 2016 What is compilation? 25 Source code
Binary compiled into run on libraries compiler and OS uses -
OSG User School 2016 Static compilation 26 Source code Static
binary statically compiled into run anywhere libraries compiler and OS -
OSG User School 2016 Static compilation workflow 28 Option 1
Static binary Submit server Execute server compile download Option 2 -
OSG User School 2016 Install software with every job •
Good for software that: - Can’t be statically compiled - Uses interpreted languages (Matlab, Python, R) - Any software with instructions for local installation • Method: write a wrapper script - Contains a list of commands to execute - Typically written in bash or perl (usually common across operating systems/versions) 30 -
OSG User School 2016 Wrapper scripts • Set up software
in the working directory - Bring along pre-built software and unpack or - Bring along source and install a fresh copy • Run software • Besides software: manage data/files in the working directory - Move or rename output - Delete installation files before job completion 31 -
OSG User School 2016 Wrapper script workflow 32 Submit server
Execute server set up run wrapper script code or pre-built install set up run set up run -
OSG User School 2016 When to pre-build? Pre-built installation •
Install once, use in multiple jobs • Faster than installing from source code within the job • Jobs must run on a computer similar to where the program was built Install with every job • Computers must have appropriate tools (compilers, libraries) for software to install • Can run on multiple systems, if these requirements are met • Longer set-up time 33 -
OSG User School 2016 Preparing your code • Where do
you compile code? Pre-build code? Test your wrapper script? • Guiding question: how computationally intensive is the task? - Computationally intensive (takes more than a few minutes, as a rule of thumb) § Run as interactive job, on a private computer/server, or with a queued job - Computationally light (runs in few minutes or less) § Run on submit server (or above options, if desired) 34 -
OSG User School 2016 Exercises • Software is a compiled
binary - Exercise 1.1: statically compile code and run (C code) - Exercise 1.2: download and run pre-compiled binary (BLAST) • Install software with each job - Exercise 1.3: create a pre-built installation, and write a wrapper script to unpack and run software (GROMACS) 35 -
OSG User School 2016 Questions? • Feel free to contact
me: - [email protected] • Now: Hands-on Exercises - 9:30-10:30am • Next: - 10:30-10:45am: Break - 10:45am-12:15pm: Other research software considerations: licenses and interpreted languages - 12:15-1:15pm: Lunch 36