Share
Embed
ORCONF 2017. State of LibreCores CI
At ORCONF 2016 we announced the work on LibreCores CI - a continuous integration service for open-source Hardware projects. During the last year we made some progress on this front, and this presentation reflects the current progress on infrastructure and development tools.
Oleg Nenashev PRO
September 09, 2017
Transcript
27 slides
-
#librecores, #ORConf Sep 09, 2017 LibreCores Free and Open Digital
Hardware State of LibreCores CI Oleg Nenashev Stefan Wallentowitz 2017 -
#librecores, #ORConf Sep 09, 2017 About us 2 @oleg_nenashev @wallento
• FOSSi, Director • OpTiMSoC • Jenkins Project • CloudBees • LibreCores -
#librecores, #ORConf Sep 09, 2017 Agenda • LibreCores CI •
State of the infrastructure* • Example: OpTiMSoC • Pitfalls we hit • Current state & future work • Demo Disclaimer: the project is in the prototyping stage 3 Oleg Stefan Oleg -
#librecores, #ORConf Sep 09, 2017 What is LibreCores CI? Continuous
Integration service for open-source HW projects 4 LibreCores Web -
#librecores, #ORConf Sep 09, 2017 LibreCore pre-history Started in 2016
Announced at ORCONF2016 5 http://bit.ly/orconf2016-librecores-ci -
#librecores, #ORConf Sep 09, 2017 Vision 2016 LibreCores CI GitHub
SSO User management LibreCores Web Top-level Web UI Project repositories GitHub BitBucket SVN GitHub is a main target system 6 CI Status Gitlab -
#librecores, #ORConf Sep 09, 2017 Vision 2016. Infrastructure Jenkins Docker
(Swarm) Other Integrated Components Physical nodes FPGAs and other HW peripherals LibreCores CI Infrastructure Project-specific Infrastructure On-demand build agents SSH, JNLP over VPN 7 -
#librecores, #ORConf Sep 09, 2017 •Dozens of build machines •Computing
grids for simulations •FPGA grids for hardware testing •Licenses for all EDA tools LibreCores does NOT* provide… *According to the current plan/resources 8 -
#librecores, #ORConf Sep 09, 2017 •Dozens of build machines •Computing
grids for simulations •FPGA grids for hardware testing •Licenses for all EDA tools LibreCores does NOT* provide… 9 BRING YOUR OWN INFRASTRUCTURE! *According to the current plan/resources -
#librecores, #ORConf Sep 09, 2017 Librecores Components Master • PoC
master: ci.librecores.org • New reusable Docker image Agents 10 • Build images with packaged tools • Examples: • Librecores CI • Librecores CI with Modules • FuseSoC + Icarus • FuseSoC + Icarus + Quartus • FuseSoC + RISCV toolchain • To be replaced by Modules Tools • LCCI CLI • Docker images for Building Tools • Shared Pipeline libraries -
#librecores, #ORConf Sep 09, 2017 Example. FuseSoC Core 11 #Jenkinsfile
import org.librecores.fusesoc.FuseSoCBuilder; OrgFuseSoCBuilder.defaultCoreCIBuild(this, repo: 'wb_sdram_ctrl', organization: 'oleg-nenashev', nodeLabels: 'docker-fusesoc-icarus' extraArgs: '--transactions 10’) https://github.com/oleg-nenashev/wb_sdram_ctrl Build Definition: -
#librecores, #ORConf Sep 09, 2017 Usage of LibreCores CI As
a service Deploy on your own Use tools • EDA tool Images • Shared Libraries for Jenkins Pipeline • Docker packages • Configuration scripts • ci.librecores.org • State: PoC It is not just a service! 12 -
#librecores, #ORConf Sep 09, 2017 •Modules •LCCI Tools •OpTiMSoC Example
Under the hood. Tool Infrastructure 13 LibreCores Free and Open Digital Hardware -
#librecores, #ORConf Sep 09, 2017 Module-based Environment • Problem: Diverse
tool requirements • Many different tools are needed • Different versions of the same tool may be used by different projects • Environment Modules: Set environment • Configured by „modulefiles“ • http://modules.sourceforge.net/ $> module load verilator/3.900 PATH=/tools/verilator/3.900/bin/:… VERILATOR_ROOT=/tools/verilator/3.900 $> module load verilator/3.902 PATH=/tools/verilator/3.902/bin/:… VERILATOR_ROOT=/tools/verilator/3.902 14 -
#librecores, #ORConf Sep 09, 2017 Module-based LibreCores CI • Library
in Jenkinsfile • Load required tools and versions • Execute shell commands in environment • https://github.com/libre cores/librecores- pipeline-lib @Library('librecoresci') import org.librecores.ci.Modules def lcci = new Modules(steps) node('librecores-ci-modules') { lcci.load(["eda/verilator/3.902"]) stage('Simulation Build') { lcci.sh 'verilator -f verilate.vc' } } 15 -
#librecores, #ORConf Sep 09, 2017 Module-based Tool Management • Standard
Docker images for LibreCores CI • Do not include tools in docker images (only basic) • Mount /tools volume from host • Tools volume • Tools in different versions • Module infrastructure • LibreCores CI provides simple Docker images to install tools • Re-producible installations • In the same OS environment (Ubuntu 16.04) as agents 16 -
#librecores, #ORConf Sep 09, 2017 LCCI Command Line • Quick
tool installation with simple command line tool • https://github.com/librecores/lcci-python • Example: pip install lcci • Simple YML configuration file • Simply install a tool • Example: lcci tools install verilator:3.902 • Install a whole „standard tool package“ • Example: lcci tools install lcci-2017.1 main: github-api-token: <TOKEN> volumes: tools: /home/lcci-tools 17 -
#librecores, #ORConf Sep 09, 2017 LCCI CLI. (FPGA) Agents •
Command: „lcci start agent fpga-nexys4ddr“ • Add agents to lcci.yaml • Each agent is registered with LCCI • Agents run the docker image • The tools volume is mapped • Connect to jenkins and authenticate • Map extra devices (for FPGA agents) agents: fpga-nexys4ddr: type: modules agent-id: <LCCI-AGENT-ID> agent-secret: <LCCI-AGENT-SECRET> devices: - /dev/bus/usb/001/003 - /dev/ttyUSB1 18 -
#librecores, #ORConf Sep 09, 2017 Example: OpTiMSoC @Library('librecoresci') import org.librecores.ci.Modules
def lcci = new Modules(steps) node('fpga-nexys4ddr') { lcci.load(["eda/xilinx/vivado/2016.4", "eda/fusesoc/1.6.1", "eda/verilator/3.902", "compilers/openrisc/gcc/5.3.0-newlib2.3.0-1", "base/python/3.5"]) stage("Checkout") { checkout scm } stage('Simulation Build') { lcci.sh 'make BUILD_DOCS=no BUILD_EXAMPLES_FPGA=no' } 19 https://github.com/optimsoc/optimsoc/blob/master/Jenkinsfile -
#librecores, #ORConf Sep 09, 2017 Example: OpTiMSoC stage('Simulation Tests') {
lcci.sh 'source objdir/dist/optimsoc-environment.sh; pytest -s -v test/systemtest/test_tutorial.py::TestTutorial' } stage('FPGA Build') { lcci.sh 'make BUILD_DOCS=no' } stage('FPGA Tests') { sh "echo 'boards:' > localconf.yaml" sh "echo ' nexys4ddr:' >> localconf.yaml" sh "echo ' device: /dev/ttyUSB1' >> localconf.yaml" lcci.sh 'source objdir/dist/optimsoc-environment.sh; export OPTIMSOC_TEST_LOCALCONF=$PWD/localconf.yaml; cat $OPTIMSOC_TEST_LOCALCONF; pytest -s -v test/systemtest/test_tutorial.py::TestTutorialFpga' } } 20 https://github.com/optimsoc/optimsoc/blob/master/Jenkinsfile -
#librecores, #ORConf Sep 09, 2017 •[Demo] •Current state •Future plans
What’s next? 21 LibreCores Free and Open Digital Hardware -
#librecores, #ORConf Sep 09, 2017 Plans - 2016 1. Setup
integration with LibreCores Web POC 2. Build definition libraries for FuseSoC YES 3. Setup flows for projects • OpTiMSoC • Other… SOME 4. Start defining usage conditions for projects TODO 5. Ensure the infrastructure can scale enough SOME 6. Start beta-testing TODO 22 -
#librecores, #ORConf Sep 09, 2017 Current state • Many plans,
not so much time • Very first prototype • https://ci.librecores.org • To be replaced by the new version • Dockerized infrastructure • https://github.com/librecores/librecores-ci • Jenkins Master • Agents 23 -
#librecores, #ORConf Sep 09, 2017 Future work 25 1. Deploy
pending changes to https://ci.librecores.org 2. Finalize reference FPGA flows • OpTiMSoC 3. Start beta-testing • LibreCores CI Instance • Standalone setups (Docker) 4. Expand Tooling Support -
#librecores, #ORConf Sep 09, 2017 Contributing • Try it •
https://hub.docker.com/r/librecores/librecores-ci-dev/ • Discussions: • Mailing list: [email protected] • Gitter: https://gitter.im/librecores/librecores-ci • GitHub: • Code repository and task tracker • https://github.com/librecores/librecores-ci 26 -
#librecores, #ORConf Sep 09, 2017 Q & A Contacts: •
Mailing list: [email protected] • Gitter: https://gitter.im/librecores/Lobby • Twitter: @oleg_nenashev, @wallento 27