Share
Embed
Transcript
37 slides
-
Zespre (Chih-Hsin) Chang, NTUST, 2024-08-03 Towards a Robust FreeBSD-based Cloud:
Porting OpenStack Components COSCUP 2024 -
About Me • Zespre (Chih-Hsin) Chang • Email: [email protected] •
Blog: https://blog.zespre.com • Roles • Senior Software Engineer at SUSE • Consultant at FreeBSD Foundation • Projects I’m currently working on • OpenStack on FreeBSD (today’s topic) • Harvester HCI • KubeVirtBMC 2 -
Agenda • Introduction • Background • Current Status • Challenges
• Roadmap • Conclusion 3 -
OpenStack on FreeBSD Project origin • CHERI (Capability Hardware Enhanced
RISC Instructions) • Managing a set of Morello evaluation boards with OpenStack Ironic • The OpenStack on FreeBSD project • Started in January 2022 • Chih-Hsin Chang & Li-Wen Hsu (lwhsu@) • Initially targeting on porting OpenStack Ironic • Pivot to VM- fi rst 4 -
OpenStack on FreeBSD Open what? • A cloud infrastructure for
virtual machines, containers, and bare metals • Consist of a stack of open-source software component 5 -
OpenStack Essentials Keystone • Authentication and authorization • Support LDAP
backend • Service discovery 7 -
OpenStack Essentials Glance & Placement • Glance • Serve VM
images and their metadata • Support various storage backends: RBD, Swift, Cinder, … 8 • Placement • Track cloud resource inventory and usage • Help other services, e.g. Nova, make the decision about resource allocation -
OpenStack Essentials Neutron • API server • Agents • `neutron-<l2-impl>-agent`:
L2 network connectivity to OpenStack resources • `neutron-l3-agent`: virtual routers and fl oating IPs • `neutron-dhcp-agent`: IP address issuance • `neutron-metadata-agent`: cloud-init metadata and user data • ML2 (Modular Layer 2) plug-ins • Type drivers: fl at, Geneve, GRE, VLAN, and VXLAN • Mechanism drivers: Open vSwitch, Linux bridge, OVN, SRIOV, MacVTap and L2 population 9 -
OpenStack Essentials Nova • API server • Scheduler • Collect
resource usage from compute nodes/Placement API • Decide what node to run the instance • Conductor • Prepare instance information based on DB entries 10 -
OpenStack Essentials Nova • Compute • Manage instance lifecycle through
hypervisor on each compute node • Hypervisor manager • Serial proxy • Provide access to instance console over WebSocket 11 -
OpenStack Essentials Ironic • Manages bare-metals in contrast to typical
Nova usage • Deployment models • Stand-alone mode • Keystone + Ironic • As a Nova virt driver 12 -
Project Status Dev environment • In-house development environment • Processors:
2 x Intel Xeon E5-2680 v4 • Motherboard: Supermicro X10DRL-i • Memory: 64 GB • Storage: 1 TB SSD • Single-node, all-in-one cluster • Remote PoC site: openstack[012] • Under construction • Targeting OpenStack 2024.1 Caracal + FreeBSD 15.0-CURRENT • Three-node HA cluster 13 -
14 So, how does it look right now? • Install
from source • Install dependencies • Pull down the (modi fi ed) source code • Build and install OpenStack packages • Each component runs in its own Python virtual environment Project Status -
Project Status So, how does it look right now? •
Keystone: source code unmodi fi ed • Glance: source code unmodi fi ed • Placement: source unmodi fi ed • Neutron • Code patches • Con fi guration: fl at network + Open vSwitch • Nova • Code patches • Con fi guration: libvirt + bhyve 15 -
Project Status So, how does it look right now? •
Limitations • No tenant network isolation • Need external DHCP service • No fl oating IPs 16 -
Events Gaining exposure • AsiaBSDCon 2023 (FreeBSD Developer Summit) •
AsiaBSDCon 2024 • BSDCan 2024 17 -
Project Status The OpenStack on FreeBSD GitHub organization • Step-by-step
build and installation guide can be found at openstack-on- freebsd/docs • Issue management is centralized in openstack-on-freebsd/admin • Source code • openstack-on-freebsd/neutron • openstack-on-freebsd/nova 19 https://github.com/openstack-on-freebsd -
Project Status The OpenStack on FreeBSD GitHub organization • Ports
• openstack-on-freebsd/openstack (kudos to chuck@) • Custom solutions • openstack-on-freebsd/socat-manager • openstack-on-freebsd/novaconsole 20 https://github.com/openstack-on-freebsd -
Challenges Computing • Nova abstracts the operations against the underlying
hypervisors • Nova virtualization driver • Well-de fi ned interfaces • Per-compute node con fi guration • Currently supported drivers • `libvirt.LibvirtDriver` • `fake.FakeDriver` • `ironic.IronicDriver` • `vmwareapi.VMwareVCDriver` • `zvm.zvmDriver` 21 virt driver -
Acts Leveraging the libvirt Driver • libvirt • Only implement
a limited set of functions for FreeBSD/bhyve • libvirt virt driver • Some operations speci fi c to bhyve are not covered by libvirt • Require a new virtualization type - bhyve 22 libvirt drv. bhyve -
Challenges Networking • L2 agents • Lack of Linux bridge
and iptables • L3 agent • Lack of iptables • DHCP agent • Lack of Linux network namespace and veth pairs • The current choice of ML2 drivers for FreeBSD • Type driver: `flat` • Mechanism driver: `openvswitch` 23 -
24 VLAN + Open vSwitch (Linux host) Flat + Open
vSwitch (FreeBSD host) -
Acts Open vSwitch on FreeBSD • Open vSwitch `datapath_type=netdev`, without
DPDK • No `openvswitch` kernel module • The combination is considered experimental (not tested thoroughly) • Performance issue • TODOs • Enable DPDK for better performance • Develop the native FreeBSD bridge agent • Replace Linux network namespace + veth(4) with vnet(9) + epair(4) for DHCP functionality • Replace Linux network namespace + iptables(8) with vnet(9) + pf(4) for virtual router functionality • Replace iptables(8) with pf(4) for security group functionality 25 -
Challenges IP Address Mismatch 1. `neutron-dhcp-agent` issue 2. Disable DHCP
for the fl at network 3. Set up a DHCP server external to the cluster 4. VMs get IP addresses from the external DHCP server 5. Source IP address does not match the one Neutron allocated • Policy enforced by the fl ow rules in the underlying Open vSwitch 6. Packets originating from VMs get dropped silently 26 -
Acts IP Address Mismatch • Workaround • Con fi gure
VM’s IP address via console • Formal solution • Make `neutron-dhcp-agent` function normally on FreeBSD • Due to this workaround, another challenge emerged… 27 -
Challenges VM consoles • cu(1) - call Unix • Required
access to the compute host • Impractical and insecure 28 -
Acts Double proxying • Introduce socat-manager • Listen on Unix
socket • Maintain TCP port to nmdm(4) mappings • Manage socat(1) processes • libvirt hook script • Take domain XML as input • Call socat-manager with parameters (port and nmdm(4) device name) 29 $ /usr/local/bin/socat \ file:/dev/nmdm21B,ispeed=9600,ospeed=9600,raw,echo=0 \ tcp-listen:10014,bind=0.0.0.0,reuseaddr,fork -
Challenges Privilege • Principle of least privilege • Run with
reduced/no privilege • Escalate when absolutely required • Operations will be translated into commands and run on the OS 31 -
Challenges The evolution of privilege mechanism in OpenStack • sudo(8)
• One-shot • All or nothing • `oslo.rootwrap` • Allow advanced fi lters • Support one-shot or daemon mode • Performance penalty • Does not allow long-lived/streaming commands 32 $ sudo command $ sudo nova—rootwrap \ /etc/nova/rootwrap.conf \ command -
Challenges The evolution of privilege mechanism in OpenStack • `oslo.privsep`
• Leverage Linux capabilities(7) • Drop root superpowers but only keep what is required • Two-process model (unprivileged and privileged) • Connected over a local communication channel • Share the same fate 33 -
Acts Fallback • Linux capabilities(7) are not available on FreeBSD
• Workaround • Fallback using rootwrap • Formal solution • Leverage FreeBSD’s own privilege management mechanism • mac(9) framework 34 -
Future Work • Develop native drivers for components that have
plug-ins architecture • Neutron • Nova • Port additional OpenStack components to FreeBSD • Ironic • Horizon • Create corresponding FreeBSD ports • Catch up on newer OpenStack releases • 2024.01 Caracal! • Continuous engagement and knowledge sharing • Performance and scalability improvements 35 -
Conclusion • Bring Linux- fi rst design to FreeBSD •
Follow the Windows path (?) • Use cases are very limited • We dropped too many things to make it viable • There are many topics/issues that need expertise • Explore network implementation equivalents • Finding a suitable privilege separation implementation • Need to formalize the changes 36