Share
Embed
Massive Bare-Metal Operating System Provisioning Improvement - OpenInfra Day Taiwan 2019/11/12
Date Huang
July 09, 2023
Transcript
26 slides
-
About me • Date Huang 黃宇強 • DozenCloud ARMv8 Cloud
• Edgecore Networks Sr. Engineer ◦ SDN Solution for Datacenter, Cloud -
Nowadays Cloud • Cloud Techs change FAST • Infrastructure-as-Code (IaC)
for System Provisioning and Service Deployment • Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment • Re-deploy Operating System to Bare-Metal Servers • Bare-Metal System Provisioning toolkit -
Nowadays Cloud • Cloud Techs change FAST • Infrastructure-as-Code (IaC)
for System Provisioning and Service Deployment • Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment • Re-deploy Operating System to Bare-Metal Servers • Bare-Metal System Provisioning toolkit -
Legacy Approach • For Massive System Provisioning • Use Unicast
/ Multicast to transfer OS image to server disks • Save temporary image in RAM or another partition or disk • Unitcast ◦ Metal as a Service (MaaS) ◦ OpenStack Ironic • Multicast ◦ Clonezilla Server Edition (Made in Taiwan) ◦ Symantec Ghost Solution Suite -
Problems need to be sloved • Cost lots of time
and Remaining Time is not predictable • Every issue will cause System Provisioning process stuck • Targets will share server’s bandwidth via Unicast (Scale Difficult, Slow down) • Targets aren’t always stable. Need to re-transfer data blocks via Multicast (Slow down) • Targets need to join deployment group and start at the same time via Multicast -
Problems need to be sloved cont’d • Unicast is HARD
to scaling up / out ◦ Server Bandwidth isn’t enough • Mutlicast / Broadcast is NOT always available ◦ Enterprise Network / Public Cloud disallow Multicast / Broadcast ◦ L2-less Datacenter (Clos Network) -
Bittorrent • Bittorrent-based System Provisioning • Avoid Multicast / Broadcast
• Async Data Transfer ◦ Don’t need to join deployment group simultaneously • Reduce Image Provider Network Loading (Server-side) -
Challenge in Bittorrent • For Peer-to-Peer file sharing, Temporary Storage
is Required ◦ Can’t assume getting data order in Bittorrent ◦ Need to store whole disk image • For Bare-Metal Provisioning, the available storage is RAM • Image size will be Limited by RAM size ◦ RAM is normally not big enough to store images and be ready to convert / restore to disk -
Brand New Mechanism - EZIO • Compatible with regular Bittorrent
Protocol ◦ Using Libtorrent-rasterbar • Seed / Proxy Image with regular Bittorrent Client ◦ e.g qBittorrent, transmission • Read/Write image from/into Disk Directly ◦ No need any temporary storage to store image for converting/restoring to disk • https://github.com/tjjh89017/ezio -
EZIO Design - Image Format • Design BT-compatible image format
• Continuous block into one file, save offset in ‘hex’ into name -
EZIO Design - Torrent • Torrent will record all continuous
block ◦ offset and length • Calculate SHA1 hash to verify -
EZIO Design - Partition Offset • ‘Piece’ is minimum unit
in Bittorrent • Cut continuous block into piece (Info is saved in torrent) -
EZIO Design - Partition Offset cont’d • When recv ‘Piece
1’, we will know data is combined with 2 blocks ◦ 0x1000000 - 0x10a2000 ◦ 0x8000000 - 0x8f5e000 • Directly read/write from disk -
EZIO Compared with Legacy Approach • Case ◦ Massive OS
Deployment to multiple Servers in Data Center ◦ OS Image is larger than 50GB • Experimental environment ◦ Network: Cisco 3560G 1Gbps ◦ Server: Dell T1700 with Intel Xeon E3-1226, 16G ram, 1TB hard disk ◦ PC Client: 32 Client, same as Server ◦ Image: Ubuntu Linux with 50GB data in disk. Multicast Image is compressed by pzstd. BT Image is raw file -
EZIO Compared with Legacy Approach cont’d • Legacy Approach (e.g.
CloneZilla Multicast Mode) ◦ Slow, 0.5~2 GiB/min (8~34 MB/s) ◦ If one server stuck, ALL servers stuck ◦ CANNOT be used in L2-less environment (Broadcast and Multicast are not available) -
EZIO Compared with Legacy Approach cont’d • Bittorrent Deployment (EZIO)
◦ Fast, 2.5~6 GiB/min (42~100 MB/s) ◦ Performance, Reliability and Flexibility Improvements in Heterogeneous Environment (e.g. 100Mbps, 1Gbps, HDD, SSD, etc) ◦ If one server stuck, others continue and no impact at all ◦ Can be used in L2-less environment -
Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio
(BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519 -
Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio
(BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519 Incresing Incresing Flat -
Easy way to try EZIO • Clonezilla Lite Server Mode
◦ https://clonezilla.org/ ◦ https://clonezilla.org/show-live-doc-content.p hp?topic=clonezilla-live/doc/12_lite_server_BT _from_dev -
Still Working on • gRPC Support to control EZIO daemon
• Working on FAST deployment in DataCenter ◦ Patch OpenStack Ironic to support BT Provisioning ◦ Patch MaaS Curtin to support BT Provisioning