Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Massive Bare-Metal Operating System Provisionin...

Massive Bare-Metal Operating System Provisioning Improvement - OpenInfra Day Taiwan 2019/11/12

Date Huang

July 09, 2023
Tweet

More Decks by Date Huang

Other Decks in Programming

Transcript

  1. About me • Date Huang 黃宇強 • DozenCloud ARMv8 Cloud

    • Edgecore Networks Sr. Engineer ◦ SDN Solution for Datacenter, Cloud
  2. Nowadays Cloud • Cloud Techs change FAST • Infrastructure-as-Code (IaC)

    for System Provisioning and Service Deployment • Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment • Re-deploy Operating System to Bare-Metal Servers • Bare-Metal System Provisioning toolkit
  3. Nowadays Cloud • Cloud Techs change FAST • Infrastructure-as-Code (IaC)

    for System Provisioning and Service Deployment • Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment • Re-deploy Operating System to Bare-Metal Servers • Bare-Metal System Provisioning toolkit
  4. Legacy Approach • For Massive System Provisioning • Use Unicast

    / Multicast to transfer OS image to server disks • Save temporary image in RAM or another partition or disk • Unitcast ◦ Metal as a Service (MaaS) ◦ OpenStack Ironic • Multicast ◦ Clonezilla Server Edition (Made in Taiwan) ◦ Symantec Ghost Solution Suite
  5. Problems need to be sloved • Cost lots of time

    and Remaining Time is not predictable • Every issue will cause System Provisioning process stuck • Targets will share server’s bandwidth via Unicast (Scale Difficult, Slow down) • Targets aren’t always stable. Need to re-transfer data blocks via Multicast (Slow down) • Targets need to join deployment group and start at the same time via Multicast
  6. Problems need to be sloved cont’d • Unicast is HARD

    to scaling up / out ◦ Server Bandwidth isn’t enough • Mutlicast / Broadcast is NOT always available ◦ Enterprise Network / Public Cloud disallow Multicast / Broadcast ◦ L2-less Datacenter (Clos Network)
  7. Bittorrent • Bittorrent-based System Provisioning • Avoid Multicast / Broadcast

    • Async Data Transfer ◦ Don’t need to join deployment group simultaneously • Reduce Image Provider Network Loading (Server-side)
  8. Challenge in Bittorrent • For Peer-to-Peer file sharing, Temporary Storage

    is Required ◦ Can’t assume getting data order in Bittorrent ◦ Need to store whole disk image • For Bare-Metal Provisioning, the available storage is RAM • Image size will be Limited by RAM size ◦ RAM is normally not big enough to store images and be ready to convert / restore to disk
  9. Brand New Mechanism - EZIO • Compatible with regular Bittorrent

    Protocol ◦ Using Libtorrent-rasterbar • Seed / Proxy Image with regular Bittorrent Client ◦ e.g qBittorrent, transmission • Read/Write image from/into Disk Directly ◦ No need any temporary storage to store image for converting/restoring to disk • https://github.com/tjjh89017/ezio
  10. EZIO Design - Image Format • Design BT-compatible image format

    • Continuous block into one file, save offset in ‘hex’ into name
  11. EZIO Design - Torrent • Torrent will record all continuous

    block ◦ offset and length • Calculate SHA1 hash to verify
  12. EZIO Design - Partition Offset • ‘Piece’ is minimum unit

    in Bittorrent • Cut continuous block into piece (Info is saved in torrent)
  13. EZIO Design - Partition Offset cont’d • When recv ‘Piece

    1’, we will know data is combined with 2 blocks ◦ 0x1000000 - 0x10a2000 ◦ 0x8000000 - 0x8f5e000 • Directly read/write from disk
  14. EZIO Compared with Legacy Approach • Case ◦ Massive OS

    Deployment to multiple Servers in Data Center ◦ OS Image is larger than 50GB • Experimental environment ◦ Network: Cisco 3560G 1Gbps ◦ Server: Dell T1700 with Intel Xeon E3-1226, 16G ram, 1TB hard disk ◦ PC Client: 32 Client, same as Server ◦ Image: Ubuntu Linux with 50GB data in disk. Multicast Image is compressed by pzstd. BT Image is raw file
  15. EZIO Compared with Legacy Approach cont’d • Legacy Approach (e.g.

    CloneZilla Multicast Mode) ◦ Slow, 0.5~2 GiB/min (8~34 MB/s) ◦ If one server stuck, ALL servers stuck ◦ CANNOT be used in L2-less environment (Broadcast and Multicast are not available)
  16. EZIO Compared with Legacy Approach cont’d • Bittorrent Deployment (EZIO)

    ◦ Fast, 2.5~6 GiB/min (42~100 MB/s) ◦ Performance, Reliability and Flexibility Improvements in Heterogeneous Environment (e.g. 100Mbps, 1Gbps, HDD, SSD, etc) ◦ If one server stuck, others continue and no impact at all ◦ Can be used in L2-less environment
  17. Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio

    (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519
  18. Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio

    (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519 Incresing Incresing Flat
  19. Easy way to try EZIO • Clonezilla Lite Server Mode

    ◦ https://clonezilla.org/ ◦ https://clonezilla.org/show-live-doc-content.p hp?topic=clonezilla-live/doc/12_lite_server_BT _from_dev
  20. Still Working on • gRPC Support to control EZIO daemon

    • Working on FAST deployment in DataCenter ◦ Patch OpenStack Ironic to support BT Provisioning ◦ Patch MaaS Curtin to support BT Provisioning