Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Prototyping Kubernetes CRI With Ruby and gRPC

remore
February 10, 2018

Prototyping Kubernetes CRI With Ruby and gRPC

Slides for my talk at RubyConf India 2018

remore

February 10, 2018
Tweet

More Decks by remore

Other Decks in Technology

Transcript

  1. How many of you guys Are using container for your

    work? Are using kubernetes for your work already? Have ever implemented your own Kubernetes CRI implementation?
  2. Docker is one of implementations of Container Runtime Container Runtime

    is something you use to manipulate container images There are many docker alternatives out there such as runc, rkt, and runv. What is important about them is, they are interchangeable since they all are compatible with OCI speci cation, which is a standard speci cation of container runtime and image. OCI stands for Open Container Initiative
  3. In Kubernetes, it is de ned like this: CRI, Container

    Runtime Interface provides a set of APIs you can call from client application. The original image is taken from http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in- kubernetes.html
  4. Now We Know Docker is a one of Container Runtime,

    but not "the only one" runtime Kubernetes is using gRPC to manage Container Runtime, and spec name for that is CRI
  5. OK Now We Know CRI Works Over gRPC. But What

    Kind of APIs Are Available Actually?
  6. CRI Protocol Buffer's IDL CRI APIs consists of 2 services(

    RuntimeService and ImageService ). In the RuntimeService , it looks like we can Run/Stop/Remove PodSandbox... service RuntimeService { // Managing PodSandbox rpc RunPodSandbox(RunPodSandboxRequest) returns (RunPodSandb rpc StopPodSandbox(StopPodSandboxRequest) returns (StopPodSa rpc RemovePodSandbox(RemovePodSandboxRequest) returns (Remov rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSa rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSa https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot
  7. And we can Create/Start/Stop/Remove Container... // Managing Containers rpc CreateContainer(CreateContainerRequest)

    returns (CreateC rpc StartContainer(StartContainerRequest) returns (StartCont rpc StopContainer(StopContainerRequest) returns (StopContain rpc RemoveContainer(RemoveContainerRequest) returns (RemoveC rpc ListContainers(ListContainersRequest) returns (ListConta rpc ContainerStatus(ContainerStatusRequest) returns (Contain rpc UpdateContainerResources(UpdateContainerResourcesRequest rpc ReopenContainerLog(ReopenContainerLogRequest) returns rpc ContainerStats(ContainerStatsRequest) returns (Container rpc ListContainerStats(ListContainerStatsRequest) returns https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.prot
  8. And misc(etc etc) // Misc rpc Version(VersionRequest) returns (VersionResponse) {}

    rpc ExecSync(ExecSyncRequest) returns (ExecSyncResponse) {} rpc Exec(ExecRequest) returns (ExecResponse) {} rpc Attach(AttachRequest) returns (AttachResponse) {} rpc PortForward(PortForwardRequest) returns (PortForwardResp rpc UpdateRuntimeConfig(UpdateRuntimeConfigRequest) returns rpc Status(StatusRequest) returns (StatusResponse) {} }
  9. Oh last but not least, we can also call List/Pull/Remove

    Image Operations: service ImageService { // Managing Image rpc ListImages(ListImagesRequest) returns (ListImagesRespons rpc ImageStatus(ImageStatusRequest) returns (ImageStatusResp rpc PullImage(PullImageRequest) returns (PullImageResponse) rpc RemoveImage(RemoveImageRequest) returns (RemoveImageResp rpc ImageFsInfo(ImageFsInfoRequest) returns (ImageFsInfoResp }
  10. Now We Know Docker is a one of Container Runtime,

    but not "the only one" runtime Kubernetes is using gRPC to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs.
  11. A Bit Closer Look at Speci cations CRI Speci cation

    de nes standard way to manage PodSandbox, Container, and Image including networking/metrics/logging requirements. OCI Runtime Speci cation de nes state, lifecycle, errors/warnings, operations such as create/start/kill/delete , and hooks. OCI Image Speci cation de nes image layout, manifest, index, lesystem layers, and image con guration etc.
  12. OK Now We Know CRI. BTW What Is It Like

    To Manage Servers Using
  13. A Very Quick Introduction As you can see, Kubernetes will

    let you deploy your arbitrary container image on any cloud service without vendor lock-in # Configure kubectl(here we use gcp just as an example) $ gcloud config set project PROJECT_ID $ gcloud container clusters get-credentials PROJECT_ID \ --zone us-central1-b # kubectl is set to use gcp # Deploy $ kubectl run myapp \ --image=remore/simple-ruby-server \ # From DockerHub --port 8080 $ kubectl expose deployment myapp --type="LoadBalancer" $ curl 130.211.134.59:8080
  14. Now We Know Docker is a one of Container Runtime,

    but not "the only one" runtime Kubernetes is using gRPC through $kubectl command to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs.
  15. Given These Understandings, Now Today's Objective Turns Out: What is

    it like to create our own gRPC server works as a CRI Shim with Ruby? “ “
  16. SHIFU: "If you only do what you can do, you

    will never be more than you are now"
  17. Today's Scenario Now is the time to get our hands

    dirty. Today we experimentally do: 1. Build Our Own Container Image 2. Build Our Own Container Runtime 3. Prototype Our Own Kubernetes CRI Implementation With Ruby and gRPC And let's wonder what Ruby can do to make this containerized world better.
  18. 1. Build Our Own Container Image First of all, container

    images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $
  19. 1. Build Our Own Container Image First of all, container

    images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $ docker save busybox > busybox.tar $
  20. 1. Build Our Own Container Image First of all, container

    images are just tarballs. $ docker pull busybox Using default tag: latest latest: Pulling from library/busybox 57310166fe88: Pull complete Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090a Status: Downloaded newer image for busybox:latest $ docker save busybox > busybox.tar $ tar xzvf busybox.tar x 5b0d5902672....e7c25bd2c3.json x ee12c7d73ed....4704362eb2/ x ee12c7d73ed....4704362eb2/VERSION x ee12c7d73ed....4704362eb2/json x ee12c7d73ed....4704362eb2/layer.tar x manifest.json x repositories
  21. 1. Build Our Own Container Image FYI: If you are

    interested in the contents: $ cat manifest.json | jq [ { "Config": "5b0d5902672....e7c25bd2c3.json", "RepoTags": [ "busybox:latest"], "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"] } ] $
  22. 1. Build Our Own Container Image FYI: If you are

    interested in the contents: $ cat manifest.json | jq [ { "Config": "5b0d5902672....e7c25bd2c3.json", "RepoTags": [ "busybox:latest"], "Layers": ["ee12c7d73ed....4704362eb2/layer.tar"] } ] $ cat repositories | jq { "busybox": { "latest": "ee12c7d73ed....4704362eb2" } }
  23. 1. Build Our Own Container Image Since container image is

    just tarballs, that means we can create one without any hassle: # In case of Debian $ mkdir /var/lib/myroot $ debootstrap wheezy \ /var/lib/myroot/ http://debianmirror.nkn.in/debian/ $ tar -c /var/lib/myroot | docker import - <image_name> $ docker push <image_name> Or of course, for example alternatively we can use Dockerfile or buildah to build OCI compliant images. e.g. Creating a OCI Container within Docker Container
  24. 2. Build Our Own Container Runtime Although there are many

    RubyGems for virtualization and containerization, there is no popular OCI-compatible container runtime written in Ruby. Most popular container runtimes are written in Go(e.g. runc, rtk) Try to search "lxc" at rubygems.org for example
  25. Introducing Haconiwa github.com/haconiwa/haconiwa Helper tools with DSL for your handmade

    linux containers, written in mruby My favorite toolkit to DIY our container work
  26. A Demo Container with Haconiwa $ touch demo.haco $ cat

    > demo.haco <<EOF > Namespace.unshare(Namespace::CLONE_NEWNS) > Namespace.unshare(Namespace::CLONE_NEWPID) > Mount.make_private "/" > Mount.bind_mount "/var/lib/myroot", "/var/lib/myroot/root" > Dir.chroot "/var/lib/myroot" > Dir.chdir "/" > c = Process.fork { > Mount.mount "proc", "/proc", :type => "proc" > Exec.exec "/bin/date" > } > pid, ret = Process.waitpid2 c > puts "Container exited with: #{ret.inspect}" > EOF
  27. A Demo Container with Haconiwa It's not OCI speci cation

    compatible, but it works anyways: # `date` command is executed inside the container $ haconiwa start ./demo.haco Mon Feb 5 01:32:03 UTC 2018 Container exited with: #<Process::Status: pid=7148,exited(0)>
  28. gRPC in Ruby World Looks like grpc rubygem, which is

    the of cial package provided by The gRPC Authors backed by CNCF, is the only way to create reliable gRPC server
  29. gRPC Server Implementation This example simply returns version number( grpc

    and runtime_services_pb rubygem is required) class RuntimeServiceServer < Runtime::RuntimeService::Service def version(request, _unused_call) Runtime::VersionResponse.new( version: "1.1", runtime_name:"foobar_runtime", runtime_version:"9.9.9", runtime_api_version:"9999" ) # Here we just use dummy values end end s = GRPC::RpcServer.new s.add_http2_port('unix:/var/run/k8s_cri_prototype.sock', :this_port_is_insecure) s.handle(RuntimeServiceServer) s.run_till_terminated
  30. Give It A Try Only a few lines of bash

    command are needed to test if this gRPC server works: BTW I have pushed this sample source code at: github.com/remore/kubernetes_cri_prototype $ git clone \ https://github.com/remore/kubernetes_cri_prototype.git $ cd ./kubernetes_cri_prototype $ bundle install $ bundle exec ruby ./server.rb & [1] 10712 $ bundle exec ruby ./demo_client.rb "Reponse Message: #<Runtime::VersionResponse:0x00000000bba6e8>"
  31. Mocking Up Real World Examples Now as the last example,

    we're going to a few more methods to RuntimeServiceServer class. This is a method corresponding to RunPodSandboxRequest : class RuntimeServiceServer def run_pod_sandbox(request, _unused_call) Runtime::RunPodSandboxResponse.new( pod_sandbox_id: "c605182c-ca3b-11e4-ad0b-525400c788eb" ) end end
  32. Mocking Up Real World Examples And this is for CreateContainerRequest

    : class RuntimeServiceServer def create_container(request, _unused_call) pipe_cmd_in, pipe_cmd_out = IO.pipe @pid = Process.spawn( "haconiwa run #{File.dirname(__FILE__)}/container.haco" :out => pipe_cmd_out, :err => pipe_cmd_out ) Process.wait @pid pipe_cmd_out.close out = pipe_cmd_in.read Process.detach @pid Runtime::CreateContainerResponse.new( container_id: "container-#{out.match(/(\d+)/).to_s}" ) end end
  33. Mocking Up Real World Examples With these preparation, we can

    respond to both RunPodSandboxRequest and CreateContainerRequest triggered by crictl, which is a CRI client simulator: # crictl -r <CRI_RUNTIME_ENDPOINT> <command> <args> # Inquire about version info $ crictl -r /var/run/k8s_cri_prototype.sock version Version: 1.1 RuntimeName: foobar_runtime RuntimeVersion: 2.2.3 RuntimeApiVersion: 9999 # Run a new pod sandbox("runp" stands for RUN Pod maybe) $ crictl -r /var/run/k8s_cri_prototype.sock runp \ test/testdata/sandbox_config.json # PodSandbox config c605182c-ca3b-11e4-ad0b-525400c788eb
  34. # This json is downloaded from # https://github.com/kubernetes-incubator/cri-o $ cat

    test/testdata/sandbox_config.json { "metadata": { "name": "podsandbox1", "uid": "redhat-test-crio", "namespace": "redhat.test.crio", "attempt": 1 }, "hostname": "crictl_host", "log_directory": "", "dns_config": {"searches": ["8.8.8.8"]}, "port_mappings": [], "resources": { "cpu": { "limits": 3, "requests": 2 }, "memory": { "limits": 50000000,
  35. Mocking Up Real World Examples And nally testing CreateContainerRequest :

    # Start to run a container $ crictl -r /var/run/k8s_cri_prototype.sock create \ "c605182c-ca3b-11e4-ad0b-525400c788eb" \#PodSandbox ID test/testdata/container_redis.json \#Container config test/testdata/sandbox_config.json #PodSandbox config container-3939 # Check if the container is running $ curl -s localhost:8000 <html><body>...</body></html> Tada! Finally we have responded to several requests sent from an of cial CRI client.
  36. Takeaway Docker is a one of Container Runtime, but not

    "the only one" runtime Kubernetes is using gRPC through $kubectl command to manage Container Runtime, and spec name for that is CRI CRI de nes 22 APIs for RuntimeService and 5 APIs for ImageService(as of 2018 Feb). Management of Container, Image, and PodSandbox are the main purpose of these APIs It looks like we can create our own CRI implementation, there are tons of todos though
  37. [Discussion] Kubernetes ecosystem is evolving rapidly. Many tooling and concepts

    are still on the way to getting matured. Much more effort is needed to improve. Good thing is essential parts of container orchestration - Container Image, Container Runtime, and Interface to Container Runtime has been standardized by CRI and OCI speci cation. I have not mentioned so much in this session, but in the real world most of the tools for kubernetes seems written in Go but not Ruby
  38. [Discussion] Potentially Ruby have something to do with this situation.

    What about the idea, for example: Write ruby script as a con guration le, just like Jsonnet , instead of writing redundant too many YAML con g les? Create your own FaaS(like AWS lambda) platform specially designed for Ruby? Write alternative to crictl sounds also interesting. It's always good to stick to a minimum set of tools until you really need it. (e.g. use grpc-gateway as an endpoint)
  39. Appendix: One Of The Best Free Resource To Learn Kubernetes

    https://kubernetes.io/docs/tutorials/kubernetes-basics/deploy-interactive/
  40. irb(main):001:0> Kubernetes.is_incubated_by => :CNCF CNCF is founded Dec 2015 as

    a part of Linux Foundation CNCF stands for "Cloud Native Computing Foundation" Cloud Native Computing is a concept of open source software stack to be: - Containerized - Dynamically orchestrated - Microservices oriented
  41. irb(main):003:0> Kubernetes.is_used_by => :most_of_cloud_vendors irb(main):004:0> Kubernetes.is_de_facto_in(2018) => true What happened

    last year was: CoreOS has moved from eet to Kubernetes Docker has started to support kubernetes Big cloud vendor such as AWS and Microsoft has started managed kubernetes service(such as AKS and EKS)