Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leveraging Ambari to Build Comprehensive Manage...

Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop Applications

Apache Conference (2015): http://sched.co/3zzs

This presentation will demonstrate how to leverage modern HTML5 technologies with the flexibility of Apache Ambari to build a comprehensive, responsive and attractive management interfaces for your Hadoop applications. In the process we will walk you through the reference implementation of an management interface for SQL-on-Hadoop application and integrate it with Apache Ambari. We will share our experience in using technologies like Google Polymer, Spring Boot and Apache Ambari.

Christian Tzolov

September 28, 2015
Tweet

More Decks by Christian Tzolov

Other Decks in Technology

Transcript

  1. Leveraging Ambari to Build Comprehensive Management UIs For Your Hadoop

    Applications by Christian Tzolov @christzolov
  2. Whoami Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD,

    Apache Committer, Crunch PMC member [email protected] blog.tzolov.net @christzolov
  3. Contents • Ambari Overview • Stacks and Services (demo) •

    Blueprints (demo) • Views (demo) • Metrics & Monitoring • On the Cloud
  4. Deployment Topology Ambari Web (REST API) Ambari Server admin node

    Ambari Agent cluster node 1 Ambari Agent cluster node 2 Ambari Agent cluster node N
  5. Service Concepts • Stack - Set of (managed) Services and

    software packages for them • Service - Set of (distributed) Components • Component - Service building block of type Master, Slave, Client
  6. Service Layout |_ services |_ <service_name> metainfo.xml metrics.json |_ configuration

    {configuration files} |_ package |_ scripts {command scripts} • metainfo.xml - defines service, components and the command management scripts • configuration - service config files used in and deployed through the command scripts • package/scripts - commandScripts and templates.
  7. Metainfo.xml <metainfo> <schemaVersion/> <services> <service> <name/> <displayName/> <comment/> <version/> <components/>

    <osSpecifics/> <commandScript/> <requiredServices/> <configuration-dependencies/> </service> </services> </metainfo> <components> <component> <name>CUST_MASTER</name> <displayName>Master</displayName> <category>MASTER</category> <cardinality>1</cardinality> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> ……… </components>
  8. Service / Command Script import sys from resource_management import *

    class Master(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Master().execute() import sys from resource_management import * class Slave(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Slave().execute()
  9. Why? • Installation Automation • Declarative definition of a cluster

    • Specify a Stack, the Component layout and the Configurations to materialize a cluster • REST API
  10. Blueprint Flow Ambari Server node 1 Cluster 1 node 2

    node N Cluster 2 . . . . Cluster Z . . . . Blueprint (json) Host Mapping (json) REST API
  11. Blueprint Deploy { "Blueprints" : { "stack_name" : "PHD", "stack_version"

    : "3.0" }, "configurations" : [ { "hawq-site" : { "properties" : { "hawq.master.port" : "5532" } } } ], "host_groups" : [ { "name" : "management_masters", "components" : [ { "name" : "NAMENODE" }, { "name" : "HAWQMASTER" } ], "cardinality" : "1" }, …… ]} POST /api/v1/blueprints/my-blueprint { "blueprint" : "hdfs-hawq-blueprint", "default_password" : "secret-password", "host_groups" : [ { "name" : "management_masters", "hosts": [ { "fqdn":"ambari.localdomain" } ] }, { "name" : "masters_workers", "hosts": [ { "fqdn":"phd1.localdomain" }, { "fqdn":"phd2.localdomain" }, { "fqdn":"phd3.localdomain" } ] } ] } POST /api/v1/clusters/my-cluster
  12. Blueprint REST API /api/v1/blueprint Cluster Logical Structure and configuration /api/v1/cluster

    Physical cluster mapping. Materializes a blueprint with real infrastructure /api/v1/clusters/<cluster name>? format=blueprint Exports blueprint for an existing cluster
  13. View Components Ambari Server Ambari Web Server Side (java) Client

    Side (js,html..) REST API VIEW PACKAGE (JAR) • Client Side - any client-side technologies • Server Side - Java application, Servlet, a REST service or ResourceProvider. • View Package - JAR bundles of the view definition and resources • View Context - Instance attributes and configuration context
  14. View Context • On the Server Side • View-instance Config

    • View-instance Attributes • Execution context, authenticated user principal ViewContext getUsername() : String getViewName() : String getInstanceName() : String getProperties() : Map<String,String> putInstanceData(…) getInstanceData(…) : String
  15. View REST API GET /api/v1/views List deployed views GET /api/v1/views/<View

    Name> List all View instances GET /api/v1/views/<View Name>/versions/<View Version> Show specific View instance POST /api/v1/views/<View Name>/versions/<View Version>/instances/<Instance Name> [{“ViewInstanceInfo":{ "properties" : { "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}] Create new View Instance
  16. Ambari Metrics System Collect, Aggregate and Serve System Metrics •

    Metric Collector - Collects and Aggregates • Metric Monitor - System-level metrics • Metric Hadoop Sinks - Service-level metrics