Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lightweight BI with Ruby, Rails, and MongoDB

Lightweight BI with Ruby, Rails, and MongoDB

As presented at Windy City Rails 2013

Coraline Ada Ehmke

September 13, 2013
Tweet

More Decks by Coraline Ada Ehmke

Other Decks in Programming

Transcript

  1. WHO AM I? A developer with a long memory and

    a longer career Sr. developer at Apartments.com Co-founder of LGBTech A lifelong learner and autodidact
  2. WHAT IS BUSINESS INTELLIGENCE? Collection & organization of mission-critical knowledge

    Historical view of business operations Tools to support decision making
  3. The foundation on which the fun stuff is built A

    science unto itself Not of great interest to most developers INFRASTRUCTURE
  4. Enable the delivery of products and services Optimize business processes

    Provide leverage for scaling the business APPLICATIONS
  5. DATA Exists to support critical business decisions Better data leads

    to better decisions ∴ Data must be as timely and useful as possible
  6. THE NAÏVE APPROACH Reporting out of the transactional database Raw

    SQL embedded in your code for “performance” Granting direct database access to stakeholders
  7. THE NAÏVE APPROACH: SHORTCOMINGS Fighting the schema with complex joins

    Poor performance Impact on production resources Chance of catastrophic data loss due to...
  8. THE ENTERPRISE APPROACH Distinct infrastructure/stack Nightly ETL (extract, transform, load)

    processes Schema designed for reporting Combination of static and dynamic reports
  9. THE ENTERPRISE APPROACH: SHORTCOMINGS 24 hour delay in data availability

    Expensive to configure and maintain Requires highly specialized resources Hard to change your mind or adapt Enterprise-y
  10. TRADITIONAL BI: THE WRONG HAMMER Waterfall approach Painful data migrations

    Brittle ETL processes No automated testing Logic embedded in the data store “Retro” user interfaces
  11. THE FOUR NOBLE GOALS OF THE LIGHTWEIGHT BI APPROACH Provide

    real-time data to support decisions Leverage familiar technology & infrastructure Use existing development staff Support an iterative, agile approach
  12. GETTING STARTED Collaborate! Determine the KPIs that actually matter Figure

    out what questions the business is asking on a daily basis
  13. TURN INFERENCES INTO FACTS Find a way to tell a

    story with your data Design your schema based on facts De-normalize like a boss
  14. PRESENT ANSWERS Provide a central, single source of truth Present

    the data wherever it’s needed Dashboard design is harder than you think Plan for iterations & ongoing collaboration
  15. BRING RUBY TO THE PARTY Makes agile, test-driven development easy

    Quickly deploy new apps Plenty of visualization libraries Powerful SQL & NoSQL ORMs Great data munging capabilities
  16. LEVERAGE MONGODB Flexible and dynamic schemas Support for native datatypes

    Powerful querying, map/ reduce, and aggregation Fast and performant Easy to scale up
  17. PARALLEL DB DEPLOYMENT Modern frameworks support multiple ORMs Use SQL

    for transactions Use NoSQL for reporting Business logic in your apps, not in your database
  18. STATISTICAL MODELS Collections of facts, not attributes Data spanning deep

    and wide object graphs De-normalized and optimized for reporting
  19. module  MemberDataProfiles    class  Performance        include  Mongoid::Document

           include  Mongoid::Timestamps        field  :member_id,                                  :type  =>  Integer        field  :annual_trunk_frequency,        :type  =>  Integer,    :default  =>  0        field  :average_trunk_value,              :type  =>  Float,        :default  =>  0.0        field  :is_customer,                              :type  =>  Boolean,    :default  =>  false        field  :keep_rate,                                  :type  =>  Float,        :default  =>  0.0        field  :last_transaction_date,          :type  =>  Date        field  :member_creation_date,            :type  =>  Date        field  :total_value,                              :type  =>  Float,        :default  =>  0.0        field  :total_number_of_trunks,        :type  =>  Integer,    :default  =>  0    end end
  20. STREAMING ETL Event-triggered, continuous data extraction Calculations are defined in

    code rather than SQL or ETL scripts Allow resource-intensive data munging to happen in the background Provide near-real-time data
  21. APIS EVERYWHERE Provide easy access to your data Allow reuse

    of data in novel ways Quickly build dashboards and data explorers Use the Faceted gem to make building APIs easy
  22. A TYPICAL SMALL APP ECOSYSTEM Persistence Service Postgres Ops Members

    Sales iOS App Inventory Outfitting Messaging Dashboard Reporting Service MongoDB Business Objects MySQL
  23. LEVIATHAN Records events from all applications Subscribes to all message

    queues Collects and displays real-time data Browse, search, & drill-down interface Longitudinal analysis with dynamic cohorts
  24. CRITICAL DATA POINTS Marketing campaign performance Member on-boarding funnels Order

    lifecycles Customer service interactions Product performance
  25. class  Event    include  Mongoid::Document    include  Mongoid::Timestamps    field

     :label    field  :application    field  :details,  :type  =>  Hash,  :default  =>  {}    def  self.record!(label,  application,  params={})        Event.create(            :label  =>  label,            :application  =>  application,            :details  =>  params        )    end    def  self.search_details(criteria={})        where("details.#{criteria.keys.first}"  =>  /#{criteria.values.first}/i)    end end