Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Collecting logs from mobile apps

Collecting logs from mobile apps

rejasupotaro

April 15, 2016
Tweet

More Decks by rejasupotaro

Other Decks in Programming

Transcript

  1. What is “fluentd”? “ ” Fluentd is an open source

    data collector, which lets us unify the data collection and consumption for a better use and understanding of data.
  2. <match td.*.*> @id td_wild type tdlog buffer_type file max_retry_wait 1h

    … </match> { tag: td.global.recipe_search_query_logs, timestamp: xxx, country: xxx, language: xxx, … } ```ruby Fluent::Logger::FluentLogger.new(tag_prefix, args).post(tag, map) ```
  3. ?

  4. Unified logging layer for mobile apps Puree has following features

    Buffering Filtering Batching Retrying Pluggable
  5. Search “chicken” public class RecipeSearchLog implements PureeLog { @SerializedName("event") String

    event; @SerializedName("keyword") String keyword; @SerializedName("order") String order; @SerializedName("page") int page; @SerializedName("per_page") int perPage; @SerializedName("total_hits") int totalHits; } Puree.send(new RecipeSearchLog(…)); {“event”:”recipe.search”, “keyword”:”chicken”,…} Track events
  6. Buffering & Batching log logs {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”,

    “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ]
  7. image size image format network type S3 fetch time image

    convert time CDN cache hit ratio local cache hit ratio decode time render time number of threads device’s CPU / RAM OS version ?
  8. { "event": "image.download", "cache": "TCP_MISS", "cache_remote": "TCP_HIT", "content_length": "17670", "content_type":

    "image/webp", "convert_time_ms": "64.445", "network_type": "4G", "s3_fetch_time_ms": "41.575", "total_time_ms": 1471, … } SELECT cache, network_type, ROUND(AVG(total_time_ms), 0) AS total_time_ms FROM image_download_logs WHERE TD_TIME_RANGE(…) GROUP BY cache, network_type SamplingFilter (1%) TimestampFilter
  9. Collecting logs from mobile apps {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”, “keyword”:”chicken”,…} {“event”:”recipe.search”,

    “keyword”:”chicken”,…} [ {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…}, {“event”:”recipe.search”, “keyword”:”chicken”,…} ] add user_id, country, … {“tag”:”td.global.recipe_search”, “user_id”:”xxx”, …} SELECT * FROM table WHERE …