Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The evolution of a rabbit

nealio82
January 21, 2015

The evolution of a rabbit

How our API call queue system evolved from a single 'select rows from the database', through a MySql simple queue and finally to using a dedicated message queue with Rabbit MQ.

Watch a video of the original presentation here: https://www.youtube.com/watch?v=iCg6t5iedEw

nealio82

January 21, 2015
Tweet

More Decks by nealio82

Other Decks in Programming

Transcript

  1. SELECT [cols] FROM `adverts_tbl` WHERE `update_flag` = 'U'; Problems: •

    Only&a&certain&amount&can&be&processed&within&a&minute • Might&only&select&a&few&rows&but&need&to&check&hundreds&of& thousands&in&table • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Data&comparison&performed&on&nonCindexed&column
  2. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag`

    = 'U'; Benefits: • Selec&ng)from)fewer)rows) • Joining)on)PKs)&)indexes • Faster!)(but)not)benchmarked)
  3. SELECT [cols] FROM `queue_tbl` q INNER JOIN `adverts_tbl` WHERE q.`update_flag`

    = 'U'; Problems: • Only&a&certain&amount&can&be&processed&within&a&minute • Big&data&read&spike&on&DB&&&CPU&spike&every&60&seconds • Complexity&increases&as&different&queue&types&become&required& (13&queue&tables)
  4. * sync progress table * sync users table * sync

    promoted tweets table * sync targeting table * campaign creation queue table * campaign update / deletion queue table * tweet creation queue table * ads queue table * ads targeting queue * ads account queue * ads tweets queue * ads embedded creatives updates table
  5. <?php use path\to\TwitterQueue; class Campaign { public function queue() {

    return new TwitterQueue('CampaignQueue', 'PromotedTweetId', $this->getId()); } // …
  6. <?php class TwitterQueue { public function __construct($queue_class, $finder_column, $entity_id =

    null) { $this->queue_class = $queue_class; $this->finder_column = $finder_column; if ($entity_id) { return $this->getQueue($entity_id); } $this->makeNewQueue(); return $this; } // …
  7. Inside'a'task fetch [x] campaigns / adverts from database loop rows

    fetch access token & other required info from database perform api call handle response & save back to db
  8. <?php use path\to\TwitterQueue; class Campaign { public function queue($action) {

    (new TwitterQueue)->Queue('Campaign', $this->createMessage($action)); } private function createMessage($action) { return json_encode(array( 'access_token' => $this->getAccount()->getAccessToken(), 'call_url' => 'https://ads.twitter.com/campaigns', 'method' => strtoupper($action), 'data' => $this->getURIParams() )); } // …
  9. <?php use PhpAmqpLib\Connection\AMQPConnection; use PhpAmqpLib\Message\AMQPMessage; class TwitterQueue { private $connection,

    $channel; public function __construct() { $this->connection = new AMQPConnection('localhost', 5672, 'guest', 'guest'); $this->channel = $this->connection->channel(); } public function Queue($queue_name, $content) { // prevent queue from auto-deleting with all the falses. $this->channel->queue_declare($queue_name, false, false, false, false); // pass $queue_name as the routing key $this->channel->basic_publish(new AMQPMessage($content), '', $queue_name); } // …