Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rong Peng - TACT: A Transfer Actor-Critic Learn...

SCEE Team
December 02, 2013

Rong Peng - TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks

SCEE Team

December 02, 2013
Tweet

More Decks by SCEE Team

Other Decks in Research

Transcript

  1. TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in

    Cellular Radio Access Networks LI RONGPENG ZHEJIANG UNIVERSITY EMAIL: [email protected] WEB: HTTP://WWW.RONGPENG.INFO
  2. Content Greener Cellular Networks Reinforcement Learning (RL) Framework for Greener

    Cellular Networks Transfer Learning (TL): Further Improvement of RL • Why it is important? • How to achieve that? • What is RL? • How to apply RL? • The Motivation • The Means • The Performance
  3. Weather in Hangzhou, the So- called Paradise in China Temperature

    of Hangzhou, Aug. 12 Highest: 41 ℃ Lowest: 28 ℃
  4. Greener Cellular Networks: “Research for the Future” Overall Energy Consumption

    of ICT: Equivalent to The Aviation Industry 2.5% Power Grid Consumption in CMCC: 81.4% 63%: Access Networks 50%-80%: 10M+ Base Stations (Year 2012) Source: China Mobile Research Institute, “Research over Networks Energy Savings Technique in Heterogeneous Networks”, Tech. Report, Nov. 2013.
  5. The Next-Generation Cellular networks Objectives: Green Wireless Network Requirement Explosive

    Traffic Demands Means: More Power? More bandwidth?  Advanced Physical Layer Technologies  Cooperative MIMO, Spatial Multiplexing, Interference Mitigation  Advanced Architecture  Cloud RAN, HetNet, Massive MMIMO Networks  “Network Intelligence” is the key.  Networks must grow and work where data is demanded. ) / 1 ( log 2 N P B C i i Channels    Increase Bandwidth Cognitive Radio More Channels MIMO Increase Power Cooperative systems Not Green! Limited Help Courtesy to the public reports of Prof. Vincent Lau (Zhejiang University Seminar, March 2013) and Dr. Jeffrey G. Andrews (University of Notre Dame Seminar, May 2011).
  6. Temporal Characteristics of Traffic Loads • Rongpeng Li, Zhifeng Zhao,

    Yan Wei, Xuan Zhou, and Honggang Zhang, “GM-PAB: A Grid-based Energy Saving Scheme with Predicted Traffic Load Guidance for Cellular Networks,” in Proceedings of IEEE ICC 2012, Ottawa, Canada, June 2012. • Rongpeng Li, Zhifeng Zhao, Xuan Zhou, and Honggang Zhang, “Energy Savings Scheme in Radio Access Network via Compressed Sensing Based Traffic Load Prediction,” Transactions on Emerging Telecommunications Technologies (ETT), Nov. 2012. BSs are deployed on the basis of peak traffic loads. How to make it green at low traffic loads?
  7. Energy Saving Scheme through BS Switching Operation Towards traffic load-aware

    BSs adaptation.  Turn some BSs into sleeping mode on the basis of minimizing the power consumption when the traffic loads are low.  Zooming other BS in a coordinated manner. To reliably predict the traffic loads is still quite challenging. One BS’s power consumption is partly related to the traffic loads within its coverage. Compared to the previous myopic schemes, a foresighted energy saving scheme is heavily needed. Actually, we can use the Reinforcement learning. BSs into Sleeping mode User Association Traffic Loads Distributed in Active BSs Power Consumption
  8. Machine Learning Supervised Learning • Data • Desired Signal/Teacher Reinforcement

    Learning • Data • Rewards/Punishments Unsupervised Learning • Only the Data
  9. Reinforcement Learning Reinforcement learning (RL) is learning by interacting with

    an environment. An RL agent learns from the consequences of its actions to maximize the accumulated reward over time, rather than from being explicitly taught and it selects its actions on basis of its past experiences (exploitation) and also by new choices (exploration), which is essentially trial and error learning. -- Scholarpedia
  10. Start S2 S3 S4 S5 Goal S7 S8 Arrows indicate

    strength between two problem states Start maze …
  11. Start S2 S3 S4 S5 Goal S7 S8 The first

    response leads to S2 … The next state is chosen by randomly sampling from the possible next states (i.e., directions).
  12. Start S2 S3 S4 S5 Goal S7 S8 Suppose the

    randomly sampled response leads to S3 …
  13. Start S2 S3 S4 S5 Goal S7 S8 At S3,

    choices lead to either S2, S4, or S7. S7 was picked (randomly)
  14. Start S2 S3 S4 S5 Goal S7 S8 And S5

    was chosen next (randomly)
  15. Start S2 S3 S4 S5 Goal S7 S8 Goal is

    reached, strengthen the associative connection between goal state and last response Next time S5 is reached, part of the associative strength is passed back to S4...
  16. Start S2 S3 S4 S5 Goal S7 S8 Let’s suppose

    after a couple of moves, we end up at S5 again
  17. Start S2 S3 S4 S5 Goal S7 S8 S5 is

    likely to lead to GOAL through strengthened route In reinforcement learning, strength is also passed back to the last state This paves the way for the next time going through maze
  18. Start S2 S3 S4 S5 Goal S7 S8 The situation

    after lots of restarts …
  19. Markov Decision Process (MDP) An MDP =< , , ,

    >  State space : ()  Action space : ()  Transition probability  Cost/Reward function A strategy , from state () to an action ) ( = () to minimize the value function starting from the state
  20. Bellman Equation and Optimal Strategy: The Methodology Bellman equation and

    optimal strategy ∗ ∗() = ∗ () = min∈ ∗ (, ) + ′∈ (′|, )∗ (′ Two important sub-problems to find the optimal strategy and the value function Action Selection Value Function Approximation
  21. Action Selection Action Selections is actually a tradeoff between exploration

    and exploitation.  Exploration: Increase the agent’s knowledge base;  exploitation: Leverage existing but under-utilized knowledge base Assume that the agent has actions to select  Greedy -Greedy  Choose the action with the largest reward with a probability of 1 −  Choose others with the largest reward with a probability of /( − 1)  Gibbs or Boltzmann Distribution  (, ) ∈ (, )  Temperature → 0: Greedy algorithm;  → ∞: Uniformly selecting the action Exploration Exploitation 3/4 1/12 1/12 1/12 9 10 1 2 Selected Action Reward/Cost 1 11 2 9 1 9 1 10 2 10
  22. Action Selection: From the Point of Game Best response argmax∈

    ) (, The discontinuities inherent in this maximization present difficulties for adaptive processes. Smooth best response ) argmax∈ ) (, + (, (, ) is a smooth, strictly differentiable concave function. If (, ) = − ) (, )(, , we can obtain the Boltzmann distribution.  By Lagrange Multiplier Algorithm, it equals that max∈ ) (, , subject to , = .
  23. State-Value Function Update/Approximation Iterations: The way to obtain a strategy

     Policy Update  State-Value Function Update Temporal Difference (TD) Error Example: +1 = 1 + 1 =1 +1 = 1 + 1 (+1 + =1 ) = 1 + 1 (+1 + ) = + + + − Newton’s Gradient Decent Method 1 3 2 4 Selected Action Reward/Cost 1 11 2 9 1 9 1 10 2 10
  24. Actor-Critic Algorithm The actor-critic algorithm encompasses three components: actor, critic,

    and environment. Actor: According to Boltzmann Distribution, select an action in a stochastic way and then executes it. Critic: Criticizes the action executed by the actor and updates the value function through TD error. (TD(0) and TD()) ( , ) = ( , ) + ⋅ ( +1 ) Value function Environment Policy Actor Critic state Cost TD error
  25. A Comparison among the Typical RL Algorithms Name TD Error

    Actor-critic • ( , ) = ( , ) + ⋅ ( + ) − SARSA (Station-Action- Reward-State-Action) • ( , ) = ( , ) + ⋅ ( + , + ) − Q-learning • ( , ) = ( , ) + ⋅ ′ ( + , ′) − State- Value Function Update Policy Update Q-Function replaces the State-Value Function It does not require a prior knowledge about the transition probability!
  26. RL Architecture for Energy Saving in Cellular Networks Environment: a

    region ℒ ∈ ℝ2 served by a set of BSs ℬ = {1, … , Controller: a BS switching operation controller to turn on/off some BSs in a centralized way; A traffic load density as = ) ( ) ( < ∞: arrival rate per unit area ) ( and file size 1 ) ( . Traffic load within BS 's coverage: = ℒ () (, ℬ )d  (, ℬ ) = 1 denotes location  is served by BS ∈ , vice versa. BS Switching Operation Controller Action BS 1: Active ⁞ BS i: Sleeping ⁞ BS N: Active Cost State Environment • Rongpeng Li, Zhifeng Zhao, Xianfu Chen, Jacques Palicot, and Honggang Zhang, “TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks,” summited to IEEE Transactions on Wireless Communications (Second Round Review). • Rongpeng Li, Zhifeng Zhao, Xianfu Chen, and Honggang Zhang, “Energy Saving through a Learning Framework in Greener Cellular Radio Access Networks,” in Proceedings of IEEE Globecom 2012, Anaheim, California, USA, Dec. 2012.
  27. Power Consumption Model and Problem Formulation All active BSs consumed

    power (, ℬ ) = ∈ℬ 1 − ) +  ∈ [0,1]: the portion of constant power consumption for BS ;  : the maximum power consumption of BS when it is fully utilized.  System load for BS ∈ ℬ : = ℒ () (, ℬ )d  System load density is defined as the fraction of time required to deliver traffic load ) ( from BS ∈ ℬ to location , namely ) () = ( ) (, ℬ . The delay optimal performance function  (, ℬ ) = ∈ℬ 1− Objection function minℬ, ) (, ℬ + ) (, ℬ  Subject to ∈ [0,1)∀ ∈ ℬ
  28. BS Traffic Load State Vector Finite state Markov process (FSMC)

    to demonstrate the traffic load variation condition; Traffic load for BS is partitioned into several parts by a boundary point ; Volume of Traffic BS 1 BS 2 BS 3 BS 4 BS 5 1 0 1 0 1 Boundary State Vector
  29. Bellman Equation • Accumulative cost () = = ∞ ()

    (), (())| = = (, ()) + ′∈ (′|, ()) ′ • Bellman equation and optimal strategy ∗ ∗() = ∗ ( = min∈ (, ) + ′∈ (′|, )∗ ′ BS Switching Operation Controller Action BS 1: Active ⁞ BS i: Sleeping ⁞ BS N: Active Cost State Environment
  30. Learning Framework based Energy Saving Scheme Detail Take the assumption

    that the system is at the beginning of stage , while the traffic load state is ().
  31. Learning Framework based Energy Saving Scheme Detail Action selection: the

    controller the controller selects an action () in state () with the probability (Boltzmann distribution) ()((), ()) = exp{((), ()) ()∈ exp {((), ()) After that, the corresponding BSs turns into sleeping mode.
  32. Learning Framework based Energy Saving Scheme Detail User association and

    data transmission: the users at location choose to connect one BS according to the following equation and start the data communication slot by slot. ∗() = argmax∈ℬon (, ℬon 1 − ) + 1 − −1 −2
  33. Learning Framework based Energy Saving Scheme Detail State-value function update:

    after the transmission part of stage , the traffic loads in each BS will change, and system will transform to state (+). A temporal difference error (()) = ()(, ) + · ((+1)) −
  34. Learning Framework based Energy Saving Scheme Detail Policy update: At

    the end of stage , “criticize” the selected action by ((), ()) ← ((), ()) − · (() . Remark: one action under a specific state can be selected with higher probability if the ``foresighted'' cost it takes is comparatively smaller.
  35. A Rethink of the Traffic Characteristics • Rongpeng Li, Zhifeng

    Zhao, Xuan Zhou, Jacques Palicot, and Honggang Zhang, “The Prediction Analysis of Cellular Radio Access Networks Traffic: From Entropy Theory To Network Practicing,” summited to IEEE Communications Magazine (Second Round Review).
  36. Transfer Actor-Critic Algorithm: Motivation Remaining issues  Temporal/Spatial relevancy in

    traffic loads  Difficulty in convergence for large station/action sets  Learning jumpstart The concept of transfer learning • Rongpeng Li, Zhifeng Zhao, Xianfu Chen, Jacques Palicot, and Honggang Zhang, “TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks,” summited to IEEE Transactions on Wireless Communications (Second Round Review).
  37. Transfer Policy Update ( 1) ( ) ( ) (

    ) ( ) ( 1) ( ) ( ) ( ) ( ) ( ) ( ) 2 2 ( , ) (1 ( ( , , ))) ( , ) ( ( , , )) ( , ) k k k o p k k k k k k k k k n e p k p k p               s a s a s a s a s a Native Policy Exotic Policy Transfer Rate • Transfer rate 2 ( , , • Incrementally decreases as the iterations run. • Diminishes the impact of exotic policy once the controller masters certain amount of information.