Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Is Rainfall Getting Heavier? Building a Weather...

Is Rainfall Getting Heavier? Building a Weather Forecasting Pipeline with Singapore Weather Station Data

Event: FOSDEM 2021 Python devroomm
Date: 7 February 2021
Location: Remote

How many seasons does a tropical country like Singapore have? Is global warming real, and is rainfall getting heavier? To answer these questions, I will show how we could use Requests and Pandas to build a data pipeline that extracts Singapore weather station data for a user-defined time period and explore the weather trends and seasons over the past few years.

Ong Chin Hwee

February 07, 2021
Tweet

More Decks by Ong Chin Hwee

Other Decks in Programming

Transcript

  1. Is Rainfall Getting Heavier? Building a Weather Forecasting Pipeline with

    Singapore Weather Station Data By: Chin Hwee Ong (@ongchinhwee) 7 February 2021 FOSDEM Python devroom
  2. About me Ong Chin Hwee 王敬惠 • Data Engineer •

    Based in sunny Singapore 🌞 • Aerospace Engineering + Computational Modelling • Loves (and contributes to) pandas @ongchinhwee
  3. We have our “four seasons”: 1. Cold and Rainy 2.

    Warm and Dry 3. Extremely Hot 4. Hot and Stormy @ongchinhwee
  4. Since 2018, Singapore had more than 20 flash floods. Majority

    of the floods were caused by intense rain. Source: PUB Singapore (https://www.pub.gov.sg/drainage/floodmanagement/recentflashfloods) @ongchinhwee
  5. Realtime Weather Readings across Singapore Real-time API on Data.gov.sg (Singapore’s

    open data portal) Open government data available under the Singapore Open Data License (Almost) minute-by-minute weather station readings @ongchinhwee
  6. “Let’s try to scrap weather data for a specific weather

    station!” “How about we scrap multi-day data from the API?” @ongchinhwee
  7. Data.gov.sg Weather Data API Scraping Scraping weather data from APIs

    via “Requests” library “Requests”: Python library for humans to send HTTP requests @ongchinhwee
  8. Data.gov.sg Weather Data API Scraping Currently supported Data.gov.sg APIs: 1.

    Air Temperature (in °C) 2. Rainfall (in mm) 3. Relative Humidity 4. Wind Direction 5. Wind Speed Scrap data for continuous time range + specific weather station @ongchinhwee
  9. Design Considerations Slow connection - retry mechanism from retrying import

    retry @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000) def get_rainfall_data_from_date(date): @ongchinhwee
  10. Design Considerations Slow connection API working but no data for

    specific date - Return empty DataFrame with same column names as if there were data for specific date @ongchinhwee
  11. Design Considerations Slow connection API working but no data for

    specific date Nested JSON to pandas DataFrame conversion - Extract desired station and readings - Concatenate them back with timestamp @ongchinhwee
  12. Time Series Analysis of Singapore Rainfall Data Selected weather station:

    Changi Weather Station (ID: S24) Analysis timeframe: 2 Dec 2016 to 31 Dec 2020 (~4 years) Objective: - Extract trend and seasonality from 5-minute rainfall data @ongchinhwee
  13. Time Series Analysis for Forecasting Analyse and forecast time series

    using “statsmodels.tsa” “statsmodels” library: Python library for statistical models, tests and exploration “statsmodels.tsa”: Model classes and functions for Time Series Analysis @ongchinhwee
  14. Time Series Analysis for Forecasting Stationarity: Stationary vs Non-Stationary -

    Augmented Dickey-Fuller (ADF) Test Patterns: Trend, Seasonality, Cycles (and Noise) - Moving Averages - STL Decomposition Autocorrelation: Relationship between a time series and a lagged version of itself @ongchinhwee
  15. Autocorrelation of Monthly Rainfall Most positive: 1st cofficient (r 1

    ); Most negative: 3rd coefficient (r 3 ) @ongchinhwee
  16. Rainfall Forecasting with ARIMA models ARIMA(p,d,q) model (AutoRegressive Integrated Moving

    Average) where: p: order of the autoregressive part; d: degree of first differencing involved; q: order of the moving average part. @ongchinhwee
  17. Rainfall Forecasting with ARIMA models 1. Apply rolling forecast technique

    with ARIMA(p, d, q) on time series data 2. Minimise root-mean-squared-error (RMSE) 3. Use optimized order parameters (p, d, q) to run rolling forecast for next N cycles a. Daily Forecast: N = 61 b. Monthly Forecast: N = 13 @ongchinhwee
  18. Key Takeaways • With climate change, rainfall patterns are becoming

    more extreme and more challenging to predict ◦ Highest rainfall in December 2019 (NE Monsoon) ◦ Higher-than-expected rainfall in May 2020 (Inter-Monsoon) - also earlier-than-expected monsoon • Rainfall data from weather station + ARIMA may not be sufficient enough to predict more “erratic” spikes in daily rainfall @ongchinhwee
  19. Reach out to me! : ongchinhwee : @ongchinhwee : hweecat

    : https://ongchinhwee.me And check out my project on: hweecat/api-scraping-nea-datasets