Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Is Rainfall Getting Heavier? Building a Weather...

Is Rainfall Getting Heavier? Building a Weather Forecasting Pipeline with Singapore Weather Station Data

Keynote at:
Event: PyCode Conference 2020
Date: 12 December 2020
Location: Remote

Premiered at:
Event: Pyjamas Conf 2020
Date: 6 December 2020
Location: Remote

How many seasons does a tropical country like Singapore have? Is rainfall getting heavier? To answer these questions, we will explore how to build a data pipeline that extracts Singapore weather station data, so that we can explore weather trends and attempt to forecast the weather using the data.

Ong Chin Hwee

December 12, 2020
Tweet

More Decks by Ong Chin Hwee

Other Decks in Programming

Transcript

  1. Is Rainfall Getting Heavier? Building a Weather Forecasting Pipeline with

    Singapore Weather Station Data By: Chin Hwee Ong (@ongchinhwee) 12 December 2020
  2. About me Ong Chin Hwee 王敬惠 • Data Engineer •

    Based in sunny Singapore • Aerospace Engineering + Computational Modelling • Loves (and contributes to) pandas @ongchinhwee
  3. We have our “four seasons”: 1. Cold and Rainy 2.

    Warm and Dry 3. Extremely Hot 4. Hot and Stormy @ongchinhwee
  4. Since 2018, Singapore had more than 20 flash floods. Majority

    of the floods were caused by intense rain. Source: PUB Singapore (https://www.pub.gov.sg/drainage/floodmanagement/recentflashfloods) @ongchinhwee
  5. Realtime Weather Readings across Singapore Real-time API on Data.gov.sg (Singapore’s

    open data portal) Open government data available under the Singapore Open Data License (Almost) minute-by-minute weather station readings @ongchinhwee
  6. “Let’s try to scrap weather data for a specific weather

    station!” “How about we scrap multi-day data from the API?” @ongchinhwee
  7. Data.gov.sg Weather Data API Scraping Scraping weather data from APIs

    via “Requests” library “Requests”: Python library for humans to send HTTP requests @ongchinhwee
  8. Data.gov.sg Weather Data API Scraping Currently supported Data.gov.sg APIs: 1.

    Air Temperature (in °C) 2. Rainfall (in mm) 3. Relative Humidity 4. Wind Direction 5. Wind Speed Scrap data for continuous time range + specific weather station @ongchinhwee
  9. Design Considerations Slow connection - retry mechanism from retrying import

    retry @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000) def get_rainfall_data_from_date(date): @ongchinhwee
  10. Design Considerations Slow connection API working but no data for

    specific date - Return empty DataFrame with same column names as if there were data for specific date @ongchinhwee
  11. Design Considerations Slow connection API working but no data for

    specific date Nested JSON to pandas DataFrame conversion - Extract desired station and readings - Concatenate them back with timestamp @ongchinhwee
  12. Time Series Analysis of Singapore Rainfall Data Selected weather station:

    Changi Weather Station (ID: S24) Analysis timeframe: 2 Dec 2016 to 30 Nov 2020 (~4 years) Objective: - Extract trend and seasonality from 5-minute rainfall data @ongchinhwee
  13. Time Series Analysis for Forecasting Analyse and forecast time series

    using “statsmodels.tsa” “statsmodels” library: Python library for statistical models, tests and exploration “statsmodels.tsa”: Model classes and functions for Time Series Analysis @ongchinhwee
  14. Time Series Analysis for Forecasting Stationarity: Stationary vs Non-Stationary -

    Augmented Dickey-Fuller (ADF) Test Patterns: Trend, Seasonality, Cycles (and Noise) - Moving Averages - STL Decomposition Autocorrelation: Relationship between a time series and a lagged version of itself @ongchinhwee
  15. Augmented Dickey-Fuller (ADF) Stationary Test from statsmodels.tsa.stattools import adfuller def

    ADF_test(timeseries): dftest = adfuller(timeseries.dropna(), autolag="AIC") print("Test statistic = {:.3f}".format(dftest[0])) print("P-value = {:.3f}".format(dftest[1])) print("Critical values :") for k, v in dftest[4].items(): print( f"\t{k}%: {v:.3f} - The data is {"not" if v < dftest[0] else ""} stationary with {100 - int(k[:-1])}% confidence") @ongchinhwee
  16. Augmented Dickey-Fuller (ADF) Stationary Test Total Daily Rainfall Test statistic

    = -5.710 P-value = 0.000 Critical values : 1%: -3.585 - The data is stationary with 99% confidence 5%: -2.928 - The data is stationary with 95% confidence 10%: -2.602 - The data is stationary with 90% confidence Monthly Daily Rainfall Test statistic = -13.590 P-value = 0.000 Critical values : 1%: -3.435 - The data is stationary with 99% confidence 5%: -2.864 - The data is stationary with 95% confidence 10%: -2.256 - The data is stationary with 90% confidence @ongchinhwee
  17. Autocorrelation of Monthly Rainfall Most positive: 1st cofficient (r 1

    ); Most negative: 3rd coefficient (r 3 ) @ongchinhwee
  18. Rainfall Forecasting with ARIMA models ARIMA(p,d,q) model (AutoRegressive Integrated Moving

    Average) where: p: order of the autoregressive part; d: degree of first differencing involved; q: order of the moving average part. @ongchinhwee
  19. Rainfall Forecasting with ARIMA models 1. Apply rolling forecast technique

    with ARIMA(p, d, q) on time series data 2. Minimise root-mean-squared-error (RMSE) 3. Use optimized order parameters (p, d, q) to run rolling forecast for next N cycles a. Daily Forecast: N = 60 b. Monthly Forecast: N = 12 @ongchinhwee
  20. Key Takeaways • With climate change, rainfall patterns are becoming

    more extreme and more challenging to predict ◦ Highest rainfall in December 2019 (NE Monsoon) ◦ Higher-than-expected rainfall in May 2020 (Inter-Monsoon) - also earlier-than-expected monsoon • Rainfall data from weather station + ARIMA may not be sufficient enough to predict more “erratic” spikes in daily rainfall @ongchinhwee
  21. Reach out to me! : ongchinhwee : @ongchinhwee : hweecat

    : https://ongchinhwee.me And check out my project on: hweecat/api-scraping-nea-datasets