Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Is Rainfall Getting Heavier? Building a Weather...

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →

Is Rainfall Getting Heavier? Building a Weather Forecasting Pipeline with Singapore Weather Station Data

Event: FOSDEM 2021 Python devroomm
Date: 7 February 2021
Location: Remote

How many seasons does a tropical country like Singapore have? Is global warming real, and is rainfall getting heavier? To answer these questions, I will show how we could use Requests and Pandas to build a data pipeline that extracts Singapore weather station data for a user-defined time period and explore the weather trends and seasons over the past few years.

Ong Chin Hwee

February 07, 2021

Transcript

56 slides
  1. Is Rainfall Getting Heavier? Building a Weather Forecasting Pipeline with

    Singapore Weather Station Data By: Chin Hwee Ong (@ongchinhwee) 7 February 2021 FOSDEM Python devroom
  2. About me Ong Chin Hwee 王敬惠 • Data Engineer •

    Based in sunny Singapore 🌞 • Aerospace Engineering + Computational Modelling • Loves (and contributes to) pandas @ongchinhwee
  3. We have our “four seasons”: 1. Cold and Rainy 2.

    Warm and Dry 3. Extremely Hot 4. Hot and Stormy @ongchinhwee
  4. Since 2018, Singapore had more than 20 flash floods. Majority

    of the floods were caused by intense rain. Source: PUB Singapore (https://www.pub.gov.sg/drainage/floodmanagement/recentflashfloods) @ongchinhwee
  5. Realtime Weather Readings across Singapore Real-time API on Data.gov.sg (Singapore’s

    open data portal) Open government data available under the Singapore Open Data License (Almost) minute-by-minute weather station readings @ongchinhwee
  6. “Let’s try to scrap weather data for a specific weather

    station!” “How about we scrap multi-day data from the API?” @ongchinhwee
  7. Data.gov.sg Weather Data API Scraping Scraping weather data from APIs

    via “Requests” library “Requests”: Python library for humans to send HTTP requests @ongchinhwee
  8. Data.gov.sg Weather Data API Scraping Currently supported Data.gov.sg APIs: 1.

    Air Temperature (in °C) 2. Rainfall (in mm) 3. Relative Humidity 4. Wind Direction 5. Wind Speed Scrap data for continuous time range + specific weather station @ongchinhwee
  9. Design Considerations Slow connection - retry mechanism from retrying import

    retry @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000) def get_rainfall_data_from_date(date): @ongchinhwee
  10. Design Considerations Slow connection API working but no data for

    specific date - Return empty DataFrame with same column names as if there were data for specific date @ongchinhwee
  11. Design Considerations Slow connection API working but no data for

    specific date Nested JSON to pandas DataFrame conversion - Extract desired station and readings - Concatenate them back with timestamp @ongchinhwee
  12. Time Series Analysis of Singapore Rainfall Data Selected weather station:

    Changi Weather Station (ID: S24) Analysis timeframe: 2 Dec 2016 to 31 Dec 2020 (~4 years) Objective: - Extract trend and seasonality from 5-minute rainfall data @ongchinhwee
  13. Time Series Analysis for Forecasting Analyse and forecast time series

    using “statsmodels.tsa” “statsmodels” library: Python library for statistical models, tests and exploration “statsmodels.tsa”: Model classes and functions for Time Series Analysis @ongchinhwee
  14. Time Series Analysis for Forecasting Stationarity: Stationary vs Non-Stationary -

    Augmented Dickey-Fuller (ADF) Test Patterns: Trend, Seasonality, Cycles (and Noise) - Moving Averages - STL Decomposition Autocorrelation: Relationship between a time series and a lagged version of itself @ongchinhwee
  15. Autocorrelation of Monthly Rainfall Most positive: 1st cofficient (r 1

    ); Most negative: 3rd coefficient (r 3 ) @ongchinhwee
  16. Rainfall Forecasting with ARIMA models ARIMA(p,d,q) model (AutoRegressive Integrated Moving

    Average) where: p: order of the autoregressive part; d: degree of first differencing involved; q: order of the moving average part. @ongchinhwee
  17. Rainfall Forecasting with ARIMA models 1. Apply rolling forecast technique

    with ARIMA(p, d, q) on time series data 2. Minimise root-mean-squared-error (RMSE) 3. Use optimized order parameters (p, d, q) to run rolling forecast for next N cycles a. Daily Forecast: N = 61 b. Monthly Forecast: N = 13 @ongchinhwee
  18. Key Takeaways • With climate change, rainfall patterns are becoming

    more extreme and more challenging to predict ◦ Highest rainfall in December 2019 (NE Monsoon) ◦ Higher-than-expected rainfall in May 2020 (Inter-Monsoon) - also earlier-than-expected monsoon • Rainfall data from weather station + ARIMA may not be sufficient enough to predict more “erratic” spikes in daily rainfall @ongchinhwee
  19. Reach out to me! : ongchinhwee : @ongchinhwee : hweecat

    : https://ongchinhwee.me And check out my project on: hweecat/api-scraping-nea-datasets

More Decks by Ong Chin Hwee

Other Decks in Programming