Share
Embed
PyConJP 2016: pandas による 時系列データ処理
Sinhrks
September 22, 2016
Transcript
64 slides
-
ࣗݾհ • @sinhrks • ۀ: σʔλੳ • OSS׆ಈ: PyData Development
Team (pandas) Dask Development Team (Dask) • GitHub: https://github.com/sinhrks -
pandasͱ • σʔλੳͷͨΊͷσʔλߏͱɺσʔλͷલॲཧ / ूܭʹ͓ ͍ͯศརͳؔ / ϝιουΛఏڙ • Rͷ
“data.frame” + α • ࡞ऀ: Wes McKinney • ϥΠηϯε: BSD • ҙຯ: PANel DAta System • GitHub: 7000↑⭐️ -
pandasΛ͏ϝϦοτ • ݱ࣮ͷ(Ԛ͍)σʔλʹରԠ • ײతͳૢ࡞ • ߴ • ࢀߟ: pandas
internals @PyConJP 2015 -
DataFrame • 2࣍ݩͷσʔλߏ: • ߦ (index) ͱ ྻ(columns) ʹϥϕϧΛ࣋ͭ •
ྻ͝ͱʹܕΛ࣋ͭ $PMVNOT *OEFY JOUܕ PCKFDUܕ -
import pandas as pd df = pd.read_csv(‘adult.csv’) df DataFrame "EVMU%BUBTFUUBLFOGSPN6$*.-3FQPTJUPSZ
-JDINBO . 6$*.BDIJOF-FBSOJOH3FQPTJUPSZ<IUUQBSDIJWFJDTVDJFEVNM>*SWJOF $"6OJWFSTJUZPG$BMJGPSOJB 4DIPPMPG*OGPSNBUJPOBOE$PNQVUFS4DJFODF $47ϑΝΠϧͷಡΈࠐΈ -
pandasͷػೳ • ϕΫτϧԽ͞Εͨܭࢉ • άϧʔϓԽ ू (split-apply-combine) • มܗ
(merge, join, concat…) • ଟ༷ͳೖग़ྗ (SQL, CSV, Excel, …) • ॊೈͳ࣌ܥྻσʔλॲཧ • ՄࢹԽ -
ڥ • όʔδϣϯ • Python 3.5.2 • pandas 0.19.0rc1 •
statsmodels 0.8.0rc1 • ໊લۭؒ import numpy as np import pandas as pd import matplotlib.pyplot as plt import statsmodels.api as sm -
ݱ࣮ͷσʔλ • ඞཁͳपظ͕ҟͳΔ • ࣍σʔλΛ݄࣍Ͱੳ͍ͨ͠ • पظతͰͳ͍ • Πϕϯτͷൃੜ͝ͱʹه͞ΕͨϩάΛੳ͍ͨ͠ •
࣌ؒͰϥϕϧ͚͞Ε͍ͯͳ͍ • ࣌Λྻͱؚͯ͠ΉੜσʔλΛɺ͋Δपظ͝ͱʹूܭͯ͠ੳ͍ͨ͠ ԿΒ͔ͷલॲཧΛߦ͍ѻ͍͍͢ܗʹ͢Δ ੜσʔλ ࣌ܥྻσʔλ -
࣌ܥྻσʔλͷ४උ values = [datetime.datetime(2001, 1, 1), datetime.datetime(2001, 2, 1), datetime.datetime(2001,
3, 1)] s = pd.Series(np.arange(3), index=values) s 2001-01-01 0 2001-02-01 1 2001-03-01 2 dtype: int64 ࣌ͷϦετ ࣌Λϥϕϧͱ͢Δ Ұ࣍ݩσʔλ 4FSJFT df = pd.DataFrame({'A': [25, 27, 30], 'B': [10, 15, 17]}, index=values) df ࣌Λϥϕϧͱ͢Δ ೋ࣍ݩσʔλ %BUB'SBNF -
࣌ܥྻσʔλͷϥϕϧ df.index DatetimeIndex(['2001-01-01', '2001-02-01', '2001-03-01'], dtype='datetime64[ns]', freq=None) df df['A'] 2001-01-01
25 2001-02-01 27 2001-03-01 30 Name: A, dtype: int64 ϥϕϧ࣌ͷܕΛ࣋ͭ %BUFUJNF*OEFY ϥϕϧ JOEFY ྻͷબ -
γʔέϯεͷੜ (pd.date_range) s = pd.Series(np.arange(10)) s 0 0 1 1
2 2 dtype: int64 s.index = pd.date_range('2001-01-01', freq='M', periods=3) s 2001-01-31 0 2001-02-28 1 2001-03-31 2 Freq: M, dtype: int64 ϥϕϧ JOEFY Λ্ॻ͖ pd.date_range('2001-01-01', freq='M', periods=3) DatetimeIndex(['2001-01-31', '2001-02-28', '2001-03-31'], dtype='datetime64[ns]', freq='M') ͔Β݄࣍Ͱݸ ͷσʔλΛ࡞ ࣌ͷϥϕϧ͕ͳ͍σʔλ -
Frequency String • ੜ͢Δ࣌ܥྻͷपظΛࢦఆ͢Δ • ଞɺશ25छྨ 'SFRVFODZ4USJOH ҙຯ "
. ݄ 8 ि % ) ࣌ 5 4 ඵ -
Frequency String pd.date_range('2016-01-01', freq='M', periods=3) DatetimeIndex(['2016-01-31', '2016-02-29', ‘2016-03-31’], dtype='datetime64[ns]', freq='M')
pd.date_range('2016-01-01', freq='MS', periods=3) DatetimeIndex(['2016-01-01', '2016-02-01', '2016-03-01'], dtype='datetime64[ns]', freq='MS') pd.date_range('2016-01-01', freq='W', periods=3) DatetimeIndex(['2016-01-03', '2016-01-10', '2016-01-17'], dtype='datetime64[ns]', freq='W-SUN') pd.date_range('2016-01-01', freq='W-TUE', periods=3) DatetimeIndex(['2016-01-05', '2016-01-12', '2016-01-19'], dtype='datetime64[ns]', freq='W-TUE') 856& ि Ր༵࢝·Γ 8ि .4݄ॳ .݄ -
࣌ͷύʔε (pd.to_datetime) • ࣌จࣈྻΛߴʹύʔε • C Parser ਖ਼نදݱ
dateutil pd.to_datetime(['2016-09-22', '2016-09-23']) DatetimeIndex(['2016-09-22', ‘2016-09-23'], dtype='datetime64[ns]', freq=None) pd.to_datetime(['September 22nd, 2016', 'September 22nd, 2016']) DatetimeIndex(['2016-09-22', ‘2016-09-22’], dtype='datetime64[ns]', freq=None) pd.to_datetime(['22 Sep 2016', '23 Sep 2016']) DatetimeIndex(['2016-09-22', ‘2016-09-23'], dtype='datetime64[ns]', freq=None) -
࣌ͷύʔε (pd.to_datetime) • ϑΥʔϚοτࢦఆʹΑΔॊೈͳύʔεՄೳ pd.to_datetime(['20169݄22', '20169݄23']) ValueError: Unknown string format
pd.to_datetime(['20169݄22', '20169݄23'], format='%Y%m݄%d') DatetimeIndex(['2016-09-22', ‘2016-09-23'], dtype='datetime64[ns]', freq=None) -
σʔλબ • Γ͍ͨ͜ͱ • 1. ͋Δ࣌Λબ͍ͨ͠ • 2. ͋ΔظؒΛબ͍ͨ͠ •
3. ͋Δ݅Λຬͨ࣌͢Λબ͍ͨ͠ -
σʔλબ idx = pd.date_range('2016-01-01', freq='D', periods=366) df = pd.DataFrame({'A': np.random.randint(100,
size=366), 'B': np.random.randint(100, size=366)}, index=idx) df.loc[datetime.datetime(2016, 1, 2)] A 12 B 64 Name: 2016-01-02 00:00:00, dtype: int64 ͷߦΛબ ݁Ռ4FSJFT df.loc['2016-01-02'] A 12 B 64 Name: 2016-01-02 00:00:00, dtype: int64 จࣈྻ࣌ͱͯ͠ѻΘΕΔ df -
݅ʹΑΔબ df.index.month df.loc[(df.index.month == 1) | (df.index.month == 3)] array([
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 12, 12, 12, 12, 12, 12, 12, 12, 12], dtype=int32) ϕΫτϧԽ͞Εͨ ϓϩύςΟΞΫηε (df.index.month == 1) | (df.index.month == 3) array([ True, True, True, True, True, True, True, ... False, False, False, False, False, False], dtype=bool) -
ϓϩύςΟΞΫηε • ࣌ͷଐੑʹԠͨ͡ॲཧ͕؆୯ʹॻ͚Δ ϓϩύςΟ ϓϩύςΟ ZFBS EBUF NPOUI UJNF EBZ
EBZPGZFBS IPVS XFFLPGZFBS NJOVUF XFFL TFDPOE EBZPGXFFL NJDSPTFDPOE XFFLEBZ OBOPTFDPOE XFFLEBZ@OBNF RVBSUFS -
࣌σʔλͷલॲཧ • Γ͍ͨ͜ͱ • 1. ࣌ͷपظΛม͍͑ͨ • 2. ܽଛΛิ͍ͨ͠ •
3. લޙͷͱൺֱ / ܭࢉ͍ͨ͠ -
ϦαϯϓϦϯά (.resample) • αϯϓϧσʔλ • ਖ਼نཚͷྦྷੵ (ϥϯμϜΥʔΫ) idx = pd.date_range('2016-09-22',
freq='H', periods=50) df = pd.DataFrame({'val': np.random.randn(50)}, index=idx) df = df.cumsum() df -
ϦαϯϓϦϯά (.resample) • ༷ʑͳू͕Մೳ ूϝιου ूϝιου ⒏MM NFEJBO CBDLpMM NJO
QBE PIMD pMMOB QSPE JOUFSQPMBUF TJ[F DPVOU TFN OVOJRVF TUE pSTU MB TVN MBTU WBS NBY -
ิ (.interpolate) • αϯϓϧσʔλ • ʹܽଛ (NaN) ΛؚΉ indexer =
np.random.randint(4, size=50) == 1 df.loc[indexer] = np.nan df ܽଛ ܽଛ -
γϑτ (.shift) • Λࢦఆ͞Εͨ periods ͚ͩͣΒ͢ • લޙͷͱͷൺֱ / ܭࢉΛ͢Δࡍʹศར
df.shift(periods=1) -
ࠩ (.diff) • ࢦఆ͞Εͨ periods ͱͷࠩΛͱΔ • df - df.shift()
ͱಉ͡ df.diff(periods=1) -
γϑτͷ͍ํ (ྫ) idx = pd.date_range('2016-09-22 10:00', freq='T', periods=50) df =
pd.DataFrame({'val': np.repeat([0, 1, 0, 1], [10, 20, 10, 10])}, index=idx) df df.index[df['val'] != df[‘val'].shift()] DatetimeIndex(['2016-09-22 10:00:00', '2016-09-22 10:10:00', '2016-09-22 10:30:00', '2016-09-22 10:40:00'], dtype='datetime64[ns]', freq=None) -
࣌σʔλͷूܭ • αϯϓϧσʔλ • ͷൃσʔλ df = pd.DataFrame({'ྔ': np.random.randint(100, size=1000),
'໊': np.random.choice(list('ABC'), 1000), 'ൃ': np.random.choice(idx, 1000)}) df -
ϓϩύςΟΞΫηε (.dt) df['ൃ'].dt.weekday df.groupby(df['ൃ'].dt.weekday).sum() 0 6 1 4 2 6
.. 997 6 998 5 999 2 Name: ൃ, dtype: int64 EUϓϩύςΟΛ௨ͯ͡ɺ ࣌ϓϩύςΟͷΞΫηε͕Մೳ -
Ϋϩεूܭ (pd.pivot_table) • pd.pivot_table + • pd.Grouper • ϓϩύςΟΞΫηε pd.pivot_table(df,
index=pd.Grouper(key='ൃ', freq='M'), columns='໊', values='ྔ', aggfunc='sum') -
ΧϨϯμʔ • pandas.tseries.offsets.CustomBusinessDay • ॕΛߟྀͨ͠ॲཧ • japandas (https://github.com/sinhrks/japandas) • JapaneseHolidayCalendar
from pandas.tseries.offsets import CustomBusinessDay import japandas cal = japandas.JapaneseHolidayCalendar() cbd = CustomBusinessDay(calendar=cal) idx = pd.DatetimeIndex(['2016-09-20', '2016-09-21', ‘2016-09-22']) idx + cbd DatetimeIndex(['2016-09-21', '2016-09-23', '2016-09-23'], dtype='datetime64[ns]', freq=None) ॕ -
ՄࢹԽ • ࣌ܥྻσʔλͷपظΛࣗಈͰௐͯ͠ϓϩοτ idx1 = pd.date_range('2016-09-01', freq='D', periods=50) df1 =
pd.DataFrame({'val1': np.random.randn(50)}, index=idx1) df1.plot() -
ՄࢹԽ • ࣌ܥྻσʔλͷपظΛࣗಈͰௐͯ͠ϓϩοτ • पظ͕ҟͳΔ߹ࣗಈௐ idx2 = pd.date_range('2016-09-01', freq='M', periods=3)
df2 = pd.DataFrame({'val1': np.random.randn(3)}, index=idx2) ax = df1.plot() df2.plot(ax=ax) -
࣌ܥྻσʔλͷ౷ܭϞσϧ • త • ࣌ܥྻͷؔΛௐ͍ͨ • কདྷͷ༧ଌΛ͍ͨ͠ • มԽ /
ҟৗΛݕ͍ͨ͠ • … • ࣌ܥྻσʔλͷཹҙ • ͋Δ࣌Ҏલͷσʔλ͔ΒͷӨڹ͕͋Δ͔ʁ • τϨϯυقઅੑ͕͋Δ͔ʁ -
࣌ܥྻϞσϧΛؚΈPythonύοέʔδ • ར༻͍ͨ͠Ϟσϧʹ߹ΘͤͯύοέʔδΛબͿ • ඞཁʹԠ͡ R Λར༻ (rpy2, pypeR) 4UBUT.PEFMT
1Z'MVY ౷ܭྔݕఆ ✅ "3*." ✅ ✅ 7"3 ✅ ✅ ("3$) ✅ TBOECPY ✅ ("4 ✅ 4UBUF4QBDF ✅ SD ✅ -
αϯϓϧσʔλ • AirPassengers • ݄࣍ͷࠃࡍઢਓ (ઍਓ) • ୯มྔɺτϨϯυͱقઅੑΛ࣋ͭ df =
pd.read_csv('airpassengers.csv', index_col=0, parse_dates=[0]) df -
౷ܭྔ • ඪຊࣗݾ૬ؔ (ACF) • ҟ࣌ؒͷڞࢄΛඪ४Խͨ͠ͷ • ඪຊภࣗݾ૬ؔ (PACF) fig,
axes = plt.subplots(1, 2) sm.tsa.graphics.plot_acf(df, ax=axes[0]); sm.tsa.graphics.plot_pacf(df, ax=axes[1]); -
౷ܭྔ • ਖ਼نཚ (ϗϫΠτϊΠζ) ͷ߹ • աڈͷͱ૬͕ؔͳ͍ wn = pd.Series(np.random.randn(100))
fig, axes = plt.subplots(1, 2) sm.tsa.graphics.plot_acf(wn, ax=axes[0]); sm.tsa.graphics.plot_pacf(wn, ax=axes[1]); -
SARIMAϞσϧ • قઅతࣗݾճؼҠಈฏۉϞσϧ • ࣗݾճؼҠಈฏۉϞσϧ (ARIMA) • + قઅมಈ (ARIMA)
• ARIMA (p, d, q) • d֊ࠩΛͱͬͨ࣌ܥྻ yt ͕ • (ऑ)ఆৗ (ฏۉɺࣗݾڞࢄ͕࣌ؒʹΑΒͣҰఆ) • ҎԼͷաఔʹै͏ • yt = c + φ1yt-1 + … + φpyp + εt + θ1εt-1 + … + θqεt-q -
୯Ґࠜݕఆ • Augmented Dickey-Fullerݕఆ sm.tsa.adfuller(df['Air passengers'])[1] 0.99188024343764114 sm.tsa.adfuller(ldf['Air passengers'])[1] 0.42236677477038415
sm.tsa.adfuller(ldf['Air passengers'].diff().dropna())[1] 0.071120548150854057 ݩσʔλ ରԽ ରԽ ֊ࠩ sm.tsa.adfuller(seasonal_adjust['Air passengers'].diff().dropna())[1] 8.0990048658604878e-09 ରԽ قઅੑআڈ ֊ࠩ -
SARIMAϞσϧͷਪఆ mod_seasonal = sm.tsa.SARIMAX(ldf, trend='c', order=(1, 1, 1), seasonal_order=(0, 1,
2, 12)) res_seasonal = mod_seasonal.fit() res_seasonal.summary() ʜ ʜ "3*."قઅͷύϥϝʔλ SD͕ඞཁ -
Ϟσϧ͔Βͷ༧ଌ pred = res_seasonal.forecast(36) pred 1961-01-01 6.110548 1961-02-01 6.052912 1961-03-01
6.174690 ... 1963-10-01 6.388955 1963-11-01 6.242262 1963-12-01 6.345214 Freq: MS, dtype: float64 ax = ldf.plot() pred.plot(ax=ax) ظઌΛ༧ଌ ݩσʔλ ༧ଌΛϓϩοτ -
։ൃϩʔυϚοϓ • ܭը • 0.19 (ݱࡏrc) 0.20 1.0
ΛϦϦʔε • pandas 1.0 • API ౚ݁ • Long Time Support • pandas 2.0 (under discussion) • Python 3.xͷΈΛαϙʔτ • 2࣍ݩҎԼͷσʔλʹಛԽ • όοΫΤϯυΛ C++ ʹҠߦ (Apache Arrow)