Using DataFrame Inputs with MSTUMP/MSTUMPED

[seanlaw]

@marcusau asked:

I have tried multi-dimensional array to mstump with stock prices and its technical indicators

#### step 2 : Feature creation
df1=df.copy()

df1['Close_pct']=np.log(df1['Close'] / df1['Close'].shift(1)).dropna()
df1['STDDEV']= ta.STDDEV(df1['Close'], timeperiod=5, nbdev=1)
### Volume Indicator Functions
df1['OBV']=ta.OBV(df1['Close'], df1['Volume'])
df1['Chaikin AD'] = ta.AD(df1['High'], df1['Low'], df1['Close'], df1['Volume'])

### momentum Indicator

macd, macdsignal, macdhist = ta.MACD(df1['Close'], fastperiod=12, slowperiod=26, signalperiod=9)
df1['macdhist']=macdhist

df1['RSI']= ta.RSI(df1['Close'], timeperiod=14)

# Volatility Indicator Functions
df1['NATR'] = ta.NATR(df1['High'], df1['Low'], df1['Close'], timeperiod=14)
df1['TRANGE'] = ta.TRANGE(df1['High'], df1['Low'], df1['Close'])
print(df1.tail())


print(list(set(df.columns) ^ set(df1.columns)))

feature_cols=list(set(df.columns) ^ set(df1.columns))
df1=df1.loc[:,feature_cols].dropna()
print(df1.head())


#Store these values in the NumPy array for using in our models later:
f_x=()
for f in feature_cols:
  f_x += (df1[f].values.reshape(-1,1),)
X = np.concatenate(f_x,axis=1)
X=X.T
print(X.shape)
>>>> (8, 2429)

window_size = 10  # Approximately, how many data points might be found in a pattern

matrix_profile, matrix_profile_indices = stumpy.mstump(X, m=window_size)

left_matrix_profile_index = matrix_profile[:, 2]
right_matrix_profile_index = matrix_profile[:, 3]

as you said, mstumpy only support 1-D data, however, what is the explanation of the results in mstumpy?

[seanlaw]

@marcusau Thank you for your question. I've moved it here so that I can address it more directly than in the previous issue where the issue was looking for documentation/tutorial help.

Prior to your issue, we didn't have anybody but myself who used mstump/mstumped on multi-dimensional data and let alone with Pandas DataFrame. DataFrame/Series support was recently supported. Firstly, I want to reassure you that the underlying matrix profile code is correct but there was an issue with handling Pandas DataFrame (Pandas Series, which are 1-dimensional, are not affected). You can read more about it in #66

Briefly, to be consistent with the 1-dimensional stump/stumped case, mstump/mstumped assumes that your data has time series stored in each row (not in each column). However, when you start with a Pandas DataFrame, typically, the time series is stored in each column. Again, the code for computing the multi-dimensional matrix profile is still correct if you pass it a NumPy array but one would've needed to transpose the input if a DataFrame were used. We've fixed this in #66 so DataFrame is automatically transposed starting in all versions greater than 1.1.0.

Please clone the repo, uninstall STUMPY, reinstall, re-run your analysis above and you should be good to go. We'll push a new version to PyPI and Conda-forge in the coming weeks. Let me know if that works!

[seanlaw]

@marcusau I believe that this is resolved now so closing this issue. Feel free to re-open if the problem persists.