Chapter 05

Returns and correlation

The project’s second half leaves rates behind for a data-handling problem: fetch it, transform it, store it, get it back unchanged.

The task

Daily closing prices and volumes for five large-cap tickers — AAPL, IBM, MSFT, GOOG and AMZN — pulled from Yahoo Finance through pandas_datareader, starting 1 January 2012. The first observations land on 3 January 2012, the first trading day of that year.

tickers = ['AAPL', 'IBM', 'MSFT', 'GOOG', 'AMZN']
all_data = {ticker: pdr.get_data_yahoo(ticker, start=datetime(2012, 1, 1))
            for ticker in tickers}

price  = pd.DataFrame({t: d['Close']  for t, d in all_data.items()})
volume = pd.DataFrame({t: d['Volume'] for t, d in all_data.items()})

Closes and volumes are split into two frames keyed by ticker, then later concatenated back into one frame with a two-level column index — price above, volume below.

Returns and cumulative returns

Daily simple returns come from the percentage change of the close, and the cumulative series is their running compounded product — a currency unit invested at the start, tracked forward:

Cumulative return

\[ R_T \;=\; \prod_{t=1}^{T}\bigl(1 + r_t\bigr) \qquad\text{where}\qquad r_t = \frac{P_t}{P_{t-1}} - 1 \]

Compounded, not summed — a 50% loss followed by a 50% gain does not return to par.

One panel per ticker on a shared scale, so the heights are directly comparable. Five lines on a single axis cannot be distinguished by colour alone at this count — separating them into panels means the labels carry the identity and the colour carries nothing.

The spread is the point. Four of the five multiplied more than twentyfold; IBM roughly doubled. Same sector, same window, wildly different outcomes — which is exactly the dispersion that makes the correlation question below worth asking.

Correlation

The correlation matrix is taken on daily returns, not on prices. This distinction matters more than it looks. Price series are non-stationary and drift upward together, so correlating them measures little beyond “both went up over ten years”. Returns are close to stationary, and their correlation measures what is actually of interest: whether these names move together day to day.

Pearson correlation

\[ \rho_{ij} \;=\; \frac{\operatorname{Cov}(r_i,\,r_j)}{\sigma_i\,\sigma_j} \]

Covariance normalised by both standard deviations, giving a number in \([-1, 1]\).

returns = price.pct_change().dropna()
returns.corr().style.background_gradient(cmap='viridis')

Every pair moves together to some degree — there are no negative or near-zero entries anywhere in the matrix. That is the useful negative result: a basket of five large-cap technology names offers far less diversification than holding five positions suggests.

One caveat on the numbers. These are full-sample averages over fourteen years. Correlation is not stable: it rises sharply in drawdowns, precisely when diversification is being relied on. A single figure per pair hides that, which is why risk work usually looks at rolling windows rather than one number.

The round-trip

The last part is deliberately unglamorous. Write one CSV per ticker holding its price and volume, then read them all back and rebuild a single frame of prices:

# out: one file per ticker
for ticker in tickers:
    pd.concat([price[ticker], volume[ticker]], axis=1,
              keys=['price', 'volume']).to_csv(f'{ticker}.csv')

# back in: recombined on the Date index
frames = (pd.read_csv(f'{ticker}.csv', usecols=['Date', 'price'],
                      index_col='Date') for ticker in tickers)
price = pd.concat(frames, axis=1)
price.columns = tickers

The round-trip is the point: the recovered frame has to match what was written, with the date index intact and the columns correctly re-labelled. Serialisation is where index handling and column naming quietly go wrong, and the check is that nothing changed on the way out and back.

The thread between the halves

The two halves look unrelated but share a shape. Each recovers something from market observations — a discount curve from prices, a correlation structure from returns — and each is validated by reconstruction: reprice the bonds, reload the frame. Getting an answer is easy; showing it survives a round-trip is the work.