Text as a State Variable in Commodity Markets: Forecasting Agricultural Futures Risk and Returns with Media News

This thesis examines whether unstructured media narratives can be treated as a meaningful, actionable data stream for forecasting in agricultural commodity futures markets-where price discovery and risk management depend not only on structured releases but also on evolving information about weather, production, policy, trade disruptions, and macro-financial spillovers. Building on the "information-to-markets" perspective that motivates the study, the thesis asks whether modern text representations-especially language-model embeddings-deliver forecasting improvements that remain credible against strong econometric benchmarks and that survive evaluation under realistic economic constraints. The empirical analysis is organised into three chapters covering (i) realized volatility forecasting, (ii) next-day return-direction forecasting and trading performance, and (iii) a direct comparison between embedding-based methods and sentiment as a compressed news proxy.

Across all chapters, the methodological approach emphasises disciplined out-of-sample design: rolling-window estimation, conservative regularisation, and benchmark comparisons chosen to reflect the state of the art in commodity forecasting. On volatility, the thesis reframes forecasting as a representation problem by converting daily news into dense embedding vectors and using regularised textual regressions with aggregation schemes that align multiple intraday articles to a daily target. Using high-frequency (5-minute) CBOT futures prices for agricultural commodities over 2009-2023, results show a clear horizon dependence: at one-day horizons, HAR-type models remain difficult to beat, while at monthly horizons embedding-based text signals consistently improve volatility forecasts and yield economically meaningful gains in volatility-timing utility. Robustness checks-including randomisation tests, architecture/model-size comparisons, and Shapley-based interpretability-support the conclusion that longer-horizon improvements are driven by semantically rich fundamentals (production, weather, and broader narratives) rather than chance correlations.

For returns, news contains statistically detectable information about next-day return direction, with performance varying by text length and model specialisation (simple lexical representations are competitive for headlines, while domain-aware transformer embeddings perform best for longer stories). However, the chapter also documents a crucial implementation gap: frictionless profitability can be substantial, but gains erode under transaction costs for models without domain adaptation, whereas finance-domain encoders remain profitable net of costs; economic value is further improved by conditioning trading on forecasted volatility regimes, underscoring state dependence and turnover constraints.

Compared directly against embeddings, sentiment proves useful for some return applications but is generally too compressed to capture the semantic depth required for robust long-horizon volatility forecasting. Embedding-based approaches produce more differentiated signals and retain stronger, more orthogonal economic value in spanning tests. Overall, the thesis contributes an integrated map of when text helps (by target and horizon), why it helps (semantic richness vs compression), and how to evaluate it (jointly on statistical loss and decision-relevant performance), reinforcing the case for treating text as a high-dimensional state variable in agricultural commodity forecasting.