Improve pandas history ergonomics: self-describing KeyErrors, DataFrame.get(symbol), lazy DataFrame on typed history - #9668

Description

Improves the ergonomics of history results in Python, targeting the most common pandas interop failure modes:

• Self-describing PandasMapper KeyErrors. The former error was opaque and cost a full backtest iteration to decode:

  KeyError: "No key found for either mapped or original key. Mapped Key: []; Original Key: []"
  

  The error now keeps the legacy wording (for backwards compatibility) and appends what was requested and what the object actually has:

  KeyError: "No key found for either mapped or original key. Mapped Key: []; Original Key: ['symbol', 'close'].
  The DataFrame has columns ['open', 'high', 'low', 'close'] and index levels ['symbol', 'time'].
  'symbol' is an index level, not a column: read it with df.index.get_level_values('symbol') or move the index levels into columns with df.reset_index()"
  

  • PandasMapper.wrap_keyerror_function now extracts the requested keys (including keys nested in list keys like df[["symbol", "close"]], which previously reported empty key lists) and appends a description of the indexed object: its columns and index levels.
  • Targeted hints for the three recurring mistakes: requesting an index level as a column (→ get_level_values/reset_index), requesting a symbol that lives in the index (→ df.loc[key]/df.xs), and requesting a known symbol that simply has no data in the frame (→ df.get(key)).
  • Message building is fully guarded: any failure while describing the frame falls back to the legacy message.

• DataFrame.get(symbol) returns the symbol's sub-frame or None. PandasMapper now extends DataFrame.get so that symbols (or cached tickers) are also looked up in the index: history.get(symbol) returns the symbol's sub-frame, or the default (None) when the symbol has no data, instead of raising. Column lookups and non-symbol keys keep the original pandas semantics.

• Typed history results expose a lazily-built DataFrame. The typed History overloads (History<T>(symbol, ...), History<T>(symbols, ...), History<T>(span/periods) and the History(symbol, ...) trade bar overloads) now return DataHistory<T> / DataHistory<DataDictionary<T>> instead of plain IEnumerable:

  • In Python, self.history[TradeBar](symbol, 10).data_frame now works instead of requiring manual conversion ('MemoizingEnumerable[TradeBar]' object has no attribute 'iterrows' was a recurring dead end).
  • DataHistory<T> implements IEnumerable<T>, so existing C# and Python consumers (iteration, LINQ) are unaffected; the pandas conversion is lazy and shares the memoized data with the enumerable, so accessing the data frame does not re-execute the history request and the result remains enumerable afterwards.
  • This mirrors the existing OptionHistory/FutureHistory/IndicatorHistory pattern, which already subclass DataHistory<T> for exactly this reason.

Deferred from the original proposal:

• Opt-in wide/unstacked data frame shapes (unstacked=True, history_series): changing the default frame shape would be breaking, and an opt-in API deserves its own design discussion. The new KeyError hints teach the existing idioms (get_level_values, reset_index, df.loc) in the meantime.
• A targeted AttributeError for pandas-only attributes on typed results (e.g. .iterrows) requires a pythonnet-side extension point for per-type attribute error hints; out of scope here.

Related Issue

N/A

Motivation and Context

MultiIndex/typed-result confusion is the dominant crash class for generated and user Python algorithms: each opaque PandasMapper KeyError costs a full backtest iteration to decode, and history.loc[symbol] on a symbol with no data kills scheduled rebalances. These changes make the errors self-explanatory and provide safe accessors, without changing any default data frame shape.

Requires Documentation Change

The typed history return type change (IEnumerable<T> → DataHistory<T>) and history.get(symbol)/.data_frame accessors could be mentioned in the history documentation.

How Has This Been Tested?

• New tests in Tests/Python/PandasIndexingTests.cs + Tests/Python/PandasTests/PandasIndexingTests.py:
  • KeyErrorDescribesMissingColumn: missing column error names the key, the available columns and the index levels.
  • KeyErrorDescribesIndexLevelKey: df[['symbol', 'lastprice']] error explains 'symbol' is an index level and suggests reset_index.
  • KeyErrorDescribesSymbolInIndex: df['spy'] error points to the 'symbol' index level and df.loc.
  • KeyErrorDescribesMissingSymbol: df.loc[symbol] for a symbol with no data suggests df.get(key).
  • GetWithSymbolReturnsSubFrame / GetWithMissingSymbolReturnsNone / GetWithColumnKeepsPandasSemantics: DataFrame.get symbol lookup, default handling, and unchanged column semantics.
• New AlgorithmHistoryTests.TypedHistoryResultsExposeADataFrame (C# and Python): single- and multi-symbol typed history results expose a data frame with the expected shape and remain enumerable after the conversion.
• All tests reproduce the old behavior (red) before the fix and pass after it.
• Full AlgorithmHistoryTests, PandasConverterTests, PandasIndexingTests and PythonUtilTests fixtures pass.
• Regression backtests CSharp/HistoryAlgorithm, Python/HistoryAlgorithm, CSharp/CustomDataTypeHistoryAlgorithm and Python/CustomDataTypeHistoryAlgorithm pass unchanged (the typed history path is exercised end-to-end).

Part of QuantConnect/Agents#305 (improvement #10: history ergonomics)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which improves implementation)
• Performance (non-breaking change which improves performance. Please add associated performance test and results)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Non-functional change (xml comments/documentation/etc)

Checklist:

• My code follows the code style of this project.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>