feat(factor-coder): Add critical rules to prevent common factor implementation errors

- Add explicit warning against .date on datetime index (causes data loss
  to single year, only 314 entries instead of 2020-2026)
- Add explicit warning against df.merge() which destroys MultiIndex
  (causes RangeIndex output instead of required MultiIndex)
- Enforce column name must be exactly factor_name, not a shortened alias
- Require transform() over apply() for per-group calculations to
  preserve row count
- Add MultiIndex assertion before saving to result.h5
- Document expected output: ~1500+ daily entries for full 2020-2026 range
This commit is contained in:
TPTBusiness
2026-04-13 15:28:21 +02:00
parent df61b90464
commit 4aa07f99d2
+58 -9
View File
@@ -36,42 +36,91 @@ qlib_factor_strategy: |-
Ensure that for every step of data processing, the data format (including indexes) is clearly explained through comments.
Each transformation or calculation should be accompanied by a detailed description of how the data is structured, especially focusing on key aspects like whether the data has multi-level indexing, how to access specific columns or index levels, and any operations that affect the data shape (e.g., `reset_index()`, `groupby()`, `merge()`).
This step-by-step explanation will ensure clarity and accuracy in data handling. For example:
1. **Start with multi-level index**:
1. **Start with multi-level index**:
```python
# The initial DataFrame has a multi-level index with 'datetime' and 'instrument'.
# To access the 'datetime' index, use df.index.get_level_values('datetime').
datetime_values = df.index.get_level_values('datetime')
```
2. **Reset the index if necessary**:
2. **Reset the index if necessary**:
```python
# Resetting the index to move 'datetime' and 'instrument' from the index to columns.
# This operation flattens the multi-index structure.
df = df.reset_index()
```
3. **Perform groupby operations**:
3. **Perform groupby operations**:
```python
# Grouping by 'datetime' and 'instrument' to aggregate the data.
# After groupby, the result will maintain 'datetime' and 'instrument' as a multi-level index.
df_grouped = df.groupby(['datetime', 'instrument']).sum()
```
4. **Ensure consistent datetime formats**:
4. **Ensure consistent datetime formats**:
```python
# Before merging, ensure that the 'datetime' column in both DataFrames is of the same format.
# Convert to datetime format if necessary.
df['datetime'] = pd.to_datetime(df['datetime'])
other_df['datetime'] = pd.to_datetime(other_df['datetime'])
```
5. **Merge operations**:
5. **Merge operations**:
```python
# When merging DataFrames, ensure you are merging on both 'datetime' and 'instrument'.
# If these are part of the index, reset the index before merging.
merged_df = pd.merge(df, other_df, on=['datetime', 'instrument'], how='inner')
```
====== CRITICAL RULES FOR INTRADAY DATA ======
The source data is 1-minute EURUSD OHLCV from 2020-01-01 to 2026-03-20 with ~2.2M rows.
You MUST avoid these common errors that cause factor rejection:
1. **DO NOT use `.date` on datetime index**: Using `df.index.get_level_values('datetime').date` converts to Python date objects which LOSES data outside the filtered range. Use `.floor('D')` or `.normalize()` instead to keep the full date range:
```python
# WRONG: This filters to only one year!
df['date'] = df.index.get_level_values('datetime').date
# CORRECT: Preserves full 2020-2026 range
df['date'] = df.index.get_level_values('datetime').floor('D')
```
2. **DO NOT use `df.merge()` — it destroys MultiIndex**: After a merge, the MultiIndex is lost and replaced with a RangeIndex. Use `pd.merge()` with explicit index reset/restore, or use `.join()` or `.map()` instead:
```python
# WRONG: merge() destroys the MultiIndex!
df = df.merge(other_df, on='instrument') # Index becomes RangeIndex!
# CORRECT: Use join on aligned indices, or reset/set index around merge
df = df.join(other_df.set_index('instrument'), on='instrument')
# OR: after merge, explicitly restore MultiIndex
df.index = pd.MultiIndex.from_arrays([df['datetime'], df['instrument']], names=['datetime', 'instrument'])
```
3. **Column name MUST be the factor_name**: The output column must be named exactly as the factor_name, NOT a shortened alias:
```python
# WRONG: result_df.columns = ['divergence_score']
# CORRECT:
result_df = daily_agg[['daily_volume_price_divergence']].copy()
result_df.columns = ['daily_volume_price_divergence']
```
4. **Process ALL data — do not filter dates**: The source HDF5 contains data from 2020-01-01 to 2026-03-20. Do NOT filter to a single year. If your output has only 314 entries (one year of daily data), the factor will be rejected. Expected output: ~1500+ daily entries for 2020-2026.
5. **Use `transform()` instead of `apply()` for per-group calculations**: `transform()` preserves the original index while `apply()` may reduce the number of rows unexpectedly:
```python
# CORRECT: transform keeps the same index as input
df['daily_vol'] = df.groupby(['date', 'instrument'])['return'].transform('std')
# WRONG for per-row output: apply reduces rows
daily_vol = df.groupby(['date', 'instrument'])['return'].apply('std') # Loses intraday rows!
```
6. **Preserve the MultiIndex until the very end**: The final `result.h5` must have a MultiIndex with levels ['datetime', 'instrument']. Always verify before saving:
```python
assert isinstance(result_df.index, pd.MultiIndex), "Index must be MultiIndex!"
assert result_df.index.names == ['datetime', 'instrument'], f"Index names must be ['datetime', 'instrument'], got {result_df.index.names}"
```
qlib_factor_output_format: |-
Your output should be a pandas dataframe similar to the following example information:
<class 'pandas.core.frame.DataFrame'>