NexQuant/rdagent/scenarios/qlib/experiment/prompts.yaml

qlib_quant_background: |-
  Quantitative investment is a data-driven approach to asset management that relies on mathematical models, statistical techniques, and computational methods to analyze financial markets and make investment decisions. Two essential components of this approach are factors and models.
  
  You are one of the most authoritative quantitative researchers at a top Wall Street hedge fund. I need your expertise to develop new factors and models that can enhance our investment returns. Based on the given context, I will ask for your assistance in designing and implementing either factors or a model.

  {% if runtime_environment is not none %}
  ====== Runtime Environment ======
  You have following environment to run the code:
  {{ runtime_environment }}
  {% endif %}

qlib_factor_background: |-
  The factor is a characteristic or variable used in quant investment that can help explain the returns and risks of a portfolio or a single asset. Factors are used by investors to identify and exploit sources of excess returns, and they are central to many quantitative investment strategies.
  Each number in the factor represents a physics value to an instrument on a day.
  User will train a model to predict the next several days return based on the factor values of the previous days.
  The factor is defined in the following parts:
  1. Name: The name of the factor.
  2. Description: The description of the factor.
  3. Formulation: The formulation of the factor.
  4. Variables: The variables or functions used in the formulation of the factor.
  The factor might not provide all the parts of the information above since some might not be applicable.
  Please specifically give all the hyperparameter in the factors like the window size, look back period, and so on. One factor should statically defines one output with a static source data. For example, last 10 days momentum and last 20 days momentum should be two different factors.

  {% if runtime_environment is not none %}
  ====== Runtime Environment ======
  You have following environment to run the code:
  {{ runtime_environment }}
  {% endif %}

qlib_factor_interface: |-
  Your python code should follow the interface to better interact with the user's system.
  Your python code should contain the following part: the import part, the function part, and the main part. You should write a main function name: "calculate_{function_name}" and call this function in "if __name__ == __main__" part. Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you.
  User will write your python code into a python file and execute the file directly with "python {your_file_name}.py". You should calculate the factor values and save the result into a HDF5(H5) file named "result.h5" in the same directory as your python file. The result file is a HDF5(H5) file containing a pandas dataframe. The index of the dataframe is the "datetime" and "instrument", and the single column name is the factor name,and the value is the factor value. The result file should be saved in the same directory as your python file.

qlib_factor_strategy: |-
  Ensure that for every step of data processing, the data format (including indexes) is clearly explained through comments.
  Each transformation or calculation should be accompanied by a detailed description of how the data is structured, especially focusing on key aspects like whether the data has multi-level indexing, how to access specific columns or index levels, and any operations that affect the data shape (e.g., `reset_index()`, `groupby()`, `merge()`).
  This step-by-step explanation will ensure clarity and accuracy in data handling. For example:
  1. **Start with multi-level index**:
    ```python
    # The initial DataFrame has a multi-level index with 'datetime' and 'instrument'.
    # To access the 'datetime' index, use df.index.get_level_values('datetime').
    datetime_values = df.index.get_level_values('datetime')
    ```

  2. **Reset the index if necessary**:
    ```python
    # Resetting the index to move 'datetime' and 'instrument' from the index to columns.
    # This operation flattens the multi-index structure.
    df = df.reset_index()
    ```

  3. **Perform groupby operations**:
    ```python
    # Grouping by 'datetime' and 'instrument' to aggregate the data.
    # After groupby, the result will maintain 'datetime' and 'instrument' as a multi-level index.
    df_grouped = df.groupby(['datetime', 'instrument']).sum()
    ```

  4. **Ensure consistent datetime formats**:
    ```python
    # Before merging, ensure that the 'datetime' column in both DataFrames is of the same format.
    # Convert to datetime format if necessary.
    df['datetime'] = pd.to_datetime(df['datetime'])
    other_df['datetime'] = pd.to_datetime(other_df['datetime'])
    ```

  5. **Merge operations**:
    ```python
    # When merging DataFrames, ensure you are merging on both 'datetime' and 'instrument'.
    # If these are part of the index, reset the index before merging.
    merged_df = pd.merge(df, other_df, on=['datetime', 'instrument'], how='inner')
    ```

  ====== CRITICAL RULES FOR INTRADAY DATA ======
  The source data is 1-minute EURUSD OHLCV from 2020-01-01 to 2026-03-20 with ~2.2M rows.
  You MUST avoid these common errors that cause factor rejection:

  1. **DO NOT use `.date` on datetime index**: Using `df.index.get_level_values('datetime').date` converts to Python date objects which LOSES data outside the filtered range. Use `.floor('D')` or `.normalize()` instead to keep the full date range:
     ```python
     # WRONG: This filters to only one year!
     df['date'] = df.index.get_level_values('datetime').date

     # CORRECT: Preserves full 2020-2026 range
     df['date'] = df.index.get_level_values('datetime').floor('D')
     ```

  2. **DO NOT use `df.merge()` — it destroys MultiIndex**: After a merge, the MultiIndex is lost and replaced with a RangeIndex. Use `pd.merge()` with explicit index reset/restore, or use `.join()` or `.map()` instead:
     ```python
     # WRONG: merge() destroys the MultiIndex!
     df = df.merge(other_df, on='instrument')  # Index becomes RangeIndex!

     # CORRECT: Use join on aligned indices, or reset/set index around merge
     df = df.join(other_df.set_index('instrument'), on='instrument')
     # OR: after merge, explicitly restore MultiIndex
     df.index = pd.MultiIndex.from_arrays([df['datetime'], df['instrument']], names=['datetime', 'instrument'])
     ```

  3. **Column name MUST be the factor_name**: The output column must be named exactly as the factor_name, NOT a shortened alias:
     ```python
     # WRONG: result_df.columns = ['divergence_score']
     # CORRECT:
     result_df = daily_agg[['daily_volume_price_divergence']].copy()
     result_df.columns = ['daily_volume_price_divergence']
     ```

  4. **Process ALL data — do not filter dates**: The source HDF5 contains data from 2020-01-01 to 2026-03-20. Do NOT filter to a single year. If your output has only 314 entries (one year of daily data), the factor will be rejected. Expected output: ~1500+ daily entries for 2020-2026.

  5. **Use `transform()` instead of `apply()` for per-group calculations**: `transform()` preserves the original index while `apply()` may reduce the number of rows unexpectedly:
     ```python
     # CORRECT: transform keeps the same index as input
     df['daily_vol'] = df.groupby(['date', 'instrument'])['return'].transform('std')

     # WRONG for per-row output: apply reduces rows
     daily_vol = df.groupby(['date', 'instrument'])['return'].apply('std')  # Loses intraday rows!
     ```

  6. **Preserve the MultiIndex until the very end**: The final `result.h5` must have a MultiIndex with levels ['datetime', 'instrument']. Always verify before saving:
     ```python
     assert isinstance(result_df.index, pd.MultiIndex), "Index must be MultiIndex!"
     assert result_df.index.names == ['datetime', 'instrument'], f"Index names must be ['datetime', 'instrument'], got {result_df.index.names}"
     ```

  7. **NEVER use same-day aggregations as the factor value — always shift by 1 day**: If your factor computes a daily aggregate (e.g. daily close return, daily OHLC range, daily volume), that aggregate is only known at end-of-day. Using it at the start of the same day is look-ahead bias. You MUST shift the daily aggregate by 1 day before forward-filling to minute bars:
     ```python
     # WRONG: look-ahead bias! Today's close return is not known at 00:00
     daily_ret = df['$close'].groupby(level='instrument').resample('1D', level='datetime').last().pct_change()
     result_df['my_factor'] = daily_ret.groupby(level='instrument').transform(lambda x: x.reindex(df.index.get_level_values('datetime'), method='ffill'))

     # CORRECT: shift by 1 trading day so factor value at day T = aggregate of day T-1
     daily_close = df.groupby([df.index.get_level_values('datetime').normalize(), df.index.get_level_values('instrument')])['$close'].last()
     daily_close.index.names = ['date', 'instrument']
     daily_ret = daily_close.groupby(level='instrument').pct_change().shift(1)  # <-- shift(1) is MANDATORY
     # then map back to minute bars via ffill
     ```
     This rule applies to ALL daily aggregations: returns, OHLC stats, volume, momentum, slopes, etc.
     **Session-based aggregations (London, NY, Asian session returns) are also daily aggregations** — the London
     session (08:00-16:00 UTC) ends at 16:00, so its return must be shifted by 1 day before use.
     Intraday rolling factors (e.g. 30-min rolling std computed at bar t using only bars t-N..t-1) do NOT need this shift.

  8. **PREFER pure intraday rolling factors**: Factors that use only a trailing window of recent bars (e.g.
     rolling(30).mean() of returns, RSI(14), Bollinger Band z-score) have NO look-ahead risk and vary every
     minute. These are the best candidates for short-horizon (96-bar) prediction. Examples:
     - Rolling 15-min / 30-min / 60-min return momentum
     - Rolling volatility (std of returns over 20-60 bars)
     - Distance of close from N-bar moving average (z-score)
     - RSI or similar oscillators computed on 1-min bars
     - VWAP deviation (requires volume — use $volume column)
     Always use `.shift(1)` on the lagged window (e.g. `rolling(N).mean().shift(1)`) to avoid using the
     current bar's own price in its own feature value.

qlib_factor_output_format: |-
  Your output should be a pandas dataframe similar to the following example information:
  <class 'pandas.core.frame.DataFrame'>
  MultiIndex: 2261923 entries, (Timestamp('2020-01-01 17:00:00'), 'EURUSD') to (Timestamp('2026-03-20 15:58:00'), 'EURUSD')
  Data columns (total 1 columns):
  #   Column            Non-Null Count  Dtype  
  ---  ------            --------------  -----  
  0   your factor name  2261923 non-null  float64
  dtypes: float64(1)
  memory usage: <ignore>
  Notice: The non-null count is OK to be different to the total number of entries since some instruments may not have the factor value on some days.
  One possible format of `result.h5` may be like following:
  datetime    instrument
  2020-01-01  EURUSD    1.094240
  2020-01-02  EURUSD    1.094280
  2020-01-03  EURUSD    1.095920
                ...
  2026-03-20  EURUSD    1.083150

qlib_factor_simulator: |-
  The factors will be sent into Qlib to train a model to predict the next several days return based on the factor values of the previous days. 
  Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.
  User will use Qlib to automatically do the following things:
  1. generate a new factor table based on the factor values.
  2. train a model like LightGBM, CatBoost, LSTM or simple PyTorch model to predict the next several days return based on the factor values.
  3. build a portfolio based on the predicted return based on a strategy.
  4. evaluate the portfolio's performance including the return, sharpe ratio, max drawdown, and so on.

qlib_factor_rich_style_description : |-
  ### R&D Agent-Qlib: Automated Quantitative Trading & Iterative Factors Evolution Demo

  #### [Overview](#_summary)

  The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making. It highlights how financial factors evolve through continuous feedback and refinement.

  #### [Automated R&D](#_rdloops)

  - **[R (Research)](#_research)**
    - Iterative development of ideas and hypotheses.
    - Continuous learning and knowledge construction.

  - **[D (Development)](#_development)**
    - Progressive implementation and code generation of factors.
    - Automated testing and validation of financial factors.

  #### [Objective](#_summary)

  To demonstrate the dynamic evolution of financial factors through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting financial factors.

qlib_factor_from_report_rich_style_description : |-
  ### R&D Agent-Qlib: Automated Quantitative Trading & Factor Extraction from Financial Reports Demo

  #### [Overview](#_summary)

  This demo showcases the process of extracting factors from financial research reports, implementing these factors, and analyzing their performance through Qlib backtest, continually expanding and refining the factor library.

  #### [Automated R&D](#_rdloops)

  - **[R (Research)](#_research)**
    - Iterative development of ideas and hypotheses from financial reports.
    - Continuous learning and knowledge construction.

  - **[D (Development)](#_development)**
    - Progressive factor extraction and code generation.
    - Automated implementation and testing of financial factors.

  #### [Objective](#_summary)

  <table border="1" style="width:100%; border-collapse: collapse;">
    <tr>
      <td>💡 <strong>Innovation </strong></td>
      <td>Tool to quickly extract and test factors from research reports.</td>
    </tr>
    <tr>
      <td>⚡ <strong>Efficiency </strong></td>
      <td>Rapid identification of valuable factors from numerous reports.</td>
    </tr>
    <tr>
      <td>🗃️ <strong>Outputs </strong></td>
      <td>Expand and refine the factor library to support further research.</td>
    </tr>
  </table>

qlib_factor_experiment_setting: |-
  | Dataset 📊 | Model 🤖    | Factors 🌟       | Data Split  🧮                                   |
  |---------|----------|---------------|-------------------------------------------------|
  | EURUSD  | LGBModel | Alpha158 Plus | Train: {{ train_start }} to {{ train_end }} <br> Valid: {{ valid_start }} to {{ valid_end }} <br> Test &nbsp;: {{ test_start }} to {{ test_end }} |


qlib_model_background: |-
  The model is a machine learning or deep learning structure used in quantitative investment to predict the returns and risks of a portfolio or a single asset. Models are employed by investors to generate forecasts based on historical data and identified factors, which are central to many quantitative investment strategies.
  Each model takes the factors as input and predicts the future returns. Usually, the bigger the model is, the better the performance would be.
  The model is defined in the following parts:
  1. Name: The name of the model.
  2. Description: The description of the model.
  3. Architecture: The detailed architecture of the model, such as neural network layers or tree structures.
  4. Hyperparameters: The hyperparameters used in the model.
  5. Training_hyperparameters: The hyperparameters used during the training process.
  6. ModelType: The type of the model, "Tabular" for tabular model and "TimeSeries" for time series model.
  The model should provide clear and detailed documentation of its architecture and hyperparameters. One model should statically define one output with a fixed architecture and hyperparameters.

  {% if runtime_environment is not none %}
  ====== Runtime Environment ======
  You have following environment to run the code:
  {{ runtime_environment }}
  {% endif %}

qlib_model_interface: |-
  Your python code should follow the interface to better interact with the user's system.
  You code should contain several parts:
  1. The import part: import the necessary libraries.
  2. A class which is a sub-class of pytorch.nn.Module. This class should should have a init function and a forward function which inputs a tensor and outputs a tensor.
  3. Set a variable called "model_cls" to the class you defined.

  The user will save your code into a python file called "model.py". Then the user imports model_cls in file "model.py" after setting the cwd into the directory:
  ```python
  from model import model_cls
  ```
  So your python code should follow the pattern:
  ```python
  class XXXModel(torch.nn.Module):
      ...
  model_cls = XXXModel
  ```

  The model can be configured as either "Tabular" for tabular models or "TimeSeries" for time series models. For a tabular model, the input shape is (batch_size, num_features), while for a time series model, the input shape is (batch_size, num_timesteps, num_features). In both cases, the output shape of the model should be (batch_size, 1).
  `num_features` will be directly set for the model based on the input data shape.
  User will initialize the tabular model with the following code:
  ```python
  model = model_cls(num_features=num_features)
  ```
  User will initialize the time series model with the following code:
  ```python
  model = model_cls(num_features=num_features, num_timesteps=num_timesteps)
  ```
  No other parameters will be passed to the model so give other parameters a default value or just make them static.

  Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you. Also, don't write main function in your python code. The user will call the forward method in the model_cls to get the output tensor.

  Please notice that your model should only use current features as input. The user will provide the input tensor to the model's forward function.


qlib_model_output_format: |-
  Your output should be a tensor with shape (batch_size, 1). 
  The output tensor should be saved in a file named "output.pth" in the same directory as your python file.
  The user will evaluate the shape of the output tensor so the tensor read from "output.pth" should be 8 numbers.

qlib_model_simulator: |-
  The models will be sent into Qlib to train and evaluate their performance in predicting future returns. Hypothesis is improved upon checking the feedback on the results. 
  Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning (RL).
  User will use Qlib to automatically perform the following tasks:
  1. Generate a baseline factor table.
  2. Train the model defined in your class Net to predict the next several days' returns based on the factor values.
  3. Build a portfolio based on the predicted returns using a specific strategy.
  4. Evaluate the portfolio's performance, including metrics such as return, IC, max drawdown, and others.
  5. Iterate on growing the hypothesis to enable model improvements based on performance evaluations and feedback.

qlib_model_rich_style_description: |-
  ### Qlib Model Evolving Automatic R&D Demo
  
  #### [Overview](#_summary)
  
  The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making in model construction in quantitative finance. It highlights how models evolve through continuous feedback and refinement.
  
  #### [Automated R&D](#_rdloops)
  
  - **[R (Research)](#_research)**
    - Iteration of ideas and hypotheses.
    - Continuous learning and knowledge construction.
  
  - **[D (Development)](#_development)**
    - Evolving code generation and model refinement.
    - Automated implementation and testing of models.
  
  #### [Objective](#_summary)
  
  To demonstrate the dynamic evolution of models through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting models. 

qlib_model_experiment_setting: |-
  | Dataset 📊 | Model 🤖    | Factors 🌟       | Data Split  🧮                                   |
  |---------|----------|---------------|-------------------------------------------------|
  | EURUSD  | RDAgent-dev | 20 factors (Alpha158)  | Train: {{ train_start }} to {{ train_end }} <br> Valid: {{ valid_start }} to {{ valid_end }} <br> Test &nbsp;: {{ test_start }} to {{ test_end }} |