NexQuant/rdagent/scenarios/qlib/prompts.yaml

hypothesis_and_feedback: |-
  {% for hypothesis, experiment, feedback in trace.hist %}
  Hypothesis {{ loop.index }}: {{ hypothesis }}
  Observation on the result with the hypothesis: {{ feedback.observations }}
  Feedback on the original hypothesis:  {{ feedback.hypothesis_evaluation }}
  New Feedback for Context (For you to agree or improve upon):  {{ feedback.new_hypothesis }}
  Reasoning for new hypothesis:  {{ feedback.reason }}
  Did changing to this hypothesis work? (focus on the change):  {{ feedback.decision }}
  {% endfor %}

hypothesis_output_format: |-
  The output should follow JSON format. The schema is as follows:
  {
    "hypothesis": "The new hypothesis generated based on the information provided.",
    "reason": "The reason why you generate this hypothesis."
  }

model_hypothesis_specification: |-
  Additional Specifications:
    
    Hypotheses should grow and evolve based on the previous hypothesis. If there is no previous hypothesis, start with something simple. Gradually Build Up Upon previous hypothesis & feedbacks. In each round, hypothesis is different. Pay attention to your previous hypothesis.

    Ensure that the hypothesis focuses on the architecture of a PyTorch model. Each hypothesis should address specific architectural choices such as the type of layers, activation functions, regularization techniques, and the overall structure of the model. Avoid hypotheses related to input features or optimization processes.

  Remember: if there is no hypothesis, start with something simple like MLP.

  Usually, a larger model works better than a smaller one. 

  Logic for generating a new hypothesis: If the previous hypothesis works, try to inherit from it and grow deeper. If the previous hypotheis doesn't work, try to make changes in the current level.

  Sample hypothesis evolution loop: (This is the entire loop, see what stage you are at. We want hypothesis to continue growing.) Levels include **Model Type**, **Layer Configuration**, **Activation Functions**, **Regularization Techniques**

    1st Round Hypothesis: The model should be a CNN. 

    2nd Round Hypothesis (If first round worked: CNN is the model type level, which means that we should extend to the next level, like layer configuration): The model should be a CNN. The CNN should have 5 convolutional layers. (Reasoning: As CNN worked, we now specify the layers specification to grow the hypothesis deeper.)

    3rd Round Hypothesis (If second round didn't work): The model should be a CNN. The CNN should have 3 convolutional layers. (Reasoning: As 5-layer structure didn't work in the 2nd round hypothesis, try something else within the layer configuration level.)

    4th Round Hypothesis (If third round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. (As last round worked, now proceed to the next level: activation functions)
    
    5th Round Hypothesis (If fourth round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.5. (Similar Reasoning & Continuing to Grow to the dropout setup)

    6th Round Hypothesis (If fourth round didn't work):  The model should be a CNN. The CNN should have 5 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.3. (Reasoning: As regularisation rate of 0.5 didn't work, we only change a new regularisation and keep the other elements that worked. This means making changes in the current level.)    

factor_hypothesis_specification: |-
  Additional Specifications:
    Hypotheses should grow and evolve based on the previous hypothesis. If there is no previous hypothesis, start with something simple. Gradually build up upon previous hypotheses and feedback.
    Ensure that the hypothesis focuses on the creation and selection of factors in quantitative finance. Each hypothesis should address specific factor characteristics such as type (momentum, value, quality), calculation methods, or inclusion criteria. Avoid hypotheses related to model architecture or optimization processes.

  Sample Hypotheses (Only learn from the format as these are not the knowledge):
    - "Include a momentum factor based on the last 12 months' returns."
    - "Add a value factor calculated as the book-to-market ratio."
    - "Incorporate a quality factor derived from return on equity (ROE)."
    - "Use a volatility factor based on the standard deviation of returns over the past 6 months."
    - "Include a sentiment factor derived from news sentiment scores."
    - "The momentum factor should be calculated using a 6-month look-back period."
    - "Combine value and momentum factors using a weighted average approach."
    - "Filter stocks by market capitalization before calculating the factors."

factor_hypothesis_specification: |-
  Additional Specifications:
    - Hypotheses should grow and evolve based on the previous hypothesis. If there is no previous hypothesis, start with something simple.
    - Gradually build upon previous hypotheses and feedback.
    - Ensure that the hypothesis focuses on the creation and selection of factors in quantitative finance.
    - Each hypothesis should address specific factor characteristics such as type (momentum, value, quality), calculation methods, or inclusion criteria.
    - Avoid hypotheses related to model architecture or optimization processes.
    - If a hypothesis can be improved further, refine it. If it achieves the desired results, explore a new direction. Previous factors exceeding SOTA (State of the Art) are preserved and combined with new factors for subsequent evaluations.

  Guiding Principles:
  1. Diversity and Depth:
    - Ensure a wide range of factor types, incorporating various financial dimensions (e.g., momentum, value, quality, volatility, sentiment).
    - Explore different calculation methods and inclusion criteria to understand their impact.
    - Consider combining multiple factors or filtering criteria for more sophisticated hypotheses.

  2. Iterative Improvement:
    - Build upon previous hypotheses, incorporating feedback and observed results.
    - Aim for continuous refinement and complexity over iterations, starting from basic factors to more advanced combinations and techniques.

  3. Contextual Relevance:
    - Tailor hypotheses to the specific financial context and current market conditions.
    - Leverage domain knowledge and recent financial research to inform hypothesis creation.

  Sample Hypotheses (Use the format for guidance, not the specific content):
  - "Include a momentum factor based on the last 12 months' returns."
  - "Add a value factor calculated as the book-to-market ratio."
  - "Incorporate a quality factor derived from return on equity (ROE)."
  - "Use a volatility factor based on the standard deviation of returns over the past 6 months."
  - "Include a sentiment factor derived from news sentiment scores."
  - "The momentum factor should be calculated using a 6-month look-back period."
  - "Combine value and momentum factors using a weighted average approach."
  - "Filter stocks by market capitalization before calculating the factors."
  - "Explore a liquidity factor based on the trading volume and bid-ask spread."
  - "Investigate the impact of an earnings surprise factor calculated from recent earnings announcements."
  - "Develop a composite factor integrating ESG (Environmental, Social, Governance) scores with traditional financial metrics."

  Detailed Workflow:
  1. Initial Hypothesis:
    - Begin with a simple factor, such as "Include a momentum factor based on the last 12 months' returns."

  2. Refine Hypothesis:
    - If the initial hypothesis is promising, refine it further, e.g., "The momentum factor should be calculated using a 6-month look-back period."

  3. Combine Factors:
    - As individual factors show potential, combine them, e.g., "Combine value and momentum factors using a weighted average approach."

  4. Contextual Adjustments:
    - Adjust factors based on market conditions or new financial insights, e.g., "Incorporate a quality factor derived from return on equity (ROE)."

  5. Advanced Hypotheses:
    - Explore sophisticated combinations or new types of factors, e.g., "Develop a composite factor integrating ESG scores with traditional financial metrics."

  Remember: If a hypothesis achieves the desired results, start a new direction while preserving the effective factors from previous hypotheses. New evaluations should combine the newly proposed factors with previously successful factors that surpassed SOTA.


factor_experiment_output_format: |-
  The output should follow JSON format. The schema is as follows:
  {
      "factor name 1": {
          "description": "description of factor 1",
          "formulation": "latex formulation of factor 1",
          "variables": {
              "variable or function name 1": "description of variable or function 1",
              "variable or function name 2": "description of variable or function 2"
          }
      },
      "factor name 2": {
          "description": "description of factor 1",
          "formulation": "latex formulation of factor 2",
          "variables": {
              "variable or function name 1": "description of variable or function 1",
              "variable or function name 2": "description of variable or function 2"
          }
      }
      # Don't add ellipsis (...) or any filler text that might cause JSON parsing errors here!
  }

model_experiment_output_format: |-
  So far please only design one model to test the hypothesis! 
  The output should follow JSON format. The schema is as follows: 
  {
    "model_name 1 (The name of the model)": {
        "description": "A detailed description of the model",
        "architecture": "A detailed description of the model's architecture, e.g., neural network layers or tree structures",
        "hyperparameters": {
            "hyperparameter_name_1": "value of hyperparameter 1",
            "hyperparameter_name_2": "value of hyperparameter 2",
            "hyperparameter_name_3": "value of hyperparameter 3"
        },
        "model_type": "Tabular or TimeSeries"  # Should be one of "Tabular" or "TimeSeries"
    },
    "model_name 2 (The name of the model)": {
        ...
    }
  }
  Usually a larger model works better than a smaller one. Hence, the parameters should be larger.

factor_feedback_generation:
  system: |-
    You are a professional result analysis assistant in data-driven R&D.
    The task is described in the following scenario:
    {{ scenario }}
    You will receive a hypothesis, multiple tasks with their factors, and some results.
    Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA results, and suggest improvements or new directions.
    Please provide detailed and constructive feedback for the future exploration.
    Please respond in JSON format, and example JSON Structure for Result Analysis:
    {
      "Observations": "Your overall observations here",
      "Feedback for Hypothesis": "Observations related to the hypothesis",
      "New Hypothesis": "Put your new hypothesis here.",
      "Reasoning": "Provide reasoning for the hypothesis here.",
      "Replace Best Result": "yes or no"
    }
  user: |-
    Target hypothesis: 
    {{ hypothesis_text }}
    Tasks and Factors:
    {{ task_details }}
    Combined Results: 
    {{ combined_result }}
    Analyze the combined result in the context of its ability to:
    1. Support or refute the hypothesis.
    2. Show improvement or deterioration compared to the SOTA experiment.

    Evaluation Metrics Explanations:
    Below are the financial meanings of each metric, which should be used to judge the results:

    - Rank ICIR: Evaluates the stability and average level of Rank IC. Rank ICIR = mean(Rank IC) / std(Rank IC).
    - 1day.excess_return_without_cost.max_drawdown: Measures the maximum loss from a peak to a trough without considering transaction costs.
    - 1day.excess_return_without_cost.information_ratio: Evaluates the excess return per unit of risk without considering transaction costs.
    - 1day.excess_return_with_cost.max_drawdown: Measures the maximum loss from a peak to a trough considering transaction costs.
    - 1day.excess_return_without_cost.annualized_return: Annualized return without considering transaction costs.
    - 1day.excess_return_with_cost.annualized_return: Annualized return considering transaction costs.
    - IC: Measures the correlation between predicted returns (\hat{y}) and actual returns (y), using Pearson correlation.
    - 1day.excess_return_with_cost.information_ratio: Evaluates the excess return per unit of risk considering transaction costs.

    When judging the results:
    1. Prioritize metrics that consider transaction costs (with cost):
    - These metrics provide a more accurate representation of real-world performance.
    2. Evaluate all metrics:
        - Compare the combined results against the current best results across all metrics to get a comprehensive view of performance.
    3. Focus on the annualized return considering transaction costs:
        - This metric is particularly important as it gives a clear picture of long-term profitability.
    4. Recommendation for replacement:
        - If the new factor demonstrates a significant improvement in the annualized return considering transaction costs, it should be recommended to replace the current best result, even if other metrics show minor variations.

    Please provide detailed feedback and recommend whether to replace the best result if the new factor proves superior.

model_feedback_generation:
  system: |-
    You are a professional result analysis assistant. You will receive a result and a hypothesis.
    Your task is to provide feedback on how well the result supports or refutes the hypothesis by judging from the observation of performance increase or decrease.
    Please provide detailed and constructive feedback. Note that as hypothesis evolve, a general trend should be that the model grows larger. 
    Example JSON Structure for Result Analysis:
    {
      "Observations": "Your overall observations here",
      "Feedback for Hypothesis": "Observations related to the hypothesis",
      "New Hypothesis": "Put your new hypothesis here.",
      "Reasoning": "Provide reasoning for the hypothesis here.",
      "Decision": <true or false>,
    }
    
    Logic for generating a new hypothesis: If the previous hypothesis works, try to inherit from it and grow deeper. If the previous hypotheis doesn't work, try to make changes in the current level.

    Sample hypothesis evolution loop: (This is the entire loop, see what stage you are at. We want hypothesis to continue growing.) Levels include **Model Type**, **Layer Configuration**, **Activation Functions**, **Regularization Techniques**

      1st Round Hypothesis: The model should be a CNN. 

      2nd Round Hypothesis (If first round worked: CNN is the model type level, which means that we should extend to the next level, like layer configuration): The model should be a CNN. The CNN should have 5 convolutional layers. (Reasoning: As CNN worked, we now specify the layers specification to grow the hypothesis deeper.)

      3rd Round Hypothesis (If second round didn't work): The model should be a CNN. The CNN should have 3 convolutional layers. (Reasoning: As 5-layer structure didn't work in the 2nd round hypothesis, try something else within the layer configuration level.)

      4th Round Hypothesis (If third round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. (As last round worked, now proceed to the next level: activation functions)
      
      5th Round Hypothesis (If fourth round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.5. (Similar Reasoning & Continuing to Grow to the dropout setup)

      6th Round Hypothesis (If fourth round didn't work):  The model should be a CNN. The CNN should have 5 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.3. (Reasoning: As regularisation rate of 0.5 didn't work, we only change a new regularisation and keep the other elements that worked. This means making changes in the current level.)    

    
  user: |-
    We are in an experiment of finding hypothesis and validating or rejecting them so that in the end we have a powerful model generated.
    Here are the context: {{context}}. 

    {% if last_hypothesis %} 
    Last Round Information:
    Hypothesis: {{last_hypothesis.hypothesis}}
    Task: {{last_task}}
    Result: {{last_result}}
    {% else %}
    This is the first round. No previous information available. As long as the performance is not too negative (eg.ICIR is greater than 0), treat it as successful. Do not set the threshold too high.  
    {% endif %} 
    
    Now let's come to this round. You will receive the result and you will evaluate if the performance increases or decreases. 
    Hypothesis: {{hypothesis.hypothesis}}
    Relevant Reasoning: {{hypothesis.reason}}
    Result: {{exp.result}}

    Compare and observe. Which result has a better return and lower risk? If the performance increases, the hypothesis should be considered positive (working). 
    Hence, with the hypotheses, relevant reasoning, and results in mind (comparison), provide detailed and constructive feedback and suggest a new hypothesis.