NexQuant/rdagent/scenarios/qlib/prompts.yaml

hypothesis_and_feedback: |-
  {% for hypothesis, experiment, feedback in trace.hist %}
  Hypothesis {{ loop.index }}: {{ hypothesis }}
  Corresponding Code (that leads to the difference in performance): {{experiment.sub_workspace_list[0].code_dict.get("model.py")}}
  Observation on the result with the hypothesis: {{ feedback.observations }}
  Feedback on the original hypothesis:  {{ feedback.hypothesis_evaluation }}
  New Feedback for Context (For you to agree or improve upon):  {{ feedback.new_hypothesis }}
  Reasoning for new hypothesis:  {{ feedback.reason }}
  Did changing to this hypothesis work? (focus on the change):  {{ feedback.decision }}
  {% endfor %}

hypothesis_output_format: |-
  The output should follow JSON format. The schema is as follows:
  {
  "hypothesis": "The new hypothesis generated based on the information provided.",
  "reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them.",
  "concise_reason": "Two-line summary. First line focuses on a concise justification for the change. Second line generalizes a knowledge statement.",
  "concise_observation": "One line summary. It focuses on the observation of the given scenario, data characteristics, or previous experiences (failures & succeses).",
  "concise_justification": "One line summary. Justify the hypothesis based on theoretical principles or initial assumptions.",
  "concise_knowledge": "One line summary. Transferable knowledge based on theoretical principles. Use conditional grammar. eg. "If...., ..; When..., .; and etc" Make sure that you state things clearly without ambiguity. Eg. avoid saying "previous hypothesis", because one wouldn't know what that is."
  }



model_hypothesis_specification: |-
  Additional Specifications:
    
    Hypotheses should grow and evolve based on the previous hypothesis. If there is no previous hypothesis, start with something simple. Gradually Build Up Upon previous hypothesis & feedbacks. In each round, hypothesis is different. Pay attention to your previous hypothesis.

    Ensure that the hypothesis focuses on the architecture of a PyTorch model. Each hypothesis should address specific architectural choices such as the type of layers, activation functions, regularization techniques, and the overall structure of the model. Avoid hypotheses related to input features or optimization processes.

  Remember: if there is no hypothesis, start with something simple like MLP.

  Usually, a larger model works better than a smaller one. 

  Logic for generating a new hypothesis: If the previous hypothesis works, try to inherit from it and grow deeper. If the previous hypotheis doesn't work, try to make changes in the current level.

  Sample hypothesis evolution loop: (This is the entire loop, see what stage you are at. We want hypothesis to continue growing.) Levels include **Model Type**, **Layer Configuration**, **Activation Functions**, **Regularization Techniques**

    1st Round Hypothesis: The model should be a CNN. 

    2nd Round Hypothesis (If first round worked: CNN is the model type level, which means that we should extend to the next level, like layer configuration): The model should be a CNN. The CNN should have 5 convolutional layers. (Reasoning: As CNN worked, we now specify the layers specification to grow the hypothesis deeper.)

    3rd Round Hypothesis (If second round didn't work): The model should be a CNN. The CNN should have 3 convolutional layers. (Reasoning: As 5-layer structure didn't work in the 2nd round hypothesis, try something else within the layer configuration level.)

    4th Round Hypothesis (If third round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. (As last round worked, now proceed to the next level: activation functions)
    
    5th Round Hypothesis (If fourth round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.5. (Similar Reasoning & Continuing to Grow to the dropout setup)

    6th Round Hypothesis (If fourth round didn't work):  The model should be a CNN. The CNN should have 5 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.3. (Reasoning: As regularisation rate of 0.5 didn't work, we only change a new regularisation and keep the other elements that worked. This means making changes in the current level.)    

factor_hypothesis_specification: |-
  1. **Type of Factor and Financial Trends:**
    - Define the type of factor introduced.
    - Explain the financial trends or market behaviors indicated by this factor.
    - Omit unnecessary or redundant details.

  2. **Simple and Effective Factors First:**
    - Start with factors that are simple and likely effective.
    - Concisely explain why these factors are expected to work.
    - Avoid complex or combined factors initially.

  3. **Gradual Complexity Increase:**
    - Introduce more complex factors as more experimental results are gathered.
    - Discuss potential advantages and complexities.
    - Combine factors only after simpler ones are tested and validated.

  4. **New Directions and Optimizations:**
    - If a new direction is needed, explain why based on financial principles, economic theories, or market behaviors.
    - Suggest only one new direction at a time for clarity.
    - If a previous hypothesis did not surpass SOTA but seems optimizable, you may continue in the same direction.
    - Highlight that factors surpassing SOTA are included in the library to avoid re-implementation.

  5. **1-3 Factors per Generation:**
    - Ensure each generation produces 1-3 factors.
    - Balance simplicity and complexity to build a robust factor library.

factor_experiment_output_format: |-
  The output should follow JSON format. The schema is as follows:
  {
      "factor name 1": {
          "description": "description of factor 1",
          "formulation": "latex formulation of factor 1",
          "variables": {
              "variable or function name 1": "description of variable or function 1",
              "variable or function name 2": "description of variable or function 2"
          }
      },
      "factor name 2": {
          "description": "description of factor 1",
          "formulation": "latex formulation of factor 2",
          "variables": {
              "variable or function name 1": "description of variable or function 1",
              "variable or function name 2": "description of variable or function 2"
          }
      }
      # Don't add ellipsis (...) or any filler text that might cause JSON parsing errors here!
  }

model_experiment_output_format: |-
  So far please only design one model to test the hypothesis! 
  The output should follow JSON format. The schema is as follows: 
  {
    "model_name 1 (The name of the model)": {
        "description": "A detailed description of the model",
        "formulation": "A LaTeX formula representing the model's formulation",
        "architecture": "A detailed description of the model's architecture, e.g., neural network layers or tree structures",
        "variables": {
            "\\hat{y}_u": "The predicted output for node u",
            "variable_name_2": "Description of variable 2",
            "variable_name_3": "Description of variable 3"
        },
        "hyperparameters": {
            "hyperparameter_name_1": "value of hyperparameter 1",
            "hyperparameter_name_2": "value of hyperparameter 2",
            "hyperparameter_name_3": "value of hyperparameter 3"
        },
        "model_type": "Tabular or TimeSeries"  # Should be one of "Tabular" or "TimeSeries"
    },
    "model_name 2 (The name of the model)": {
        ...
    }
  }
  Usually a larger model works better than a smaller one. Hence, the parameters should be larger.

factor_feedback_generation:
  system: |-
    You are a professional financial result analysis assistant in data-driven R&D. 
    The task is described in the following scenario:

    {{ scenario }}
    
    You will receive a hypothesis, multiple tasks with their factors, their results, and the SOTA result. 
    Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA (State of the Art) results, and suggest improvements or new directions.
    
    Please understand the following operation logic and then make your feedback that is suitable for the scenario:
      1. Logic Explanation:
          - If the previous hypothesis factor surpasses the SOTA, include this factor in the SOTA factor library.
          - New experiments will generate new factors, which will be combined with the factors in the SOTA library.
          - These combined factors will be backtested and compared against the current SOTA to continuously iterate.
      2. Development Directions:
          - New Direction:
              - Propose a new factor direction for exploration and development.
          - Optimization of Existing Direction:
              - If the previous experiment's factor replaced the SOTA, suggest further improvements to that factor.
              - Clearly specify the differences in name and improvements compared to the previous factor.
          - Continued Research:
              - If the previous experiment's factor did not replace the SOTA, suggest ways to optimize and develop factors in this direction.
      3. Final Goal:
          - The ultimate goal is to continuously accumulate factors that surpass each iteration to maintain the best SOTA.
    
      When judging the results:
      1. **Recommendation for Replacement:**
        - If the new factor shows a significant improvement in the annualized return without transaction costs, recommend it to replace the current best result.
        - If the annualized return and any other single metric are better than SOTA, recommend the replacement.
        - Minor variations in other metrics are acceptable as long as the annualized return improves.

    Consider Changing Direction for Significant Gaps with SOTA:
      - If the new results significantly differ from the SOTA, consider exploring a new direction.
      - Avoid re-implementing previous factors as those that surpassed SOTA are already included in the factor library and will be used in each run.

    Please provide detailed and constructive feedback for future exploration.
    Respond in JSON format. Example JSON structure for Result Analysis:
    {
      "Observations": "Your overall observations here",
      "Feedback for Hypothesis": "Observations related to the hypothesis",
      "New Hypothesis": "Your new hypothesis here",
      "Reasoning": "Reasoning for the new hypothesis",
      "Replace Best Result": "yes or no"
    }
  user: |-
    Target hypothesis: 
    {{ hypothesis_text }}
    Tasks and Factors:
    {% for task in task_details %}
      - {{ task.factor_name }}: {{ task.factor_description }}
        - Factor Formulation: {{ task.factor_formulation }}
        - Variables: {{ task.variables }}
        - Factor Implementation: {{ task.factor_implementation }}
        {% if task.factor_implementation == "False" %}
        **Note: This factor was not implemented in the current experiment. Only the hypothesis for implemented factors can be verified.**
        {% endif %}
    {% endfor %}
    Combined Results: 
    {{ combined_result }}
    
    Analyze the combined result in the context of its ability to:
    1. Support or refute the hypothesis.
    2. Show improvement or deterioration compared to the SOTA experiment.

    Evaluation Metrics Explanations:
    Below are the financial meanings of each metric, which should be used to judge the results:

    - 1day.excess_return_without_cost.max_drawdown: Measures the maximum loss from a peak to a trough without considering transaction costs. (the smaller the better)
    - 1day.excess_return_without_cost.information_ratio: Evaluates the excess return per unit of risk without considering transaction costs. (the bigger the better)
    - 1day.excess_return_without_cost.annualized_return: Annualized return without considering transaction costs. (the bigger the better)
    - IC: Measures the correlation between predicted returns (\hat{y}) and actual returns (y), using Pearson correlation. (the bigger the better)

    When judging the results:
      1. **Recommendation for Replacement:**
        - If the new factor shows a significant improvement in the annualized return without transaction costs, recommend it to replace the current best result.
        - If the annualized return and any other single metric are better than SOTA, recommend the replacement.
        - Minor variations in other metrics are acceptable as long as the annualized return improves.

    Consider Changing Direction for Significant Gaps with SOTA:
      - If the new results significantly differ from the SOTA, consider exploring a new direction.
      - Avoid re-implementing previous factors as those that surpassed SOTA are already included in the factor library and will be used in each run.

    Note: Only factors with 'Factor Implementation' as True are implemented and tested in this experiment. If 'Factor Implementation' is False, the hypothesis for that factor cannot be verified in this run.

model_feedback_generation:
  system: |-
    You are a professional result analysis assistant. You will receive a result and a hypothesis.
    Your task is to provide feedback on how well the result supports or refutes the hypothesis by judging from the observation of performance increase or decrease.
    Please provide detailed and constructive feedback. Note that as hypothesis evolve, a general trend should be that the model grows larger. 
    Example JSON Structure for Result Analysis:
    {
      "Observations": "Your overall observations here",
      "Feedback for Hypothesis": "Observations related to the hypothesis",
      "New Hypothesis": "Put your new hypothesis here.",
      "Reasoning": "Provide reasoning for the hypothesis here.",
      "Decision": <true or false>,
    }

    Focus on the changes in hypothesis and justify why do hypothesis evolve like this. Also, increase complexity as the hypothesis evolves  (give more layers, more neurons, and etc)
    
    Logic for generating a new hypothesis: If the previous hypothesis works, try to inherit from it and grow deeper. If the previous hypotheis doesn't work, try to make changes in the current level.

    Sample hypothesis evolution loop: (This is the entire loop, see what stage you are at. We want hypothesis to continue growing.) Levels include **Model Type**, **Layer Configuration**, **Activation Functions**, **Regularization Techniques**

      1st Round Hypothesis: The model should be a CNN. 

      2nd Round Hypothesis (If first round worked: CNN is the model type level, which means that we should extend to the next level, like layer configuration): The model should be a CNN. The CNN should have 5 convolutional layers. (Reasoning: As CNN worked, we now specify the layers specification to grow the hypothesis deeper.)

      3rd Round Hypothesis (If second round didn't work): The model should be a CNN. The CNN should have 3 convolutional layers. (Reasoning: As 5-layer structure didn't work in the 2nd round hypothesis, try something else within the layer configuration level.)

      4th Round Hypothesis (If third round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. (As last round worked, now proceed to the next level: activation functions)
      
      5th Round Hypothesis (If fourth round worked): The model should be a CNN. The CNN should have 3 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.5. (Similar Reasoning & Continuing to Grow to the dropout setup)

      6th Round Hypothesis (If fourth round didn't work):  The model should be a CNN. The CNN should have 5 convolutional layers. Use Leaky ReLU activation for all layers. Use dropout regularization with a rate of 0.3. (Reasoning: As regularisation rate of 0.5 didn't work, we only change a new regularisation and keep the other elements that worked. This means making changes in the current level.)    

    
  user: |-
    We are in an experiment of finding hypothesis and validating or rejecting them so that in the end we have a powerful model generated.
    Here are the context: {{context}}. 

    {% if last_hypothesis %} 
    Last Round Information:
    Hypothesis: {{last_hypothesis.hypothesis}}
    Task: {{last_task}}
    Code Implemented: {{last_code}}
    Result: {{last_result}}
    {% else %}
    This is the first round. No previous information available. As long as the performance is not too negative (eg.ICIR is greater than 0), treat it as successful. Do not set the threshold too high.  
    {% endif %} 
    
    Now let's come to this round. You will receive the result and you will evaluate if the performance increases or decreases. 
    Hypothesis: {{hypothesis.hypothesis}}
    Experiment Setup: {{exp.sub_tasks[0]}}
    Code Implemented: {{exp.sub_workspace_list[0].code_dict.get("model.py")}}
    Relevant Reasoning: {{hypothesis.reason}}
    Result: {{exp.result}}

    Compare and observe. Which result has a better return and lower risk? If the performance increases, the hypothesis should be considered positive (working). 
    Hence, with the hypotheses, relevant reasoning, and results in mind (comparison), provide detailed and constructive feedback and suggest a new hypothesis.