mirror of
https://github.com/NicolasBohn/NexQuant.git
synced 2026-07-27 15:37:44 +00:00
feat: Fix 1min data integration and centralize all prompts
- Fix daily/1min contradiction in factor_experiment_loader prompts - Rename daily_pv.h5 to intraday_pv.h5 (generate.py, utils.py, README) - Fix FactorDatetimeDailyEvaluator to accept 1min bars as correct - Add _write_run_log() to log every factor attempt to results/logs/ - Add _ensure_results_dirs() to create all result directories - Extract all 44 prompt YAML files to prompts/ centralized directory - Add prompts/INDEX.md for navigation Tests: 93 passed
This commit is contained in:
TPTBusiness
@@ -9,7 +9,7 @@ NOTE: **key is always "data" for all hdf5 files **.
|
||||
# Here is a short description about the data
|
||||
| Filename | Description |
|
||||
| -------------- | -----------------------------------------------------------------|
|
||||
| "daily_pv.h5" | EURUSD 1-minute OHLCV intraday data (2020-2026). |
|
||||
| "intraday_pv.h5" | EURUSD 1-minute OHLCV intraday data (2020-2026). |
|
||||
|
||||
# For different data, We have some basic knowledge for them
|
||||
|
||||
|
||||
+2
-2
@@ -11,10 +11,10 @@ qlib.init(provider_uri="~/.qlib/qlib_data/eurusd_1min_data")
|
||||
fields = ["$open", "$close", "$high", "$low", "$volume"]
|
||||
data = (D.features(["EURUSD"], fields, start_time="2022-03-14", end_time="2026-03-20", freq="1min")
|
||||
.swaplevel().sort_index())
|
||||
data.to_hdf("./daily_pv_all.h5", key="data")
|
||||
data.to_hdf("./intraday_pv_all.h5", key="data")
|
||||
data_debug = (D.features(["EURUSD"], fields, start_time="2024-01-01", end_time="2026-03-20", freq="1min")
|
||||
.swaplevel().sort_index())
|
||||
data_debug.to_hdf("./daily_pv_debug.h5", key="data")
|
||||
data_debug.to_hdf("./intraday_pv_debug.h5", key="data")
|
||||
print(f"Done: {data.shape[0]} rows")
|
||||
"""],
|
||||
capture_output=False
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# Predix Prompts Index
|
||||
|
||||
Centralized location for all LLM prompts used in the Predix trading system.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
prompts/
|
||||
├── standard_prompts.yaml # Main EURUSD trading prompts (Factor Discovery, Evolution, Model Coder)
|
||||
├── local/ # Your improved prompts (NOT in Git!)
|
||||
├── patches/ # Override patches for Qlib scenarios
|
||||
│ ├── qlib_experiment_prompts.yaml
|
||||
│ ├── qlib_rd_loop_prompts.yaml
|
||||
│ └── qlib_scenarios_prompts.yaml
|
||||
├── app/ # Application-level prompts
|
||||
│ ├── ci/prompts.yaml # CI/CD prompts
|
||||
│ ├── qlib_rd_loop/prompts.yaml # Qlib RD Loop hypothesis generation
|
||||
│ ├── utils/prompts.yaml # APE prompts
|
||||
│ └── finetune/prompts.yaml # Finetune prompts
|
||||
├── components/ # Component prompts
|
||||
│ ├── agent/prompts.yaml # Context7 MCP documentation search
|
||||
│ ├── proposal/prompts.yaml # Hypothesis proposal generation
|
||||
│ ├── coder/
|
||||
│ │ ├── factor_coder/prompts.yaml # Factor code evaluator
|
||||
│ │ ├── model_coder/prompts.yaml # Model code evaluator
|
||||
│ │ ├── rl/prompts.yaml # RL trading coder (Chinese)
|
||||
│ │ ├── CoSTEER/prompts.yaml # Component analysis
|
||||
│ │ ├── finetune/prompts.yaml # LLM finetuning coder
|
||||
│ │ └── data_science/ # Data science pipeline
|
||||
│ │ ├── ensemble/prompts.yaml
|
||||
│ │ ├── feature/prompts.yaml
|
||||
│ │ ├── model/prompts.yaml
|
||||
│ │ ├── pipeline/prompts.yaml
|
||||
│ │ ├── raw_data_loader/prompts.yaml
|
||||
│ │ ├── share/prompts.yaml
|
||||
│ │ └── workflow/prompts.yaml
|
||||
├── scenarios/ # Scenario-specific prompts
|
||||
│ ├── qlib/ # Qlib EURUSD trading
|
||||
│ │ ├── prompts.yaml # Main Qlib scenario
|
||||
│ │ ├── experiment/prompts.yaml
|
||||
│ │ └── factor_experiment_loader/prompts.yaml
|
||||
│ ├── data_science/ # Data science scenarios
|
||||
│ │ ├── dev/prompts.yaml
|
||||
│ │ ├── runner/dev/prompts.yaml
|
||||
│ │ ├── proposal/exp_gen/prompts.yaml
|
||||
│ │ ├── proposal/exp_gen/prompts_v2.yaml # Largest file (82KB)
|
||||
│ │ ├── proposal/exp_gen/select/prompts.yaml
|
||||
│ │ └── scen/prompts.yaml
|
||||
│ ├── finetune/ # LLM finetuning
|
||||
│ │ ├── dev/prompts.yaml
|
||||
│ │ ├── proposal/prompts.yaml
|
||||
│ │ └── scen/prompts.yaml
|
||||
│ ├── kaggle/ # Kaggle competition
|
||||
│ │ ├── prompts.yaml
|
||||
│ │ ├── experiment/prompts.yaml
|
||||
│ │ └── knowledge_management/prompts.yaml
|
||||
│ ├── rl/ # Reinforcement learning (Chinese)
|
||||
│ │ ├── dev/prompts.yaml
|
||||
│ │ └── proposal/prompts.yaml
|
||||
│ └── general_model/prompts.yaml
|
||||
└── utils/ # Utility prompts
|
||||
└── prompts.yaml # Filter redundant text
|
||||
```
|
||||
|
||||
## Active Prompts for EURUSD Trading
|
||||
|
||||
The following prompts are actively used in the `rdagent fin_quant` trading loop:
|
||||
|
||||
| Priority | File | Purpose |
|
||||
|----------|------|---------|
|
||||
| 1 | `standard_prompts.yaml` | Factor Discovery, Factor Evolution, Model Coder, Trading Strategy |
|
||||
| 2 | `rdagent/app/qlib_rd_loop/prompts.yaml` | Hypothesis generation for Qlib RD Loop |
|
||||
| 3 | `rdagent/scenarios/qlib/prompts.yaml` | Qlib scenario: hypothesis feedback, output format |
|
||||
| 4 | `rdagent/scenarios/qlib/factor_experiment_loader/prompts.yaml` | Factor viability, relevance, duplicate checks |
|
||||
| 5 | `rdagent/scenarios/qlib/experiment/prompts.yaml` | Qlib experiment background, factor interface |
|
||||
| 6 | `rdagent/components/coder/factor_coder/prompts.yaml` | Code evaluation, final decision |
|
||||
| 7 | `patches/qlib_scenarios_prompts.yaml` | EURUSD-specific overrides (1min data, market sessions) |
|
||||
| 8 | `patches/qlib_rd_loop_prompts.yaml` | EURUSD hypothesis generation overrides |
|
||||
|
||||
## Key Changes (April 2026)
|
||||
|
||||
- **Fixed:** All "daily frequency" references changed to "intraday 1-minute bars"
|
||||
- **Fixed:** `daily_pv.h5` renamed to `intraday_pv.h5` in data descriptions
|
||||
- **Fixed:** `FactorDatetimeDailyEvaluator` now accepts 1min-30min bars as correct for EURUSD
|
||||
|
||||
## Total Files: 44 YAML files
|
||||
## Total Size: ~486 KB
|
||||
@@ -0,0 +1,117 @@
|
||||
generate_lint_command_template: |
|
||||
Please generate a command to lint or format a {language} repository.
|
||||
Here are some information about different linting tools ```{linting_tools}```
|
||||
linting_system_prompt_template: |
|
||||
You are a software engineer. You can write code to a high standard and are adept at solving {language} linting problems.
|
||||
session_manual_template: |
|
||||
There are some problems with the code you provided, please modify the code again according to the instruction and return the errors list you modified.
|
||||
|
||||
Instruction:
|
||||
{operation}
|
||||
|
||||
Your response format should be like this:
|
||||
|
||||
```python
|
||||
<modified code>
|
||||
```
|
||||
|
||||
```json
|
||||
{{
|
||||
"errors": ["<Line Number>:<Error Start Position> <Error Code>", ...]
|
||||
}}
|
||||
```
|
||||
session_normal_template: |
|
||||
Please modify this code snippet based on the lint info. Here is the code snippet:
|
||||
```Python
|
||||
{code}
|
||||
```
|
||||
|
||||
-----Lint info-----
|
||||
{lint_info}
|
||||
-------------------
|
||||
|
||||
The lint info contains one or more errors. Different errors are separated by blank lines. Each error follows this format:
|
||||
-----Lint info format-----
|
||||
<Line Number>:<Error Start Position> <Error Code> <Error Message>
|
||||
<Error Position (maybe multiple lines)>
|
||||
<Helpful Information (sometimes have)>
|
||||
--------------------------
|
||||
The error code is an abbreviation set by the checker for ease of describing the error. The error position includes the relevant code around the error, and the helpful information provides useful information or possible fix method.
|
||||
|
||||
Please simply reply the code after you fix all linting errors. You should be aware of the following:
|
||||
1. The indentation of the code should be consistent with the original code.
|
||||
2. You should just replace the code I provided you, which starts from line {start_line} to line {end_line}.
|
||||
3. You'll need to add line numbers to the modified code which starts from {start_lineno}.
|
||||
4. You don't need to add comments to explain your changes.
|
||||
Please wrap your code with following format:
|
||||
|
||||
```python
|
||||
<your code..>
|
||||
```
|
||||
session_start_template: |
|
||||
Please modify the Python code based on the lint info.
|
||||
Due to the length of the code, I will first tell you the entire code, and then each time I ask a question, I will extract a portion of the code and tell you the error information contained in this code segment.
|
||||
You need to fix the corresponding error in the code segment and return the code that can replace the corresponding code segment.
|
||||
|
||||
The Python code is from a complete Python project file. Each line of the code is annotated with a line number, separated from the original code by three characters ("<white space>|<white space>"). The vertical bars are aligned.
|
||||
Here is the complete code, please be prepared to fix it:
|
||||
```Python
|
||||
{code}
|
||||
```
|
||||
suffix2language_template: |
|
||||
Here are the files suffix in one code repo: {suffix}.
|
||||
Please tell me the programming language used in this repo and which language has linting-tools.
|
||||
Your response should follow this template:
|
||||
{{
|
||||
"languages": <languages list>,
|
||||
"languages_with_linting_tools": <languages with lingting tools list>
|
||||
}}
|
||||
user_get_files_contain_lint_commands_template: |
|
||||
You get a file list of a repository. Some files may contain linting rules or linting commands defined by repo authors.
|
||||
Here are the file list:
|
||||
```
|
||||
{file_list}
|
||||
```
|
||||
|
||||
Please find all files that may correspond to linting from it.
|
||||
Please respond with the following JSON template:
|
||||
{{
|
||||
"files": </path/to/file>,
|
||||
}}
|
||||
user_get_makefile_lint_commands_template: |
|
||||
You get a Makefile which contains some linting rules. Here are its content:
|
||||
```
|
||||
{file_text}
|
||||
```
|
||||
Please find executable commands about linting from it.
|
||||
Please respond with the following JSON template:
|
||||
{{
|
||||
"commands": ["python -m xxx --params"...],
|
||||
}}
|
||||
user_template_for_code_snippet: |
|
||||
Please modify the Python code based on the lint info.
|
||||
-----Python Code-----
|
||||
{code}
|
||||
---------------------
|
||||
|
||||
-----Lint info-----
|
||||
{lint_info}
|
||||
-------------------
|
||||
|
||||
The Python code is a snippet from a complete Python project file. Each line of the code is annotated with a line number, separated from the original code by three characters ("<white space>|<white space>"). The vertical bars are aligned.
|
||||
|
||||
The lint info contains one or more errors. Different errors are separated by blank lines. Each error follows this format:
|
||||
-----Lint info format-----
|
||||
<Line Number>:<Error Start Position> <Error Code> <Error Message>
|
||||
<Error Context (multiple lines)>
|
||||
<Helpful Information (last line)>
|
||||
--------------------------
|
||||
The error code is an abbreviation set by the checker for ease of describing the error. The error context includes the relevant code around the error, and the helpful information suggests possible fixes.
|
||||
|
||||
Please simply reply the code after you fix all linting errors.
|
||||
The code you return does not require line numbers, and should just replace the code I provided you, and does not require comments.
|
||||
Please wrap your code with following format:
|
||||
|
||||
```python
|
||||
<your code..>
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
prev_model_eval:
|
||||
system: |-
|
||||
You are a data scientist tasked with evaluating code generation.
|
||||
|
||||
You will receive the following information:
|
||||
- The implemented code
|
||||
|
||||
Focus on these aspects:
|
||||
- Check if the code load the model in the "prev_model/" subfolder.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information. ."
|
||||
"return_checking": "Detect whether the model is loaded from 'prev_model/' subfolder and finetune is prepared based on prev model.",
|
||||
"code": "The code has explicity load the model from 'prev_model/' subfolder and prepares finetune based on prev model.",
|
||||
"final_decision": <true or false in boolean type; only return true when ensuring that the code loads the model from 'prev_model/' subfolder and prepares finetune based on prev model.>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
------------ The implemented code ------------
|
||||
{{code}}
|
||||
@@ -0,0 +1,56 @@
|
||||
hypothesis_generation:
|
||||
system: |-
|
||||
You are an expert quantitative researcher specialized in FX (foreign exchange) trading,
|
||||
specifically EURUSD intraday strategies on 1-MINUTE bars.
|
||||
|
||||
EURUSD domain knowledge you must apply:
|
||||
- Data frequency: 1-minute bars (96 bars = 1 day, 16 bars = 16 minutes)
|
||||
- London session (08:00-12:00 UTC): highest volatility, trending behavior — favor momentum strategies
|
||||
- NY session (13:00-17:00 UTC): second volatility peak, also trending
|
||||
- Asian session (00:00-07:00 UTC): low volatility, mean-reverting behavior
|
||||
- London/NY overlap (13:00-17:00 UTC): strongest directional moves of the day
|
||||
- Weekend gap risk: avoid holding positions after Friday 20:00 UTC
|
||||
- Spread cost: ~1.5 bps per trade — strategies must minimize unnecessary entries
|
||||
- EURUSD is mean-reverting on short windows (<1h), trending on longer (>4h)
|
||||
- Key macro drivers: ECB/Fed rate decisions, NFP (first Friday of month), CPI releases
|
||||
|
||||
Available model types you can propose:
|
||||
- TimeSeries: LSTM, GRU, TCN (Temporal Convolutional Network), Transformer, PatchTST
|
||||
- Tabular: XGBoost, LightGBM, RandomForest (on engineered features)
|
||||
- Hybrid: CNN+LSTM, XGBoost+LSTM ensemble
|
||||
- Statistical: Regime-switching (HMM), Kalman filter
|
||||
|
||||
Available features in the dataset:
|
||||
- OHLCV: open, high, low, close, volume (1min bars)
|
||||
- Returns: ret_1, ret_4, ret_8, ret_16, ret_96
|
||||
- Technical: rsi_14, macd_hist, adx_14, atr_14, bb_pct, stoch_k, cci_14
|
||||
- Volatility: vol_real_4, vol_real_16, vol_ratio, zscore_ret_96
|
||||
- Time/Session: hour, is_london, is_ny, is_overlap, hour_sin, hour_cos
|
||||
- Lags: rsi_14_lag1-8, macd_hist_lag1-8, bb_pct_lag1-8
|
||||
|
||||
Your hypothesis must:
|
||||
1. Specify which session(s) the strategy targets
|
||||
2. Name which model type to use and why it fits EURUSD
|
||||
3. Include a session filter (is_london / is_ny)
|
||||
4. Include a spread filter (only trade when expected |return| > 0.0003)
|
||||
5. Specify target: classification (fwd_sign_4) or regression (fwd_ret_4)
|
||||
|
||||
Please ensure your response is in JSON format:
|
||||
{
|
||||
"hypothesis": "A clear and concise trading hypothesis for EURUSD 1min.",
|
||||
"reason": "Detailed explanation including session, model choice, and expected edge.",
|
||||
"model_type": "One of: TimeSeries / Tabular / XGBoost",
|
||||
"target_session": "london / ny / asian / all",
|
||||
"expected_arr_range": "e.g. 8-12%"
|
||||
}
|
||||
|
||||
user: |-
|
||||
Previously tried approaches and their results:
|
||||
{{ factor_descriptions }}
|
||||
|
||||
Additional context:
|
||||
{{ report_content }}
|
||||
|
||||
Generate a NEW hypothesis that is meaningfully different from what has been tried.
|
||||
Focus on approaches that have NOT been tested yet.
|
||||
Target: beat current best ARR of 9.62%.
|
||||
@@ -0,0 +1,119 @@
|
||||
ape:
|
||||
system: |-
|
||||
We'll provide you with a pair of Chat QA about data science.
|
||||
We are creating solutions for a Kaggle Competition based on the answers.
|
||||
Good questions are crucial for getting good answers.
|
||||
Please suggest how to improve the question.
|
||||
You can analyze based on these aspects:
|
||||
- Is the question complete (is all the information needed to answer the question provided?)
|
||||
|
||||
The conversation will be provided in the following format:
|
||||
|
||||
<question>
|
||||
<part1>
|
||||
...text to describe the question...
|
||||
</part1>
|
||||
<part2>
|
||||
...text to describe the question...
|
||||
</part2>
|
||||
</question>
|
||||
|
||||
<answer>
|
||||
...text to describe the answer.
|
||||
</answer>
|
||||
|
||||
You response should be very concorete and concise(less than 20 words) and focuse on the mentioned aspects, like
|
||||
```
|
||||
Info Missing: the question ask for changing code, but it does not provide the description of current code.
|
||||
```
|
||||
Please be very conversatiive when you propose improvements. Only propose improvements when it becomes impossible to give the answer.
|
||||
|
||||
Don't propose conerete modifications
|
||||
|
||||
user: |-
|
||||
<question>
|
||||
<part1>
|
||||
{{system}}
|
||||
</part1>
|
||||
<part2>
|
||||
{{user}}
|
||||
</part2>
|
||||
</question>
|
||||
|
||||
<answer>
|
||||
{{answer}}
|
||||
</answer>
|
||||
|
||||
optional: |-
|
||||
If you want to suggest modification on the question. Please follow the *SEARCH/REPLACE block* Rules!!!! It is optional.
|
||||
Please make it concise and less than 20 lines!!!
|
||||
|
||||
# *SEARCH/REPLACE block* Rules:
|
||||
|
||||
Every *SEARCH/REPLACE block* must use this format:
|
||||
1. The *FULL* file path alone on a line, verbatim. No bold asterisks, no quotes around it, no escaping of characters, etc.
|
||||
2. The opening fence and code language, eg: ```python
|
||||
3. The start of search block: <<<<<<< SEARCH
|
||||
4. A contiguous chunk of lines to search for in the existing source code
|
||||
5. The dividing line: =======
|
||||
6. The lines to replace into the source code
|
||||
7. The end of the replace block: >>>>>>> REPLACE
|
||||
8. The closing fence: ```
|
||||
|
||||
Use the *FULL* file path, as shown to you by the user.
|
||||
|
||||
Every *SEARCH* section must *EXACTLY MATCH* the existing file content, character for character, including all comments, docstrings, etc.
|
||||
If the file contains code or other data wrapped/escaped in json/xml/quotes or other containers, you need to propose edits to the literal contents of the file, including the container markup.
|
||||
|
||||
*SEARCH/REPLACE* blocks will *only* replace the first match occurrence.
|
||||
Including multiple unique *SEARCH/REPLACE* blocks if needed.
|
||||
Include enough lines in each SEARCH section to uniquely match each set of lines that need to change.
|
||||
|
||||
Keep *SEARCH/REPLACE* blocks concise.
|
||||
Break large *SEARCH/REPLACE* blocks into a series of smaller blocks that each change a small portion of the file.
|
||||
Include just the changing lines, and a few surrounding lines if needed for uniqueness.
|
||||
Do not include long runs of unchanging lines in *SEARCH/REPLACE* blocks.
|
||||
|
||||
Only create *SEARCH/REPLACE* blocks for files that the user has added to the chat!
|
||||
|
||||
To move code within a file, use 2 *SEARCH/REPLACE* blocks: 1 to delete it from its current location, 1 to insert it in the new location.
|
||||
|
||||
Pay attention to which filenames the user wants you to edit, especially if they are asking you to create a new file.
|
||||
|
||||
If you want to put code in a new file, use a *SEARCH/REPLACE block* with:
|
||||
- A new file path, including dir name if needed
|
||||
- An empty `SEARCH` section
|
||||
- The new file's contents in the `REPLACE` section
|
||||
|
||||
To rename files which have been added to the chat, use shell commands at the end of your response.
|
||||
|
||||
If the user just says something like "ok" or "go ahead" or "do that" they probably want you to make SEARCH/REPLACE blocks for the code changes you just proposed.
|
||||
The user will say when they've applied your edits. If they haven't explicitly confirmed the edits have been applied, they probably want proper SEARCH/REPLACE blocks.
|
||||
|
||||
You are diligent and tireless!
|
||||
You NEVER leave comments describing code without implementing it!
|
||||
You always COMPLETELY IMPLEMENT the needed code!
|
||||
|
||||
|
||||
ONLY EVER RETURN CODE IN A *SEARCH/REPLACE BLOCK*!
|
||||
Examples of when to suggest shell commands:
|
||||
|
||||
- If you changed a self-contained html file, suggest an OS-appropriate command to open a browser to view it to see the updated content.
|
||||
- If you changed a CLI program, suggest the command to run it to see the new behavior.
|
||||
- If you added a test, suggest how to run it with the testing tool used by the project.
|
||||
- Suggest OS-appropriate commands to delete or rename files/directories, or other file system operations.
|
||||
- If your code changes add new dependencies, suggest the command to install them.
|
||||
- Etc.
|
||||
|
||||
Here is a example of SEARCH/REPLACE BLOCK to change a function implementation to import.
|
||||
|
||||
<<<<<<< SEARCH
|
||||
def hello():
|
||||
"print a greeting"
|
||||
|
||||
print("hello")
|
||||
=======
|
||||
from hello import hello
|
||||
|
||||
>>>>>>> REPLACE
|
||||
# - Is there any ambiguity in the question?
|
||||
@@ -0,0 +1,59 @@
|
||||
# Context7 MCP Enhanced Query Prompts
|
||||
|
||||
system_prompt: |-
|
||||
You are a helpful assistant.
|
||||
You help to user to search documentation based on error message and provide API reference information.
|
||||
|
||||
context7_enhanced_query_template: |-
|
||||
ERROR MESSAGE:
|
||||
{{error_message}}
|
||||
{{context_info}}
|
||||
IMPORTANT INSTRUCTIONS:
|
||||
1. ENVIRONMENT: The running environment is FIXED and unchangeable - DO NOT suggest pip install, conda install, or any environment modifications.
|
||||
2. DOCUMENTATION SEARCH REQUIREMENTS:
|
||||
- Search for official API documentation related to the error
|
||||
- Focus on parameter specifications, method signatures, and usage patterns
|
||||
- Find compatible alternatives if the original API doesn't exist
|
||||
- Consider the current code context and maintain consistency with existing architecture
|
||||
- Provide API reference information, NOT complete code solutions
|
||||
3. TOOL USAGE REQUIREMENTS:
|
||||
- ⚠️ CRITICAL: For EVERY call to 'resolve-library-id', you MUST follow it with A CORRESPONDING call to 'get-library-docs'
|
||||
- If you call 'resolve-library-id' N times, you MUST call 'get-library-docs' N times (one for each library you found)
|
||||
- Complete the full workflow: resolve → get-docs → analyze → respond
|
||||
- Do NOT provide final answers without first getting detailed documentation via 'get-library-docs'
|
||||
- If 'get-library-docs' returns "Documentation not found" or 404 error, you should never provide guidance based on the library information from 'resolve-library-id'
|
||||
4. RESPONSE FORMAT:
|
||||
- Start with a brief explanation of the root cause
|
||||
- Provide relevant API documentation excerpts
|
||||
- List available parameters and their descriptions
|
||||
- Show method signatures and basic usage patterns
|
||||
- If multiple API options exist, document all viable alternatives
|
||||
5. STRICT CONSTRAINTS:
|
||||
- DO NOT provide complete working code replacements
|
||||
- DO NOT suggest hardware configuration changes (CPU/GPU)
|
||||
- DO NOT recommend architecture or framework changes
|
||||
- DO NOT provide performance optimization suggestions
|
||||
- ONLY provide API documentation and parameter information
|
||||
6. AVOID: Complete code solutions, environment setup, hardware recommendations, architecture suggestions, or performance advice.
|
||||
{{timm_trigger_text}}
|
||||
Example response format:
|
||||
```
|
||||
The error occurs because [brief explanation].
|
||||
API Documentation:
|
||||
- Method: library.function_name(param1, param2, ...)
|
||||
- Parameters:
|
||||
* param1 (type): description
|
||||
* param2 (type): description
|
||||
- Usage pattern: Basic syntax without complete implementation
|
||||
- Alternative APIs (if applicable): list of alternative methods with signatures
|
||||
```
|
||||
Please search the documentation and provide API reference information only.
|
||||
# Special case templates
|
||||
timm_special_case: |-
|
||||
7.⚠️ Special Case: If the code error is indeed related to the 'timm' library (PyTorch Image Models), you MUST return the library ID '/huggingface/pytorch-image-models' as the most relevant match, even if other candidates exist.
|
||||
# Code context template
|
||||
code_context_template: |-
|
||||
CURRENT CODE CONTEXT:
|
||||
```python
|
||||
{{full_code}}
|
||||
```
|
||||
@@ -0,0 +1,10 @@
|
||||
|
||||
analyze_component_prompt_v1_system: |-
|
||||
User is getting a new task that might consist of the components below (given in component_index: component_description):
|
||||
{{all_component_content}}
|
||||
|
||||
You should find out what components does the new task have, and put their indices in a list.
|
||||
Please response the critic in the json format. Here is an example structure for the JSON output, please strictly follow the format:
|
||||
{
|
||||
"component_no_list": the list containing indices of components.
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
ensemble_coder:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
## Task Description
|
||||
Currently, you are working on model ensemble implementation. Your task is to write a Python function that combines multiple model predictions and makes final decisions.
|
||||
|
||||
Your specific task as follows:
|
||||
{{ task_desc }}
|
||||
|
||||
## Competition Information for This Task
|
||||
{{ competition_info }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
## Relevant Information for This Task
|
||||
{% endif %}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------- Successful Implementations for Similar Models ---------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict["ensemble.py"] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------- Previous Failed Attempts ---------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict["ensemble.py"] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Guidelines
|
||||
1. The function's code is associated with several other functions including a data loader, feature engineering, and model training. all codes are as follows:
|
||||
{{ all_code }}
|
||||
2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
|
||||
{% include "scenarios.data_science.share:guidelines.coding" %}
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
--------- Code Specification ---------
|
||||
{{ code_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
--------- Former code ---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
--------- Feedback to former code ---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes.
|
||||
{% endif %}
|
||||
|
||||
|
||||
ensemble_eval:
|
||||
system: |-
|
||||
You are a data scientist responsible for evaluating ensemble implementation code generation.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Ensemble Code
|
||||
```python
|
||||
{{ code }}
|
||||
```
|
||||
|
||||
## Testing Process
|
||||
The ensemble code is tested using the following script:
|
||||
```python
|
||||
{{ test_code }}
|
||||
```
|
||||
You will analyze the execution results based on the test output provided.
|
||||
|
||||
{% if workflow_stdout is not none %}
|
||||
### Whole Workflow Consideration
|
||||
The ensemble code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.
|
||||
|
||||
**Workflow Code:**
|
||||
```python
|
||||
{{ workflow_code }}
|
||||
```
|
||||
|
||||
You should evaluate both the ensemble test results and the overall workflow results. **Approve the code only if both tests pass.**
|
||||
{% endif %}
|
||||
|
||||
The metric used for scoring the predictions:
|
||||
**{{ metric_name }}**
|
||||
|
||||
## Evaluation Criteria
|
||||
- You will be given the standard output (`stdout`) from the ensemble test and, if applicable, the workflow test.
|
||||
- Code should have no try-except blocks because they can hide errors.
|
||||
- Check whether the code implement the scoring process using the given metric.
|
||||
- The stdout includes the local variable values from the ensemble code execution. Check whether the validation score is calculated correctly.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe how well the ensemble executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
|
||||
"return_checking": "Detail the checks performed on the ensemble results, including shape and value validation.",
|
||||
"code": "Assess code quality, readability, and adherence to specifications.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
user: |-
|
||||
--------- Ensemble test stdout ---------
|
||||
{{ stdout }}
|
||||
{% if workflow_stdout is not none %}
|
||||
--------- Whole workflow test stdout ---------
|
||||
{{ workflow_stdout }}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,131 @@
|
||||
feature_coder:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Competition Information for This Task
|
||||
{{ competition_info }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
## Relevant Information for This Task
|
||||
{% endif %}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------- Successful Implementations for Similar Models ---------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict["feature.py"] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------- Previous Failed Attempts ---------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict["feature.py"] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Guidelines
|
||||
1. If feature engineering is unnecessary or should be combined with model training, you may skip this step.
|
||||
2. Be cautious of any column drop in the code. Dropping a column easily without any more attempts, it may not be a good practice.
|
||||
3. The function input is the output of the following data loader:
|
||||
```python
|
||||
{{ data_loader_code }}
|
||||
```
|
||||
4. **Additional Guidance:**
|
||||
- If a previous attempt exists, improve upon it without repeating mistakes.
|
||||
- If errors indicate a missing file, find a way to download it or implement an alternative solution.
|
||||
- You should avoid using logging module to output information in your generated code, and instead use the print() function.
|
||||
5. You should use the following cache decorator to cache the results of the function:
|
||||
```python
|
||||
from joblib import Memory
|
||||
memory = Memory(location='{% include "scenarios.data_science.share:scen.cache_path" %}', verbose=0)
|
||||
@memory.cache```
|
||||
6. Coding tricks:
|
||||
- If the input consists of a batch of file paths and you need to modify the file contents to complete your feature engineering task, you can accomplish your feature engineering task by modifying these files and creating new files in a subfolder within "{% include "scenarios.data_science.share:scen.cache_path" %}" (this path is persistent, otherwise you may lose your created file). Then the new file paths are returned.
|
||||
|
||||
{% include "scenarios.data_science.share:guidelines.coding" %}
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
--------- Code Specification ---------
|
||||
{{ code_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
--------- Former code ---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
--------- Feedback to former code ---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes.
|
||||
{% endif %}
|
||||
|
||||
|
||||
feature_eval:
|
||||
system: |-
|
||||
You are a data scientist responsible for evaluating feature engineering code generation.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Feature Engineering Code
|
||||
```python
|
||||
{{ code }}
|
||||
```
|
||||
|
||||
## Testing Process
|
||||
The feature engineering code is tested using the following script:
|
||||
```python
|
||||
{{ test_code }}
|
||||
```
|
||||
You will analyze the execution results based on the test output provided.
|
||||
|
||||
{% if workflow_stdout is not none %}
|
||||
### Whole Workflow Consideration
|
||||
The feature engineering code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.
|
||||
|
||||
**Workflow Code:**
|
||||
```python
|
||||
{{ workflow_code }}
|
||||
```
|
||||
|
||||
You should evaluate both the feature engineering test results and the overall workflow results. **Approve the code only if both tests pass.**
|
||||
{% endif %}
|
||||
|
||||
## Evaluation Criteria
|
||||
You will be given the standard output (`stdout`) from the feature engineering test and, if applicable, the workflow test.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe how well the feature engineering executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
|
||||
"return_checking": "Evaluate the correctness and integrity of processed data, checking for missing values, incorrect transformations, and data consistency.",
|
||||
"code": "Assess code quality, readability, and adherence to specifications. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for optimization.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
--------- Feature engineering test stdout ---------
|
||||
{{ stdout }}
|
||||
{% if workflow_stdout is not none %}
|
||||
--------- Whole workflow test stdout ---------
|
||||
{{ workflow_stdout }}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,186 @@
|
||||
model_coder:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Competition Information for This Task
|
||||
{{ competition_info }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
## Relevant Information for This Task
|
||||
{% endif %}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------- Successful Implementations for Similar Models ---------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict[similar_successful_knowledge.target_task.name ~ '.py'] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------- Previous Failed Attempts ---------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict[former_failed_knowledge.target_task.name ~ '.py'] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Guidelines
|
||||
1. The function's input is from the output of a feature engineering function whose input is the output of a data loading function. The data loader function and feature engineering function code is as follows:
|
||||
--------- Data Loader Code ---------
|
||||
{{ data_loader_code }}
|
||||
--------- Feature Engineering Code ---------
|
||||
{{ feature_code }}
|
||||
2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
|
||||
3. If the model can both be implemented by PyTorch and Tensorflow, please use pytorch for broader compatibility.
|
||||
4. You should use the following cache decorator to cache the results of the function:
|
||||
```python
|
||||
from joblib import Memory
|
||||
memory = Memory(location='{% include "scenarios.data_science.share:scen.cache_path" %}', verbose=0)
|
||||
@memory.cache``
|
||||
{% include "scenarios.data_science.share:guidelines.coding" %}
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
The file name should be the model name described in the model task in the format "{task_name}.py". You should always follow this name format.
|
||||
{% else %}
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
user_general: |-
|
||||
--------- Code Specification ---------
|
||||
{{ code_spec }}
|
||||
|
||||
--------- Former model code ---------
|
||||
{% if latest_model_code|length == 0 %}
|
||||
So far the workspace is empty. No model code has been implemented yet.
|
||||
{% else %}
|
||||
{{ latest_model_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
--------- Feedback to former code ---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
model_eval:
|
||||
system: |-
|
||||
You are a data scientist responsible for evaluating model building code generation.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Model Building Code
|
||||
```python
|
||||
{{ code }}
|
||||
```
|
||||
|
||||
## Testing Process
|
||||
The model building code is tested using the following script:
|
||||
```python
|
||||
{{ test_code }}
|
||||
```
|
||||
|
||||
### Execution Phases
|
||||
The model is tested in two phases:
|
||||
|
||||
1. Initial Training Phase:
|
||||
- The model receives **train and valid inputs** with **empty hyperparameters**.
|
||||
- The focus is on verifying whether the model successfully trains and produces **valid outputs and hyperparameter outputs**.
|
||||
|
||||
2. Retraining Phase:
|
||||
- The model receives **train and test inputs** (without valid inputs).
|
||||
- The hyperparameters generated from the first phase are passed back for **retraining**.
|
||||
|
||||
|
||||
### Key Requirements for Approval
|
||||
A model can only be approved if it meets all of the following conditions:
|
||||
1. Hyperparameter Handling
|
||||
- If hyperparameters are returned, they must include an early stop round.
|
||||
- The hyperparameters must be correctly utilized in the model for retraining.
|
||||
- If the early stop round is provided, it must be used in the model implementation.
|
||||
2. The model output shape must strictly match the specifications in `spec.md`.
|
||||
|
||||
{% if workflow_stdout is not none %}
|
||||
### Whole Workflow Consideration
|
||||
The model building code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.
|
||||
|
||||
**Workflow Code:**
|
||||
```python
|
||||
{{ workflow_code }}
|
||||
```
|
||||
|
||||
You should evaluate both the model building test results and the overall workflow results. **Approve the code only if both tests pass.**
|
||||
{% endif %}
|
||||
|
||||
## Evaluation Criteria
|
||||
You will be given the standard output (`stdout`) from the model building test and, if applicable, the workflow test.
|
||||
[Note] If no stdout for model buidling test is provided, the model failed due to a timeout or out-of-memory error. You should analyze potential optimizations.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe how well the model building executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
|
||||
"return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in spec.md. You also need to check whether the hyperparameters used for retraining are correctly returned during the test execution of the model.",
|
||||
"code": "Assess code quality, readability, and adherence to specifications. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for optimization.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
--------- Model building test stdout ---------
|
||||
{{ stdout }}
|
||||
{% if workflow_stdout is not none %}
|
||||
--------- Whole workflow test stdout ---------
|
||||
{{ workflow_stdout }}
|
||||
{% endif %}
|
||||
|
||||
model_eval_rm:
|
||||
system: |-
|
||||
You are a data scientist responsible for evaluating model removal process.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
{% if workflow_stdout is not none %}
|
||||
## Whole Workflow Consideration
|
||||
The model building code is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.
|
||||
|
||||
**Workflow Code:**
|
||||
```python
|
||||
{{ workflow_code }}
|
||||
```
|
||||
|
||||
You should evaluate both the model removal test results and the overall workflow results. **Approve the code only if both tests pass.**
|
||||
{% endif %}
|
||||
|
||||
## Evaluation Criteria
|
||||
You will be given the standard output (`stdout`) from the model removal test and, if applicable, the workflow test.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe how well the model removal executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
|
||||
"return_checking": "Check the generated value, including whether the value is generated and comparing the shape of the model output with the requirement in spec.md.",
|
||||
"code": "Assess code quality, readability, and adherence to specifications.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
--------- Model removal test stdout ---------
|
||||
{{ stdout }}
|
||||
{% if workflow_stdout is not none %}
|
||||
--------- Whole workflow test stdout ---------
|
||||
{{ workflow_stdout }}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,347 @@
|
||||
pipeline_coder:
|
||||
system: |-
|
||||
You are a grandmaster-level data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
Your task is to generate robust, debuggable, and iteration-friendly code for data science pipelines, following a strict, stepwise process.
|
||||
|
||||
**Important Context**: You are working on sample datasets and your code will go through automated iterations. Design your code to be iteration-friendly with comprehensive print statements and clear debugging information to facilitate the automatic improvement process.
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## The runtime environment your code will running on
|
||||
{{ runtime_environment }}
|
||||
|
||||
{% if package_info is not none %}
|
||||
To help you write the runnable code, the user has provided the package information which contains the package names and versions.
|
||||
You should be careful about the package versions, as the code will be executed in the environment with the specified version and the api might be different from the latest version.
|
||||
The user might provide the packages the environment doesn't have, you should avoid using any of them.
|
||||
## Package Information
|
||||
{{ package_info }}
|
||||
{% endif %}
|
||||
|
||||
## Hyperparameters Specification
|
||||
Follow the hyperparameter choices if they are specified in the task description, unless they are unreasonable or incorrect.
|
||||
In this case, refer to the guidelines below for appropriate adjustments:
|
||||
{% include "scenarios.data_science.share:spec.hyperparameter" %}
|
||||
|
||||
# Specification your code should follow
|
||||
{{ spec }}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
## Previous Failed Attempts
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
# Workflow Overview
|
||||
You must complete the following stages in order.
|
||||
|
||||
## Data Loading
|
||||
- Load the dataset strictly from `{% include "scenarios.data_science.share:scen.input_path" %}` as described in the **Data Folder Description**. DO NOT attempt to load data from the current directory (`./`).
|
||||
- When loading data files, you may use try-except blocks to handle scenarios where files might be missing or in different formats. However, if no data is successfully loaded, this indicates an incorrect file path or reading method that should be fixed rather than bypassed.
|
||||
- **Important Note on Error Handling**: Beyond data loading, avoid using try-except blocks to hide or suppress errors in data processing, analysis, or model training. All errors should be properly diagnosed and fixed at their source to ensure code robustness and reliability.
|
||||
|
||||
## Exploratory Data Analysis (EDA) (Required)
|
||||
Please follow this systematic methodology (in the required schema) for your analysis.
|
||||
1. Initial Data Assessment & Sanitization:
|
||||
- Data shape
|
||||
- First 5 rows
|
||||
- Data types per column
|
||||
- Missing values per column
|
||||
- Unique values per column
|
||||
- Target variable distribution
|
||||
- Any other relevant insights
|
||||
|
||||
2. Detailed Feature Analysis (A Non-Exhaustive Guide):
|
||||
For Numerical & Categorical Features:
|
||||
- Central Tendency & Dispersion
|
||||
- Distribution Shape & Imbalance
|
||||
- Outliers & Anomalies
|
||||
- Cardinality & Granularity
|
||||
For Text Features:
|
||||
- Text Granularity & Scale
|
||||
- Core Content & Topicality
|
||||
- Linguistic Structure & Style
|
||||
- Vocabulary Richness & Redundancy
|
||||
|
||||
3. The EDA part should be drafted in plain text sending to standard output with command print or other similar functions with no more than ten thousand characters in the following schema:
|
||||
=== Start of EDA part ===
|
||||
{EDA content}
|
||||
=== End of EDA part ===
|
||||
User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1]
|
||||
- An evaluation agent will help to check whether the EDA part is added correctly.
|
||||
- During the EDA part, you should try to avoid any irrelevant information sending to the standard output.
|
||||
{% include "scenarios.data_science.share:guidelines.coding" %}
|
||||
|
||||
{% if enable_model_dump %}
|
||||
## Model Dumping
|
||||
{% include "components.coder.data_science.share.prompts:dump_model_coder.guideline" %}
|
||||
{% endif %}
|
||||
|
||||
{% if enable_debug_mode %}
|
||||
## Debug Mode
|
||||
Your code will be executed in a debug mode with following command:
|
||||
```bash
|
||||
python main.py --debug
|
||||
```
|
||||
Please simulate the following code to check whether the code is running in debug mode:
|
||||
```python
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--debug', action='store_true', help='Run in debug mode')
|
||||
args = parser.parse_args()
|
||||
DEBUG = False
|
||||
if args.debug:
|
||||
DEBUG = True
|
||||
```
|
||||
In debug mode, you should only sample ten percent of the training data and run the minimum epochs to quickly test the correctness of the code.
|
||||
In debug mode, you should implement a timer to measure the time taken for your debug configuration and estimate the time required for the full run. Your timer should only measure the time taken for the training part, not the data loading or feature engineering part.
|
||||
For example:
|
||||
```python
|
||||
# Read data, feature engineering, etc.
|
||||
start_time = time.time()
|
||||
# Train your model
|
||||
end_time = time.time()
|
||||
debug_time = end_time - start_time
|
||||
# post processing, saving model, etc.
|
||||
```
|
||||
In debug mode, your code should run faster, so the environment will set a shorter time limit than the standard time limit for your code.
|
||||
For example, you can sample ten percent of the training data and run for one epoch, then the full run with ten epochs will take one hundred times the time taken for the debug run. The scale is calculated by yourself depending on the data sampling and epoch number you choose. If your full run enables early stopping, the scale should be smaller considering the early stopping will stop the training earlier than the full epochs.
|
||||
Be careful about the train-valid split strategy. Stratified related split is highly risk since the data has some categories with only one sample. If you use Stratified related split, you should consider using a try-except block to catch the error and use a different split strategy if the error occurs. Example code:
|
||||
```python
|
||||
try:
|
||||
fold_indices = StratifiedKFold(...).split(train_X, train_y) or StratifiedShuffleSplit or StratifiedSubsetSampler etc.
|
||||
except Exception as e:
|
||||
fold_indices = KFold(...).split(train_X, train_y) or other split strategy
|
||||
```
|
||||
You should sample the data after train valid split. When you split the data after sampling, you might get a class with only one sample which might cause the split strategy to fail.
|
||||
Your debug code should run exactly the same as the full run, except for the data sampling and epoch number, to ensure the correctness of the code.
|
||||
You should print total time and estimated time in standard output using print function in the following schema:
|
||||
=== Start of Debug Information ===
|
||||
debug_time: time_taken_for_debug_run_in_seconds (e.g., 'debug_time: 10.0')
|
||||
estimated_time: estimated_time_for_full_run_in_seconds (e.g., 'estimated_time: 100.0')
|
||||
=== End of Debug Information ===
|
||||
User will use the following code to match: re.search(r"(.*?)=== Start of Debug Information ===(.*)=== End of Debug Information ===", stdout, re.DOTALL).groups()[1]
|
||||
Notice, data sampling should only be applied in debug mode. Always use the full data in the full run!
|
||||
Example code:
|
||||
```python
|
||||
if args.debug:
|
||||
sample_size = int(0.1 * len(train_dataset)) # 10% for debug
|
||||
else:
|
||||
sample_size = len(train_dataset)
|
||||
```
|
||||
In debug mode, to increase efficiency, you only need to perform inference on the first sample of the test set to generate a valid prediction for `submission.csv`. For all other samples in the test set, you should use a placeholder value (e.g., 0 or a default value) to fill the prediction column. This ensures that the generated `submission.csv` has the same number of rows as the full run and passes the format check.
|
||||
Example code:
|
||||
```python
|
||||
all_preds = []
|
||||
for i, batch in enumerate(test_loader):
|
||||
# In debug mode, use placeholders for all batches after the first one to improve efficiency.
|
||||
if args.debug and i > 0:
|
||||
# The shape and data type of the placeholder must match the model's actual output.
|
||||
# Here, we assume `predictions` is a NumPy array.
|
||||
placeholder = np.zeros_like(predictions)
|
||||
all_preds.append(placeholder)
|
||||
continue
|
||||
|
||||
# In full mode, or for the first batch in debug mode, perform actual model inference.
|
||||
predictions = model.predict(batch)
|
||||
all_preds.append(predictions)
|
||||
|
||||
# final_predictions = np.concatenate(all_preds)
|
||||
# ... then create and save submission.csv
|
||||
```
|
||||
You should be very careful about the label classes number in the debug mode. The label classes should be the same as the full run even when you are in the debug mode. The label classes number is often used to build the model.
|
||||
{% endif %}
|
||||
|
||||
## General Guidelines
|
||||
1. Code correctness is the top priority. Ensure your code is runnable and produces the expected output even if some task requirements are not fully met because the task itself might contain some errors like the wrong package name or wrong package function names.
|
||||
2. Use the print() function for all output; do not use the logging module.
|
||||
3. **Avoid all hard-coded values (e.g., fixed dataset sizes)**. Always use proportions for data splitting and similar operations, never absolute numbers.
|
||||
4. Add informative print statements at key steps to facilitate debugging and automated iteration.
|
||||
5. For model training, use reasonable epoch numbers. ALWAYS implement early stopping with proper conditions: sufficient epochs completed, loss reaching sufficiently low value, and no improvement for patience period. Save best model checkpoints based on validation performance.
|
||||
6. Except in debug mode, ALWAYS use all available data; do not sample or subset the data due to resource limitations. If resources are insufficient, print the issue honestly rather than compromising data integrity.
|
||||
7. Do not use tqdm or similar progress bar tools.
|
||||
8. **Try-except blocks are ONLY allowed when reading files. If no files are successfully read, it indicates incorrect file paths or reading methods, not a try-except issue. Try-except is PROHIBITED elsewhere in the code. Assert statements are PROHIBITED throughout the entire code.**
|
||||
9. ATTENTION: ALWAYS use the best saved model (not necessarily final epoch) for predictions. **NEVER create dummy/placeholder submissions (e.g., all 1s, random values)**. If training fails, report failure honestly rather than generating fake submission files.
|
||||
10. You should ALWAYS generate the complete code rather than partial code.
|
||||
11. If the task contains any user instructions, you must strictly follow them. User instructions have the highest priority and should be followed even if they conflict with other specifications or guidelines.
|
||||
12. Strictly follow all specifications and general guidelines described above.
|
||||
|
||||
### Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Competition Information
|
||||
{{ competition_info }}
|
||||
|
||||
# Data Folder Description (All path are relative to the data folder, i.e. "{% include "scenarios.data_science.share:scen.input_path" %}")
|
||||
{{ folder_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
# Former code
|
||||
```
|
||||
{{ latest_code }}
|
||||
```
|
||||
{% if latest_code_feedback is not none %}
|
||||
## Feedback to former code
|
||||
{{ latest_code_feedback }}
|
||||
|
||||
## Improvement Planning
|
||||
Before modifying the code, carefully analyze the feedback and identify no more than three key areas requiring changes. Plan your modifications strategically:
|
||||
1. Prioritize the most critical issues that directly affect code execution, correctness, or stability.
|
||||
2. Focus on improvements with the highest impact on functionality and reliability.
|
||||
3. Preserve existing working components. Do not modify parts of the code that are already correct, in order to avoid introducing new errors.
|
||||
|
||||
The previous version of the code contained errors. You must correct these issues based on the provided information and ensure you do not repeat the same mistakes.
|
||||
|
||||
{% else %}
|
||||
## Improvement Planning
|
||||
Before enhancing the code, thoroughly analyze what aspects can be improved and identify no more than three key areas for enhancement. Plan your improvements strategically:
|
||||
1. Focus on improvements related to performance, robustness, or feature engineering.
|
||||
2. Enhance code clarity and debugging capabilities to facilitate maintenance and troubleshooting.
|
||||
3. Optimize model configuration or validation strategy to improve overall effectiveness.
|
||||
|
||||
The previous version of the code is correct. You should improve the code based on the provided task while ensuring that unrelated parts remain unchanged.
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
pipeline_eval:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
You will be provided with:
|
||||
1. A detailed competition scenario description.
|
||||
2. A task description outlining the step-by-step process for the code, along with a specification of the code structure.
|
||||
3. A code implementation and its execution output.
|
||||
Your task is to rigorously evaluate the code implementation against the provided scenario and task description, ensuring it meets all requirements, adheres to the specified structure, and executes successfully.
|
||||
|
||||
## Evaluation Aspects
|
||||
|
||||
### Execution Success
|
||||
- Goal: Ensure the code executes successfully without any errors.
|
||||
- Notes:
|
||||
- Model performance is not evaluated in this step; focus solely on successful execution.
|
||||
- Warnings are acceptable if they do not interfere with successful code execution.
|
||||
- If the code execute successfully:
|
||||
- Proceed to Step 2.
|
||||
- If the code does not execute successfully:
|
||||
- Set the "final_decision" to false.
|
||||
{% if enable_mcp_documentation_search %}
|
||||
- Given that my package/environment is fixed and unchangeable, first you should go through the code and the execution output,if the problem could be solved by looking up the official documentation to confirm feature/API availability, compatible usage, or official alternatives in the fixed environment, set the "requires_documentation_search" to true.
|
||||
{% endif %}
|
||||
- Write complete analysis in the "execution" field.
|
||||
|
||||
### Competition Alignment
|
||||
- Goal: Confirm strict adherence to the competition's evaluation rules and experimental setup.
|
||||
- Guidelines:
|
||||
- Analyze whether the experimental setup and code may cause misalignment between validation and test performance.
|
||||
- Confirm strict adherence to the competition's evaluation rules listed in `scenario`:
|
||||
- The metric implementation must exactly match scenario requirements (metric value itself is not the focus).
|
||||
- Prediction methodologies must be consistent between validation and test datasets.
|
||||
- No shortcuts or fold-specific strategies should be applied inconsistently.
|
||||
- Check for corner-case consistency.
|
||||
- Avoid hard-coded values; use proportions for data splitting and similar operations.
|
||||
- If no issues are found:
|
||||
- Begin the "code" with `[Code analysis]`, providing a detailed analysis of the code quality, readability, and adherence to specifications.
|
||||
- If discrepancies or risks are found:
|
||||
- Set the "final_decision" to false.
|
||||
- Begin the "code" with `[Evaluation error]`, explicitly document any evaluation alignment issues causing experiment failure.
|
||||
|
||||
{% if debug_mode %}
|
||||
### Debug Mode Compliance
|
||||
- Goal: Ensure the code follows debug mode requirements.
|
||||
- Guidelines:
|
||||
- Sufficient debugging information (print statements, clear error messages) should be included to facilitate automatic improvement processes.
|
||||
- The code should be executed in debug mode with the command `python main.py --debug`.
|
||||
- In debug mode, the code should sample ten percent of the data and run the minimum epochs to quickly test the correctness of the code.
|
||||
- Check whether the code follows these requirements. If not, emphasize it in your feedback and reject this implementation.
|
||||
- Execution time and estimated time for the full run should be checked. Estimated time should not be too large to finish in the given time limit.
|
||||
- Consider the early stopping mechanism in the code. The estimated time could be very large but early stopping could stop the training earlier than the full epochs.
|
||||
- Debug time should be reasonable and the estimated time should be reasonable based on the debug time.
|
||||
- Data sampling should only be applied in debug mode. Always use the full data in the full run.
|
||||
- The label classes number should be the same as the full run even in debug mode.
|
||||
- If the code passes this step: Proceed to Next Aspects.
|
||||
- If the code does not pass this step: Clearly document the debug mode compliance issues and reject the implementation.{% endif %}
|
||||
|
||||
|
||||
### Submission File Format Check
|
||||
{% if mle_check %}
|
||||
- The user has done a format check for your submission. Since you didn't sample any test data, your debug mode output should be the same format as the full run.
|
||||
- The user will put the check result in the "Submission check" section of the execution output.
|
||||
- If the submission check returns a 'Submission is valid' or similar message, despite some warning messages, you should give the conclusion that the code executed successfully. If no other code related issues are found, set the "final_decision" to true.
|
||||
- If the submission check returns an error message, you should set the "final_decision" to false and clearly document the issues in the "return_checking" field.
|
||||
{% elif is_sub_enabled %}
|
||||
- Goal: Verify that the code correctly generates the final submission in the expected format and that the submission is authentic.
|
||||
- Guidelines:
|
||||
- The submission file must strictly match the required structure (correct columns, index format, data types). The index names and column names must be identical to the format specified in the Competition Information's '====== Submission Format ======' section.
|
||||
- Rigorously verify that the submission file was produced by genuine model inference and successful code execution, not by cheating, fallback or exception-handling mechanisms.
|
||||
- The submission must be generated from genuine model predictions using the best saved model—never empty, constant, random, or hard-coded values.
|
||||
- Submissions must reflect authentic model outputs; any form of fabrication, cheating, or simulated results is strictly prohibited and grounds for rejection.
|
||||
- Cross-check both code logic and stdout to ensure predictions originate from real model inference, not from error recovery or placeholder code paths.
|
||||
- Only check the format of the submission since only part of the data is provided; the submission might have a different index than expected due to data sampling.
|
||||
- Verify honest failure reporting if training issues occur.
|
||||
- If the code passes this step, Finalize evaluation.
|
||||
- If the code does not pass this step:
|
||||
- Set the "final_decision" to false and clearly document the issues in the "return_checking" field.
|
||||
{% else %}
|
||||
Submission File Format Check is not conducted since no target submission format is provided. You should consider this submission file is valid.
|
||||
{% endif %}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
### Step 6: Similar Successful Implementations to help Code Improvement
|
||||
The user has done several similar tasks and get some successful implementations. These code might not be implemented to the same task, but they are similar to your task and they might work well on your dataset.
|
||||
Please refer to these successful implementation and provide your suggestions in your response on how to correct your current code based on these successful implementations.
|
||||
## Successful Implementations for Similar Tasks
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Similar Task {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
Please respond with your feedback in the following JSON format without anything else.
|
||||
```json
|
||||
{
|
||||
{% if enable_mcp_documentation_search %}
|
||||
"requires_documentation_search": <true/false>,
|
||||
{% endif %}"execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information. If errors occurred, analyze the root causes: (1) Are they fundamental algorithmic/approach issues, or (2) Implementation details that can be easily fixed, or (3) Environment/dependency problems?",
|
||||
"return_checking": "Examine the generated files by cross-referencing the code logic and stdout output. Verify: (1) Format matches required submission format (index, column names, CSV content); (2) **File generation authenticity**: Is the file genuinely produced by successful model execution, or is it a result of exception handling/fallback mechanisms? Cite specific code sections and stdout evidence.",
|
||||
"code": "Begin explicitly with [Code analysis] or [Evaluation error]. Provide structured analysis: (1) **Technical Appropriateness**: Does the chosen approach (algorithms, data processing, validation strategy) match this problem's data characteristics and competition requirements? (2) **Effective Components**: What specific parts work well and why are they effective for this problem type? (3) **Issues & Improvements**: Identify concrete problems and suggest actionable improvement directions (without providing actual code). (4) **Code Quality**: Assess readability, structure, and adherence to specifications.",
|
||||
{% if enable_mcp_documentation_search %}
|
||||
"error_message": "If the code execution has problems, extract the error information in the following format, otherwise set to empty string: ### TRACEBACK: <full relevant traceback extracted from execution output> ### SUPPLEMENTARY_INFO: <only if TRACEBACK is unclear - copy exact code fragments: import statements, variable=value assignments, function calls with parameters as they appear in code>",
|
||||
{% endif %}"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
user: |-
|
||||
# Competition Information
|
||||
{{ scenario }}
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Task Specification for Code Structure
|
||||
{{ spec }}
|
||||
|
||||
# Code
|
||||
```
|
||||
{{ code }}
|
||||
```
|
||||
|
||||
## Execution Output
|
||||
```
|
||||
{{ stdout }}
|
||||
```
|
||||
@@ -0,0 +1,402 @@
|
||||
|
||||
spec:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
Currently, you are working on a Kaggle competition project.
|
||||
This project involves analyzing data and building models to beat other competitors, with the code being generated by large language models.
|
||||
|
||||
The runtime environment you are working in includes the following libraries and their respective versions:
|
||||
{{ runtime_environment }}
|
||||
|
||||
Your overall task is provided below:
|
||||
{{ task_desc }}
|
||||
|
||||
Your task is to write five specification texts (in markdown format) for the following tasks, based on the competition information provided
|
||||
- Data loading (and preprocessing)
|
||||
- Feature Engineering
|
||||
- Model Building
|
||||
- Ensemble
|
||||
- The overall workflow
|
||||
|
||||
The specifications for each step should be tailored to the competition information provided.
|
||||
|
||||
Your specification should consists two parts:
|
||||
1. The function definition in code format, including type annotations and a clear, complete docstring that describes the function's purpose, input parameters, return value, and any relevant exceptions.
|
||||
2. Additional information or notes that the coder should consider while implementing the function.
|
||||
|
||||
Your specifications should include only the function definition and docstring, without any code implementation or inline comments.
|
||||
|
||||
## Competition Information for This Task
|
||||
{{ competition_info }}
|
||||
|
||||
----------- Folder Description (All path are relative to the data folder) ---------
|
||||
- Ensure that all columns in sample_submission can be generated.
|
||||
{{ folder_spec }}
|
||||
|
||||
user:
|
||||
data_loader: |-
|
||||
Data loader specification text should follow these detailed requirements:
|
||||
1. Function Interface:
|
||||
- Function Name: `load_data`
|
||||
- Input: No input arguments.
|
||||
- Output:
|
||||
- `X` (DT, define based on competition information): Feature matrix for training data.
|
||||
- `y` (DT): Target vector for training data.
|
||||
- `X_test` (DT): Feature matrix for test data.
|
||||
- `test_ids` (DT): Identifiers for the test data.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Specify the data source location (`{% include "scenarios.data_science.share:scen.input_path" %}`).
|
||||
- Clearly define the structure and type of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
2. Notes:
|
||||
- Update `DT` (data type) based on the specific competition dataset. This can include `pd.DataFrame`, `np.array`, `torch.Tensor`, etc.
|
||||
- Only set the DT of variables without inferring the shape of these variables since you don't know the shape of the data.
|
||||
|
||||
Responsibilities and notes of an implemented data loader that aligns with the generated specification.
|
||||
{% include "scenarios.data_science.share:component_spec.DataLoadSpec" %}
|
||||
|
||||
{% if latest_spec %}
|
||||
6. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.
|
||||
|
||||
feature: |-
|
||||
Feature engineering specification text should adhere to the following requirements:
|
||||
1. Function Interface:
|
||||
- Function Name: `feat_eng`
|
||||
- Parameters:
|
||||
- `X` (DT): Train data to be transformed.
|
||||
- `y` (DT): Train label data.
|
||||
- `X_test` (DT): Test data.
|
||||
- Output:
|
||||
- `X_transformed` (DT): Transformed train data.
|
||||
- `y_transformed` (DT): Transformed train label data.
|
||||
- `X_test_transformed` (DT): Transformed test data.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Clarify the input parameters and their data types.
|
||||
- Define the structure and format of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
|
||||
2. Precautions for Feature Engineering:
|
||||
- Well handle the shape of the data:
|
||||
- The sample size of the train data and the test data should be the same in all scenarios.
|
||||
- To some tabular or time-series data, you may add or remove some columns so your inferred column number may be unsure.
|
||||
- For scenarios where each dimension does not have a special meaning (like image, audio, and so on), the input shape and the output shape should be exactly the same in most cases unless there is a compelling reason to change them.
|
||||
- Integration with the Model Pipeline:
|
||||
- If feature engineering is deferred to the model pipeline for better overall performance, state explicitly that it will be handled at the model stage.
|
||||
- Model-related operations should not be implemented in this step. (e.g., it uses tools combined with models like torch.Dataset with rich data transformation/augmentation)
|
||||
- Otherwise, ensure this function applies all required transformations while avoiding data leakage.
|
||||
- General Considerations:
|
||||
- Ensure scalability for large datasets.
|
||||
- Handle missing values and outliers appropriately (e.g., impute, remove, or replace).
|
||||
- Ensure consistency between feature data types and transformations.
|
||||
- Prevent data leakage: Do not use information derived from the test set when transforming training data.
|
||||
- Domain-Specific Features:
|
||||
- Apply logic for competition-specific features (e.g., text vectorization, image augmentations, categorical encoding).
|
||||
|
||||
3. Code Standards:
|
||||
- Avoid using progress bars (e.g., `tqdm`) in the implementation.
|
||||
|
||||
4. Notes:
|
||||
- Align `DT` (data type) definitions with those in the Data Loader specification.
|
||||
- GPU and multiprocessing are available and are encouraged to use for accelerating transformations.
|
||||
- Only set the DT of variables without inferring the shape of these variables since you don't know the shape of the data.
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.
|
||||
|
||||
model: |-
|
||||
Model building specification text should adhere to the following requirements:
|
||||
|
||||
1. Function Interface:
|
||||
- Function Name: `model_workflow`
|
||||
- Parameters:
|
||||
- `X` (DT): Training feature data.
|
||||
- `y` (DT): Training label data.
|
||||
- `val_X` (Optional[DT]): Validation feature data.
|
||||
- `val_y` (Optional[DT]): Validation label data.
|
||||
- `test_X` (Optional[DT]): Test feature data.
|
||||
- `hyper_params` (dict): Dictionary of hyperparameters for model configuration.
|
||||
- Output:
|
||||
- `pred_val` (Optional[DT]): Predictions on validation data.
|
||||
- `pred_test` (Optional[DT]): Predictions on test data.
|
||||
- `hyper_params` (dict): Updated dictionary of hyperparameters after training.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Clarify the input parameters and their data types.
|
||||
- Define the structure and format of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
|
||||
2. Code Standards:
|
||||
- Do not use progress bars (e.g., `tqdm`) in the implementation.
|
||||
|
||||
3. Precautions:
|
||||
- Ensure input arrays (`X`, `y`, `val_X`, `val_y`, `test_X`) have consistent dimensions and shapes.
|
||||
- Use default values for hyperparameters if `hyper_params` is not provided.
|
||||
- Train the model on `X` and `y`.
|
||||
- Evaluate the model using `val_X` and `val_y` if validation data is available.
|
||||
- If `test_X` is provided, generate predictions for it.
|
||||
|
||||
4. Notes:
|
||||
- Align `DT` (data type) with the definitions used in Feature Engineering specifications.
|
||||
- The device has GPU support, so you are encouraged to use it for training if necessary to accelerate the process.
|
||||
- Some data transformations/augmentations can be included in this step (e.g., data tools provided by TensorFlow and Torch)
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.
|
||||
|
||||
ensemble: |-
|
||||
Ensemble specification text adhere to the following requirements:
|
||||
1. Function Interface:
|
||||
- Function Name: `ensemble_workflow`
|
||||
- Parameters:
|
||||
- `test_preds_dict` (Dict[str, DT]): A dictionary of test predictions from different models. The key is the model file name.
|
||||
- `val_preds_dict` (Dict[str, DT]): A dictionary of validation predictions from different models. The key is the model file name.
|
||||
- `val_label` (DT): Validation label.
|
||||
- Output:
|
||||
- `final_pred` (DT): Ensemble prediction for the test data.
|
||||
- Docstring Requirements:
|
||||
- Describe the purpose of the function.
|
||||
- Clarify the input parameters and their data types.
|
||||
- Define the structure and format of the output.
|
||||
- Inferred data shape to each input and output data variables. To uncertain dimension, use -1.
|
||||
|
||||
2. Precautions:
|
||||
- Input Validation:
|
||||
- Ensure all predictions in `test_preds_dict` and `val_preds_dict` have consistent shapes and dimensions.
|
||||
- Verify that `val_label` is provided and matches the length of `val_preds_dict` predictions.
|
||||
- Handle empty or invalid inputs gracefully with appropriate error messages.
|
||||
- Metric Calculation and Storage:
|
||||
- Calculate the metric (mentioned in the evaluation section of the competition information) for each model and ensemble strategy on valid, and save the results in `scores.csv`, e.g.:
|
||||
```python
|
||||
scores = {}
|
||||
for model_name, val_pred in val_preds_dict.items():
|
||||
scores[model_name] = calculate_metric(val_label, val_pred)
|
||||
|
||||
...
|
||||
some code about ensemble strategy
|
||||
...
|
||||
ensemble_val_pred = ...
|
||||
|
||||
ensemble_score = calculate_metric(val_label, ensemble_val_pred)
|
||||
scores["ensemble"] = ensemble_score # Ensure "ensemble" is explicitly stored
|
||||
|
||||
scores_df = pd.DataFrame(scores.items(), columns=["Model", <metric_name>])
|
||||
scores_df.to_csv("scores.csv", index=False)
|
||||
```
|
||||
- Even if only one model is present, compute the ensemble score and store it under `"ensemble"`.
|
||||
|
||||
3. Code Standards:
|
||||
- Do not use progress bars (e.g., tqdm) in the code.
|
||||
|
||||
4. Notes:
|
||||
- Align `DT` (data type) definitions with those used in model specifications.
|
||||
- Ensure flexibility to handle multiple ensemble strategies based on competition requirements.
|
||||
- Only set the DT of variables without inferring the shape of these variables since you don't know the shape of the data.
|
||||
|
||||
{% if latest_spec %}
|
||||
5. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
You should return the specification in markdown format directly, while the **function definition** within it should be in code format, tailored to the Competition Information, with detailed explanations provided in the docstring.
|
||||
|
||||
workflow: |-
|
||||
{% include "scenarios.data_science.share:component_spec.Workflow" %}
|
||||
|
||||
{% if latest_spec %}
|
||||
7. Former Specification:
|
||||
{{ latest_spec }}
|
||||
You should follow the provided specifications to improve this task.
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
You should return the specification in markdown format directly.
|
||||
You should create the rules based on the competition information instead of copying the requirements.
|
||||
|
||||
data_loader_coder:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
## Relevant Information for This Task
|
||||
{% endif %}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------- Successful Implementation Examples for Similar Task ---------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Example {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------- Previous Failed Attempts ---------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Guidelines
|
||||
1. Ensure that the dataset is loaded strictly from `{% include "scenarios.data_science.share:scen.input_path" %}`, following the exact folder structure described in the **Data Folder Description**, and do not attempt to load data from the current directory (`./`).
|
||||
2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
|
||||
3. You should use the following cache decorator to cache the results of the function:
|
||||
```python
|
||||
from joblib import Memory
|
||||
memory = Memory(location='{% include "scenarios.data_science.share:scen.cache_path" %}', verbose=0)
|
||||
@memory.cache```
|
||||
{% include "scenarios.data_science.share:guidelines.coding" %}
|
||||
|
||||
## Exploratory Data Analysis (EDA) part(Required):
|
||||
- Before returning the data, you should always add an EDA part describing the data to help the following steps understand the data better.
|
||||
- The EDA part should include but not limited in the following information in plain text:
|
||||
- The shape of the data.
|
||||
- The first 5 rows of the data.
|
||||
- The data types of each column.
|
||||
- The number of missing values in each column.
|
||||
- The number of unique values in each column.
|
||||
- The distribution of the target variable.
|
||||
- Any other information that you think is important for the following steps.
|
||||
- The EDA part should be drafted in plain text sending to standard output with command print or other similar functions with no more than ten thousand characters in the following schema:
|
||||
=== Start of EDA part ===
|
||||
{ You EDA output content }
|
||||
=== End of EDA part ===
|
||||
User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1]
|
||||
- An evaluation agent will help to check whether the EDA part is added correctly.
|
||||
- During the EDA part, you should try to avoid any irrelevant information sending to the standard output.
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
--------- Competition Information ---------
|
||||
{{ competition_info }}
|
||||
|
||||
--------- Code Specification ---------
|
||||
{{ code_spec }}
|
||||
|
||||
--------- Data Folder Description (All path are relative to the data folder, i.e. "{% include "scenarios.data_science.share:scen.input_path" %}") ---------
|
||||
{{ folder_spec }}
|
||||
|
||||
{% if latest_code %}
|
||||
--------- Former code ---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
--------- Feedback to former code ---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes.
|
||||
{% endif %}
|
||||
|
||||
You should strictly follow the code specifications provided by the specification to implement the function.
|
||||
|
||||
|
||||
data_loader_eval:
|
||||
system: |-
|
||||
You are a data scientist responsible for evaluating data loader code for a Kaggle-style machine learning competition project.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Data Loader Code
|
||||
The data loader code is located in `load_data.py`:
|
||||
```python
|
||||
{{ code }}
|
||||
```
|
||||
|
||||
## Testing Process
|
||||
The data loader is tested using the following script:
|
||||
```python
|
||||
{{ test_code }}
|
||||
```
|
||||
|
||||
{% if workflow_stdout is not none %}
|
||||
### Whole Workflow Consideration
|
||||
The data loader is part of the whole workflow. The user has executed the entire pipeline and provided additional stdout.
|
||||
|
||||
**Workflow Code:**
|
||||
{{ workflow_code }}
|
||||
|
||||
You should evaluate both the data loader test results and the overall workflow execution. **Approve the code only if both tests pass.**
|
||||
{% endif %}
|
||||
|
||||
## Evaluation Criteria
|
||||
You will be given the standard output (`stdout`) from the data loader test and, if applicable, the workflow test.
|
||||
|
||||
## Exploratory Data Analysis (EDA) Part evaluation
|
||||
- The code has also generated some EDA output to help understand the data better.
|
||||
- The EDA part should be drafted in plain text sending to standard output with command print or other similar functions with no more than ten thousand characters in the following schema:
|
||||
=== Start of EDA part ===
|
||||
{ You EDA output content }
|
||||
=== End of EDA part ===
|
||||
User will use the following code to match: re.search(r"(.*?)=== Start of EDA part ===(.*)=== End of EDA part ===", stdout, re.DOTALL).groups()[1]
|
||||
- The EDA part should include but not limited in the following information in plain text:
|
||||
- The shape of the data.
|
||||
- The first 5 rows of the data.
|
||||
- The data types of each column.
|
||||
- The number of missing values in each column.
|
||||
- The number of unique values in each column.
|
||||
- The distribution of the target variable.
|
||||
- Any other information that you think is important for the following steps.
|
||||
You will be given the EDA output, your job is to check whether the output contains the required and sufficient information. If no EDA output is provided, you should consider it as a failure. Put this evaluation result in the return_checking part.
|
||||
|
||||
Your response must follow this structured JSON format:
|
||||
```json
|
||||
{
|
||||
"execution": "Describe how well the data loader executed, including any errors or issues encountered. Append all error messages and full traceback details without summarizing or omitting any information.",
|
||||
"return_checking": "Evaluate the correctness and integrity of the loaded data. Check for issues like missing values, incorrect data types, outliers, or formatting inconsistencies.",
|
||||
"code": "Assess code quality, readability, and adherence to best practices. Consider efficiency, including whether the code utilizes multi-threading or GPU acceleration for faster data loading.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
--------- Data loader test stdout ---------
|
||||
{{ stdout }}
|
||||
--------- Data loader EDA stdout ---------
|
||||
{% if eda_output is not none %}
|
||||
{{ eda_output }}
|
||||
{% else %}
|
||||
No EDA output is provided.
|
||||
{% endif %}
|
||||
{% if workflow_stdout is not none %}
|
||||
--------- Whole workflow test stdout ---------
|
||||
{{ workflow_stdout }}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,123 @@
|
||||
dump_model_coder:
|
||||
guideline: |-
|
||||
Your code will be executed in a inference mode with following command:
|
||||
```bash
|
||||
python main.py --inference
|
||||
```
|
||||
Please dump the model in a "models/" subfolder in the first running, and the script rerun performs inference without needing to retrain the model when running the code again.
|
||||
In inference Mode, the script MUST NOT load any training data.
|
||||
If there are parameters generated from the training data that might be needed for inference on test data, please save them in the "models/" subfolder as well.
|
||||
If no test set is provided, reserve a portion of the data as your test set and save the generated test files in the models/ subfolder for use in submission and inference.
|
||||
Make sure that the required files, like submission.csv and scores.csv, are created without model training step through loading the saved model and test data file directly.
|
||||
|
||||
|
||||
dump_model_eval:
|
||||
system: |-
|
||||
You are a data scientist tasked with evaluating code generation. You've developed a Kaggle competition code that can produce a submission file.
|
||||
The code should follow the guideline below:
|
||||
{% include "components.coder.data_science.share.prompts:dump_model_coder.guideline" %}
|
||||
|
||||
You will receive the following information:
|
||||
- The implemented code
|
||||
- The stdout from running the code
|
||||
- The file list in "models/" subfolder
|
||||
- The scores.csv file generated during both training and inference (if it exists)
|
||||
|
||||
Focus on these aspects:
|
||||
- Check if the code saves the model in the "models/" subfolder.
|
||||
- Check if the code saves the test data in the "models/" subfolder when there is no test data specified.
|
||||
- Ensure that when the code is rerun in inference mode, it skips the training process and loads the model from the "models/" subfolder for direct inference.
|
||||
- Verify that there is no training activity in the output.
|
||||
- Verify that the script does not load the original training data.
|
||||
- Ensure that even if you skip the model training by loading saved models, the files like scores.csv and submission.csv are still correctly created.
|
||||
- The model's performance should remain consistent and not vary unreasonably between training and inference.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information. Carefully check the stdout to ensure that when the code is rerun, it skips the training process and loads the model from the 'models/' subfolder for direct inference. Append the information that makes you think that the model is still being retrained when rerunning the code."
|
||||
"return_checking": "Verify the generated files include necessary files. Make sure scores.csv file does not change unreasonably between training and inference",
|
||||
"code": "The code has explicity dump the model into 'models/' subfolder; When the modes files are already in 'models/' subfolder, the code will explicity skip the training process.",
|
||||
"final_decision": <true or false in boolean type; only return true when ensuring that the code saves the model in a 'models/' subfolder, and the script rerun performs inference without needing to retrain the model.>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
------------ The implemented code ------------
|
||||
{{code}}
|
||||
|
||||
------------ The stdout from running the code ------------
|
||||
{{stdout}}
|
||||
|
||||
------------ File opened by the code ------------
|
||||
{{opened_trace_lines}}
|
||||
|
||||
------------ The file list in "models/" subfolder ------------
|
||||
{% for f in model_folder_files %}
|
||||
- {{ f }}
|
||||
{% endfor %}
|
||||
|
||||
------------ The scores.csv file generated ------------
|
||||
# Training:
|
||||
{{scores_content_before}}
|
||||
|
||||
# Inference:
|
||||
{{scores_content_after}}
|
||||
|
||||
|
||||
docdev:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %} Your task is to create documentation for a data science solution.
|
||||
|
||||
You will be given:
|
||||
- a list of files in the folder.
|
||||
- content from some important files.
|
||||
|
||||
Please explain the trained models in the "models/" folder. The training and inference processes are detailed in the `main.py` file. The models' evaluation results are in `scores.csv`. Please respond with a markdown file that includes the following information:
|
||||
- Explain the purpose of each model. If some models are part of a group (like those from cross-validation), describe them together.
|
||||
- Provide key details for each model group:
|
||||
- Important training parameters
|
||||
- Model details
|
||||
- Performance of each model
|
||||
|
||||
Be brief. Mention the file path when you introduce files.
|
||||
Don't introduce anything other than models.
|
||||
|
||||
{% include "utils.agent.tpl:MarkdownOut" %}
|
||||
|
||||
user: |-
|
||||
--------------- The file list in the workspace ---------------
|
||||
{% for f in file_li %}
|
||||
- {{ f }}
|
||||
{% endfor %}
|
||||
|
||||
--------------- File content of each file ---------------
|
||||
{% for fname, content in key_files.items() %}
|
||||
File Path: {{fname}}
|
||||
```
|
||||
{{content}}
|
||||
```
|
||||
{% endfor %}
|
||||
|
||||
notebookconverter:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %} Your task is to provide a summary for a data science solution.
|
||||
|
||||
You will be given:
|
||||
- The original implementation plan for the script.
|
||||
- A Python script that contains code and output.
|
||||
|
||||
Your task is to generate markdown content that includes a title and a short paragraph summarizing the technique in model training, the type of model produced and any other noteworthy details in the solution.
|
||||
|
||||
The return content should be like the format below(Please note that "````" is used to avoid confliction of "```" in markdown file)
|
||||
````markdown
|
||||
# <The title of the notebook>
|
||||
<the content of markdown file>
|
||||
````
|
||||
|
||||
user: |-
|
||||
--------------- The implementation plan ---------------
|
||||
{{plan}}
|
||||
|
||||
--------------- The Python script content ---------------
|
||||
{{code}}
|
||||
@@ -0,0 +1,137 @@
|
||||
workflow_coder:
|
||||
system: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
Here is the competition information for this task:
|
||||
{{ competition_info }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 or queried_former_failed_knowledge|length != 0 %}
|
||||
## Relevant Information for This Task
|
||||
{% endif %}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------- Successful Implementations for Similar Models ---------
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Model {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.file_dict["main.py"] }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------- Previous Failed Attempts ---------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.file_dict["main.py"] }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Guidelines
|
||||
1. Understand the User's Code Structure
|
||||
- The user has written different Python functions that can load and preprocess data, execute feature engineering, train models, and ensemble them.
|
||||
- Each functionality is in a separate Python file.
|
||||
2. Your task is only to integrate the existing processes of load_data, feature, model, and ensemble into a complete workflow. Do not edit or modify the existing Python files. The final step should output the predictions in the required format.
|
||||
3. The user may provide specific code organization rules and instructions. Ensure that the integration follows the given framework and structure.
|
||||
4. After predicting the output, print the shape and other information of the output to stdout to help the evaluator assess the code.
|
||||
5. You should avoid using logging module to output information in your generated code, and instead use the print() function.
|
||||
{% include "scenarios.data_science.share:guidelines.coding" %}
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
--------- Code Specification ---------
|
||||
{{ code_spec }}
|
||||
|
||||
--------- load data code ---------
|
||||
file: load_data.py
|
||||
{{ load_data_code }}
|
||||
|
||||
--------- feature engineering code ---------
|
||||
file: feature.py
|
||||
{{ feature_code }}
|
||||
|
||||
--------- model training code ---------
|
||||
Attention: The input and output of the model function is flexible. Training dataset is necessary, but validation and test dateset might be optional. The hyperparameters can either be passed as arguments or be set as default values in the function. You need to use the function correctly.
|
||||
All model files share the same function name. Please import the model files with their name like: from {file_name} import {function_name}
|
||||
{{ model_codes }}
|
||||
|
||||
--------- ensemble code ---------
|
||||
Note, we will check the index of the score.csv, so please use the model name as the index to feed into ensemble function.
|
||||
file: ensemble.py
|
||||
{{ ensemble_code }}
|
||||
|
||||
{% if latest_code %}
|
||||
--------- Former code ---------
|
||||
{{ latest_code }}
|
||||
{% if latest_code_feedback is not none %}
|
||||
--------- Feedback to former code ---------
|
||||
{{ latest_code_feedback }}
|
||||
{% endif %}
|
||||
The former code contains errors. You should correct the code based on the provided information, ensuring you do not repeat the same mistakes.
|
||||
{% endif %}
|
||||
|
||||
workflow_eval:
|
||||
system: |-
|
||||
You are a data scientist responsible for evaluating workflow code generation.
|
||||
|
||||
## Task Description
|
||||
The user is trying to build a workflow in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
The main code generation task is as follows:
|
||||
{{ task_desc }}
|
||||
|
||||
The user provides workflow information and its components.
|
||||
The details on how to structure the workflow are given in the specification file:
|
||||
```markdown
|
||||
{{ spec }}
|
||||
```
|
||||
|
||||
This workflow integrates multiple stages, including:
|
||||
- Data loading
|
||||
- Feature engineering
|
||||
- Model training
|
||||
- Ensembling
|
||||
|
||||
## Evaluation Scope
|
||||
Your focus is to check whether the workflow code:
|
||||
1. Executes successfully, correctly organizing components and generating a final submission.
|
||||
2. Generates predictions in the correct format, ensuring they align with the **sample submission** structure!
|
||||
|
||||
[Note]
|
||||
1. The individual components (data loading, feature engineering, model tuning, etc.) have already been evaluated by the user. You should only evaluate and improve the workflow code, unless there are critical issues in the components.
|
||||
2. Model performance is NOT a concern in this evaluation—only correct execution and formatting matter.
|
||||
3. As long as the execution does not exceed the time limit, ensure that the code uses cross-validation to split the training data and train the model. If cross-validation is not used, mention it in the execution section and set `final_decision` to `false`.
|
||||
|
||||
## Evaluation Criteria
|
||||
You will be given the workflow execution output (`stdout`) to determine correctness.
|
||||
|
||||
Please respond with your feedback in the following JSON format and order
|
||||
```json
|
||||
{
|
||||
"execution": "Describe whether the main workflow executed successfully, correctly integrating all components and generating the final submission. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information.",
|
||||
"return_checking": "Verify the generated files, particularly the submission file. Ensure that its format matches the sample submission, checking the index, column names, and CSV content.",
|
||||
"code": "Provide feedback on code quality, readability, and adherence to the given specifications.",
|
||||
"final_decision": <true/false>
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
--------- Workflow test stdout ---------
|
||||
{{ stdout }}
|
||||
--------- Workflow code generated by user ---------
|
||||
{{ code }}
|
||||
@@ -0,0 +1,209 @@
|
||||
|
||||
evaluator_code_feedback_v1_system: |-
|
||||
User is trying to implement some factors in the following scenario:
|
||||
{{ scenario }}
|
||||
User will provide you the information of the factor.
|
||||
|
||||
Your job is to check whether user's code is align with the factor and the scenario.
|
||||
The user will provide the source python code and the execution error message if execution failed.
|
||||
The user might provide you the ground truth code for you to provide the critic. You should not leak the ground truth code to the user in any form but you can use it to provide the critic.
|
||||
|
||||
User has also compared the factor values calculated by the user's code and the ground truth code. The user will provide you some analyze result comparing two output. You may find some error in the code which caused the difference between the two output.
|
||||
|
||||
If the ground truth code is provided, your critic should only consider checking whether the user's code is align with the ground truth code since the ground truth is definitely correct.
|
||||
If the ground truth code is not provided, your critic should consider checking whether the user's code is reasonable and correct.
|
||||
|
||||
Notice that your critics are not for user to debug the code. They are sent to the coding agent to correct the code. So don't give any following items for the user to check like "Please check the code line XXX".
|
||||
|
||||
You suggestion should not include any code, just some clear and short suggestions. Please point out very critical issues in your response, ignore non-important issues to avoid confusion. If no big issue found in the code, you can response "No critics found".
|
||||
|
||||
You should provide the suggestion to each of your critic to help the user improve the code. Please response the critic in the following format. Here is an example structure for the output:
|
||||
critic 1: The critic message to critic 1
|
||||
critic 2: The critic message to critic 2
|
||||
|
||||
evaluator_code_feedback_v1_user: |-
|
||||
--------------Factor information:---------------
|
||||
{{ factor_information }}
|
||||
--------------Python code:---------------
|
||||
{{ code }}
|
||||
--------------Execution feedback:---------------
|
||||
{{ execution_feedback }}
|
||||
{% if value_feedback is not none %}
|
||||
--------------Factor value feedback:---------------
|
||||
{{ value_feedback }}
|
||||
{% endif %}
|
||||
{% if gt_code is not none %}
|
||||
--------------Ground truth Python code:---------------
|
||||
{{ gt_code }}
|
||||
{% endif %}
|
||||
|
||||
evolving_strategy_factor_implementation_v1_system: |-
|
||||
User is trying to implement some factors in the following scenario:
|
||||
{{ scenario }}
|
||||
Your code is expected to align the scenario in any form which means The user needs to get the exact factor values with your code as expected.
|
||||
|
||||
To help you write the correct code, the user might provide multiple information that helps you write the correct code:
|
||||
1. The user might provide you the correct code to similar factors. Your should learn from these code to write the correct code.
|
||||
2. The user might provide you the failed former code and the corresponding feedback to the code. The feedback contains to the execution, the code and the factor value. You should analyze the feedback and try to correct the latest code.
|
||||
3. The user might provide you the suggestion to the latest fail code and some similar fail to correct pairs. Each pair contains the fail code with similar error and the corresponding corrected version code. You should learn from these suggestion to write the correct code.
|
||||
|
||||
Your must write your code based on your former latest attempt below which consists of your former code and code feedback, you should read the former attempt carefully and must not modify the right part of your former code.
|
||||
|
||||
Notice that you should not add any other text before or after the json format.
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Your former latest attempt:---------------
|
||||
=====Code to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].implementation.all_codes }}
|
||||
=====Feedback to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].feedback }}
|
||||
{% endif %}
|
||||
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
|
||||
evolving_strategy_factor_implementation_v2_user: |-
|
||||
--------------Target factor information:---------------
|
||||
{{ factor_information_str }}
|
||||
|
||||
{% if queried_similar_error_knowledge|length != 0 %}
|
||||
{% if error_summary_critics is none %}
|
||||
Recall your last failure, your implementation met some errors.
|
||||
When doing other tasks, you met some similar errors but you finally solve them. Here are some examples:
|
||||
{% for error_content, similar_error_knowledge in queried_similar_error_knowledge %}
|
||||
--------------Factor information to similar error ({{error_content}}):---------------
|
||||
{{ similar_error_knowledge[0].target_task.get_task_information() }}
|
||||
=====Code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[0].implementation.all_codes }}
|
||||
=====Success code to former code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[1].implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
Recall your last failure, your implementation met some errors.
|
||||
After reviewing some similar errors and their solutions, here are some suggestions for you to correct your code:
|
||||
{{error_summary_critics}}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
Here are some success implements of similar component tasks, take them as references:
|
||||
--------------Correct code to similar factors:---------------
|
||||
{% for similar_successful_knowledge in queried_similar_successful_knowledge %}
|
||||
=====Factor {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if latest_attempt_to_latest_successful_execution is not none %}
|
||||
You have tried to correct your former failed code but still met some errors. Here is the latest attempt to the latest successful execution, try not to get the same error to your new code:
|
||||
=====Your latest attempt=====
|
||||
{{ latest_attempt_to_latest_successful_execution.implementation.all_codes }}
|
||||
=====Feedback to your latest attempt=====
|
||||
{{ latest_attempt_to_latest_successful_execution.feedback }}
|
||||
{% endif %}
|
||||
|
||||
evolving_strategy_error_summary_v2_system: |-
|
||||
User is trying to implement some factors in the following scenario:
|
||||
{{ scenario }}
|
||||
User is doing the following task:
|
||||
{{factor_information_str}}
|
||||
|
||||
You have written some code but it meets errors like the following:
|
||||
{{code_and_feedback}}
|
||||
|
||||
The user has found some tasks that met similar errors, and their final correct solutions.
|
||||
Please refer to these similar errors and their solutions, provide some clear, short and accurate critics that might help you solve the issues in your code.
|
||||
|
||||
You suggestion should not include any code, just some clear and short suggestions. Please point out very critical issues in your response, ignore non-important issues to avoid confusion. If no big issue found in the code, you can response "No critics found".
|
||||
|
||||
[NOTE]
|
||||
1. When processing data, avoid time leakage.
|
||||
|
||||
Please response the critic in the following format. Here is an example structure for the output:
|
||||
critic 1: The critic message to critic 1
|
||||
critic 2: The critic message to critic 2
|
||||
|
||||
evolving_strategy_error_summary_v2_user: |-
|
||||
{% if queried_similar_error_knowledge|length != 0 %}
|
||||
{% for error_content, similar_error_knowledge in queried_similar_error_knowledge %}
|
||||
--------------Factor information to similar error ({{error_content}}):---------------
|
||||
{{ similar_error_knowledge[0].target_task.get_task_information() }}
|
||||
=====Code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[0].implementation.all_codes }}
|
||||
=====Success code to former code with similar error ({{error_content}}):=====
|
||||
{{ similar_error_knowledge[1].implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
|
||||
select_implementable_factor_system: |-
|
||||
User is trying to implement some factors in the following scenario:
|
||||
{{ scenario }}
|
||||
Your job is to help the user select the easiest-to-implement factors. Some factors may be difficult to implement due to a lack of information or excessive complexity. The user will provide the number of factors you should pick and information about the factors, including their descriptions, formulas, and variable explanations.
|
||||
User will provide you the former attempt to implement the factor and the feedback to the implementation. You need to carefully review your previous attempts. Some factors have been repeatedly tried without success. You should consider discarding these factors.
|
||||
Please analyze the difficulties of the each factors and provide the reason and response the indices of selected implementable factor in the json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"Analysis": "Analyze the difficulties of the each factors and provide the reason why the factor can be implemented or not."
|
||||
"selected_factor": "The indices of selected factor index in the list, like [0, 2, 3].The length should be the number of factor left after filtering.",
|
||||
}
|
||||
|
||||
select_implementable_factor_user: |-
|
||||
Number of factor you should pick: {{ factor_num }}
|
||||
{% for factor_info in sub_tasks %}
|
||||
=============Factor index:{{factor_info[0]}}:=============
|
||||
=====Factor name:=====
|
||||
{{ factor_info[1].factor_name }}
|
||||
=====Factor description:=====
|
||||
{{ factor_info[1].factor_description }}
|
||||
=====Factor formulation:=====
|
||||
{{ factor_info[1].factor_formulation }}
|
||||
{% if factor_info[2]|length != 0 %}
|
||||
--------------Your former attempt:---------------
|
||||
{% for former_attempt in factor_info[2] %}
|
||||
=====Code to attempt {{ loop.index }}=====
|
||||
{{ former_attempt.implementation.all_codes }}
|
||||
=====Feedback to attempt {{ loop.index }}=====
|
||||
{{ former_attempt.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
|
||||
evaluator_output_format_system: |-
|
||||
User is trying to implement some factors in the following scenario:
|
||||
{{ scenario }}
|
||||
User will provide you the format of the output. Please help to check whether the output is align with the format.
|
||||
Please respond in the JSON format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"output_format_decision": True,
|
||||
"output_format_feedback": "The output format is correct."
|
||||
}
|
||||
|
||||
|
||||
evaluator_final_decision_v1_system: |-
|
||||
User is trying to implement some factors in the following scenario:
|
||||
{{ scenario }}
|
||||
User has finished evaluation and got some feedback from the evaluator.
|
||||
The evaluator run the code and get the factor value dataframe and provide several feedback regarding user's code and code output. You should analyze the feedback and considering the scenario and factor description to give a final decision about the evaluation result. The final decision concludes whether the factor is implemented correctly and if not, detail feedback containing reason and suggestion if the final decision is False.
|
||||
|
||||
The implementation final decision is considered in the following logic:
|
||||
1. If the value and the ground truth value are exactly the same under a small tolerance, the implementation is considered correct.
|
||||
2. If the value and the ground truth value have a high correlation on ic or rank ic, the implementation is considered correct.
|
||||
3. If no ground truth value is provided, the implementation is considered correct if the code executes successfully (assuming the data provided is correct). Any exceptions, including those actively raised, are considered faults of the code. Additionally, the code feedback must align with the scenario and factor description. The implementation cannot be considered correct if the code execution failed, no matter what the reason is.
|
||||
|
||||
Please response the critic in the json format. Here is an example structure for the JSON output, please strictly follow the format:
|
||||
{
|
||||
"final_decision": True,
|
||||
"final_feedback": "The final feedback message",
|
||||
}
|
||||
|
||||
evaluator_final_decision_v1_user: |-
|
||||
--------------Factor information:---------------
|
||||
{{ factor_information }}
|
||||
--------------Execution feedback:---------------
|
||||
{{ execution_feedback }}
|
||||
--------------Code feedback:---------------
|
||||
{{ code_feedback }}
|
||||
--------------Factor value feedback:---------------
|
||||
{{ value_feedback }}
|
||||
@@ -0,0 +1,742 @@
|
||||
data_coder:
|
||||
system: |-
|
||||
You are a world-class data engineer specializing in preparing training data for large language model fine-tuning.
|
||||
Your expertise includes processing various data formats and converting them to the Alpaca format required by LlamaFactory.
|
||||
|
||||
# Part 1: Context
|
||||
|
||||
## 1.1 Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
## 1.2 Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## 1.3 Available Datasets
|
||||
The following datasets are available for processing:
|
||||
{{ dataset_info }}
|
||||
|
||||
## 1.4 Priority Rules (CRITICAL)
|
||||
**Task Description requirements are MANDATORY.** You MUST implement all data processing requirements specified in the Task Description exactly as described.
|
||||
|
||||
# Part 2: Output Specification
|
||||
|
||||
## 2.1 Alpaca Format Definition
|
||||
Your script must output a JSON file named `data.json` in the current working directory (`{{ workspace_path }}`).
|
||||
The output must be in Alpaca format: a JSON array where each element has:
|
||||
- `instruction`: The instruction or prompt for the model (required, non-empty)
|
||||
- `input`: Optional additional context (can be empty string)
|
||||
- `output`: The expected response from the model (required, non-empty)
|
||||
|
||||
## 2.2 Output Example
|
||||
```json
|
||||
[
|
||||
{
|
||||
"instruction": "Translate the following English text to French.",
|
||||
"input": "Hello, how are you?",
|
||||
"output": "Bonjour, comment allez-vous?"
|
||||
},
|
||||
{
|
||||
"instruction": "Summarize the following article.",
|
||||
"input": "Article content here...",
|
||||
"output": "Summary of the article..."
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## 2.3 Data Quality Awareness (IMPORTANT)
|
||||
- Raw datasets may contain low-quality, noisy, or incorrect samples
|
||||
- It is better to DISCARD questionable samples than to include them in training data
|
||||
- When encountering samples that are ambiguous, malformed, or have inconsistent answers, prefer filtering them out
|
||||
- A smaller but high-quality dataset is more valuable than a larger noisy one
|
||||
- High filtering rate is acceptable and expected - it means the script is doing quality control properly
|
||||
|
||||
## 2.4 Data Validation Rules
|
||||
Before writing the final data.json, implement these validations:
|
||||
|
||||
### 2.4.1 Answer Consistency Check (CRITICAL)
|
||||
- Verify generated answer matches expected answer
|
||||
- Prefer string normalization over LLM when feasible
|
||||
- Answer format varies by task (e.g., `\boxed{}` for math, JSON for structured, code output for programming)
|
||||
- Filter samples with mismatched answers
|
||||
|
||||
### 2.4.2 Over-length Filtering (MANDATORY)
|
||||
- Filter out samples where `total_tokens > max_position_embeddings`
|
||||
- Do NOT truncate - filter instead
|
||||
- See Part 6 for COT-specific validation requirements
|
||||
|
||||
# Part 3: Script Implementation Requirements
|
||||
|
||||
## 3.1 Basic Conventions
|
||||
1. Read data from `{{ datasets_path }}` directory (mounted read-only)
|
||||
2. Use standard Python libraries (json, csv, os, pathlib) when possible
|
||||
3. Handle file encoding properly (use utf-8)
|
||||
4. Include error handling for file operations
|
||||
5. Print progress information to stdout for debugging
|
||||
6. **IMPORTANT**: Your script MUST support the `--debug` command-line argument (see 3.2). Other than `--debug`, do NOT expect any other command-line arguments.
|
||||
|
||||
## 3.2 Debug Mode (CRITICAL)
|
||||
Your script MUST support `--debug` for fast validation:
|
||||
- Sampling/filtering is pure code operation (no LLM), so it runs completely in both modes
|
||||
- `--debug`: Process ~100 samples through LLM pipeline, print actual sampled total
|
||||
- No flag: Process ALL sampled data through LLM pipeline
|
||||
|
||||
### Debug Mode Example
|
||||
```python
|
||||
import random
|
||||
|
||||
# Step 1: Run complete sampling/filtering (fast, no LLM) - runs in BOTH modes
|
||||
sampled_data = apply_sampling_strategy(raw_data) # e.g., 50000 → 2000
|
||||
|
||||
# Step 2: Limit LLM processing in debug mode only
|
||||
if args.debug:
|
||||
samples_to_process = random.sample(sampled_data, min(100, len(sampled_data)))
|
||||
else:
|
||||
samples_to_process = sampled_data
|
||||
|
||||
# Step 3: Show the actual number of sampled items (Do not estimate; count the exact number of samples that will be processed when not in debug mode.)
|
||||
print(f"Sampled data size from raw: {len(sampled_data)} / {len(raw_data)}") # Actual training data size
|
||||
```
|
||||
|
||||
## 3.3 Logging Convention
|
||||
Only print progress at 20%, 40%, 60%, 80%, 100%. No per-item logs.
|
||||
|
||||
## 3.4 Output Statistics Format
|
||||
Your script should print statistics at the end of execution:
|
||||
|
||||
### Script Execution Summary (REQUIRED)
|
||||
```
|
||||
# Debug mode (--debug):
|
||||
========== SUMMARY ==========
|
||||
Total output samples: {actual_output}
|
||||
Sampled data size from raw: {sampled_count} / {raw_count}
|
||||
Debug samples processed: {debug_processed_count}
|
||||
Estimated full output: ~{int(actual_output / debug_processed_count * sampled_count)}
|
||||
Output file: {{ workspace_path }}data.json
|
||||
=============================
|
||||
|
||||
# Full mode (no --debug):
|
||||
========== SUMMARY ==========
|
||||
Total output samples: {actual_output}
|
||||
Sampled data size from raw: {sampled_count} / {raw_count}
|
||||
Output file: {{ workspace_path }}data.json
|
||||
=============================
|
||||
```
|
||||
|
||||
### CoT Quality Statistics (REQUIRED for COT tasks)
|
||||
```
|
||||
========== COT QUALITY STATS ==========
|
||||
COT format check: {with_think_tags}/{total} have <think> tags
|
||||
Over-length filtered: {count} ({percentage}%)
|
||||
Answer consistency check: {passed}/{total} passed
|
||||
Length distribution: p25={}, p50={}, p75={}, p99={}
|
||||
=======================================
|
||||
```
|
||||
|
||||
# Part 4: Scope Clarification (IMPORTANT)
|
||||
**Your script should ONLY handle data processing and output data.json.**
|
||||
- DO NOT generate training configuration files (e.g., train.yaml, training_config.json)
|
||||
- DO NOT include training scripts or fine-tuning code
|
||||
- DO NOT save any files other than data.json
|
||||
- Training configuration will be handled separately by another component
|
||||
|
||||
# Part 5: LLM API Usage Guide
|
||||
|
||||
## 5.1 Model Pool - Load Balancing
|
||||
**All models have INDEPENDENT quotas** - distribute load evenly across models!
|
||||
|
||||
```python
|
||||
import os, json
|
||||
import litellm; litellm.suppress_debug_info = True
|
||||
from litellm import completion
|
||||
|
||||
STRONG_MODELS = json.loads(os.getenv("STRONG_MODEL_POOL", "[]")) # CoT generation
|
||||
WEAK_MODELS = json.loads(os.getenv("WEAK_MODEL_POOL", "[]")) # simple/fast tasks
|
||||
|
||||
# Default timeout for API calls (in seconds)
|
||||
API_TIMEOUT = 120
|
||||
|
||||
def call_llm(messages, models, start_idx=0, timeout=API_TIMEOUT):
|
||||
"""Load-balanced LLM call with timeout. Use start_idx to distribute across models."""
|
||||
if not models:
|
||||
raise RuntimeError("Model pool is empty. Set STRONG_MODEL_POOL/WEAK_MODEL_POOL env vars.")
|
||||
last_err = None
|
||||
for i in range(len(models)):
|
||||
model = models[(start_idx + i) % len(models)]
|
||||
try:
|
||||
resp = completion(model=model, messages=messages, drop_params=True, timeout=timeout)
|
||||
return resp.choices[0].message.content
|
||||
except Exception as e:
|
||||
last_err = e
|
||||
continue
|
||||
raise RuntimeError(f"All models failed. Last error: {last_err}")
|
||||
```
|
||||
|
||||
## 5.2 Timeout & Efficiency (CRITICAL)
|
||||
- Set `timeout=120` for API calls to prevent blocking on complex problems
|
||||
- If timeout after retries, skip sample and continue
|
||||
- Prefer string/regex over LLM for validation (answer check, structure check) when possible
|
||||
|
||||
## 5.3 Concurrency - CRITICAL
|
||||
**MANDATORY**: Use `ThreadPoolExecutor(max_workers={{ api_max_workers }})` for parallel sample processing.
|
||||
- DO NOT use `os.cpu_count()` - it limits parallelism unnecessarily
|
||||
- The value {{ api_max_workers }} is intentional for maximizing API throughput
|
||||
- Pass `start_idx=sample_index % len(models)` to distribute load evenly
|
||||
|
||||
```python
|
||||
with ThreadPoolExecutor(max_workers={{ api_max_workers }}) as executor: # NOT os.cpu_count()!
|
||||
futures = {executor.submit(process_sample, i, sample, i % len(STRONG_MODELS)): i
|
||||
for i, sample in enumerate(samples)}
|
||||
```
|
||||
|
||||
# Part 6: CoT Processing Guide (CRITICAL)
|
||||
## 6.1 CoT Output Requirement (MANDATORY)
|
||||
**CRITICAL: ALL training data MUST include Chain-of-Thought reasoning in output field.**
|
||||
|
||||
### Why This Matters
|
||||
- Models learn to reason by seeing reasoning examples
|
||||
- Direct answers (A/B/C/D, True/False) provide NO training signal for reasoning
|
||||
|
||||
### Generation Process
|
||||
- Ask LLM to provide step-by-step reasoning before the final answer
|
||||
- Good: "Explain your reasoning step by step, then give the final answer"
|
||||
- Bad: "Output with <think> tags" (models will refuse)
|
||||
- Let LLM generate reasoning naturally
|
||||
|
||||
### Output Format
|
||||
{% if force_think_token %}
|
||||
- Your script MUST wrap LLM output into `<think>...</think>` format
|
||||
- Format: `<think>{reasoning}</think>{answer}`
|
||||
- The **answer** (content AFTER `</think>`) must follow **Benchmark Description**
|
||||
- DO NOT ask for `<think>` tags in prompts (models refuse this)
|
||||
{% else %}
|
||||
- If base model is NOT a thinking model (no native `<think>` token), DO NOT add `<think>` tags
|
||||
- Output must contain step-by-step reasoning (CoT)
|
||||
{% endif %}
|
||||
- **Answer format must follow Benchmark Description**
|
||||
|
||||
## 6.2 Post-Processing Validation
|
||||
{% if force_think_token %}
|
||||
- **Structure check**: `"<think>" in output and "</think>" in output`
|
||||
{% endif %}
|
||||
- **Content check**: Output must contain reasoning (not just direct answer)
|
||||
- **Answer check**: Answer format must match Benchmark Description
|
||||
|
||||
# Part 7: Previous Failed Attempts
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
# Part 8: Response Format
|
||||
Provide ONLY the Python script in a markdown code block:
|
||||
```python
|
||||
# Your complete Python script here
|
||||
```
|
||||
|
||||
Do NOT add explanations before or after the code block.
|
||||
|
||||
user: |-
|
||||
Please generate a Python script that processes the available datasets and outputs a `data.json` file in Alpaca format.
|
||||
|
||||
The script will be executed in two modes:
|
||||
1. **Debug mode (coding phase):** `python {{ workspace_path }}process_data.py --debug` - process 100 samples for fast validation
|
||||
2. **Full mode (running phase):** `python {{ workspace_path }}process_data.py` - generates all samples for training
|
||||
|
||||
Dataset files are located at: {{ datasets_path }}
|
||||
|
||||
## Detailed Dataset Descriptions
|
||||
{% for ds_name, ds_desc in involved_dataset_folder_desc.items() %}
|
||||
### Dataset: {{ ds_name }}
|
||||
(Note: All file paths for this dataset are relative to `{{ datasets_path }}{{ ds_name }}/`)
|
||||
{{ ds_desc }}
|
||||
{% endfor %}
|
||||
|
||||
Output file should be: {{ workspace_path }}data.json
|
||||
|
||||
{% if latest_code %}
|
||||
## Previous Data Processing Script
|
||||
```python
|
||||
{{ latest_code }}
|
||||
```
|
||||
|
||||
{% if latest_feedback is not none %}
|
||||
## Feedback on Previous Script
|
||||
{{ latest_feedback }}
|
||||
|
||||
Please improve the 'Previous Data Processing Script' based on the feedback above. Do not create a new script. Consider the feedback carefully and make necessary corrections. If the feedback asks for more information or logging, make sure to include that in your revised script to help the evaluator to better assess your implementation.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
Please create a new Data Processing Script based on the task description.
|
||||
{% endif %}
|
||||
|
||||
**IMPORTANT**: Make sure your script supports the `--debug` argument as described in the system prompt.
|
||||
|
||||
finetune_coder:
|
||||
system: |-
|
||||
You are a world-class machine learning engineer specializing in large language model fine-tuning using LlamaFactory.
|
||||
Your expertise includes creating optimal LlamaFactory configuration files for various fine-tuning scenarios.
|
||||
|
||||
# Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
## Previous Failed Attempts
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
|
||||
=====Code:=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback:=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Available Fine-tuning Methods
|
||||
{{ available_methods }}
|
||||
|
||||
## Shared Parameters
|
||||
These parameters apply to all fine-tuning methods:
|
||||
{{ shared_params }}
|
||||
|
||||
## Method-Specific Parameters
|
||||
{% for method, params_desc in methods_specific_params.items() %}
|
||||
{{ params_desc }}
|
||||
{% endfor %}
|
||||
|
||||
## Priority Rules (CRITICAL)
|
||||
**Task Description parameters are MANDATORY.** You MUST use exactly the hyperparameter values specified in the Task Description. Guidelines below are defaults only - they apply ONLY when task description does not specify a value.
|
||||
|
||||
## Requirements
|
||||
1. Create a LlamaFactory configuration file named `train.yaml`
|
||||
2. Based on the hypothesis provided by the user, select the most appropriate fine-tuning method
|
||||
3. Generate full training configuration (no sample limit)
|
||||
4. Ensure all parameters are valid for LlamaFactory
|
||||
5. **Adaptive Logging Configuration (CRITICAL)**:
|
||||
- Set `logging_strategy` to 'steps' for consistent monitoring
|
||||
- Calculate `logging_steps` adaptively:
|
||||
* Check `stdout_summary` in data_stats for `Estimated full output` (NOT `total_samples` which is debug mode count)
|
||||
* total_steps = estimated_full × num_epochs / (batch_size × gradient_accumulation_steps × num_gpus)
|
||||
* Target 20-50 log entries total
|
||||
6. **Validation and Checkpoint Strategy (CRITICAL for best model selection)**:
|
||||
- **Validation Split**: Set `val_size` to split a portion of training data for validation. Choose ratio based on dataset size and task needs.
|
||||
- **Save Strategy**: Choose `save_strategy` ('steps' or 'epoch') based on training duration. MUST ensure `eval_strategy` == `save_strategy`.
|
||||
- If using 'steps', set `save_steps` based on estimated full output appropriately, DON'T set it very low or high.
|
||||
- set 'per_device_eval_batch_size' appropriately to speed up eval without OOM.
|
||||
- **Best Model Selection**: Use `load_best_model_at_end: true` with `save_total_limit: 1` to automatically keep and load the best checkpoint based on eval_loss. Note: `save_total_limit` will be force-injected to 1.
|
||||
7. If the former configuration faces error, please make sure to fix the error while aligning with the task. If these two goals conflict, please prioritize fixing the error.
|
||||
|
||||
## Configuration Principle
|
||||
**ONLY include parameters you want to change from defaults**
|
||||
If a parameter's default value matches your intention, OMIT it entirely
|
||||
This prevents unnecessary dependencies and keeps configuration clean
|
||||
Example: if `mixture_of_depths` defaults to `false` and you don't need it, DO NOT include it
|
||||
|
||||
## Output Format
|
||||
You MUST output the YAML configuration in a standard markdown code block:
|
||||
```yaml
|
||||
model_name_or_path: /path/to/model
|
||||
stage: sft
|
||||
...
|
||||
```
|
||||
|
||||
Do NOT add explanations before or after the YAML block.
|
||||
|
||||
user: |-
|
||||
## Path Configuration
|
||||
- dataset_dir: "{{ datasets_path }}"
|
||||
- output_dir: "./output" (auto-injected, you can omit this)
|
||||
- model_name_or_path: "{{ models_path }}{{ base_model }}"
|
||||
- tokenized_path: "{{ workspace_path }}tokenized_cache"
|
||||
|
||||
## Critical Configuration Rules
|
||||
- dataset: MUST be "processed_data" (this is the dataset name in dataset_info.json)
|
||||
- model_name_or_path: use local model path instead of HuggingFace model identifier
|
||||
- dataset_info.json is located at: "{{ datasets_path }}dataset_info.json" (contains the "processed_data" entry)
|
||||
- template: NEVER set to "auto" or "none" - these are invalid values.
|
||||
- For Qwen series model, set to "qwen", and for Qwen3 series model especially, set to "qwen3".
|
||||
- For other models, DO NOT include this field (LlamaFactory auto-detects from tokenizer).
|
||||
- tokenized_path: MUST set to "{{ workspace_path }}tokenized_cache" (datasets directory is read-only mounted)
|
||||
- batch_size: Be aware that `auto_find_batch_size` can cause synchronization issues in multi-GPU (DDP) training. Consider setting `per_device_train_batch_size` explicitly if training hangs
|
||||
- flash_attn: For models supporting flash attention2 (e.g., Qwen series, llama series), set to "fa2" to enhance training speed and reduce memory usage
|
||||
{% if deepspeed_path %}- deepspeed: If number of GPUs > 1, use DeepSpeed with ZeRO Stage 2 or 3 for memory optimization. specifically, set to "{{ deepspeed_path }}ds_z3_config.json" for ZeRO Stage 3, otherwise use "{{ deepspeed_path }}ds_z2_config.json" for ZeRO Stage 2{% endif %}
|
||||
- **IMPORTANT Compatibility Rules**:
|
||||
- `pissa_init: true` is NOT compatible with DeepSpeed ZeRO-3. If using ZeRO-3, do NOT set pissa_init to true
|
||||
- If you need PiSSA initialization, use ZeRO Stage 2 instead of ZeRO Stage 3
|
||||
- `load_best_model_at_end: true` requires `eval_strategy` == `save_strategy` (both "steps" or both "epoch"). Always set both to the same value.
|
||||
|
||||
{% if force_think_token %}
|
||||
{% if has_think_token is defined and not has_think_token %}
|
||||
## Special Token Configuration for CoT Training
|
||||
The base model does NOT have `<think>` token in its vocabulary.
|
||||
To train with Chain-of-Thought reasoning format (output like `<think>reasoning</think>answer`), you MUST add special tokens AND train the new embeddings:
|
||||
```yaml
|
||||
new_special_tokens: ["<think>", "</think>"]
|
||||
resize_vocab: true
|
||||
additional_target: embed_tokens,lm_head # MANDATORY for LoRA/QLoRA when resize_vocab=true! And Full Training does not need this field.
|
||||
```
|
||||
This ensures `<think>` and `</think>` are tokenized as single tokens, not split into subwords.
|
||||
{% elif has_think_token is defined and has_think_token %}
|
||||
## Special Token Note
|
||||
The base model already supports `<think>` token natively. No need to add special tokens for CoT training.
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
{# When force_think_token=false, no special token configuration needed #}
|
||||
|
||||
{% if data_stats %}
|
||||
## Processed Data Statistics (from debug mode)
|
||||
{{ data_stats }}
|
||||
|
||||
**Your Task**: Implement the training configuration specified in the task description.
|
||||
|
||||
- Follow task requirements first (method, batch size, epochs, cutoff_len, etc.)
|
||||
- Apply technical constraints only when task doesn't specify:
|
||||
- `cutoff_len`: ≤ min(max_position_embeddings, memory limit, data p99)
|
||||
- `per_device_train_batch_size`: Choose based on Memory Estimates table
|
||||
- `gradient_accumulation_steps`: Adjust for stable training (effective_batch = batch × accum × gpus)
|
||||
- Validation setup: `val_size`, `eval_strategy` == `save_strategy`, `load_best_model_at_end: true`
|
||||
{% endif %}
|
||||
|
||||
{% if latest_code %}
|
||||
## Previous Configuration
|
||||
```yaml
|
||||
{{ latest_code }}
|
||||
```
|
||||
|
||||
{% if latest_feedback is not none %}
|
||||
## Feedback on Previous Configuration
|
||||
{{ latest_feedback }}
|
||||
|
||||
Please improve the configuration based on the feedback above and the hypothesis.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
Please create a new configuration for the model {{ base_model }} based on the hypothesis above.
|
||||
|
||||
**Remember to include ALL required fields:**
|
||||
- stage: sft
|
||||
- finetuning_type: [select appropriate method based on hypothesis]
|
||||
- do_train: true
|
||||
- model_name_or_path: {{ models_path }}{{ base_model }}
|
||||
- dataset: processed_data
|
||||
- dataset_dir: {{ datasets_path }}
|
||||
- tokenized_path: {{ workspace_path }}tokenized_cache
|
||||
{% endif %}
|
||||
|
||||
user_test_params: |-
|
||||
Now, please provide a set of "test parameters" that will be merged into the above configuration specifically for the DEBUG/MICRO-BATCH test phase.
|
||||
|
||||
The debug phase runs on a very small subset (~10 samples).
|
||||
You need to override parameters that adapt to the dataset for quick debugging the yaml config.
|
||||
|
||||
**Example for Test Parameters:**
|
||||
- Set `num_train_epochs` to 1.
|
||||
- Set `max_samples` to a very small number.
|
||||
|
||||
**Output Format:**
|
||||
Output ONLY the YAML block for these test parameters:
|
||||
```yaml
|
||||
num_train_epochs: 1
|
||||
...
|
||||
```
|
||||
|
||||
finetune_eval:
|
||||
system: |-
|
||||
You are a world-class machine learning engineer specializing in evaluating fine-tuning configurations for large language models using LlamaFactory.
|
||||
Your expertise includes validating LlamaFactory configuration files to ensure they meet all necessary requirements for successful fine-tuning.
|
||||
|
||||
You will be provided with:
|
||||
1. A detailed scenario description which requires a fine-tuning LLM.
|
||||
2. A yaml configuration file named `train.yaml` created for LlamaFactory fine-tuning.
|
||||
3. A structured execution summary (JSON format) containing: status, exit_code, errors, training metrics, and warnings.
|
||||
4. The files generated during the execution.
|
||||
5. Some other yaml configuration for similar tasks which might help you better provide feedback and possible corrections.
|
||||
|
||||
Your task is to:
|
||||
1. Check the execution summary to determine if the run succeeded.
|
||||
2. validate the provided `train.yaml` configuration file to ensure it adheres to the required standards for LlamaFactory fine-tuning using the specified method.
|
||||
3. Provide clear and concise feedback on any issues found in the configuration file or execution logs.
|
||||
4. Suggest specific corrections or improvements if any issues are identified.
|
||||
|
||||
You must give a false final decision only if:
|
||||
- The execution fails with non-zero exit code.
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
### Similar Successful Implementations to help training config Improvement
|
||||
The user has done several similar tasks and get some successful implementations. These yaml configurations might not be implemented to the same task, but they are similar to your task and they might work well on your task.
|
||||
Please refer to these successful implementation and provide your suggestions in your response on how to correct your current code based on these successful implementations.
|
||||
## Successful Implementations for Similar Tasks
|
||||
====={% for similar_successful_knowledge in queried_similar_successful_knowledge %} Similar Task {{ loop.index }}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Yaml configurations:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
# Important Notice
|
||||
- You may find that the execution is short with limited data and iterations. This is expected as we are only validating the configuration file's correctness and not performing full-scale training. Don't treat this as a failure. Also do not put this information in your feedback.
|
||||
|
||||
## Output Format
|
||||
Please respond with your feedback in the following JSON format without anything else.
|
||||
```json
|
||||
{
|
||||
"execution": "State if run succeeded. If errors, include all messages verbatim. Classify cause: algorithm, implementation, or environment."
|
||||
"return_checking": "Plain text. Examine the generated files from the user input. Does the output contains a fine-tuned model or expected artifacts? If not, specify what is missing or incorrect.",
|
||||
"code": "Plain text. Use short simple sentences: say if approach fits task, what works, main issues, brief improvement suggestions."
|
||||
"final_decision": <true/false>, # Final decision on whether the configuration is acceptable for full data fine-tuning
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
# Scenario Information
|
||||
{{ scenario }}
|
||||
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
# Yaml Configuration File
|
||||
```yaml
|
||||
{{ code_yaml }}
|
||||
|
||||
## Execution Summary (Structured)
|
||||
```json
|
||||
{{ stdout }}
|
||||
```
|
||||
|
||||
## Workspace Files
|
||||
{{ workspace_files }}
|
||||
|
||||
data_eval:
|
||||
system: |-
|
||||
You are a data quality expert for LLM fine-tuning using LlamaFactory.
|
||||
Your expertise includes evaluating training data quality and validating data processing scripts.
|
||||
|
||||
You will evaluate:
|
||||
1. **Data format correctness**: Alpaca format requires instruction, input (optional), output fields
|
||||
2. **Data quality**: length distribution, duplicates, semantic reasonableness
|
||||
3. **Alignment with task objectives**: whether the data matches what the task requires
|
||||
4. **Code logic correctness**: whether the processing script is well-designed
|
||||
|
||||
## The Main Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
## Similar Successful Data Processing Examples
|
||||
The following are successful data processing implementations for similar tasks:
|
||||
{% for knowledge in queried_similar_successful_knowledge %}
|
||||
### Example {{ loop.index }}:
|
||||
**Task:** {{ knowledge.target_task.get_task_information() }}
|
||||
**Code:**
|
||||
```python
|
||||
{{ knowledge.implementation.file_dict.get("process_data.py", "N/A") }}
|
||||
```
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
## Debug Mode Context (IMPORTANT)
|
||||
This evaluation runs during the CODING phase in DEBUG MODE.
|
||||
- The script is executed with `--debug` flag to process only ~100 samples for fast validation
|
||||
- Sample count less than 100 is EXPECTED and should NOT be considered a quality issue
|
||||
- Focus on evaluating:
|
||||
1. Data format correctness (Alpaca format)
|
||||
2. Data quality of the generated samples
|
||||
3. Script logic correctness (will it work in full mode?)
|
||||
- Do NOT fail the evaluation just because sample count is low
|
||||
|
||||
## Evaluation Criteria
|
||||
- **Format**: All samples must have non-empty instruction and output fields
|
||||
- **Length**: instruction/output should be reasonable length (not too short or excessively long)
|
||||
- **Duplicates**: High duplicate ratio indicates data quality issues
|
||||
- **Semantic**: instruction should be a question/task, output should be an answer/response
|
||||
- **Alignment**: Data should match the task's training objective
|
||||
|
||||
## CoT Quality Evaluation (Task-Adaptive)
|
||||
**IMPORTANT: CoT quality ≠ CoT length. Adapt criteria based on task type from README metadata.**
|
||||
|
||||
**Check README's `CoT Quality Assessment` section for `task_type` and `quality_ready` fields.**
|
||||
|
||||
1. **Over-length Check** (Report only):
|
||||
- Report percentage of samples exceeding `max_position_embeddings`
|
||||
- High over-length ratio is a warning sign, but NOT an automatic failure if the script handles it correctly
|
||||
|
||||
2. **Answer Consistency Check** (Informational):
|
||||
- Note: The data processing script already filters for answer consistency
|
||||
- If the script implements answer verification, trust its filtering logic
|
||||
- Only flag as issue if the SCRIPT lacks answer verification logic entirely
|
||||
|
||||
3. **Structure Quality Check** (Task-adaptive):
|
||||
- **Math/Code**: Look for step-by-step markers, verification, backtracking
|
||||
- **Chemistry/Structured**: Look for JSON structure or "Step N:" format (short but structured is OK)
|
||||
- **General**: No strict structure requirement
|
||||
|
||||
4. **Length Assessment** (Informational only):
|
||||
- Report length distribution for reference
|
||||
- Length alone should NOT determine pass/fail
|
||||
- Different tasks have different natural length distributions
|
||||
|
||||
5. **Polish Quality Assessment**:
|
||||
- All data must be polished before use
|
||||
- If README shows `baseline_quality: high`: verify enrichment was applied
|
||||
- If README shows `baseline_quality: low`: verify full generation/rewrite was done
|
||||
- Check polish met the requirements in `polish_strategy`
|
||||
|
||||
**Include in return_checking:**
|
||||
- "Task type: {type}, Quality ready: {ready}"
|
||||
- "CoT stats: p50={}, over-length={X}%, structure quality={Y}%"
|
||||
- Assessment based on task-appropriate criteria
|
||||
|
||||
## Hard Check Criteria (AUTOMATIC FAIL if not met)
|
||||
{% if force_think_token %}
|
||||
### 1. COT Format Verification (HARD FAIL)
|
||||
- EVERY sample MUST contain `<think>` and `</think>` tags
|
||||
- Content AFTER `</think>` must be non-empty
|
||||
|
||||
**Rejection:** "FAIL: {X} samples missing <think> tags."
|
||||
{% else %}
|
||||
### 1. COT Format Verification (HARD FAIL)
|
||||
- Output must contain reasoning content (not just a direct answer)
|
||||
- Answer format must match **Benchmark Description**
|
||||
- Do NOT reject for reasoning quality or answer correctness
|
||||
|
||||
**Rejection:** "FAIL: {X}% of samples are direct answers without reasoning."
|
||||
{% endif %}
|
||||
|
||||
### 2. Sample Count Check
|
||||
- Debug mode should generate ~100 samples
|
||||
- Estimated full run samples should be at most {{ upper_data_size_limit }}
|
||||
- Reject if either criteria is not met
|
||||
|
||||
## Final Decision Guidelines
|
||||
**Core Principle: Strict on COT format, lenient on reasoning quality and answer correctness.**
|
||||
|
||||
- **Approve (true)** if:
|
||||
- Script runs successfully (exit_code == 0)
|
||||
- At least 1 sample is generated
|
||||
{% if force_think_token %}- ALL samples have `<think>` and `</think>` tags (MANDATORY){% else %}- ALL samples contain reasoning content (not just direct answers){% endif %}
|
||||
- Data format is correct (Alpaca format with instruction/output)
|
||||
|
||||
- **Reject (false)** if ANY of these:
|
||||
- Script fails to run (exit_code != 0)
|
||||
- Zero samples are generated
|
||||
{% if force_think_token %}- **ANY sample missing `<think>` or `</think>` tags (HARD FAIL)**{% else %}- **ANY sample missing reasoning content (just direct answer)**{% endif %}
|
||||
- Data format is fundamentally broken
|
||||
- **Data does NOT match task description requirements**
|
||||
|
||||
- **Do NOT reject** for:
|
||||
- Low sample count in debug mode (expected)
|
||||
- Moderate quality variations in individual samples
|
||||
- Length distribution not matching ideal patterns
|
||||
- High filtering rate (script doing its job)
|
||||
|
||||
## Important Note
|
||||
- Do not summarize the code into your feedback and DO NOT copy the task description also. Only provide new insights based on your evaluation.
|
||||
- If you think the current logging information is not sufficient to find out the issues, please specify what additional logging information is needed in your feedback and put this information in 'code' block. The user will add further provide you the additional logging information in the next iteration.
|
||||
- Do not write any code in your response, use plain text only.
|
||||
|
||||
## Output Format
|
||||
Respond with JSON only (no markdown code block):
|
||||
{
|
||||
"execution": "Script execution status and data generation result. Include exit code and any errors.",
|
||||
"return_checking": "Data quality analysis: format validation, length distribution assessment, duplicate ratio, semantic issues found; Hard check criteria: does the solution meet the hard check criteria",
|
||||
"code": "Code issues and specific improvement suggestions. What works well, what needs fixing.",
|
||||
"final_decision": true/false
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
{% if script_code %}
|
||||
|
||||
# Data Processing Script (for debugging)
|
||||
```python
|
||||
{{ script_code }}
|
||||
```
|
||||
{% endif %}
|
||||
{% if stdout %}
|
||||
|
||||
# Execution Output ({% if exit_code != 0 %}error logs{% else %}summary{% endif %})
|
||||
```
|
||||
Exit code: {{ exit_code }}
|
||||
{{ stdout }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
# Data Statistics
|
||||
```json
|
||||
{{ data_stats }}
|
||||
```
|
||||
|
||||
# Sample Data ({{ sample_count }} samples from total {{ total_samples }}) [DEBUG MODE]
|
||||
```json
|
||||
{{ data_samples }}
|
||||
```
|
||||
|
||||
runner_eval:
|
||||
system: |-
|
||||
You are a world-class ML engineer evaluating LLM fine-tuning results.
|
||||
|
||||
## Your Task
|
||||
Analyze the training run information and determine if the experiment succeeded.
|
||||
|
||||
## Evaluation Criteria (for final_decision)
|
||||
1. **Execution Success**: Did training complete without errors? Check exit_code and model outputs.
|
||||
2. **Benchmark Execution**: Did benchmark run successfully? Check benchmark results availability.
|
||||
|
||||
## Loss Analysis (for improvement suggestions ONLY - does NOT affect final_decision)
|
||||
- Analyze loss trajectory: Is loss decreasing steadily? Any signs of overfitting?
|
||||
- Use this information ONLY to provide suggestions in the "code" field
|
||||
- Loss patterns should NEVER cause final_decision to be false
|
||||
|
||||
## Error Categories (if failed)
|
||||
- **Timeout (exit_code=124)**: Process was killed due to timeout. Check "failed_stage" and "timeout" fields in stdout:
|
||||
- If failed_stage is "data_processing": Data processing script timed out. This is often due to LLM API calls for CoT data generation taking too long.
|
||||
- If failed_stage is "training": Training timed out.
|
||||
- **OOM**: GPU memory exhaustion - suggest batch size/model changes
|
||||
- **CUDA**: Driver/device issues - suggest environment checks
|
||||
- **Config**: Invalid parameters - suggest specific fixes
|
||||
- **Data**: Dataset issues - suggest data pipeline fixes
|
||||
|
||||
## Output Format
|
||||
Respond with JSON only:
|
||||
{
|
||||
"execution": "Execution status: SUCCESS or FAILED with category [OOM/CUDA/Config/Data]. Include key metrics or error details.",
|
||||
"return_checking": "If success: benchmark analysis. If failed: what failed and expected behavior.",
|
||||
"code": "Configuration assessment and improvement suggestions",
|
||||
"final_decision": true/false // Set to true as long as training succeeded (exit_code=0) and benchmark ran successfully
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
# Training Configuration
|
||||
```yaml
|
||||
{{ config_yaml }}
|
||||
```
|
||||
|
||||
# Execution Info
|
||||
- Exit Code: {{ exit_code }}
|
||||
- Model Output Files: {{ model_files_status }}
|
||||
{% if failed_stage %}- Failed Stage: {{ failed_stage }}
|
||||
- Stage Timeout Config: {{ timeout_seconds }} seconds
|
||||
{% endif %}
|
||||
|
||||
# Benchmark Results
|
||||
```json
|
||||
{{ benchmark_result }}
|
||||
```
|
||||
|
||||
# Loss History (train loss and eval_loss if validation enabled)
|
||||
```json
|
||||
{{ loss_history }}
|
||||
```
|
||||
{% include "components.coder.finetune.prompts:runner_eval.train_output" %}
|
||||
|
||||
train_output: |-
|
||||
# Training Output (key information extracted from stdout)
|
||||
```
|
||||
{{ stdout }}
|
||||
```
|
||||
@@ -0,0 +1,165 @@
|
||||
extract_model_formulation_system: |-
|
||||
offer description of the proposed model in this paper, write a latex formula with variable as well as the architecture of the model. the format should be like
|
||||
{
|
||||
"model_name (The name of the model)": {
|
||||
"description": "A detailed description of the model",
|
||||
"formulation": "A LaTeX formula representing the model's formulation",
|
||||
"architecture": "A detailed description of the model's architecture, e.g., neural network layers or tree structures",
|
||||
"variables": {
|
||||
"\\hat{y}_u": "The predicted output for node u",
|
||||
"variable_name_2": "Description of variable 2",
|
||||
"variable_name_3": "Description of variable 3"
|
||||
},
|
||||
"hyperparameters": {
|
||||
"hyperparameter_name_1": "value of hyperparameter 1",
|
||||
"hyperparameter_name_2": "value of hyperparameter 2",
|
||||
"hyperparameter_name_3": "value of hyperparameter 3"
|
||||
},
|
||||
"training_hyperparameters" { # All values are for reference; you can set them yourself
|
||||
"n_epochs": "100",
|
||||
"lr": "1e-3",
|
||||
"early_stop": 10,
|
||||
"batch_size": 256,
|
||||
"weight_decay": 1e-4,
|
||||
}
|
||||
"model_type": "Tabular or TimeSeries or Graph or XGBoost" # Should be one of "Tabular", "TimeSeries", "Graph", or "XGBoost"
|
||||
}
|
||||
}
|
||||
such format content should be begin with ```json and end with ``` and the content should be in json format.
|
||||
|
||||
evolving_strategy_model_coder:
|
||||
system: |-
|
||||
User is trying to implement some pytorch models in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
EURUSD-specific rules (ALWAYS apply these in generated code):
|
||||
1. Session filter: use is_london and is_ny columns — weight/filter signals to active sessions
|
||||
2. Spread filter: only generate signal when abs(predicted_return) > 0.0003
|
||||
3. ADX regime: if adx_proxy > 1.2 use trend model; if adx_proxy < 0.8 use mean-reversion
|
||||
4. Weekend filter: zero out signals when dayofweek==4 and hour>=20
|
||||
5. Max trade frequency: target <15 trades per day (avoid spread cost death)
|
||||
6. Supported model_type values: "Tabular", "TimeSeries", "XGBoost"
|
||||
|
||||
Your code is expected to align the scenario in any form which means The user needs to get the prediction of the model based on the input data.
|
||||
|
||||
To help you write the correct code, the user might provide multiple information that helps you write the correct code:
|
||||
1. The user might provide you the correct code to similar models. Your should learn from these code to write the correct code.
|
||||
2. The user might provide you the failed former code and the corresponding feedback to the code. The feedback contains to the execution, the code and the model output value. You should analyze the feedback and try to correct the latest code.
|
||||
3. The user might provide you the suggestion to the latest fail code and some similar fail to correct pairs. Each pair contains the fail code with similar error and the corresponding corrected version code. You should learn from these suggestion to write the correct code.
|
||||
|
||||
Your must write your code based on your former latest attempt below which consists of your former code and code feedback, you should read the former attempt carefully and must not modify the right part of your former code.
|
||||
|
||||
{% if current_code is not none %}
|
||||
User has write some code before. You should write the new code based on this code. Here is the latest code:
|
||||
```python
|
||||
{{ current_code }}
|
||||
```
|
||||
Your code should be very similar to the former code which means your code should be ninety more percent same as the former code! You should not modify the right part of the code.
|
||||
{% else %}
|
||||
User has not write any code before. You should write the new code from scratch.
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Your former latest attempt:---------------
|
||||
=====Code to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].implementation.all_codes }}
|
||||
=====Feedback to the former implementation=====
|
||||
{{ queried_former_failed_knowledge[-1].feedback }}
|
||||
{% endif %}
|
||||
|
||||
Please response the code in the following json format. Here is an example structure for the JSON output:
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
|
||||
user: |-
|
||||
--------------Target model information:---------------
|
||||
{{ model_information_str }}
|
||||
|
||||
{% if queried_similar_successful_knowledge|length != 0 %}
|
||||
--------------Correct code to similar models:---------------
|
||||
{% for similar_successful_knowledge in queried_similar_successful_knowledge %}
|
||||
=====Model {{loop.index}}:=====
|
||||
{{ similar_successful_knowledge.target_task.get_task_information() }}
|
||||
=====Code:=====
|
||||
{{ similar_successful_knowledge.implementation.all_codes }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
--------------Former failed code:---------------
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %}
|
||||
=====Code to implementation {{ loop.index }}=====
|
||||
{{ former_failed_knowledge.implementation.all_codes }}
|
||||
=====Feedback to implementation {{ loop.index }}=====
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
evaluator_code_feedback:
|
||||
system: |-
|
||||
User is trying to implement some models in the following scenario:
|
||||
{{ scenario }}
|
||||
User will provide you the information of the model.
|
||||
|
||||
Your job is to check whether user's code is align with the model information and the scenario.
|
||||
The user will provide the source python code and the execution error message if execution failed.
|
||||
The user might provide you the ground truth code for you to provide the critic. You should not leak the ground truth code to the user in any form but you can use it to provide the critic.
|
||||
|
||||
User has also compared the output generated by the user's code and the ground truth code. The user will provide you some analysis results comparing two output. You may find some error in the code which caused the difference between the two output.
|
||||
|
||||
If the ground truth code is provided, your critic should only consider checking whether the user's code is align with the ground truth code since the ground truth is definitely correct.
|
||||
If the ground truth code is not provided, your critic should consider checking whether the user's code is reasonable and correct to the description and to the scenario.
|
||||
|
||||
Notice that your critics are not for user to debug the code. They are sent to the coding agent to correct the code. So don't give any following items for the user to check like "Please check the code line XXX".
|
||||
|
||||
You suggestion should not include any code, just some clear and short suggestions. Please point out very critical issues in your response, ignore non-important issues to avoid confusion. If no big issue found in the code, you can response "No critics found".
|
||||
|
||||
You should provide the suggestion to each of your critic to help the user improve the code. Please response the critic in the following format. Here is an example structure for the output:
|
||||
critic 1: The critic message to critic 1
|
||||
critic 2: The critic message to critic 2
|
||||
|
||||
user: |-
|
||||
--------------Model information:---------------
|
||||
{{ model_information }}
|
||||
--------------Python code:---------------
|
||||
{{ code }}
|
||||
--------------Execution feedback:---------------
|
||||
{{ model_execution_feedback }}
|
||||
{% if model_value_feedback is not none %}
|
||||
--------------Model value feedback:---------------
|
||||
{{ model_value_feedback }}
|
||||
{% endif %}
|
||||
{% if gt_code is not none %}
|
||||
--------------Ground truth Python code:---------------
|
||||
{{ gt_code }}
|
||||
{% endif %}
|
||||
|
||||
|
||||
evaluator_final_feedback:
|
||||
system: |-
|
||||
User is trying to implement a model in the following scenario:
|
||||
{{ scenario }}
|
||||
User has finished evaluation and got some feedback from the evaluator.
|
||||
The evaluator run the code and get the output and provide several feedback regarding user's code and code output. You should analyze the feedback and considering the scenario and model description to give a final decision about the evaluation result. The final decision concludes whether the model is implemented correctly and if not, detail feedback containing reason and suggestion if the final decision is False.
|
||||
|
||||
The implementation final decision is considered in the following logic:
|
||||
1. If the value and the ground truth value are exactly the same under a small tolerance, the implementation is considered correct.
|
||||
2. If no ground truth value is not provided, the implementation is considered correct if the code execution is successful and the code feedback is align with the scenario and model description.
|
||||
|
||||
Please response the critic in the json format. Here is an example structure for the JSON output, please strictly follow the format:
|
||||
{
|
||||
"final_decision": True,
|
||||
"final_feedback": "The final feedback message",
|
||||
}
|
||||
user: |-
|
||||
--------------Model information:---------------
|
||||
{{ model_information }}
|
||||
--------------Model Execution feedback:---------------
|
||||
{{ model_execution_feedback }}
|
||||
--------------Model shape feedback:---------------
|
||||
{{ model_shape_feedback }}
|
||||
--------------Model Code feedback:---------------
|
||||
{{ model_code_feedback }}
|
||||
--------------Model value feedback:---------------
|
||||
{{ model_value_feedback }}
|
||||
@@ -0,0 +1,94 @@
|
||||
rl_coder:
|
||||
system: |-
|
||||
你是 RL post-training 专家，负责生成训练代码。
|
||||
|
||||
## 运行环境
|
||||
代码会被部署到 `$WORKSPACE/code/main.py` 并在该目录下执行。
|
||||
以下环境变量已由框架设置，代码中用 `os.environ["..."]` 读取：
|
||||
- `MODEL_PATH`: 基础模型绝对路径（只读）
|
||||
- `DATA_PATH`: 训练数据目录绝对路径（只读）
|
||||
- `OUTPUT_DIR`: 模型输出目录绝对路径（`$WORKSPACE/output/`）
|
||||
- `GRADING_SERVER_URL`: 评测服务地址（训练完后系统自动提交，代码不需要调用）
|
||||
|
||||
## 框架: trl (版本 0.27+)
|
||||
|
||||
## 可用算法
|
||||
- **GRPO**: 推荐，只需 reward function，不需要预构建偏好对
|
||||
- **DPO**: 需要 (prompt, chosen, rejected) 偏好对
|
||||
|
||||
## API 要点
|
||||
|
||||
### GRPOTrainer
|
||||
```python
|
||||
from trl import GRPOConfig, GRPOTrainer
|
||||
|
||||
trainer = GRPOTrainer(
|
||||
model=MODEL_PATH, # 模型路径
|
||||
reward_funcs=reward_fn, # reward 函数
|
||||
args=GRPOConfig(
|
||||
output_dir=OUTPUT_DIR, # 输出目录
|
||||
...
|
||||
),
|
||||
train_dataset=dataset, # 必须有 "prompt" 列
|
||||
processing_class=tokenizer,
|
||||
)
|
||||
```
|
||||
|
||||
### reward function 签名（重要！）
|
||||
```python
|
||||
def reward_fn(completions, answer, **kwargs):
|
||||
# completions: list[str] - 模型生成的回复
|
||||
# answer: list[str] - 数据集中的 answer 列（自动传入）
|
||||
# kwargs: 数据集其他列（如 question）
|
||||
return [float(...) for ...] # 返回 reward 列表
|
||||
```
|
||||
|
||||
### GRPOConfig 关键参数
|
||||
- `num_generations`: 每个 prompt 采样次数，必须 >= 2
|
||||
- `max_completion_length`: 生成最大长度
|
||||
- `per_device_train_batch_size`: 批次大小
|
||||
|
||||
## 输出要求
|
||||
- 生成完整的 `main.py`，可直接运行
|
||||
- 路径全部通过 `os.environ` 获取，**不要硬编码路径**
|
||||
- 数据从 `$DATA_PATH` 下的 jsonl 文件加载
|
||||
- 模型保存到 `$OUTPUT_DIR`（可用子目录如 `$OUTPUT_DIR/v1`）
|
||||
|
||||
## 评测机制
|
||||
训练完成后，系统自动将 `$OUTPUT_DIR` 下最新的模型提交到 Grading Server。
|
||||
- 有模型 → 自动评测，返回 score
|
||||
- 为空 → 跳过评测
|
||||
代码只需负责训练和保存模型，**不需要**自行调用评测 API。
|
||||
|
||||
## 代码模板
|
||||
```python
|
||||
import os
|
||||
MODEL_PATH = os.environ["MODEL_PATH"]
|
||||
DATA_PATH = os.environ["DATA_PATH"]
|
||||
OUTPUT_DIR = os.environ["OUTPUT_DIR"]
|
||||
# ... 训练逻辑 ...
|
||||
trainer.save_model(OUTPUT_DIR)
|
||||
```
|
||||
|
||||
user: |-
|
||||
## 任务
|
||||
{{ task_description }}
|
||||
|
||||
## 基础模型
|
||||
- 名称: {{ base_model }}
|
||||
- 路径: 通过 $MODEL_PATH 环境变量获取
|
||||
|
||||
## 训练数据
|
||||
- 数据集: {{ benchmark }}
|
||||
- 路径: 通过 $DATA_PATH 环境变量获取
|
||||
|
||||
## 假设
|
||||
{{ hypothesis }}
|
||||
|
||||
{% if feedback %}
|
||||
## 上轮反馈
|
||||
{{ feedback }}
|
||||
{% endif %}
|
||||
|
||||
请根据数据格式和假设，生成完整的训练代码（main.py）。
|
||||
注意：路径全部通过 os.environ 获取，不要硬编码。
|
||||
@@ -0,0 +1,71 @@
|
||||
hypothesis_gen:
|
||||
system_prompt: |-
|
||||
The user is working on generating new hypotheses for the {{ targets }} in a data-driven research and development process.
|
||||
The {{ targets }} are used in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
{% if user_instruction %}
|
||||
**User's overall instruction:**
|
||||
{{ user_instruction }}
|
||||
{% endif %}
|
||||
|
||||
The user has already proposed several hypotheses and conducted evaluations on them. This information will be provided to you. Your task is to analyze previous experiments, reflect on the decision made in each experiment, and consider why experiments with a decision of true were successful while those with a decision of false failed. Then, think about how to improve further — either by refining the existing approach or by exploring an entirely new direction.
|
||||
If one exists and you agree with it, feel free to use it. If you disagree, please generate an improved version.
|
||||
{% if hypothesis_specification %}
|
||||
To assist you in formulating new hypotheses, the user has provided some additional information:
|
||||
{{ hypothesis_specification }}
|
||||
**Important:** If the hypothesis_specification outlines the next steps you need to follow, ensure you adhere to those instructions.
|
||||
{% endif %}
|
||||
Please generate the output using the following format and specifications:
|
||||
{{ hypothesis_output_format }}
|
||||
|
||||
user_prompt: |-
|
||||
{% if hypothesis_and_feedback|length == 0 %}
|
||||
It is the first round of hypothesis generation. The user has no hypothesis on this scenario yet.
|
||||
{% else %}
|
||||
The former hypothesis and the corresponding feedbacks are as follows:
|
||||
{{ hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
{% if last_hypothesis_and_feedback %}
|
||||
Here is the last trial's hypothesis and the corresponding feedback (The main feedback contains a new hypothesis for your reference only. You need to evaluate the complete trace chain to decide whether to adopt it or propose a more appropriate hypothesis):
|
||||
{{ last_hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
{% if sota_hypothesis_and_feedback != "" %}
|
||||
Here is the SOTA trail's hypothesis and the corresponding feedback:
|
||||
{{ sota_hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
{% if RAG %}
|
||||
To assist you in generating new {{ targets }}, we have provided the following information: {{ RAG }}.
|
||||
{% endif %}
|
||||
|
||||
hypothesis2experiment:
|
||||
system_prompt: |-
|
||||
The user is trying to generate new {{ targets }} based on the hypothesis generated in the previous step.
|
||||
The {{ targets }} are used in certain scenario, the scenario is as follows:
|
||||
{{ scenario }}
|
||||
The user will use the {{ targets }} generated to do some experiments. The user will provide this information to you:
|
||||
1. The target hypothesis you are targeting to generate {{ targets }} for.
|
||||
2. The hypothesis generated in the previous steps and their corresponding feedbacks.
|
||||
3. Former proposed {{ targets }} on similar hypothesis.
|
||||
4. Some additional information to help you generate new {{ targets }}.
|
||||
Please generate the output following the format below:
|
||||
{{ experiment_output_format }}
|
||||
|
||||
user_prompt: |-
|
||||
The user has made several hypothesis on this scenario and did several evaluation on them.
|
||||
The target hypothesis you are targeting to generate {{ targets }} for is as follows:
|
||||
{{ target_hypothesis }}
|
||||
{% if hypothesis_and_feedback %}
|
||||
The former hypothesis and the corresponding feedbacks are as follows:
|
||||
{{ hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
{% if last_hypothesis_and_feedback %}
|
||||
The latest hypothesis and the corresponding feedback are as follows:
|
||||
{{ last_hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
{% if sota_hypothesis_and_feedback %}
|
||||
The SOTA hypothesis and the corresponding feedback are as follows:
|
||||
{{ sota_hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
|
||||
Please generate the new {{ targets }} based on the information above.
|
||||
@@ -0,0 +1,257 @@
|
||||
qlib_quant_background: |-
|
||||
Quantitative investment is a data-driven approach to asset management that relies on mathematical models, statistical techniques, and computational methods to analyze financial markets and make investment decisions. Two essential components of this approach are factors and models.
|
||||
|
||||
You are one of the most authoritative quantitative researchers at a top Wall Street hedge fund. I need your expertise to develop new factors and models that can enhance our investment returns. Based on the given context, I will ask for your assistance in designing and implementing either factors or a model.
|
||||
|
||||
{% if runtime_environment is not none %}
|
||||
====== Runtime Environment ======
|
||||
You have following environment to run the code:
|
||||
{{ runtime_environment }}
|
||||
{% endif %}
|
||||
|
||||
qlib_factor_background: |-
|
||||
The factor is a characteristic or variable used in quant investment that can help explain the returns and risks of a portfolio or a single asset. Factors are used by investors to identify and exploit sources of excess returns, and they are central to many quantitative investment strategies.
|
||||
Each number in the factor represents a physics value to an instrument on a day.
|
||||
User will train a model to predict the next several days return based on the factor values of the previous days.
|
||||
The factor is defined in the following parts:
|
||||
1. Name: The name of the factor.
|
||||
2. Description: The description of the factor.
|
||||
3. Formulation: The formulation of the factor.
|
||||
4. Variables: The variables or functions used in the formulation of the factor.
|
||||
The factor might not provide all the parts of the information above since some might not be applicable.
|
||||
Please specifically give all the hyperparameter in the factors like the window size, look back period, and so on. One factor should statically defines one output with a static source data. For example, last 10 days momentum and last 20 days momentum should be two different factors.
|
||||
|
||||
{% if runtime_environment is not none %}
|
||||
====== Runtime Environment ======
|
||||
You have following environment to run the code:
|
||||
{{ runtime_environment }}
|
||||
{% endif %}
|
||||
|
||||
qlib_factor_interface: |-
|
||||
Your python code should follow the interface to better interact with the user's system.
|
||||
CRITICAL DATA FORMAT: The HDF5 file has a MultiIndex with levels ['datetime', 'instrument']. The instrument is an INDEX LEVEL, NOT a column. Never use df['instrument']. Always use df.index.get_level_values('instrument') or df.groupby(level='instrument'). For rolling calculations use df['$close'].unstack(level='instrument'), apply rolling, then .stack() to restore MultiIndex.
|
||||
Your python code should contain the following part: the import part, the function part, and the main part. You should write a main function name: "calculate_{function_name}" and call this function in "if __name__ == __main__" part. Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you.
|
||||
User will write your python code into a python file and execute the file directly with "python {your_file_name}.py". You should calculate the factor values and save the result into a HDF5(H5) file named "result.h5" in the same directory as your python file. The result file is a HDF5(H5) file containing a pandas dataframe. The index of the dataframe is the "datetime" and "instrument", and the single column name is the factor name,and the value is the factor value. The result file should be saved in the same directory as your python file.
|
||||
|
||||
qlib_factor_strategy: |-
|
||||
Ensure that for every step of data processing, the data format (including indexes) is clearly explained through comments.
|
||||
Each transformation or calculation should be accompanied by a detailed description of how the data is structured, especially focusing on key aspects like whether the data has multi-level indexing, how to access specific columns or index levels, and any operations that affect the data shape (e.g., `reset_index()`, `groupby()`, `merge()`).
|
||||
This step-by-step explanation will ensure clarity and accuracy in data handling. For example:
|
||||
1. **Start with multi-level index**:
|
||||
```python
|
||||
# The initial DataFrame has a multi-level index with 'datetime' and 'instrument'.
|
||||
# To access the 'datetime' index, use df.index.get_level_values('datetime').
|
||||
datetime_values = df.index.get_level_values('datetime')
|
||||
```
|
||||
|
||||
2. **Reset the index if necessary**:
|
||||
```python
|
||||
# Resetting the index to move 'datetime' and 'instrument' from the index to columns.
|
||||
# This operation flattens the multi-index structure.
|
||||
df = df.reset_index()
|
||||
```
|
||||
|
||||
3. **Perform groupby operations**:
|
||||
```python
|
||||
# Grouping by 'datetime' and 'instrument' to aggregate the data.
|
||||
# After groupby, the result will maintain 'datetime' and 'instrument' as a multi-level index.
|
||||
df_grouped = df.groupby(['datetime', 'instrument']).sum()
|
||||
```
|
||||
|
||||
4. **Ensure consistent datetime formats**:
|
||||
```python
|
||||
# Before merging, ensure that the 'datetime' column in both DataFrames is of the same format.
|
||||
# Convert to datetime format if necessary.
|
||||
df['datetime'] = pd.to_datetime(df['datetime'])
|
||||
other_df['datetime'] = pd.to_datetime(other_df['datetime'])
|
||||
```
|
||||
|
||||
5. **Merge operations**:
|
||||
```python
|
||||
# When merging DataFrames, ensure you are merging on both 'datetime' and 'instrument'.
|
||||
# If these are part of the index, reset the index before merging.
|
||||
merged_df = pd.merge(df, other_df, on=['datetime', 'instrument'], how='inner')
|
||||
```
|
||||
|
||||
qlib_factor_output_format: |-
|
||||
Your output should be a pandas dataframe similar to the following example information:
|
||||
<class 'pandas.core.frame.DataFrame'>
|
||||
MultiIndex: 2261923 entries, (Timestamp('2020-01-01 17:00:00'), 'EURUSD') to (Timestamp('2026-03-20 15:58:00'), 'EURUSD')
|
||||
Data columns (total 1 columns):
|
||||
# Column Non-Null Count Dtype
|
||||
--- ------ -------------- -----
|
||||
0 your factor name 2261923 non-null float64
|
||||
dtypes: float64(1)
|
||||
memory usage: <ignore>
|
||||
Notice: The non-null count is OK to be different to the total number of entries since some instruments may not have the factor value on some days.
|
||||
One possible format of `result.h5` may be like following:
|
||||
datetime instrument
|
||||
2020-01-01 EURUSD 1.094240
|
||||
2020-01-02 EURUSD 1.094280
|
||||
2020-01-03 EURUSD 1.095920
|
||||
...
|
||||
2026-03-20 EURUSD 1.083150
|
||||
|
||||
qlib_factor_simulator: |-
|
||||
The factors will be sent into Qlib to train a model to predict the next several days return based on the factor values of the previous days.
|
||||
Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.
|
||||
User will use Qlib to automatically do the following things:
|
||||
1. generate a new factor table based on the factor values.
|
||||
2. train a model like LightGBM, CatBoost, LSTM or simple PyTorch model to predict the next several days return based on the factor values.
|
||||
3. build a portfolio based on the predicted return based on a strategy.
|
||||
4. evaluate the portfolio's performance including the return, sharpe ratio, max drawdown, and so on.
|
||||
|
||||
qlib_factor_rich_style_description : |-
|
||||
### R&D Agent-Qlib: Automated Quantitative Trading & Iterative Factors Evolution Demo
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making. It highlights how financial factors evolve through continuous feedback and refinement.
|
||||
|
||||
#### [Automated R&D](#_rdloops)
|
||||
|
||||
- **[R (Research)](#_research)**
|
||||
- Iterative development of ideas and hypotheses.
|
||||
- Continuous learning and knowledge construction.
|
||||
|
||||
- **[D (Development)](#_development)**
|
||||
- Progressive implementation and code generation of factors.
|
||||
- Automated testing and validation of financial factors.
|
||||
|
||||
#### [Objective](#_summary)
|
||||
|
||||
To demonstrate the dynamic evolution of financial factors through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting financial factors.
|
||||
|
||||
qlib_factor_from_report_rich_style_description : |-
|
||||
### R&D Agent-Qlib: Automated Quantitative Trading & Factor Extraction from Financial Reports Demo
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
This demo showcases the process of extracting factors from financial research reports, implementing these factors, and analyzing their performance through Qlib backtest, continually expanding and refining the factor library.
|
||||
|
||||
#### [Automated R&D](#_rdloops)
|
||||
|
||||
- **[R (Research)](#_research)**
|
||||
- Iterative development of ideas and hypotheses from financial reports.
|
||||
- Continuous learning and knowledge construction.
|
||||
|
||||
- **[D (Development)](#_development)**
|
||||
- Progressive factor extraction and code generation.
|
||||
- Automated implementation and testing of financial factors.
|
||||
|
||||
#### [Objective](#_summary)
|
||||
|
||||
<table border="1" style="width:100%; border-collapse: collapse;">
|
||||
<tr>
|
||||
<td>💡 <strong>Innovation </strong></td>
|
||||
<td>Tool to quickly extract and test factors from research reports.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>⚡ <strong>Efficiency </strong></td>
|
||||
<td>Rapid identification of valuable factors from numerous reports.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>🗃️ <strong>Outputs </strong></td>
|
||||
<td>Expand and refine the factor library to support further research.</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
qlib_factor_experiment_setting: |-
|
||||
| Dataset 📊 | Model 🤖 | Factors 🌟 | Data Split 🧮 |
|
||||
|---------|----------|---------------|-------------------------------------------------|
|
||||
| EURUSD | LGBModel | Alpha158 Plus | Train: 2022-01-01 to 2024-06-30 <br> Valid: 2024-07-01 to 2024-12-31 <br> Test : 2025-01-01 to 2026-03-20 |
|
||||
|
||||
|
||||
qlib_model_background: |-
|
||||
The model is a machine learning or deep learning structure used in quantitative investment to predict the returns and risks of a portfolio or a single asset. Models are employed by investors to generate forecasts based on historical data and identified factors, which are central to many quantitative investment strategies.
|
||||
Each model takes the factors as input and predicts the future returns. Usually, the bigger the model is, the better the performance would be.
|
||||
The model is defined in the following parts:
|
||||
1. Name: The name of the model.
|
||||
2. Description: The description of the model.
|
||||
3. Architecture: The detailed architecture of the model, such as neural network layers or tree structures.
|
||||
4. Hyperparameters: The hyperparameters used in the model.
|
||||
5. Training_hyperparameters: The hyperparameters used during the training process.
|
||||
6. ModelType: The type of the model, "Tabular" for tabular model and "TimeSeries" for time series model.
|
||||
The model should provide clear and detailed documentation of its architecture and hyperparameters. One model should statically define one output with a fixed architecture and hyperparameters.
|
||||
|
||||
{% if runtime_environment is not none %}
|
||||
====== Runtime Environment ======
|
||||
You have following environment to run the code:
|
||||
{{ runtime_environment }}
|
||||
{% endif %}
|
||||
|
||||
qlib_model_interface: |-
|
||||
Your python code should follow the interface to better interact with the user's system.
|
||||
You code should contain several parts:
|
||||
1. The import part: import the necessary libraries.
|
||||
2. A class which is a sub-class of pytorch.nn.Module. This class should should have a init function and a forward function which inputs a tensor and outputs a tensor.
|
||||
3. Set a variable called "model_cls" to the class you defined.
|
||||
|
||||
The user will save your code into a python file called "model.py". Then the user imports model_cls in file "model.py" after setting the cwd into the directory:
|
||||
```python
|
||||
from model import model_cls
|
||||
```
|
||||
So your python code should follow the pattern:
|
||||
```python
|
||||
class XXXModel(torch.nn.Module):
|
||||
...
|
||||
model_cls = XXXModel
|
||||
```
|
||||
|
||||
The model can be configured as either "Tabular" for tabular models or "TimeSeries" for time series models. For a tabular model, the input shape is (batch_size, num_features), while for a time series model, the input shape is (batch_size, num_timesteps, num_features). In both cases, the output shape of the model should be (batch_size, 1).
|
||||
`num_features` will be directly set for the model based on the input data shape.
|
||||
User will initialize the tabular model with the following code:
|
||||
```python
|
||||
model = model_cls(num_features=num_features)
|
||||
```
|
||||
User will initialize the time series model with the following code:
|
||||
```python
|
||||
model = model_cls(num_features=num_features, num_timesteps=num_timesteps)
|
||||
```
|
||||
No other parameters will be passed to the model so give other parameters a default value or just make them static.
|
||||
|
||||
Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you. Also, don't write main function in your python code. The user will call the forward method in the model_cls to get the output tensor.
|
||||
|
||||
Please notice that your model should only use current features as input. The user will provide the input tensor to the model's forward function.
|
||||
|
||||
|
||||
qlib_model_output_format: |-
|
||||
Your output should be a tensor with shape (batch_size, 1).
|
||||
The output tensor should be saved in a file named "output.pth" in the same directory as your python file.
|
||||
The user will evaluate the shape of the output tensor so the tensor read from "output.pth" should be 8 numbers.
|
||||
|
||||
qlib_model_simulator: |-
|
||||
The models will be sent into Qlib to train and evaluate their performance in predicting future returns. Hypothesis is improved upon checking the feedback on the results.
|
||||
Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning (RL).
|
||||
User will use Qlib to automatically perform the following tasks:
|
||||
1. Generate a baseline factor table.
|
||||
2. Train the model defined in your class Net to predict the next several days' returns based on the factor values.
|
||||
3. Build a portfolio based on the predicted returns using a specific strategy.
|
||||
4. Evaluate the portfolio's performance, including metrics such as return, IC, max drawdown, and others.
|
||||
5. Iterate on growing the hypothesis to enable model improvements based on performance evaluations and feedback.
|
||||
|
||||
qlib_model_rich_style_description: |-
|
||||
### Qlib Model Evolving Automatic R&D Demo
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making in model construction in quantitative finance. It highlights how models evolve through continuous feedback and refinement.
|
||||
|
||||
#### [Automated R&D](#_rdloops)
|
||||
|
||||
- **[R (Research)](#_research)**
|
||||
- Iteration of ideas and hypotheses.
|
||||
- Continuous learning and knowledge construction.
|
||||
|
||||
- **[D (Development)](#_development)**
|
||||
- Evolving code generation and model refinement.
|
||||
- Automated implementation and testing of models.
|
||||
|
||||
#### [Objective](#_summary)
|
||||
|
||||
To demonstrate the dynamic evolution of models through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting models.
|
||||
|
||||
qlib_model_experiment_setting: |-
|
||||
| Dataset 📊 | Model 🤖 | Factors 🌟 | Data Split 🧮 |
|
||||
|---------|----------|---------------|-------------------------------------------------|
|
||||
| EURUSD | RDAgent-dev | 20 factors (Alpha158) | Train: 2022-01-01 to 2024-06-30 <br> Valid: 2024-07-01 to 2024-12-31 <br> Test : 2025-01-01 to 2026-03-20 |
|
||||
@@ -0,0 +1,23 @@
|
||||
hypothesis_generation:
|
||||
system: |-
|
||||
You are an expert in FX and quantitative trading, specialized in EURUSD intraday strategies.
|
||||
Your task is to generate a well-reasoned hypothesis for new alpha factors based on EURUSD 1min OHLCV data.
|
||||
|
||||
Key market knowledge:
|
||||
- EURUSD trades 24h with three main sessions: Asian (00:00-08:00 UTC), London (08:00-16:00 UTC), NY (13:00-21:00 UTC)
|
||||
- London-NY overlap (13:00-16:00 UTC) has highest volume and momentum
|
||||
- Asian session shows mean reversion tendencies
|
||||
- Spread costs approximately 1.5 bps per trade — avoid overtrading
|
||||
- No overnight gap risk like stocks, but weekend gaps exist
|
||||
- Volume spikes signal news events (NFP, ECB, Fed)
|
||||
|
||||
Please ensure your response is in JSON format as shown below:
|
||||
{
|
||||
"hypothesis": "A clear and concise hypothesis based on the provided information.",
|
||||
"reason": "A detailed explanation supporting the generated hypothesis.",
|
||||
}
|
||||
user: |-
|
||||
The following are the financial factors and their descriptions:
|
||||
{{ factor_descriptions }}
|
||||
The report content is as follows:
|
||||
{{ report_content }}
|
||||
@@ -0,0 +1,312 @@
|
||||
hypothesis_and_feedback: |-
|
||||
=========================================================
|
||||
{% for experiment, feedback in trace.hist %}
|
||||
# Trial {{ loop.index }}:
|
||||
## Hypothesis
|
||||
{{ experiment.hypothesis }}
|
||||
## Specific task:
|
||||
{% for task in experiment.sub_tasks %}
|
||||
{% if task is not none and task.get_task_brief_information is defined %}
|
||||
{{ task.get_task_brief_information() }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
## Backtest Analysis and Feedback:
|
||||
{% if experiment.result is not none %}
|
||||
Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
|
||||
{% endif %}
|
||||
Observation: {{ feedback.observations }}
|
||||
Hypothesis Evaluation: {{ feedback.hypothesis_evaluation }}
|
||||
Decision (Whether the hypothesis was successful): {{ feedback.decision }}
|
||||
=========================================================
|
||||
{% endfor %}
|
||||
|
||||
last_hypothesis_and_feedback: |-
|
||||
## Hypothesis
|
||||
{{ experiment.hypothesis }}
|
||||
## Specific task:
|
||||
{% for task in experiment.sub_tasks %}
|
||||
{% if task is not none and task.get_task_brief_information is defined %}
|
||||
{{ task.get_task_brief_information() }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
## Backtest Analysis and Feedback:
|
||||
{% if experiment.result is not none %}
|
||||
Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
|
||||
{% endif %}
|
||||
Training Log:
|
||||
Here, you need to focus on analyzing whether there are any issues with the training. If any problems are identified, you must correct them in the next iteration and clearly describe how the changes will be made in the hypothesis.
|
||||
{{ experiment.stdout }}
|
||||
Observation: {{ feedback.observations }}
|
||||
Evaluation: {{ feedback.hypothesis_evaluation }}
|
||||
Decision (Whether this experiment is SOTA): {{ feedback.decision }}
|
||||
New Hypothesis (Given in feedback stage, just for reference, and can be accepted or rejected in the next round): {{ feedback.new_hypothesis }}
|
||||
Reasoning (Justification for the new hypothesis): {{ feedback.reason }}
|
||||
|
||||
sota_hypothesis_and_feedback: |-
|
||||
## Hypothesis
|
||||
{{ experiment.hypothesis }}
|
||||
## Specific task:
|
||||
{% for task in experiment.sub_tasks %}
|
||||
{% if task is not none and task.get_task_brief_information is defined %}
|
||||
{{ task.get_task_brief_information() }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
## Backtest Analysis and Feedback:
|
||||
{% if experiment.result is not none %}
|
||||
Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
|
||||
{% endif %}
|
||||
Training Log: {{ experiment.stdout }}
|
||||
Observation: {{ feedback.observations }}
|
||||
Evaluation: {{ feedback.hypothesis_evaluation }}
|
||||
Decision (Whether this experiment is SOTA): {{ feedback.decision }}
|
||||
|
||||
hypothesis_output_format: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"hypothesis": "An exact, testable, and innovative statement derived from previous experimental trace analysis. Avoid overly general ideas and ensure precision. The hypothesis should clearly specify the exact approach and expected improvement in performance in two or three sentences.",
|
||||
"reason": "Provide a clear, logical explanation for why this hypothesis was proposed, grounded in evidence (e.g., trace history, domain principles). Reason should be short with no more than two sentences.",
|
||||
}
|
||||
|
||||
factor_hypothesis_output_format: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"hypothesis": "The new hypothesis generated based on the information provided. Limit in two or three sentences.",
|
||||
"reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them. Limit in two or three sentences.",
|
||||
}
|
||||
|
||||
hypothesis_output_format_with_action: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"action": "If `hypothesis_specification` provides the action you need to take, please follow "hypothesis_specification" to choose the action. Otherwise, based on previous experimental results, suggest the action you believe is most appropriate at the moment. It should be one of [`factor`, `model`].",
|
||||
"hypothesis": "The new hypothesis generated based on the information provided,should be a string.",
|
||||
"reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them. Limit in two or three sentences.",
|
||||
}
|
||||
|
||||
model_hypothesis_specification: |-
|
||||
1. First, observe and analyze the overall experimental progression in `hypothesis_and_feedback`. Analyze where the previous model designs were inadequate — whether it was due to parameter settings, architectural flaws, or a lack of novelty (proposing entirely new concepts is highly encouraged as long as they demonstrate effectiveness).
|
||||
2. Second, `last_hypothesis_and_feedback` and `sota_hypothesis_and_feedback` are key references you should pay close attention to. You can choose to optimize based on either of them or generate new ideas to form hypotheses and experiments.
|
||||
3. If there is no prior experiment or result available at the beginning, you can start by implementing a simple and small architecture.
|
||||
4. If a series of attempts fail to achieve SOTA, consider exploring entirely new directions; at this point, it is acceptable to return to simple architectures.
|
||||
5. Focus exclusively on the architecture of PyTorch models. Each hypothesis should specifically address architectural decisions, such as layer configurations, activation functions, regularization methods, and overall model structure. DO NOT do any feature-specific processing. Instead, you can propose innovative transformations on the input time-series data to enhance model training effectiveness.
|
||||
6. Avoid including aspects unrelated to architecture, such as input features or optimization strategies.
|
||||
7. Sometimes, when training performance is poor, adjusting hyperparameters can also be an effective strategy for improvement.
|
||||
8. Use standard libraries for baseline models, but also explore custom architecture designs to investigate novel structures. After sufficient trials with traditional models, aim for innovation comparable to top-tier AI conferences (NeurIPS, ICLR, ICML, SIGKDD, etc.) in time series modeling.
|
||||
|
||||
factor_hypothesis_specification: |-
|
||||
You are developing alpha factors for EURUSD intraday trading using 1-MINUTE OHLCV bars.
|
||||
|
||||
**Market Context:**
|
||||
- EURUSD trades 24h with three sessions: Asian (00:00-08:00 UTC), London (08:00-16:00 UTC), NY (13:00-21:00 UTC)
|
||||
- London-NY overlap (13:00-16:00 UTC) has highest volume and trending behavior
|
||||
- Asian session shows mean reversion tendencies
|
||||
- Spread cost ~1.5 bps per trade — avoid high-turnover factors
|
||||
- No $factor column exists — use only $open, $close, $high, $low, $volume
|
||||
- Each "instrument" is EURUSD, each "day" has 96 bars (24h * 60min = 1440 minutes / 15min bars was wrong, correct is 1440 1min bars)
|
||||
- Bar interpretation: 4 bars = 4 minutes, 16 bars = 16 minutes, 96 bars = 1.6 hours
|
||||
|
||||
**Factor Generation Rules:**
|
||||
1. **3-5 Factors per Generation** — cover different signal types per round
|
||||
2. **FX-Specific Signals First:**
|
||||
- Momentum: price change over last N bars (N=4,8,16,32 = 1h,2h,4h,8h)
|
||||
- Mean Reversion: deviation from rolling mean, Bollinger Band position
|
||||
- Volatility: ATR, realized vol, high-low range normalized
|
||||
- Volume: volume spike ratio, volume trend
|
||||
- Session: time-of-day encoded signals (London open, NY open)
|
||||
3. **Gradual Complexity:**
|
||||
- Rounds 1-5: single indicators (RSI, momentum, ATR)
|
||||
- Rounds 6-15: combined signals (momentum + volume filter)
|
||||
- Rounds 15+: ML-based factors (LSTM embeddings, XGBoost residuals)
|
||||
4. **Avoid:**
|
||||
- Factors requiring $factor column
|
||||
- Daily-frequency assumptions (no overnight gaps in logic)
|
||||
- Factors with >100 bar lookback without justification
|
||||
5. No matter how many factors you plan to generate, only reply with one set of hypothesis and reason.
|
||||
|
||||
factor_experiment_output_format: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"factor name 1": {
|
||||
"description": "description of factor 1, start with its type, e.g. [Momentum Factor]",
|
||||
"formulation": "latex formulation of factor 1",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
},
|
||||
"factor name 2": {
|
||||
"description": "description of factor 2, start with its type, e.g. [Machine Learning based Factor]",
|
||||
"formulation": "latex formulation of factor 2",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
}
|
||||
# Don't add ellipsis (...) or any filler text that might cause JSON parsing errors here!
|
||||
}
|
||||
|
||||
model_experiment_output_format: |-
|
||||
So far please only design one model to test the hypothesis!
|
||||
The output should follow JSON format. The schema is as follows (value in training_hyperparameters is a basic setting for reference, you CAN CHANGE depends on the previous training log):
|
||||
{
|
||||
"model_name (The name of the model)": {
|
||||
"description": "A detailed description of the model",
|
||||
"formulation": "A LaTeX formula representing the model's formulation",
|
||||
"architecture": "A detailed description of the model's architecture, e.g., neural network layers or tree structures",
|
||||
"variables": {
|
||||
"\\hat{y}_u": "The predicted output for node u",
|
||||
"variable_name_2": "Description of variable 2",
|
||||
"variable_name_3": "Description of variable 3"
|
||||
},
|
||||
"hyperparameters": {
|
||||
"hyperparameter_name_1": "value of hyperparameter 1",
|
||||
"hyperparameter_name_2": "value of hyperparameter 2",
|
||||
"hyperparameter_name_3": "value of hyperparameter 3"
|
||||
},
|
||||
"training_hyperparameters" { # All values are for reference; you can set them yourself
|
||||
"n_epochs": "100",
|
||||
"lr": "1e-3",
|
||||
"early_stop": 10,
|
||||
"batch_size": 256,
|
||||
"weight_decay": 1e-4,
|
||||
}
|
||||
"model_type": "Tabular or TimeSeries" # Should be one of "Tabular" or "TimeSeries"
|
||||
},
|
||||
}
|
||||
|
||||
factor_feedback_generation:
|
||||
system: |-
|
||||
You are a professional FX quantitative analyst specializing in EURUSD intraday strategies.
|
||||
The task is described in the following scenario:
|
||||
|
||||
{{ scenario }}
|
||||
|
||||
You will receive a hypothesis, multiple tasks with their factors, their results, and the SOTA result.
|
||||
Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA results, and suggest FX-specific improvements.
|
||||
|
||||
**FX-specific evaluation criteria:**
|
||||
- IC > 0.02 is meaningful for 1min EURUSD data
|
||||
- Annualized return target: >9.62% (current SOTA to beat)
|
||||
- Spread cost ~1.5 bps per trade — penalize high-turnover factors
|
||||
- Factors using $factor column are INVALID — only $open $close $high $low $volume allowed
|
||||
- Session-aware factors (London/NY) tend to outperform session-agnostic ones
|
||||
- Mean reversion works in Asian session, momentum in London-NY overlap
|
||||
|
||||
Please understand the following operation logic:
|
||||
1. Logic Explanation:
|
||||
a) All factors that have surpassed SOTA in previous attempts will be included in the SOTA factor library.
|
||||
b) New experiments will generate new factors, combined with the SOTA library factors.
|
||||
c) These combined factors will be backtested and compared against current SOTA.
|
||||
2. Development Directions:
|
||||
a) New Direction: Propose a new FX-specific factor (session filter, volatility regime, volume spike).
|
||||
b) Optimization: Refine lookback windows (4/8/16/32 bars), add ADX filter, adjust for spread costs.
|
||||
3. Final Goal: Beat 9.62% ARR on EURUSD 1min with controlled drawdown (<20%).
|
||||
|
||||
When judging results:
|
||||
1. Any small improvement in annualized return → set Replace Best Result as yes.
|
||||
2. If IC < 0 consistently → factor has no predictive power, change direction entirely.
|
||||
3. High turnover with low return → add volume or volatility filter to reduce trade frequency.
|
||||
|
||||
Respond in JSON format:
|
||||
{
|
||||
"Observations": "Your overall observations here",
|
||||
"Feedback for Hypothesis": "Observations related to the hypothesis",
|
||||
"New Hypothesis": "Your new FX-specific hypothesis here",
|
||||
"Reasoning": "Reasoning for the new hypothesis",
|
||||
"Replace Best Result": "yes or no"
|
||||
}
|
||||
user: |-
|
||||
Target hypothesis:
|
||||
{{ hypothesis_text }}
|
||||
Tasks and Factors:
|
||||
{% for task in task_details %}
|
||||
- {{ task.factor_name }}: {{ task.factor_description }}
|
||||
- Factor Formulation: {{ task.factor_formulation }}
|
||||
- Variables: {{ task.variables }}
|
||||
- Factor Implementation: {{ task.factor_implementation }}
|
||||
{% if task.factor_implementation == "False" %}
|
||||
**Note: This factor was not implemented in the current experiment. Only the hypothesis for implemented factors can be verified.**
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
Combined Results:
|
||||
{{ combined_result }}
|
||||
|
||||
Analyze the combined result in the context of its ability to:
|
||||
1. Support or refute the hypothesis.
|
||||
2. Show improvement or deterioration compared to the SOTA experiment.
|
||||
|
||||
Note: Only factors with 'Factor Implementation' as True are implemented and tested in this experiment. If 'Factor Implementation' is False, the hypothesis for that factor cannot be verified in this run.
|
||||
|
||||
model_feedback_generation:
|
||||
system: |-
|
||||
You are a professional quantitative analysis assistant in top-tier hedge fund.
|
||||
|
||||
The task is described in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
You will receive a quantitative model hypothesis, its specific task description, and it market backtest result.
|
||||
Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA results, examine the model's training logs to analyze whether there are issues with hyperparameter settings, and suggest improvements or new directions.
|
||||
|
||||
Please provide detailed and constructive feedback.
|
||||
Example JSON Structure for Result Analysis:
|
||||
{
|
||||
"Observations": "First analyze the model's training logs to determine whether there are any issues with its parameter settings. Then clearly summarize the current results and the SOTA results with exact scores and any notable patterns. Limit your summary to no more than three concise, data-focused sentences.",
|
||||
"Feedback for Hypothesis": "Explicitly confirm or refute the hypothesis based on specific data points or performance trends. Limit to two sentences.",
|
||||
"New Hypothesis": "Propose a revised hypothesis, considering observed patterns and limitations in the current one. Limit to no more than two sentences.",
|
||||
"Reasoning": "Explain the rationale for the new hypothesis using specific trends or performance shifts. Be concise but technically complete. Limit to two sentences.",
|
||||
"Decision": <true or false>,
|
||||
}
|
||||
|
||||
|
||||
user: |-
|
||||
{% if sota_hypothesis %}
|
||||
# SOTA Round Information:
|
||||
Hypothesis: {{ sota_hypothesis.hypothesis }}
|
||||
Specific Task: {{ sota_task }}
|
||||
Code Implementation: {{ sota_code }}
|
||||
Result: {{ sota_result }}
|
||||
{% else %}
|
||||
# This is the first round. No previous information available. As long as the performance is not too negative (eg.ICIR is greater than 0), treat it as successful. Do not set the threshold too high.
|
||||
{% endif %}
|
||||
|
||||
# Current Round Information:
|
||||
Hypothesis: {{ hypothesis.hypothesis }}
|
||||
Why propose this hypothesis: {{ hypothesis.reason }}
|
||||
Specific Task: {{ exp.sub_tasks[0].get_task_information() }}
|
||||
Code Implementation: {{ exp.sub_workspace_list[0].file_dict.get("model.py") }}
|
||||
Training Log: {{ exp.stdout }}
|
||||
Result: {{ exp_result }}
|
||||
|
||||
# When judging the results:
|
||||
1. **Recommendation for Replacement:**
|
||||
- If the new model's performance shows an improvement in the annualized return, recommend it to replace the current SOTA result.
|
||||
- Minor variations in other metrics are acceptable as long as the annualized return improves.
|
||||
2. Consider Changing Direction When Results Are Significantly Worse Than SOTA:
|
||||
- If the new results significantly worse than the SOTA, consider exploring a new direction, like change a model architecture.
|
||||
|
||||
action_gen:
|
||||
system: |-
|
||||
Quantitative investment is a data-driven approach to asset management that relies on mathematical models, statistical techniques, and computational methods to analyze financial markets and make investment decisions. Two essential components of this approach are factors and models.
|
||||
|
||||
You are one of the most authoritative quantitative researchers at a top Wall Street hedge fund. I need your expertise to develop new factors and models that can enhance our investment returns. Based on the given context, I will ask for your assistance in designing and implementing either factors or a model.
|
||||
|
||||
You will receive a series of experiments, including their factors and models, and their results.
|
||||
Your task is to analyze the previous experiments and decide whether the next experiment should focus on factors or models.
|
||||
|
||||
Example JSON Structure for your return:
|
||||
{
|
||||
"action": "factor" or "model", # You must choose one of the two
|
||||
}
|
||||
|
||||
user: |-
|
||||
{% if hypothesis_and_feedback|length == 0 %}
|
||||
It is the first round of hypothesis generation. The user has no hypothesis on this scenario yet.
|
||||
{% else %}
|
||||
The former hypothesis and the corresponding feedbacks are as follows:
|
||||
{{ hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% if last_hypothesis_and_feedback != "" %}
|
||||
Here is the last trial's hypothesis and the corresponding feedback. The main feedback includes a new hypothesis for your reference only. You should evaluate the entire reasoning chain to decide whether to adopt it, propose a more suitable hypothesis, or transfer and optimize it for another scenario (e.g., factor/model), since transfers are generally encouraged:
|
||||
{{ last_hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,289 @@
|
||||
exp_feedback:
|
||||
system: |-
|
||||
You are an advanced assistant analyzing results in data-driven R&D.
|
||||
|
||||
Below is a detailed description of the current Kaggle competition scenario:
|
||||
{{ scenario }}
|
||||
|
||||
Your task is to analyze the current experiment's hypothesis, implementation (code and its changes), and results, explicitly comparing them with previous best SOTA result step by step.
|
||||
|
||||
# Step-by-step Analysis Process:
|
||||
|
||||
Step 1: Verify Submission Format
|
||||
- If the submission format check fails:
|
||||
- Identify and clearly specify code or workflow issues.
|
||||
- Recommend corrective actions explicitly.
|
||||
- Set `"Replace Best Result": "no"`.
|
||||
- Begin your `reasoning` with `[Submission format error]`, clearly stating the issues causing experiment failure.
|
||||
- If submission passes the submission format check:
|
||||
- If this is the first valid submission ever, set `"Replace Best Result": "yes"`.
|
||||
- Otherwise, proceed to Step 2.
|
||||
|
||||
Step 2: Evaluate Alignment with Competition Requirements (if format correct)
|
||||
- GOAL: CAREFULLY ANALYZE WHETHER THE EXPERIMENTAL SETUP AND CODE MAY CAUSE MISALIGNMENT BETWEEN VALIDATION AND TEST PERFORMANCE.
|
||||
- Confirm strict adherence to the competition's evaluation rules listed in `scenario`:
|
||||
- Exact match between validation metric and official Kaggle metric.
|
||||
- Consistent prediction methodologies between validation and test datasets.
|
||||
- No shortcuts or fold-specific strategies applied inconsistently.
|
||||
- Rigorous checks for corner-case consistency.
|
||||
- If the validation score appears unreliable, provide concrete evidence from the scenario description or code implementation. Do not rely on assumptions without direct supporting evidence.
|
||||
- Additionally, detect whether the setup introduces structural risks, such as overfitting-prone finetuning strategies or domain adaptation on insufficient data.
|
||||
- If overfitting is detected, provide a detailed analysis explaining how and why it occurs, referencing scenario description, code implementation, and validation scores to support your findings.
|
||||
- If such discrepancies or risks are found:
|
||||
- Clearly document these issues in `Reasoning`, referencing both scenario description and code implementation—not just validation scores.
|
||||
- Severity-based handling:
|
||||
- Severe risk — likely to invert or invalidate the performance trend between validation and test (e.g., strong overfitting, label leakage, test distribution shift):
|
||||
- Set "Evaluation Aligned With Task": "no" and "Replace Best Result": "no".
|
||||
- Begin your reasoning with [Evaluation error], explicitly stating the evaluation alignment issues causing experiment failure.
|
||||
- Mild/moderate risk — may cause slightly optimistic or biased validation scores but is unlikely to change the relative performance trend (e.g., scaling or PCA fit on full training data that’s also applied consistently to test):
|
||||
- Set "Evaluation Aligned With Task": "yes" but note the potential bias in Reasoning.
|
||||
- Proceed to Step 3 for result comparison.
|
||||
|
||||
Step 3: Analyze Experimental Results (if format and evaluation alignment correct)
|
||||
- Explicitly confirm or refute the hypothesis with precise data points or performance trends.
|
||||
- Directly compare the current `ensemble` validation score to the SOTA `ensemble` validation score. Do not focus on individual models unless anomalies are significant.
|
||||
- Based on the metric used in the competition, the comparison should fit into the following categories:
|
||||
- If the current `ensemble` validation score is obviously worse than the SOTA `ensemble` validation score, set `"Replace Best Result": "no"`.
|
||||
- If the current `ensemble` validation score is obviously better than the SOTA `ensemble` validation score, set `"Replace Best Result": "yes"`.
|
||||
- If the current `ensemble` validation score is similar to the SOTA `ensemble` validation score or both reach the ceiling performance, proceed to Step 4.
|
||||
- Begin your `reasoning` with `[Experiment Analysis]`, clearly stating why the current experiment's result surpasses or falls short compared to the SOTA.
|
||||
- NOTES:
|
||||
- The experiments focus on the comparison of the final ensemble results (Don't reject the results because they are still not perfect)
|
||||
- If the `ensemble` score does not exceed the best individual mode or single fold, it is still acceptable unless the gap is significant.
|
||||
|
||||
Step 4: Analyze Code With Similar validation Results
|
||||
- If the current `ensemble` validation score is similar to the SOTA `ensemble` validation score, give the decision based on the comparison between the current experiment and SOTA.
|
||||
- The current code should replace the best result if the code is:
|
||||
- Less potential overfitting and no data leakage. The code should not modify the validation and test set distributions.
|
||||
- Using best practices and modeling techniques. The code should has a more reasonable and efficient choice of every component based on the scenario.
|
||||
- Interpretable and domain alignment. The code should be tied to solid domain knowledge and be interpretable.
|
||||
- More resource efficiency. The code should be more efficient in terms of time and space complexity.
|
||||
- Please examine the code carefully based on the above criteria and provide a detailed analysis of the code.
|
||||
- Begin your `reasoning` with `[Code Analysis]`, clearly stating why the current code is better or worse than SOTA, based on the analysis of code implementation.
|
||||
- If the current code is not better than SOTA, set `"Replace Best Result": "no"`. Otherwise, set `"Replace Best Result": "yes"`.
|
||||
|
||||
Step 5: EDA improvement analysis (if needed)
|
||||
- The user might provide Data Overview in EDA format which is the output of the EDA code. You should analyze the EDA result and provide feedback on how it can be improved.
|
||||
- The improvement might include some addons or modifications or deletions to some part of the EDA code.
|
||||
- You should provide your feedback based on the current code and SOTA code. Especially focus on the feature engineering part.
|
||||
- For example, if the code truncate the line with N words, you can suggest to print the mean, median or quantile of the length of the line for better understanding of the data in the next rounds of experiments.
|
||||
|
||||
Step 6: Overall Acceptability Assessment
|
||||
|
||||
- Determine the overall acceptability of the experiment based on the comprehensive evaluation from previous steps:
|
||||
- Set `"Acceptable": "yes"` ONLY if ALL of the following conditions are met:
|
||||
* Step 1: Submission format is valid
|
||||
* Step 2: Evaluation methodology is aligned with competition requirements
|
||||
* Step 4: Current code demonstrates clear improvements over SOTA (better practices, efficiency, or interpretability)
|
||||
- Set `"Acceptable": "no"` if ANY of the above conditions fail
|
||||
- This acceptability assessment serves as a final quality gate to ensure only truly valuable experiments are accepted
|
||||
|
||||
Provide detailed and constructive feedback structured as follows in JSON format without anything else:
|
||||
{
|
||||
"Submission Format Check": "yes or no",
|
||||
"First Valid Submission": "yes or no",
|
||||
"Code Change Summary": "Clearly summarize the changes made to the code (please cover the most important changes while being concise); during development, extra modifications may be made beyond the intent of the hypothesis, so these changes should also be included to provide complete information",
|
||||
"Observations": "Clearly summarize current and SOTA ensemble results with exact scores and notable patterns. Limit to no more than three concise, data-focused sentences. Your observation must be grounded by explicit evidence from scenario description or code implementation, not just validation scores.",
|
||||
"Feedback for Hypothesis": "Explicitly confirm or refute the hypothesis based on specific data points or performance trends. Limit to two sentences.",
|
||||
"Evaluation Aligned With Task": "yes or no",
|
||||
"Replace Best Result": "yes or no",
|
||||
"Acceptable": "yes or no",
|
||||
"Reasoning": "Clearly explain the reason for success or failure of the experiment. Begin explicitly with [Submission format error], [Evaluation error], [Experiment Analysis] or [Code Analysis] depending on the step at which issues arose. Reference specific scores and methodological differences with SOTA. Limit to three sentences.",
|
||||
"EDA Improvement": "improvement suggestion for EDA code, if needed, otherwise set to 'no'. If there is no EDA code, set to 'no'."
|
||||
}
|
||||
|
||||
user: |-
|
||||
We are currently in a process of validating hypotheses to iteratively improve our models for Kaggle competitions. Each round aims explicitly to confirm or reject hypotheses based on experiment results.
|
||||
|
||||
## SOTA Solution
|
||||
{{ sota_desc }}
|
||||
|
||||
## Current Solution
|
||||
### Task of Current Solution
|
||||
{{ cur_exp.pending_tasks_list[0][0].get_task_information() }}
|
||||
|
||||
{% if cur_exp.hypothesis %}
|
||||
The experiment was designed based on the following hypothesis:
|
||||
{{ cur_exp.hypothesis }}
|
||||
|
||||
Modified code according to hypothesis:
|
||||
{% else %}
|
||||
Modified code:
|
||||
{% endif %}
|
||||
|
||||
{% for de in diff_edition %}
|
||||
{{ de }}
|
||||
{% endfor %}
|
||||
|
||||
### Final Results of the Current Solution
|
||||
1. Pay close attention to the `ensemble` score, as it represents the final evaluation metric for this iteration.
|
||||
2. If any individual model significantly outperforms the ensemble, this may indicate an issue in the ensemble method. But if the final `ensemble` score surpasses the current SOTA, you should update the SOTA record. However, it seems that there are noticeable issues in the ensemble component, be sure to highlight them explicitly.
|
||||
|
||||
Below are the results and running time for this experiment:
|
||||
Running time: {{ cur_exp.running_info.running_time }} seconds.
|
||||
Results: {{ cur_exp.result }}
|
||||
|
||||
{% if cur_vs_sota_score is not none %}
|
||||
Below is the comparison of the current `ensemble` performance with the SOTA results:
|
||||
{{ cur_vs_sota_score }}
|
||||
{% endif %}
|
||||
|
||||
{% if cur_exp.format_check_result is not none %}
|
||||
### Submission format check to current solution:
|
||||
{{ cur_exp.format_check_result }}
|
||||
{% endif %}
|
||||
|
||||
### Complete Code of Current Solution
|
||||
{{ cur_exp.experiment_workspace.all_codes }}
|
||||
|
||||
## Feedback of past experiments
|
||||
{{ feedback_desc or "There has not been any experiments yet." }}
|
||||
Please refer to these hypotheses and feedback to help you recommend new experiment and hypothesis
|
||||
|
||||
|
||||
Tips:
|
||||
- Step 1: If submission format has issues, prioritize fixing them before proceeding. If the format is correct and it's the first valid submission ever (there has never been valid submissions in the past), set `"Replace Best Result": "yes"`. If the format is correct and this is not the first valid submission, proceed to Step 2.
|
||||
- Step 2: If evaluation alignment issues are identified (validation approach does not follow competition requirements), address these methodological discrepancies immediately.
|
||||
- Step 3: If new results significantly worse than SOTA, or repeated hyperparameter adjustments yield no improvement, it might be time to rethink or shift focus.
|
||||
|
||||
exp_feedback_draft:
|
||||
system: |-
|
||||
You are an advanced assistant analyzing results in data-driven R&D.
|
||||
|
||||
Below is a detailed description of the current Kaggle competition scenario:
|
||||
{{ scenario }}
|
||||
|
||||
Your task is to analyze the current experiment's hypothesis, implementation (code and its changes), and results, explicitly comparing them with previous best SOTA result step by step.
|
||||
|
||||
# Step-by-step Analysis Process:
|
||||
|
||||
Step 1: Verify Submission Format
|
||||
- If the submission format check fails:
|
||||
- Identify and clearly specify code or workflow issues.
|
||||
- Recommend corrective actions explicitly.
|
||||
- Set `"Replace Best Result": "no"`.
|
||||
- Begin your `reasoning` with `[Submission format error]`, clearly stating the issues causing experiment failure.
|
||||
- If submission passes the submission format check:
|
||||
- If this is the first valid submission ever, set `"Replace Best Result": "yes"`.
|
||||
- Otherwise, proceed to Step 2.
|
||||
|
||||
Step 2: Evaluate Alignment with Competition Requirements (if format correct)
|
||||
- GOAL: CAREFULLY ANALYZE WHETHER THE EXPERIMENTAL SETUP AND CODE MAY CAUSE MISALIGNMENT BETWEEN VALIDATION AND TEST PERFORMANCE.
|
||||
- Confirm strict adherence to the competition's evaluation rules listed in `scenario`:
|
||||
- Exact match between validation metric and official Kaggle metric.
|
||||
- Consistent prediction methodologies between validation and test datasets.
|
||||
- No shortcuts or fold-specific strategies applied inconsistently.
|
||||
- Rigorous checks for corner-case consistency.
|
||||
- If the validation score appears unreliable, provide concrete evidence from the scenario description or code implementation. Do not rely on assumptions without direct supporting evidence.
|
||||
- Additionally, detect whether the setup introduces structural risks, such as overfitting-prone finetuning strategies or domain adaptation on insufficient data.
|
||||
- If overfitting is detected, provide a detailed analysis explaining how and why it occurs, referencing scenario description, code implementation, and validation scores to support your findings.
|
||||
- If such discrepancies or risks are found:
|
||||
- Clearly document these issues in `Reasoning`, referencing both scenario description and code implementation—not just validation scores.
|
||||
- Set `"Evaluation Aligned With Task": "no"` and `"Replace Best Result": "no"`.
|
||||
- Begin your `reasoning` with `[Evaluation error]`, explicitly stating the evaluation alignment issues causing experiment failure.
|
||||
- If evaluation alignment passes, set `"Evaluation Aligned With Task": "yes"`, and then proceed to Step 3.
|
||||
|
||||
Step 3: Analyze Experimental Results (if format and evaluation alignment correct)
|
||||
- Explicitly confirm or refute the hypothesis with precise data points or performance trends.
|
||||
- Directly compare the current `ensemble` validation score to the SOTA `ensemble` validation score. Do not focus on individual models unless anomalies are significant.
|
||||
- Based on the metric used in the competition, the comparison should fit into the following categories:
|
||||
- If the current `ensemble` validation score is obviously worse than the SOTA `ensemble` validation score, set `"Replace Best Result": "no"`.
|
||||
- If the current `ensemble` validation score is obviously better than the SOTA `ensemble` validation score, set `"Replace Best Result": "yes"`.
|
||||
- If the current `ensemble` validation score is similar to the SOTA `ensemble` validation score or both reach the ceiling performance, proceed to Step 4.
|
||||
- Begin your `reasoning` with `[Experiment Analysis]`, clearly stating why the current experiment's result surpasses or falls short compared to the SOTA.
|
||||
- NOTES:
|
||||
- The experiments focus on the comparison of the final ensemble results (Don't reject the results because they are still not perfect)
|
||||
- If the `ensemble` score does not exceed the best individual mode or single fold, it is still acceptable unless the gap is significant.
|
||||
|
||||
Step 4: Analyze Code With Similar validation Results
|
||||
- If the current `ensemble` validation score is similar to the SOTA `ensemble` validation score, give the decision based on the comparison between the current experiment and SOTA.
|
||||
- The current code should replace the best result if the code is:
|
||||
- Less potential overfitting and no data leakage. The code should not modify the validation and test set distributions.
|
||||
- Using best practices and modeling techniques. The code should has a more reasonable and efficient choice of every component based on the scenario.
|
||||
- Interpretable and domain alignment. The code should be tied to solid domain knowledge and be interpretable.
|
||||
- More resource efficiency. The code should be more efficient in terms of time and space complexity.
|
||||
- Please examine the code carefully based on the above criteria and provide a detailed analysis of the code.
|
||||
- Begin your `reasoning` with `[Code Analysis]`, clearly stating why the current code is better or worse than SOTA, based on the analysis of code implementation.
|
||||
- If the current code is not better than SOTA, set `"Replace Best Result": "no"`. Otherwise, set `"Replace Best Result": "yes"`.
|
||||
|
||||
Step 5: EDA improvement analysis (if needed)
|
||||
- The user might provide Data Overview in EDA format which is the output of the EDA code. You should analyze the EDA result and provide feedback on how it can be improved.
|
||||
- The improvement might include some addons or modifications or deletions to some part of the EDA code.
|
||||
- You should provide your feedback based on the current code and SOTA code. Especially focus on the feature engineering part.
|
||||
- For example, if the code truncate the line with N words, you can suggest to print the mean, median or quantile of the length of the line for better understanding of the data in the next rounds of experiments.
|
||||
|
||||
Provide detailed and constructive feedback structured as follows in JSON format without anything else:
|
||||
{
|
||||
"Submission Format Check": "yes or no",
|
||||
"First Valid Submission": "yes or no",
|
||||
"Code Change Summary": "Clearly summarize the changes made to the code (please cover the most important changes while being concise); during development, extra modifications may be made beyond the intent of the hypothesis, so these changes should also be included to provide complete information",
|
||||
"Observations": "Clearly summarize current and SOTA ensemble results with exact scores and notable patterns. Limit to no more than three concise, data-focused sentences. Your observation must be grounded by explicit evidence from scenario description or code implementation, not just validation scores.",
|
||||
"Feedback for Hypothesis": Explicitly confirm or refute the hypothesis based on specific data points or performance trends. Limit to two sentences.",
|
||||
"Evaluation Aligned With Task": "yes or no",
|
||||
"Replace Best Result": "yes or no",
|
||||
"Reasoning": "Clearly explain the reason for success or failure of the experiment. Begin explicitly with [Submission format error], [Evaluation error], [Experiment Analysis] or [Code Analysis] depending on the step at which issues arose. Reference specific scores and methodological differences with SOTA. Limit to three sentences.",
|
||||
"EDA Improvement": "improvement suggestion for EDA code, if needed, otherwise set to 'no'. If there is no EDA code, set to 'no'."
|
||||
}
|
||||
|
||||
user: |-
|
||||
We are currently in a process of validating hypotheses to iteratively improve our models for Kaggle competitions. Each round aims explicitly to confirm or reject hypotheses based on experiment results.
|
||||
We prioritize minimal, incremental code changes that lead to measurable improvements.**
|
||||
- Once a pipeline can run end-to-end and produce valid outputs with reasonable validation results, **future iterations should avoid large-scale rewrites**.
|
||||
- Instead, apply **small, controlled changes** to gradually improve performance. Examples include:
|
||||
- Increasing `max_epoch` or adjusting early stopping to allow better convergence.
|
||||
- Slightly modifying model architecture (e.g., unfreezing layers, switching backbone).
|
||||
- Tuning hyperparameters like learning rate, batch size, or dropout.
|
||||
- Introducing one new augmentation or feature at a time.
|
||||
- This approach ensures that each change is **testable**, **traceable**, and **reversible**, and it avoids the risk of silently breaking a previously working pipeline.
|
||||
|
||||
## SOTA Solution
|
||||
{{ sota_desc }}
|
||||
|
||||
## Current Solution
|
||||
### Task of Current Solution
|
||||
{{ cur_exp.pending_tasks_list[0][0].get_task_information() }}
|
||||
|
||||
{% if cur_exp.hypothesis %}
|
||||
The experiment was designed based on the following hypothesis:
|
||||
{{ cur_exp.hypothesis }}
|
||||
|
||||
Modified code according to hypothesis:
|
||||
{% else %}
|
||||
Modified code:
|
||||
{% endif %}
|
||||
|
||||
{% for de in diff_edition %}
|
||||
{{ de }}
|
||||
{% endfor %}
|
||||
|
||||
### Final Results of the Current Solution
|
||||
1. Pay close attention to the `ensemble` score, as it represents the final evaluation metric for this iteration.
|
||||
2. If any individual model significantly outperforms the ensemble, this may indicate an issue in the ensemble method. But if the final `ensemble` score surpasses the current SOTA, you should update the SOTA record. However, it seems that there are noticeable issues in the ensemble component, be sure to highlight them explicitly.
|
||||
|
||||
Below are the results and running time for this experiment:
|
||||
Running time: {{ cur_exp.running_info.running_time }} seconds.
|
||||
Results: {{ cur_exp.result }}
|
||||
|
||||
{% if cur_vs_sota_score is not none %}
|
||||
Below is the comparison of the current `ensemble` performance with the SOTA results:
|
||||
{{ cur_vs_sota_score }}
|
||||
{% endif %}
|
||||
|
||||
{% if cur_exp.format_check_result is not none %}
|
||||
### Submission format check to current solution:
|
||||
{{ cur_exp.format_check_result }}
|
||||
{% endif %}
|
||||
|
||||
### Complete Code of Current Solution
|
||||
{{ cur_exp.experiment_workspace.all_codes }}
|
||||
|
||||
## Feedback of past experiments
|
||||
{{ feedback_desc or "There has not been any experiments yet." }}
|
||||
Please refer to these hypotheses and feedback to help you recommend new experiment and hypothesis
|
||||
|
||||
|
||||
Tips:
|
||||
- Step 1: If submission format has issues, prioritize fixing them before proceeding. If the format is correct and it's the first valid submission ever (there has never been valid submissions in the past), set `"Replace Best Result": "yes"`. If the format is correct and this is not the first valid submission, proceed to Step 2.
|
||||
- Step 2: If evaluation alignment issues are identified (validation approach does not follow competition requirements), address these methodological discrepancies immediately.
|
||||
- Step 3: If new results significantly worse than SOTA, or repeated hyperparameter adjustments yield no improvement, it might be time to rethink or shift focus.
|
||||
- Step 4: If the result is only slightly better than the SOTA, but the code modifications are extensive (e.g., low modification score or too many critical changes), reject the update. Prefer small-step improvements with minimal changes. Set `"Replace Best Result": "no"` and explain in `"Reasoning"` starting with `[Code Change Too Large]`.
|
||||
@@ -0,0 +1,349 @@
|
||||
hypothesis_gen: # It is deprecated now, please refer to direct_exp_gen
|
||||
system: |-
|
||||
The user is working on generating new hypotheses for the {{ targets }} in a data-driven research and development process.
|
||||
The {{ targets }} are used in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
The user has already proposed several hypotheses and conducted evaluations. This information will be provided to you. Your task is to:
|
||||
1. Review the existing hypotheses and their evaluation results: Determine if any existing hypotheses are valid and worth pursuing further.
|
||||
2. Decide on the next step: Based on the results and reasoning, decide whether:
|
||||
- To propose a new direction, diverging from the current focus.
|
||||
- To refine and deepen the exploration of the current hypothesis or direction.
|
||||
3. If refining an existing hypothesis: Provide clear adjustments or additional details to enhance its focus.
|
||||
4. If proposing a new hypothesis: Ensure it is distinct and addresses any gaps or shortcomings in the current approach.
|
||||
|
||||
The current component to focus on is: {{ component }}.
|
||||
{% if hypothesis_specification %}
|
||||
To assist in hypothesis formulation, the user has provided additional information: {{ hypothesis_specification }}.
|
||||
Important: If the hypothesis_specification outlines specific next steps, ensure that you follow those instructions carefully.
|
||||
{% endif %}
|
||||
Please generate the output using the following format and specifications:
|
||||
{{ hypothesis_output_format }}
|
||||
|
||||
user: |-
|
||||
{% if exp_and_feedback_desc|length == 0 %}
|
||||
This is the first round of hypothesis generation. The user has not yet proposed any hypotheses for this scenario.
|
||||
{% else %}
|
||||
This is not the first round. The user has already proposed several hypotheses and conducted evaluations.
|
||||
|
||||
The previous hypotheses and their corresponding feedback are as follows (focus on the most recent hypothesis, its derived insights, and reasoning):
|
||||
{{ exp_and_feedback_desc }}
|
||||
{% endif %}
|
||||
|
||||
In addition, generate relevant reasoning and distilled knowledge keys.
|
||||
For these keys, especially the knowledge section, provide detailed context specific to the scenario to enhance domain understanding, rather than offering general knowledge.
|
||||
|
||||
hypothesis_model: # It is deprecated now, please refer to direct_exp_gen
|
||||
system: |-
|
||||
The user is working on generating new hypotheses for the {{ targets }} in a data-driven research and development process.
|
||||
The {{ targets }} are used in the following scenario:
|
||||
{{ scenario }}
|
||||
{% if model_enough %}
|
||||
There are sufficient models available ({{ model_info | length }} models). Your task is to choose one of the existing models for further tuning or optimization. Based on the model's information:
|
||||
{{ model_info }}
|
||||
Ensure the hypothesis is specific, actionable, and well-justified.
|
||||
{% else %}
|
||||
The number of available models is insufficient ({{ model_info | length }} models). Your task is to first decide whether to:
|
||||
- Tune an existing model: Select one of the current models for further tuning and improvement.
|
||||
- Add a new model: Introduce a new model to expand the hypothesis space.
|
||||
Based on the current model information:
|
||||
{{ model_info }}
|
||||
Make a decision and proceed accordingly:
|
||||
- If you decide to tune an existing model, select the most promising one and generate a new hypothesis.
|
||||
- If you decide to add a new model, specify the type of model you would add and generate a new hypothesis related to the new model.
|
||||
{% endif %}
|
||||
{% if hypothesis_specification %}
|
||||
To assist in hypothesis formulation, the user has provided additional information: {{ hypothesis_specification }}.
|
||||
Important: If the hypothesis_specification outlines specific next steps, ensure that you follow those instructions carefully.
|
||||
{% endif %}
|
||||
Please generate the output using the following format and specifications:
|
||||
{{ hypothesis_output_format }}
|
||||
|
||||
hypothesis_and_feedback: |-
|
||||
{% for experiment, feedback in hist %}
|
||||
Hypothesis {{ loop.index }}
|
||||
The experiment is design driven by hypothesis : {{ experiment.hypothesis }}
|
||||
Observation on the result with the hypothesis: {{ feedback.observations }}
|
||||
Feedback on the original hypothesis: {{ feedback.hypothesis_evaluation }}
|
||||
Did changing to this hypothesis work? (focus on the change): {{ feedback.decision }}
|
||||
{% endfor %}
|
||||
|
||||
task_gen:
|
||||
system: |-
|
||||
The user is trying to generate new {{ targets }} based on the hypothesis generated in the previous step.
|
||||
The {{ targets }} are used in certain scenario, the scenario is as follows:
|
||||
{{ scenario }}
|
||||
|
||||
{% if task_specification is not none %}
|
||||
The user has wrote some specification for the {{ targets }}. The specification is as follows:
|
||||
{{ task_specification }}
|
||||
Your task should adhere to the specification above.
|
||||
{% endif %}
|
||||
|
||||
{% if hypothesis is none %}
|
||||
Since we are at the very beginning stage, we plan to start with a very simple task. For example, the feature engineering can only implement the function that outputs the raw data without any transformation. The model component uses the most suitable type of model for the task, but a relatively basic version. The ensemble component only uses the simplest ensemble method. The main focus at this stage is to build the first runnable version of the solution.
|
||||
{% else %}
|
||||
The user will use the {{ targets }} generated to do some experiments. The user will provide this information to you:
|
||||
1. The target hypothesis you are targeting to generate {{ targets }} for.
|
||||
2. The hypothesis generated in the previous steps and their corresponding feedbacks.
|
||||
3. Former proposed {{ targets }} on similar hypothesis.
|
||||
4. Some additional information to help you generate new {{ targets }}.
|
||||
{% endif %}
|
||||
|
||||
Please generate the output following the format below:
|
||||
{{ task_output_format }}
|
||||
|
||||
user: |-
|
||||
{% if workspace_code %}
|
||||
Here is a list of all the filenames and their corresponding content in the workspace:
|
||||
{{workspace_code}}
|
||||
{% endif %}
|
||||
|
||||
{% if former_task_desc is not none %}
|
||||
The user has made several task on this scenario but didn't get the expected result due to wrong implementation or just bad luck. The former task is as follows:
|
||||
{{ former_task_desc }}
|
||||
Please avoid generating similar task to the former task to avoid the same mistake and boost efficiency.
|
||||
|
||||
{% if targets == "Model" %}
|
||||
Based on the feedback from previous experiment failures, if the failure was due to exceeding the time limit or memory constraints, start with the smallest model size or choose alternative algorithms or methods with significantly lower time or space complexity instead of using a neural network. You can then iteratively refine and optimize the model in later stages.
|
||||
{% endif %}
|
||||
|
||||
{% endif %}
|
||||
|
||||
{% if hypothesis is not none %}
|
||||
The user has made several hypothesis on this scenario and did several evaluation on them.
|
||||
The target hypothesis you are targeting to generate {{ targets }} for is as follows:
|
||||
{{ hypothesis }}
|
||||
The former hypothesis and the corresponding feedbacks are as follows:
|
||||
{{ exp_and_feedback_desc }}
|
||||
Please generate the new {{ targets }} based on the information above.
|
||||
{% else %}
|
||||
Please generate the new {{ targets }} task.
|
||||
{% endif %}
|
||||
|
||||
task_gen_model: # It is deprecated now, please refer to direct_exp_gen
|
||||
system: |-
|
||||
{% if hypothesis is not none %}
|
||||
The user is trying to generate new {{ targets }} based on the hypothesis generated in the previous step.
|
||||
{% else %}
|
||||
The user is trying to generate new {{ targets }} based on the information provided.
|
||||
{% endif %}
|
||||
The {{ targets }} are used in certain scenario, the scenario is as follows:
|
||||
{{ scenario }}
|
||||
|
||||
{% if hypothesis is not none %}
|
||||
The user will use the {{ targets }} generated to do some experiments. The user will provide this information to you:
|
||||
1. The target hypothesis you are targeting to generate {{ targets }} for.
|
||||
2. The hypothesis generated in the previous steps and their corresponding feedbacks.
|
||||
3. Former proposed {{ targets }} on similar hypothesis.
|
||||
4. Some additional information to help you generate new {{ targets }}.
|
||||
{% endif %}
|
||||
Please generate the output following the format below:
|
||||
{{ task_output_format }}
|
||||
|
||||
user: |-
|
||||
{% if hypothesis is not none %}
|
||||
The user has made several hypothesis on this scenario and did several evaluation on them.
|
||||
The target hypothesis you are targeting to generate {{ targets }} for is as follows:
|
||||
{{ hypothesis }}
|
||||
The former hypothesis and the corresponding feedbacks are as follows:
|
||||
{{ exp_and_feedback_desc }}
|
||||
Please generate the new {{ targets }} based on the information above.
|
||||
{% else %}
|
||||
Please generate the new {{ targets }} task.
|
||||
{% endif %}
|
||||
|
||||
direct_exp_gen:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
|
||||
The user is working on creating a solution for a Kaggle competition. Your task is to first suggest a hypothesis and then design a task to enhance the current best solution based on that hypothesis.
|
||||
|
||||
The component to focus on for the next hypothesis is already determined as: {{ component }}.
|
||||
It will be used in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
# Step1: Hypothesis Proposal
|
||||
The user has already proposed several hypotheses and conducted evaluations on them. This information will be provided to you later.
|
||||
|
||||
## Hypothesis Specification
|
||||
To assist you in formulating new hypotheses, the user has provided some additional information:
|
||||
{{ hypothesis_specification }}
|
||||
|
||||
## Guidelines
|
||||
Important: If the Hypothesis Specification outlines the next steps you need to follow, ensure you adhere to those instructions.
|
||||
|
||||
[Partial Response Format 1] Your generated output should contain key-value pairs adhering to the following format and specifications:
|
||||
{{ hypothesis_output_format }}
|
||||
Also generate the relevant keys for the reasoning and the distilled knowledge that follows. For those keys, in particular for knowledge, explain in the context of the specific scenario to build up domain knowledge in the specific field rather than general knowledge.
|
||||
|
||||
# Step2: Task Design
|
||||
|
||||
The user is trying to generate new {{ targets }} based on the hypothesis generated in the previous step.
|
||||
|
||||
## Task Specification
|
||||
The scope of the {{ targets }} can be described by a interface specification as follows:
|
||||
```markdown
|
||||
{{ task_specification }}
|
||||
```
|
||||
|
||||
## Guidelines
|
||||
The user will use the {{ targets }} generated to do some experiments. The user will provide this information to you:
|
||||
1. The target hypothesis you are targeting to generate {{ targets }} for.
|
||||
2. The hypothesis generated in the previous steps and their corresponding feedbacks.
|
||||
3. Former proposed {{ targets }} on similar hypothesis.
|
||||
4. Some additional information to help you generate new {{ targets }}.
|
||||
|
||||
[Partial Response Format 2] Your generated output should contain key-value pairs adhering to the following format and specifications:
|
||||
{{ task_output_format }}
|
||||
|
||||
{% if workflow_check %}
|
||||
# Step3: Workflow update
|
||||
Since components have dependencies, the workflow should be updated to reflect the changes made to the target component. Please also decide whether the workflow needs to be updated and provide a brief description of the change task.
|
||||
[Partial Response Format 3] Your generated workflow description should be a simple text and the following agent will do the implementation. If you think the workflow should not be updated, just respond with "No update needed".
|
||||
{% endif %}
|
||||
|
||||
Your response should contain two parts: the hypothesis proposal and the task design. Please follow the format and specifications provided below:
|
||||
{
|
||||
"hypothesis_proposal": [Partial Response Format 1],
|
||||
"task_design": [Partial Response Format 2],
|
||||
{% if workflow_check %}"workflow_update": [Partial Response Format 3], {% endif %}
|
||||
}
|
||||
|
||||
user: |-
|
||||
# All former experiments and their feedbacks
|
||||
{{ exp_and_feedback_list_desc }}
|
||||
|
||||
{% if targets == "Model" %}
|
||||
Based on the feedback from previous experiment failures, if the failure was due to exceeding the time limit or memory constraints, start with the smallest model size or choose alternative algorithms or methods with significantly lower time or space complexity instead of using a neural network. You can then iteratively refine and optimize the model in later stages.
|
||||
|
||||
Here is the SOTA solution:
|
||||
{{ sota_exp_desc }}
|
||||
Pay attention to the **Results** section. If there are sufficient models available and there is a model with a significantly worse score, consider removing that model. In this case, `model_name` in task_design should be the model you are going to remove (the name must be the same as the name in the model column in the **Results** section), and `description` should start with "Model removal".
|
||||
|
||||
Otherwise, if the number of available models is insufficient. Your task is to first decide whether to:
|
||||
a. Tune an existing model: Select one of the current models for further tuning and improvement.
|
||||
b. Add a new model: Introduce a new model to expand the hypothesis space.
|
||||
|
||||
The information of the model is described by the code of workspace.
|
||||
|
||||
Then, based on your decision, proceed with the corresponding actions accordingly:
|
||||
a. If you decide to tune an existing model, select the existing model file and generate a new hypothesis.
|
||||
b. If you decide to add a new model, specify the type of model you would add and generate a new hypothesis related to the new model.
|
||||
|
||||
When building the model, if the runtime permits, consider incorporating hyperparameter search methods to improve performance.
|
||||
{% endif %}
|
||||
|
||||
{% if last_exp_diff %}
|
||||
# Here are the differences between the latest version of implementation and the current best version of implementation
|
||||
It is presented in diff format, highlighting changes from the best version to the latest version.
|
||||
{{ last_exp_diff }}
|
||||
{% endif %}
|
||||
|
||||
component_gen:
|
||||
system: |-
|
||||
You are a Kaggle Grander Master. You are going to provide a solution for a kaggle competition.
|
||||
|
||||
# Here is the description of the competition scenario:
|
||||
{{ scenario }}
|
||||
|
||||
# Here is the current best version of implementation:
|
||||
{{ sota_exp_desc }}
|
||||
[Notice] Pay attention to the **Results** section. If there is a model with a significantly worse score, consider removing that model.
|
||||
|
||||
{% if last_exp_diff %}
|
||||
# Here are the differences between the latest version of implementation and the current best version of implementation
|
||||
It is presented in diff format, highlighting changes from the best version to the latest version.
|
||||
{{ last_exp_diff }}
|
||||
{% endif %}
|
||||
|
||||
You will be provided the feedback for the latest implementation.
|
||||
|
||||
Please select the component you are going to improve the sota implementation.
|
||||
# Here is the brief description of the components you can select:
|
||||
{{ component_desc }}
|
||||
|
||||
Please generate the output in JSON format following the format below:
|
||||
{% include "scenarios.data_science.proposal.exp_gen.prompts:output_format.component" %}
|
||||
|
||||
user: |-
|
||||
Here are the former experiments and their feedbacks:
|
||||
{{ exp_and_feedback_list_desc }}
|
||||
|
||||
Please choose the most proper component to focus on based on the information above. Please balance the exploration and exploitation.
|
||||
Avoid selecting the same component more than 5 times in a row to ensure that the chosen component is not overly repetitive.
|
||||
|
||||
exp_and_feedback: |-
|
||||
{% for experiment, feedback in trace.hist[-10:] %}
|
||||
## Experiment {{ loop.index }}
|
||||
Experiment are focusing on task: {{ experiment.pending_tasks_list[0][0] }}
|
||||
{% if experiment.hypothesis %}
|
||||
The experiment is design driven by hypothesis : {{ experiment.hypothesis }}
|
||||
Observation on the result with the hypothesis: {{ feedback.observations }}
|
||||
{% endif %}
|
||||
Feedback on the original hypothesis: {{ feedback.hypothesis_evaluation }}
|
||||
Did changing to this hypothesis work? (focus on the change): {{ feedback.decision }}
|
||||
{% endfor %}
|
||||
|
||||
hypothesis_specification: |-
|
||||
1. The hypothesis should be precise, testable, and directly actionable. Avoid general or vague statements. For example, "tuning a model" is too broad, whereas "increasing the learning rate to 0.1 in the LightGBM model will improve performance" is specific and actionable.
|
||||
2. Each hypothesis should focus on a single direction per experiment. Avoid proposing multiple possibilities within the same hypothesis, such as "this may work in case A or case B." Research and development can be approached at different levels (shallow or deep), but each experimental loop should validate only one specific idea.
|
||||
3. The hypothesis should based on current SOTA solution. The user will conduct experiments based on the SOTA solution to test whether the hypothesis improves performance in this specific competition.
|
||||
|
||||
output_format:
|
||||
component: |-
|
||||
{
|
||||
"reason": "The reason why you chose this component. Based on the current status and former trials, 1) why this component is the most promising one to focus on. 2) Why the component is the right place to apply your idea."
|
||||
"component": "The component you suggest to focus on. It must be one of ['DataLoadSpec', 'FeatureEng', 'Model', 'Ensemble', 'Workflow']."
|
||||
}
|
||||
hypothesis: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"component": "If "hypothesis_specification" provides the component you need to take, please follow "hypothesis_specification" to choose the component. Otherwise, based on previous experimental results, suggest the component you believe is most appropriate at the moment. It should be one of ["DataLoadSpec", "FeatureEng", "Model", "Ensemble", "Workflow"]",
|
||||
"hypothesis": "A concise, testable statement derived from previous experimental outcomes. Limit it to one or two sentences that clearly specify the expected change or improvement in the <component>'s performance.",
|
||||
"reason": "A brief explanation, also in one or two sentences, outlining the rationale behind the hypothesis. It should reference specific trends or failures from past experiments and explain how the proposed approach may address these issues.",
|
||||
"concise_reason": "Two-line summary. First line focuses on a concise justification for the change. Second line generalizes a knowledge statement.",
|
||||
"concise_observation": "One line summary. It focuses on the observation of the given scenario, data characteristics, or previous experiences (failures & success).",
|
||||
"concise_justification": "One line summary. Justify the hypothesis based on theoretical principles or initial assumptions.",
|
||||
"concise_knowledge": "One line summary. Transferable knowledge based on theoretical principles. Use conditional grammar. eg. "If...., ..; When..., .; and etc" Make sure that you state things clearly without ambiguity. Eg. avoid saying "previous hypothesis", because one wouldn't know what that is."
|
||||
}
|
||||
data_loader: |-
|
||||
Design a specific and detailed data loader task based on the given hypothesis. The output should be detailed enough to directly implement the corresponding code.
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"description": "A precise and comprehensive description of the overall data loader for the data science workflow",
|
||||
}
|
||||
feature: |-
|
||||
Design a specific and detailed feature engineering task based on the given hypothesis. The output should be detailed enough to directly implement the corresponding code.
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"description": "A precise and comprehensive description of feature engineering task",
|
||||
}
|
||||
model: |-
|
||||
Design a specific and detailed model task based on the given hypothesis. The output should be detailed enough to directly implement the corresponding code.
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"model_name": "model name, must start with 'model_' and only contain letters, numbers, and underscores",
|
||||
"description": "A precise and comprehensive description of the model. Start with [Model building/tuning] or [Model removal].",
|
||||
}
|
||||
ensemble: |-
|
||||
Design a specific and detailed ensemble task based on the given hypothesis. The output should be detailed enough to directly implement the corresponding code.
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"description": "A precise and comprehensive description of the ensemble",
|
||||
}
|
||||
workflow: |-
|
||||
Design a specific and detailed workflow task based on the given hypothesis. The output should be detailed enough to directly implement the corresponding code.
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"description": "A precise and comprehensive description of the main workflow script (`main.py`)",
|
||||
}
|
||||
pipeline: |-
|
||||
Design a specific and detailed Pipeline task based on the given hypothesis. The output should be detailed enough to directly implement the corresponding code.
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"description": "A detailed, step-by-step implementation guide for `main.py` that synthesizes planned modifications and code structure into a comprehensive coding plan. Must be formatted in Markdown with level-3 headings (###) organizing logical sections, key decision points, and implementation steps. Should provide sufficient detail covering implementation flow, algorithms, data handling, and key logic points for unambiguous developer execution.",
|
||||
"packages": ["package1", "package2", ...] # Optional, list of packages needed for the task. If no packages are needed, leave it empty.
|
||||
}
|
||||
@@ -0,0 +1,972 @@
|
||||
scenario_problem:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
The user is improving a Kaggle competition implementation iteratively. Each new iteration (trace) is typically a modification of the current overall State-of-the-Art (SOTA) solution. If a new trace's performance surpasses the current SOTA, it establishes a new SOTA. Otherwise, it is considered a failed experiment.
|
||||
|
||||
You will be provided with:
|
||||
1. A detailed competition scenario description;
|
||||
2. The overall current SOTA implementation and its associated feedback, which represents the best-performing experiment from the entire history provided up to this point.
|
||||
|
||||
Your task is to analyze the provided information (primarily the scenario and current SOTA, if available) and identify a concise list of **Key Challenges** or **Core Problems** relevant to achieving success in this competition and improving the target metric. Aim for **FEWER BUT BETTER** challenges (e.g., 2-3 critical challenges), focusing on the most impactful aspects that can be methodically addressed.
|
||||
|
||||
### Core Analysis Dimensions for Identifying Challenges
|
||||
- **Gap Identification**: (If successful past solutions or common winning strategies are known/inferred) Examine what implicitly addressed problems or unexploited avenues these successful approaches highlight. These gaps can represent current challenges.
|
||||
- **Domain-Implementation Coherence Check**: Identify instances where technical approaches might violate domain constraints, oversimplify complex relationships, or miss domain-specific nuances. These incoherencies are challenges.
|
||||
{% if plan.draft is false %}- **SOTA Alignment Analysis**: Systematically compare the current SOTA implementation against dataset properties and domain knowledge to identify discrepancies or areas representing core challenges to overcome for enhancement.
|
||||
{% else %}- **Scenario-First Focus**: Since SOTA implementation is available, the **primary identified challenge** should be foundational. It should focus on establishing a **reasonable baseline** that directly addresses the core task and evaluation metric. Avoid overly complex initial challenges.
|
||||
{% endif %}
|
||||
|
||||
{% if sibling_hypotheses is not none %}
|
||||
### Diversity To Your Siblings
|
||||
You are working on exploration traces in parallel with others. To maximize exploration efficiency, your identified problems **Must** be **diverse** from those being explored in other traces.
|
||||
Here are the problems and hypotheses from your siblings:
|
||||
{% for hyp in sibling_hypotheses %}
|
||||
=== Sibling {{ loop.index }} Hypothesis ===
|
||||
{{ hyp }}
|
||||
{% endfor %}
|
||||
Your generated problems **MUST** guide the agent towards different approaches, for example, different backbone models, different feature engineering methods, different ensemble strategies, different workflow optimizations, focus on efficiency etc. Avoid proposing challenges that would likely result in solutions similar to those listed above.
|
||||
{% endif %}
|
||||
|
||||
## Key Challenges / Core Problems
|
||||
You **MUST** categorize each identified challenge into one of the following two types. This categorization should be based on the primary driver or nature of the challenge:
|
||||
1. **Dataset-Driven Challenge**: Challenges primarily derived from addressing or leveraging inherent structural or statistical properties of the dataset (e.g., mitigating imbalance, managing high dimensionality, specific feature engineering needs for data types like text or time-series, handling missing data, transforming skewed distributions, accounting for collinearity or outliers).
|
||||
2. **Domain-Informed Challenge**: Challenges primarily derived from correctly applying actionable knowledge specific to the competition's domain. This includes the correct interpretation of data patterns based on domain context, domain-specific feature engineering, adhering to known domain constraints, or avoiding invalid assumptions that data analysis alone might not reveal.
|
||||
|
||||
### Specification for each Identified Challenge
|
||||
1. The challenge should be specific and fine-grained. Avoid general or vague statements.
|
||||
2. The challenge should be technical or methodological. Focus on design and implementation strategies that need to be solved, not simple runtime bugs (unless the bug points to a deeper architectural challenge or a persistent efficiency problem).
|
||||
3. The challenge must be strictly aligned with the improvement of the target metric.
|
||||
{% if plan.draft is true %}4. If no SOTA is available, at least one identified challenge must guide the creation of a baseline model that is feasible, potentially competitive, and able to run to completion.{% endif %}
|
||||
|
||||
|
||||
{% if problem_output_format is not none %}
|
||||
### Output Format
|
||||
{{ problem_output_format }}
|
||||
{% else %}
|
||||
Please response in json format.
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Current SOTA Implementation
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
feedback_problem:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
The user is improving a Kaggle competition implementation iteratively through traces. Each new trace is a modification of the State-of-the-Art (SOTA) implementation that was current at the time that trace was initiated. If a new trace's performance surpasses the SOTA it aimed to improve upon, it becomes the new SOTA. If not, it is considered a failed experiment.
|
||||
|
||||
You will be provided with:
|
||||
1. A detailed competition scenario description;
|
||||
2. A history of previous successfully experiments and their associated feedbacks, indexed or ordered from oldest to newest; the latest SOTA experiment accumulates all the improvements from the previous successful experiments.
|
||||
3. A history of previous failed experiments and their associated feedbacks, chronologically ordered, where each failed experiment did not surpass the SOTA that was current at the time of its execution. The failed experiments are based on the current SOTA implementation and are used to propose hypotheses for further performance improvements.
|
||||
4. The overall current SOTA implementation and its associated feedback, which represents the best-performing experiment from the entire history provided up to this point.
|
||||
|
||||
Your task is to analyze all this provided historical information and extract **Key Learnings and Unresolved Challenges** from the experiment history. These should guide concrete improvements in subsequent iterations.
|
||||
|
||||
## Key Learnings and Unresolved Challenges
|
||||
|
||||
{% if inject_diverse %}
|
||||
### Focus on Diversity!!
|
||||
Diversity is very critical in the analysis of scenario problems. You should closely check the history of previous experiments and feedbacks, and try to explore the problems/hypotheses that are not covered by the previous experiments.
|
||||
1. Check the previous experiments and feedbacks to find the problems that are not covered by the previous experiments.
|
||||
2. Check the current SOTA implementation and feedback to find the problems that are not covered by the current SOTA implementation.
|
||||
3. Do not do incremental exploration on the previous problems.
|
||||
{% endif %}
|
||||
|
||||
### Definition
|
||||
Key Learnings and Unresolved Challenges are specific, fine-grained technical or methodological observations, persistent issues, or patterns identified within previous experiments or the current SOTA implementation. These are primarily derived from explicit feedback, code analysis, or patterns in the trace history, and should highlight problems that need solving or learnings that should inform future hypotheses.
|
||||
|
||||
### Guidelines for Identification
|
||||
Here are guidelines to help you identify these Learnings and Challenges:
|
||||
|
||||
1. **Feedback Analysis**:
|
||||
- **Explicit Issues/Suggestions as Challenges**: Extract critical issues, errors (especially those pointing to deeper problems like resource limits or incorrect submission formats if not easily fixed), or direct suggestions from feedback that represent unresolved problems.
|
||||
- **Implicit Gaps as Challenges**: Infer unaddressed points, shortcomings, or areas for improvement implied by feedback that constitute ongoing challenges.
|
||||
- **Time/Memory Constraints as Critical Challenges**: If previous experiments indicate failures due to time/memory limitations, or inefficient resource usage, this **MUST** be listed as a critical challenge. This includes identifying if the current SOTA or failed experiments are too complex for the given time limits.
|
||||
|
||||
2. **Implementation Review (of SOTA or relevant past experiments)**:
|
||||
- **Suboptimal Design as Challenges**: Identify potentially suboptimal feature selection, model architecture, hyperparameters, ensemble strategy, training/validation processes that appear as recurring problems or limit performance, framing them as challenges to be addressed.
|
||||
- **Common Implementation Issues**: Note the coding issues that are blocking for receiving a reasonable result. For example, the submission format was repeatedly incorrect despite attempts to fix it, this is an unresolved challenge related to the implementation.
|
||||
|
||||
3. **Trace History Analysis (Trends & Patterns as Challenges)**:
|
||||
- **Persistent Issues/Errors as Challenges**: Flag unresolved negative patterns, errors (e.g., recurrent `zipfile.BadZipFile`, CUDA label errors, submission format mismatches if they persist after attempts to fix), or suboptimal outcomes that recur across multiple experiment traces. These represent core unresolved challenges.
|
||||
- **Ineffective/Partial Fixes**: Highlight if previous changes intended to solve a problem were only partially successful or ineffective, meaning the core challenge remains.
|
||||
- **Unexplored Promising Directions**: Identify potentially valuable approaches (e.g., alternative feature sets, different model families, advanced optimization techniques) that were hinted at by feedback, briefly tried without full exploration, or represent logical next steps given the trajectory of past experiments.
|
||||
- **Constraint Violations/Inefficiencies as Challenges**: Explicitly note any unaddressed time or memory constraint violations or significant computational inefficiencies as critical challenges that need strategic solutions.
|
||||
|
||||
### Specification for each Learning/Challenge
|
||||
1. The Learning/Challenge must be specific, actionable, and evidence-based (tied to feedback, code, or trace history).
|
||||
2. It should focus on technical or methodological problems that need solving.
|
||||
3. Clearly state the learning or articulate the challenge.
|
||||
4. Addressing the challenge or applying the learning should have a plausible positive impact on the target metric or successful execution.
|
||||
5. The challenge must be strictly aligned with the improvement of the target metric.
|
||||
|
||||
{% if sibling_hypotheses is not none %}
|
||||
### Diversity To Your Siblings
|
||||
You are working on exploration traces in parallel with others. To maximize exploration efficiency, your identified problems **Must** be **diverse** from those being explored in other traces.
|
||||
Here are the problems and hypotheses from your siblings:
|
||||
{% for hyp in sibling_hypotheses %}
|
||||
=== Sibling {{ loop.index }} Hypothesis ===
|
||||
{{ hyp }}
|
||||
{% endfor %}
|
||||
Your generated problems **MUST** guide the agent towards different approaches, for example, different backbone models, different feature engineering methods, different ensemble strategies, different workflow optimizations, focus on efficiency etc. Avoid proposing challenges that would likely result in solutions similar to those listed above.
|
||||
{% endif %}
|
||||
|
||||
{% if problem_output_format is not none %}
|
||||
### Output Format
|
||||
{{ problem_output_format }}
|
||||
{% else %}
|
||||
Please response in json format.
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Previous Experiments and Feedbacks
|
||||
{{ exp_and_feedback_list_desc }}
|
||||
|
||||
# Current SOTA Implementation
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
hypothesis_gen:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
The user is iteratively improving a Kaggle competition implementation. Each new iteration (trace) is a modification of the current State-of-the-Art (SOTA). If a new trace surpasses the current SOTA, it becomes the new SOTA. Otherwise, it's a failed experiment.
|
||||
You will be provided with:
|
||||
1. A detailed competition scenario description.
|
||||
2. A history of previous successfully experiments and their associated feedbacks, indexed or ordered from oldest to newest; the latest SOTA experiment accumulates all the improvements from the previous successful experiments.
|
||||
3. A history of previous failed experiments and their associated feedbacks, chronologically ordered, where each failed experiment did not surpass the SOTA that was current at the time of its execution. The failed experiments are based on the current SOTA implementation and are used to propose hypotheses for further performance improvements.
|
||||
4. The current SOTA implementation and feedback (the latest successful experiment).
|
||||
5. A list of identified **Challenges** from history), which we will refer to as "Identified Challenges" below.
|
||||
|
||||
Your task is to perform two main steps:
|
||||
1. **Hypothesis Proposal**: For each relevant Identified Challenge, propose one specific, testable hypothesis.
|
||||
2. **Hypothesis Evaluation**: Evaluate each proposed hypothesis across multiple dimensions.
|
||||
|
||||
{% if enable_idea_pool %}
|
||||
To help you propose hypotheses, the user may provide a list of ideas for each Identified Challenge. These ideas are methods or techniques from successful SOTA implementations in other competitions.
|
||||
Evaluate these ideas: they might help address the Identified Challenges and improve the current SOTA. You must decide whether to use them. If you adapt a provided idea for a specific Challenge into your hypothesis, ensure you clearly state this by setting the 'inspired' flag to True for that hypothesis.
|
||||
{% endif %}
|
||||
|
||||
# Task 1: Hypothesis Proposal
|
||||
First note that the user might provide a list of challenges containing duplicates. You should only propose one hypothesis for each unique challenge. If a challenge is a duplicate of a previous one, you can skip it.
|
||||
For each Identified Challenge, propose one hypothesis corresponding to the Challenge, aimed at improving the current SOTA implementation or establishing a robust initial SOTA.
|
||||
|
||||
## 1.1. Steps to Hypothesize
|
||||
Follow these steps to formulate effective hypotheses:
|
||||
|
||||
1. **Understanding the Challenge**:
|
||||
- Analyze the Identified Challenge to understand its root cause and potential impact on the competition's target metric or successful execution.
|
||||
- If the Challenge stems from past experiments (SOTA or failed), review the specifics of those experiments to ensure the proposed hypothesis offers a novel, more effective, or correctly implemented solution.
|
||||
- If the Challenge relates to persistent problems from failed experiments (e.g., experiments consistently failed due to time/memory constraints, or recurrent errors like incorrect data loading or submission formats), your hypothesis MUST propose a direct and robust tentative solution.
|
||||
{% if plan.draft is true %}
|
||||
2. **Drafting the First Implementation (if no SOTA exists)**:
|
||||
- If there is no SOTA implementation yet (i.e., you are drafting the first implementation based on a foundational Challenge identified in the previous step), your primary hypothesis should focus on developing a baseline model that directly addresses the foundational Challenge and can run to completion reliably.
|
||||
- This initial hypothesis should define the core data processing, feature engineering, model choice, and submission generation steps in a clear and executable way. Avoid introducing unnecessary complexity in the first version, but you are not restricted to overly simple models—a reasonable, competitive baseline is acceptable as long as it is likely to run reliably.
|
||||
{% endif %}
|
||||
{% if plan.draft is true %}3{% else %}2{% endif %}. **Actionable Changes**:
|
||||
- If a Challenge involves underperforming models (e.g., in an ensemble), propose specific actions like removing or replacing those models.
|
||||
- If a Challenge relates to hyperparameter tuning, recommend a specific method or strategy (e.g., "Use Optuna to perform hyperparameter tuning on the LightGBM model to address the 'suboptimal hyperparameter' challenge").
|
||||
- If a Challenge points to data loading, preprocessing, or submission format errors, the hypothesis must detail the exact changes required to rectify these issues.
|
||||
{% if enable_idea_pool %}
|
||||
4. **Idea Reference**: Provided ideas are methods, techniques, or tricks from high-performing implementations in other competitions addressing similar problems. Use them as inspiration if you find them suitable for the current Challenge.
|
||||
{% endif %}
|
||||
|
||||
## 1.2. Guidelines for Writing Hypotheses
|
||||
|
||||
1. **Be Specific and Decisive**:
|
||||
- Clearly state the exact, unambiguous change(s) being proposed. Avoid vague goals like "improve the model" or "optimize the pipeline."
|
||||
- The hypothesis must propose a single, clear course of action. Do not suggest alternatives (e.g., "try method A or method B").
|
||||
- The hypothesis statement must be direct and definitive, without phrases like "for example," "e.g.," "might involve," "consider," "try," or "explore."
|
||||
- The hypothesis must be more informative and decisive than the Challenge it addresses. It should not simply restate the Challenge or suggest a general approach without specifics.
|
||||
2. **Ensure Testability and Actionability**:
|
||||
- The hypothesis must describe an action or change that can be practically implemented and tested.
|
||||
- If the hypothesis is about improving SOTA, it should clearly state the expected improvement, typically related to a measurable performance metric or successful execution.
|
||||
- If the hypothesis is about establishing the first solution, it should clearly outline the expected outcome -- RUNNABILITY and CORRECTNESS. Prioritize getting a valid submission out, even with a very basic model or pipeline.
|
||||
3. **Align with Current SOTA and Identified Challenges**:
|
||||
- The hypothesis must be directly relevant to improving the *current* State-of-the-Art (SOTA) implementation or establishing a new SOTA if none exists.
|
||||
- It must directly address one of the `Identified Challenges` provided as input.
|
||||
4. **Maintain Singular Focus within Hypothesis**:
|
||||
- If a hypothesis involves multiple adjustments, these must be tightly correlated and contribute to a single, unified conceptual change addressing the core of the Identified Challenge.
|
||||
- Avoid bundling multiple independent or unrelated ideas into a single hypothesis. Each hypothesis should test one core concept.
|
||||
5. **Address the Overall Pipeline (for Pipeline-Focused Tasks)**:
|
||||
- The hypothesis should address improvements to the end-to-end pipeline.
|
||||
- It can propose coordinated changes across multiple parts of the SOTA implementation if these are necessary to achieve a significant pipeline-level improvement to address the Challenge. (Note: Even for pipeline-focused hypotheses, you will still select the single *most relevant* primary component tag during the evaluation task.)
|
||||
|
||||
{% if former_user_instructions_str is not none %}
|
||||
## 1.3. Mandatory Consideration of Past User Instructions
|
||||
The user has provided specific instructions in previous experiments. These instructions may contain critical insights or constraints that must be considered when formulating your hypotheses. Carefully review the following past user instructions and ensure that your proposed hypotheses align with these directives:
|
||||
{{ former_user_instructions_str }}
|
||||
{% endif %}
|
||||
|
||||
# Task 2: Hypothesis Evaluation
|
||||
After proposing one hypothesis for each relevant Identified Challenge, evaluate each one.
|
||||
|
||||
## 2.1. Evaluation Instruction
|
||||
For each individual hypothesis you proposed in Task 1, perform the following two evaluation steps:
|
||||
|
||||
1. **Assign a Component Tag:** Assign a single component tag to the hypothesis. Choose the **single most relevant** tag from the official list below, even if the hypothesis appears to touch upon multiple areas. Use the following detailed descriptions to understand the scope and boundaries of each component.
|
||||
|
||||
- **`DataLoadSpec`**: Responsible for loading raw competition data, ensuring data is converted to the correct types, and potentially providing an initial exploratory data analysis (EDA) summary. (e.g., fixing `zipfile.BadZipFile` by improving loading logic).
|
||||
- **`FeatureEng`**: Focuses on transforming raw data into meaningful features suitable for model consumption. Key responsibilities include maintaining data shape consistency, preventing data leakage during feature creation, and optimizing features for model performance. Feature engineering should be model-agnostic.
|
||||
- **`Model`**: Involves model building (developing new models to address the problem), model tuning (optimizing existing models for better performance), or model removal. This component also handles data operations or augmentations closely tied to a specific model framework (e.g., PyTorch `Datasets` & `DataLoaders`, TensorFlow `tf.data`, or fixing CUDA label errors by ensuring correct label mapping before loss calculation).
|
||||
- **`Ensemble`**: Combines predictions from multiple models using various ensemble strategies.
|
||||
- **`Workflow`**: Integrates all pipeline components, orchestrating the flow from data loading through to final output generation (e.g., correcting `submission.csv` column names or structure, managing overall pipeline execution logic for efficiency).
|
||||
|
||||
2. **Score the Hypothesis:** For each hypothesis, provide a score from 1 (lowest/worst) to 10 (highest/best) on each of the following five dimensions. Base your scores on all provided information.
|
||||
- **Challenge-Hypothesis Alignment (Score: 1-10):** How directly and effectively does the hypothesis address the core issues of the `Identified Challenge` it targets? A higher score means a stronger, more direct alignment.
|
||||
- **Expected Impact (Score: 1-10):** What is the estimated magnitude of improvement (e.g., in the primary competition metric, efficiency, robustness, or successful execution) if this hypothesis is successfully implemented? Higher scores for greater positive impact.
|
||||
- **Novelty (Score: 1-10):** How innovative or original is this hypothesis when compared to the approaches and ideas evident in the `previous SOTA experiments` and `previous failed experiments`? Assign a score of 1 if the hypothesis is a repeat or substantially similar to a previously attempted hypothesis (whether successful or failed), UNLESS the previous attempt clearly failed due to a trivial implementation bug and the current hypothesis proposes the correct implementation of the same core idea.
|
||||
- **Feasibility (Score: 1-10):** How easily and practically can this hypothesis be implemented and *run to completion* within the existing SOTA codebase and operational constraints (e.g., allowed time for training/inference, available compute resources, overall complexity)? Higher scores for easier implementation and higher likelihood of successful execution.
|
||||
- **Risk-Reward Balance (Score: 1-10):** Considering the potential for significant improvement (reward) versus the probability of failure, negative side-effects, or excessive resource consumption (risk), how optimal is this balance? A high score indicates a favorable balance.
|
||||
- **Prioritization for Critical Challenges:** If a hypothesis directly and credibly addresses a **critical Challenge that caused prior experiment failures** (e.g., timeout, persistent data loading errors, incorrect submission format preventing any score), its **Expected Impact** and **Risk-Reward Balance** should generally be scored highly (e.g., 8-10), and **Feasibility** should also be high if the proposed solution is indeed simpler, more direct, or more efficient. This ensures such critical hypotheses are prioritized.
|
||||
{%if enable_simple_hypothesis%}
|
||||
3. Please generate 3 hypotheses, as concise as possible, no more than 2 sentences each.
|
||||
{% endif %}
|
||||
{%if generate_unique_hypothesis %}
|
||||
We are now at the beginning stage. Please generate hypotheses that are as unique as possible.
|
||||
Each hypothesis should handle a different component. For example, you can generate four distinct hypotheses for:
|
||||
- DataLoadSpec
|
||||
- FeatureEng
|
||||
- Model
|
||||
- Workflow
|
||||
The goal is for these components together to form a complete code solution. Avoid generating complex ensemble methods (e.g., 5-fold CV or stacked models) at this stage.
|
||||
Special requirements for Hypotheses:
|
||||
- They must be extremely simple, trivial, and easy to implement — something that can be tested quickly with minimal code changes.
|
||||
- Avoid "trick-like" operations, such as freezing layers in the model.
|
||||
- For **DataLoadSpec**:
|
||||
- Especially in Computer Vision(CV) competitions where datasets are often very large, carefully analyze the dataset size. If the dataset is too large, propose sampling a reasonable subset for quick experiments.
|
||||
- For **audio competitions**, consider first converting the audio data into images (e.g., spectrograms) and then applying CV-based methods for modeling.
|
||||
{% endif %}
|
||||
|
||||
{% if sibling_hypotheses is not none %}
|
||||
### Diversity To Your Siblings
|
||||
You are working on exploration traces in parallel with others. To maximize exploration efficiency, your proposed hypotheses **Must** be **diverse** from those being explored in other traces.
|
||||
Here are the problems and hypotheses from your siblings:
|
||||
{% for hyp in sibling_hypotheses %}
|
||||
=== Sibling {{ loop.index }} Hypothesis ===
|
||||
{{ hyp }}
|
||||
{% endfor %}
|
||||
Your generated hypotheses **MUST** guide the agent towards different approaches, for example, different backbone models, different feature engineering methods, different ensemble strategies, different workflow optimizations, focus on efficiency etc. Avoid proposing hypotheses that are similar to those listed above.
|
||||
{% endif %}
|
||||
|
||||
{% if inject_diverse %}
|
||||
# Focus on Diversity!!
|
||||
Diversity is very critical in the analysis of scenario problems. You should closely check the history of previous experiments and feedbacks, and try to explore the problems/hypotheses that are not covered by the previous experiments.
|
||||
1. Check the previous experiments and feedbacks to find the problems that are not covered by the previous experiments.
|
||||
2. Check the current SOTA implementation and feedback to find the problems that are not covered by the current SOTA implementation.
|
||||
3. Think out of the box and explore the hypothesis that are not covered by the previous experiments and feedbacks, but are reasonable and aligned with the identified problems.
|
||||
4. Do not do incremental exploration on the previous problems, like lightgbm -> xgboost, or 1dCNN -> 2dCNN. Totally different hypothesis on model\data\feature\ensemble\workflow level are welcomed.
|
||||
{% endif %}
|
||||
|
||||
{% if plan.suggest_model_architecture is true %}
|
||||
## Current focus: Find the best model architecture!
|
||||
The user has chose to focus on finding the best model architecture so far. This means if no problems are critical, you should focus on proposing a hypothesis that suggests a new model architecture or a significant change to the existing model architecture. This is the primary focus of the current iteration.
|
||||
If the problem contains a critical challenge, you should still propose a hypothesis that addresses the critical challenge.
|
||||
{% elif plan.suggest_ensemble is true %}
|
||||
## Current focus: Try to find the best ensemble strategy!
|
||||
The user has chose to focus on finding the best ensemble strategy so far. This means if no problems are critical, you should focus on proposing a hypothesis that suggests a new ensemble strategy or try to increase the cross validation folds or the number of models in the ensemble. This is the primary focus of the current iteration.
|
||||
Some scenarios like computer vision tasks may not typically use ensemble strategies, so you can ignore this focus if it does not apply.
|
||||
If the problem contains a critical challenge, you should still propose a hypothesis that addresses the critical challenge.
|
||||
{% endif %}
|
||||
|
||||
{% if hypothesis_output_format is not none %}
|
||||
## Final Output Format in JSON Schema:
|
||||
{{ hypothesis_output_format }}
|
||||
{% else %}
|
||||
Please response in json format.
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Previous Experiments and Feedbacks
|
||||
{{ exp_and_feedback_list_desc }}
|
||||
|
||||
# Current SOTA Implementation
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
# Identified Challenges{% if enable_idea_pool %} with Sampled Ideas{% endif %}
|
||||
{{ problems }}
|
||||
|
||||
{% if knowledge %}
|
||||
# Some reference knowledge from the community
|
||||
{{ knowledge }}
|
||||
{% endif %}
|
||||
|
||||
hypothesis_critique:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
You are an expert critic evaluating machine learning hypotheses for Kaggle competition improvement.
|
||||
|
||||
For each hypothesis, provide a focused critique that identifies key issues and suggests improvements while preserving the experimental nature of hypotheses.
|
||||
|
||||
## Three Core Evaluation Areas:
|
||||
|
||||
### 1. Feasibility Assessment
|
||||
- **Technical Risk**: Major implementation challenges or resource constraints that could cause failure
|
||||
- **Integration Issues**: Conflicts with existing code or pipeline components
|
||||
- **Constraint Violations**: Whether this respects competition time/memory limits based on historical patterns
|
||||
|
||||
### 2. Alignment Check
|
||||
- **Problem-Solution Fit**: Does this actually address the root cause of the identified challenge?
|
||||
- **Metric Impact**: Will this meaningfully improve the competition's evaluation metric?
|
||||
- **Historical Context**: Has similar approaches been tried? Key learnings from past attempts?
|
||||
- **Innovation vs History Balance**: Distinguish between implementation failures (worth retrying with improvements) vs fundamental approach failures (multiple attempts failed due to core unsuitability - should avoid)
|
||||
|
||||
### 3. Improvement Direction
|
||||
- **Clarity Issues**: If vague, identify specific methods or strategies that address the core problem
|
||||
- **Alternative Strategies**: If implementation is problematic, identify concrete alternative approaches within the current framework such as switching from simple to weighted ensemble
|
||||
- **Risk Mitigation**: Recommend specific validation strategies or safeguards for high-risk aspects
|
||||
- **Competition Context**: This is a Kaggle competition where strong performance may come from novel approaches, but also from incremental improvements and careful optimization. Balance innovation with practical enhancements.
|
||||
|
||||
## CRITICAL Guidance Rules
|
||||
|
||||
- Be specific about methods and strategies, but avoid over-specifying implementation parameters. Suggest clear approaches like "use weighted ensemble instead of simple averaging" rather than exact values like "set weights=[0.3, 0.7]".
|
||||
- Focus on suggesting CLEAR METHODS and APPROACHES that lead to decisive hypotheses.
|
||||
- Avoid Overfitting to History: Learn from past failures but don't over-constrain innovation. Distinguish between implementation failures (worth retrying with improvements) and fundamental approach failures (should be avoided).
|
||||
|
||||
### Examples:
|
||||
|
||||
**Good Critiques:**
|
||||
- "The hypothesis lacks specificity about which ensemble method to use. Consider weighted averaging based on validation performance rather than simple averaging, given the model performance disparities."
|
||||
- "This hypothesis proposes LSTM for tabular data. History shows 3 consecutive failures with different LSTM implementations, and tabular data lacks sequential structure. Consider graph-based approaches instead to capture feature relationships."
|
||||
|
||||
**Poor Critiques:**
|
||||
- "Set max_depth=10, learning_rate=0.05, and use 500 trees." (too specific)
|
||||
- "This might not work." (too vague)
|
||||
- "LSTM is innovative, let's try again with different hyperparameters." (ignores fundamental mismatch)
|
||||
|
||||
{% if critique_output_format is not none %}
|
||||
## Output Format
|
||||
{{ critique_output_format }}
|
||||
{% else %}
|
||||
Please response in json format.
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Previous Experiments and Feedbacks
|
||||
{{ exp_and_feedback_list_desc }}
|
||||
|
||||
# Current SOTA Implementation
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
# Hypotheses to Critique
|
||||
{{ hypotheses_formatted }}
|
||||
|
||||
hypothesis_rewrite:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
You are an expert hypothesis rewriter specializing in iterative improvement of machine learning solutions for Kaggle competitions.
|
||||
|
||||
## Task
|
||||
Transform each **original hypothesis and its critique** into a **single, specific, testable technical hypothesis** that can be implemented immediately.
|
||||
|
||||
## Core Principles
|
||||
1. **Actionable Critique** – Apply insights from the critique, but the final text must stand alone with **no meta‑discussion** of the critique itself.
|
||||
2. **Standalone Justification** – Ground every technical decision in dataset characteristics, available compute budget, and competition constraints.
|
||||
3. **Decisive Specificity** – Remove all ambiguity; propose one clear action.
|
||||
4. **Innovation Preservation** – Maintain the innovative core of the original hypothesis while addressing implementation concerns. Avoid reverting to conventional approaches unless absolutely necessary.
|
||||
5. **CRITICAL - Avoid Overfitting to Critique** – Apply critique insights thoughtfully without over-constraining innovation. Balance addressing identified issues with preserving the exploratory value of bold ideas.
|
||||
{% if enable_scale_check %}6. The user is currently working on a continuous exploration on the task. It's typical that we first try in small scale and in some certain point we will scale up the solution.
|
||||
The user will tell you how much time have they spent on the task so far and all the former trials. You should consider whether to scale up the solution based on the current situation. You should put this conclusion in each hypothesis's appendix section.
|
||||
Typical scaling method includes:
|
||||
- Increasing the model architecture complexity.
|
||||
- Increasing the number of models to ensemble.
|
||||
- Increasing the number of features.
|
||||
- Increasing the number of cross validation folds.
|
||||
- Increasing the number of epochs for training.
|
||||
- Increasing the batch size for training.
|
||||
In the beginning stage, you should instruct to build low scale solutions which avoid the upper methods. After sufficient exploration iterations to approach the end of the time limit, you can suggest to scale up the solution in your response.
|
||||
Scaling is no connection to the debugging process. It's related to the whole solution's complexity. Please include this in every hypothesis you rewrite.
|
||||
{% endif %}
|
||||
|
||||
## Guidelines for Writing Rewritten Hypotheses
|
||||
|
||||
1. **Critique-Informed Specificity**:
|
||||
- Address technical gaps identified in the critique and replace vague terms with specific algorithms, methods, or parameters.
|
||||
- Transform general suggestions from the critique into concrete, implementable actions.
|
||||
- If the critique highlighted feasibility issues, propose alternative approaches that maintain the hypothesis's core intent while being more practical.
|
||||
- The rewritten hypothesis must be more specific than the original, incorporating the critique's guidance without explicitly referencing it.
|
||||
|
||||
2. **Standalone Technical Justification**:
|
||||
- Ground every technical decision in observable dataset characteristics (e.g., data size, feature types, class distribution).
|
||||
- Reference competition constraints (time limits, evaluation metrics, submission format) to justify approach choices.
|
||||
- Ensure the hypothesis can be understood and implemented without needing to read the original hypothesis or critique.
|
||||
- Include rationale for why the specific method/algorithm chosen is suitable for the current scenario.
|
||||
|
||||
3. **Enhanced Actionability and Precision**:
|
||||
- Replace any remaining ambiguity with decisive technical choices (e.g., "ensemble method" → "weighted averaging based on validation performance").
|
||||
- Specify validation strategies that will confirm the hypothesis's effectiveness.
|
||||
- Define clear success criteria or expected outcomes that can be measured.
|
||||
- If the original hypothesis bundled multiple ideas, focus on the most impactful one identified through the critique.
|
||||
|
||||
4. **Risk Mitigation and Implementation Clarity**:
|
||||
- If the critique identified implementation risks, incorporate specific mitigation strategies into the rewritten hypothesis.
|
||||
- Address resource constraint concerns by proposing efficient alternatives or optimizations.
|
||||
- Ensure the hypothesis addresses root causes rather than symptoms, as guided by the critique analysis.
|
||||
- Make the hypothesis robust against common failure modes identified in the critique.
|
||||
|
||||
5. **Pipeline Integration and Component Focus**:
|
||||
- Clearly specify how the proposed changes integrate with existing SOTA components.
|
||||
- Maintain focus on the primary component while ensuring compatibility with the overall pipeline.
|
||||
- If the critique suggested coordination across multiple components, organize these as a unified technical approach rather than separate changes.
|
||||
- Ensure the rewritten hypothesis preserves successful aspects of the current SOTA while addressing identified weaknesses.
|
||||
|
||||
6. **Innovation and Historical Learning**:
|
||||
- Apply critique insights to enhance sound innovative ideas while avoiding repeated fundamental failures identified in the analysis.
|
||||
- **Competition Context**: This is a Kaggle competition where strong performance may come from novel approaches or incremental improvements. Enhance both innovative ideas and practical optimizations based on the critique analysis.
|
||||
|
||||
{% if sibling_hypotheses is not none %}
|
||||
### Diversity To Your Siblings
|
||||
You are working on exploration traces in parallel with others. To maximize exploration efficiency, your rewritten hypotheses **Must** be **diverse** from those being explored in other traces.
|
||||
Here are the problems and hypotheses from your siblings:
|
||||
{% for hyp in sibling_hypotheses %}
|
||||
=== Sibling {{ loop.index }} Hypothesis ===
|
||||
{{ hyp }}
|
||||
{% endfor %}
|
||||
Your rewritten hypotheses **MUST** guide the agent towards different approaches, for example, different backbone models, different feature engineering methods, different ensemble strategies, different workflow optimizations, focus on efficiency etc. Avoid proposing hypotheses that are similar to those listed above.
|
||||
{% endif %}
|
||||
|
||||
{% if former_user_instructions_str is not none %}
|
||||
# Mandatory Consideration of Past User Instructions
|
||||
The user has provided specific instructions in previous experiments. These instructions may contain critical insights or constraints that must be considered when rewriting your hypotheses. Carefully review the following past user instructions and ensure that your rewritten hypotheses align with these directives:
|
||||
{{ former_user_instructions_str }}
|
||||
{% endif %}
|
||||
|
||||
{% if rewrite_output_format is not none %}
|
||||
## Output Format
|
||||
{{ rewrite_output_format }}
|
||||
{% else %}
|
||||
Please response in json format.
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Previous Experiments and Feedbacks
|
||||
{{ exp_and_feedback_list_desc }}
|
||||
|
||||
# Current SOTA Implementation
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
# Original Hypotheses and Their Critiques
|
||||
{{ hypothesis_critique_pairs }}
|
||||
|
||||
{% if time_status is not none %}
|
||||
# Time Status
|
||||
{{ time_status }}
|
||||
{% endif %}
|
||||
|
||||
|
||||
hypothesis_select:
|
||||
system: |-
|
||||
You are a Kaggle Grandmaster with deep expertise in model evaluation and decision making.
|
||||
Your task: Return the most appropriate hypothesis to improve the current solution in this experiment.
|
||||
## Hypothesis Source
|
||||
hypothesis_candidates are the hypotheses proposed in the current experiment. Please give them priority:
|
||||
{{hypothesis_candidates}}
|
||||
|
||||
{%if sota_flag %}
|
||||
SOTA score: {{current_sota_score}}
|
||||
{% if current_sota_score_in_current_trace == -1 %}
|
||||
Current SOTA score in this experiment: None.
|
||||
{% else %}
|
||||
Current SOTA score in this experiment: {{ current_sota_score_in_current_trace }}
|
||||
{% endif %}
|
||||
|
||||
{% if selected_extra_hypo_l and selected_extra_hypo_l|length > 0 %}
|
||||
The following are additional hypotheses that have been approved by other experiments.
|
||||
If any of these hypotheses have a SOTA score significantly higher than the current SOTA score in this experiment, you may want to prioritize considering them:
|
||||
Additional hypotheses (may include those corresponding to the SOTA score):
|
||||
{% for item in selected_extra_hypo_l %}
|
||||
{{ loop.index }}. {{ item[0] }} (score: {{ "%.3f"|format(item[1]) }})
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% else %}
|
||||
|
||||
{% if current_sota_score_in_current_trace == -1 %}
|
||||
{% if selected_extra_hypo_l and selected_extra_hypo_l|length > 0 %}
|
||||
The current SOTA score in this experiment is unavailable. Carefully examine the portion of the hypothesis associated with the SOTA score and incorporate any insights it provides.
|
||||
The following are additional hypotheses that have been approved by other experiments.
|
||||
You can also serve as references and are part of the Hypothesis Source to help you quickly reach or surpass the SOTA score:
|
||||
Additional hypotheses (may include those corresponding to the SOTA score):
|
||||
{% for item in selected_extra_hypo_l %}
|
||||
{{ loop.index }}. {{ item[0] }} (score: {{ "%.3f"|format(item[1]) }})
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
{%endif %}
|
||||
|
||||
|
||||
- The list `hypothesis_candidates` is for REFERENCE ONLY.
|
||||
- You may:
|
||||
1. Select one hypothesis directly from the candidates.
|
||||
2. Modify an existing hypothesis from the candidates.
|
||||
3. Create a new hypothesis, considering the current stage, by integrating advantages from multiple candidates or from historical hypotheses.
|
||||
|
||||
## Hypothesis Generation Guidelines
|
||||
## Stage Constraints
|
||||
|
||||
{% if use_ratio < 10 %}
|
||||
### Stage = Draft
|
||||
- This stage is focused on rapid, easy-to-implement hypotheses. Performance gains can be modest, but the code must be simple and safe to integrate.
|
||||
- You may take one of three actions:
|
||||
|
||||
1. **Select one hypothesis directly from the candidates**
|
||||
- Ideal for: *Simple, quick-to-implement hypotheses — minimal code changes, modest gains acceptable.*
|
||||
- Guidance: Pick an existing hypothesis that addresses the current bottleneck or potential improvement without modification. This is the fastest way to produce working code.
|
||||
|
||||
2. **Modify an existing hypothesis from the candidates**
|
||||
- Ideal for: *Focus on small, targeted tweaks such as loss function, learning rate schedule, light data augmentation, or minor architecture adjustments.*
|
||||
- Guidance: Make small adjustments to an existing hypothesis to better fit the current code or dataset. Examples include:
|
||||
- Tuning hyperparameters (including learning rate, batch size, and number of epochs)
|
||||
- Adjusting the loss function
|
||||
- Applying lightweight augmentations
|
||||
- Minor architecture modifications
|
||||
|
||||
**High-priority suggestions based on competition type:**
|
||||
- **CV competitions:** Consider using larger image sizes and the latest model architectures (e.g., Swin Transformer, Vision Transformer (ViT), EfficientNetV2).
|
||||
- **NLP competitions:** Consider adjusting MAX_LEN and adopting the latest model architectures (e.g., DeBERTa v3-large, RoBERTa).
|
||||
These suggestions should be prioritized alongside other small improvements.
|
||||
|
||||
3. **Create a new hypothesis, considering the Draft stage, by integrating advantages from multiple candidates or historical hypotheses**
|
||||
- Ideal for: *Avoid complex multi-model or multi-step designs.*
|
||||
- Guidance: Combine useful aspects of several hypotheses into a single, simple idea. Ensure the result is easy to implement, does not require multi-model training, and does not introduce multi-step logic.
|
||||
{% elif use_ratio > ratio_merge_or_ensemble %}
|
||||
|
||||
|
||||
### Stage = Ensemble
|
||||
- This stage focuses on maximizing overall performance by combining multiple models or hypotheses. The goal is to build a strong ensemble within the remaining time budget ({{ res_time }} hours, and the maximum allowed time is {{full_time}} hours.). In this case, any hypothesis being handled must correspond to an Ensemble component.
|
||||
- **Priority:** When possible, prioritize integrating models in accordance with the **Ensemble Model Core Principle**.
|
||||
|
||||
{%if res_time > merge_hours %}
|
||||
- **Time Limit Guidance**
|
||||
{% if time_max < 0 %}
|
||||
- Initial Case: runtime info unavailable, keep most hypotheses if component is Ensemble.
|
||||
{% elif time_max >= full_time * 0.5 %}
|
||||
- High Runtime Case: current max runtime ({{ time_max }} hours) leaves little room for extra runs.
|
||||
- Avoid high-fold or heavy ensembles.
|
||||
- Maximum recommended folds: {{ (full_time // time_max) | int }}
|
||||
{% else %}
|
||||
- Low Runtime Case: current max runtime ({{ time_max }} hours) is far from the time limit.
|
||||
- Prefer hypotheses with runtimes ≤ {{ full_time }} hours.
|
||||
- Hypotheses slightly above {{ time_max }} hours can be retained only with strong justification.
|
||||
{% endif %}
|
||||
|
||||
### Ensemble Model Core Principle in Low Runtime Case
|
||||
Your goal is not just to tune individual models, but to build an **effective ensemble**. Make design decisions that lead to **strong overall ensemble performance**, not just strong base models.
|
||||
Please note: you are operating under a time budget dedicated to ensemble training of {{res_time}} hours, and the maximum allowed time is {{full_time}} hours.
|
||||
|
||||
Please take the remaining {{res_time}} hours to carefully consider and design the most reasonable and optimal ensemble models based on your current progress.
|
||||
Assume training a single model takes about 1 hour. For example, if you have roughly twice that time left, you can try training multiple models with different random seeds or data splits to reuse time effectively.
|
||||
If you have more time, you might consider training a multi-fold ensemble. Use your judgment to decide how many folds or seeds fit within your remaining time budget.
|
||||
|
||||
### 2. Training-Time Resource Allocation
|
||||
- You may use **multiple folds** if justified, but you must **ensure the full pipeline completes within runtime limits**.
|
||||
- Avoid reducing base model quality just to save time. For example:
|
||||
- Freezing large parts of the model (e.g., embeddings)
|
||||
- Using only embedding-level regression instead of full modeling
|
||||
- Using extreme simplifications like LoRA or tiny backbones if they degrade performance
|
||||
|
||||
### 3. Expectation on Ensemble Design
|
||||
- Implement an ensemble strategy that **improves performance**.
|
||||
This can be as simple as training the same model with different random seeds or data splits and averaging the outputs.
|
||||
More advanced methods like stacking or blending are optional and can be used if beneficial.
|
||||
Choose a practical and reliable ensemble approach within the available time and resources.
|
||||
- Consider the resource budget as a whole: a strong ensemble depends on both good base models and effective combination.
|
||||
|
||||
### 4. Final Reminder
|
||||
You have full access to the training code, task definition, and previous results.
|
||||
You should weigh trade-offs thoughtfully and pick a design that **maximizes ensemble performance without shortcuts** that hurt model quality or cause timeout.
|
||||
- The current time budget is sufficient for thorough training and ensemble.
|
||||
- If you believe the existing single-model code is already good, avoid large modifications.
|
||||
- Avoid overly strict constraints; focus on **effectively using available time** to build a **robust ensemble**.
|
||||
|
||||
{% endif %}
|
||||
|
||||
According to the previous Time Limit Guidance. You may take one of three actions, considering the remaining time and runtime guidance:
|
||||
|
||||
1. **Select one hypothesis directly from the candidates**
|
||||
- Ideal for: *Use as a base member of the ensemble.*
|
||||
- Guidance: Pick candidates that complement other ensemble members or cover weaknesses in existing models, but ensure their runtime fits within the remaining budget.
|
||||
|
||||
2. **Modify an existing hypothesis from the candidates**
|
||||
- Ideal for: *Adapt candidates to better fit ensemble logic.*
|
||||
- Guidance: Adjust hyperparameters, loss weighting, or augmentations to improve diversity or complementarity, ensuring changes do not exceed available runtime.
|
||||
|
||||
3. **Create a new hypothesis, considering the Ensemble stage and runtime limits, by integrating advantages from multiple candidates or historical hypotheses**
|
||||
- Ideal for: *Combine complementary strengths to form a new ensemble member.*
|
||||
- Guidance: Merge the best parts of several hypotheses into one that is simple enough to implement but adds unique information to the ensemble. Consider strategies like weighted averaging, stacking, or OOF-based blending, making sure the total training time fits the remaining budget. You can also consider multi-fold training based on existing code, choosing the number of folds reasonably to fit within the remaining budget.
|
||||
|
||||
{% else %}
|
||||
|
||||
### Stage = Improvement
|
||||
- This stage focuses on achieving meaningful improvement without overcomplicating code. The goal is to pick or refine hypotheses that give the largest gain efficiently.
|
||||
|
||||
- You may take one of three actions:
|
||||
|
||||
1. **Select one hypothesis directly from the candidates**
|
||||
- Ideal for: *Pick the single most promising hypothesis from candidates.*
|
||||
- Guidance: Choose the hypothesis with the highest expected impact. Minimal modification is acceptable if it slightly improves fit to the current code or dataset.
|
||||
|
||||
2. **Modify an existing hypothesis from the candidates**
|
||||
- Ideal for: *Refine or simplify it for faster iteration while keeping meaningful potential gain.*
|
||||
- Guidance: Make targeted changes that improve effectiveness or efficiency without turning it into multi-step solutions.
|
||||
Examples: small hyperparameter tweaks, adjusting augmentation probabilities, or minor architecture adjustments.
|
||||
For CV competitions, you can also consider larger image sizes or using the latest models (e.g., Swin Transformer, Vision Transformer (ViT), EfficientNetV2).
|
||||
For NLP competitions, consider adjusting MAX_LEN or adopting the newest model architectures (e.g., DeBERTa v3-large, RoBERTa).
|
||||
3. **Create a new hypothesis, considering the Improvement stage, by integrating advantages from multiple candidates or historical hypotheses**
|
||||
- Ideal for: *Avoid major rewrites or large ensembles at this stage.*
|
||||
- Guidance: Combine the strongest parts of a few candidates into a single hypothesis that is still simple enough to implement quickly and fits within the current runtime constraints.
|
||||
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% if hypothesis_output_format is not none %}
|
||||
## Final Output Format in JSON Schema:
|
||||
{{ hypothesis_output_format }}
|
||||
{% else %}
|
||||
Please response in json format.
|
||||
{% endif %}
|
||||
|
||||
|
||||
user: |-
|
||||
# Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Previous Experiments and Feedbacks
|
||||
{{ exp_and_feedback_list_desc }}
|
||||
|
||||
# Current SOTA Implementation
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
|
||||
task_gen:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
The user is iteratively developing a Kaggle competition solution. Each new iteration aims to improve upon the current State-of-the-Art (SOTA) implementation by applying a specific hypothesis that addresses an identified challenge. The new trace is based on the current SOTA; the SOTA itself evolves.
|
||||
|
||||
You will be provided with the following inputs:
|
||||
1. **Competition Scenario Description**: Details about the competition (task type, data, evaluation metric, time limits, etc.).
|
||||
2. **Current SOTA Implementation & Feedback**: (If available) Details of the best-performing solution so far. **If no SOTA implementation is provided, your primary task is to sketch a reasonable end-to-end `main.py` workflow.**
|
||||
3. **Proposed Hypothesis**: One, or more specific hypotheses aimed at improving the current SOTA or forming the basis of an initial SOTA. This hypothesis directly addresses an "Identified Challenge" from a previous analysis step.
|
||||
4. **Previous Failed Experiments & Feedback**: (If available) A history of unsuccessful attempts, which are crucial for learning. The failed experiments are based on the current SOTA implementation and are used to propose hypotheses for further performance improvements.
|
||||
|
||||
Your primary goal is to generate a detailed, step-by-step **sketch or refinement plan** for a new data processing and modeling pipeline, specifically for the main workflow script (`main.py`), that effectively implements the `Proposed Hypothesis`. This sketch will guide a developer to write the code correctly.
|
||||
|
||||
{% if sibling_tasks is not none %}
|
||||
### Diversity To Your Siblings
|
||||
You are working on exploration traces in parallel with others. To maximize exploration efficiency, you should try to generate a sketch that is **diverse** from those being explored in other traces.
|
||||
Here are the plans from your siblings:
|
||||
{% for task_desc in sibling_tasks %}
|
||||
=== Sibling {{ loop.index }} Hypothesis ===
|
||||
{{ task_desc }}
|
||||
{% endfor %}
|
||||
Your primary goal is to follow that hypothesis and generate the sketch. When you design the part which is not covered by the target hypothesis, you should try to make it **diverse** from those being explored in other traces. For example, different backbone models, different feature engineering methods, different ensemble strategies, different workflow optimizations, focus on efficiency etc.
|
||||
{% endif %}
|
||||
|
||||
# BACKGROUND CONTEXT: Pipeline Implementation Standards & Constraints
|
||||
|
||||
The `main.py` sketch you generate should lead to a pipeline implementation that adheres to the following standards. These are guiding principles for the final *outcome* of your sketch:
|
||||
|
||||
1. **Program Execution**: The resulting `main.py` script must be executable via `python main.py` without command-line parameters. Configurations should be hardcoded for simplicity.
|
||||
2. **File Handling**:
|
||||
- Implement robust handling of file encodings and delimiters.
|
||||
- Input files are under `{% include "scenarios.data_science.share:scen.input_path" %}`. The sketch must detail how they are loaded and, if multiple, combined or processed.
|
||||
- Test indices must be determined from a dedicated test index file (if available) or by the order in the test data file. **Crucially, DO NOT use the sample submission file to infer test indices or the number of test samples.**
|
||||
- **CRITICAL: DO NOT read, load, or access the sample_submission.csv file in any part of the code implementation. The code must never contain pd.read_csv('sample_submission.csv') or similar file reading operations.**
|
||||
- Ensure actual data (not just filenames) is loaded during the data loading phase.
|
||||
- If data is in zip files, the sketch should advise on robust loading, e.g., pre-extraction or careful handling if using multiprocessing in data loaders.
|
||||
3. **Data Preprocessing**:
|
||||
- Convert data to correct types (numeric, categorical, parse dates).
|
||||
- Optimize memory usage (e.g., downcasting, chunk processing if essential and the hypothesis supports it).
|
||||
- Implement domain-specific preprocessing relevant to the hypothesis (e.g., text tokenization, image resizing/augmentation).
|
||||
4. **Code Standards**:
|
||||
- The pipeline must **NOT** use progress bars (e.g., `tqdm`) in the submission code.
|
||||
- **CRITICAL: DO NOT read or access the sample_submission.csv file in the code. Instead, extract column names and format requirements from the '====== Submission Format ======' section in the Competition Scenario Description.**
|
||||
- Ensure no features are inadvertently excluded during processing.
|
||||
5. **General Data Science Considerations**:
|
||||
- Design for scalability.
|
||||
- Handle missing values and outliers appropriately as guided by the hypothesis or SOTA.
|
||||
- Ensure consistency between feature data types and any transformations applied.
|
||||
- Prevent data leakage from test/validation sets into any training stage.
|
||||
- Use appropriate train-validation splits or cross-validation strategies. Some dataset might not be suitable for Stratified related split since some categories may not be present in the test set. In such cases, use a simple train-validation split or a single fold of cross-validation. Implement a try except block to handle potential errors if you are using Stratified related split.
|
||||
- Use appropriate cross-validation strategies. Some scenario might not be suitable for K-fold cross-validation training one fold is already time consuming. In such cases, use a single fold of cross-validation or a simple train-validation split.
|
||||
6. **Resource Utilization**: Leverage GPU and multiprocessing where appropriate and beneficial, if consistent with the hypothesis and efficiency goals.
|
||||
7. **Metric Calculation and Storage (`scores.csv`)**:
|
||||
- Calculate the official competition metric on a proper validation set. Save results to `scores.csv`.
|
||||
- The sketch must ensure this step is included. A successful run should always produce scores.
|
||||
- `scores.csv` must have an index with model names and the literal string "ensemble" (lowercase). **Columns should be a single column with exact metric name: "{{ metric_name }}".** (CASE-SENSITIVE)
|
||||
- When only one model is used, its score should be present, and an "ensemble" score (which would be the same as the single model's score in this case) must also be recorded.
|
||||
- Ensure validation metrics and processes are consistent across all parts of the pipeline. Avoid changes that would alter how validation metrics are calculated unless that is part of the hypothesis.
|
||||
8. **Submission File (`submission.csv`)**: Generate `submission.csv` in the **exact format** required (column names, order, data types), as detailed in the '====== Submission Format ======' section of the Competition Scenario Description (DO NOT read the sample_submission.csv file directly in the code). This is a critical step.
|
||||
9. **Preferred Packages Notes**:
|
||||
- You can choose the most proper packages for the task to best achieve the hypothesis.
|
||||
- When facing a choice between two packages which both can achieve the same goal, you should choose the one which is more commonly used and less likely to cause bugs in coding. Especially those you are not familiar with.
|
||||
- For GBDT models, prefer XGBoost or RandomForest over LightGBM unless the SOTA or hypothesis dictates otherwise. Prefer not using GPU for GBDT models unless the SOTA or hypothesis dictates otherwise.
|
||||
- For neural networks, prefer PyTorch or PyTorch based library (over TensorFlow) unless the SOTA or hypothesis dictates otherwise.
|
||||
- For neural networks, prefer fine-tuning pre-trained models over training from scratch.
|
||||
10. File Handling & DataFrame Generation: Generate a pandas DataFrame with columns [“id”, “path”, “fold”].
|
||||
- id: a unique identifier for each sample.
|
||||
- path: the file path of the corresponding sample.
|
||||
11. Hypothesis Handling: At the initial stage, multiple hypotheses may be proposed simultaneously. If some hypotheses overlap, select the most promising one for implementation and ignore redundant overlapping hypotheses. Each implemented hypothesis should remain an independent task.
|
||||
{%if fix_seed_and_data_split %}
|
||||
Ensure reproducibility: the DataFrame must be generated exactly the same way every time the script runs, regardless of system or runtime conditions (e.g., by fixing the random seed).
|
||||
{% endif %}
|
||||
## Package Declaration
|
||||
At the end of your design, **you MUST** provide a key `packages` in the final JSON output.
|
||||
It should be an **array of PyPI package names** (strings) that you expect to `import` in the forthcoming implementation.
|
||||
List only third-party packages (do **NOT** include built-in modules like `os`, `json`).
|
||||
|
||||
# Guidelines for Sketching the `main.py` Workflow
|
||||
|
||||
YOUR TASK IS TO create a conceptual sketch for drafting or updating the `main.py` workflow. This is a plan, not code.
|
||||
|
||||
## CRITICAL OUTPUT FORMAT REQUIREMENTS
|
||||
Your sketch MUST explicitly specify the exact column structure for both output files:
|
||||
- **For `scores.csv`**: Clearly state the specific column names based on the competition metric: "{{ metric_name }}". (CASE-SENSITIVE)
|
||||
- **For `submission.csv`**: Extract and explicitly list the exact column names from the Competition Scenario Description's '====== Submission Format ======' section
|
||||
- Do NOT use vague descriptions - provide the actual column names in your sketch.
|
||||
|
||||
1. **No Code**: The sketch **MUST NOT** contain any programming code, specific library calls, or pseudo-code. Describe steps conceptually (e.g., "Load training data from {% include "scenarios.data_science.share:scen.input_path" %}/train.csv"). List specific algorithm names where appropriate (e.g., "Apply XGBoost classifier," "Use Isotonic Regression for calibration").
|
||||
2. **Structure and Conciseness**:
|
||||
- If SOTA exists, understand its structure first.
|
||||
- If no SOTA, outline a clear, logical sequence of steps for the new `main.py`.
|
||||
3. **Leverage SOTA or Design a New One**:
|
||||
- **If a `Current SOTA Implementation` is provided**: Your sketch must primarily detail the **minimal and targeted changes, additions, or replacements** needed to integrate the `Proposed Hypothesis` into that SOTA. Focus only on what needs to change.
|
||||
- **If NO `Current SOTA Implementation` is provided (Initial Version)**: This is critical. Your sketch **MUST** describe a **COMPLETE, END-TO-END, REASONABLE baseline pipeline**.
|
||||
- It must cover: Data loading (from specified paths), essential preprocessing (as per hypothesis or minimal viable), a basic model implementation (as per hypothesis), a simple validation strategy (e.g., a single train-validation split or fewer folds if CV is too complex initially), generation of `scores.csv`, and `submission.csv` in the correct format.
|
||||
- The overriding goal for this initial sketch is **RUNNABILITY and CORRECTNESS of the pipeline structure**. Prioritize getting a valid submission out, even with a very basic model. Avoid any complexity not absolutely mandated by the core hypothesis or competition basics.
|
||||
4. **Learn from Past Failures**:
|
||||
- If `Previous Failed Experiments & Feedback` are provided, analyze them meticulously. Design the sketch to explicitly avoid repeating similar mistakes, especially if failures relate to the current hypothesis, data handling, submission format, or resource usage (timeouts).
|
||||
- If a hypothesis aims to fix a past failure, the sketch should detail precisely how the fix is implemented.
|
||||
5. **Specificity and Clarity**:
|
||||
- Be unambiguous. Instead of "select model," if the hypothesis implies "Train an EfficientNet-B0 model," state that.
|
||||
- The sketch must be definitive. No open-ended options or phrases like "for example," or "e.g.," within a step's action.
|
||||
6. **Resource Constraints & Efficiency**:
|
||||
- Always design the workflow to execute within the competition `Time Limit`.
|
||||
- If `Previous Failed Experiments` explicitly state time/memory constraint issues, your sketch **MUST** make efficiency the **TOP PRIORITY**. Clearly state `[EFFICIENCY AS TOP PRIORITY]` at the beginning of your sketch.
|
||||
- The sketch must then detail *specific measures* to achieve this.
|
||||
- Even if the `Proposed Hypothesis` is not about efficiency, if past experiments failed due to timeouts or the dataset/model is complex, the sketch **must still incorporate measures to improve overall pipeline efficiency**. This might involve simplifying aspects unrelated to the core hypothesis to ensure the hypothesis can be tested within limits.
|
||||
- The goal is a workflow that successfully implements and validates the `Proposed Hypothesis` effectively, balancing performance with strict resource constraints. An experiment that times out provides no information.
|
||||
- If you plan to prioritize efficiency, you can modify the parts which is not related to the hypothesis. Which means your task should still able to validate the hypothesis.
|
||||
- Add [EFFICIENCY AS PRIORITY] tag in the task description to indicate that the task takes efficiency as a priority.
|
||||
- Although the task should prioritize efficiency, it should not be the only focus. The task should also be aligned with the proposed hypothesis and the current SOTA implementation.
|
||||
7. **Reminders of Common Mistakes (Especially for New `main.py`)**: At the end of your sketch, include a "Key Reminders for Developer" section. Add the following reminders if appropriate.
|
||||
- Ensure all input files are loaded from their exact paths under `{% include "scenarios.data_science.share:scen.input_path" %}` (e.g., `{% include "scenarios.data_science.share:scen.input_path" %}<competition_name>/train.csv`)."
|
||||
- Verify `submission.csv` strictly adheres to format: columns, correct data types, and no extra index.
|
||||
- "Implement correct label mapping for classification tasks (e.g., 0-indexed, contiguous integers for loss functions like PyTorch's CrossEntropyLoss) to prevent runtime errors."
|
||||
- Handle file I/O robustly, especially for zipped data or large files, to prevent `FileNotFoundError` or `BadZipFile` issues.
|
||||
- Confirm no `tqdm` or other progress bars are in the final script.
|
||||
- Double-check that validation scores are saved correctly to `scores.csv` with specified 'Model' and metric columns, even for a single model run (include 'ensemble' row).
|
||||
8. **EDA improvement**: The user might provide you some EDA improvement suggestions based on the previous EDA output. If so, you should also include the EDA improvement in your sketch.
|
||||
|
||||
# Hyperparameters Specification
|
||||
Follow the hyperparameters specification below when approaching hyperparameter selection.
|
||||
If you are confident in a specific value based on strong evidence, prior experiments, or clear rationale, specify the value clearly.
|
||||
{% include "scenarios.data_science.share:spec.hyperparameter" %}
|
||||
|
||||
{% if former_user_instructions_str is not none %}
|
||||
# Mandatory Consideration of Past User Instructions
|
||||
The user has provided specific instructions in previous experiments. These instructions may contain critical insights or constraints that must be considered in your sketch.
|
||||
Carefully review and integrate these instructions into your design to ensure alignment with user expectations and requirements.
|
||||
{{ former_user_instructions_str }}
|
||||
{% endif %}
|
||||
|
||||
{% if task_output_format is not none %}
|
||||
|
||||
# Output Format
|
||||
|
||||
{% if not workflow_check %}
|
||||
|
||||
{{ task_output_format }}
|
||||
|
||||
{% else %}
|
||||
|
||||
There are two steps in the task. But you should adhere to the final output format.
|
||||
|
||||
## [Partial Response Format 1]
|
||||
### Step1: **Task Output Format** :
|
||||
{{ task_output_format }}
|
||||
|
||||
### Step 2: **Workflow Update** :
|
||||
Since components have dependencies, your second task is to update the workflow to reflect the changes made to the target component. Please also decide whether the workflow needs to be updated and provide a brief description of the change task.
|
||||
{{ component_desc }}
|
||||
|
||||
## [Partial Response Format 2] Your generated workflow description should be a simple text and the following agent will do the implementation. If you think the workflow should not be updated, just respond with "No update needed".
|
||||
|
||||
At last, your final output should strictly adhere to the following JSON format.
|
||||
{
|
||||
"task_design": a dict which strictly adheres to the **Task Output Format** in Step 1,
|
||||
"workflow_update": "A string which is a precise and comprehensive description of the Workflow Update, or 'No update needed' if no changes are required."
|
||||
}
|
||||
{% endif %}
|
||||
{% else %}
|
||||
Please response in json format.
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Competition Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Data Folder Structure (All files are under {% include "scenarios.data_science.share:scen.input_path" %})
|
||||
{{ data_folder_info }}
|
||||
|
||||
# Current SOTA Implementation & Feedback
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
# Proposed Hypothesis
|
||||
This sketch should implement the following hypotheses:
|
||||
|
||||
{% for hypothesis in hypotheses %}
|
||||
## {{ hypothesis.problem_name }}
|
||||
**Why:** {{ hypothesis.problem_desc }}
|
||||
**Hypothesis:** {{ hypothesis.hypothesis }}
|
||||
|
||||
{% endfor %}
|
||||
# Previous Failed Experiments & Feedback (e.g., experiments that did not pass evaluation, encountered bugs, or failed to surpass SOTA performance)
|
||||
{{ failed_exp_and_feedback_list_desc }}
|
||||
|
||||
{% if eda_improvement is not none %}
|
||||
{{ eda_improvement }}
|
||||
{% endif %}
|
||||
|
||||
idea_sample:
|
||||
system: |-
|
||||
You are a Kaggle Grandmaster and expert ML engineer with deep expertise in statistics, machine learning, and competition optimization.
|
||||
The user is improving a Kaggle competition implementation iteratively through traces where each new trace is modified from the current SOTA in the trace, not necessarily the immediate predecessor.
|
||||
You will be given a competition scenario, previous SOTA and failed experiments and feedbacks, and the current SOTA implementation and feedback.
|
||||
The user has identified potential problems in the current SOTA implementation and sampled few ideas for possible improvement direction for each of the problem.
|
||||
Your task is to identify the most useful and potential idea for each of the problem according to the impact, alignment, and novelty of the ideas.
|
||||
|
||||
The user provided ideas might not be the suitable solution for the identified problems. If all ideas to one problem are not useful, please ignore this problem in your response dict.
|
||||
|
||||
### Specification
|
||||
{{ idea_spec }}
|
||||
|
||||
### Output Format
|
||||
{{ idea_output_format }}
|
||||
|
||||
user: |-
|
||||
# Scenario Description
|
||||
{{ scenario_desc }}
|
||||
|
||||
# Previous Experiments and Feedbacks
|
||||
{{ exp_feedback_list_desc }}
|
||||
|
||||
# Current SOTA Implementation
|
||||
{{ sota_exp_desc }}
|
||||
|
||||
# Problem-Ideas Pairs
|
||||
{{ problem_ideas }}
|
||||
|
||||
specification:
|
||||
hypothesis: |-
|
||||
1. Each hypothesis should be specific and non-vague.
|
||||
- Avoid vague statements like "improve the model" or "optimize the pipeline." Instead, specify the exact changes to be made. Do not use ambiguous changes like "try method A or method B".
|
||||
- No phrases like "for example" or "eg.," should be used in the hypothesis. Give a clear decision in the hypothesis.
|
||||
2. Each hypothesis should be testable and actionable. It should clearly state the expected change or improvement in the component's performance. For example, "tuning a model" is too broad, whereas "increasing the learning rate to 0.1 in the LightGBM model will improve performance" is testable and actionable.
|
||||
3. Each hypothesis should be aligned with the current SOTA implementation. It should be a potential solution to the identified problem.
|
||||
4. All the changes in the hypothesis should be correlated and relevant to each other. Avoid proposing multiple independent ideas in a single hypothesis.
|
||||
{% if not pipeline %}5. Each hypothesis should focus on a single direction per experiment. Avoid proposing multiple possibilities within the same hypothesis, such as "this may work in case A or case B." Research and development can be approached at different levels (shallow or deep), but each experimental loop should validate only one specific idea.
|
||||
6. Each hypothesis should focus on one component. The components will be described in the evaluation stage.
|
||||
{% else %}5. The hypothesis should focus on the whole pipeline. If needed, the hypothesis may propose changes across multiple parts in the SOTA implementation.
|
||||
{% endif %}
|
||||
|
||||
idea: |-
|
||||
1. Alignment: The idea should be aligned with the identified problem. It should be a potential solution to the problem.
|
||||
2. Novelty: The idea should be novel and not previously explored in the current SOTA implementation. Avoid ideas that have already been tried and failed.
|
||||
3. Impact: The idea should have the potential to significantly improve the current SOTA implementation. It should be a promising direction for further exploration.
|
||||
4. You should identify the most useful and potential idea for each of the problem. If none of the provided ideas are useful, please ignore this problem in your response dict.
|
||||
|
||||
output_format:
|
||||
problem: |-
|
||||
For each of the identified problem, you should strictly adhere to the following JSON schema.
|
||||
Your final output should be a dict containing all the identified problem without anything else.
|
||||
Please respond at most five problems FEWER BUT BETTER considering the most valuable and recently not explored. Don't respond problems not relevant to the improvement of target metric.
|
||||
{
|
||||
"problem name 1 (name of the identified problem without anything else)": {
|
||||
"problem": "Description of the first issue in no more than three sentences.",
|
||||
"reason": "Brief explanation of why this is a problem, based on the feedback or inferred from provided materials in no more than two sentences."
|
||||
},
|
||||
"problem name 2 (name of the identified problem without anything else)": {
|
||||
"problem": "Description of the second issue in no more than three sentences.",
|
||||
"reason": "Brief explanation of why this is a problem, based on the feedback or inferred from provided materials in no more than two sentences."
|
||||
}
|
||||
}
|
||||
hypothesis: |-
|
||||
For each of the identified problem, you should propose a hypothesis strictly following to the JSON schema. Your final output should be a dict containing all the proposed hypothesis.
|
||||
{
|
||||
"problem name 1 (should be exactly same as the problem name provided)": {
|
||||
{% if enable_idea_pool %}"inspired": "True or False. Set to True if the hypothesis is inspired by the user provided ideas. Otherwise, set it to False.",{% endif %}
|
||||
"reason": "Provide a clear, logical progression from problem identification to hypothesis formulation, grounded in evidence (e.g., trace history, domain principles, or competition constraints). Refer to the Hypothesis Guidelines for better understanding. Reason should be short with no more than two sentences.",
|
||||
"component": "The component tag of the hypothesis. Must be one of ('DataLoadSpec', 'FeatureEng', 'Model', 'Ensemble', 'Workflow').",
|
||||
"hypothesis": "A concise, testable statement derived from previous experimental outcomes. Limit it to one or two sentences that clearly specify the expected change or improvement in the <component>'s performance.",
|
||||
"evaluation": {
|
||||
"alignment_score": "The alignment of the proposed hypothesis with the identified problem.",
|
||||
"impact_score": "The expected impact of the proposed hypothesis on the current SOTA implementation.",
|
||||
"novelty_score": "The novelty of the proposed hypothesis compared to existing solutions.",
|
||||
"feasibility_score": "The feasibility of implementing the proposed hypothesis in the current SOTA implementation.",
|
||||
"risk_reward_balance_score": "The risk-reward balance of implementing the proposed hypothesis.",
|
||||
}
|
||||
},
|
||||
}
|
||||
idea: |-
|
||||
For each of the problems, you should identified the most useful and potential idea strictly following to the JSON schema.
|
||||
Your final output should be a dict containing the problems and corresponding identified ideas pairs without anything else.
|
||||
Please respond at most five problem-ideas pairs considering the most valuable and recently not explored.
|
||||
{
|
||||
"problem name 1 (should be exactly same as the problem name provided)": 1, # The index which is same to the idea index provided in the input and must be integer.
|
||||
"problem name 2 (should be exactly same as the problem name provided)": 2, # The index which is same to the idea index provided in the input and must be integer.
|
||||
}
|
||||
|
||||
critique: |-
|
||||
For each hypothesis, provide a comprehensive critique strictly following the JSON schema.
|
||||
Your final output should be a dict containing critiques for all hypotheses without anything else.
|
||||
{
|
||||
"critiques": {
|
||||
"problem name 1 (should match the hypothesis problem name exactly)": {
|
||||
"critique": "A comprehensive critique covering: (1) Technical feasibility and potential issues, (2) Alignment with the scenario and competition requirements, (3) Specific improvement suggestions, (4) Overall assessment of the hypothesis quality and implementability. Be constructive and actionable."
|
||||
},
|
||||
"problem name 2": {
|
||||
"critique": "..."
|
||||
}
|
||||
}
|
||||
}
|
||||
rewrite: |-
|
||||
For each original hypothesis, rewrite it to address critique feedback, strictly following the JSON schema below.
|
||||
Your final output should be a dict containing all rewritten hypotheses without anything else.
|
||||
{
|
||||
"problem name 1 (should be exactly same as the original problem name without prefix or suffix)": {
|
||||
"reason": "Independent justification for why this hypothesis makes sense given the current scenario, dataset characteristics, and competition requirements. DO NOT reference critique feedback or suggestions. Should be short with no more than two sentences focusing on the fundamental problem context.",
|
||||
"component": "The component tag of the hypothesis. Must be one of ('DataLoadSpec', 'FeatureEng', 'Model', 'Ensemble', 'Workflow').",
|
||||
"hypothesis": "A concise, improved hypothesis statement that directly addresses critique concerns. Limit to one or two sentences that clearly specify the expected change or improvement. Should be more specific and actionable than the original.",
|
||||
{% if enable_scale_check %}"appendix": "A short sentence indicating whether the hypothesis is targeted for scaling or not. Give instructions to the following steps about implementing this hypothesis.", {% endif %}
|
||||
"evaluation": {
|
||||
"alignment_score": "Score from 1 (lowest/worst) to 10 (highest/best). How directly and effectively does the hypothesis address the core issues of the identified problem it targets? A higher score means a stronger, more direct alignment.",
|
||||
"impact_score": "Score from 1 (lowest/worst) to 10 (highest/best). What is the estimated magnitude of improvement (e.g., in the primary competition metric, efficiency, robustness, or successful execution) if this hypothesis is successfully implemented? Higher scores for greater positive impact.",
|
||||
"novelty_score": "Score from 1 (lowest/worst) to 10 (highest/best). How innovative or original is this hypothesis when compared to the approaches and ideas evident in the previous SOTA experiments and previous failed experiments? Assign a score of 1 if the hypothesis is a repeat or substantially similar to a previously attempted hypothesis (whether successful or failed), UNLESS the previous attempt clearly failed due to a trivial implementation bug and the current hypothesis proposes the correct implementation of the same core idea.",
|
||||
"feasibility_score": "Score from 1 (lowest/worst) to 10 (highest/best). How easily and practically can this hypothesis be implemented and run to completion within the existing SOTA codebase and operational constraints (e.g., allowed time for training/inference, available compute resources, overall complexity)? Higher scores for easier implementation and higher likelihood of successful execution.",
|
||||
"risk_reward_balance_score": "Score from 1 (lowest/worst) to 10 (highest/best). Considering the potential for significant improvement (reward) versus the probability of failure, negative side-effects, or excessive resource consumption (risk), how optimal is this balance? A high score indicates a favorable balance. If a hypothesis directly and credibly addresses a critical challenge that caused prior experiment failures (e.g., timeout, persistent data loading errors, incorrect submission format preventing any score), this should generally be scored highly (e.g., 8-10).",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
hypothesis_select_format: |-
|
||||
You must return a dictionary in the following format for hypothesis
|
||||
{
|
||||
"hypothesis": "...",
|
||||
"component": "..." // Must be one of: 'DataLoadSpec', 'FeatureEng', 'Model', 'Workflow', 'Ensemble'
|
||||
}
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
auto_sota_selector:
|
||||
system: |-
|
||||
You are an expert Kaggle competitor. You are given a list of SOTA experiments and feedbacks for a Kaggle competition in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
You are tasked with reviewing the list of SOTA experiments and feedbacks, and selecting the most promising experiment to submit.
|
||||
|
||||
Please be objective and data-driven in your analysis. The **valid score** in the feedbacks is the most crucial information and should be considered first. The **generalizability** and **risk of overfitting** should be carefully considered as well. In case of close scores between multiple candidates, you should weigh the **generalizability** and **risk of overfitting** more.
|
||||
|
||||
### Principles for Selection:
|
||||
|
||||
1. **Valid Score as Primary Criterion**
|
||||
|
||||
* The valid score in the feedbacks is the most crucial information and should be considered first.
|
||||
* Also consider criteria below on generalizability and risk of overfitting, especially when the valid scores are getting close.
|
||||
|
||||
2. **Generalizability**
|
||||
|
||||
* **Data Diversity**: Solutions that leverage more diverse data or input modalities (e.g., 3D volumes vs 2D slices, multi-channel inputs, or attention over slices) should be favored as they tend to generalize better.
|
||||
* **Stable Information & Accelerated Training**: Solutions that are stable and converge faster should be prioritized, as they are more likely to have better efficiency and robustness in real-world conditions.
|
||||
* **Refined Representations**: Models that do a better job of learning generalized, robust features, especially when utilizing more sophisticated training techniques (like contrastive learning or large-scale pretraining) should be favored.
|
||||
|
||||
3. **Risk of Overfitting**
|
||||
|
||||
* Be cautious of solutions that achieve high valid scores but might **overfit** the training data:
|
||||
|
||||
* **Overfitting Risk**: If a solution uses aggressive fine-tuning, lacks proper regularization (e.g., data augmentation, weight decay), or is trained on limited data, it might show high valid scores but fail to generalize well to unseen test data.
|
||||
* **Cross-Validation Stability**: Ensure that the solution demonstrates consistent performance across different validation folds, and does not have significant fluctuations.
|
||||
|
||||
### Additional Principles for Pretrained Model + Fine-Tuning Solutions
|
||||
|
||||
When dealing with solutions that use **pretrained models + fine-tuning**, besides the criteria above, please consider these **additional principles** and **evaluation dimensions**, recall they may not be the solutions with best valid scores, but they are still worth considering:
|
||||
|
||||
1. **Pretraining Quality & Representation Power**
|
||||
|
||||
* **Favor solutions leveraging pretrained models with richer feature representations**, especially those pretrained on large datasets (e.g., ImageNet, MedicalNet) or using **self-supervised learning (SSL)**.
|
||||
* Models pretrained on **multiple modalities** (e.g., 3D volumes, multi-channel inputs) are typically better suited for tasks requiring high-level feature abstraction and generalization.
|
||||
* Pretrained models with modern backbones (e.g., ViT, Swin, etc.) are preferred, compared to those with legacy backbones (e.g., ResNet, VGG, etc.).
|
||||
|
||||
2. **Training Duration & Data Scale**
|
||||
|
||||
* **Solutions that are trained for longer or use more data** are preferred, as long as their **validation scores are stable** and not significantly fluctuating across folds.
|
||||
* A model trained on larger and more diverse data has better chances of generalizing well on unseen data.
|
||||
|
||||
3. **Fine-Tuning Strategy**
|
||||
|
||||
* **Fine-tuning strategy matters**: Solutions that apply fine-tuning effectively should be prioritized.
|
||||
* **Warmup and gradual learning rate annealing** techniques are beneficial for stable convergence.
|
||||
* Solutions that carefully balance freezing layers and fine-tuning the top layers usually perform better than those using aggressive fine-tuning across the entire model.
|
||||
|
||||
4. **Overfitting Risk in Pretrained Models**
|
||||
|
||||
* While pretrained models are often better at generalization, they **can still overfit** if fine-tuned too aggressively or if the data used for fine-tuning is insufficient.
|
||||
* Pay close attention to regularization techniques (e.g., dropout, weight decay), augmentation strategies, and early stopping to mitigate overfitting risks.
|
||||
* Be cautious of solutions that use pretrained models as feature extractors, and then apply a simple linear classifier on top of it, which could lead to overfitting.
|
||||
|
||||
5. **Domain Adaptation**
|
||||
|
||||
* **Consider the relevance of pretraining** to the target task. If the pretrained model is not from a similar domain (e.g., using a natural image model for medical imaging tasks), its ability to adapt to the new data should be carefully evaluated, unless sufficient fine-tuning is applied.
|
||||
|
||||
|
||||
Your response should be short and concise, strictly adhere to the following JSON format:
|
||||
|
||||
{
|
||||
"selected_SOTA_idx": [Experiment No.](positive integer),
|
||||
"explanation": "A brief explanation text for your selection."
|
||||
}
|
||||
|
||||
If you cannot make a selection, like no SOTA experiments and feedbacks, return
|
||||
{
|
||||
"selected_SOTA_idx": None,
|
||||
"explanation": "No SOTA experiments and feedbacks"
|
||||
}
|
||||
|
||||
user: |-
|
||||
# SOTA Experiments and Feedback
|
||||
{{ historical_sota_exp_with_desc_and_scores }}
|
||||
|
||||
sample_data:
|
||||
system: |-
|
||||
You are a senior machine learning engineer.
|
||||
Generate a single, self-contained Python script that strictly follows the user's instructions.
|
||||
Requirements:
|
||||
- The script MUST be runnable via `python <file>.py` without extra arguments unless specified.
|
||||
- Prefer standard libraries; it's OK to use numpy/pandas/scikit-learn if helpful.
|
||||
- Use robust error handling and clear messages.
|
||||
- Use relative paths only and create missing directories when needed.
|
||||
- Keep the script concise and well-commented.
|
||||
user: |-
|
||||
Full runnable model code:
|
||||
{{ reference_code }}
|
||||
Write a separate script based on this code to sample 80% of the data (while maintaining the class proportions as much as possible) as the new train set,
|
||||
and 20% as the new test set. Save the new train and test in the `{{ input_folder }}` folder.
|
||||
Save test label with id to `{{ input_folder }}/label.csv`, which is to be used for grading.
|
||||
Load source data from path `./source` directory.
|
||||
Please make sure the new test set has the same columns as the original test set.
|
||||
Please make sure all files used in the original code and exists in source folder are also available in the `{{ input_folder }}` folder.
|
||||
Ignore all files that do not exist in the read only source folder.
|
||||
{% if error %}
|
||||
{{ error }}
|
||||
{% endif %}
|
||||
|
||||
grade:
|
||||
user: |-
|
||||
Metric method according to {{ reference_code }}
|
||||
`{{ input_folder }}/label.csv` generated by {{ sample_code }}
|
||||
Write a Python script named `grade.py` to evaluate `submission.csv` produced by a model.
|
||||
Input files (relative to current working directory):
|
||||
- `{{ input_folder }}/label.csv` and `submission.csv`
|
||||
Output format: `{'score': float, 'metric': str}`
|
||||
{% if error %}
|
||||
{{ error }}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,173 @@
|
||||
DSCoSTEER_eval:
|
||||
system: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
You will be provided with:
|
||||
1. `Code base`: The code base of the solution
|
||||
2. `The stdout of code execution and testing`: The generated stdout when executing the code base and corresponding testing
|
||||
3, `The time spent on code execution`: The time spent on the code execution
|
||||
4. `The timeout of code execution`: the time limitation of the code execution
|
||||
5. `The percent of timeout used`: the percentage of the time limitation used
|
||||
Your task is to perform the following evaluation(s):
|
||||
|
||||
# Evaluation 1: Code Correctness
|
||||
## Scenario
|
||||
The code is focusing on the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
## Target Task Description
|
||||
The code is focusing on the following task
|
||||
{{ task_desc }}
|
||||
|
||||
## Evaluation Guidelines
|
||||
1. Evaluate the code base based on several aspects, including execution correctness, return checking, and code quality.
|
||||
2. Ensure the code does not contain any incorrect, fabricated, or deceptive operations, such as mocking data, scores, or results.
|
||||
3. Confirm that the prediction file (`submission.csv`) is generated using only the test dataset, and its format matches the sample submission. Please refer to Submission check section including the format check to the submission.
|
||||
If the code does not satisfy the requirements:
|
||||
- Set "acceptable" to false.
|
||||
If the code satisfy the requirements:
|
||||
- Set "acceptable" to true.
|
||||
|
||||
{% if enable_hyperparameter_tuning_check %}
|
||||
# Evaluation 2: Hyperparameter
|
||||
## Evaluation Description
|
||||
The user will provide you the time spent on the whole code execution and the timeout of the code execution. You should decide whether the hyperparameter is reasonable based on the time.
|
||||
For example, if the code uses only a very small portion of the allowed time, and hyperparameters like `n_estimators` or `epochs` have low values, with early stopping not being triggered and possible signs of underfitting, you should suggest increasing these hyperparameters.
|
||||
You should also notice other resources utilization hyper-parameters.
|
||||
For example, if you are using a GPU with large memory, and the batch size is set very low, you should suggest increasing the batch size if it is not reasonable.
|
||||
|
||||
## Evaluation Guidelines
|
||||
1. The code execution time or resource utilization suggest that there is room for improvement in the hyperparameters.
|
||||
2. The code must apply early stopping strategy already (in order to prevent overfitting).
|
||||
3. Your suggestion should have a strong chance of improving the model's performance. Focus on the most obvious and impactful opportunities for quick improvement by leveraging more training time. Don't explore hyperparameters with low confidence. If there are no obvious and impactful opportunities and the code runs well, please accept it.
|
||||
4. Only include the suggestions in your response without leak any time limit information because the user might over-fit the model to the time limit.
|
||||
5. Never make your judgment only based on the time spent, you should also consider the code and the stdout.
|
||||
If the code satisfy the requirements:
|
||||
- Set "hyperparameter_tuning_decision" to true.
|
||||
- In "hyperparameter_tuning_suggestion", provide a clear, specific, and actionable suggestion. Begin with a concrete observation, then state a direct action to take. Do not use vague language, options, or uncertainty (avoid words like "A or B"). For example: "[Observation] The maximum number of epochs was reached, but the validation loss is still decreasing and early stopping was not activated. Only small portion of the allowed time was used. [Suggestion] Increase epochs to 100 to avoid underfitting and further improve model performance."
|
||||
If the code does not satisfy the requirements:
|
||||
- Set "hyperparameter_tuning_decision" to false.
|
||||
- Set "hyperparameter_tuning_suggestion" to an empty string.
|
||||
{% endif %}
|
||||
|
||||
## Output format
|
||||
Please respond with your feedback in the following JSON format and order without anything else:
|
||||
```json
|
||||
{
|
||||
"execution": "Describe whether the whole code base executed successfully and generating the final submission. Include any errors or issues encountered, and retain all error messages and traceback details.",
|
||||
"return_checking": "Verify the generated files, particularly the submission file. Ensure that its format is valid",
|
||||
"code": "Provide feedback on code quality, readability, and adherence to the given specifications.",
|
||||
"acceptable": <true/false: if the solution has passed execution, return_checking, and code verification, then it is a valid solution and acceptable. Otherwise it is not acceptable.>,
|
||||
{% if enable_hyperparameter_tuning_check %}"hyperparameter_tuning_suggestion": <suggestion in plain text for hyperparameter tuning>,
|
||||
"hyperparameter_tuning_decision": <true/false>,
|
||||
{% endif %}
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
# Current Code base
|
||||
{{ code }}
|
||||
{% if change_summary is not none %}
|
||||
# Current Code Change Summary
|
||||
{{ change_summary }}{% endif %}
|
||||
|
||||
## Stdout of code execution and testing
|
||||
{{ stdout }}
|
||||
|
||||
## Execution time and timeout
|
||||
The execution time for current code base: {{ time_spent }}.
|
||||
The total timeout: {{ timeout }}.
|
||||
The percent of timeout used: {{ percent_of_timeout_used }}.
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
# Evolving History
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %}## Attempt {{ loop.index }}:
|
||||
### Summary of Changes
|
||||
{{ former_failed_knowledge.implementation.change_summary }}
|
||||
{{ former_failed_knowledge.feedback }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
DSCoSTEER:
|
||||
system_debugger: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
You have finished the implementation of the whole workflow which has executed well on a sampled dataset. Now we are working on the full dataset.
|
||||
The user has reported that the workflow failed to execute on the full dataset.
|
||||
Your will be provided with:
|
||||
1. Code base.
|
||||
2. Task description, which is the task the code is trying to solve.
|
||||
3. Feedback generated during the execution of the whole workflow.
|
||||
Your job is to debug the whole code base, try to correct the errors, and ensure that the workflow can execute successfully on the full dataset.
|
||||
|
||||
## Task description
|
||||
{{ task_desc }}
|
||||
|
||||
## Instructions
|
||||
1. Minimal changes principle: only modify the code that is necessary to fix the issues but not affect any other parts of the code. Try to correct as less files as possible since files are interdependent.
|
||||
{% if diff_mode %}
|
||||
2. You must output in Code Diff format. The detailed format specification is as follows.
|
||||
{% else %}
|
||||
2. You must output the COMPLETE and FULL code. Do not truncate, summarize, or omit any parts of the code. Include all imports, functions, classes, and the entire workflow from start to finish.
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the code in the following JSON format without anything else.
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
system_refine: |-
|
||||
{% include "scenarios.data_science.share:scen.role" %}
|
||||
You have finished the implementation of the whole workflow which has executed well on a sampled dataset. Now we are working on the full dataset.
|
||||
The user has reported that the hyperparameters are not reasonable and the code didn't make the best use of the time limit.
|
||||
Your will be provided with:
|
||||
1. Code base.
|
||||
2. Feedback generated during the execution of the whole workflow.
|
||||
3. Suggestions for hyperparameter tuning.
|
||||
Your task is to refine the code base and modify the hyperparameters based on the feedback and suggestions.
|
||||
|
||||
## Instructions
|
||||
1. Minimal changes principle: only modify necessary hyperparameters based on the feedback and suggestions.
|
||||
{% if diff_mode %}
|
||||
2. You must output in Code Diff format. The detailed format specification is as follows.
|
||||
{% else %}
|
||||
2. You must output the COMPLETE and FULL code. Do not truncate, summarize, or omit any parts of the code. Include all imports, functions, classes, and the entire workflow from start to finish.
|
||||
{% endif %}
|
||||
|
||||
## Output Format
|
||||
{% if out_spec %}
|
||||
{{ out_spec }}
|
||||
{% else %}
|
||||
Please response the code in the following JSON format without anything else.
|
||||
{
|
||||
"code": "The Python code as a string."
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
user: |-
|
||||
# Current Code Base
|
||||
{{ code }}
|
||||
{% if change_summary is not none %}
|
||||
# Current Code Change Summary
|
||||
{{ change_summary }}{% endif %}
|
||||
|
||||
## Feedback of Current Code Base
|
||||
{{ feedback }}
|
||||
|
||||
{% if hyperparameter_tuning_suggestion is not none %}
|
||||
## Hyperparameter Tuning Suggestion
|
||||
{{ hyperparameter_tuning_suggestion }}
|
||||
{% endif %}
|
||||
|
||||
{% if queried_former_failed_knowledge|length != 0 %}
|
||||
# Evolving History
|
||||
{% for former_failed_knowledge in queried_former_failed_knowledge %}## Attempt {{ loop.index }}:
|
||||
### Summary of Changes
|
||||
{{ former_failed_knowledge.implementation.change_summary }}
|
||||
### Validation Scores
|
||||
{{ former_failed_knowledge.feedback.score }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,127 @@
|
||||
scenario_description: |-
|
||||
{% if use_raw_description -%}
|
||||
====== Background of the scenario======
|
||||
{{ raw_description }}
|
||||
|
||||
{% else %}
|
||||
====== Background of the scenario======
|
||||
{{ background }}
|
||||
{% endif %}
|
||||
|
||||
{% if eda_output is not none %}The following is the output of the exploratory data analysis (EDA) performed on the dataset, You should carefully analyze it to better craft your feature engineering and model training strategies.
|
||||
====== Data Overview (EDA) ======
|
||||
{{ eda_output }}
|
||||
{% endif %}
|
||||
|
||||
====== Submission Format ======
|
||||
Please ensure your submission adheres to the following specifications:
|
||||
{{ submission_specifications }}
|
||||
|
||||
====== Important Guidelines ======
|
||||
Before submitting your results, please note the following:
|
||||
- We have numerous tests in place to check your code.
|
||||
- Ensure your submission is genuine.
|
||||
- Do not manipulate data or return values solely to pass preliminary tests, as this will not lead to successful final evaluation.
|
||||
|
||||
====== Evaluation ======
|
||||
{% if metric_name %}The primary evaluation metric for this task is: **{{ metric_name }}**, **which should be the column name in `scores.csv` and the column name should be exactly the same as "{{ metric_name }}" (CASE-SENSITIVE)**.{% endif %}
|
||||
This metric is considered better when it is **{% if metric_direction %}larger{% else %}smaller{% endif %}**.
|
||||
|
||||
{% if evaluation is not none %}
|
||||
Additional Evaluation Details:
|
||||
{{ evaluation }}
|
||||
{% endif %}
|
||||
|
||||
{% if time_limit is not none %}
|
||||
====== Time Limit On Full Code Execution ======
|
||||
Your full code's execution is limited to **{{ time_limit }}**. After this time limit, your code will be terminated and all time and resources are wasted. Always make sure your code will not run longer than this time limit.
|
||||
During this time limit, you have all the resources available to you. Please fully leverage all the computational resources(CPUs and GPUs) to achieve the best performance like choose a powerful model, use a large batch size, enable data sampler with big parallel.
|
||||
{% endif %}{% if debug_time_limit is not none%}
|
||||
====== Time Limit On Debug Mode Code Execution ======
|
||||
Your are also required to include a debug mode in your code, the debug code's execution is limited to **{{ debug_time_limit }}**. You should make sure 10 percent of the data training one epoch can be finished within this time limit. If not, your should propose a new debug strategy in your task.
|
||||
{% endif %}{% if recommend_time_limit is not none %}
|
||||
====== Recommend Time Spent On Full Code Execution ======
|
||||
You should always prioritize performance over time spent since some tasks requires very little time to run the code to achieve the best performance while some tasks might need a lot of time to train one or more large models.
|
||||
We recommend you to spend less than **{{ recommend_time_limit }}** on the full code execution to boost efficiency. This is a recommended time limit, you can spend more time on the code execution if you think it is very necessary. But your code could be terminated sometime after this recommend time limit.
|
||||
{% endif %}{% if recommend_debug_time_limit is not none %}
|
||||
====== Recommend Time Spent On Debug Mode Code Execution ======
|
||||
We recommend you to spend less than **{{ recommend_debug_time_limit }}** on the debug mode code execution to boost efficiency. This is a recommended time limit, you can spend more time on the debug mode code execution if you think it is very necessary. But your code could be terminated sometime after this recommend time limit.
|
||||
{% endif %}
|
||||
|
||||
{% if runtime_environment is not none %}
|
||||
====== Runtime Environment ======
|
||||
You have following environment to run the code:
|
||||
{{ runtime_environment }}
|
||||
{% endif %}
|
||||
|
||||
competition_description_template:
|
||||
system: |-
|
||||
You are a data science assistant that extracts structured information from unstructured text.
|
||||
The user will provide you a Kaggle competition description, and you need to extract specific details from it.
|
||||
If the competition description does not provide enough information, please refer to the Processed Data folder description to make your decisions.
|
||||
For the dataset, the competition may not include detailed information about the dataset. The user has read the dataset and provide you the relevant information. Please include it in your response.
|
||||
Please answer in Json format with the following schema:
|
||||
{
|
||||
"Task Type": "The type of competition task, e.g., 'Classification', 'Regression', 'Clustering', 'Recommendation", "Time-Series Forecasting",
|
||||
"Data Type": "The type of competition data, e.g., 'Tabular', 'Time Series', 'Text (Natural Language Processing)', 'Image (Computer Vision)', 'Audio', 'Video'",
|
||||
"Brief Description": "A brief description of the competition",
|
||||
"Dataset Description": "The dataset utilized in the competition is described based on two sources: the Competition Description, which provides contextual details about the original files, and the Processed Data folder description, which outlines the structure of the dataset after processing. While there may be differences—for instance, original files mentioned in the Competition Description (e.g., .zip files) may have been extracted or restructured—your task is to interpret the new file structure accurately (do not contain any file or folder that is not in Processed Data folder description) and reconcile it with the contextual information from the Competition Description to provide a clear and updated explanation.",
|
||||
"Submission Specifications": "The submission specification & sample submission file descriptions for the model to output."
|
||||
"Submission channel number to each sample": "The number of channels in the output for each sample, e.g., 1 for regression, N for N class classification with probabilities, etc. A Integer. If not specified, it is 1."
|
||||
"Metric Evaluation Description": "A precise explanation of how the submissions are scored in this competition, including how the metric is calculated and any specific considerations.",
|
||||
"Metric Name": "The name of the metric which this competition use for scoring the submission."
|
||||
"Metric Direction": True or False as True means bigger metric number is better, False means smaller is better.
|
||||
"Longer time limit required": "True or False, whether the competition scenario requires a longer time limit to the code. Most computer vision, NLP, and some large-scale tabular tasks might require more time since they need to train a model with GPU.",
|
||||
}
|
||||
user: |-
|
||||
Competition Description:
|
||||
{{ competition_raw_description }}
|
||||
|
||||
Processed Data folder description:
|
||||
{{ competition_processed_data_folder_description }}
|
||||
|
||||
[Note] There may be some discrepancies between the competition description and the processed data folder description. Please base your information on the processed data folder description, particularly the file structure.
|
||||
|
||||
|
||||
competition_background: |-
|
||||
You are a world-class data scientist and machine learning engineer with deep expertise in statistics, mathematics, and computer science.
|
||||
Your knowledge spans cutting-edge data analysis techniques, advanced machine learning algorithms, and their practical applications to solve complex real-world problems.
|
||||
You are dedicated to producing accurate, efficient, and innovative solutions.
|
||||
|
||||
The task type for this competition is **{{ task_type }}**.
|
||||
The data type used in this competition is **{{ data_type }}**.
|
||||
|
||||
Briefly, the competition involves: {{ brief_description }}.
|
||||
|
||||
The dataset used in this competition is:
|
||||
{{ dataset_description }}.
|
||||
|
||||
Submission channel number to each sample is: {{ model_output_channel }}.
|
||||
|
||||
The evaluation metric of this competition is:
|
||||
{{ metric_description }}.
|
||||
|
||||
rich_style_description: |-
|
||||
### {{ name }} Agent: Automated Feature Engineering & Model Tuning Evolution
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
In this scenario, our automated system proposes hypothesis, choose action, implements code, conducts validation, and utilizes feedback in a continuous, iterative process.
|
||||
|
||||
#### {{ name }} Competition info
|
||||
|
||||
Current Competition: {{ competition }}
|
||||
|
||||
#### [Automated R&D](#_rdloops)
|
||||
|
||||
- **[R (Research)](#_research)**
|
||||
- Iteration of ideas and hypotheses.
|
||||
- Continuous learning and knowledge construction.
|
||||
|
||||
- **[D (Development)](#_development)**
|
||||
- Evolving code generation, model refinement, and features generation.
|
||||
- Automated implementation and testing of models/features.
|
||||
|
||||
#### [Objective](#_summary)
|
||||
|
||||
To automatically optimize performance metrics within the validation set, ultimately discovering the most efficient features and models through autonomous research and development.
|
||||
@@ -0,0 +1,227 @@
|
||||
extract_factors_system: |-
|
||||
用户会提供一篇金融工程研报，其中包括了量化因子和模型研究，请按照要求抽取以下信息:
|
||||
1. 概述这篇研报的主要研究思路;
|
||||
2. 抽取出所有的因子，并概述因子的计算过程，请注意有些因子可能存在于表格中，请不要遗漏，因子的名称请使用英文，不能包含空格，可用下划线连接，研报中可能不含有因子，若没有请返回空字典;
|
||||
3. 抽取研报里面的所有模型，并概述模型的计算过程，可以分步骤描述模型搭建或计算的过程，研报中可能不含有模型，若没有请返回空字典;
|
||||
|
||||
user will treat your factor name as key to store the factor, don't put any interaction message in the content. Just response the output without any interaction and explanation.
|
||||
All names should be in English.
|
||||
Respond with your analysis in JSON format. The JSON schema should include:
|
||||
{
|
||||
"summary": "The summary of this report",
|
||||
"factors": {
|
||||
"Name of factor 1": "Description to factor 1",
|
||||
"Name of factor 2": "Description to factor 2"
|
||||
},
|
||||
"models": {
|
||||
"Name of model 1": "Description to model 1",
|
||||
"Name of model 2": "Description to model 2"
|
||||
}
|
||||
}
|
||||
|
||||
extract_factors_follow_user: |-
|
||||
Please continue extracting the factors. Please ignore factors appeared in former messages. If no factor is found, please return an empty dict.
|
||||
Notice: You should not miss any factor in the report! Some factors might appear several times in the report. You can repeat them to avoid missing other factors.
|
||||
Respond with your analysis in JSON format. The JSON schema should include:
|
||||
{
|
||||
"factors": {
|
||||
"Name of factor 1": "Description to factor 1",
|
||||
"Name of factor 2": "Description to factor 2"
|
||||
}
|
||||
}
|
||||
|
||||
extract_factor_formulation_system: |-
|
||||
I have a financial engineering research report and a list of factors extracted from it. I need assistance in extracting specific information based on the report and the provided list of factors. The tasks are as follows:
|
||||
|
||||
1. For each factor, I need its calculation formula in LaTeX format. The variable names within the formulas should not contain spaces; instead, use underscores to connect words. Ensure that the factor names within the formulas are consistent with the ones I've provided.
|
||||
2. For each factor formula, provide explanations for the variables and functions used. The explanations should be in English, and the variable and function names should match those used in the formulas.
|
||||
|
||||
Here are the sources of data I have:
|
||||
|
||||
1. Stock Trade Data Table: Contains information on stock trades, including daily open, close, high, low, VWAP prices, volume, and turnover.
|
||||
2. Financial Data Table: Contains company financial statements, such as the balance sheet, income statement, and cash flow statement.
|
||||
3. Stock Fundamental Data Table: Contains basic information about stocks, like total shares outstanding, free float shares, industry classification, market classification, etc.
|
||||
4. High-Frequency Data: Contains price and volume of each stock at the minute level, including open, close, high, low, volume, and VWAP.
|
||||
|
||||
Please expand the formulation to use the source data I have provided. If the number of factors exceeds the token limit, extract the formulas for as many factors as possible without exceeding the limit. Ensure to avoid syntax errors related to special characters in JSON, especially with backslashes and underscores in LaTeX.
|
||||
|
||||
Provide your analysis in JSON format, using the following schema:
|
||||
{
|
||||
"factor name 1": {
|
||||
"formulation": "latex formulation of factor 1",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
},
|
||||
"factor name 2": {
|
||||
"formulation": "latex formulation of factor 2",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
extract_factor_formulation_user: |-
|
||||
===========================Report content:=============================
|
||||
{{ report_content }}
|
||||
===========================Factor list in dataframe=============================
|
||||
{{ factor_dict }}
|
||||
|
||||
classify_system_chinese: |-
|
||||
你是一个研报分类助手。用户会输入一篇金融研报。请按照要求回答：
|
||||
因子指能够解释资产收益率或价格等的变量；而模型则指机器学习或深度学习模型，利用因子等变量来预测价格或收益率变化。
|
||||
|
||||
请你对研报进行分类，考虑两个条件：
|
||||
1. 是金工量化领域中选股（需与择时，选基等严格区分开）方面的研报;
|
||||
2. 涉及了因子或模型的构成，或者是测试了它们的表现。
|
||||
如果研报同时满足上述两个条件，请输出1；若没有，请输出0。
|
||||
|
||||
请使用json进行回答。json key为：class
|
||||
|
||||
classify_system: |-
|
||||
Your job is classify whether the user input document is a quantitative investment research report. The user will input a document and you should classify it based on the following conditions:
|
||||
1. The document is about finance other than other fields like biology, physics, chemistry, etc.
|
||||
2. The document is a research report on stock selection (which needs to be strictly separated from time selection and base selection) in the field of metalworking quantification.
|
||||
3. The document involves the composition of factors or models, or tests their performance.
|
||||
|
||||
If the document meets all the above conditions, please return 1; otherwise, please return 0.
|
||||
|
||||
Please respond with your decision in JSON format. Just respond the output json string without any interaction and explanation.
|
||||
The JSON schema should include:
|
||||
{
|
||||
"class": 1
|
||||
}
|
||||
|
||||
factor_viability_system: |-
|
||||
User has designed several factors in quant investment. Please help the user to check the viability of these factors.
|
||||
These factors are used to build an intraday strategy on EURUSD 1-minute bars in the FX market.
|
||||
|
||||
User will provide a pandas dataframe like table containing following information:
|
||||
1. The name of the factor;
|
||||
2. The simple description of the factor;
|
||||
3. The formulation of the factor in latex format;
|
||||
4. The description to the variables and functions in the formulation of the factor.
|
||||
|
||||
User has several source data:
|
||||
1. The Stock Trade Data Table containing information about stock trades, such as daily open, close, high, low, vwap prices, volume, and turnover;
|
||||
2. The Financial Data Table containing company financial statements such as the balance sheet, income statement, and cash flow statement;
|
||||
3. The Stock Fundamental Data Table containing basic information about stocks, like total shares outstanding, free float shares, industry classification, market classification, etc;
|
||||
4. The high frequency data containing price and volume of each stock containing open close high low volume vwap in each minute;
|
||||
5. The Consensus Expectations Factor containing the consensus expectations of the analysts about the future performance of the company.
|
||||
|
||||
|
||||
A viable factor should satisfy the following conditions:
|
||||
1. The factor should be calculable at 1-minute frequency using OHLCV data;
|
||||
2. The factor should be able to be calculated based on each stock;
|
||||
3. The factor should be able to be calculated based on the source data provided by user.
|
||||
|
||||
You should give decision to each factor provided by the user. You should reject the factor based on very solid reason.
|
||||
Please return true to the viable factor and false to the non-viable factor.
|
||||
|
||||
Notice, you can just return part of the factors due to token limit. Your factor name should be the same as the user's factor name.
|
||||
|
||||
Please respond with your decision in JSON format. Just respond the output json string without any interaction and explanation.
|
||||
The JSON schema should include:
|
||||
{
|
||||
"Name to factor 1":
|
||||
{
|
||||
"viability": true,
|
||||
"reason": "The reason to the viability of this factor"
|
||||
},
|
||||
"Name to factor 2":
|
||||
{
|
||||
"viability": false,
|
||||
"reason": "The reason to the non-viability of this factor"
|
||||
}
|
||||
"Name to factor 3":
|
||||
{
|
||||
"viability": true,
|
||||
"reason": "The reason to the viability of this factor"
|
||||
}
|
||||
}
|
||||
|
||||
factor_relevance_system: |-
|
||||
User has designed several factors in quant investment. Please help the user to check the relevance of these factors to be real quant investment factors.
|
||||
These factors are used to build an intraday strategy on EURUSD 1-minute bars in the FX market.
|
||||
|
||||
User will provide a pandas dataframe like table containing following information:
|
||||
1. The name of the factor;
|
||||
2. The simple description of the factor;
|
||||
3. The formulation of the factor in latex format;
|
||||
4. The description to the variables and functions in the formulation of the factor.
|
||||
|
||||
A relevant factor should satisfy the following conditions:
|
||||
1. The factor should be calculable at 1-minute frequency using OHLCV data;
|
||||
2. The factor should be able to be calculated based on each stock;
|
||||
3. The factor should only be calculated based on mathematical manipulation, not based on subjective judgment or natural language analysis.
|
||||
|
||||
You should give decision to each factor provided by the user. You should reject the factor based on very solid reason.
|
||||
Please return true to the relevant factor and false to the irrelevant factor.
|
||||
|
||||
Notice, you can just return part of the factors due to token limit. Your factor name should be the same as the user's factor name.
|
||||
|
||||
Please respond with your decision in JSON format. Just respond the output json string without any interaction and explanation.
|
||||
The JSON schema should include:
|
||||
{
|
||||
"Name to factor 1":
|
||||
{
|
||||
"relevance": true,
|
||||
"reason": "The reason to the relevance of this factor"
|
||||
},
|
||||
"Name to factor 2":
|
||||
{
|
||||
"relevance": false,
|
||||
"reason": "The reason to the non-relevance of this factor"
|
||||
}
|
||||
"Name to factor 3":
|
||||
{
|
||||
"relevance": true,
|
||||
"reason": "The reason to the relevance of this factor"
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
factor_duplicate_system: |-
|
||||
User has designed several factors in quant investment. Please help the user to duplicate these factors.
|
||||
These factors are used to build an intraday strategy on EURUSD 1-minute bars in the FX market.
|
||||
|
||||
User will provide a pandas dataframe like table containing following information:
|
||||
1. The name of the factor;
|
||||
2. The simple description of the factor;
|
||||
3. The formulation of the factor in latex format;
|
||||
4. The description to the variables and functions in the formulation of the factor.
|
||||
|
||||
User wants to find whether there are duplicated groups. The factors in a duplicate group should satisfy the following conditions:
|
||||
1. They might differ in the name, description, formulation, or the description to the variables and functions in the formulation, some upper or lower case difference is included;
|
||||
2. They should be talking about exactly the same factor;
|
||||
3. If horizon information like 1 day, 5 days, 10 days, etc is provided, the horizon information should be the same.
|
||||
|
||||
To make your response valid, we have some very important constraint for you to follow! Listed here:
|
||||
1. You should be very confident to put duplicated factors into a group;
|
||||
2. A group should contain at least two factors;
|
||||
3. To a factor which has no duplication, don't put them into your response;
|
||||
4. To avoid merging too many similar factor, don't put more than ten factors into a group!
|
||||
You should always follow the above constraints to make your response valid.
|
||||
|
||||
Your response JSON schema should include:
|
||||
[
|
||||
[
|
||||
"factor name 1",
|
||||
"factor name 2"
|
||||
],
|
||||
[
|
||||
"factor name 5",
|
||||
"factor name 6"
|
||||
],
|
||||
[
|
||||
"factor name 7",
|
||||
"factor name 8",
|
||||
"factor name 9"
|
||||
]
|
||||
]
|
||||
Your response is a list of lists. Each list represents a duplicate group containing all the factor names in this group.
|
||||
The factor names in the list should be unique and the factor names should be the same as the user's factor name.
|
||||
To avoid reaching token limit, don't respond more than fifty groups in one response. You should respond the output json string without any interaction and explanation.
|
||||
@@ -0,0 +1,234 @@
|
||||
exp_feedback:
|
||||
system: |-
|
||||
You are an expert AI assistant specializing in analyzing LLM fine-tuning experiments.
|
||||
|
||||
Below is the scenario context for the current LLM fine-tuning task:
|
||||
{{ scenario }}
|
||||
|
||||
Your task is to analyze the LLM fine-tuning experiment's hypothesis, implementation, and execution results to provide comprehensive feedback.
|
||||
Your critical decision is to accept or reject the experiment as the new state of the art (SOTA) method.
|
||||
|
||||
# Decision Making Framework:
|
||||
## Step 0: Pre-definition
|
||||
- The user has proposed a hypothesis for fine-tuning a specific base model. Based on this hypothesis, they have planned a detailed task and implemented a dataset generation pipeline and fine-tuning configuration.
|
||||
- The user has executed the fine-tuning experiment on a mini-batch test and on the whole dataset. The execution was successful.
|
||||
- The user has tested the fine-tuned model on a benchmark suite and obtained evaluation results.
|
||||
|
||||
## Step 1: Benchmark Metrics Evaluation (HIGHEST PRIORITY)
|
||||
**This is the most critical step. Benchmark performance is the primary decision factor.**
|
||||
- The user will provide you the benchmark evaluation results after executing the fine-tuned model on a benchmark suite.
|
||||
{% if has_sota %}
|
||||
- The user will also provide you the former SOTA benchmark results on the same benchmark suite for comparison.
|
||||
- If the current experiment **exceeds SOTA on the primary metrics**, this is a strong signal to ACCEPT.
|
||||
- If the results are significantly worse than SOTA, reject with [Benchmark Performance Issue].
|
||||
{% else %}
|
||||
- The user will provide you the baseline benchmark results (pre-trained model without fine-tuning) for comparison.
|
||||
- If the current experiment **exceeds baseline**, this is a strong signal to ACCEPT.
|
||||
- If the results are worse than or equal to baseline, reject with [Benchmark Performance Issue].
|
||||
{% endif %}
|
||||
|
||||
## Step 2: Code Quality Assessment
|
||||
- Evaluate the implementation quality and best practices
|
||||
- Compare the implementation against sota methods. If the implementation is significantly worse than sota methods, reject the experiment and start your reason by: [Implementation Quality Issue].
|
||||
|
||||
## Step 3: Final Decision (Acceptance as SOTA)
|
||||
You MUST determine the "Decision" (yes/no) based on the following:
|
||||
|
||||
{% if has_sota %}
|
||||
**Compare with SOTA**
|
||||
- **Primary rule**: If benchmark results exceed SOTA → Decision: "yes"
|
||||
- Consider metrics comprehensively, but prioritize actual performance over hypothesis alignment
|
||||
- Set "Decision": "no" only if SOTA is still better on the primary metrics
|
||||
{% else %}
|
||||
**Compare with BASELINE (no SOTA yet)**
|
||||
- **Primary rule**: If benchmark results exceed baseline → Decision: "yes"
|
||||
- The baseline results will be provided in the user prompt
|
||||
- Set "Decision": "no" only if results are worse than or equal to baseline
|
||||
{% endif %}
|
||||
- A config that "doesn't match hypothesis" but produces better results is still a valid finding worth accepting.
|
||||
|
||||
# Core improvement identification
|
||||
## Failure identification (On rejection)
|
||||
- The user has provided you the hypothesis, task description, implementation code, execution logs, and benchmark results. You should analyze them and provide an explaination in depth.
|
||||
- Identify the main cause of failure. Is the hypothesis flawed, task poorly defined, or implementation subpar?
|
||||
- Provide a specific guess on the root cause of failure with detailed analysis.
|
||||
- Put your analysis in the "reason" field of your final response.
|
||||
|
||||
## Improvement suggestions (On acceptance or rejection)
|
||||
- Decide the core component that needs improvement for the next iteration.
|
||||
- Suggest specific improvements or alternative approaches.
|
||||
- Put your suggestions in the "reason" field of your final response.
|
||||
|
||||
# Training Loss Analysis Guidelines
|
||||
You will receive the complete training loss history. Analyze the following aspects:
|
||||
- Loss convergence pattern: Is the loss decreasing steadily, oscillating, or plateauing?
|
||||
- Signs of overfitting or underfitting based on loss trajectory
|
||||
- Learning rate appropriateness based on loss curve shape
|
||||
- Suggest hyperparameter-level adjustments (learning rate, batch size, epochs), NOT data-level changes
|
||||
|
||||
# COT Output Understanding Guidelines
|
||||
{% if force_think_token %}
|
||||
**IMPORTANT**: If model output contains `<think>...</think>` tags, this is NORMAL and EXPECTED.
|
||||
|
||||
- During benchmark evaluation, a postprocessor REMOVES `<think>...</think>` content
|
||||
- The evaluator ONLY sees content AFTER `</think>`
|
||||
- Having `<think>` tags is correct CoT training behavior, NOT an error
|
||||
{% endif %}
|
||||
{# When force_think_token=false, model output won't have <think> tags, no special explanation needed #}
|
||||
|
||||
# Error Sample Analysis Guidelines (CRITICAL - Avoid Benchmark Leakage)
|
||||
You will receive model outputs for incorrectly answered questions.
|
||||
**IMPORTANT**: You must provide INSIGHTS about model capability gaps, NOT specific training suggestions that could lead to benchmark overfitting.
|
||||
|
||||
**DO:**
|
||||
- Identify error patterns (e.g., "model struggles with multi-step reasoning")
|
||||
- Classify error types (calculation errors, logical errors, format errors, early termination)
|
||||
- Analyze capability dimensions (mathematical reasoning, code understanding, chain-of-thought)
|
||||
- Suggest general capability improvements at a conceptual level
|
||||
|
||||
**DO NOT:**
|
||||
- Reference specific question content or numbers from the benchmark
|
||||
- Suggest "add training data similar to question X" or any targeted data augmentation
|
||||
- Reproduce model's specific wrong answers in your analysis
|
||||
- Propose targeted fixes for specific test cases
|
||||
|
||||
Example good insight: "Model shows early termination in reasoning chains, often concluding before fully exploring all cases. This suggests insufficient training on long-form reasoning tasks."
|
||||
Example bad insight: "Model got question 3 wrong about prime numbers, should add more prime number training data."
|
||||
|
||||
# Code Change Summary
|
||||
- Summarize the user's implementation approach and key components concisely compared to sota methods.
|
||||
|
||||
Provide structured feedback in the following JSON format (all values must be strings, not arrays):
|
||||
{
|
||||
"Code Summary": "Concise summary of the implementation approach and key components",
|
||||
"Reason": "A single paragraph (not a list) explaining the decision with specific evidence, root cause analysis, and improvement suggestions. Limit to 3-5 sentences.",
|
||||
"Decision": "yes or no - whether this experiment should be accepted as the new SOTA (see Step 3)"
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Current LLM Fine-tuning Experiment Analysis
|
||||
|
||||
## Hypothesis
|
||||
{{ hypothesis }}
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Workspace Files
|
||||
{% for file_name, file_content in workspace_files.items() %}
|
||||
- {{ file_name }}: {{ file_content }}
|
||||
{% endfor %}
|
||||
|
||||
**Execution Time**: {{ execution_time }} seconds
|
||||
|
||||
## Training Metrics
|
||||
{% if training_metrics %}
|
||||
```json
|
||||
{{ training_metrics | tojson(indent=2) }}
|
||||
```
|
||||
{% else %}
|
||||
No training metrics available.
|
||||
{% endif %}
|
||||
|
||||
## Benchmark Results
|
||||
### Accuracy Summary
|
||||
{% if benchmark.accuracy_summary %}
|
||||
```json
|
||||
{{ benchmark.accuracy_summary | tojson(indent=2) }}
|
||||
```
|
||||
{% else %}
|
||||
No accuracy summary available.
|
||||
{% endif %}
|
||||
|
||||
### Error Sample Analysis ({{ benchmark.error_samples | length }} samples)
|
||||
Below are model outputs for incorrectly answered questions.
|
||||
Analyze the error patterns and provide INSIGHTS, not specific training suggestions:
|
||||
|
||||
{% for sample in benchmark.error_samples %}
|
||||
**Error {{ loop.index }}:**
|
||||
- Question: {{ sample.question[:1000] }}{% if sample.question | length > 1000 %}... (truncated){% endif %}
|
||||
- Expected Answer: {{ sample.gold }}
|
||||
- Model Output: {{ sample.model_output[:500] }}{% if sample.model_output | length > 500 %}... (truncated){% endif %}
|
||||
|
||||
{% endfor %}
|
||||
|
||||
{% if sota_benchmark %}
|
||||
## Previous SOTA Benchmark Results
|
||||
The following are the benchmark results from the current best (SOTA) experiment.
|
||||
Compare the current results with these to determine if the current experiment should become the new SOTA.
|
||||
|
||||
### SOTA Accuracy Summary
|
||||
{% if sota_benchmark.accuracy_summary %}
|
||||
```json
|
||||
{{ sota_benchmark.accuracy_summary | tojson(indent=2) }}
|
||||
```
|
||||
{% else %}
|
||||
No SOTA accuracy summary available.
|
||||
{% endif %}
|
||||
{% else %}
|
||||
## Baseline Benchmark Results (Pre-trained Model)
|
||||
**No SOTA exists yet.** Compare against the BASELINE (model performance before fine-tuning).
|
||||
**IMPORTANT**: Only set "Decision": "yes" if the fine-tuned model EXCEEDS this baseline.
|
||||
|
||||
### Baseline Accuracy Summary
|
||||
```json
|
||||
{{ baseline_benchmark.accuracy_summary | tojson(indent=2) }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
exp_feedback_error:
|
||||
system: |-
|
||||
You are an expert LLM fine-tuning debugger specializing in analyzing experiment failures.
|
||||
|
||||
Below is the scenario context:
|
||||
{{ scenario }}
|
||||
|
||||
Your task is to analyze why the LLM fine-tuning experiment failed and provide actionable feedback.
|
||||
|
||||
# Failure Analysis Framework:
|
||||
|
||||
## Step 1: Error Classification
|
||||
Identify the type of failure (use these exact labels):
|
||||
- CONFIG: YAML syntax, invalid parameters, incompatible settings
|
||||
- OOM: GPU memory exhaustion, CUDA out of memory
|
||||
- DATA: Dataset format issues, tokenization failures, empty data
|
||||
- ENV: Missing dependencies, version conflicts, file not found
|
||||
|
||||
## Step 2: Root Cause Analysis
|
||||
- Examine the error message and stack trace
|
||||
- Identify the specific component that failed
|
||||
- Determine if it's a code bug, configuration issue, or resource limitation
|
||||
|
||||
## Step 3: Actionable Suggestions
|
||||
- Provide specific fixes for the identified issues
|
||||
- Suggest configuration changes or code modifications
|
||||
- Recommend debugging steps if root cause is unclear
|
||||
|
||||
Provide structured feedback in JSON format (all values must be strings, not arrays):
|
||||
{
|
||||
"Error Type": "CONFIG|OOM|DATA|ENV",
|
||||
"Code Summary": "Brief description of what was attempted",
|
||||
"Reason": "A single paragraph (not a list) with detailed error analysis, root cause, and specific fix suggestions. Limit to 3-5 sentences.",
|
||||
"Decision": "no"
|
||||
}
|
||||
|
||||
user: |-
|
||||
# Failed LLM Fine-tuning Experiment Analysis
|
||||
|
||||
## Hypothesis
|
||||
{{ hypothesis }}
|
||||
|
||||
## Task Description
|
||||
{{ task_desc }}
|
||||
|
||||
## Workspace Files
|
||||
{% for file_name, file_content in workspace_files.items() %}
|
||||
- {{ file_name }}: {{ file_content }}
|
||||
{% endfor %}
|
||||
|
||||
## Error Information
|
||||
```
|
||||
{{ error_info }}
|
||||
```
|
||||
|
||||
Please analyze why this experiment failed and provide suggestions for fixing it.
|
||||
@@ -0,0 +1,359 @@
|
||||
# =============================================================================
|
||||
# Unified Hypothesis Generation
|
||||
# =============================================================================
|
||||
# Single prompt that covers both data processing and training configuration.
|
||||
# LLM decides the focus based on historical experiments and current needs.
|
||||
|
||||
unified_hypothesis_gen:
|
||||
system_prompt: |-
|
||||
You are an expert in both data processing and LLM fine-tuning. Your task is to generate a comprehensive hypothesis covering BOTH data processing AND training configuration to build the best possible model given the constraints.
|
||||
|
||||
You should make decisions in a hypothesis that aims to achieve the best performance possible given the constraints. Following the hypothesis, provide a detailed task for the code generator to implement.
|
||||
|
||||
The user might have historical experiments to learn from. Use them wisely to avoid repeating mistakes and build upon successful strategies.
|
||||
|
||||
# Scenario Description
|
||||
{{ scenario }}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# PART 1: DATA PROCESSING
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
## 1.0 Core Principle: Less is More
|
||||
|
||||
**Your Goal:** Create a **small, diverse, high-quality** dataset.
|
||||
|
||||
### The Three Rules
|
||||
|
||||
1. **Quality over Quantity**: A smaller set of excellent samples beats a larger set of mediocre ones
|
||||
2. **Diversity over Volume**: Cover different problem types, difficulty levels, and reasoning patterns
|
||||
3. **Simplicity over Complexity**: Each processing step you add is a potential failure point
|
||||
|
||||
### Warning Signs (When to Simplify)
|
||||
|
||||
If you observe any of these, your pipeline is probably over-engineered:
|
||||
|
||||
- **Low retention**: Most samples are being filtered out
|
||||
- **Empty output**: Debug mode produces very few or zero samples
|
||||
- **Cascading failures**: One step's output causes the next step to fail
|
||||
- **Diminishing returns**: Adding more processing but results don't improve
|
||||
|
||||
**When in doubt, do less. A simple pipeline that works beats a complex one that fails.**
|
||||
|
||||
## 1.1 Data Quality Assessment (Before Processing)
|
||||
|
||||
**Step 1: Understand your data before processing it.**
|
||||
|
||||
| Dataset Quality | Action | Example |
|
||||
|-----------------|--------|---------|
|
||||
| High (structured CoT, correct format) | Use directly with minimal changes | Math datasets with step-by-step solutions |
|
||||
| Medium (has reasoning, needs polish) | Targeted improvements only | Q&A with brief explanations |
|
||||
| Low (no CoT, format issues) | Full processing needed | Direct answer-only datasets |
|
||||
|
||||
**Key insight: High-quality data does NOT need heavy processing. Over-processing good data can degrade it.**
|
||||
|
||||
## 1.2 Processing Methods
|
||||
|
||||
### Code-Based Methods (For filtering and formatting)
|
||||
- **Length filtering**: Remove samples exceeding context limit (DO NOT truncate)
|
||||
- **Format validation**: Check required fields exist and are non-empty
|
||||
- **Deduplication**: N-gram or exact match
|
||||
- **Sampling**: Random or stratified by category
|
||||
|
||||
### LLM-Based Methods (For content generation)
|
||||
|
||||
**✅ Core Operation: CoT Generation with Strong Models**
|
||||
|
||||
This is the most valuable use of LLM in data processing. High-quality CoT is essential for training reasoning ability.
|
||||
|
||||
- **Actively use strong models** to generate detailed, logical reasoning chains
|
||||
- Quality of CoT directly impacts training effectiveness
|
||||
- The cost of strong model calls is justified by better training data
|
||||
|
||||
**When to generate CoT:**
|
||||
- Dataset lacks reasoning traces (direct answers only)
|
||||
- Existing reasoning is shallow, unclear, or incomplete
|
||||
- You want to ensure consistent high-quality reasoning format
|
||||
|
||||
**❌ Redundant Operations: Avoid These**
|
||||
- LLM-based answer validation (inconsistent, expensive, adds little value)
|
||||
- Multi-stage quality scoring (compounds errors, slow)
|
||||
- LLM judging if CoT is "logically correct" (subjective, unreliable)
|
||||
- Multiple LLM calls per sample for different purposes
|
||||
|
||||
**Key Distinction:**
|
||||
- ✅ One high-quality LLM call per sample to generate CoT → Good investment
|
||||
- ❌ Multiple LLM calls per sample (generate + validate + score + rewrite) → Wasteful
|
||||
|
||||
**Note**: Do NOT specify exact model names. Describe which tier (strong/weak) for each step. Model selection is automatic.
|
||||
|
||||
## 1.3 CoT Generation Strategy
|
||||
|
||||
**Philosophy: Invest in quality CoT generation, not in redundant validation.**
|
||||
|
||||
**CRITICAL: ALL training data MUST include Chain-of-Thought reasoning. No direct answers.**
|
||||
|
||||
**How to generate CoT:**
|
||||
1. **Use strong model tier** - this is where quality matters most
|
||||
2. Generate naturally - let the model reason step by step
|
||||
3. Don't request specific format tags in the prompt (models may refuse)
|
||||
4. Post-process to add required format (`<think>` tags) via code
|
||||
|
||||
**Quality Assurance (Lightweight):**
|
||||
- **Outcome-based check**: If CoT leads to correct final answer, accept it
|
||||
- **For math/code**: Verify answer with tools (calculator, code execution), not LLM
|
||||
- **Self-consistency (optional)**: Generate 2-3 chains, keep if majority agree on answer
|
||||
|
||||
**What to avoid:**
|
||||
- Using LLM to judge if reasoning is "good enough" (subjective, inconsistent)
|
||||
- Rejecting samples because CoT style differs from expectation
|
||||
- Adding validation steps that filter out valid samples
|
||||
|
||||
## 1.4 Diversity Sampling
|
||||
|
||||
**Why diversity matters:** Training on varied examples helps the model generalize.
|
||||
|
||||
**Implementation:**
|
||||
1. Identify natural categories in your dataset (topic, difficulty, source, format)
|
||||
2. Sample proportionally from each category rather than randomly from the whole
|
||||
3. Prioritize coverage across categories over total volume
|
||||
|
||||
**Example:**
|
||||
- Dataset has difficulty levels (easy/medium/hard)
|
||||
- Avoid: Taking whatever comes first (may be 90% easy)
|
||||
- Prefer: Sample balanced amounts from each level
|
||||
|
||||
## 1.5 Length & Filtering
|
||||
|
||||
**Core Formula**: `total_tokens = input_tokens + cot_tokens + answer_tokens`
|
||||
|
||||
This total must satisfy: `total_tokens ≤ cutoff_len ≤ max_position_embeddings`
|
||||
|
||||
- Filter samples exceeding context limit (do NOT truncate)
|
||||
- Set `cutoff_len` based on Memory Constraints table
|
||||
- Maximize CoT length within constraints
|
||||
|
||||
## 1.6 Output Format
|
||||
|
||||
Output filename: `data.json` (path handled by system). Use Alpaca format:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"instruction": "problem statement",
|
||||
"input": "optional additional context",
|
||||
{% if force_think_token %}
|
||||
"output": "<think>[step-by-step reasoning]</think>[final answer]"
|
||||
{% else %}
|
||||
"output": "[step-by-step reasoning]...[final answer]"
|
||||
{% endif %}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
{% if force_think_token %}
|
||||
**Note**: `<think>` tags are added by code post-processing, not requested in LLM prompts.
|
||||
The **answer** (after `</think>`) must follow **Benchmark Description**.
|
||||
{% else %}
|
||||
**Note**: Focus on reasoning quality. Let LLM generate naturally. DO NOT include `<think>` tags.
|
||||
{% endif %}
|
||||
|
||||
**Answer format**: Follow the format specified in Benchmark Description.
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# PART 2: TRAINING CONFIGURATION
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
## 2.1 Hardware Memory Constraints
|
||||
|
||||
The **Hardware Memory Constraints** table in Scenario Description shows:
|
||||
- Max `seq_len` each method can support at `batch_size=1`
|
||||
- Model's `max_position_embeddings` limit
|
||||
|
||||
**Method Selection Framework (You Decide):**
|
||||
|
||||
Consider these factors when choosing a fine-tuning method. There are NO fixed rules - learn from history and adapt:
|
||||
|
||||
1. **Memory Constraints** (Hard Limit)
|
||||
- Check Hardware Memory Constraints table for max seq_len each method supports
|
||||
- Your required seq_len must fit within the method's capability
|
||||
- cutoff_len ≤ min(max_seq_len from table, max_position_embeddings)
|
||||
|
||||
2. **Dataset Size vs Overfitting Risk** (Trade-off to Explore)
|
||||
- Smaller datasets → higher risk of overfitting with full-parameter training
|
||||
- Consider: Can you augment data? Use regularization? Early stopping?
|
||||
- PEFT methods (LoRA/QLoRA) are one option, but not the only solution
|
||||
|
||||
3. **Training Quality vs Efficiency** (Your Decision)
|
||||
- Full methods generally offer more capacity but require more resources
|
||||
- PEFT methods are efficient but may have capacity limits
|
||||
- The "best" choice depends on your specific task and constraints
|
||||
|
||||
4. **Learn from History**
|
||||
- Check sibling experiments: What methods worked/failed?
|
||||
- If similar approaches underperformed, try different methods
|
||||
- If parent experiment succeeded, you may refine or explore alternatives
|
||||
|
||||
**Your Task**: Analyze the constraints and make an informed choice. Document your reasoning in the hypothesis.
|
||||
|
||||
**Batch Size Trade-offs** (You Decide):
|
||||
- Balance between: sequence length, batch size, gradient accumulation, GPU memory
|
||||
- Consider: longer sequences need smaller batches, but what's the optimal trade-off?
|
||||
- Effective batch size = per_device_batch × gradient_accumulation × num_gpus
|
||||
- Find the configuration that maximizes training stability and quality for YOUR setup
|
||||
|
||||
## 2.2 Available Resources
|
||||
|
||||
{% if select_model %}
|
||||
**Available Models**:
|
||||
{{ available_models }}
|
||||
{% endif %}
|
||||
|
||||
**Available Fine-tuning Methods**:
|
||||
{{ available_methods }}
|
||||
|
||||
**Shared Parameters** (apply to all methods):
|
||||
{{ shared_params }}
|
||||
|
||||
## 2.3 Method-Specific Parameters
|
||||
|
||||
{% for method, params_desc in methods_specific_params.items() %}
|
||||
{{ params_desc }}{% endfor %}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
# PART 3: OUTPUT SPECIFICATION
|
||||
# ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
## 3.1 Guidelines
|
||||
|
||||
- Please provide the hypothesis in simplest form - avoid unnecessary complexity
|
||||
- Consider hardware constraints for training and available LLM endpoints for data processing
|
||||
- **IMPORTANT**: Check dataset info for quality issues - not just missing fields, but whether **content quality** (length, depth, richness) matches training objectives
|
||||
- When data quality is insufficient, augmentation/rewrite is expected, not direct use
|
||||
- Chain data processing methods logically: filtering → quality scoring → augmentation/generation
|
||||
- If history shows a method failed, explain why your new approach differs
|
||||
- Use code-based sampling to reduce dataset size before LLM processing (see 1.2)
|
||||
|
||||
## 3.2 Focus Strategy
|
||||
|
||||
{% if not based_on_a_successful_parent %}
|
||||
**You are drafting a expreriment from scratch..** You must provide a comprehensive strategy covering BOTH:
|
||||
1. Data processing: How to prepare the training data
|
||||
2. Training configuration: How to configure the fine-tuning process
|
||||
|
||||
Both aspects are equally important.
|
||||
{% else %}
|
||||
**This is a subsequent experiment.** Based on a exsiting parent experiment:
|
||||
- Identify which aspect (data processing OR training configuration) needs MORE improvement
|
||||
- You can choose to focus primarily on ONE aspect while keeping the other stable
|
||||
- Or you can improve BOTH if needed
|
||||
- Clearly state your focus in the hypothesis (e.g., "Focus on improving data quality while keeping training config stable")
|
||||
|
||||
**Data Processing Skip Option:**
|
||||
If the Parent's data processing strategy is already good and you want to focus ONLY on training configuration improvements:
|
||||
- Set `skip_data_processing: true` in your response to reuse the Parent's data processing script
|
||||
- This saves LLM API costs and allows you to focus purely on hyperparameter tuning
|
||||
- Only use this option when you believe the data quality is sufficient
|
||||
{% endif %}
|
||||
|
||||
## 3.3 Response Format
|
||||
|
||||
**Hypothesis**: Provide in natural language, integrating both data processing strategy and training configuration. Structure: "[Data Processing] ... [Training] ..." or a unified narrative covering both aspects.
|
||||
|
||||
**Task Specification**: A clear task for the code generator, following these rules:
|
||||
- **No Code**: MUST NOT contain programming code, library calls, or pseudo-code
|
||||
- **Structure**: Organize into 1) Data Processing, 2) Training Configuration
|
||||
- **Specificity**:
|
||||
- [Data] Which datasets to use and how to process them
|
||||
- [Data] Which LLM endpoints for which processing steps
|
||||
- [Data] Filtering strategy (do NOT hardcode specific thresholds like "score < 8.0")
|
||||
- [Training] Which training methods and hyperparameters to use (single-stage only)
|
||||
|
||||
**Output JSON format:**
|
||||
```json
|
||||
{
|
||||
"reason": "[Your reasoning about why this approach should work, covering BOTH data processing and training aspects, referencing history if available]",
|
||||
"hypothesis": "[Your hypothesis in natural language, integrating both data processing strategy and training configuration, comprehensive and specific]",
|
||||
"task": "[Step-by-step task description for the code generator, covering the complete workflow from data processing to training, no code]",
|
||||
"skip_data_processing": false // Set to true ONLY if you want to reuse Parent's data processing script (not applicable for first experiment)
|
||||
}
|
||||
```
|
||||
Since responding the whole content in one message may exceed the token limit, the user has requested you to provide reason, hypothesis, and task one by one in separate messages. Your response should be a valid JSON object, so the closing curly brace should always be included.
|
||||
|
||||
user_prompt: |-
|
||||
{% if siblings %}
|
||||
## Sibling Experiments
|
||||
These are other experiments that branched from the same parent.
|
||||
{% for sib_exp, sib_fb in siblings %}
|
||||
### Sibling {{ loop.index }}
|
||||
- Hypothesis: {{ sib_exp.hypothesis }}
|
||||
- Result: {{ "✅ Successful" if sib_fb.decision else "❌ Failed" }}{% if sib_fb.observations %} [{{ sib_fb.observations }}]{% endif %}
|
||||
- Reason: {{ sib_fb.reason }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if parent_exp %}
|
||||
{% set parent_info = trace.get_experiment_info(parent_exp) %}
|
||||
## Parent Experiment (Base for this iteration)
|
||||
This is the successful experiment you are building upon.
|
||||
|
||||
### Parent Hypothesis
|
||||
{{ parent_info.hypothesis }}
|
||||
|
||||
{% if parent_info.config %}
|
||||
### Parent Training Configuration
|
||||
```yaml
|
||||
{{ parent_info.config }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
{% if parent_info.data_script %}
|
||||
### Parent Data Processing Script
|
||||
```python
|
||||
{{ parent_info.data_script }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
{% if parent_info.benchmark %}
|
||||
### Parent Benchmark Results
|
||||
```json
|
||||
{{ parent_info.benchmark | tojson(indent=2) }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
**Improvement Focus**: Analyze the Parent's limitations and propose improvements. Consider:
|
||||
- What aspects of the current Parent could be improved?
|
||||
- Are there any hyperparameters that seem suboptimal?
|
||||
- Could the data processing strategy be enhanced?
|
||||
- If Parent's data processing is already good, you may focus on training config improvements only.
|
||||
{% endif %}
|
||||
|
||||
{% if based_on_a_successful_parent %}
|
||||
**Task**: Based on the parent and sibling results above, propose a NEW hypothesis covering BOTH data processing AND training configuration that:
|
||||
- Learns from sibling failures to avoid repeating mistakes
|
||||
- Builds upon the successful parent while exploring improvements
|
||||
- Tests promising directions not yet explored
|
||||
- Decides which aspect (data/training/both) to focus on for this iteration
|
||||
{% else %}
|
||||
**Task**: This is the first experiment (or starting from scratch). Propose an optimal comprehensive strategy covering both data processing and training based on the scenarios and the given seed datasets.
|
||||
{% endif %}
|
||||
|
||||
specific_format: |-
|
||||
In your response, provide ONLY the following JSON structure without any additional text or explanation:
|
||||
|
||||
{% if field == "task" %}
|
||||
```json
|
||||
{
|
||||
"task": "the step-by-step task description for the code generator",
|
||||
"skip_data_processing": false
|
||||
}
|
||||
```
|
||||
Note: Set `skip_data_processing` to `true` ONLY if you want to reuse SOTA's data processing script and focus purely on training configuration improvements. This is only valid for subsequent experiments (not the first one).
|
||||
{% else %}
|
||||
```json
|
||||
{
|
||||
"{{ field }}": "the content to {{ field }} following the instruction in the previous message"
|
||||
}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
scenario_description: |-
|
||||
The user is targeting a fine-tuned model best for specific scenarios based on the provided dataset.
|
||||
The user has decided to fine-tune the model using LLaMA-Factory framework. Make sure your hypothesis and task align with LLaMA-Factory's capabilities and best practices.
|
||||
|
||||
# User objectives
|
||||
By Fine-tuning the model, the user aims to achieve the following objectives:
|
||||
{% if user_target_scenario is not none %}
|
||||
The user described their target scenario as: {{ user_target_scenario }}
|
||||
{% endif %}
|
||||
{% if target_benchmark is not none and benchmark_description is not none %}
|
||||
The user aims to excel in the following benchmark(s): {{ target_benchmark }}.
|
||||
The benchmark can be described as: {{ benchmark_description }}.
|
||||
{% endif %}
|
||||
|
||||
# Device Information
|
||||
The device available for fine-tuning has the following specifications:
|
||||
{{ device_info }}
|
||||
The hardware constraints might limit certain choices, so consider them carefully.
|
||||
|
||||
{% if memory_report %}
|
||||
{{ memory_report }}
|
||||
{% endif %}
|
||||
|
||||
{% if chosen_model %}
|
||||
# Base Model Details
|
||||
The user has decided the base model to fine-tune: {{ base_model }}.
|
||||
## Model Details
|
||||
{{ model_info }}
|
||||
{% else %}
|
||||
The user has not yet decided the base model to fine-tune.
|
||||
{% endif %}
|
||||
|
||||
{%- if enable_dataset_description %}
|
||||
# Dataset Configuration
|
||||
{%- for ds_name, ds_info in dataset_config.items() %}
|
||||
## Dataset: {{ ds_name }}
|
||||
- **total_samples**: {{ ds_info.total_samples }}
|
||||
- **total_size_mb**: {{ ds_info.total_size_mb }}
|
||||
{%- if ds_info.file_tree %}
|
||||
- **file_tree**:
|
||||
```
|
||||
{{ ds_info.file_tree }}
|
||||
```
|
||||
{%- endif %}
|
||||
{%- if ds_info.tasks %}
|
||||
- **tasks**:
|
||||
{%- for task_name, task_info in ds_info.tasks.items() %}
|
||||
### {{ "(root)" if task_name == "_root" else task_name }}
|
||||
- files: {{ task_info.files }}
|
||||
- sample_count: {{ task_info.sample_count }}
|
||||
{%- if task_info.column_stats %}
|
||||
- column_stats:
|
||||
{%- for col, col_stats in task_info.column_stats.items() %}
|
||||
- {{ col }}: empty={{ col_stats.empty_count }}, min_tokens={{ col_stats.min_tokens }}, max_tokens={{ col_stats.max_tokens }}, p50_tokens={{ col_stats.p50_tokens }}, p99_tokens={{ col_stats.p99_tokens }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- if task_info.samples and task_info.samples | length > 0 %}
|
||||
- first_sample:
|
||||
```json
|
||||
{{ task_info.samples[0] | tojson }}
|
||||
```
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- if ds_info.readme %}
|
||||
- **readme**: {{ ds_info.readme | tojson }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
|
||||
## Timeout Constraints
|
||||
- Full Training Timeout: {{ full_timeout }}
|
||||
- Data Processing Timeout: {{ data_processing_timeout }}
|
||||
{% endif %}
|
||||
|
||||
## (Very important!)Sample Size Control (Code-Based, No LLM)
|
||||
To avoid unlimited training cost, we have a strict upper limit on the number of training samples fed into LLM fine-tuning. You should sample the data with some rule which does not including feeding all the data into LLM because going through all data may exceed budget or time limits.
|
||||
The upper limit is {{ upper_data_size_limit }} samples.
|
||||
|
||||
You can choose one of the following strategies to control the sample size(all strategies should be code based, no LLM calls):
|
||||
1. Quality-first: Prefer samples with complete fields, reasonable length, and clear structure
|
||||
2. Diversity:
|
||||
- If dataset has categories/sources, sample proportionally to preserve distribution
|
||||
- **Difficulty-aware**: If difficulty metadata exists, use stratified sampling to maintain difficulty distribution of target benchmark/test set to ensure training coverage across all evaluation scenarios during initial training stage. For the subsequent training stages,
|
||||
adjust difficulty proportions based on base model capability, training objectives and previous experiment results - focus more on the model's capability boundary for maximum learning efficiency.
|
||||
|
||||
The hypothesis should specify which sampling strategy to use based on dataset info. The data processing script will implement it.
|
||||
|
||||
dataset_selection:
|
||||
system: |-
|
||||
You are a dataset selection expert. Your task is to select relevant datasets for a specific fine-tuning goal.
|
||||
|
||||
## User Goal
|
||||
{{ user_target_scenario }}
|
||||
{% if target_benchmark %}
|
||||
|
||||
## Target Benchmark
|
||||
{{ target_benchmark }}
|
||||
{{ benchmark_description }}
|
||||
{% endif %}
|
||||
|
||||
## Selection Guidelines
|
||||
- Select datasets that are directly relevant to the user's target scenario
|
||||
- Consider domain alignment (e.g., math datasets for math reasoning tasks)
|
||||
- Consider task type alignment (e.g., reasoning datasets for reasoning tasks)
|
||||
- When uncertain, include the dataset (better to have false positives than miss relevant data)
|
||||
|
||||
## Output Format
|
||||
Return a JSON object:
|
||||
```json
|
||||
{
|
||||
"selected_datasets": ["dataset1", "dataset2"],
|
||||
"reasoning": "Brief explanation of why these datasets were selected"
|
||||
}
|
||||
```
|
||||
|
||||
user: |-
|
||||
## Available Datasets
|
||||
{% for ds in datasets %}
|
||||
### {{ ds.name }}
|
||||
- **total_samples**: {{ ds.total_samples }}
|
||||
- **total_size_mb**: {{ ds.total_size_mb }}
|
||||
{%- if ds.tasks %}
|
||||
- **tasks**:
|
||||
{%- for task_name, task_info in ds.tasks.items() %}
|
||||
#### {{ "(root)" if task_name == "_root" else task_name }}
|
||||
- sample_count: {{ task_info.sample_count }}
|
||||
{%- if task_info.column_stats %}
|
||||
- column_stats:
|
||||
{%- for col, col_stats in task_info.column_stats.items() %}
|
||||
- {{ col }}: p50={{ col_stats.p50_tokens }}, p99={{ col_stats.p99_tokens }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- if ds.readme %}
|
||||
- **readme**: {{ ds.readme }}
|
||||
{%- endif %}
|
||||
|
||||
{% endfor %}
|
||||
|
||||
Please select the datasets most relevant to the user's fine-tuning goal.
|
||||
@@ -0,0 +1,89 @@
|
||||
general_model_background: |-
|
||||
The general model is a flexible and comprehensive framework designed to integrate factor-based, model-based, and graph-based approaches in quantitative investment. It allows users to define custom models that leverage various financial factors to predict the returns and risks of portfolios or single assets. These models are central to many advanced quantitative investment strategies and can be adapted to a wide range of use cases, from factor-based alpha generation to complex deep learning predictions.
|
||||
|
||||
Each general model incorporates the following components:
|
||||
1. Name: The name of the model.
|
||||
2. Description: A detailed description of the model.
|
||||
3. Factors: The financial factors used as inputs, including their definitions and formulations.
|
||||
4. Architecture: The structure of the machine learning, deep learning, or graph-based model.
|
||||
5. Hyperparameters: The hyperparameters used in the model, such as learning rate, number of epochs, etc.
|
||||
6. ModelType: The type of the model, "Tabular" for tabular data, "TimeSeries" for time series data, or "Graph" for graph data.
|
||||
The general model should provide clear and detailed documentation of its factors, architecture, and hyperparameters. Each model should have a fixed architecture and hyperparameters to ensure reproducibility and consistency.
|
||||
|
||||
general_model_interface: |-
|
||||
Your python code should follow the interface to better interact with the user's system. It should be a pytorch model.
|
||||
Your code should contain several parts:
|
||||
1. The import part: import the necessary libraries.
|
||||
2. A class which is a sub-class of pytorch.nn.Module. This class should have an init function and a forward function which inputs a tensor and outputs a tensor.
|
||||
3. Set a variable called "model_cls" to the class you defined.
|
||||
|
||||
The user will save your code into a python file called "model.py". Then the user imports model_cls in file "model.py" after setting the cwd into the directory:
|
||||
```python
|
||||
from model import model_cls
|
||||
|
||||
So your python code should follow the pattern:
|
||||
|
||||
class XXXModel(torch.nn.Module):
|
||||
...
|
||||
model_cls = XXXModel
|
||||
|
||||
The model has three types, "Tabular" for tabular data, "TimeSeries" for time series data, and "Graph" for graph data.
|
||||
|
||||
The input shape to a tabular model is (batch_size, num_features).
|
||||
The input shape to a time series model is (batch_size, num_features, num_timesteps).
|
||||
The input to a graph model are two tensors.
|
||||
node_feature: a tensor of shape (batch_size, num_features)
|
||||
edge_index: a tensor of shape (2, num_edges)
|
||||
|
||||
The batch_size is a dynamic value which is determined by the input of the forward function.
|
||||
|
||||
The output shape of the model should be (batch_size, 1).
|
||||
|
||||
The "num_features", "num_timesteps" are static and will be provided to the model through the init function.
|
||||
|
||||
User will initialize the tabular model with the following code:
|
||||
|
||||
model = model_cls(num_features=num_features)
|
||||
|
||||
User will initialize the time series model with the following code:
|
||||
|
||||
model = model_cls(num_features=num_features, num_timesteps=num_timesteps)
|
||||
|
||||
User will initialize the graph model with the following code:
|
||||
|
||||
model = model_cls(num_features=num_features)
|
||||
|
||||
|
||||
No other parameters will be passed to the model, so give other parameters a default value or make them static.
|
||||
|
||||
When dealing with a time series model, remember to permute the input tensor since the input tensor is in the shape of (batch_size, num_features, num_timesteps) and a normal time series model is expecting the input tensor in the shape of (batch_size, num_timesteps, num_features).
|
||||
|
||||
Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you. Also, don't write a main function in your python code. The user will call the forward method in the model_cls to get the output tensor.
|
||||
|
||||
Please note that your model should only use current features as input. The user will provide the input tensor to the model's forward function.
|
||||
|
||||
general_model_output_format: |-
|
||||
Your output should be a tensor with shape (batch_size, 1).
|
||||
The output tensor should be saved in a file named "output.pth" in the same directory as your python file.
|
||||
The user will evaluate the shape of the output tensor, so the tensor read from "output.pth" should be 8 numbers.
|
||||
|
||||
general_model_simulator: |-
|
||||
The models are not loaded and backtested. That said, pay attention to its architecture.
|
||||
|
||||
general_model_rich_style_description: |-
|
||||
### [Model Research & Development Co-Pilot](#_scenario)
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
This demo automates the extraction and development of PyTorch models from academic papers. It supports various model types through two main components: Reader and Coder.
|
||||
|
||||
#### [Workflow Components](#_rdloops)
|
||||
|
||||
1. **[Reader](#_research)**
|
||||
- Extracts model information from papers, including architectures and parameters.
|
||||
- Converts content into a structured format using Large Language Models.
|
||||
|
||||
2. **[Evolving Coder](#_development)**
|
||||
- Translates structured information into executable PyTorch code.
|
||||
- Ensures correct tensor shapes with an evolving coding mechanism.
|
||||
- Refines the code to match source specifications.
|
||||
@@ -0,0 +1,323 @@
|
||||
kg_description_template:
|
||||
system: |-
|
||||
You are an assistant that extracts structured information from unstructured text.
|
||||
The user will provide you a Kaggle competition description, and you need to extract specific details from it.
|
||||
For the dataset, the competition may not include detailed information about the dataset. The user has read the dataset and provide you the relevant information. Please include it in your response.
|
||||
Please answer in Json format with the following schema:
|
||||
{
|
||||
"Competition Type": "The type of competition, e.g., 'Classification', 'Regression', 'Clustering', 'Prediction", "Time-Series Forecasting",
|
||||
"Competition Description": "A brief description of the competition",
|
||||
"Target Description": "A description of the target variable to be predicted",
|
||||
"Competition Features": "Two-line description of the overall features involved within the competition as background."
|
||||
"Submission Specifications": "The submission specification & sample submission csv descriptions for the model to output."
|
||||
"Submission channel number to each sample": "The number of channels in the output for each sample, e.g., 1 for regression, N for N class classification with probabilities, etc. A Integer. If not specified, it is 1."
|
||||
"Metric Evaluation Description": "A brief description of the metrics used in the evaluation. Please note that if `evaluation_metric_direction` is True, it indicates that higher values are better; if False, lower values are preferred."
|
||||
}
|
||||
Since these might be very similar column names in data like one_hot_encoded columns, you can use some regex to group them together.
|
||||
|
||||
|
||||
user: |-
|
||||
Competition Description:
|
||||
{{ competition_descriptions }}
|
||||
The raw data information:
|
||||
{{ raw_data_information }}
|
||||
Evaluation_metric_direction:
|
||||
{{ evaluation_metric_direction }}
|
||||
|
||||
kg_background: |-
|
||||
You are solving a data science tasks and the type of the competition is {{ competition_type }}.
|
||||
The competition description is: {{competition_description}}.
|
||||
|
||||
We provide an overall script in file: train.py. The user will run the train.py script along with several feature and model scripts to train several model to get a good performance on this task.
|
||||
|
||||
The train.py script is as follows:
|
||||
```python
|
||||
{{ train_script }}
|
||||
```
|
||||
|
||||
The final output of our pipeline is from a ensemble of up to four models. Each model is trained on a different subset of the data.
|
||||
The four model types are: XGBoost, RandomForest, LightGBM and Neural Network (A Pytorch model).
|
||||
About the Neural Network model, You can try different architectures and hyperparameters to improve the performance. You can even use a pytorch model to ensemble the other three types of models. Try to open your mind on the NN model.
|
||||
|
||||
The data is extracted from the competition dataset, focusing on relevant attributes in {{ competition_features }}.
|
||||
|
||||
The user firstly designs and implements a feature book for each model. The feature book is a combination of several features and feature groups.
|
||||
The feature book is built from:
|
||||
- Raw features: The raw features are the original features from the dataset.
|
||||
- generated features: The generated features are the features that are calculated based on the raw features according to some formulations. The calculation should be align with some physical or logical meaning. Don't just simply apply some numeric operations to the raw features.
|
||||
- feature groups: The feature groups are preprocessed group of features from the raw features like normalization, one hot encoding, etc.
|
||||
The feature or feature group is defined in the following parts:
|
||||
- Name: The name of the feature or feature group.
|
||||
- Description: A description of the feature or feature group.
|
||||
- Formulation: The formulation of the feature or feature group.
|
||||
- Variables: The variable list used in the formulation. Notice: The variable should be a specific feature in the dataset. Please make sure the feature name is exactly the same as the feature name in the dataset.
|
||||
|
||||
For each model, the user will design and implement the model in a separate script.
|
||||
The model is defined in the following parts:
|
||||
- Name: The name of the model.
|
||||
- Description: A description of the model.
|
||||
- Architecture: The detailed architecture of the model, such as neural network layers or tree structures.
|
||||
- ModelType: The type of the model, which should be one of ["XGBoost", "RandomForest", "LightGBM", "NN"].
|
||||
The model should provide clear and detailed documentation of its architecture and hyperparameters.
|
||||
|
||||
The user tries to optimize the performance iteratively by employing one of the feature related or model related action items:
|
||||
- Feature related:
|
||||
- "Feature engineering": The user will design several new tasks and implement several new features. The new feature might only affect the model using all the feature book.
|
||||
- "Feature processing": The user will design a new task to process the feature book like normalization or one hot encoding to improve the model performance. Any processing with help of a deep model is not included in this task.
|
||||
- Model related:
|
||||
- "Model feature selection": The user will modify one model to select the part of the features from the feature book to improve the model performance.
|
||||
- "Model tuning": The user will tune the hyperparameters of XGBoost, RandomForest or LightGBM or build or improve the NN model to improve the model performance.
|
||||
Notice: You can automatically optimize the hyperparameters of the model using some library when training the model. Since we don't have a lot of time to train the model, please use a small number of trials to optimize the hyperparameters.
|
||||
Our validation set split is not deterministic, so when you are using hyperparameter tuning, you can merge training and validation and use cross validation method to tune the hyperparameters.
|
||||
One you have determine the best model parameter, you should retrain the model on all training and validation set to get the final model.
|
||||
|
||||
For each loop, you need to help user decide which action item to choose and provide the corresponding code to implement the action item.
|
||||
|
||||
kg_feature_interface: |-
|
||||
Your code should contain several parts:
|
||||
1. The import part: import the necessary libraries.
|
||||
2. A class that contains the feature engineering logic.
|
||||
The class should have the following methods:
|
||||
- fit: This method should fit the feature engineering model to the training data.
|
||||
- transform: This method should transform the input data and return it.
|
||||
For some tasks like generating new features, the fit method may not be necessary. Please pass this function as a no-op.
|
||||
3. A variable called feature_engineering_cls that contains the class name.
|
||||
The input to 'fit' is the training data in pandas dataframe, and the input to 'transform' is the data to be transformed in pandas dataframe.
|
||||
The original columns should be excluded from the returned DataFrame.
|
||||
|
||||
Notice: Since we have a very big dataset, the feature engineering should be efficient and fast. Otherwise, please sufficiently exploit the multiprocessing or parallel computing to speed up the feature engineering process!
|
||||
|
||||
Exception handling will be managed externally, so avoid using try-except blocks in your code. The user will handle any exceptions that arise and provide feedback as needed.
|
||||
|
||||
The feat_eng function can be one of the following:
|
||||
- Feature engineering: This function calculated one new feature based on the existing raw data.
|
||||
- Feature processing: This function processes the existing raw data like normalization or one hot encoding and return the processed data in the form of a pandas DataFrame.
|
||||
|
||||
Here is an example of how your Python code should be structured:
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
class FeatureEngineeringName:
|
||||
def fit(self, train_df: pd.DataFrame):
|
||||
"""
|
||||
Fit the feature engineering model to the training data.
|
||||
For example, for one hot encoding, this would involve fitting the encoder to the training data.
|
||||
For feature scaling, this would involve fitting the scaler to the training data.
|
||||
"""
|
||||
return self
|
||||
|
||||
def transform(self, X: pd.DataFrame):
|
||||
"""
|
||||
Transform the input data.
|
||||
"""
|
||||
return X
|
||||
return X.mean(axis=1).to_frame("mean_feature") # Example feature engineering
|
||||
return X.fillna(0) # Example feature processing
|
||||
|
||||
feature_engineering_cls = FeatureEngineeringName
|
||||
```
|
||||
|
||||
To Note:
|
||||
Top 0. I have already completed the encoded labeling process, so please avoid any one-hot encoding or similar operations in the future. Focus instead on targeted and efficient feature engineering techniques, such as normalizing float-type features, filtering based on specific categories, or other concise transformations that can be quickly implemented and tested without unnecessary complexity. Also, ensure that the index of the output DataFrame matches the original DataFrame's index, and that the number of columns remains consistent across train, validation, and test sets.
|
||||
1. Ensure that your code meets these requirements and produces a feature-engineered DataFrame that contains only the newly engineered columns, aligning with the user's data and objectives.
|
||||
2. Ensure that the index of the output DataFrame matches the index of the original DataFrame. For example:
|
||||
Incorrect: `normalized_df = pd.DataFrame(normalized_features, columns=X.columns)`
|
||||
Correct: `normalized_df = pd.DataFrame(normalized_features, columns=X.columns, index=X.index)`
|
||||
3. Ensure consistency in column count across train, validation, and test sets post-feature engineering. For example, fit PCA on the training set and apply the same transformation to validation and test sets to keep the number of columns aligned, and use OneHotEncoder may also cause different number of columns.
|
||||
4. Ensure that the generation of new features does not drastically increase the number of columns, which can slow down data processing. For example, avoid creating pairwise interactions for all features, as this would lead to a quadratic increase in the number of columns.
|
||||
5. Avoids raising a `ValueError` or any other exceptions that could interrupt the main program's flow. The code should not include checks that could potentially lead to a `ValueError`. Instead, focus on writing robust and fault-tolerant feature engineering functions that handle edge cases and missing data gracefully, without stopping the program.
|
||||
6. Specific categories of features can be filtered, and processing can be applied to those categories. For example, normalization can be applied to float-type features, but such processing should not be done on one-hot encoded features.
|
||||
7. You are participating in a Kaggle competition and need data engineering ideas that are small, efficient, and quick to execute. Your suggestions should avoid unnecessary complexity or excessive processing time. Focus on delivering concise, impactful transformations or preprocessing steps that improve model performance with minimal resource usage. Please suggest clear, targeted approaches that can be implemented and tested rapidly.
|
||||
|
||||
kg_model_interface: |-
|
||||
Your code should contain several parts:
|
||||
1. The import part: import the necessary libraries.
|
||||
2. A function called fit() that trains the model and returns the trained model.
|
||||
The function should take the following arguments:
|
||||
- X_train: The training features as a pandas DataFrame.
|
||||
- y_train: The training labels as a pandas Series.
|
||||
- X_valid: The validation features as a pandas DataFrame.
|
||||
- y_valid: The validation labels as a pandas Series.
|
||||
The function should return the trained model.
|
||||
3. A function called predict() that makes predictions using the trained model.
|
||||
The function should take the following arguments:
|
||||
- model: The trained model.
|
||||
- X: The features as a pandas DataFrame.
|
||||
The function should return the predicted probabilities or boolean predictions in numpy.ndarray format.
|
||||
Please refer to the train.py script to verify whether the output should be a class label or a probability!
|
||||
|
||||
Here are some examples of how your Python code should be structured:
|
||||
|
||||
{% if tag == "XGBoost" or tag is none %}
|
||||
For XGBoost:
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import xgboost
|
||||
from xgboost import DMatrix
|
||||
|
||||
def fit(
|
||||
X_train: pd.DataFrame, y_train: pd.Series, X_valid: pd.DataFrame, y_valid: pd.Series
|
||||
) -> xgboost.Booster:
|
||||
dtrain = DMatrix(X_train, label=y_train)
|
||||
dvalid = DMatrix(X_valid, label=y_valid)
|
||||
params = ... # Set parameters to XGBoost model
|
||||
model = xgboost.train(params, dtrain, num_boost_round=100)
|
||||
y_pred = model.predict(dvalid)
|
||||
|
||||
accuracy = ... # Calculate accuracy
|
||||
return model
|
||||
|
||||
|
||||
def predict(model: xgboost.Booster, X: pd.DataFrame) -> np.ndarray:
|
||||
dtest = DMatrix(X)
|
||||
y_pred = model.predict(dtest)
|
||||
|
||||
return y_pred
|
||||
```
|
||||
{% endif %}
|
||||
{% if tag == "RandomForest" or tag is none %}
|
||||
For RandomForest:
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
|
||||
from sklearn.metrics import accuracy_score
|
||||
|
||||
def fit(
|
||||
X_train: pd.DataFrame, y_train: pd.Series, X_valid: pd.DataFrame, y_valid: pd.Series
|
||||
) -> RandomForestClassifier | RandomForestRegressor:
|
||||
model = RandomForestClassifier(...) # fir classification tasks
|
||||
model = RandomForestRegressor(...) # for regression tasks
|
||||
model.fit(X_train, y_train, ...) # Train the model
|
||||
|
||||
return model
|
||||
|
||||
|
||||
def predict(model: RandomForestClassifier | RandomForestRegressor, X: pd.DataFrame) -> np.ndarray:
|
||||
y_pred = model.predict(X)
|
||||
|
||||
return y_pred
|
||||
```
|
||||
{% endif %}
|
||||
{% if tag == "LightGBM" or tag is none %}
|
||||
For LightGBM:
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from lightgbm import LGBMClassifier, LGBMRegressor
|
||||
|
||||
def fit(
|
||||
X_train: pd.DataFrame, y_train: pd.Series, X_valid: pd.DataFrame, y_valid: pd.Series
|
||||
) -> LGBMClassifier | LGBMRegressor:
|
||||
model = LGBMClassifier(...) # for classification tasks, please add parameters here
|
||||
model = LGBMRegressor(...) # for regression tasks, please add parameters here
|
||||
|
||||
model.fit(X=X_train, y=y_train, eval_set=[(X_valid, y_valid)])
|
||||
return model
|
||||
|
||||
|
||||
def predict(model: LGBMClassifier | LGBMRegressor, X: pd.DataFrame) -> np.ndarray:
|
||||
y_pred = model.predict(X)
|
||||
|
||||
return y_pred
|
||||
```
|
||||
{% endif %}
|
||||
{% if tag == "NN" or tag is none %}
|
||||
For Neural Network:
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch.utils.data import DataLoader, TensorDataset
|
||||
|
||||
|
||||
class NNModel(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super(Model, self).__init__()
|
||||
# Define your model here
|
||||
|
||||
def forward(self, x):
|
||||
# Define the forward pass
|
||||
return x
|
||||
|
||||
def fit(X_train: pd.DataFrame, y_train: pd.DataFrame, X_valid: pd.DataFrame, y_valid: pd.DataFrame) -> torch.nn.Module:
|
||||
model = NNModel() # Initialize the model, You can write your own model class
|
||||
|
||||
optimizer = torch.optim.Adam(model.parameters(), lr=0.01) # Example optimizer, you can use any optimizer
|
||||
criterion = torch.nn.CrossEntropyLoss() # Example loss function, you can use any loss function
|
||||
|
||||
train_loader = DataLoader(TensorDataset(X_train, y_train), batch_size=64, shuffle=True)
|
||||
valid_loader = DataLoader(TensorDataset(X_valid, y_valid), batch_size=64, shuffle=False)
|
||||
|
||||
# Example training loop, you can customize this loop as per your requirement
|
||||
for epoch in range(10):
|
||||
model.train()
|
||||
for X_batch, y_batch in train_loader:
|
||||
optimizer.zero_grad()
|
||||
outputs = model(X_batch)
|
||||
loss = criterion(outputs, y_batch)
|
||||
loss.backward()
|
||||
optimizer.step()
|
||||
|
||||
model.eval()
|
||||
y_pred = []
|
||||
with torch.no_grad():
|
||||
for X_batch, _ in valid_loader:
|
||||
outputs = model(X_batch)
|
||||
y_pred.extend(outputs.squeeze().tolist())
|
||||
|
||||
y_pred = torch.tensor(y_pred)
|
||||
accuracy = (y_pred == y_valid).float().mean()
|
||||
# You can early stop based on the validation, please customize this as per your requirement
|
||||
return model
|
||||
|
||||
|
||||
def predict(model: torch.nn.Module, X: pd.DataFrame) -> np.ndarray:
|
||||
X = torch.tensor(X.values).float()
|
||||
model.eval()
|
||||
with torch.no_grad():
|
||||
y_pred = model(X).squeeze().numpy()
|
||||
|
||||
return y_pred
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
kg_feature_simulator: |-
|
||||
The data preprocessing method you provide will be used to prepare data by processing it, concatenating the results with other features, and removing unnecessary features before training the model.
|
||||
The processed data will then be used for model training and prediction.
|
||||
|
||||
User will use your data preprocessing method to do the following steps:
|
||||
1. Execute your Python files to process the data. (what you need to do)
|
||||
2. Concatenate the processed features with other features and the original data.
|
||||
3. Remove any unnecessary features before training the model.
|
||||
4. Train a model such as LightGBM, CatBoost, LSTM, or a simple PyTorch model using the processed data.
|
||||
5. Evaluate the performance of your preprocessing method and provide feedback.
|
||||
|
||||
kg_feature_output_format: |-
|
||||
The output should be a pandas DataFrame with the new features. The columns should be the new features, and the rows should correspond to the number of samples in the input DataFrame.
|
||||
Sample output dataframe info:
|
||||
<class 'pandas.core.frame.DataFrame'>
|
||||
Index: {Same to the input DataFrame}
|
||||
Data columns (total N columns):
|
||||
# Column Dtype
|
||||
--- ------ -----
|
||||
0 feature_name_0 float64
|
||||
1 feature_name_1 float64
|
||||
dtypes: float64(N)
|
||||
memory usage: {Memory usage of the output DataFrame}
|
||||
|
||||
kg_model_output_format: |-
|
||||
For model related tasks, the output should be an np.ndarray with the appropriate number of predictions.
|
||||
Please refer to the train.py script to verify whether the output should be a class label or a probability!
|
||||
{% if channel == 1 %}
|
||||
For each sample, the output should be a single value (e.g., (8, 1) if there are 8 samples).
|
||||
{% else %}
|
||||
For each sample, the output should be multiple values with {{ channel }} numbers (e.g., (8, {{ channel }}) if there are 8 samples).
|
||||
{% endif %}
|
||||
|
||||
kg_model_simulator: |-
|
||||
The models will be trained on the competition dataset and evaluated on their ability to predict the target. Metrics like accuracy and AUC-ROC is used to evaluate the model performance.
|
||||
Model performance will be iteratively improved based on feedback from evaluation results.
|
||||
Your output should follow some requirements to submit to the competition:
|
||||
{{ submission_specifications }}
|
||||
@@ -0,0 +1,93 @@
|
||||
extract_kaggle_knowledge_prompts:
|
||||
system: |-
|
||||
You are a Kaggle competition expert with extensive experience in analyzing high-ranking Kaggle notebooks and competition strategies.
|
||||
Your task is to summarize or infer key information such as the competition name, task type, and specific techniques employed in the notebook or strategy.
|
||||
For each provided content, you are expected to extract valuable insights and organize the analysis in the structured format outlined below.
|
||||
|
||||
Please provide the analysis in the following JSON format:
|
||||
{
|
||||
"content": "Put the provided content here",
|
||||
"title": "extracted title, if available",
|
||||
"competition_name": "extracted competition name",
|
||||
"task_category": "extracted task type, e.g., Classification, Regression",
|
||||
"field": "field of focus, e.g., Feature Engineering, Modeling",
|
||||
"ranking": "extracted ranking, if available",
|
||||
"score": "extracted score or metric, if available"
|
||||
}
|
||||
|
||||
user: |-
|
||||
High-ranking Kaggle notebooks or competition strategies: {{ file_content }}
|
||||
|
||||
extract_kaggle_knowledge_from_feedback_prompts:
|
||||
system: |-
|
||||
You are a Kaggle competition expert with extensive experience in analyzing Kaggle notebooks and competition strategies.
|
||||
Your task is to summarize or infer key information such as the competition name, task type, and specific techniques employed in the notebook or strategy.
|
||||
For each provided content, you are expected to extract valuable insights and organize the analysis in the structured format outlined below.
|
||||
|
||||
Please provide the analysis in the following JSON format:
|
||||
{
|
||||
"content": "all provided content",
|
||||
"title": "extracted title, if available",
|
||||
"competition_name": "extracted competition name",
|
||||
"task_category": "extracted task type, e.g., Classification, Regression",
|
||||
"field": "field of focus, e.g., Feature Engineering, Modeling",
|
||||
"ranking": "extracted ranking, if available",
|
||||
"score": "extracted score or metric, if available"
|
||||
}
|
||||
|
||||
user: |-
|
||||
Experiment strategy: {{ experiment_strategy }}
|
||||
|
||||
|
||||
extract_knowledge_graph_from_document:
|
||||
system: |-
|
||||
You are helping the user extract knowledge from a document.
|
||||
{% if scenario %}
|
||||
The user is working on data science competitions in Kaggle, with the following scenario: {{ scenario }}
|
||||
{% else %}
|
||||
The user is working on general data science competitions on Kaggle.
|
||||
{% endif %}
|
||||
|
||||
The user has identified valuable documents from other experts and requires your help to extract meaningful insights from them.
|
||||
|
||||
Considering each document might contain several valuable insights, you need to extract them one by one and organize them in a structured format.
|
||||
|
||||
You should return a dict containing a single knowledge which includes several fields:
|
||||
1. The competition the document is related to.
|
||||
2. The hypothesis the document is trying to prove. Containing a type to the hypothesis and very detailed explanation to the hypothesis. The type should be one from ["Feature engineering", "Feature processing", "Model feature selection", "Model tuning"].
|
||||
3. Detailed experiments the document has conducted.
|
||||
4. Any related code snippets related to the hypothesis if available.
|
||||
5. The conclusion to this knowledge. A bool value indicating whether the hypothesis is proved or not is required. More explainable conclusion is also needed.
|
||||
|
||||
Please provide the analysis in the following JSON format:
|
||||
{
|
||||
"competition": "(Plain text) extracted competition information, including the competition name, type, description, target, and features (If no specific competition name or other fields are found, leave them blank).",
|
||||
"hypothesis":
|
||||
{
|
||||
"type": "one of the hypothesis types from ['Feature engineering', 'Feature processing', 'Model feature selection', 'Model tuning']",
|
||||
"explanation": "(Plain text) extracted detailed explanation to the hypothesis"
|
||||
},
|
||||
"experiments": "(Plain text) Detailed descriptions of the experiments conducted in the document, which can be listed in bullet points.",
|
||||
"code": "extracted code snippets if available",
|
||||
"conclusion":
|
||||
{
|
||||
"proved": "bool value indicating whether the hypothesis is proved or not",
|
||||
"explanation": "(Plain text) extracted detailed explanation to the conclusion"
|
||||
}
|
||||
}
|
||||
All fields are required so don't miss any key in the schema. The document might not contain all the fields, so you should extract as much information as possible. If a field is not available, please put "N/A" in the field.
|
||||
|
||||
If you find no valuable insights in the document, please return an empty dict.
|
||||
|
||||
user: |-
|
||||
Document content: {{ document_content }}
|
||||
|
||||
refine_with_LLM:
|
||||
system: |-
|
||||
You are an experienced data science expert and an assistant, helping the user evaluate and improve content.
|
||||
|
||||
user: |-
|
||||
Here is the target: {{ target }}.
|
||||
Please evaluate whether the following RAG query result aligns with the target.
|
||||
If it does not, simply respond with "There are no relevant RAG results to support."
|
||||
RAG query result: {{ text }}.
|
||||
@@ -0,0 +1,366 @@
|
||||
KG_hypothesis_gen_RAG: |-
|
||||
The user has proposed several hypothesis and conducted experiments to validate them.
|
||||
The hypothesis can divided into two categories:
|
||||
1. Insights: These are the observations user did to other similar problems. You can either apply the same hypothesis or modify them to fit the current problem.
|
||||
2. Experience: These are former hypothesis and experiments user did to the current problem. You can either continue to improve the hypothesis or change to a new one.
|
||||
|
||||
{% if insights %}
|
||||
The insights are as follows:
|
||||
{% for insight in insights %}
|
||||
Insight: {{ loop.index }}
|
||||
- hypothesis: {{ insight.hypothesis }}
|
||||
- experiments: {{ insight.experiments }}
|
||||
- conclusion: {{ insight.conclusion }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
{% if experiences %}
|
||||
The experiences are as follows:
|
||||
{% for experience in experiences %}
|
||||
Experience: {{ loop.index }}
|
||||
- hypothesis: {{ experience.hypothesis }}
|
||||
- experiments: {{ experience.experiments }}
|
||||
- conclusion: {{ experience.conclusion }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
hypothesis_and_feedback: |-
|
||||
{% for experiment, feedback in trace.hist[-10:] %}
|
||||
Hypothesis {{ loop.index }}: {{ experiment.hypothesis }}
|
||||
Observation on the result with the hypothesis: {{ feedback.observations }}
|
||||
Feedback on the original hypothesis: {{ feedback.hypothesis_evaluation }}
|
||||
Did changing to this hypothesis work? (focus on the change): {{ feedback.decision }}
|
||||
{% endfor %}
|
||||
|
||||
hypothesis_output_format: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"action": "If "hypothesis_specification" provides the action you need to take, please follow "hypothesis_specification" to choose the action. Otherwise, based on previous experimental results, suggest the action you believe is most appropriate at the moment. It should be one of ["Feature engineering", "Feature processing", "Model feature selection", "Model tuning"]"
|
||||
"hypothesis": "The new hypothesis generated based on the information provided.",
|
||||
"reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them.",
|
||||
"concise_reason": "Two-line summary. First line focuses on a concise justification for the change. Second line generalizes a knowledge statement.",
|
||||
"concise_observation": "One line summary. It focuses on the observation of the given scenario, data characteristics, or previous experiences (failures & succeses).",
|
||||
"concise_justification": "One line summary. Justify the hypothesis based on theoretical principles or initial assumptions.",
|
||||
"concise_knowledge": "One line summary. Transferable knowledge based on theoretical principles. Use conditional grammar. eg. "If...., ..; When..., .; and etc" Make sure that you state things clearly without ambiguity. Eg. avoid saying "previous hypothesis", because one wouldn't know what that is."
|
||||
}
|
||||
|
||||
hypothesis_specification:
|
||||
Feature engineering: |-
|
||||
Action: Feature engineering
|
||||
|
||||
Description: We engineer the features for the sake of best model performance on the basis of engineering the most influential features.
|
||||
|
||||
1. Type of Feature and Data Characteristics:
|
||||
- Clearly define the type of feature being introduced.
|
||||
- Explain what data characteristics or patterns this feature captures.
|
||||
- Keep descriptions focused, avoiding redundant details to ensure clarity.
|
||||
|
||||
2. Simple and Effective Features First:
|
||||
- Start by introducing features that are simple yet likely to be effective.
|
||||
- Provide a concise explanation of why these features are expected to perform well.
|
||||
- Avoid complex or combined features during the initial stages.
|
||||
|
||||
3. Gradual Complexity Increase:
|
||||
- After initial feature testing, introduce more complex features.
|
||||
- Discuss both the potential benefits and any additional complexities of these features.
|
||||
- Begin combining features only after simpler ones have been tested and validated.
|
||||
|
||||
4. New Directions and Optimizations:
|
||||
- If results suggest a need for a new approach, explain why, using data analysis, domain knowledge, or observed patterns.
|
||||
- Propose one new direction per iteration for clarity and focus.
|
||||
- If a previous hypothesis did not surpass the previous best but shows promise, continue in the same direction with optimizations.
|
||||
- Emphasize that features that outperform previous best results are added to the feature library, avoiding redundant work.
|
||||
|
||||
5. 1-3 Feature Tasks per Generation:
|
||||
- Each generation should produce 1-3 feature tasks.
|
||||
- Maintain a balance between simplicity and complexity to develop a diverse and robust feature library.
|
||||
|
||||
Feature processing: |-
|
||||
Action: Feature processing
|
||||
|
||||
1. Feature Transformation and Normalization:
|
||||
- Clearly define any transformations applied to features (e.g., scaling, normalization, log transforms).
|
||||
- Explain how these transformations improve the data's suitability for the model.
|
||||
- Ensure transformations do not introduce unnecessary complexity early on.
|
||||
|
||||
2. Handling Missing Values and Outliers:
|
||||
- Define any imputation methods used for missing data (e.g., mean, median, or more complex methods).
|
||||
- Explain how outliers are handled (e.g., clipping, removal, or transformation).
|
||||
- Ensure these processes are straightforward, enhancing data quality without overcomplicating early feature processing.
|
||||
|
||||
3. Feature Interactions and Combinations:
|
||||
- After testing individual features, introduce combinations or interactions.
|
||||
- Discuss the potential advantages of feature interaction terms (e.g., polynomial or multiplicative features).
|
||||
- Ensure interactions are only applied after simpler, individual features have been processed.
|
||||
|
||||
4. 1-3 Feature Tasks per Generation:
|
||||
- Each generation should produce 1-3 feature tasks.
|
||||
- Maintain a balance between simplicity and complexity to develop a diverse and robust feature library.
|
||||
|
||||
Model feature selection: |-
|
||||
Action: Model feature selection
|
||||
|
||||
1. Selection based on model_type:
|
||||
- Specify which features are being selected and explain why, considering the model type (e.g., NN, Random Forest, LightGBM, XGBoost).
|
||||
- Ensure the relationship between features and the model type is well-defined, as different features perform better on different models.
|
||||
|
||||
2. Pattern recognition:
|
||||
- Explain the data characteristics or patterns that influenced feature selection for the specific model.
|
||||
- Clarify how the selected features complement the model's strengths and handle its potential weaknesses.
|
||||
|
||||
Model tuning: |-
|
||||
Action: Model tuning
|
||||
|
||||
1. Overview:
|
||||
- Clearly explain your hypothesis.
|
||||
- Which model are you tuning (one of the four types)?
|
||||
- How are you revising it, and why?
|
||||
- What are the innovations?
|
||||
- Base your hypothesis on previous structures and your understanding of the model code.
|
||||
- "Tuning" includes changing the model architecture or hyperparameters.
|
||||
|
||||
2. Focus on Architecture and/or Hyperparameter Tuning:
|
||||
- Concentrate on designing new model architectures one at a time, hyperparameter tuning, or both.
|
||||
- Each hypothesis should introduce a novel architecture or a significant modification to an existing one.
|
||||
- Leverage prior experiences and hypothesis history.
|
||||
- If necessary, write source code manually to implement innovations beyond existing packages.
|
||||
|
||||
3. Specific to Model Type:
|
||||
- Tuning must be specific to the model types available in our workspace (e.g., Neural Networks, XGBoost, Random Forest, LightGBM).
|
||||
- Clearly define the model type and the architecture or tuning being introduced.
|
||||
- Ensure the changes align with data characteristics and the model's strengths or limitations.
|
||||
|
||||
4. Rationale Behind Architecture and Tuning:
|
||||
- Explain the reasoning behind your architectural design or tuning approach.
|
||||
- Justify how the new structure or parameter changes more effectively capture data patterns and improve learning efficiency.
|
||||
|
||||
feature_experiment_output_format: |-
|
||||
According to the hypothesis, please help user design one or more feature engineering tasks.
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"factor or group name 1": {
|
||||
"description": "description of factor or group name 1",
|
||||
"formulation": "latex formulation of factor or group name 1",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
},
|
||||
"factor or group name 2": {
|
||||
"description": "description of factor or group name 2",
|
||||
"formulation": "latex formulation of factor or group name 2",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
}
|
||||
# Don't add ellipsis (...) or any filler text that might cause JSON parsing errors here!
|
||||
}
|
||||
|
||||
model_experiment_output_format: |-
|
||||
According to the hypothesis, please help user design one model task.
|
||||
We only build one model from four main model types: ["XGBoost", "RandomForest", "LightGBM", "NN"].
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"model_name": "model_name",
|
||||
"description": "A detailed description of the model",
|
||||
"architecture": "A detailed description of the model's architecture, e.g., neural network layers or tree structures",
|
||||
"hyperparameters": {
|
||||
"hyperparameter_name_1": "value of hyperparameter 1",
|
||||
"hyperparameter_name_2": "value of hyperparameter 2",
|
||||
"hyperparameter_name_3": "value of hyperparameter 3"
|
||||
},
|
||||
"model_type": "Please select only **one** model type from the following four options: XGBoost, RandomForest, LightGBM, or NN. The selected model must be unique and used as the **primary model**. You may choose an auxiliary model for support or optimization on specific tasks if necessary, but the primary model must come from the provided options."
|
||||
|
||||
}
|
||||
|
||||
kg_feedback_generation_user: |-
|
||||
We are in a process of finding and validating hypotheses to build a powerful model. Each round aims to confirm or reject hypotheses based on results.
|
||||
|
||||
The SOTA solution for the task is as follows:
|
||||
Features and its corresponding channel: {{ sota_features }}
|
||||
Models and its corresponding code: {{ sota_models }}
|
||||
Final result of the SOTA solution (we select the best-performing model's metric as the final result): {{ sota_result }}
|
||||
{% if sota_sub_results %}
|
||||
Sub-results of all sub-models: {{ sota_sub_results }}
|
||||
{% endif %}
|
||||
|
||||
Current solution to be evaluated:
|
||||
Hypothesis: {{ current_hypothesis }}
|
||||
Reasoning: {{ current_hypothesis_reason }}
|
||||
Current target action: {{ current_target_action }}
|
||||
Experiments conducted and their code: {{ current_sub_exps_to_code }}
|
||||
Final result of the current solution (we select the best-performing model's metric as the final result): {{ current_result }}
|
||||
{% if current_sub_results %}
|
||||
Sub-results of all sub-models: {{ current_sub_results }}
|
||||
{% endif %}
|
||||
|
||||
A more detailed comparison between the current solution and the SOTA solution:
|
||||
{{ combined_result }}
|
||||
|
||||
Some information about comparing the current solution with the SOTA solution:
|
||||
{{ evaluation_description }}
|
||||
|
||||
{% if last_hypothesis_and_feedback %}
|
||||
The user has made some hypothesis and conducted experiments to validate them, and the results are as follows:
|
||||
hypothesis: {{ last_hypothesis_and_feedback[0].hypothesis }}
|
||||
feedback decision: {{ last_hypothesis_and_feedback[1].decision }}
|
||||
reason: {{ last_hypothesis_and_feedback[1].reason }}
|
||||
{% endif %}
|
||||
Please refer to these hypothesis and feedback to help you recommend new hypothesis
|
||||
|
||||
Consider Changing Direction for Significant Gaps with the Best Result and the last round:
|
||||
- If the new results significantly differ from SOTA, consider a new direction.
|
||||
- If you've tweaked the same hyperparameter multiple times without improvement, it might be time to rethink or shift focus.
|
||||
- If it is model tuning, focus on comparing the SOTA's Sub-results of all sub-models: {{ sota_sub_results }} with the current experiment's Sub-results of all sub-models: {{ current_sub_results }}. For example, identify which model is currently the best, which model was adjusted in this experiment, and whether the adjustment was effective. Determine if there is potential to continue with this model or if another model shows more promise.
|
||||
|
||||
model_tuning_feedback_generation:
|
||||
system: |-
|
||||
You are an advanced assistant for analyzing results in data-driven R&D, in the context of designing machine learning models.
|
||||
The task is described in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
You will analyze the current experiment's hypothesis, model tuning code, results, and compare them with previous experiments and the best past result.
|
||||
Your feedback should:
|
||||
1. Confirm if the current result supports or refutes the hypothesis.
|
||||
2. Compare with previous best results.
|
||||
3. Suggest improvements or new directions. Stay innovative and adaptive.
|
||||
|
||||
Please provide detailed and constructive feedback. Note that as hypothesis evolve, a general trend should be that the model grows larger.
|
||||
Example JSON Structure for Result Analysis:
|
||||
{
|
||||
"Observations": "Your overall observations here",
|
||||
"Feedback for Hypothesis": "Observations related to the hypothesis",
|
||||
"New Hypothesis": "Your new hypothesis here",
|
||||
"Reasoning": "Reasoning for the new hypothesis",
|
||||
"Replace Best Result": "yes or no"
|
||||
}
|
||||
|
||||
Hypothesis Evolution Logic:
|
||||
- If the current hypothesis works, make the model more complex (e.g., add layers, neurons, etc.).
|
||||
- If a hypothesis works, build on it. If not, adjust at the same level before growing deeper. Think step by step and make changes. Act innovatively.
|
||||
- If it doesn't, modify elements at the current level (e.g., adjust regularization, change features).
|
||||
|
||||
Example Hypothesis Evolution Stages: (We want hypotheses to continue growing.) Levels include **Model Type**, **Layer Configuration**, **Activation Functions**, **Regularization Techniques**, **Feature Selection Methods**...
|
||||
- Initial Hypothesis: Use CNN with no feature selection.
|
||||
- Next Level (if successful): Add 5 convolutional layers, use all features.
|
||||
- Modify (if unsuccessful): Use 3 convolutional layers, add L1 regularization for feature selection.
|
||||
- Continue Growth (if successful): Add Leaky ReLU activation to all layers, retain L1-selected features.
|
||||
- Further Growth (if successful): Add dropout regularization (0.5 rate), retain L1 features.
|
||||
- Adjust (if unsuccessful): Use 5 layers, Leaky ReLU, dropout 0.3 rate.
|
||||
|
||||
factor_feedback_generation:
|
||||
system: |-
|
||||
You are a professional data feature engineering assistant in data-driven R&D.
|
||||
The task is described in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
You will receive a hypothesis, multiple tasks with their features, their results, and the best previous result.
|
||||
Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous best results, and suggest improvements or new directions.
|
||||
|
||||
Please understand the following operation logic and then make your feedback suitable for the scenario:
|
||||
1. Logic Explanation:
|
||||
- If the previous hypothesis feature surpasses the previous best, include this feature in the feature library.
|
||||
- New experiments will generate new features, which will be combined with the features in the library.
|
||||
- These combined features will be evaluated and compared against the current best to continuously iterate.
|
||||
2. Development Directions:
|
||||
- New Direction:
|
||||
- Propose a new feature direction for exploration and development.
|
||||
- Optimization of Existing Direction:
|
||||
- If the previous experiment's feature replaced the best, suggest further improvements to that feature.
|
||||
- Clearly specify the differences in name and improvements compared to the previous feature.
|
||||
- Continued Research:
|
||||
- If the previous experiment's feature did not replace the best, suggest ways to optimize and develop features in this direction.
|
||||
3. Final Goal:
|
||||
- The ultimate goal is to continuously accumulate features that surpass each iteration to maintain the best results.
|
||||
|
||||
Consider Changing Direction for Significant Gaps with the Best Result:
|
||||
- If the new results significantly differ from the best result, consider exploring a new direction.
|
||||
- Avoid re-implementing previous features as those that surpassed the best are already included in the feature library and will be used in each run.
|
||||
Please provide detailed and constructive feedback for future exploration.
|
||||
Respond in JSON format. Example JSON structure for Result Analysis:
|
||||
{
|
||||
"Observations": "Your overall observations here",
|
||||
"Feedback for Hypothesis": "Observations related to the hypothesis",
|
||||
"New Hypothesis": "Your new hypothesis here",
|
||||
"Reasoning": "Reasoning for the new hypothesis",
|
||||
"Replace Best Result": "yes or no"
|
||||
}
|
||||
|
||||
feature_selection_feedback_generation:
|
||||
system: |-
|
||||
You are a professional feature selection assistant for machine learning models. Your task is to analyze the current feature selection strategy, evaluate its effectiveness, and suggest improvements.
|
||||
The task is described in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
In your feedback, consider:
|
||||
1. How effective is the current feature selection strategy?
|
||||
2. Are there any patterns in the selected or discarded features that might inform future selections?
|
||||
3. How might we refine or change the feature selection approach to improve model performance?
|
||||
4. Are there any domain-specific considerations that should inform our feature selection?
|
||||
|
||||
Provide detailed and constructive feedback, focusing on actionable insights for feature selection improvement.
|
||||
|
||||
Respond in JSON format. Example JSON structure for Result Analysis:
|
||||
{
|
||||
"Observations": "Your overall observations here",
|
||||
"Feedback for Hypothesis": "Observations related to the hypothesis",
|
||||
"New Hypothesis": "Your new hypothesis here",
|
||||
"Reasoning": "Reasoning for the new hypothesis",
|
||||
"Replace Best Result": "yes or no"
|
||||
}
|
||||
|
||||
|
||||
model_feature_selection:
|
||||
system: |-
|
||||
You are an assistant for model feature selection in machine learning. Your task is to understand the current feature groups and choose the most relevant features for the model to get the best performance.
|
||||
|
||||
The user is currently working on a Kaggle competition scenario as follows:
|
||||
{{ scenario }}
|
||||
|
||||
The user is now working on the following model type:
|
||||
{{ model_type }}
|
||||
|
||||
The user will give you several feature groups and their descriptions. Your task is to select the most relevant features for the model to achieve the best performance. You should consider the following:
|
||||
1. How well do the selected features support the scenario?
|
||||
2. Are there any features that might be redundant or noisy?
|
||||
|
||||
Please answer the chosen group index in JSON format. Example JSON structure for Result Analysis:
|
||||
{
|
||||
"Selected Group Index": [1, 3, 5], # List of selected group indices, notice: the index starts from 1
|
||||
}
|
||||
|
||||
user: |-
|
||||
Current feature groups:
|
||||
{% for feature in feature_groups %}
|
||||
Group {{ loop.index }}:
|
||||
{{ feature }}
|
||||
{% endfor %}
|
||||
|
||||
gen_knowledge_from_code_mini_case:
|
||||
system: |-
|
||||
You were a proficient data scientist.
|
||||
user: |-
|
||||
The following notebook (contain markdown part and code part) is a high-performing solution for a kaggle competition.
|
||||
Please answer the following questions one by one and **as detailed as possible**.
|
||||
Make sure that another data scientist can exactly reproduce this copy of code based on your answer.
|
||||
Focus on the training process.
|
||||
|
||||
(1) Please give a summary of the overall design.
|
||||
(2) What is the overall model architecture? Please use a long article to answer this question as accurately and in detail as possible.
|
||||
(3) How are the important hyper-parameters setting in this code?
|
||||
(4) What is the optimization objective?
|
||||
(5) What advanced machine learning technique does this copy of code use?
|
||||
(6) What other important tricks do you think play an important role for high performance?
|
||||
|
||||
Note that make sure the answers are directly included from the code or markdown text, rather than based on your assumption.
|
||||
|
||||
--------------------
|
||||
{{ notebook }}
|
||||
--------------------
|
||||
|
||||
gen_knowledge_from_code_RDAgent:
|
||||
system: |-
|
||||
You were a proficient data scientist.
|
||||
user: |-
|
||||
TODO...
|
||||
@@ -0,0 +1,256 @@
|
||||
qlib_quant_background: |-
|
||||
Quantitative investment is a data-driven approach to asset management that relies on mathematical models, statistical techniques, and computational methods to analyze financial markets and make investment decisions. Two essential components of this approach are factors and models.
|
||||
|
||||
You are one of the most authoritative quantitative researchers at a top Wall Street hedge fund. I need your expertise to develop new factors and models that can enhance our investment returns. Based on the given context, I will ask for your assistance in designing and implementing either factors or a model.
|
||||
|
||||
{% if runtime_environment is not none %}
|
||||
====== Runtime Environment ======
|
||||
You have following environment to run the code:
|
||||
{{ runtime_environment }}
|
||||
{% endif %}
|
||||
|
||||
qlib_factor_background: |-
|
||||
The factor is a characteristic or variable used in quant investment that can help explain the returns and risks of a portfolio or a single asset. Factors are used by investors to identify and exploit sources of excess returns, and they are central to many quantitative investment strategies.
|
||||
Each number in the factor represents a physics value to an instrument on a day.
|
||||
User will train a model to predict the next several days return based on the factor values of the previous days.
|
||||
The factor is defined in the following parts:
|
||||
1. Name: The name of the factor.
|
||||
2. Description: The description of the factor.
|
||||
3. Formulation: The formulation of the factor.
|
||||
4. Variables: The variables or functions used in the formulation of the factor.
|
||||
The factor might not provide all the parts of the information above since some might not be applicable.
|
||||
Please specifically give all the hyperparameter in the factors like the window size, look back period, and so on. One factor should statically defines one output with a static source data. For example, last 10 days momentum and last 20 days momentum should be two different factors.
|
||||
|
||||
{% if runtime_environment is not none %}
|
||||
====== Runtime Environment ======
|
||||
You have following environment to run the code:
|
||||
{{ runtime_environment }}
|
||||
{% endif %}
|
||||
|
||||
qlib_factor_interface: |-
|
||||
Your python code should follow the interface to better interact with the user's system.
|
||||
Your python code should contain the following part: the import part, the function part, and the main part. You should write a main function name: "calculate_{function_name}" and call this function in "if __name__ == __main__" part. Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you.
|
||||
User will write your python code into a python file and execute the file directly with "python {your_file_name}.py". You should calculate the factor values and save the result into a HDF5(H5) file named "result.h5" in the same directory as your python file. The result file is a HDF5(H5) file containing a pandas dataframe. The index of the dataframe is the "datetime" and "instrument", and the single column name is the factor name,and the value is the factor value. The result file should be saved in the same directory as your python file.
|
||||
|
||||
qlib_factor_strategy: |-
|
||||
Ensure that for every step of data processing, the data format (including indexes) is clearly explained through comments.
|
||||
Each transformation or calculation should be accompanied by a detailed description of how the data is structured, especially focusing on key aspects like whether the data has multi-level indexing, how to access specific columns or index levels, and any operations that affect the data shape (e.g., `reset_index()`, `groupby()`, `merge()`).
|
||||
This step-by-step explanation will ensure clarity and accuracy in data handling. For example:
|
||||
1. **Start with multi-level index**:
|
||||
```python
|
||||
# The initial DataFrame has a multi-level index with 'datetime' and 'instrument'.
|
||||
# To access the 'datetime' index, use df.index.get_level_values('datetime').
|
||||
datetime_values = df.index.get_level_values('datetime')
|
||||
```
|
||||
|
||||
2. **Reset the index if necessary**:
|
||||
```python
|
||||
# Resetting the index to move 'datetime' and 'instrument' from the index to columns.
|
||||
# This operation flattens the multi-index structure.
|
||||
df = df.reset_index()
|
||||
```
|
||||
|
||||
3. **Perform groupby operations**:
|
||||
```python
|
||||
# Grouping by 'datetime' and 'instrument' to aggregate the data.
|
||||
# After groupby, the result will maintain 'datetime' and 'instrument' as a multi-level index.
|
||||
df_grouped = df.groupby(['datetime', 'instrument']).sum()
|
||||
```
|
||||
|
||||
4. **Ensure consistent datetime formats**:
|
||||
```python
|
||||
# Before merging, ensure that the 'datetime' column in both DataFrames is of the same format.
|
||||
# Convert to datetime format if necessary.
|
||||
df['datetime'] = pd.to_datetime(df['datetime'])
|
||||
other_df['datetime'] = pd.to_datetime(other_df['datetime'])
|
||||
```
|
||||
|
||||
5. **Merge operations**:
|
||||
```python
|
||||
# When merging DataFrames, ensure you are merging on both 'datetime' and 'instrument'.
|
||||
# If these are part of the index, reset the index before merging.
|
||||
merged_df = pd.merge(df, other_df, on=['datetime', 'instrument'], how='inner')
|
||||
```
|
||||
|
||||
qlib_factor_output_format: |-
|
||||
Your output should be a pandas dataframe similar to the following example information:
|
||||
<class 'pandas.core.frame.DataFrame'>
|
||||
MultiIndex: 2261923 entries, (Timestamp('2020-01-01 17:00:00'), 'EURUSD') to (Timestamp('2026-03-20 15:58:00'), 'EURUSD')
|
||||
Data columns (total 1 columns):
|
||||
# Column Non-Null Count Dtype
|
||||
--- ------ -------------- -----
|
||||
0 your factor name 2261923 non-null float64
|
||||
dtypes: float64(1)
|
||||
memory usage: <ignore>
|
||||
Notice: The non-null count is OK to be different to the total number of entries since some instruments may not have the factor value on some days.
|
||||
One possible format of `result.h5` may be like following:
|
||||
datetime instrument
|
||||
2020-01-01 EURUSD 1.094240
|
||||
2020-01-02 EURUSD 1.094280
|
||||
2020-01-03 EURUSD 1.095920
|
||||
...
|
||||
2026-03-20 EURUSD 1.083150
|
||||
|
||||
qlib_factor_simulator: |-
|
||||
The factors will be sent into Qlib to train a model to predict the next several days return based on the factor values of the previous days.
|
||||
Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.
|
||||
User will use Qlib to automatically do the following things:
|
||||
1. generate a new factor table based on the factor values.
|
||||
2. train a model like LightGBM, CatBoost, LSTM or simple PyTorch model to predict the next several days return based on the factor values.
|
||||
3. build a portfolio based on the predicted return based on a strategy.
|
||||
4. evaluate the portfolio's performance including the return, sharpe ratio, max drawdown, and so on.
|
||||
|
||||
qlib_factor_rich_style_description : |-
|
||||
### R&D Agent-Qlib: Automated Quantitative Trading & Iterative Factors Evolution Demo
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making. It highlights how financial factors evolve through continuous feedback and refinement.
|
||||
|
||||
#### [Automated R&D](#_rdloops)
|
||||
|
||||
- **[R (Research)](#_research)**
|
||||
- Iterative development of ideas and hypotheses.
|
||||
- Continuous learning and knowledge construction.
|
||||
|
||||
- **[D (Development)](#_development)**
|
||||
- Progressive implementation and code generation of factors.
|
||||
- Automated testing and validation of financial factors.
|
||||
|
||||
#### [Objective](#_summary)
|
||||
|
||||
To demonstrate the dynamic evolution of financial factors through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting financial factors.
|
||||
|
||||
qlib_factor_from_report_rich_style_description : |-
|
||||
### R&D Agent-Qlib: Automated Quantitative Trading & Factor Extraction from Financial Reports Demo
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
This demo showcases the process of extracting factors from financial research reports, implementing these factors, and analyzing their performance through Qlib backtest, continually expanding and refining the factor library.
|
||||
|
||||
#### [Automated R&D](#_rdloops)
|
||||
|
||||
- **[R (Research)](#_research)**
|
||||
- Iterative development of ideas and hypotheses from financial reports.
|
||||
- Continuous learning and knowledge construction.
|
||||
|
||||
- **[D (Development)](#_development)**
|
||||
- Progressive factor extraction and code generation.
|
||||
- Automated implementation and testing of financial factors.
|
||||
|
||||
#### [Objective](#_summary)
|
||||
|
||||
<table border="1" style="width:100%; border-collapse: collapse;">
|
||||
<tr>
|
||||
<td>💡 <strong>Innovation </strong></td>
|
||||
<td>Tool to quickly extract and test factors from research reports.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>⚡ <strong>Efficiency </strong></td>
|
||||
<td>Rapid identification of valuable factors from numerous reports.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>🗃️ <strong>Outputs </strong></td>
|
||||
<td>Expand and refine the factor library to support further research.</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
qlib_factor_experiment_setting: |-
|
||||
| Dataset 📊 | Model 🤖 | Factors 🌟 | Data Split 🧮 |
|
||||
|---------|----------|---------------|-------------------------------------------------|
|
||||
| EURUSD | LGBModel | Alpha158 Plus | Train: {{ train_start }} to {{ train_end }} <br> Valid: {{ valid_start }} to {{ valid_end }} <br> Test : {{ test_start }} to {{ test_end }} |
|
||||
|
||||
|
||||
qlib_model_background: |-
|
||||
The model is a machine learning or deep learning structure used in quantitative investment to predict the returns and risks of a portfolio or a single asset. Models are employed by investors to generate forecasts based on historical data and identified factors, which are central to many quantitative investment strategies.
|
||||
Each model takes the factors as input and predicts the future returns. Usually, the bigger the model is, the better the performance would be.
|
||||
The model is defined in the following parts:
|
||||
1. Name: The name of the model.
|
||||
2. Description: The description of the model.
|
||||
3. Architecture: The detailed architecture of the model, such as neural network layers or tree structures.
|
||||
4. Hyperparameters: The hyperparameters used in the model.
|
||||
5. Training_hyperparameters: The hyperparameters used during the training process.
|
||||
6. ModelType: The type of the model, "Tabular" for tabular model and "TimeSeries" for time series model.
|
||||
The model should provide clear and detailed documentation of its architecture and hyperparameters. One model should statically define one output with a fixed architecture and hyperparameters.
|
||||
|
||||
{% if runtime_environment is not none %}
|
||||
====== Runtime Environment ======
|
||||
You have following environment to run the code:
|
||||
{{ runtime_environment }}
|
||||
{% endif %}
|
||||
|
||||
qlib_model_interface: |-
|
||||
Your python code should follow the interface to better interact with the user's system.
|
||||
You code should contain several parts:
|
||||
1. The import part: import the necessary libraries.
|
||||
2. A class which is a sub-class of pytorch.nn.Module. This class should should have a init function and a forward function which inputs a tensor and outputs a tensor.
|
||||
3. Set a variable called "model_cls" to the class you defined.
|
||||
|
||||
The user will save your code into a python file called "model.py". Then the user imports model_cls in file "model.py" after setting the cwd into the directory:
|
||||
```python
|
||||
from model import model_cls
|
||||
```
|
||||
So your python code should follow the pattern:
|
||||
```python
|
||||
class XXXModel(torch.nn.Module):
|
||||
...
|
||||
model_cls = XXXModel
|
||||
```
|
||||
|
||||
The model can be configured as either "Tabular" for tabular models or "TimeSeries" for time series models. For a tabular model, the input shape is (batch_size, num_features), while for a time series model, the input shape is (batch_size, num_timesteps, num_features). In both cases, the output shape of the model should be (batch_size, 1).
|
||||
`num_features` will be directly set for the model based on the input data shape.
|
||||
User will initialize the tabular model with the following code:
|
||||
```python
|
||||
model = model_cls(num_features=num_features)
|
||||
```
|
||||
User will initialize the time series model with the following code:
|
||||
```python
|
||||
model = model_cls(num_features=num_features, num_timesteps=num_timesteps)
|
||||
```
|
||||
No other parameters will be passed to the model so give other parameters a default value or just make them static.
|
||||
|
||||
Don't write any try-except block in your python code. The user will catch the exception message and provide the feedback to you. Also, don't write main function in your python code. The user will call the forward method in the model_cls to get the output tensor.
|
||||
|
||||
Please notice that your model should only use current features as input. The user will provide the input tensor to the model's forward function.
|
||||
|
||||
|
||||
qlib_model_output_format: |-
|
||||
Your output should be a tensor with shape (batch_size, 1).
|
||||
The output tensor should be saved in a file named "output.pth" in the same directory as your python file.
|
||||
The user will evaluate the shape of the output tensor so the tensor read from "output.pth" should be 8 numbers.
|
||||
|
||||
qlib_model_simulator: |-
|
||||
The models will be sent into Qlib to train and evaluate their performance in predicting future returns. Hypothesis is improved upon checking the feedback on the results.
|
||||
Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning (RL).
|
||||
User will use Qlib to automatically perform the following tasks:
|
||||
1. Generate a baseline factor table.
|
||||
2. Train the model defined in your class Net to predict the next several days' returns based on the factor values.
|
||||
3. Build a portfolio based on the predicted returns using a specific strategy.
|
||||
4. Evaluate the portfolio's performance, including metrics such as return, IC, max drawdown, and others.
|
||||
5. Iterate on growing the hypothesis to enable model improvements based on performance evaluations and feedback.
|
||||
|
||||
qlib_model_rich_style_description: |-
|
||||
### Qlib Model Evolving Automatic R&D Demo
|
||||
|
||||
#### [Overview](#_summary)
|
||||
|
||||
The demo showcases the iterative process of hypothesis generation, knowledge construction, and decision-making in model construction in quantitative finance. It highlights how models evolve through continuous feedback and refinement.
|
||||
|
||||
#### [Automated R&D](#_rdloops)
|
||||
|
||||
- **[R (Research)](#_research)**
|
||||
- Iteration of ideas and hypotheses.
|
||||
- Continuous learning and knowledge construction.
|
||||
|
||||
- **[D (Development)](#_development)**
|
||||
- Evolving code generation and model refinement.
|
||||
- Automated implementation and testing of models.
|
||||
|
||||
#### [Objective](#_summary)
|
||||
|
||||
To demonstrate the dynamic evolution of models through the Qlib platform, emphasizing how each iteration enhances the accuracy and reliability of the resulting models.
|
||||
|
||||
qlib_model_experiment_setting: |-
|
||||
| Dataset 📊 | Model 🤖 | Factors 🌟 | Data Split 🧮 |
|
||||
|---------|----------|---------------|-------------------------------------------------|
|
||||
| EURUSD | RDAgent-dev | 20 factors (Alpha158) | Train: {{ train_start }} to {{ train_end }} <br> Valid: {{ valid_start }} to {{ valid_end }} <br> Test : {{ test_start }} to {{ test_end }} |
|
||||
@@ -0,0 +1,300 @@
|
||||
hypothesis_and_feedback: |-
|
||||
=========================================================
|
||||
{% for experiment, feedback in trace.hist %}
|
||||
# Trial {{ loop.index }}:
|
||||
## Hypothesis
|
||||
{{ experiment.hypothesis }}
|
||||
## Specific task:
|
||||
{% for task in experiment.sub_tasks %}
|
||||
{% if task is not none and task.get_task_brief_information is defined %}
|
||||
{{ task.get_task_brief_information() }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
## Backtest Analysis and Feedback:
|
||||
{% if experiment.result is not none %}
|
||||
Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
|
||||
{% endif %}
|
||||
Observation: {{ feedback.observations }}
|
||||
Hypothesis Evaluation: {{ feedback.hypothesis_evaluation }}
|
||||
Decision (Whether the hypothesis was successful): {{ feedback.decision }}
|
||||
=========================================================
|
||||
{% endfor %}
|
||||
|
||||
last_hypothesis_and_feedback: |-
|
||||
## Hypothesis
|
||||
{{ experiment.hypothesis }}
|
||||
## Specific task:
|
||||
{% for task in experiment.sub_tasks %}
|
||||
{% if task is not none and task.get_task_brief_information is defined %}
|
||||
{{ task.get_task_brief_information() }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
## Backtest Analysis and Feedback:
|
||||
{% if experiment.result is not none %}
|
||||
Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
|
||||
{% endif %}
|
||||
Training Log:
|
||||
Here, you need to focus on analyzing whether there are any issues with the training. If any problems are identified, you must correct them in the next iteration and clearly describe how the changes will be made in the hypothesis.
|
||||
{{ experiment.stdout }}
|
||||
Observation: {{ feedback.observations }}
|
||||
Evaluation: {{ feedback.hypothesis_evaluation }}
|
||||
Decision (Whether this experiment is SOTA): {{ feedback.decision }}
|
||||
New Hypothesis (Given in feedback stage, just for reference, and can be accepted or rejected in the next round): {{ feedback.new_hypothesis }}
|
||||
Reasoning (Justification for the new hypothesis): {{ feedback.reason }}
|
||||
|
||||
sota_hypothesis_and_feedback: |-
|
||||
## Hypothesis
|
||||
{{ experiment.hypothesis }}
|
||||
## Specific task:
|
||||
{% for task in experiment.sub_tasks %}
|
||||
{% if task is not none and task.get_task_brief_information is defined %}
|
||||
{{ task.get_task_brief_information() }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
## Backtest Analysis and Feedback:
|
||||
{% if experiment.result is not none %}
|
||||
Backtest Result: {{ experiment.result.loc[["IC", "1day.excess_return_without_cost.annualized_return", "1day.excess_return_without_cost.max_drawdown"]] }}
|
||||
{% endif %}
|
||||
Training Log: {{ experiment.stdout }}
|
||||
Observation: {{ feedback.observations }}
|
||||
Evaluation: {{ feedback.hypothesis_evaluation }}
|
||||
Decision (Whether this experiment is SOTA): {{ feedback.decision }}
|
||||
|
||||
hypothesis_output_format: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"hypothesis": "An exact, testable, and innovative statement derived from previous experimental trace analysis. Avoid overly general ideas and ensure precision. The hypothesis should clearly specify the exact approach and expected improvement in performance in two or three sentences.",
|
||||
"reason": "Provide a clear, logical explanation for why this hypothesis was proposed, grounded in evidence (e.g., trace history, domain principles). Reason should be short with no more than two sentences.",
|
||||
}
|
||||
|
||||
factor_hypothesis_output_format: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"hypothesis": "The new hypothesis generated based on the information provided. Limit in two or three sentences.",
|
||||
"reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them. Limit in two or three sentences.",
|
||||
}
|
||||
|
||||
hypothesis_output_format_with_action: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"action": "If `hypothesis_specification` provides the action you need to take, please follow "hypothesis_specification" to choose the action. Otherwise, based on previous experimental results, suggest the action you believe is most appropriate at the moment. It should be one of [`factor`, `model`].",
|
||||
"hypothesis": "The new hypothesis generated based on the information provided,should be a string.",
|
||||
"reason": "The reason why you generate this hypothesis. It should be comprehensive and logical. It should cover the other keys below and extend them. Limit in two or three sentences.",
|
||||
}
|
||||
|
||||
model_hypothesis_specification: |-
|
||||
1. First, observe and analyze the overall experimental progression in `hypothesis_and_feedback`. Analyze where the previous model designs were inadequate — whether it was due to parameter settings, architectural flaws, or a lack of novelty (proposing entirely new concepts is highly encouraged as long as they demonstrate effectiveness).
|
||||
2. Second, `last_hypothesis_and_feedback` and `sota_hypothesis_and_feedback` are key references you should pay close attention to. You can choose to optimize based on either of them or generate new ideas to form hypotheses and experiments.
|
||||
3. If there is no prior experiment or result available at the beginning, you can start by implementing a simple and small architecture.
|
||||
4. If a series of attempts fail to achieve SOTA, consider exploring entirely new directions; at this point, it is acceptable to return to simple architectures.
|
||||
5. Focus exclusively on the architecture of PyTorch models. Each hypothesis should specifically address architectural decisions, such as layer configurations, activation functions, regularization methods, and overall model structure. DO NOT do any feature-specific processing. Instead, you can propose innovative transformations on the input time-series data to enhance model training effectiveness.
|
||||
6. Avoid including aspects unrelated to architecture, such as input features or optimization strategies.
|
||||
7. Sometimes, when training performance is poor, adjusting hyperparameters can also be an effective strategy for improvement.
|
||||
8. Use standard libraries for baseline models, but also explore custom architecture designs to investigate novel structures. After sufficient trials with traditional models, aim for innovation comparable to top-tier AI conferences (NeurIPS, ICLR, ICML, SIGKDD, etc.) in time series modeling.
|
||||
|
||||
factor_hypothesis_specification: |-
|
||||
1. **1-5 Factors per Generation:**
|
||||
- Ensure each generation produces 1-5 factors.
|
||||
- Balance simplicity and complexity to build a robust factor library.
|
||||
- Make full use of the financial data provided to you instead of focusing solely on a specific field.
|
||||
2. **Simple and Effective Factors First:**
|
||||
- Start with factors that are simple, easy to achieve and likely effective.
|
||||
- Concisely explain why these factors are expected to work.
|
||||
- Avoid complex or combined factors initially.
|
||||
3. **Gradual Complexity Increase:**
|
||||
- Introduce more complex factors (e.g. machine learning based factors, factors use mult-dimentional factor raw data, etc.) as more experimental results are gathered.
|
||||
- Combine factors only after simpler ones are tested and validated.
|
||||
4. **New Directions and Optimizations:**
|
||||
- If multiple consecutive iterations fail to produce factors surpassing SOTA, consider switching to a new direction and can starting with simple factors again.
|
||||
- If optimizing a specific type of factor, proceed from simple to complex.
|
||||
5. Note
|
||||
- Highlight that factors surpassing SOTA are included in the library to avoid re-implementation.
|
||||
- No matter how many factors you plan to generate, only reply with one set of hypothesis and reason. The hypothesis can include the proposal of multiple factors at the same time.
|
||||
|
||||
factor_experiment_output_format: |-
|
||||
The output should follow JSON format. The schema is as follows:
|
||||
{
|
||||
"factor name 1": {
|
||||
"description": "description of factor 1, start with its type, e.g. [Momentum Factor]",
|
||||
"formulation": "latex formulation of factor 1",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
},
|
||||
"factor name 2": {
|
||||
"description": "description of factor 2, start with its type, e.g. [Machine Learning based Factor]",
|
||||
"formulation": "latex formulation of factor 2",
|
||||
"variables": {
|
||||
"variable or function name 1": "description of variable or function 1",
|
||||
"variable or function name 2": "description of variable or function 2"
|
||||
}
|
||||
}
|
||||
# Don't add ellipsis (...) or any filler text that might cause JSON parsing errors here!
|
||||
}
|
||||
|
||||
model_experiment_output_format: |-
|
||||
So far please only design one model to test the hypothesis!
|
||||
The output should follow JSON format. The schema is as follows (value in training_hyperparameters is a basic setting for reference, you CAN CHANGE depends on the previous training log):
|
||||
{
|
||||
"model_name (The name of the model)": {
|
||||
"description": "A detailed description of the model",
|
||||
"formulation": "A LaTeX formula representing the model's formulation",
|
||||
"architecture": "A detailed description of the model's architecture, e.g., neural network layers or tree structures",
|
||||
"variables": {
|
||||
"\\hat{y}_u": "The predicted output for node u",
|
||||
"variable_name_2": "Description of variable 2",
|
||||
"variable_name_3": "Description of variable 3"
|
||||
},
|
||||
"hyperparameters": {
|
||||
"hyperparameter_name_1": "value of hyperparameter 1",
|
||||
"hyperparameter_name_2": "value of hyperparameter 2",
|
||||
"hyperparameter_name_3": "value of hyperparameter 3"
|
||||
},
|
||||
"training_hyperparameters" { # All values are for reference; you can set them yourself
|
||||
"n_epochs": "100",
|
||||
"lr": "1e-3",
|
||||
"early_stop": 10,
|
||||
"batch_size": 256,
|
||||
"weight_decay": 1e-4,
|
||||
}
|
||||
"model_type": "Tabular or TimeSeries" # Should be one of "Tabular" or "TimeSeries"
|
||||
},
|
||||
}
|
||||
|
||||
factor_feedback_generation:
|
||||
system: |-
|
||||
You are a professional financial result analysis assistant in data-driven R&D.
|
||||
The task is described in the following scenario:
|
||||
|
||||
{{ scenario }}
|
||||
|
||||
You will receive a hypothesis, multiple tasks with their factors, their results, and the SOTA result.
|
||||
Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA (State of the Art) results, and suggest improvements or new directions.
|
||||
|
||||
Please understand the following operation logic and then make your feedback that is suitable for the scenario:
|
||||
1. Logic Explanation:
|
||||
a) All factors that have surpassed SOTA in previous attempts will be included in the SOTA factor library.
|
||||
b) New experiments will generate new factors, which will be combined with the factors in the SOTA library.
|
||||
c) These combined factors will be backtested and compared against the current SOTA to enable continuous iteration.
|
||||
2. Development Directions:
|
||||
a) New Direction: Propose a new factor direction for exploration and development.
|
||||
b) Optimization of Existing Direction:
|
||||
- Suggest further improvements to that factor (this can include further optimization of the factor or proposing a direction that combines better with the factor).
|
||||
- Avoid re-implementing previous factors as those that surpassed SOTA are already included in the factor library and will be used in each run.
|
||||
3. Final Goal: To continuously accumulate factors that surpass each iteration to maintain the best SOTA.
|
||||
|
||||
When judging the results:
|
||||
1. Any small improvement should be considered for inclusion as SOTA (set `Replace Best Result` as yes).
|
||||
2. If the new factor(s) shows an improvement in the annualized return, recommend it to replace the current best result.
|
||||
3. Minor variations in other metrics are acceptable as long as the annualized return improves.
|
||||
|
||||
Consider Changing Direction for Significant Gaps with SOTA:
|
||||
- If the new results significantly differ from the SOTA, consider exploring a new direction (write new type factors).
|
||||
- Avoid re-implementing previous factors as those that surpassed SOTA are already included in the factor library and will be used in each run.
|
||||
|
||||
Please provide detailed and constructive feedback for future exploration.
|
||||
Respond in JSON format. Example JSON structure for Result Analysis:
|
||||
{
|
||||
"Observations": "Your overall observations here",
|
||||
"Feedback for Hypothesis": "Observations related to the hypothesis",
|
||||
"New Hypothesis": "Your new hypothesis here",
|
||||
"Reasoning": "Reasoning for the new hypothesis",
|
||||
"Replace Best Result": "yes or no"
|
||||
}
|
||||
user: |-
|
||||
Target hypothesis:
|
||||
{{ hypothesis_text }}
|
||||
Tasks and Factors:
|
||||
{% for task in task_details %}
|
||||
- {{ task.factor_name }}: {{ task.factor_description }}
|
||||
- Factor Formulation: {{ task.factor_formulation }}
|
||||
- Variables: {{ task.variables }}
|
||||
- Factor Implementation: {{ task.factor_implementation }}
|
||||
{% if task.factor_implementation == "False" %}
|
||||
**Note: This factor was not implemented in the current experiment. Only the hypothesis for implemented factors can be verified.**
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
Combined Results:
|
||||
{{ combined_result }}
|
||||
|
||||
Analyze the combined result in the context of its ability to:
|
||||
1. Support or refute the hypothesis.
|
||||
2. Show improvement or deterioration compared to the SOTA experiment.
|
||||
|
||||
Note: Only factors with 'Factor Implementation' as True are implemented and tested in this experiment. If 'Factor Implementation' is False, the hypothesis for that factor cannot be verified in this run.
|
||||
|
||||
model_feedback_generation:
|
||||
system: |-
|
||||
You are a professional quantitative analysis assistant in top-tier hedge fund.
|
||||
|
||||
The task is described in the following scenario:
|
||||
{{ scenario }}
|
||||
|
||||
You will receive a quantitative model hypothesis, its specific task description, and it market backtest result.
|
||||
Your feedback should specify whether the current result supports or refutes the hypothesis, compare it with previous SOTA results, examine the model's training logs to analyze whether there are issues with hyperparameter settings, and suggest improvements or new directions.
|
||||
|
||||
Please provide detailed and constructive feedback.
|
||||
Example JSON Structure for Result Analysis:
|
||||
{
|
||||
"Observations": "First analyze the model's training logs to determine whether there are any issues with its parameter settings. Then clearly summarize the current results and the SOTA results with exact scores and any notable patterns. Limit your summary to no more than three concise, data-focused sentences.",
|
||||
"Feedback for Hypothesis": "Explicitly confirm or refute the hypothesis based on specific data points or performance trends. Limit to two sentences.",
|
||||
"New Hypothesis": "Propose a revised hypothesis, considering observed patterns and limitations in the current one. Limit to no more than two sentences.",
|
||||
"Reasoning": "Explain the rationale for the new hypothesis using specific trends or performance shifts. Be concise but technically complete. Limit to two sentences.",
|
||||
"Decision": <true or false>,
|
||||
}
|
||||
|
||||
|
||||
user: |-
|
||||
{% if sota_hypothesis %}
|
||||
# SOTA Round Information:
|
||||
Hypothesis: {{ sota_hypothesis.hypothesis }}
|
||||
Specific Task: {{ sota_task }}
|
||||
Code Implementation: {{ sota_code }}
|
||||
Result: {{ sota_result }}
|
||||
{% else %}
|
||||
# This is the first round. No previous information available. As long as the performance is not too negative (eg.ICIR is greater than 0), treat it as successful. Do not set the threshold too high.
|
||||
{% endif %}
|
||||
|
||||
# Current Round Information:
|
||||
Hypothesis: {{ hypothesis.hypothesis }}
|
||||
Why propose this hypothesis: {{ hypothesis.reason }}
|
||||
Specific Task: {{ exp.sub_tasks[0].get_task_information() }}
|
||||
Code Implementation: {{ exp.sub_workspace_list[0].file_dict.get("model.py") }}
|
||||
Training Log: {{ exp.stdout }}
|
||||
Result: {{ exp_result }}
|
||||
|
||||
# When judging the results:
|
||||
1. **Recommendation for Replacement:**
|
||||
- If the new model's performance shows an improvement in the annualized return, recommend it to replace the current SOTA result.
|
||||
- Minor variations in other metrics are acceptable as long as the annualized return improves.
|
||||
2. Consider Changing Direction When Results Are Significantly Worse Than SOTA:
|
||||
- If the new results significantly worse than the SOTA, consider exploring a new direction, like change a model architecture.
|
||||
|
||||
action_gen:
|
||||
system: |-
|
||||
Quantitative investment is a data-driven approach to asset management that relies on mathematical models, statistical techniques, and computational methods to analyze financial markets and make investment decisions. Two essential components of this approach are factors and models.
|
||||
|
||||
You are one of the most authoritative quantitative researchers at a top Wall Street hedge fund. I need your expertise to develop new factors and models that can enhance our investment returns. Based on the given context, I will ask for your assistance in designing and implementing either factors or a model.
|
||||
|
||||
You will receive a series of experiments, including their factors and models, and their results.
|
||||
Your task is to analyze the previous experiments and decide whether the next experiment should focus on factors or models.
|
||||
|
||||
Example JSON Structure for your return:
|
||||
{
|
||||
"action": "factor" or "model", # You must choose one of the two
|
||||
}
|
||||
|
||||
user: |-
|
||||
{% if hypothesis_and_feedback|length == 0 %}
|
||||
It is the first round of hypothesis generation. The user has no hypothesis on this scenario yet.
|
||||
{% else %}
|
||||
The former hypothesis and the corresponding feedbacks are as follows:
|
||||
{{ hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% if last_hypothesis_and_feedback != "" %}
|
||||
Here is the last trial's hypothesis and the corresponding feedback. The main feedback includes a new hypothesis for your reference only. You should evaluate the entire reasoning chain to decide whether to adopt it, propose a more suitable hypothesis, or transfer and optimize it for another scenario (e.g., factor/model), since transfers are generally encouraged:
|
||||
{{ last_hypothesis_and_feedback }}
|
||||
{% endif %}
|
||||
@@ -0,0 +1,63 @@
|
||||
exp_feedback:
|
||||
system: |-
|
||||
你是 RL post-training 专家，负责分析实验结果并生成反馈。
|
||||
|
||||
## 分析维度
|
||||
1. 训练是否成功完成
|
||||
2. 代码质量和实现正确性
|
||||
3. 是否达成假设目标
|
||||
4. 改进建议
|
||||
|
||||
## 输出要求
|
||||
JSON 格式：{"decision": true/false, "reason": "...", "suggestions": "..."}
|
||||
- decision: true 表示接受当前实验，false 表示拒绝
|
||||
- reason: 决策原因
|
||||
- suggestions: 下一步改进建议
|
||||
|
||||
user: |-
|
||||
## 假设
|
||||
{{ hypothesis }}
|
||||
|
||||
## 任务描述
|
||||
{{ task_desc }}
|
||||
|
||||
## 执行结果
|
||||
- exit_code: {{ exit_code }}
|
||||
- running_time: {{ running_time }}s
|
||||
{% if stdout %}
|
||||
- stdout (前1000字符):
|
||||
{{ stdout[:1000] }}
|
||||
{% endif %}
|
||||
{% if benchmark %}
|
||||
## Benchmark 结果
|
||||
{{ benchmark }}
|
||||
{% endif %}
|
||||
|
||||
{% if exception %}
|
||||
## 异常信息
|
||||
{{ exception }}
|
||||
{% endif %}
|
||||
|
||||
请分析实验结果并给出反馈。
|
||||
|
||||
exp_feedback_error:
|
||||
system: |-
|
||||
你是 RL post-training 专家，负责分析失败的实验。
|
||||
|
||||
## 常见错误类型
|
||||
- ImportError: 缺少依赖库
|
||||
- SyntaxError: 代码语法错误
|
||||
- RuntimeError: 运行时错误（OOM、CUDA 等）
|
||||
- API 不兼容: 库版本问题
|
||||
|
||||
## 输出要求
|
||||
JSON 格式：{"error_type": "...", "root_cause": "...", "fix_suggestion": "..."}
|
||||
|
||||
user: |-
|
||||
## 假设
|
||||
{{ hypothesis }}
|
||||
|
||||
## 错误信息
|
||||
{{ error_info }}
|
||||
|
||||
请分析错误原因并给出修复建议。
|
||||
@@ -0,0 +1,80 @@
|
||||
hypothesis_gen:
|
||||
system: |-
|
||||
你是 RL post-training 专家，负责生成训练假设。
|
||||
|
||||
## 核心目标
|
||||
**提升模型在 benchmark 上的分数**，这是唯一目标。
|
||||
|
||||
## 运行环境
|
||||
代码由系统自动部署到 `$WORKSPACE/code/` 并执行。
|
||||
环境变量（已由框架设置，代码中直接 `os.environ` 读取）：
|
||||
- `MODEL_PATH`: 基础模型路径（只读）
|
||||
- `DATA_PATH`: 训练数据路径（只读）
|
||||
- `OUTPUT_DIR`: 模型输出目录（`$WORKSPACE/output/`）
|
||||
- `GRADING_SERVER_URL`: 评测服务地址
|
||||
|
||||
## 评测机制
|
||||
训练完成后，系统自动将 `$OUTPUT_DIR` 下最新的模型提交到 Grading Server 评测。
|
||||
- `$OUTPUT_DIR` 下有模型 → 自动提交评测，返回 score
|
||||
- `$OUTPUT_DIR` 为空 → 跳过评测
|
||||
- 可用子目录区分版本（如 `output/v1/`、`output/v2/`），系统取最新的
|
||||
|
||||
## 策略选择
|
||||
|
||||
### 情况1：首次运行 / 代码一直失败（exit_code≠0）
|
||||
- 生成简单、稳定的训练代码
|
||||
- 目标：让代码能跑通（exit_code=0）
|
||||
- 可以先不保存模型，验证链路
|
||||
|
||||
### 情况2：代码稳定但没有评测分数
|
||||
- **说明训练没有保存模型到 $OUTPUT_DIR**
|
||||
- 现在应该生成**正式训练**假设
|
||||
- 必须保存模型到 $OUTPUT_DIR
|
||||
|
||||
### 情况3：已有评测分数，需要优化
|
||||
- 关注超参数调优
|
||||
- 尝试不同算法或配置
|
||||
- 每次改动一个变量，便于归因
|
||||
|
||||
## 可用算法
|
||||
- **GRPO**: 推荐，数学推理效果好，不需要偏好对
|
||||
- DPO: 需要 (chosen, rejected) 偏好对
|
||||
- PPO/RLOO: 其他选择
|
||||
|
||||
## 框架
|
||||
- trl (版本 0.27+): GRPOTrainer, DPOTrainer, PPOTrainer
|
||||
|
||||
## 输出要求
|
||||
JSON 格式：
|
||||
{
|
||||
"hypothesis": "具体的训练策略描述",
|
||||
"reason": "为什么这样做，基于历史分析",
|
||||
"algorithm": "GRPO/DPO/PPO/RLOO",
|
||||
"is_formal_training": true/false
|
||||
}
|
||||
|
||||
- is_formal_training=true: 正式训练，会保存模型到 $OUTPUT_DIR
|
||||
- is_formal_training=false: 调试/验证，不保存模型
|
||||
|
||||
user: |-
|
||||
## 基础模型
|
||||
{{ base_model }}
|
||||
|
||||
## 历史实验
|
||||
{% if trace_summary %}
|
||||
{{ trace_summary }}
|
||||
|
||||
**请分析历史：**
|
||||
1. exit_code 情况：有多少次成功(0)/失败(非0)？
|
||||
2. benchmark 分数：是数字还是 None？
|
||||
- 如果是 None：说明没有保存模型，需要正式训练
|
||||
- 如果是数字：可以基于此优化
|
||||
3. 错误模式：是否有重复的错误？如何避免？
|
||||
{% else %}
|
||||
无历史实验（首次运行）
|
||||
- 建议：生成简单稳定的 GRPO 训练代码
|
||||
- 目标：先让代码跑通，验证训练链路
|
||||
{% endif %}
|
||||
|
||||
请生成下一轮实验假设。
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
filter_redundant_text:
|
||||
system: |
|
||||
You are an assistant designed to analyze and filter text containing training log messages, repeated warning messages, and progress bar outputs. Your task is to examine the text and determine whether these patterns are present.
|
||||
1. Training log messages should be evaluated based on their usefulness—logs that contain meaningful training metrics such as loss or accuracy reported at each epoch should be retained, while redundant messages, such as those repeatedly reporting NaN values or iteration numbers without valuable information, should be removed.
|
||||
2. For warning messages, **only one occurrence of each unique message should be kept**, eliminating any duplicates.
|
||||
3. Additionally, any visual progress indicators, such as ASCII-based progress bars or dynamic percentage updates, should be removed. Once these patterns are identified, you should generate appropriate regex expressions to filter them out.
|
||||
4. Don't remove useful information that is not duplicated.
|
||||
5. Lastly, indicate whether substitution is needed in `needs_sub` field. If the input exceeds a token limit, the system will provide only a shortened portion of the text.
|
||||
|
||||
Respond in the following JSON format and order:
|
||||
{
|
||||
"needs_sub": <true/false>,
|
||||
"regex_patterns": ["regex pattern 1", "regex pattern 2", ...]
|
||||
}
|
||||
user: |
|
||||
The following text contains stdout:
|
||||
|
||||
{{ stdout }}
|
||||
|
||||
Check if the text contains training log messages, repeated warning messages, and progress bar patterns. If patterns are found, provide a list of regex patterns to filter them. Otherwise, indicate that substitution is not needed.
|
||||
@@ -236,12 +236,14 @@ class FactorDatetimeDailyEvaluator(FactorEvaluator):
|
||||
)
|
||||
|
||||
time_diff = pd.to_datetime(gen_df.index.get_level_values("datetime")).to_series().diff().dropna().unique()
|
||||
if pd.Timedelta(minutes=1) in time_diff:
|
||||
# For EURUSD 1min trading, intraday data (1min to 30min bars) is CORRECT
|
||||
if any(pd.Timedelta(minutes=m) in time_diff for m in range(1, 31)):
|
||||
return (
|
||||
"The generated dataframe is not daily. The implementation is definitely wrong. Please check the implementation.",
|
||||
False,
|
||||
"The generated dataframe uses intraday frequency (1min-30min bars). This is correct for EURUSD intraday trading.",
|
||||
True,
|
||||
)
|
||||
return "The generated dataframe is daily.", True
|
||||
# Daily data would also be acceptable for some strategies
|
||||
return "The generated dataframe uses daily or non-standard frequency. Verify this matches the factor specification.", True
|
||||
|
||||
|
||||
class FactorRowCountEvaluator(FactorEvaluator):
|
||||
|
||||
@@ -1,11 +1,19 @@
|
||||
from pathlib import Path
|
||||
"""
|
||||
Qlib Factor Runner - Executes factor backtests in Docker.
|
||||
|
||||
NOTE: The @cache_with_pickle decorator was REMOVED from develop() because:
|
||||
- Backtests should ALWAYS run fresh — caching causes stale results
|
||||
- Each hypothesis may have different code even with same task info
|
||||
- Docker-level caching (QlibDockerConf.enable_cache=False) is sufficient
|
||||
- The pickle cache caused 240+ factor generations but ZERO Docker backtests
|
||||
"""
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import pandas as pd
|
||||
from pandarallel import pandarallel
|
||||
|
||||
from rdagent.core.conf import RD_AGENT_SETTINGS
|
||||
from rdagent.core.utils import cache_with_pickle
|
||||
|
||||
pandarallel.initialize(verbose=1)
|
||||
|
||||
@@ -61,12 +69,17 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
|
||||
IC_max = IC_max.unstack().max(axis=0)
|
||||
return new_feature.iloc[:, IC_max[IC_max < 0.99].index]
|
||||
|
||||
@cache_with_pickle(CachedRunner.get_cache_key, CachedRunner.assign_cached_result)
|
||||
def develop(self, exp: QlibFactorExperiment) -> QlibFactorExperiment:
|
||||
"""
|
||||
Generate the experiment by processing and combining factor data,
|
||||
then passing the combined data to Docker for backtest results.
|
||||
|
||||
NOTE: @cache_with_pickle decorator was REMOVED. Every experiment
|
||||
triggers a fresh Docker backtest — no cached results are used.
|
||||
"""
|
||||
# Ensure all results directories exist
|
||||
self._ensure_results_dirs()
|
||||
|
||||
if exp.based_experiments and exp.based_experiments[-1].result is None:
|
||||
logger.info(f"Baseline experiment execution ...")
|
||||
exp.based_experiments[-1] = self.develop(exp.based_experiments[-1])
|
||||
@@ -228,6 +241,9 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to save results to database: {e}")
|
||||
|
||||
# Always write a log entry for every run (success or failure)
|
||||
self._write_run_log(exp, result)
|
||||
|
||||
return exp
|
||||
|
||||
def _validate_result(self, exp, result) -> dict:
|
||||
@@ -688,3 +704,99 @@ class QlibFactorRunner(CachedRunner[QlibFactorExperiment]):
|
||||
# Mark factor as rejected by protection
|
||||
exp.rejected_by_protection = True
|
||||
exp.protection_reason = protection_result.reason
|
||||
|
||||
def _write_run_log(self, exp, result) -> None:
|
||||
"""
|
||||
Write a log entry for EVERY run (success or failure) to results/logs/.
|
||||
|
||||
This ensures we have a record of every factor attempt, even if it failed
|
||||
validation or had no valid metrics.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
exp : QlibFactorExperiment
|
||||
The experiment object
|
||||
result : pd.Series or dict or None
|
||||
Backtest result (can be None if execution failed)
|
||||
"""
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
factor_name = "unknown"
|
||||
if hasattr(exp, 'hypothesis') and exp.hypothesis is not None:
|
||||
factor_name = getattr(exp.hypothesis, 'hypothesis', 'unknown')
|
||||
|
||||
# Build log entry
|
||||
log_entry = {
|
||||
"factor_name": factor_name,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"status": "unknown",
|
||||
"ic": None,
|
||||
"sharpe": None,
|
||||
"annualized_return": None,
|
||||
"max_drawdown": None,
|
||||
"win_rate": None,
|
||||
"rejected_by_protection": getattr(exp, 'rejected_by_protection', False),
|
||||
"protection_reason": getattr(exp, 'protection_reason', None),
|
||||
}
|
||||
|
||||
# Extract metrics if available
|
||||
if result is not None:
|
||||
if hasattr(result, 'get'): # pd.Series or dict
|
||||
ic_val = result.get('IC', result.get('ic', None))
|
||||
log_entry['ic'] = self._safe_float(ic_val) if ic_val is not None else None
|
||||
|
||||
sharpe_val = result.get('1day.excess_return_with_cost.shar',
|
||||
result.get('1day.excess_return_with_cost.sharpe',
|
||||
result.get('sharpe', None)))
|
||||
log_entry['sharpe'] = self._safe_float(sharpe_val) if sharpe_val is not None else None
|
||||
|
||||
ann_ret = result.get('1day.excess_return_with_cost.annualized_return',
|
||||
result.get('annualized_return', None))
|
||||
log_entry['annualized_return'] = self._safe_float(ann_ret) if ann_ret is not None else None
|
||||
|
||||
mdd = result.get('1day.excess_return_with_cost.max_drawdown',
|
||||
result.get('max_drawdown', None))
|
||||
log_entry['max_drawdown'] = self._safe_float(mdd) if mdd is not None else None
|
||||
|
||||
wr = result.get('win_rate', None)
|
||||
log_entry['win_rate'] = self._safe_float(wr) if wr is not None else None
|
||||
|
||||
# Determine status
|
||||
if log_entry['ic'] is not None or log_entry['sharpe'] is not None:
|
||||
log_entry['status'] = "success"
|
||||
elif getattr(exp, 'rejected_by_protection', False):
|
||||
log_entry['status'] = "rejected_protection"
|
||||
else:
|
||||
log_entry['status'] = "no_valid_metrics"
|
||||
else:
|
||||
log_entry['status'] = "execution_failed"
|
||||
log_entry['reason'] = "Result was None"
|
||||
|
||||
# Write to results/logs/
|
||||
try:
|
||||
project_root = Path(__file__).parent.parent.parent.parent.parent
|
||||
log_dir = project_root / "results" / "logs"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# One file per day
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
log_file = log_dir / f"factor_runs_{today}.jsonl"
|
||||
|
||||
with open(log_file, "a") as f:
|
||||
f.write(json.dumps(log_entry, ensure_ascii=False) + "\n")
|
||||
|
||||
logger.info(
|
||||
f"Run log written for '{factor_name[:50]}': "
|
||||
f"status={log_entry['status']}, IC={log_entry['ic']}, Sharpe={log_entry['sharpe']}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to write run log: {e}")
|
||||
|
||||
def _ensure_results_dirs(self) -> None:
|
||||
"""Ensure all results directories exist."""
|
||||
from pathlib import Path
|
||||
project_root = Path(__file__).parent.parent.parent.parent.parent
|
||||
for subdir in ["results/runs", "results/factors", "results/logs", "results/backtests", "results/db"]:
|
||||
(project_root / subdir).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -10,7 +10,7 @@ NOTE: **key is always "data" for all hdf5 files **.
|
||||
|
||||
| Filename | Description |
|
||||
| -------------- | -----------------------------------------------------------------|
|
||||
| "daily_pv.h5" | EURUSD 1-minute price and volume data (2020-2026). |
|
||||
| "intraday_pv.h5" | EURUSD 1-minute OHLCV intraday data (2020-2026). |
|
||||
|
||||
|
||||
# For different data, We have some basic knowledge for them
|
||||
|
||||
@@ -12,7 +12,7 @@ fields = ["$open", "$close", "$high", "$low", "$volume"]
|
||||
# Start: 2020-01-01, End: 2026-03-20
|
||||
data = D.features(instruments, fields, freq="1min").swaplevel().sort_index()
|
||||
|
||||
data.to_hdf("./daily_pv_all.h5", key="data")
|
||||
data.to_hdf("./intraday_pv_all.h5", key="data")
|
||||
|
||||
|
||||
# Debug-Daten: Nur letzte ~100 Instrumente für schnelleres Testing
|
||||
@@ -32,8 +32,8 @@ data_debug = (
|
||||
.sort_index()
|
||||
)
|
||||
|
||||
data_debug.to_hdf("./daily_pv_debug.h5", key="data")
|
||||
data_debug.to_hdf("./intraday_pv_debug.h5", key="data")
|
||||
|
||||
print(f"Generated daily_pv_all.h5 with {len(data)} rows")
|
||||
print(f"Generated daily_pv_debug.h5 with {len(data_debug)} rows")
|
||||
print(f"Generated intraday_pv_all.h5 with {len(data)} rows")
|
||||
print(f"Generated intraday_pv_debug.h5 with {len(data_debug)} rows")
|
||||
print(f"Date range: {data.index.min()} to {data.index.max()}")
|
||||
|
||||
@@ -21,19 +21,19 @@ def generate_data_folder_from_qlib():
|
||||
entry=f"python generate.py",
|
||||
)
|
||||
|
||||
assert (Path(__file__).parent / "factor_data_template" / "daily_pv_all.h5").exists(), (
|
||||
"daily_pv_all.h5 is not generated. It means rdagent/scenarios/qlib/experiment/factor_data_template/generate.py is not executed correctly. Please check the log: \n"
|
||||
assert (Path(__file__).parent / "factor_data_template" / "intraday_pv_all.h5").exists(), (
|
||||
"intraday_pv_all.h5 is not generated. It means rdagent/scenarios/qlib/experiment/factor_data_template/generate.py is not executed correctly. Please check the log: \n"
|
||||
+ execute_log
|
||||
)
|
||||
assert (Path(__file__).parent / "factor_data_template" / "daily_pv_debug.h5").exists(), (
|
||||
"daily_pv_debug.h5 is not generated. It means rdagent/scenarios/qlib/experiment/factor_data_template/generate.py is not executed correctly. Please check the log: \n"
|
||||
assert (Path(__file__).parent / "factor_data_template" / "intraday_pv_debug.h5").exists(), (
|
||||
"intraday_pv_debug.h5 is not generated. It means rdagent/scenarios/qlib/experiment/factor_data_template/generate.py is not executed correctly. Please check the log: \n"
|
||||
+ execute_log
|
||||
)
|
||||
|
||||
Path(FACTOR_COSTEER_SETTINGS.data_folder).mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy(
|
||||
Path(__file__).parent / "factor_data_template" / "daily_pv_all.h5",
|
||||
Path(FACTOR_COSTEER_SETTINGS.data_folder) / "daily_pv.h5",
|
||||
Path(__file__).parent / "factor_data_template" / "intraday_pv_all.h5",
|
||||
Path(FACTOR_COSTEER_SETTINGS.data_folder) / "intraday_pv.h5",
|
||||
)
|
||||
shutil.copy(
|
||||
Path(__file__).parent / "factor_data_template" / "README.md",
|
||||
@@ -42,8 +42,8 @@ def generate_data_folder_from_qlib():
|
||||
|
||||
Path(FACTOR_COSTEER_SETTINGS.data_folder_debug).mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy(
|
||||
Path(__file__).parent / "factor_data_template" / "daily_pv_debug.h5",
|
||||
Path(FACTOR_COSTEER_SETTINGS.data_folder_debug) / "daily_pv.h5",
|
||||
Path(__file__).parent / "factor_data_template" / "intraday_pv_debug.h5",
|
||||
Path(FACTOR_COSTEER_SETTINGS.data_folder_debug) / "intraday_pv.h5",
|
||||
)
|
||||
shutil.copy(
|
||||
Path(__file__).parent / "factor_data_template" / "README.md",
|
||||
|
||||
@@ -97,7 +97,7 @@ classify_system: |-
|
||||
|
||||
factor_viability_system: |-
|
||||
User has designed several factors in quant investment. Please help the user to check the viability of these factors.
|
||||
These factors are used to build a daily frequency strategy in EURUSD 1min intraday FX market.
|
||||
These factors are used to build an intraday strategy on EURUSD 1-minute bars in the FX market.
|
||||
|
||||
User will provide a pandas dataframe like table containing following information:
|
||||
1. The name of the factor;
|
||||
@@ -114,7 +114,7 @@ factor_viability_system: |-
|
||||
|
||||
|
||||
A viable factor should satisfy the following conditions:
|
||||
1. The factor should be able to be calculated in daily frequency;
|
||||
1. The factor should be calculable at 1-minute frequency using OHLCV data;
|
||||
2. The factor should be able to be calculated based on each stock;
|
||||
3. The factor should be able to be calculated based on the source data provided by user.
|
||||
|
||||
@@ -145,7 +145,7 @@ factor_viability_system: |-
|
||||
|
||||
factor_relevance_system: |-
|
||||
User has designed several factors in quant investment. Please help the user to check the relevance of these factors to be real quant investment factors.
|
||||
These factors are used to build a daily frequency strategy in EURUSD 1min intraday FX market.
|
||||
These factors are used to build an intraday strategy on EURUSD 1-minute bars in the FX market.
|
||||
|
||||
User will provide a pandas dataframe like table containing following information:
|
||||
1. The name of the factor;
|
||||
@@ -154,7 +154,7 @@ factor_relevance_system: |-
|
||||
4. The description to the variables and functions in the formulation of the factor.
|
||||
|
||||
A relevant factor should satisfy the following conditions:
|
||||
1. The factor should be able to be calculated in daily frequency;
|
||||
1. The factor should be calculable at 1-minute frequency using OHLCV data;
|
||||
2. The factor should be able to be calculated based on each stock;
|
||||
3. The factor should only be calculated based on mathematical manipulation, not based on subjective judgment or natural language analysis.
|
||||
|
||||
@@ -186,7 +186,7 @@ factor_relevance_system: |-
|
||||
|
||||
factor_duplicate_system: |-
|
||||
User has designed several factors in quant investment. Please help the user to duplicate these factors.
|
||||
These factors are used to build a daily frequency strategy in EURUSD 1min intraday FX market.
|
||||
These factors are used to build an intraday strategy on EURUSD 1-minute bars in the FX market.
|
||||
|
||||
User will provide a pandas dataframe like table containing following information:
|
||||
1. The name of the factor;
|
||||
|
||||
Reference in New Issue
Block a user