feat: enable finetune llm (#1055)

* feat: start with previous workspace

* feat: finetune llm

* add PrevModelLoadEvaluator

---------

Co-authored-by: Young <afe.young@gmail.com>
Co-authored-by: v-jianwan <v-jianwan@microsoft.com>
Co-authored-by: you-n-g <you-n-g@users.noreply.github.com>

docs/scens/catalog.rst

@@ -43,3 +43,4 @@ The supported scenarios are listed below:
     model_agent_fin
     model_copilot_general
     data_science
+    finetune

docs/scens/finetune.rst

@@ -0,0 +1,163 @@
+.. _finetune_agent:
+
+=============================
+Fine-tuning an Existing Model
+=============================
+
+## **🎯 Scenario: Continue Training on a Pre-trained Model**
+
+In this workflow the **Data Science Agent** starts from a *previously trained* model (and its training script), performs additional fine-tuning on new data, and then re-uses the updated weights for subsequent inference runs.
+
+🚧 Directory Structure
+
+Your competition folder (here called ``custom_data``) must contain **one extra sub-directory** named ``prev_model`` where you keep the old weights and the code that produced them:
+
+.. code-block:: text
+
+   ds_data
+   └── custom_data
+       ├── train.csv
+       ├── test.csv
+       ├── sample_submission.csv      # optional
+       ├── description.md             # optional
+       ├── sample.py                  # optional
+       └── prev_model                 # ← NEW
+           ├── models/                #   previous checkpoints (e.g. *.bin, *.pt, *.ckpt)
+           └── main.py                  #   training/inference scripts you used before
+
+If your competition provides custom grading/validation scripts, keep them under ``ds_data/eval/custom_data`` exactly as before.
+
+🔧 Environment Setup
+~~~~~~~~~~~~~~~~~~~~~~
+
+Add or update the following variables in **.env** (examples shown):
+
+.. code-block:: sh
+
+   # required for all Data-Science runs
+   dotenv set DS_LOCAL_DATA_PATH <your local path>/ds_data
+
+   # optional: choose docker / conda, etc.
+   dotenv set DS_CODER_COSTEER_ENV_TYPE docker
+
+🚀 How It Works at Runtime
+
+1. **First run**
+
+   * `rdagent` detects `prev_model/models`.
+   * It loads the latest checkpoint and prepare the fine-tuning based on code found under `prev_model/*.py` (or your own pipeline if you override it).
+   * Fine-tuned weights are written to `./workspace_input/models`.
+
+2. **Subsequent runs**
+
+   * When you execute `python ./workspace_input/main.py`, the script first looks for a checkpoint in `./workspace_input/models`.
+   * If found, it **skips fine-tuning** and goes straight to prediction / submission generation.
+
+⏰ Managing Timeouts
+
+
+By default:
+
+* **Debug loop**: 1 hour (``DS_DEBUG_TIMEOUT=3600`` seconds)  
+* **Full run**  : 3 hours (``DS_FULL_TIMEOUT=10800`` seconds)
+
+Override either value in **.env**:
+
+.. code-block:: sh
+
+   # give the debug loop 45 min and the full loop 6 h
+   dotenv set DS_DEBUG_TIMEOUT 2700
+   dotenv set DS_FULL_TIMEOUT 21600
+
+- 🚀 **Run the Application**
+
+  - You can directly run the application by using the following command:
+    
+    .. code-block:: sh
+
+        dotenv run -- python rdagent/app/finetune/data_science/loop.py --competition <Competition ID>
+
+  - Then, you can run the test set score corresponding to each round of the loop.
+
+    .. code-block:: sh
+
+        dotenv run -- python rdagent/log/mle_summary.py grade <url_to_log>
+
+    Here, <url_to_log> refers to the parent directory of the log folder generated during the run.
+
+- 📥 **Visualize the R&D Process**
+
+  - We provide a web UI to visualize the log. You just need to run:
+
+    .. code-block:: sh
+
+        streamlit run rdagent/log/ui/dsapp.py
+
+  - Then you can input the log path and visualize the R&D process.
+
+🔍 MLE-bench Guide: Running ML Engineering via MLE-bench
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- 📝 **MLE-bench Overview**
+
+  - MLE-bench is a comprehensive benchmark designed to evaluate the ML engineering capabilities of AI systems using real-world scenarios. The dataset comprises 75 Kaggle competitions. Since Kaggle does not provide held-out test sets for these competitions, the benchmark includes preparation scripts that split the publicly available training data into new training and test sets, and grading scripts are provided for each competition to accurately evaluate submission scores.
+
+- 🔧 **Set up Environment for MLE-bench**
+
+  - Running R&D-Agent on MLE-bench is designed for full automation. There is no need for manual downloads and data preparation. Simply set the environment variable ``DS_IF_USING_MLE_DATA`` to True.  
+
+  - At runtime, R&D-Agent will automatically build the Docker image specified at ``rdagent/scenarios/kaggle/docker/mle_bench_docker/Dockerfile``. This image is responsible for downloading the required datasets and grading files for MLE-bench.  
+  
+  - Note: The first run may take longer than subsequent runs as the Docker image and data are being downloaded and set up for the first time.
+
+    .. code-block:: sh
+
+        dotenv set DS_LOCAL_DATA_PATH <your local directory>/ds_data
+        dotenv set DS_IF_USING_MLE_DATA True
+
+- 🔨 **Configuring the Kaggle API**
+
+  - Downloading Kaggle competition data requires the Kaggle API. You can set up the Kaggle API by following these steps:
+  
+    - Register and login on the `Kaggle <https://www.kaggle.com/>`_ website.
+
+    - Click on the avatar (usually in the top right corner of the page) -> ``Settings`` -> ``Create New Token``, A file called ``kaggle.json`` will be downloaded.
+
+    - Move ``kaggle.json`` to ``~/.config/kaggle/``
+
+    - Modify the permissions of the ``kaggle.json`` file.
+
+      .. code-block:: sh
+
+        chmod 600 ~/.config/kaggle/kaggle.json
+
+  - For more information about Kaggle API Settings, refer to the `Kaggle API <https://github.com/Kaggle/kaggle-api>`_.
+
+
+- 🔩 **Setting the Environment Variables for MLE-bench**
+
+  - In addition to auto-downloading the benchmark data, you must also configure the runtime environment for executing the competition code.  
+  - Use the environment variable ``DS_CODER_COSTEER_ENV_TYPE`` to select the execution mode:
+    
+    • When set to docker (the default), RD-Agent utilizes the official Kaggle Docker image (``gcr.io/kaggle-gpu-images/python:latest``) to ensure that all required packages are available.  
+    • If you prefer to use a custom Docker setup, you can modify the configuration using ``DS_DOCKER_IMAGE`` or ``DS_DOCKERFILE_FOLDER_PATH``.  
+    • Alternatively, if your competition work only demands basic libraries, you may set ``DS_CODER_COSTEER_ENV_TYPE`` to conda. In this mode, you must create a local conda environment named “kaggle” and pre-install the necessary packages. RD-Agent will execute the competition code within this “kaggle” conda environment.
+
+    .. code-block:: sh
+
+      # Configure the runtime environment: choice between 'docker' (default) or 'conda'
+      dotenv set DS_CODER_COSTEER_ENV_TYPE docker
+
+- **Additional Guidance**
+
+  - **Combine different LLM Models at R&D Stage**
+
+    - You can combine different LLM models at the R&D stage. 
+
+    - By default, when you set environment variable ``CHAT_MODEL``, it covers both R&D stages. When customizing the model for the development stage, you can set:
+    
+    .. code-block:: sh
+
+      # This example sets the model to "o3-mini". For some models, the reasoning effort shoule be set to "None".
+      dotenv set LITELLM_CHAT_MODEL_MAP '{"coding":{"model":"o3-mini","reasoning_effort":"high"},"running":{"model":"o3-mini","reasoning_effort":"high"}}'
+

rdagent/app/finetune/data_science/conf.py

@@ -0,0 +1,40 @@
+import os
+
+from pydantic_settings import SettingsConfigDict
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.core.conf import RD_AGENT_SETTINGS, ExtendedBaseSettings
+
+
+class DSFinetuneScen(ExtendedBaseSettings):
+    model_config = SettingsConfigDict(env_prefix="FT_", protected_namespaces=())
+    scen: str = "rdagent.app.finetune.data_science.scen.DSFinetuneScen"
+    """
+    Scenario class for data science tasks.
+    - For Kaggle competitions, use: "rdagent.scenarios.data_science.scen.KaggleScen"
+    - For custom data science scenarios, use: "rdagent.scenarios.data_science.scen.DataScienceScen"
+    - For LLM finetune scenarios, use: "rdagent.app.finetune.llm.scen.LLMFinetuneScen"
+    - For Data science finetune scenarios, use: "rdagent.app.finetune.data_science.scen.DSFinetuneScen"
+    """
+
+    debug_timeout: int = 3600
+    """The timeout limit for running on debugging data"""
+    full_timeout: int = 10800
+    """The timeout limit for running on full data"""
+
+    coder_on_whole_pipeline: bool = True
+    enable_model_dump: bool = True
+    app_tpl: str = "app/finetune/data_science/tpl"
+
+
+def update_settings(competition: str):
+    """
+    Update the RD_AGENT_SETTINGS with the values from DS_FINETUNE_SETTINGS.
+    """
+    DS_FINETUNE_SETTINGS = DSFinetuneScen()
+    RD_AGENT_SETTINGS.app_tpl = DS_FINETUNE_SETTINGS.app_tpl
+    os.environ["DS_CODER_COSTEER_EXTRA_EVALUATOR"] = '["rdagent.app.finetune.share.eval.PrevModelLoadEvaluator"]'
+    for field_name, new_value in DS_FINETUNE_SETTINGS.model_dump().items():
+        if hasattr(DS_RD_SETTING, field_name):
+            setattr(DS_RD_SETTING, field_name, new_value)
+    DS_RD_SETTING.competition = competition

rdagent/app/finetune/data_science/loop.py

@@ -0,0 +1,40 @@
+import asyncio
+from pathlib import Path
+
+import fire
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.app.finetune.data_science.conf import update_settings
+from rdagent.core.utils import import_class
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.data_science.loop import DataScienceRDLoop
+
+
+def main(
+    model: str | None = None,
+    competition: str | None = None,
+):
+    """
+    Parameters
+    ----------
+    competition :
+        Competition name.
+
+    Auto R&D Evolving loop for models finetune.
+    You can continue running a session by using the command:
+    .. code-block:: bash
+        dotenv run -- python rdagent/app/finetune/data_science/loop.py --competition aerial-cactus-identification
+    """
+    if not competition:
+        raise Exception("Please specify competition name.")
+
+    model_folder = Path(DS_RD_SETTING.local_data_path) / competition / "prev_model"
+    if not model_folder.exists():
+        raise Exception(f"Please put the model path to {model_folder}.")
+    update_settings(competition)
+    rd_loop: DataScienceRDLoop = DataScienceRDLoop(DS_RD_SETTING)
+    asyncio.run(rd_loop.run())
+
+
+if __name__ == "__main__":
+    fire.Fire(main)

rdagent/app/finetune/data_science/scen.py

@@ -0,0 +1,20 @@
+from pathlib import Path
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.core.scenario import Scenario
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.data_science.scen import DataScienceScen
+from rdagent.scenarios.data_science.scen.utils import describe_data_folder_v2
+from rdagent.utils.agent.tpl import T
+
+
+class DSFinetuneScen(DataScienceScen):
+    """DSFinetuneScen Scenario"""
+
+    def _get_data_folder_description(self) -> str:
+        folder_desc = describe_data_folder_v2(
+            Path(DS_RD_SETTING.local_data_path) / self.competition,
+            show_nan_columns=DS_RD_SETTING.show_nan_columns,
+            max_length=20000,  # more context for model script
+        )
+        return folder_desc

rdagent/app/finetune/data_science/tpl/components/coder/data_science/pipeline/prompts.yaml

@@ -0,0 +1,4 @@
+pipeline_coder:
+  system: |-
+    {% include "rdagent.components.coder.data_science.pipeline.prompts:pipeline_coder.system" %}
+    NOTE: Ensure that base model form `{% include "scenarios.data_science.share:scen.input_path" %}prev_model` is correctly loaded, you are supposed to finetune the base model. 

rdagent/app/finetune/data_science/tpl/scenarios/data_science/proposal/exp_gen/prompts_v2.yaml

@@ -0,0 +1,6 @@
+task_gen:
+  system: |-
+    {% include "rdagent.scenarios.data_science.proposal.exp_gen.prompts_v2:task_gen.system" %}
+    NOTE: You MUST load base model form `{% include "scenarios.data_science.share:scen.input_path" %}prev_model`. Your main goal is to finetune it. 
+
+  

rdagent/app/finetune/llm/conf.py

@@ -0,0 +1,43 @@
+import os
+
+from pydantic_settings import SettingsConfigDict
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.core.conf import RD_AGENT_SETTINGS, ExtendedBaseSettings
+
+
+class LLMFinetuneScen(ExtendedBaseSettings):
+    model_config = SettingsConfigDict(env_prefix="FT_", protected_namespaces=())
+    scen: str = "rdagent.app.finetune.llm.scen.LLMFinetuneScen"
+    """
+    Scenario class for data science tasks.
+    - For Kaggle competitions, use: "rdagent.scenarios.data_science.scen.KaggleScen"
+    - For custom data science scenarios, use: "rdagent.scenarios.data_science.scen.DataScienceScen"
+    - For LLM finetune scenarios, use: "rdagent.app.finetune.llm.scen.LLMFinetuneScen"
+    - For Data science finetune scenarios, use: "rdagent.app.finetune.data_science.scen.DSFinetuneScen"
+    """
+
+    hypothesis_gen: str = "rdagent.app.finetune.llm.proposal.FinetuneExpGen"
+    """Hypothesis generation class"""
+
+    debug_timeout: int = 36000
+    """The timeout limit for running on debugging data"""
+    full_timeout: int = 360000
+    """The timeout limit for running on full data"""
+
+    coder_on_whole_pipeline: bool = True
+    enable_model_dump: bool = True
+    app_tpl: str = "app/finetune/llm/tpl"
+
+
+def update_settings(competition: str):
+    """
+    Update the RD_AGENT_SETTINGS with the values from LLM_FINETUNE_SETTINGS.
+    """
+    LLM_FINETUNE_SETTINGS = LLMFinetuneScen()
+    RD_AGENT_SETTINGS.app_tpl = LLM_FINETUNE_SETTINGS.app_tpl
+    os.environ["DS_CODER_COSTEER_EXTRA_EVALUATOR"] = '["rdagent.app.finetune.share.eval.PrevModelLoadEvaluator"]'
+    for field_name, new_value in LLM_FINETUNE_SETTINGS.model_dump().items():
+        if hasattr(DS_RD_SETTING, field_name):
+            setattr(DS_RD_SETTING, field_name, new_value)
+    DS_RD_SETTING.competition = competition

rdagent/app/finetune/llm/loop.py

@@ -0,0 +1,40 @@
+import asyncio
+from pathlib import Path
+
+import fire
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.app.finetune.llm.conf import update_settings
+from rdagent.core.utils import import_class
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.data_science.loop import DataScienceRDLoop
+
+
+def main(
+    model: str | None = None,
+    dataset: str | None = None,
+):
+    """
+    Parameters
+    ----------
+    dataset :
+        Dateset name, used for finetune.
+
+    Auto R&D Evolving loop for models finetune.
+    You can continue running a session by using the command:
+    .. code-block:: bash
+        dotenv run -- python rdagent/app/finetune/llm/loop.py --dataset shibing624/alpaca-zh
+    """
+    if not dataset:
+        raise Exception("Please specify dataset name.")
+
+    model_folder = Path(DS_RD_SETTING.local_data_path) / dataset / "prev_model"
+    if not model_folder.exists():
+        raise Exception(f"Please put the model path to {model_folder}.")
+    update_settings(dataset)
+    rd_loop: DataScienceRDLoop = DataScienceRDLoop(DS_RD_SETTING)
+    asyncio.run(rd_loop.run())
+
+
+if __name__ == "__main__":
+    fire.Fire(main)

rdagent/app/finetune/llm/prompts.yaml

@@ -0,0 +1,13 @@
+scenario_description: |-
+  ------Background of the scenario------
+  You are a world-class machine learning engineer. Your task is to finetune a model on the given dataset using QLoRA method.
+  ------Dataset Description------
+  {{ raw_description }}
+
+competition_background: |-
+  ## QLoRA Fine-Tuning
+  You are a world-class machine learning engineer and prompt engineer specializing in parameter-efficient fine-tuning of large language models using **QLoRA**. Your expertise includes 4-bit quantization, low-rank adaptation, and maximizing performance on GPU clusters. You are committed to building accurate, resource-efficient, and robust LLMs.
+
+  - **Fine-Tuning Method**: QLoRA (4-bit quantized LoRA)  
+  - **Training Dataset**:  
+    > {{ raw_description }}

rdagent/app/finetune/llm/proposal.py

@@ -0,0 +1,46 @@
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.core.proposal import ExpGen
+from rdagent.core.scenario import Scenario
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.data_science.experiment.experiment import DSExperiment
+from rdagent.scenarios.data_science.proposal.exp_gen.base import DSHypothesis, DSTrace
+from rdagent.scenarios.data_science.proposal.exp_gen.proposal import DSProposalV2ExpGen
+from rdagent.utils.agent.tpl import T
+
+
+class FinetuneExpGen(DSProposalV2ExpGen):
+    def gen(
+        self,
+        trace: DSTrace,
+    ) -> DSExperiment:
+        component_desc = T("scenarios.data_science.share:component_description_in_pipeline").r()
+
+        if (sota_exp_fb := trace.sota_experiment_fb()) is None:
+            sota_exp, fb_to_sota_exp = None, None
+        else:
+            sota_exp, fb_to_sota_exp = sota_exp_fb
+
+        if not isinstance(sota_exp, DSExperiment):
+            eda_output = None
+        else:
+            eda_output = sota_exp.experiment_workspace.file_dict.get("EDA.md", None)
+        scenario_desc = self.scen.get_scenario_all_desc(eda_output=eda_output)
+
+        # TODO: this is a over simplified version. More features will be added after more survey
+        sota_exp_desc = "No previous SOTA experiments available."
+        failed_exp_feedback_list_desc = "No previous experiments available."
+
+        return self.task_gen(
+            component_desc=component_desc,
+            scenario_desc=scenario_desc,
+            sota_exp_desc=sota_exp_desc,
+            sota_exp=sota_exp,
+            hypotheses=[
+                DSHypothesis(
+                    component="Model",
+                )
+            ],
+            pipeline=True,
+            failed_exp_feedback_list_desc=failed_exp_feedback_list_desc,
+            fb_to_sota_exp=fb_to_sota_exp,
+        )

rdagent/app/finetune/llm/scen.py

@@ -0,0 +1,87 @@
+from pathlib import Path
+
+from rdagent.app.data_science.conf import DS_RD_SETTING
+from rdagent.core.scenario import Scenario
+from rdagent.log import rdagent_logger as logger
+from rdagent.scenarios.data_science.scen import DataScienceScen
+from rdagent.scenarios.data_science.scen.utils import describe_data_folder_v2
+from rdagent.utils.agent.tpl import T
+
+
+class LLMFinetuneScen(DataScienceScen):
+    """LLMFinetuneScen Scenario"""
+
+    def __init__(self, competition: str) -> None:
+        self._download_data(competition=competition)
+        super().__init__(competition)
+        self._analysis_competition_description()
+
+    def _get_data_folder_description(self) -> str:
+        folder_desc = describe_data_folder_v2(
+            Path(DS_RD_SETTING.local_data_path) / self.competition, show_nan_columns=DS_RD_SETTING.show_nan_columns
+        )
+        return folder_desc
+
+    def _download_data(self, competition: str):
+        """
+        Download dateset from Hugging Face Hub
+
+        Parameters
+        ----------
+        - competition (str): Dateset ID, like "shibing624/alpaca-zh".
+        """
+        save_path = f"{DS_RD_SETTING.local_data_path}/{competition}"
+        if Path(save_path).exists():
+            logger.info(f"{save_path} already exists.")
+        else:
+            logger.info(f"Downloading {competition} to {save_path}")
+            try:
+                from huggingface_hub import snapshot_download
+
+                snapshot_download(
+                    repo_id=competition,
+                    repo_type="dataset",
+                    local_dir=save_path,
+                    local_dir_use_symlinks=False,
+                )
+            except ImportError:
+                raise ImportError(
+                    "Please install huggingface_hub first. "
+                    'You can install it with `pip install -U "huggingface_hub[cli]"`.'
+                )
+            except Exception as e:
+                logger.error(f"Error when downloading {competition}: {e}")
+                raise e
+
+    def _get_description(self):
+        if (fp := Path(f"{DS_RD_SETTING.local_data_path}/{self.competition}/README.md")).exists():
+            logger.info(f"{self.competition}/Found README.md, loading from local file.")
+            return fp.read_text()
+
+    def _get_direction(self):
+        return True
+
+    @property
+    def rich_style_description(self) -> str:
+        raise NotImplementedError
+
+    @property
+    def background(self) -> str:
+        background_template = T(".prompts:competition_background")
+        background_prompt = background_template.r(
+            raw_description=self.raw_description,
+        )
+        return background_prompt
+
+    def get_competition_full_desc(self) -> str:
+        return T(".prompts:scenario_description").r(
+            raw_description=self.raw_description,
+        )
+
+    def get_scenario_all_desc(self, eda_output=None) -> str:
+        """
+        eda_output depends on dynamic .md files from current workspace, not fixed.
+        """
+        return T(".prompts:scenario_description").r(
+            raw_description=self.raw_description,
+        )

rdagent/app/finetune/llm/tpl/components/coder/data_science/pipeline/prompts.yaml

@@ -0,0 +1,71 @@
+pipeline_coder:
+  system: |-
+    You are a world-class ML engineer specializing in parameter-efficient LLM fine-tuning with QLoRA.
+    Design a single-file `main.py` that:
+      • Loads a pretrained model from `./workspace_input/prev_model`.  
+      • Attaches 4-bit LoRA adapters, runs fine-tuning, evaluates on the validation set.  
+      • Uses `print()` for progress and debug output (no `logging` or progress bars).  
+      • Wraps file reads in `try/except` only to catch missing files—do not suppress other errors.  
+      • Hardcodes all paths and hyperparameters—no CLI parsing.  
+      • Is directly executable via `python main.py`.
+
+    ## Task Description
+    {{ task_desc }}
+
+    ## The runtime environment your code will running on
+    {{ runtime_environment }}
+
+    {% if queried_former_failed_knowledge|length != 0 %}
+    --------- Previous Failed Attempts ---------
+    {% for former_failed_knowledge in queried_former_failed_knowledge %} Attempt {{ loop.index }}:
+    =====Code:=====
+    {{ former_failed_knowledge.implementation.all_codes }}
+    =====Feedback:=====
+    {{ former_failed_knowledge.feedback }}
+    {% endfor %}
+    {% endif %}
+
+    ## Guidelines
+    1. Ensure that the dataset is loaded strictly from `{% include "scenarios.data_science.share:scen.input_path" %}`, following the exact folder structure described in the **Data Folder Description**, and do not attempt to load data from the current directory (`./`).
+    2. You should avoid using logging module to output information in your generated code, and instead use the print() function.
+    3. You should be very careful about the try catch block in your code. You may use it to handle missing files in data reading, but you should not use it to handle the errors in your code. Especially use it to bypass the errors in your code. Directly solve the errors in your code instead of using try catch block to bypass them.
+    4. Initialize random seeds and specify device (`cpu`/`cuda`) for reproducibility.  
+    5. Ensure `main.py` runs end-to-end: training → validation → save `./scores.csv`.  
+    6. Save finetuned adapter to `./models/` directory.
+    7. When run the code again, the code will skip finetune process and directly load the finetuned adapter from `./models/` directory.
+
+    {% if enable_debug_mode %}
+    Your code will be executed in a debug mode with following command: 
+    ```bash
+    python main.py --debug
+    ```
+    In debug mode, you should only sample smallest possible subset from the training data and run the minimum epochs to quickly test the correctness of the code.
+    In debug mode, you should implement a timer to measure the time taken for your debug configuration and estimate the time required for the full run.
+    For example, you can sample smallest possible subset from the training data and run for one epoch, then the full run with ten epochs will take one hundred times the time taken for the debug run. The scale is calculated by yourself depending on the data sampling and epoch number you choose. If your full run enables early stopping, the scale should be smaller considering the early stopping will stop the training earlier than the full epochs.
+    You should sample the data after train valid split. When you split the data after sampling, you might get a class with only one sample which might cause the split strategy to fail.
+    Your debug code should run exactly the same as the full run, except for the data sampling and epoch number, to ensure the correctness of the code.
+    You should print total time and estimated time in standard output using print function in the following schema:
+    === Start of Debug Information ===
+    debug_time: time_taken_for_debug_run_in_seconds (e.g., 'debug_time: 10.0')
+    estimated_time: estimated_time_for_full_run_in_seconds (e.g., 'estimated_time: 100.0')
+    === End of Debug Information ===
+    User will use the following code to match: re.search(r"(.*?)=== Start of Debug Information ===(.*)=== End of Debug Information ===", stdout, re.DOTALL).groups()[1]
+    Notice, data sampling should only be applied in debug mode. Always use the full data in the full run!
+    Example code:
+    ```python
+    if args.debug:
+      sample_size = int(0.01 * len(train_dataset))  # 1% for debug
+    else:
+      sample_size = len(train_dataset)
+    ```
+    {% endif %}
+
+    ## Output Format
+    {% if out_spec %}
+    {{ out_spec }}
+    {% else %}
+    Please response the full runable code in the following json format. Here is an example structure for the JSON output:
+    {
+        "code": "The Python code as a string."
+    }
+    {% endif %}

rdagent/app/finetune/llm/tpl/scenarios/data_science/prompts.yaml

@@ -0,0 +1,10 @@
+system: |-
+    You are a world-class ML engineer specializing in parameter-efficient LLM fine-tuning with QLoRA.
+    Design a single-file `main.py` that:
+      • Loads a pretrained model from `./workspace_input/prev_model`.  
+      • Attaches 4-bit LoRA adapters, runs fine-tuning, evaluates on the validation set.  
+      • Uses `print()` for progress and debug output (no `logging` or progress bars).  
+      • Wraps file reads in `try/except` only to catch missing files—do not suppress other errors.  
+      • Hardcodes all paths and hyperparameters—no CLI parsing.  
+      • Is directly executable via `python main.py`.
+   

rdagent/app/finetune/llm/tpl/scenarios/data_science/proposal/exp_gen/prompts_v2.yaml

@@ -0,0 +1,82 @@
+scenario_problem:
+  system: |-
+    You are a world-class machine learning and prompt engineer specializing in parameter-efficient fine-tuning of large language models using QLoRA (4-bit quantized LoRA adapters).  
+    Each iteration (trace) represents one training run or adapter update. If an iteration’s validation metric exceeds the current best, it becomes the new SOTA adapter; otherwise it is a failed experiment.
+    Your task is to analyze the scenario (and SOTA, if given) and identify a concise list of **2–3 Key Challenges** that most critically limit fine-tuning performance.
+
+    ### Core Analysis Dimensions
+    1. **Adapter-Model Alignment**  
+       Compare current LoRA adapter configuration against model capacity and task complexity.  
+    2. **Optimization Dynamics**  
+       Identify where training diverges or plateaus (e.g. LR too high/low, quantization noise).  
+    3. **Data-Model Coherence**  
+       Spot mismatches between the dataset’s characteristics and model input preprocessing or sequence length.  
+
+    ## Key Challenges / Core Problems
+    Categorize each challenge as one of:
+    - **Data-Driven Challenge**  
+      Issues in dataset size, domain mismatch, label noise, sequence length distribution, etc.
+    - **Model-Optimization Challenge**  
+      LoRA rank selection, quantization artifacts, learning rate schedule, gradient accumulation, etc.
+
+    ### For Each Challenge
+    1. Be **specific** and **actionable**.  
+    2. Focus on **methodological** aspects, not trivial bugs.  
+    3. Directly tie to improving the **target metric**.  
+    4. If no SOTA exists, include at least one challenge that guides building a minimal baseline adapter.
+
+    {% if task_output_format is not none %}
+    {% endif %}
+
+task_gen:
+  system: |-
+    You are an expert in LLM fine-tuning with QLoRA. Each iteration applies a specific hypothesis to improve the current adapter (SOTA) or establish an initial adapter.
+
+    **Inputs**:
+      - Scenario: base model, task, data path, evaluation metric  
+      - Current SOTA adapter & feedback (if any)  
+      - Proposed Hypothesis  
+      - Failed runs feedback (if any)
+
+    **Your task**: Outline a conceptual plan for `main.py` that implements the Proposed Hypothesis.
+
+    **Standards**:
+      - Run via `python main.py` with no CLI args; configs are hard-coded.  
+      - No code or pseudo-code—describe each step in plain language.  
+      - Do **not** use progress bars.  
+      - Do **not** infer test indices from sample files.
+
+    **Sketch**:
+      1. **Load Data**  
+         - Read train/validation files from the given data path.  
+         - Tokenize or preprocess inputs for the model.
+
+      2. **Initialize Model & Adapter**  
+         - Load the base LLM.  
+         - Attach a QLoRA adapter.
+
+      3. **Train with Hypothesis**  
+         - Apply the hypothesis change (e.g., modify learning schedule, adapter config).  
+         - Train and validate iteratively.
+
+      4. **Validate & Record**  
+         - Compute the metric on validation set.  
+         - Save results to `scores.csv` (with adapter name and “ensemble”).
+
+      5. **Generate Submission**  
+         - Write `submission.jsonl` or `.csv` matching the competition format exactly.
+
+    **Key Reminders for Developer**:
+      - Hard-code all paths; do not rely on sample files for indices.  
+      - Ensure tokenizer and model names match.  
+      - Validate output formats for `scores.csv` and `submission`.  
+      - Handle file I/O robustly (e.g., zipped data).
+
+    {% if task_output_format is not none %}
+    ## [Partial Response Format 1] Task Output Format:
+    {{ task_output_format }}
+    Your final output should strictly adhere to the following JSON format. 
+    {
+      "task_design": ---The dict corresponding to task output format---,
+    }
+    {% endif %}

rdagent/app/finetune/llm/tpl/scenarios/data_science/scen/prompts.yaml

@@ -0,0 +1,18 @@
+competition_description_template:
+  system: |-
+    You are a data science assistant that extracts structured information from unstructured text.
+    The user will provide you a description of an LLM fine-tuning project, and you need to extract specific details from it.
+    For the dataset, the user has already reviewed and provided any additional context—include that information in your response.
+    Please answer in JSON format with the following schema:
+    {
+      "Task Type":       "The type of fine-tuning task, e.g., 'Question Answering', 'Text Classification', 'Summarization', 'Translation', 'Code Generation'",
+      "Data Type":       "The type of data used for fine-tuning, e.g., 'Text (Natural Language)', 'Code', 'Multimodal', 'Dialogue'",
+      "Brief Description": "A concise summary of the fine-tuning project and its objectives",
+      "Dataset Description": "A description of the dataset as organized in the Processed Data folder: list files, formats, sizes, and any pre-processing steps applied, reconciled with contextual details from the project description",
+      "Training Specifications": "Details of the fine-tuning setup, including base model name, number of epochs, batch size, learning rate, optimizer, and any scheduler or early-stopping rules",
+      "Output Format":   "The expected model output format per sample (e.g., single label, probability distribution over N classes, generated text sequence)",
+      "Channels per Sample": "An integer indicating output dimensionality per example (e.g., 1 for single regression value, N for N-class probabilities, variable for generated text)",
+      "Evaluation Metric Description": "A precise explanation of how model performance is measured, including the formula or procedure used",
+      "Metric Name":     "The name of the evaluation metric (e.g., 'Accuracy', 'ROUGE-L', 'BLEU', 'F1'), please only choose one metric name",
+      "Metric Direction": true or false  // true if higher is better, false if lower is better
+    }

rdagent/app/finetune/share/eval.py

@@ -0,0 +1,52 @@
+from pathlib import Path
+
+from rdagent.components.coder.CoSTEER.evaluators import (
+    CoSTEEREvaluator,
+    CoSTEERSingleFeedback,
+)
+from rdagent.core.experiment import FBWorkspace, Task
+from rdagent.core.scenario import Scenario
+from rdagent.utils.agent.tpl import T
+from rdagent.utils.agent.workflow import build_cls_from_json_with_retry
+
+
+class PrevModelLoadEvaluator(CoSTEEREvaluator):
+    """This evaluator checks whether the code actually loads a model from `prev_model`."""
+
+    def __init__(self, scen: Scenario):
+        super().__init__(scen)
+
+    def evaluate(
+        self, target_task: Task, implementation: FBWorkspace, gt_implementation: FBWorkspace, *args, **kwargs
+    ) -> CoSTEERSingleFeedback:
+        data_source_path = T("scenarios.data_science.share:scen.input_path").r()
+        prev_model_dir = Path(data_source_path) / "prev_model"
+
+        # 1) Inspect the code itself for references to prev_model loading
+        code_str = implementation.file_dict["main.py"]
+        code_contain_prev = "prev_model" in code_str
+        print(f"Code references prev_model: {code_contain_prev}")
+        if not code_contain_prev:
+            err = (
+                "No evidence found that your code loads a model from `prev_model`. "
+                "Please check that you are calling the correct load function "
+                f"and pointing it to the `{prev_model_dir}` directory."
+            )
+            return CoSTEERSingleFeedback(
+                execution=err,
+                return_checking=err,
+                code=err,
+                final_decision=False,
+            )
+
+        system_prompt = T(".prompts:prev_model_eval.system").r()
+        user_prompt = T(".prompts:prev_model_eval.user").r(
+            code=implementation.all_codes,
+        )
+
+        csfb = build_cls_from_json_with_retry(
+            CoSTEERSingleFeedback,
+            system_prompt=system_prompt,
+            user_prompt=user_prompt,
+        )
+        return csfb

rdagent/app/finetune/share/prompts.yaml

@@ -0,0 +1,23 @@
+prev_model_eval:
+  system: |-
+    You are a data scientist tasked with evaluating code generation. 
+
+    You will receive the following information:
+    - The implemented code
+
+    Focus on these aspects:
+    - Check if the code load the model in the "prev_model/" subfolder.
+
+    Please respond with your feedback in the following JSON format and order
+    ```json
+    {
+        "execution": "Describe whether the code executed successfully. Include any errors or issues encountered, and append all error messages and full traceback details without summarizing or omitting any information. ."
+        "return_checking": "Detect whether the model is loaded from 'prev_model/' subfolder and finetune is prepared based on prev model.",
+        "code": "The code has explicity load the model from 'prev_model/' subfolder and prepares finetune based on prev model.",
+        "final_decision": <true or false in boolean type; only return true when ensuring that the code loads the model from 'prev_model/' subfolder and prepares finetune based on prev model.>
+    }
+    ```
+
+  user: |-
+    ------------ The implemented code ------------ 
+    {{code}}

rdagent/components/coder/data_science/conf.py

@@ -22,6 +22,8 @@ class DSCoderCoSTEERSettings(CoSTEERSettings):
     max_seconds_multiplier: int = 4
     env_type: str = "docker"
     # TODO: extract a function for env and conf.
+    extra_evaluator: list[str] = []
+    """Extra evaluators to use"""
 
     extra_eval: list[str] = []
     """

rdagent/components/coder/data_science/pipeline/__init__.py

@@ -141,6 +141,8 @@ class PipelineCoSTEER(DSCoSTEER):
         eval_l = [PipelineCoSTEEREvaluator(scen=scen)]
         if DS_RD_SETTING.enable_model_dump:
             eval_l.append(ModelDumpEvaluator(scen=scen, data_type="sample"))
+        for evaluator in settings.extra_evaluator:
+            eval_l.append(import_class(evaluator)(scen=scen))
 
         for extra_eval in DSCoderCoSTEERSettings().extra_eval:
             kls = import_class(extra_eval)

rdagent/components/coder/data_science/pipeline/prompts.yaml

@@ -216,7 +216,6 @@ pipeline_coder:
     {% endif %}
     {% endif %}
 
-
 pipeline_eval:
   system: |-
     {% include "scenarios.data_science.share:scen.role" %}

rdagent/core/conf.py

@@ -102,5 +102,8 @@ class RDAgentSettings(ExtendedBaseSettings):
     def is_force_subproc(self) -> bool:
         return self.subproc_step or self.get_max_parallel() > 1
 
+    # Template:
+    app_tpl: str | None = None  # for application to override the default template, example: "app/fintune/tpl"
+
 
 RD_AGENT_SETTINGS = RDAgentSettings()

rdagent/log/storage.py

@@ -107,7 +107,7 @@ class FileStorage(Storage):
     def truncate(self, time: datetime) -> None:
         for file in self.path.glob("**/*.pkl"):
             timestamp = datetime.strptime(file.stem, "%Y-%m-%d_%H-%M-%S-%f").replace(tzinfo=timezone.utc)
-            if timestamp > time:
+            if timestamp > time.replace(tzinfo=timezone.utc):
                 file.unlink()
 
         _remove_empty_dir(self.path)

rdagent/scenarios/data_science/scen/utils.py

@@ -294,7 +294,12 @@ class FileTreeGenerator:
     Smart file tree generator with symlink handling and intelligent truncation.
     """
 
-    def __init__(self, max_lines: int = 200, priority_files: Set[str] = None, hide_base_name: bool = True):
+    def __init__(
+        self,
+        max_lines: int = 200,
+        priority_files: Set[str] = None,
+        hide_base_name: bool = True,
+    ):
         """
         Initialize the file tree generator.
 
@@ -566,19 +571,26 @@ class DataFolderDescriptor:
                     file_name = str(fn)
 
                 try:
-                    if fn.suffix == ".csv":
-                        out.append(preview_csv(fn, file_name, simple=simple, show_nan_columns=show_nan_columns))
-                    elif fn.suffix == ".json":
-                        out.append(preview_json(fn, file_name))
-                    elif fn.suffix == ".parquet":
-                        out.append(preview_parquet(fn, file_name, simple=simple, show_nan_columns=show_nan_columns))
-                    elif fn.suffix in plaintext_files:
-                        if get_file_len_size(fn)[0] < 30:
-                            with open(fn) as f:
-                                content = f.read()
-                                if fn.suffix in code_files:
-                                    content = f"```\n{content}\n```"
-                                out.append(f"-> {file_name} has content:\n\n{content}")
+                    if "prev_model" in file_name:
+                        # NOTE: for finetune model.
+                        if fn.suffix == ".py" and "test" not in file_name:
+                            out.append(f"### {file_name}:")
+                            out.append(fn.read_text(encoding="utf-8"))
+                    else:
+                        if fn.suffix == ".csv":
+                            out.append(preview_csv(fn, file_name, simple=simple, show_nan_columns=show_nan_columns))
+                        elif fn.suffix == ".json":
+                            out.append(preview_json(fn, file_name))
+                        elif fn.suffix == ".parquet":
+                            out.append(preview_parquet(fn, file_name, simple=simple, show_nan_columns=show_nan_columns))
+                        elif fn.suffix in plaintext_files:
+                            if get_file_len_size(fn)[0] < 30:
+                                with open(fn) as f:
+                                    content = f.read()
+                                    if fn.suffix in code_files:
+                                        content = f"```\n{content}\n```"
+                                    out.append(f"-> {file_name} has content:\n\n{content}")
+
                 except Exception as e:
                     out.append(f"-> {file_name}: Error reading file - {str(e)[:100]}")
 

rdagent/utils/agent/tpl.py

@@ -11,10 +11,11 @@ from typing import Any
 import yaml
 from jinja2 import Environment, FunctionLoader, StrictUndefined
 
+from rdagent.core.conf import RD_AGENT_SETTINGS
 from rdagent.log import rdagent_logger as logger
 
 DIRNAME = Path(__file__).absolute().resolve().parent
-PROJ_PATH = DIRNAME.parent.parent
+PROJ_PATH = DIRNAME.parent.parent  # rdagent
 
 
 def get_caller_dir(upshift: int = 0) -> Path:
@@ -43,11 +44,22 @@ def load_content(uri: str, caller_dir: Path | None = None, ftype: str = "yaml")
     # load file_path with priorities.
     if path_part.startswith("."):
         file_path_l = [caller_dir / f"{path_part[1:].replace('.', '/')}.{ftype}"]
+        if RD_AGENT_SETTINGS.app_tpl is not None:
+            file_path_l.insert(0, PROJ_PATH / RD_AGENT_SETTINGS.app_tpl / file_path_l[0].relative_to(PROJ_PATH))
     else:
         file_path_l = [
             Path(path_part.replace(".", "/")).with_suffix(f".{ftype}"),
             (PROJ_PATH / path_part.replace(".", "/")).with_suffix(f".{ftype}"),
         ]
+        # NOTE: for application's template to override the default template
+        if RD_AGENT_SETTINGS.app_tpl is not None:
+            file_path_l.insert(
+                0, (PROJ_PATH / RD_AGENT_SETTINGS.app_tpl / path_part.replace(".", "/")).with_suffix(f".{ftype}")
+            )
+            # NOTE: when we can both load tpl from tpl; to avoid recursive extension.
+            # e.g. we want app_tpl/a.b.c extend rdagent/a.b.c;  so we allow specifying in a upper
+            #        level. for example,  rdagent.a.b.c;
+            file_path_l.insert(0, (PROJ_PATH.parent / path_part.replace(".", "/")).with_suffix(f".{ftype}"))
 
     for file_path in file_path_l:
         try: